0 votes
in Web Development by (130 points)

1 Answer

0 votes
by (220 points)

Basic format:

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

Blocking all web crawlers from all content

User-agent: * 
Disallow: /

Allowing all web crawlers access to all content

User-agent: * 
Disallow: 

Blocking a specific web crawler from a specific folder

User-agent: Googlebot 
Disallow: /example-subfolder/
...