robots.txt
robots.txt
Specifies to search robots what catalogs to take for indexing should not be. If it is empty or does not exist, then everything can be taken.
Details about robots.txt
Specifies to search robots what catalogs to take for indexing should not be. If it is empty or does not exist, then everything can be taken.
Example
User-agent: *Disallow:/news
# Prohibit all robots from indexing links,
# That start with/news User-agent: StackRambler
User-agent: Aport
Disallow:/eng
Disallow:/news
#Rambler and Aport to forbid indexing of links,
# That start with/news and/eng User-agent: Yandex
Disallow:
#Yandex alloud all.
FILE GENERATOR of ROBOTS.TXT
Analyze robots.txt
- google.com/webmasters/tools/robots-t... - Google, Robots.txt file verification tool
- websiteplanet.com/ru/webtools/robots... - Website Planet
Webmasters
- webmaster.yandex.ru/tools/robotstxt/... - Analyze robots.txt - Here you can find out what's wrong with us, if we have something wrong in the robots.txt file, learn more about robots.txt.
