|
|
Create robots.txtrobots.txt Specifies to search robots what catalogs to take for indexing should not be. If it is empty or does not exist, then everything can be taken.
Details about robots.txt
ExampleUser-agent: *Disallow:/news # Prohibit all robots from indexing links, # That start with/news User-agent: StackRambler User-agent: Aport Disallow:/eng Disallow:/news #Rambler and Aport to forbid indexing of links, # That start with/news and/eng User-agent: Yandex Disallow: #Yandex alloud all. FILE GENERATOR of ROBOTS.TXT | For a web master What are looking for? Site traffic statistics META tags .htaccess CHMOD Mail Form Secure Email Colors in HTML Determinant of download speed Builder RSS stream in JavaScript Encrypt HTML in javascript |