Robots.txt is a file that exists on the basis directory of each website and may be used to instruct search engines like yahoo on which directories/data files of the website they will crawl and consist of of their index. The content material of the learned website page, and also the https://technik.forum