Social networking Influence: Well known traits on social websites can spill more than into search engines like google, affecting the look for volume of related keywords. txt file is then parsed and can instruct the robotic as to which internet pages aren't to be crawled. Being a search engine crawler https://www.youtube.com/watch?v=CWAXJPBjGjE