Robots.txt is really a file that exists on the foundation directory of each website and may be used to instruct search engines like yahoo on which directories/files of the website they can crawl and involve of their index. No matter if you're employed on Web optimization yourself, delegate it to https://news.rafeeg.ae