Robots.txt is a file that exists on the basis directory of each Web page and may be used to instruct search engines like google and yahoo on which directories/data files of the website they are able to crawl and incorporate in their index. I'm happy to get messages from institutions, https://news.rafeeg.ae