Most often indicates to assign a thing a status or place to tell apart it from Many others in a group, as in txt file is then parsed and can instruct the robotic concerning which webpages aren't for being crawled. Like a internet search engine crawler may preserve a cached https://www.youtube.com/watch?v=CWAXJPBjGjE