Dignity nobility high quality condition highness standing course caste status superiority priority manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robot concerning which pages are certainly not for being crawled. For a search engine crawler may well https://www.youtube.com/watch?v=7cBHAtTOxoI