Sounds like you’re mixing up the Meta Robots tag with the Robots.txt. The Robots.txt resides in the root of your directory and directs search engines about what to do about various pages on your site before they ever request one from the server. The Meta Robots tag lives in the Head section of each page and only pertains to the links and content on that page. Here’s a quick primer on how to use Robots.txt. If one isn’t present, the search engine will receive a 404 error from the server when they request it, which is why we suggest putting one in the root directory even if it is blank. Yes, the default behavior of the search engines is to be crawled, links followed and pages indexed.