#this disallows all spiders from everything

User-agent: *
Disallow: /