User-agent: ia_archiver Disallow: / User-agent: CCBot Disallow: / User-agent: Scrapy Disallow: /