User-agent: Googlebot Sitemap: https://www.mariecurie.org.uk/sitemap.xml Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: bingbot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Googlebot-Mobile Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Googlebot-News Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Googlebot-Video Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Googlebot-Image Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Mediapartners-Google Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: AdsBot-Google Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: msnbot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: msnbot-media Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: facebookexternalhit Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: MJ12bot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Twitterbot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: LinkedInBot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Pinterest Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: bitlybot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Feedfetcher-Google Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-Agent: SemrushBot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-Agent: SemrushBot-SA Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: Yahoo! Slurp Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: DuckDuckBot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: AhrefsBot Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-agent: AhrefsSiteAudit Disallow: /episerver/CMS/* Disallow: /Util/* Disallow: /episerver/Find/ User-Agent: W3C-checklink Disallow: User-Agent: iisbot Disallow: User-agent: Screaming Frog SEO Spider Disallow: # All undefined bots - Disallow website Crawl. User-agent: * Disallow: / # Version 3.3