User-agent: sentibot Sitemap: https://www.consultancy.eu/sitemap_index.xml Crawl-delay: 20 #wait 20 seconds before requesting the next page User-agent: * Sitemap: https://www.consultancy.eu/sitemap_index.xml Crawl-delay: 5 #wait 5 seconds before requesting the next page