User-agent: * Allow: / # Allow major search engines to access everything User-agent: Googlebot User-agent: Bingbot User-agent: Yahoo User-agent: Baiduspider Allow: / # Sitemaps Sitemap: https://www.joinsafedeal.com/sitemap.xml # Crawl-delay directive to prevent servers from being overloaded Crawl-delay: 5