User-agent: * Disallow: /supervisor-contact-list/ # Block all web crawlers User-agent: * Disallow: / # Optional: block some known spammy bots explicitly User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Yandex Disallow: / User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / User-agent: msnbot Disallow: / User-agent: bingbot Disallow: /