ScanoBot crawler policy

Who we are

ScanoBot is the web crawler of Scano Labs, Inc., the company behind Scano, a media-monitoring and media-relations platform. It identifies itself on every request with this User-Agent string:

User-Agent: ScanoBot/1.0 (+https://scano.me/bot; bot@scano.me)

A request that claims to be ScanoBot but does not carry this exact string is not ours. If you see one, please tell us at bot@scano.me.

What ScanoBot fetches

We build a directory of journalists: who writes where, for media-relations software. To do that, ScanoBot reads from news outlets:

• public article pages, only to read the byline, title and publish date;
• author and staff pages, and the directories that link to them;
• sitemaps and RSS feeds, to find those pages.

We keep only professionally published details: the journalist's name, the outlet, the public author page, recent article titles, URLs and dates, and the work contact details the outlet itself publishes on that author page. We do not store article text, images or headshots.

ScanoBot does not log in, create accounts, fill forms or solve CAPTCHAs. It does not bypass login walls or paywalls and does not use headless browsers.

How often

• at most one request every 2 seconds per host, and slower when your robots.txt sets a larger Crawl-delay;
• conditional requests (If-None-Match / If-Modified-Since), so unchanged pages cost you almost nothing;
• retries only on 429 and 503, honouring Retry-After.

If your site answers with a bot challenge or blocks the connection, we stop visiting it and never try to get around the block.

How to block or opt out

ScanoBot obeys the Robots Exclusion Protocol (robots.txt) for the token ScanoBot, falling back to the * rules. To exclude it from your whole site:

User-agent: ScanoBot
Disallow: /

To keep it out of one section only, or to slow it down:

User-agent: ScanoBot
Disallow: /staff/
Crawl-delay: 10

Changes to robots.txt take effect within 24 hours.

Journalists: to have your entry removed, or to stop us collecting it, e-mail bot@scano.me with the URL of your author page. The removal applies across all Scano products and is recorded so a later crawl does not re-create the entry. Outlets may ask for all their staff entries to be removed the same way.

Contact

Crawler questions, opt-out and erasure requests: bot@scano.me. We answer within 5 business days.