PolarisNewsBot · Polaris News

Requests identifying themselves as PolarisNewsBot/0.1 (+https://polarisnews.ca/about/bot) come from Polaris News, a Canadian news publication that reports from primary sources. The short version: we read a small, fixed list of feeds and pages, we obey robots.txt, we ask only for what has changed, and we will remove your site on request without argument.

How to make it stop

Two ways, and both work.

Add this to your robots.txt and we will stop on our next pass, which is within the hour:

User-agent: PolarisNewsBot
Disallow: /

Or write to bot@polarisnews.ca with the hostname and we will remove the source from our registry entirely. You do not need to give a reason, and we will confirm when it is done.

What it does

It does not crawl. Polaris reads a registry of 83 specific addresses: feeds, release indexes and data endpoints. Each one was admitted deliberately by a person, and each is published in full. It does not wander from them: nothing it reads causes it to fetch a page it was not given.
One exception, and it is operator-driven. When a registered address turns out to be a newsroom page rather than a feed, an operator can ask Polaris to find the feed behind it. That makes up to fifteen requests to that site, once, following links from the page we were already given, which is the same thing a person would do by hand. It never runs on its own and never leaves the host it was pointed at.
It asks for changes, not copies. Once we have seen an address, every later request carries If-None-Match and If-Modified-Since from the previous visit, whichever of them you gave us. If nothing has changed, your server answers 304 and sends no body, so most of our requests cost you a response header. The first request to an address necessarily carries neither.
It visits on a schedule, not on demand. The default is once every 15 minutes per address, and most are slower than that. Nothing a reader does on our site triggers a request to yours.
It backs off when it is unwelcome. After repeated failures an address is polled exponentially less often, up to a ceiling of once a day. A server that returns errors is not retried at full rate.
It reads one image per item, at most, and only when image capture is switched on, which it is not by default. We do not download an image unless a person has first established that your terms permit us to publish it.

What it does not do

It does not ignore robots.txt. Compliance is switched on, and turning it off requires a separate confirmation in our admin interface that says exactly what it means.
It does not submit forms, run JavaScript, or log in.
It does not try to reach anything it was not explicitly given.
It does not train a model on what it reads. Retrieved material is used to write and to verify an article, and is kept as evidence for that article.
It does not republish your reporting. Polaris reports from documents issued by the organizations responsible for events. What we publish cites the document and links to it.

If it misbehaved

If you have seen request rates that do not match the description above, we would genuinely like to know. Send the log lines to bot@polarisnews.ca. A bot behaving differently from its own documentation is a bug we would want to fix, and your log is better evidence of it than anything we can see from our side.