GavvioBot · Gavvio

GavvioBot is the program Gavvio uses to read public web pages at its customers' request. A customer asks, for example, for an analysis of the pages that rank on Google for a keyword of their business, and GavvioBot reads those pages once.

How it identifies itself

On every request the user agent is exactly GavvioBot/1.0 (+https://gavvio.com/bot). It never presents itself as a browser.
Its robots.txt token is GavvioBot.
Requests are signed (HTTP Message Signatures, RFC 9421, in the Web Bot Auth profile). The public key is at app.gavvio.com/.well-known/http-message-signatures-directory.
The addresses it connects from are listed at gavvio.com/bot/ips.json.

What it does

It reads a site's robots.txt before any page and obeys it (RFC 9309), including Crawl-delay.
It makes one request at a time to a site, with at least two seconds between requests. When a site answers 429 or 503, it waits for as long as it is asked to.
It keeps a page it read for 24 hours, and after that asks whether the page changed (ETag and Last-Modified) before downloading it again.
It reads text only. It does not download images, video or fonts.
When a page has no text without running scripts, it may open it once in an automated browser, with the same user agent and no cookies from other pages.
Each Gavvio customer has a daily page limit.

What it never does

It does not sign in, fill in forms or click anything.
It does not solve or work around challenges, CAPTCHAs or blocks. When a site refuses it, the page is recorded as refused.
It does not change its user agent, rotate addresses or use proxy networks.
It does not read social network pages. Social posts are read only through each platform's official API.
It does not pay for content automatically.

Use by artificial intelligence

Gavvio does not train AI models on the pages it reads. A page's text may be sent to a language model to produce an analysis for the customer who asked for the read. GavvioBot reads and honours the preferences a site states:

Content-Signal: ai-input=no in robots.txt, or tdm-reservation: 1 (header or meta tag): the page may be read, but its text is not sent to any model.
Content-Signal: ai-train=no, or noai in the robots meta tag or the X-Robots-Tag header: the text is never stored in a customer's knowledge base nor used for training.

How to block it or slow it down

To block GavvioBot on the whole site, add to robots.txt:

User-agent: GavvioBot
Disallow: /

To ask for a longer interval between requests (in seconds):

User-agent: GavvioBot
Crawl-delay: 10

The change applies from the next read of robots.txt, which happens at most once a day per site.

How long we keep it

The text of a page is deleted within 30 days. The record that the read happened (address, date and result) is deleted within 12 months.

Contact

If GavvioBot caused a problem on your site, or if you have a question, write to[email protected]. Include the site's address and, if you can, the time of the requests.