Bot information
The Sitehistory.fyi crawler
We are a snapshot crawler that loads the pages a site owner has listed, once per schedule (usually once a day), like a desktop browser and like a phone.
We only visit domains whose owner has verified them with us, through Google Search Console, a confirmed company email address, or a DNS TXT record. See how verification works.
What we do
- Typical volume is your number of URLs x 2 page loads per day.
- We do not log in or submit forms. Cookie banners: by default we keep them and give no consent, so we capture what a first-time visitor sees. Only if the site owner chose "accept" or "decline" for a job do we click the banner's "accept all" or "reject all / only necessary" control; the choice "remove" hides the banner without consenting. No other click is ever made.
- We read
robots.txtof the page's own host before every scheduled run (the answer is cached for 24 hours). A job thatrobots.txtdisallows for our token is not captured and the run records the reason. Only the site owner can override this for a verified domain, in the property settings. - We abort analytics and ad measurement requests, so we do not show up in your numbers.
How to recognise us
| Signal | Value |
|---|---|
| User-agent token | Sitehistory.fyi/0.1 (+https://sitehistory.fyi/bot)A normal Chrome user agent ending in this token. |
| robots.txt token | SiteHistoryWe follow the rules in |
| Static IP addresses |
|
| Optional secret header | X-Snapshot-Token: <your project token> |
How to allow us
The safest rule is always our IP addresses and the secret header together. Step by step instructions per protection vendor are on the allowlisting page.
How to stop us
- Remove the verification record or delete the property: job runs stop at once.
- Or block our IP addresses.
Googlebot profiles: what they are not
If a site owner switches them on, they send Google's published Googlebot user-agent strings from our IP address. They show what the site's own code sends to a request with a Googlebot user agent (dynamic rendering, prerender services, user-agent switches, accidental cloaking). They cannot show what real Googlebot gets where the site or its CDN verifies Googlebot by IP or reverse DNS. Such requests may also trip "fake Googlebot" protections even on an allowlisted IP. Every such job run carries this note.
The wording above is the note stored with every Googlebot-profile job run, so you can quote it when you read the history.
Abuse contact
Report abuse or ask us to stop visiting your site: abuse@sitehistory.fyi (our legal details are in the imprint).