REGOVBot

About the RE:GOV crawler

REGOVBot visits a government website only when someone asks RE:GOV to reimagine it. Here is exactly what it fetches, the rules it follows, and how a site can block it or ask for a takedown.

Last updated October 28, 2026. Questions: GitHub issues or rmbergonia.com.

01Who it is
REGOVBot is the crawler of RE:GOV, an independent, non-commercial civic-tech project. It identifies itself in every request with this user agent:
```
Mozilla/5.0 (compatible; REGOVBot/1.0; +https://regov.app/bot) Chrome/120
```
It runs only when a person submits a government address on regov.app. It does not crawl the web on its own, and it does not index sites for search or train models.
02What it fetches
- Public pages of the one site requested: up to 30 pages, at most three links deep, about one request a second.
- Citizen’s Charter PDFs linked from those pages, and the WordPress REST API when the site exposes it.
- No logins, no forms, no search boxes. It never solves a captcha or works around bot protection; when it meets one, it stops.
- RE:GOV accepts at most 7 new sites a day in total, and does not crawl the same site twice within 24 hours.
03robots.txt
REGOVBot follows robots.txt as set out in RFC 9309: rules for REGOVBot if the file names it, otherwise the rules for every crawler (*), with Allow and Disallow, wildcards, and Crawl-delay (up to 30 seconds). If robots.txt cannot be read because the server errors, it stays out.
If robots.txt asks REGOVBot, or every crawler, to stay away from the whole site, RE:GOV does not rebuild that site at all, not even from the Wayback Machine. To block only REGOVBot:
```
User-agent: REGOVBot
Disallow: /
```
04Terms of use
Before it collects any page, REGOVBot opens the site’s terms of use, legal notice or disclaimer linked from the homepage, and Muse Spark reads it. If the terms forbid automated access (crawling, scraping, data mining) or republishing the content without permission, the run stops there and the person who asked sees the clause that applies.
A plain “all rights reserved” line, or a request to credit the source, does not stop a run: RE:GOV links every rebuilt page to its original. Works of the Philippine government carry no copyright (Republic Act 8293, section 176), but a site’s own terms are still followed.
05Sites that are down or block bots
When a site is down or behind bot protection, RE:GOV rebuilds it from the Internet Archive’s public Wayback Machine captures instead, under the robots.txt and terms of use that the archive captured for that site.
06Block it or take a concept down
Any site owner can ask RE:GOV to stop, and sites found to forbid it in their terms are added too. The domain goes on a do-not-crawl list: new requests for it are refused, and existing concepts, screenshots and reports of it stop being served. Requests are honoured promptly.
Open an issue at github.com/rockyru/regov/issues or contact the maintainer via rmbergonia.com. The crawler’s code is open at src/pipeline/crawl.ts.

REGOVBot visits a government website only when someone asks RE:GOV to reimagine it. Here is exactly what it fetches, the rules it follows, and how a site can block it or ask for a takedown.

Last updated October 28, 2026. Questions: GitHub issues or rmbergonia.com.

01Who it is

REGOVBot is the crawler of RE:GOV, an independent, non-commercial civic-tech project. It identifies itself in every request with this user agent:

Mozilla/5.0 (compatible; REGOVBot/1.0; +https://regov.app/bot) Chrome/120

It runs only when a person submits a government address on regov.app. It does not crawl the web on its own, and it does not index sites for search or train models.

02What it fetches

Public pages of the one site requested: up to 30 pages, at most three links deep, about one request a second.
Citizen’s Charter PDFs linked from those pages, and the WordPress REST API when the site exposes it.
No logins, no forms, no search boxes. It never solves a captcha or works around bot protection; when it meets one, it stops.
RE:GOV accepts at most 7 new sites a day in total, and does not crawl the same site twice within 24 hours.

03robots.txt

REGOVBot follows robots.txt as set out in RFC 9309: rules for REGOVBot if the file names it, otherwise the rules for every crawler (*), with Allow and Disallow, wildcards, and Crawl-delay (up to 30 seconds). If robots.txt cannot be read because the server errors, it stays out.

If robots.txt asks REGOVBot, or every crawler, to stay away from the whole site, RE:GOV does not rebuild that site at all, not even from the Wayback Machine. To block only REGOVBot:

User-agent: REGOVBot
Disallow: /

04Terms of use

Before it collects any page, REGOVBot opens the site’s terms of use, legal notice or disclaimer linked from the homepage, and Muse Spark reads it. If the terms forbid automated access (crawling, scraping, data mining) or republishing the content without permission, the run stops there and the person who asked sees the clause that applies.

A plain “all rights reserved” line, or a request to credit the source, does not stop a run: RE:GOV links every rebuilt page to its original. Works of the Philippine government carry no copyright (Republic Act 8293, section 176), but a site’s own terms are still followed.

05Sites that are down or block bots

When a site is down or behind bot protection, RE:GOV rebuilds it from the Internet Archive’s public Wayback Machine captures instead, under the robots.txt and terms of use that the archive captured for that site.

06Block it or take a concept down

Any site owner can ask RE:GOV to stop, and sites found to forbid it in their terms are added too. The domain goes on a do-not-crawl list: new requests for it are refused, and existing concepts, screenshots and reports of it stop being served. Requests are honoured promptly.

Open an issue at github.com/rockyru/regov/issues or contact the maintainer via rmbergonia.com. The crawler’s code is open at src/pipeline/crawl.ts.