Checking registry listings without scraping anyone
If you ship an MCP server, being findable matters: the official MCP registry, mcp.so, Glama — each is a channel users actually browse. MCPatrol shows a LISTED / NOT LISTED status for your servers on each of them. The official registry has a public API, so that one's easy. The community registries don't. The obvious move is to scrape their search pages.
We skipped the scraper. Both sites disallow their search endpoints in
robots.txt, and scraped search results are a maintenance headache regardless —
markup shifts, rate limits, blocked IPs. There's a more stable interface sitting right next
to the rule that says don't.
The front door: sitemaps
The same robots.txt files point to something better: published sitemaps —
machine-readable lists of every page the site wants discovered, servers included.
So MCPatrol's backend does this instead:
- Once a day, fetch each registry's sitemap index and its server-listing children. That's a handful of XML requests per site per day — less traffic than one human visitor clicking around.
- Cache every listed URL (about 18,000 entries for mcp.so alone) in our own database. All matching happens against the cache, so checking your server's status costs the registries nothing.
- Match by slug against your server's name, then confirm a candidate by fetching just that one allowed page and checking it references your server's host.
- A descriptive User-Agent — the service name and a contact URL, standard crawler practice, so the requests are easy to place in a server log (and easy to block, if an operator ever wants to).
Edge cases that made it interesting
- mcp.so paginates its server sitemap through query parameters
(
sitemap.xml?section=servers&page=N) — child sitemaps that don't end in.xml. Detecting a sitemap index by its root XML element instead of its URL suffix fixed that. - Glama lists remote servers under a different path
(
/mcp/connectors/…) than the rest (/mcp/servers/…). Path allowlists per registry, not one global pattern. - A failed fetch returns an empty list — which must never wipe yesterday's cache. A registry having a bad hour shouldn't flip every listing to NOT LISTED.
When the answer isn’t known
When the cache can't answer — first sync hasn't run, the sitemap was unreachable, the match is ambiguous — the app shows UNKNOWN, not a guess. A listing status you can't trust is worse than none.
None of this is clever. Sitemaps exist so software can find these pages; this is just reading them on a slow clock.