Running mcpreflight inside a gated weekly pipeline
The longgame case · published 2026-10-30 · self-run, not a customer engagement
Most "production use" claims for a small open-source tool are unfalsifiable. This page is the opposite: every number below can be pulled from a public repository, and the one line where the tool runs is explicitly marked UNVERIFIED. What follows is the pipeline, the four gates a run must pass, the real HTTP failures that occurred, and what mcpreflight contributed to each step.
What longgame is
longgame tracks 13 public-data series (climate disclosure dockets, grid interconnection queues, AI Act registers, Form D filings, agent payment directories, and the MCP ecosystem). No human types a number into it: each line either collects from a public source with the generation date attached, or it is quarantined with a published reason. Writing a zero to fill a gap is treated as fabrication, not as a missing value.
One of those 13 lines — mcpreflight-weekly — is the MCP ecosystem census, and mcpreflight is the scanner behind it. On top of the census sits the number we publish separately: of the 35,327 remote endpoints returned by the complete 2026-11-02 registry sweep, 20,701 (58.6%) completed a clean anonymous handshake, 14,626 (41.4%) gave no anonymous response, and 167 (0.5%) answered but broke the protocol. Method and limits of that count.
What the pipeline demands of a scanner
A scheduled collector breaks in ways a demo never does. Four gates run before anything publishes, and a scanner output that fails any of them stops the round:
| Gate | Question it asks | How it hit the MCP output |
|---|---|---|
| verify | Does the page exist, carry a generation date, and match a recomputed value? | A census page with no dated number is rejected — an undated scan is not a finding |
| contracts | Schema, units, de-duplication, non-emptiness | 13 dataset contracts / 105 assertions. The fixture set also carries a scanner payload with a string toolCount and one with an empty results array; an empty array cannot pass as a successful scan, and a string is not a count |
| fixtures | Do known-good and known-bad samples land on the right side? | 39 constructed samples, 39/39 as expected: normal / empty / missing finalStatus / string toolCount / duplicate rows / HTTP 429 body |
| fault injection | If a fault is injected into a copy, is it actually blocked? | 6 fault classes injected nightly; 6/6 blocked, real data hash unchanged before and after |
The fixture set is the part that ages well: http-429.json exists because rate limiting is a normal outcome, not an exception, and the empty and duplicate samples exist because "we scanned and found nothing" and "we scanned twice and counted twice" are the two failure modes that quietly inflate a census.
Real failures, and what "handled" meant
These are collector-side incidents recorded in the pipeline's own event log, kept here because a scanner faces the same surfaces. Every row links to an event with a timestamp, an HTTP status and a byte count — none of it reconstructed afterwards.
| When (UTC) | Surface | Observed | What the pipeline did |
|---|---|---|---|
| 2026-10-30 09:06:48 | California ARB docket page | HTTP 403, 919 B challenge body | Retried once (403 is in the retryable set with 202/417/429); recovered at 09:10:34 with 58,613 B, wrote a data-source-recovered event, series uninterrupted |
| 2026-10-30 09:09:34 | Second ARB page | HTTP 403, 919 B | Same path; recovered at 09:10:43 with 61,221 B |
| ongoing | ARB SB 261 sub-volume | 417 / 202 to non-browser clients | Declared tolerate:true with the reason in-repo: record the event, never block the round, never write a zero |
| ongoing | CME compute-futures path | HTTP 404, path retired | Exempted the same way, and the line still refuses a success exit rather than backfilling an old value |
Read plainly: 403 and 417 are treated as environmental interception, not as "the source is dead". A 919-byte challenge page is not data, and it is not zero. The rule that made this survivable is narrow on purpose — a page failure only fails the round if it is neither tolerated nor a known anti-bot status, or if every page on the line is down. Before that rule existed, one anti-bot page silently took an entire week of collection and publishing with it.
The uncomfortable part
mcpreflight did not cause any of the above, and it is not a monitoring system. It runs read-only: two JSON-RPC requests per endpoint, no tools called, no credentials, no payments. What it gave the pipeline was a reproducible per-endpoint verdict — PASS / AUTH_GATED / UNKNOWN / FAIL — that could be schema-checked, fixture-tested and fault-injected like any other dataset. What it did not give was any way to distinguish "requires OAuth" from "dead on arrival" without trying and failing; that ambiguity is the reason 41.4% is published as an upper bound rather than a death count.
The line is also marked UNVERIFIED in the pipeline's own status file. That is not modesty, it is the honest state: the census is produced by us, checked by our own gates, and has never been recomputed by an independent party. It sits at evidence level E0 next to the other twelve lines.
Reproduce it
npx @eidonze/mcpreflight schema <endpoint-url> # single endpoint, current version 0.5.1 git clone https://github.com/xka0085-byte/longgame && cd longgame npm i && node scripts/assurance-gate.mjs # verify / contracts / fixtures / fault-injection
The pipeline exports a self-contained check snapshot each round, including exit codes, sample inventory, per-file sha256 and first-observed timestamps, so the gate result can be recomputed rather than trusted: longgame/assurance/artifact-2026-10-30.json. The 2026-10-30 gate result was passed:true at head df87cd6 with input fingerprint ef6f2815f12d0053….
Links
- longgame — collection and gate method (the counterpart page)
- github.com/xka0085-byte/longgame
- github.com/xka0085-byte/mcpreflight · npm
- How many MCP servers are dead? · mcpreflight home