002 — Hardening Mythic

For one week we stopped building Mythic and tried to break it instead — black-box against a live server, white-box where the black box wasn't enough. The week produced twelve findings. Every one is fixed, pinned by a regression test, and re-verified against its original reproduction before we called it done. This entry also records the two M1 roadmap items that landed in the same pass: key rotation and a Redis store.

Key rotation without downtime

Until this week Mythic signed every decision token with a single Ed25519 key; rotating it meant a restart and invalidated every live token. The server now keeps a key ring with exactly one active signer (ADR-0006). Rotation is time-based and runs in-process (MYTHIC_KEY_MAX_AGE, default 720h). Retired keys stay advertised in the JWKS for MYTHIC_KEY_RETENTION (default 24h) — deliberately longer than any token they signed can remain valid, so in-flight tokens never break mid-rotation:

FIG. 1 — Retired keys outlive the tokens they signed. Rotation never invalidates a live token.

The keystore is a JSON file written atomically; legacy 32-byte seed files still load and migrate on first save. Origins keep verifying tokens locally, selecting the key by the kid claim — no coordination call needed.

A store that survives more than one node

The in-memory store bound mythicd to a single node (ADR-0009). MYTHIC_STORE=redis selects a Redis backend: single-use challenges via GETDEL (atomic fetch-and-delete) and pressure counters as sorted sets pruned by one Lua script. Both backends implement sliding-window counters, which closes the fixed-window boundary burst. The Store interface now returns errors: an unreachable Redis refuses startup, and a runtime outage surfaces as 503 store_unavailable on /v1 while /healthz stays open for load balancers. A shared contract suite runs against both backends (memory and miniredis) so their semantics cannot drift.

One caveat in writing: the store is shared, signing keys are not. A token from node A does not verify against node B's JWKS. Multi-node token verification needs a shared keystore (KMS/HSM) — M5, not solved by this change.

Twelve findings, grouped

Each row: what the attack showed, then what changed. Every fix carries a regression test; every fix was re-run against its original reproduction.

Area	Cause	Fix
Transport & limits
Rate-limit budget	Budget was consumed after validation — invalid-JSON and unknown-site-key floods ran unthrottled (600 requests at 225 req/s, zero 429s).	Budget is now counted before validation.
IPv6 pressure key	Limiter keyed IPv6 at /128 — one /64 controlled 2⁶⁴ identities (5,000 requests from a single block, zero hits).	Counters aggregate IPv6 to /64 (ADR-0007).
Connection timeouts	No body/read/idle timeouts — a slow client held connections indefinitely.	`ReadTimeout`, `WriteTimeout`, `IdleTimeout` are set.
Challenge economics
Solve-time floor	Floor modeled a 50k H/s worker; measured `crypto.subtle` workers run ~400k H/s — honest clients were flagged implausible.	Floor is now a low quantile of honest solve time (`2^d / (2M × 32)`): false positives ≤ ~6% worst case, GPU-class answers trip it ~96% (ADR-0008).
Delayed verification	Any client could simply delay verification past the wall-clock floor — the floor cannot catch patience.	Documented in the threat model as the behavioral layer's job (M2), not the floor's.
Tokens & configuration
Token TTL	The invariant "tokens never outlive their challenge" was unenforced — a 1-hour token was issued from a 5-second challenge.	`TokenTTL` is now clamped to `ChallengeTTL`.
Score clamp	Hostile tuning (`MYTHIC_FAST_SOLVE_PENALTY=500`) leaked a risk score of 500 into signed tokens.	Scores clamp to 0–100 at the engine boundary; penalties above 100 refuse to start.
Config validation	A negative `MYTHIC_BASE_DIFFICULTY` issued challenges any nonce satisfied.	Configuration is validated at startup; out-of-range values refuse to serve.
Base64 canonical form	Go's base64 decoder silently discards the unused low bits of a trailing character, so distinct token strings decoded to the same bytes and verified — breaking string-level token identity. Caught by the tamper-resistance mutation suite on its first run.	`VerifyToken` enforces canonical base64: re-encoding the decoded bytes must reproduce the input exactly, otherwise the token is rejected.
State handling
Keystore concurrency	Two processes sharing one keystore collided on the same temp file or silently split-brained their signing keys.	Temp names are process-unique; saves refuse to overwrite a file changed since load.
JSON trailing bytes	Trailing bytes after a JSON value were silently ignored.	Trailing bytes are rejected.
Counter saturation	The sliding-window counter stored one entry per hit — including rate-limited ones. A million hits from one identity cost ~16.4 MB of heap and 595 s of CPU.	Counters saturate at `IPLimit+1`: the same flood costs +0.00 MB and 714 ms.

The score-clamp fix is worth showing because it sits at a trust boundary — the value that ends up signed inside tokens:

// The score is a 0-100 contract (token claim `risk`,
// dashboards, tuning docs). Hostile tuning must clamp
// here, not leak 500 into tokens (red-team H3).
if score > 100 {
    score = 100
}
if score < 0 {
    score = 0
}

Verification — and what stays open

Every finding has a reproduction; every fix has a regression test (go test ./... -race, plus the SDK typecheck and tests), and every fix was re-verified against the original reproduction. What remains open is documented in the threat model, not in our heads: patient solvers that delay verification (M2, behavioral signals), distributed attacks with fresh identities (partly addressed since by M1 network fingerprints — see 003), and per-node keystores (M5, shared KMS/HSM).

— XEY LABS

Hardening Mythic.

Key rotation without downtime

A store that survives more than one node

Twelve findings, grouped

Verification — and what stays open