github
closed
No calls in the window.
State per-instance — Redis is not configured
Site reliability
What this site
knows about itself.
Rate, errors and duration for its own endpoints, measured by the code that serves them. The circuit breakers in front of its two flaky upstreams, with every state change they have made. And an experiment that drives the production breaker through an outage, in your browser, so you can check that it does what the description says.
RED, from shared state.
Rate, errors, duration — the three numbers that describe a request-serving system. The hard part here was not measuring them, it was making them survive being serverless: an in-process counter lives in one instance’s memory, and a scrape reaches whichever instance the platform picks, so the series jumps between unrelated totals. Each instance therefore accumulates deltas and appends them to Postgres, where they sum correctly no matter how many instances ran. Percentiles come from fixed histogram buckets rather than stored averages, because a mean latency is the one number that never describes anyone.
No requests recorded in the last 60 minutes. Metrics are flushed at most once every 15 seconds per instance, so a freshly deployed build reads empty until the first instrumented route is called — which is a different thing from an outage, and this page will not pretend otherwise.
Buckets are fixed at 10, 25, 50, 100, 250, 500, 1000, 2500, 5000 and +∞ milliseconds. They are a one-way door — change them and every row written beforehand measures something else — so they were chosen for what this app actually does: 10ms is a cache hit, 100ms is a Neon round trip from the same region, 5s means an upstream is hanging. Instrumentation is opt-in per route, so a route absent from this table is one nobody wrapped, not one nobody called.
Circuit breakers.
Two upstreams sit behind one: GitHub, and LeetCode’s unofficial endpoint, which throttles serverless IPs often enough that the failure path is the ordinary path rather than the rare one. The breaker opens on a failure ratio and not a failure count, because a threshold of five failures means something completely different at ten requests an hour than at ten thousand, and this site sees both.
leetcode
closed
No calls in the window.
State per-instance — Redis is not configured
No transitions recorded. Both upstreams have stayed inside the failure threshold since the breaker was deployed — which is the boring outcome, and the one worth having.
Break it yourself.
The breaker exercised below is the one described above — the same module, imported into this page rather than reimplemented for it. Only the dependency is simulated. Time is simulated too, so ten minutes of cooldowns and probes resolve instantly; every random decision comes from a seeded generator, so a seed and a scenario reproduce a run exactly. Chaos engineering you cannot replay is an anecdote, not an experiment.
The dependency dies completely at call 20 and comes back at call 70. The clearest view of open → half-open → closed.
- Calls refused
- 3025% of all calls
- Latency not spent
- 1.2son calls that never left
- Reached the dependency
- 9022% of those failed
- Ended
- Closedafter 120 calls
3 state changes
closed → opencall 39, 39s in — 20/40 failed (50% ≥ 50%)half-open → half-opencall 70, 70s in — half-open probehalf-open → closedcall 71, 71s in — 2 consecutive probes succeeded
Policy in force, and it is the one guarding the real integrations: opens above 50% failures over a 60s window, never below 5 samples, probes after 30s, closes on 2 consecutive successes.
Traces, and what they cost.
W3C Trace Context in, OTLP over HTTP out — both implemented directly rather than pulled in as an SDK, which is the same trade this codebase already made for its Redis and S3 clients: the protocol is small and completely specified, so speak it. The consequence is worth stating plainly: there is no auto-instrumentation. A span exists because a call site asked for one.
No collector is configured, so spans stay in this instance's memory. The list below is therefore whatever landed on the instance that rendered this page — a debugging aid, not an observability backend, and it will look empty or partial precisely because it is honest about that.
No spans on this instance yet. Spans are recorded around guarded dependency calls, so one appears the first time this instance fetches GitHub or LeetCode stats.
What is real, and what is not.
Four claims about this page, including the two that are limitations. Open any of them.
The breaker is realIn production
It guards both integration fetchers in production. When it opens, the stats panels render last-good data through the existing stale-payload path rather than an error state — and they do it immediately instead of spending eight seconds rediscovering that the upstream is down.
The metrics are realPrometheus
Every row is written by the code serving the request, and /api/metrics exposes them as a Prometheus counter and histogram. The counter resets when the retention sweep prunes, which is a counter reset — rate() is built to detect exactly that, so the type is honest.
The chaos is simulated, deliberatelyIn your tab
Faults are injected into a dependency that exists only in your tab. There is a server-side arm that injects into the real integration path, and it stays off unless CHAOS_ENABLED=1 — it is for a preview deploy, where the question is whether the fallback renders, not for production, where the answer would cost a visitor their page.
The trace buffer is per-instanceA real limit
Without a collector, spans live in the memory of one serverless instance and the list above shows whichever instance answered. That is the same limitation that made in-process counters unusable, and it is not solved here — it is solved by configuring OTEL_EXPORTER_OTLP_ENDPOINT, which is a URL, not a code change.