All Dashboards Green, All Screens Black

On June 8, 2026, newly loaded web clients began showing black screens instead of livestream video. For our platform, that’s a serious problem: it’s pretty hard to sell Pokémon cards that no one can see.

It turned out that a third-party script we use for client-side security monitoring wasn’t bundled with our release, but was rather fetched live from the provider’s CDN. When the provider updated the script all new web clients fetched it, and it included an additional URL validation which our video provider didn’t pass.

The result? Black screens for users of the affected web clients, with most internal dashboards staying green. 3,000 users were impacted in the first 30 minutes of the incident.

One system did catch it. Our end-to-end service-level objective (E2E SLO) monitoring was already in production and paged the owning teams within five minutes. Here’s what it saw.

The real problem

Most of our dashboards stayed green because they monitor component-level health: a server, an endpoint, a specific function. These are all useful, but can all be healthy while the actual user experience is completely broken. This problem gets worse the more external dependencies there are, or the more sophisticated an experience you want to deliver.

In complex, integrated product experiences like ours, a “small” problem can have an outsize impact on the user experience. Measuring this requires a different approach to observability, namely, to model the user journey across multiple surfaces that must be true for a customer to have a good experience.

So how do we measure this in a complex distributed application?

Joining a livestream is not one thing

Joining a livestream sounds like one action, but the user expects at least three things:

• Video to be playing
• Auction details to be shown
• Chat to be visible and up to date

Each of those can succeed or fail completely independently of the other two. Our video depends on third-party providers and CDN network, auctions are handled by one backend service, while chat depends on a different one, both pushing messages asynchronously when auctions start and new messages are sent.

Users need all three parts to render. We set off to make our monitoring measure the same.

Watching the whole trip

Each user scenario follows the same pattern: there’s a trigger, usually an action a user takes, then maybe a few intermediate steps, and finally a set of expected outcomes. In the livestream join scenario, a user tapping a livestream tile is a trigger, server authorizing the user is a step, and finally the user seeing chat messages, running auction, and video playing are outcomes.

We treat each user action as part of a story. A story starts with a trigger and succeeds only if all steps and required outcomes arrive within a fixed time. We decided to encode such “stories” as protobuf messages, each containing a correlation key, allowing us to decide if user attempt is successful or not.

Whatnot already had an infrastructure that handled event reporting from our backend services and clients, which we decided to use here. Now every step of the user journey emits events, which are correlated by the nodes’ key.

All of the emitted events are handled by our events ingestion service and forwarded to Kafka, from where they go to ClickHouse. Using randomized keys with every story allows not only for performant correlation of events related to each individual attempt, but also allows us to deduplicate stories easily, which matters as our event pipeline delivers events at least once.

Every minute, a background worker then finds triggers that have reached a 60-second deadline. It then collects all events with the same story key and checks, based on the schema, whether every required outcome arrived.

Below you can see an example tick, where a period [t-2, t-1) was processed, where we search for all triggers and correlate events forward. Notice how starting with triggers avoids reprocessing stories that were already processed in previous ticks, even if their events overlap with future tick windows.

The background worker then emits story metrics to Datadog. Each minute, every client with at least one trigger contributes one result. That result is successful only if all required outcomes arrive. The SLI is the percentage of successful client results.

Proven in production

Thanks to this system not all dashboards remained green during the June 8 web incident. Because web applications suddenly stopped emitting outcome events related to the video stream, all stories in the incident window come back incomplete.

Because each of our SLOs has a matching burn rate alert, within 5 minutes the story-based SLO turned red and paged owning teams. The responsible teams started the investigation immediately. If it wasn’t for this new capability, we would most likely only learn about the outage from users, which is the worst kind of detection.

And that’s only one example of incidents caught by this technology. At the time of writing E2E SLOs guard 30 core features across many teams, and the adoption is with 1–2 new SLOs added each month.

What production taught us

Building the first version was only part of the work. Production traffic changed how we defined success, counted results, and processed events:

• Measure users, not attempts. A user with a poor connection can retry an action 100 times. If we count every attempt, one user can dominate the SLI. We now produce one result for each client in each minute and use that client’s worst result. The SLI therefore shows the percentage of affected users without hiding poor experiences.
• Treat client telemetry as production infrastructure. The system depends on clients delivering outcome events. We now use local storage, compression, and robust retries in our client SDKs. We also replaced a third-party ingestion path, which provided P95 latency below five minutes, with an internal system that provides P99 latency of one second. One limitation remains: a fully offline client cannot appear in a live graph.
• Use product rules to define success. Some journeys end early for valid reasons. For example, a seller can block a user from joining a livestream. We added additional event roles for these expected failures so that they do not reduce the SLI. We also use a 60-second deadline. If a story takes longer, we count it as a failure because the experience has already failed for the user.
• Plan for real devices and production scale. Well-instrumented stories reach about 99.5% SLI, not the 99.99% that we expect from many backend systems. Devices and client networks add failure modes that we do not fully control. The system now processes hundreds of thousands of story events each minute. This volume is also pushing our minute-based ClickHouse queries toward their limit, so stream processing is our next step.

Back to black screen

The dreaded black screen exposed a gap in component-level monitoring. Async messaging, client rendering, and third-party scripts can all break a journey while individual services remain healthy.

Not anymore. E2E SLOs watch each person’s trip instead of any one system, and nothing stays hidden. Our reliability is no longer defined by infrastructure boundaries, but rather by user and product outcomes, capturing our users experience in real-time.

If building reliable, real-time systems that power live commerce and bring communities together sounds interesting to you, we’re hiring!