@testdino/playwright - Streaming Playwright Results Out of CI

The problem

A Playwright suite that runs on one machine produces one report. A Playwright suite that runs sharded across a CI matrix, or split by hand across several jobs, produces fragments. Each fragment has its own start time, its own pass and fail counts, and no shared identity. Nothing in the fragments says they belong to the same run. The second cost is timing. A report is written when a process exits, so the feedback arrives after the slowest shard finishes. On a long matrix, the first failure and the first sight of that failure can be twenty minutes apart. The engineer who triggered the run has moved on. TestDino is a CI test-observability platform. This case study covers the client half: the reporter and CLI that runs inside the customer's pipeline, collects results, and delivers them. It is published on npm as tdpw, currently at v2.6.2, Node 18 and above, so every claim below is installable and checkable. The brief was narrow and strict:

Install as a Playwright reporter, or run as a CLI wrapper with no edit to playwright.config.ts at all.
Stream each test result as it completes, not a report uploaded after the run.
Never block or slow the run it is observing.
Reassemble a sharded or split run into one coherent run on the server side.

I was the principal author across nine months and set the architecture. Five other engineers worked on the CLI and metadata layers alongside me. The design decisions below are the ones I set and held.

Delivery under an unreliable transport

Streaming results live means the reporter is now a network client sitting inside someone else's build. That is a hostile place to be. The broker can be unreachable, the runner can be on a constrained network, and the build must not fail because telemetry failed. Delivery is dual-transport. Kafka is primary, HTTP is the fallback, and transport health is tracked rather than guessed. The part that took the longest to settle was not the failover. It was the return. A naive implementation retries Kafka on the next event, fails, falls back again, and thrashes on every single test result while the broker is flapping. So the return path is deliberately asymmetric: the client drops to HTTP on failure immediately, but only returns to Kafka after a configurable run of consecutive HTTP successes. Recovery is evidence-based, and a broker that is up and down every few seconds costs one failed attempt rather than hundreds. The HTTP path is byte-aware, not count-aware. Batches cap at 900 events and 4 MiB, sized deliberately under the documented server limits of 1000 events and 5 MiB. Counting events alone is not enough: a run with large attachment metadata hits the byte ceiling long before the event ceiling, and the failure mode of getting that wrong is a rejected batch, which means lost results in the middle of a run. Failures return a typed error taxonomy rather than a generic throw. The distinction matters at the boundary. An authentication error is permanent and should stop retrying. A batch-too-large error should split and retry. A transient network error should back off. One error type collapses those into the same behaviour and makes the wrong choice in two cases out of three.

Two problems that shaped the design

Two constraints did more to shape this codebase than the transport work did.

Enqueue cannot block

test:end is a Playwright reporter callback on the hot path of the run. Anything awaited there is time added to the suite, and a transport that is hung is a suite that is hung. So the buffer has an intentional sync and async split. Enqueue is synchronous and fire-and-forget: it takes the result, puts it in the buffer, and returns. Flush, batching, transport selection, and retry all happen on the async side, away from the callback. The split is not an optimisation, it is a correctness boundary. It means the worst case for a dead broker is buffered events and a slow drain, not a stalled test run.

Silence is not death

Long tests are normal in end-to-end suites. A test with a 5 to 10 minute timeout emits nothing for minutes at a time. An activity-based reaper on the server side cannot tell that apart from a shard whose runner was terminated. Both look identical: no events. Marking the long test dead produces a false failure. Waiting long enough to be safe means a genuinely dead shard hangs the run for the length of the longest possible timeout. The fix is a liveness heartbeat from the CLI, with two conditions that matter more than the heartbeat itself. It emits only during silence, so a busy run pays nothing for it. It emits only while the process is actually alive, so a crashed or killed shard stops emitting and is still detected as dead. A heartbeat that survives the thing it reports on is worse than no heartbeat, because it converts a detectable failure into a silent one.

Reassembling a split run

Sharding and splitting are not the same problem. Playwright's own --shard gives the reporter shard identity for free, so those fragments can be stitched without the user doing anything. Manual splitting, where a team divides specs across jobs themselves, gives the reporter nothing. There is no signal anywhere in the process that says "you are split 2 of 5". It has to be declared. Declaring it introduces a configuration problem, because the value can arrive from four places: a CLI flag, testdino.config.ts, reporter options, or the environment. They resolve through an explicit precedence chain, flag first, then config file, then reporter options, then env, so a value set in CI is predictable rather than dependent on load order. Two rules guard it. Validation is both-or-neither: an index without a total, or a total without an index, is rejected at startup rather than producing a run that aggregates wrong and looks fine. And the wire contract for split identity is frozen, because the server aggregates on those fields and a rename in the client silently breaks reassembly for every version already installed in the field.

Working across ten CI providers

Every result needs provenance: which commit, which branch, which pull request, which build. The reporter resolves that across 10 providers: GitHub Actions, GitLab CI, CircleCI, Azure DevOps, Jenkins, AWS CodeBuild, Bitbucket Pipelines, Buildkite, Harness, and TeamCity. The non-obvious rule is precedence. Provider environment variables are preferred over local git, and that is the opposite of what looks correct. Reading the repository seems like the more authoritative source, but the checkout step in a pull-request build frequently rewrites the ref into a detached merge commit that exists only inside CI. Trusting git there attributes results to a commit nobody can find. The provider's own variables describe the build that was actually requested, so they win. Coverage is merged in the same layer. Istanbul coverage files from each shard are collected with glob filtering, path-normalised so the same source file reported from different working directories resolves to one entry, then merged into a single coverage view for the run.

Scale and ownership

The published package is roughly 8,800 lines of TypeScript in src across 9 modules, with roughly 20,200 lines of test code: about 1,042 test cases across 42 files. The ratio is deliberate. This is a library that runs inside other teams' pipelines, where a regression does not surface as a bug report, it surfaces as a customer whose CI is broken. On ownership, the honest version: this is a multi-contributor codebase, not a solo project. Over nine months I authored 120 of 189 commits, 63%, and hold 63% of surviving blame in src and 57% in tests. Five other engineers contributed, mainly on the CLI and metadata layers. Squash merges compress multiple commits into one, so those counts are directional rather than exact. What is not directional is the architecture. The transport health model, the sync and async buffer split, the heartbeat conditions, the config precedence chain, and the frozen wire contract are decisions I set and held across the nine months, including through the parts where holding them was the harder option. The package is public. npm install tdpw and the source, the test suite, and the version history are all inspectable without taking my word for any of it.

Related projects

Driveline TCO - Alternative Drivetrain Comparison & Route Analysis Platform

Client engagement for a Munich automotive firm. Engineering lead on structural improvements to an existing TCO comparison platform spanning seven drivetrain concepts for heavy commercial vehicles - consumption, total cost of ownership, route feasibility. Bilingual UI, 115 tests, deterministic models. Delivered ahead of schedule.

Read case study

Saral - AI-Powered Recruiting Search Platform

Designed the platform architecture and owned backend, infrastructure, and the search data flow. Semantic candidate search over 119k profiles with progressive backfill, LLM fallback, and per-page credit billing.

Read case study View project

TestDino - AI-Powered Test Intelligence Platform

Forward deployed engineering from day one. Architected a SaaS platform handling 2M+ test executions for 80+ organizations. Built the backend, led the team, shipped to production.

Read case study View project

TestGenX - AI-Ready Parallel Test Execution Engine

Platform engineering project: Built a distributed test execution system from scratch. 200 parallel VMs, 7 hours saved weekly, zero-touch automation.

Read case study View project