Testing pendula pre & post release

The problem

Pendula was an Australian customer-engagement company. Its platform let a non-engineer build two-way messaging flows and run them against their own customers: a trigger fires, a flow runs, an action calls back, a message goes out. The messages went out under the customer's name, so a flow that stopped running was their customers not hearing from them, and nobody finding out from a stack trace.

Most of that is asynchronous and eventually consistent, so very little can be checked with a single request and response. A request that starts a flow is answered with an acknowledgement, and the work happens afterwards. Asserting on that acknowledgement proves the platform accepted the request and nothing else. I owned how the platform was tested before each release and how it was watched in production, across two suites and the pipelines that ran them.

Continuous canary and load testing

The regression suite validated a release candidate before it shipped and said nothing about the hours after. What covered those hours was infrastructure monitoring, which reports that the services are up. A flow can stop running while every service it depends on is up and green. I proposed a second suite for that gap and built it: synthetic traffic through real flows, continuously, checking that each message came back correctly and in time.

The platform is asynchronous, so there is no single response to assert on: a request starts a workflow that runs in the background and calls back later. Instead of asserting a value, each iteration verifies a full round trip. It generates one correlation id and threads it through three hops: a baseline request logged immediately, a primary request that starts a real workflow, and the platform's callback when that workflow finishes. Two of those reach the canary server and are recorded, and matching them by id is what confirms the message completed the round trip.

// one correlation id, threaded through both requests
const correlationId = uuidv4()
const t = Date.now()

// baseline: recorded immediately
http.post(REQUEST_BIN_URL,
  JSON.stringify({ correlationId, t, canaryType }))

// primary: start a real workflow, tell it where to call back
http.post(WEBHOOK_URL, JSON.stringify({
  outboundUrl: `${CALLBACK_URL}?correlationId=${correlationId}&t=${t}`,
}))

// callback: the same id returns, matched and recorded
app.post('/webhook/callback', (req, res) => {
  const { correlationId, t, canaryType } = req.query
  const responseTime = Date.now() - Number(t)
  db.run(insertLog, [correlationId, canaryType, 'callback', responseTime])
  res.sendStatus(200)
})

K6 generated the load. I compiled a custom K6 binary with an xk6 StatsD extension, so load results were emitted as metrics into the same observability pipeline as everything else rather than a separate report.

Results are stored in an in-memory DuckDB table, keyed by correlation id. The data is short-lived and queried in aggregate, so an in-memory analytical database fit: fast queries, no database server to run, and a scheduled job that deletes records past a retention window. A management endpoint exports the database on demand.

Metrics went through OpenTelemetry, with the exporters chosen by environment variable: OTLP over HTTP to the collector, and a console exporter for local debugging. The K6 metrics took the other path, into StatsD through the custom binary.

The Express server was built to run in production. Its /ping check answers 503 until the database is initialised, so the load balancer holds traffic off an instance that would drop callbacks. It handles SIGTERM by draining in-flight requests, with a forced exit twenty seconds later so a stuck connection cannot hold the process open. The keep-alive timeout is 65 seconds and the headers timeout one second above it, both set above the load balancer's idle timeout so the balancer closes idle connections rather than the server racing it and answering a request that is already gone.

Five canaries ran. Four ran concurrently against a development environment and one against production, each isolated, so a failure in one does not stop the others. Each type carries its own grace period before a missing callback counts against it, because a callback arriving late is normal and a callback not arriving is not. The database keeps a rolling window, a scheduled job deletes anything past it, and the export endpoint sits on a separate management port rather than on the ingress that receives callbacks.

One of the five does not use K6 at all. The trigger canary has to hold a subscription open to receive what it fired, and a K6 iteration is a short-lived virtual user with nowhere to hold one. It runs as a plain Node process against the same server and writes the same rows, so it reads identically in the data.

BDD regression suite

The regression suite is Cucumber.js and TypeScript, with each scenario mapped to a real user path. Because the platform is event-driven and eventually consistent, most steps cannot assert immediately. A step authenticates, drives the GraphQL API, subscribes to the event stream with RxJS, then polls once per second until the workflow has run, before validating the result.

The ceilings are per step rather than global. A synchronous read, fetching the published flows and checking one of them is there, gets five seconds, because if it has not answered by then it is broken. An asynchronous path, a trigger queuing or a flow completing, gets five minutes, which is long enough that a slow run is not reported as a failure. Two dedicated tenants exist for the suite, one outbound and one inbound, so a run cannot touch real customer data and inbound messages have somewhere to land.

// the platform answers late, so poll instead of asserting
async function pollUntil(cond, intervalMs, maxWaitMs) {
  const start = Date.now()
  while (Date.now() - start < maxWaitMs) {
    if (await cond()) return true
    await sleep(intervalMs)
  }
  return false
}

For real-time paths, a scenario registers its own action handler, fires a trigger carrying a unique id, and confirms the handler receives it and echoes it back before checking the flow's recorded events. Runtime types are validated at the boundary with io-ts decoders. Logs are structured with Winston. The suite is containerised as a multi-stage Docker build and run through a Go-Task Taskfile.

Scenarios carry one of nine tags, so a run can take a single category or all of them, each against what it exercised:

CI/CD pipelines

Three workflows. The first is how anyone ran the regression suite. It is a manual dispatch with a dropdown of the nine categories plus a combined smoke run, so an engineer checking a release picks a category from the Actions tab rather than cloning the repository and assembling a dozen environment variables. The staging secrets are injected by the workflow, and cloud credentials are assumed through a role only for the streaming suite that needs them, so the other eight run without them at all. The smoke run is a matrix with fail-fast disabled, because the useful output of a pre-release check is every category that failed, not the first one.

The second builds only what changed. It reads the repository's project directories at runtime and generates the path-filter config from them, rather than keeping a hand-written list that goes stale the first time someone adds a project. Changed projects build in a matrix and push to the registry, tagged by pull request and by commit, and a sticky comment on the pull request carries the image references and the digest so a reviewer can pull the exact build. Sticky, so ten pushes leave one comment rather than ten.

The third runs on merge. The same matrix build runs, the tags are collected across the matrix jobs, and a repository_dispatch event, authenticated as a GitHub App, tells a separate deploy repository to roll the images out. Building the private dependencies needs a registry token, which is mounted as a BuildKit secret rather than passed as a build argument, so it is not left behind in a layer of the published image.

Design decisions

A round trip, not a response. The obvious assertion is on the response to the request that starts a flow. That request is answered with an acknowledgement, so the assertion passes whenever the platform is accepting work, including while every flow behind it has stopped running. Threading one correlation id through a baseline record and the platform's own callback costs a server to receive callbacks and a database to match them in. It is the version that fails when the platform goes quiet.

An in-memory database, not a database server. Records live for a day and every question asked of them is an aggregate over a short window: how many callbacks matched, at what response times, per canary type. That is an analytical query over a small table, which DuckDB answers inside the process with nothing to deploy, secure or pay for. A restart loses the window, which is the cost, and it is affordable because the same numbers have already left as metrics. The raw rows are still reachable through an export endpoint when someone wants to look at them.

Load results into the platform's own pipeline. K6 writes its own summary at the end of a run, which is a report somebody has to remember to open. I compiled a custom K6 binary with the xk6 StatsD extension so the load figures arrived on the same dashboards as the platform's own metrics, where they sit next to what they are meant to be compared against.

One canary outside the load runner. The trigger canary has to hold a subscription open to receive what it fired, and a K6 iteration is a short-lived virtual user with nowhere to hold one. Rather than wrap the client to fit the runner, or weaken that canary into a fire-and-forget check, it runs as a plain Node process against the same server and writes the same rows. The cost is a second execution path to maintain.

A grace period, not an immediate failure. Callbacks arrive late by design, so each canary type carries a window before a missing callback counts against it. Without one the suite reports failures that resolve themselves a few seconds later, and a monitor that cries wolf gets muted. The trade is that for the length of that window a slow flow and a stopped flow look identical, which is a delay in detection I accepted deliberately.

Where it stands

When I left in October 2025 the regression suite covered nine categories and ran from a dispatch menu, so a release could be smoke tested by anyone on the team without a local setup, and five canaries ran continuously with their results on the same dashboards as the platform's own metrics. Pendula was acquired by Smart Communications that year.

The nearest public thing I own is drift-tests, the same BDD approach on my own project.

Two limits are worth stating. Four of the five canaries ran against a development environment and one against production, so most of the continuous signal was pre-production: it caught a broken deployment, not a customer-facing incident, and extending the production coverage was the next piece of work. And the five-minute ceilings that keep the regression suite from reporting slow runs as failures also mean a genuine failure costs the full five minutes before it says so, which puts the suite in CI rather than in anyone's edit loop.