Testing

Cloud has two kinds of tests. Unit, render, and behavior tests run anywhere. Integration tests need real infrastructure and run only when you point them at it explicitly.

Run every check

bun run check
bun run test

bun run check verifies dependencies, import boundaries, package cycles, service API contracts, localization, capability presentation, CSS architecture, formatting, fixture short IDs, the application set, the CI gate's job list, and every package typecheck. A package typecheck covers its scripts/ as well as src/, so a package's build, smoke, and verification scripts break the gate when they drift from the code they drive. The only exceptions are scripts a package excludes by name in its tsconfig.json and lists in its scripts README, currently three manual Grids tools. The root scripts/ directory is not typechecked. bun run test runs every workspace in its own process and reports the integration files it skipped.

Both commands are heavy: each takes one of the machine's heavy-run slots, waits when all are taken, and says so. By default there is one slot, so only one heavy run proceeds at a time. CI runs without slots. Package typechecks use two checker threads each, and the typecheck rule runs as many packages in parallel as the run's share of memory allows (about 8 GiB per parallel type checker). Before a pull request, the focused checks in What CI runs are usually enough.

The localization check also rejects hard-coded prose in frontends whose text comes only from message catalogs, currently Mail: JSX text and expressions, labels, titles, descriptions, placeholders, and toast or prompt messages with two or more words. It reads every branch of a conditional, and in a template literal a substitution next to a word counts as a second word. Single words such as product names or protocol labels stay allowed.

The capability presentation check compiles the capability declaration each built-in application passes to app.start() and requires a German title and description for every Type, Query, Action, Command, and Universal Search tag. The Assistant, approvals, search, and the capability catalog show these texts to German readers.

Heavy runs

Set CLOUD_HEAVY_SLOTS=<n> in your shell profile to allow n heavy runs at once. Use about one slot per 16 GiB of memory and per 6 CPU cores, whichever gives fewer: 1 on a 16 GiB laptop, 2 at 32 GiB and 12 cores, or 4 at 64 GiB and 24 cores. The typecheck rule divides the machine's memory by the slot count, still budgeting about 8 GiB per parallel type checker. More slots mean fewer type checkers per run.

Run other heavy commands through bun --no-env-file scripts/heavy.ts <command>: browser test files, docker build, performance scripts, or integration files run by hand. The flag keeps the checkout's .env away from the command, and the helper refuses to start without it. The helper takes a slot the same way and marks the command with CLOUD_HEAVY_LOCK=held, so a nested bun run check or bun run test does not wait again:

bun --no-env-file scripts/heavy.ts docker build -f packages/mail/Dockerfile .
bun --no-env-file scripts/heavy.ts bun test <file>.browser.test.ts

The older form flock /tmp/cloud-heavy-run.lock env CLOUD_HEAVY_LOCK=held <command> still works and occupies the first slot on hosts whose temp directory is /tmp, so it also waits for integration runs.

A run whose environment sets a CLOUD_TEST_* target always takes the first slot. Integration runs stay one at a time on the machine even with several slots, because checkouts share the Valkey and Filegate test targets. Other runs take the first slot only when every other slot is busy. When all slots are busy, a run waits for one slot picked by its process ID, so it can keep waiting while another slot frees up first.

Write behavior tests

A behavior test renders Solid components into a happy-dom document and drives them like a user. Name it *.behavior.test.ts or *.behavior.test.tsx and put it anywhere in a workspace package. bun run test finds every such file and runs each package's behavior tests in a suite of their own, <package> behavior, with browser conditions and the Solid DOM preload (packages/ui/test/solid-dom-preload.ts), started from the repository root so a package's server-rendering preload does not apply. Package test scripts leave these files out with --path-ignore-patterns '**/*.behavior.test.*'.

To run one file directly:

bun --no-env-file test --isolate --conditions=browser \
  --preload ./packages/ui/test/solid-dom-preload.ts \
  ./packages/core/src/pages/admin/CacheNotice.behavior.test.ts

Guard browser-only tests with isServer from solid-js/web, so a plain bun test skips them instead of failing. bun run check fails when a test that branches on isServer is not named *.behavior.test.*, or when the runner would not pick up a behavior test.

Load heavy components once at module scope, not inside the first test: the first import runs the Solid transform over the component's source graph, and that time otherwise counts against the 5 s test timeout. A plain module-scope import does not work, because the @k2b/ui browser build needs a document while its modules evaluate. Load the component under a temporary harness from packages/ui/test/dom.ts, after any top-level mock.module calls:

const load = async () => {
  const dom = createDomTestHarness();
  try {
    return (await import("./RecordsView")).default;
  } finally {
    dom.cleanup();
  }
};
const RecordsView = isServer ? undefined : await load();

A browser test has the same 5 s budget for its pages, documents, and scans; see Test in Chromium and WebKit.

Solid attaches its delegated event listeners, such as the one for click, to the document that exists while a module evaluates, here the temporary one. A test that clicks therefore calls delegateEvents(["click"], dom.document) for its own document. packages/grids/src/frontend/_components/records-view/RecordsView.behavior.test.tsx loads its component after module mocks and registers delegated events in each test.

Pin the clock

Tests that render dates use a fixed instant. In behavior tests, use setSystemTime(new Date("2026-10-14T10:00:00.000Z")) from bun:test and restore the clock with setSystemTime() after each test.

In browser tests, fixtures stamp their data from a fixed clock, and each context gets an init script that replaces the page's Date with one that starts at that clock and flows from there. The Chat harness starts at chatTestNow, exposes its flowing clock as harness.now(), and pins pages with pinPageClock in packages/chat/src/frontend/ChatWorkspace.browser.test.ts. A page built with setContent keeps the window of its blank page, so the continuous month test in @k2b/ui runs the same function with page.evaluate before setContent. Use performance.now() for durations and deadlines.

Do not pin the date with Playwright's clock. setFixedTime and setSystemTime also install its fake clock, which replaces timers, animation frames, and performance. Scroll- and frame-driven UI then runs on a timer instead of the page's rendering, so a test sees state that lags only in the test. The Chat read-state tests failed in WebKit this way, and so did the continuous month title (#1139).

The test-clock check flags .clock.setFixedTime( and .clock.setSystemTime( calls on a page or context. A line comment // playwright clock: <reason> on the same line or directly above allows a pin in a test whose subject is fake timers. The reason must be non-empty. clock.install(), runFor, and fastForward stay allowed for deliberate timer control, and setSystemTime from bun:test is not a finding.

Measure browser timing on the browser's clock. Playwright events such as request reach the test process late under load, so tests that assert a browser throttle or delay record performance.now() in the page, as the Chat read-state tests do with chatReadStarts in an init script that wraps fetch. Wait for browser-side windows with a timer inside the page. The test-clock check also rejects clock reads inside Playwright event listeners.

The test-clock check rejects new new Date() and Date.now() reads in UI tests, fixtures, harnesses, and suites. A line comment // real clock: <reason> on the same line or directly above allows a read that needs the real clock. The reason must be non-empty. Known findings live in scripts/checks/test-clock.baseline. Its counts can only decrease; bun scripts/check.ts test-clock --fix lowers counts and removes entries when findings are gone.

Wait for conditions, not time

A test waits for the state it needs: poll with a deadline, await an event, a recorded close, or a log. A fixed sleep assumes the state arrived and is still there. The live-routes slow-client test waits for the server's 1013 close before it resumes the client. A test about the moment before a debounce or delay fires acts in the same task as the trigger, as the Notebooks Timeline scroll-restore test does, because a check after a real-time wait that the delay has not fired yet fails on a slow runner. Wait for the state you assert, not for a step before it: the continuous month test waits until the grid rests at the week and the header names the month, because the view names the month in an animation frame after the scroll event.

A live region that a component fills after a timer is read with a bounded wait, such as waitForFunction on its text, never once after settled(). settled() waits for fonts, animations, and frames, not for timers. The outcome line of ChatOutcome in @k2b/ui empties its role="status" region and fills it 40 ms later, so that the same words spoken again are announced. The answering case in action-question.browser.test.ts holds the fixture's answer, waits until the status names the pending line, and only then releases the answer (#1572).

The test-waits check rejects new awaited fixed waits of at least 1,000 ms with a numeric literal in test files and browser suites (*.test.ts(x), *.browser-suite.ts(x)). It also rejects a polling loop whose attempt count decides the outcome, such as for (let i = 0; i < 100 && writes.length < 2; i++) await sleep(20). That loop is a hidden 2 s budget, and a loaded runner outlasts it (#2204). Wake the wait from the condition's own signal instead, and let the test's timeout bound it. A line comment // fixed wait: <reason> on the same line or directly above allows a wait whose duration is the subject of the test. The reason must be non-empty. Known findings live in scripts/checks/test-waits.baseline. Its counts can only decrease; bun scripts/check.ts test-waits --fix lowers counts and removes entries when findings are gone.

Scroll-driven animations, such as the navigation label fade in @k2b/ui, keep running and never finish. Before waiting for page animations to stop or awaiting their finished promises, select only the document timeline with document.getAnimations().filter((animation) => animation.timeline === document.timeline). Apply the same filter to element calls with { subtree: true }, which can include navigation labels. The animation-settle check requires an actual .timeline property access in the surrounding statement of document-wide and subtree calls in tests, browser suites, and browser harnesses, or a leading // animation-settle: all timelines comment (optionally followed by a reason) for deliberate inspection of all timelines.

Every enhanced link starts a document view transition. While one is active, WebKit sends pointer input to <html>, so a click is lost, and the transition can stay active after its own animations ended, so settled() may return before it. After a navigation, a test awaits viewTransitionEnded(page) from packages/ui/test/browser.ts before its next click or Back. See the Info and thread tests in packages/chat/src/frontend.

On a wide screen, a closing AppWorkspace detail panel slides out first, and only 140 ms later the content takes its room back in one step. Playwright's stability check can pass before that step, so a press can start on a control that then moves, and the click goes to an ancestor. After closing a detail panel, a test moves the pointer off the content that moves, so no hover starts while it settles, and awaits settled(page) before its next press. See the Info reopening test in packages/chat/src/frontend/ChatWorkspace.browser.test.ts (#2487).

After a pointer move, a test waits until the target matches :hover, or for the control that the hover reveals, before it reads what the hover changes. WebKit can apply :hover after mouse.move() resolves, and a reveal that only changes visibility starts no animation for settled() to wait for. See hover() in packages/ui/src/layout/sidebar-row-label-room.browser.test.ts.

A browser test that opens its own context or page gives itself a budget, as the Chat suites do (test(name, run, 30_000)), and keeps the page's shorter limit for each step (page.setDefaultTimeout(5_000)). Bun's default of 5 s is meant for one step, not for a page load, a mount, and a flow.

Before a test reads styles or geometry after an action, it awaits settled(page) from packages/ui/test/browser.ts or waits for its own condition. settled() waits for fonts, for the running finite animations on the document timeline, and then for two frames. It does not wait for infinite, scroll-driven, or paused animations, because they never finish. When the page starts new finite animations in that time, settled() throws and names them: a page that keeps animating is not settled, and the test must wait for its own condition. settled() does not change media emulation. Tests that are not about motion pass reducedMotion: "reduce" to newPage() or newContext(). See open() in packages/oauth/src/frontend/authorization-page.browser.test.ts (#1476, #1499).

The settle-reads check flags a style or geometry read that follows an action with no settle in between. It checks *.browser.test.ts, *.browser-suite.ts, and *.browser-harness.tsx files. A read is boundingBox(), or an evaluate() or evaluateAll() callback that uses getComputedStyle, getBoundingClientRect, getClientRects, checkVisibility, or elementFromPoint. Actions are navigation, setContent, clicks, key presses, hover, fill, focus, dispatchEvent, mouse, keyboard, and touchscreen calls, emulateMedia, setViewportSize, and newPage. Settles are settled(), waitForFunction, expect.poll, locator waitFor, keptThrough, and mostAtOnce, when the test awaits them. The check reads each function body in source order, nested blocks included. A call to a named helper in the same file counts as what the helper does: a read before its first action or settle, and its last action or settle. A helper whose return type names Page counts as an action unless it ends in a settle.

Known findings live in scripts/checks/settle-reads.baseline. Its counts can only decrease; bun scripts/check.ts settle-reads --fix lowers counts and removes entries when findings are gone. A line comment // settled: <reason> on the same line or directly above allows a read whose subject is the first frame or the unsettled moment, such as a read of the server-rendered frame. The reason must be non-empty. On a read inside a helper, it also covers the helper's callers.

The cookie-store check rejects direct cookies() reads in tests, browser suites, and browser harnesses. Use storedCookie(page, name) for cookies that page scripts write; see Engine differences. A leading // cookie-store: <reason> line comment allows a deliberate store read, such as an HttpOnly cookie set by a server response. The reason must be non-empty.

Test in Chromium and WebKit

Layout, focus, touch, and paint need a real engine, so these tests drive one through Playwright. They are named *.browser.test.ts, except a few behavior tests that start a browser. Each one starts its browser with launchBrowser() from packages/ui/test/browser.ts:

import { browserName, launchBrowser } from "../../test/browser";

const browser = await launchBrowser();

TEST_BROWSER chooses the engine: chromium, the default, or webkit, the engine of Safari and of every browser on iOS. bun run check fails when a test starts a Playwright browser type directly, or when a Playwright test does not import the launcher directly as .../test/browser, without a file extension: bun run test --browser finds the tests by that import. The Assistant artifact suites are the only exception: the ones that still start Google Chrome run only nightly. The artifact suites that import the launcher run like any other browser test.

bun run test --browser runs only these tests, each file in a process of its own, and fails when it finds none. Files that also need CLOUD_TEST_* targets, such as the OAuth consent test, run with --integration instead. --filter narrows the run to a package or a file name.

When a test times out, bun test stops every process the test file started, including the shared browser. launchBrowser() starts a new browser for the next newPage() or newContext(), so one hung test fails alone. Closing a page or context of the killed browser succeeds, so the error that the hung step reports after the timeout names what it waited for, not the cleanup in finally. Open pages and contexts in each test, not in beforeAll: a page kept across tests closes with its browser. packages/ui/test/browser.browser.test.ts guards this recovery.

browser.close() from launchBrowser() is bounded. It asks the browser to exit and waits up to 2 s, then kills its process group and waits up to 2 s until every process is gone. This keeps afterAll inside Bun's 5 s hook timeout even when exit hangs, so the next file never starts beside a browser still shutting down (#1473). A browser from TEST_BROWSER_ENDPOINT runs elsewhere and only gets the 2 s graceful close. A close that needed the kill and a launch still pending after 4 s print a launchBrowser: line with their durations next to the failure. A failed launch carries its duration in the error. A process group still alive after the kill fails the hook with those timings. packages/ui/test/browser.browser.test.ts guards a hung close and the next launch.

bun run test --browser first runs packages/ui/test/browser-warmup.ts. It draws real @k2b/ui surfaces once, so a fresh engine's one-time costs, such as WebKit's shader compile in a new Playwright container (#2149), do not fall inside the first test's 5 s. It starts its own browser, shares only on-disk caches with the suites, and has its own bounded budget. When it fails, no suite runs.

Open each URL state that a test checks in its own page. Navigate a loaded page in place only when the test checks navigation itself: reload, Back, Forward, or a link. In #1367, an in-place goto on an idle Notebooks page stalled before load in two WebKit CI runs, and a Back to a graph page in a third. The cause was a collapsed receive queue on the CI runner's kernel; see When a page never fires load.

States that a test sets with setContent, a fixture call, or another change inside the page share one page per test. Open the page at the start of the test, change the state inside it, and close the page at the end. Each new page and each new document has a fixed cost. On a busy machine, a WebKit page takes about 300 ms to open. Each setContent parses an inlined dist/styles.css (857 KB) again, and each document runs its addScriptTag bundle again. These costs fit the 5 s test timeout on an idle machine, but not under three heavy slots (#1485, #1497). When many states use one stylesheet, load it once and replace the body for each state, as show() does in packages/ui/src/styles/touch-targets.browser.test.ts. An exhaustive check, such as a tap on every pixel, looks only where its result can change, and a test compares it with the full check; see takenPixels in the same file.

When goto, reload, goBack, or goForward on a launcher page fails or times out, the error appends a navigation diagnosis. It names the call and how long it waited, the URL before the call and at failure, and up to five recent responses with their status, method, and URL. It also reports whether a document committed, the navigation response status, DOMContentLoaded and load events, open requests, failed requests with their error, and a main-thread probe. A navigation cut off when Bun's test timeout closes the browser carries the same evidence instead of a cleanup error; the probe reports that the page or browser is closed. Do not call Playwright between catching an error and reporting it: after the browser closed, Playwright rejects every call with one shared error object and rewrites its message, so the reported error names the later call. No commit with an open document request points to the server or network. A commit with an open subresource points to that request. A silent main thread after a commit points to a blocked page; see When a WebKit page stops answering. Before a commit, Chromium holds the probe until the navigation ends, so there silence alone proves nothing. A failed request with WebKit encountered an internal error adds a hint; see When WebKit reports an internal error. packages/ui/test/browser.browser.test.ts guards the diagnosis for an uncommitted navigation, a committed navigation whose load never fires, a navigation that fails with request errors, and a navigation cut off by Bun's test timeout.

A step that scrolls the page, such as focus() on an element below the fold, dispatches its scroll event on the next frame. Chart inspection, popovers, and menus that close on scroll can then undo the result of the next step. Let one frame pass after such a step (requestAnimationFrame in page.evaluate) before a step whose result a scroll can undo. The scroll also moves content under a mouse pointer that an earlier click left in place, and WebKit then sends a pointermove without any mouse movement, up to about 100 ms later. Do not mix clicks into a keyboard flow that scrolls: press the control by keyboard instead. See showFirst and the repeated enhancement case in packages/notebooks/src/frontend/lib/notebook-charts.browser.test.ts.

WebKit sends no pointerleave when the element under the pointer leaves the document, as when a row drops a line while the pointer rests on it. A state that lasts from pointerenter to pointerleave, such as a paused timer or a list that holds still, then stays after the pointer went. Confirm such a state with the next pointerover on the document: when its target is outside the container, the pointer has left. To test it, rest the mouse on the element that the change removes, make the change, and move the mouse away in the next step, before WebKit updates the hover. See pointAt in packages/filesv2/src/frontend/UploadPanel.browser.test.ts and the held toasts in packages/ui/src/feedback/toast.ts.

A fake server started by a test records a request when it arrives. The page may have changed already, as seen in WebKit on a loaded machine. Never read the newest recorded request after a click or key press. Note how many requests have arrived before the step, wait for the request it causes, and compare that one. See requests.next in packages/dashboard/src/frontend/dashboard-home.browser.test.ts. An assertion that no request was sent can only cover what has already arrived.

Use keptThrough(locator, change) from packages/ui/test/browser.ts to check that a mutation keeps an element and its height. Never read boundingBox() right after a change: it does not wait and returns null for an element that a refresh is replacing. See the claim test in packages/spaces/src/frontend/[id]/_components/kanban/kanban-board.browser-suite.ts.

Use mostAtOnce(page, selector, change) from packages/ui/test/browser.ts to check that Undo or a refresh never shows an entity twice and ends with exactly one. Scope a locator that reads the entity to its row, not to the page. Await Playwright calls and read their values: expect(await box.isChecked()). In Playwright tests, never use expect(promise).resolves or .rejects. Besides hiding the reason for a rejection, Bun's matcher waits in a nested event loop that runs Playwright's channel messages out of order and can close the browser connection. This shows as Cannot find object to "message": jsonPipe@… and a next test that fails within milliseconds (#1502). The browser-test-promises check enforces this rule. Await promises directly and assert on their values; for an expected rejection, capture the reason and assert on it. See the tray Undo test in packages/spaces/src/frontend/[id]/_components/calendar/calendar-tray.browser-suite.ts.

Every page from launchBrowser() answers Cloud's own capability catalog read (GET /api/capabilities/v1/catalog) with an empty catalog, in the page and without a request. Shared browser modules make this read by themselves, such as the file-provider discovery while a page is idle, so a fixture that serves only its own endpoints would see it at random. A test file that serves a catalog, counts its reads, or fails it on purpose launches with launchBrowser({ serveShellCatalog: true }); its pages then send the read to the fixture. Other methods and paths always reach the fixture.

Collect page errors with pageErrors(page) from packages/ui/test/browser.ts and assert expect(errors).toEqual([]); new tests do not register a pageerror listener of their own. The page-errors check enforces this rule; its baseline lists older suites that still have their own listener. While a page waits for its next document, WebKit refuses every load that the page starts, such as a stream that opens again after the navigation cancelled it. It logs the refusal as Fetch API cannot load <url> due to access control checks., also for a same-origin request whose promise is handled (playwright#43120). Playwright reports that log as a page error, so a test that navigates fails at random. pageErrors() ignores only that report, only in WebKit, only for a URL with the origin of the page, and only while a goto, reload, goBack, or goForward of the test runs. It reports everything else: a real CORS failure, which names another origin, an unhandled rejection, a refusal during a navigation that the page starts itself, such as after a click, and every other error. packages/ui/test/browser.browser.test.ts guards these cases in Chromium and WebKit.

Run WebKit locally

Playwright's WebKit needs system libraries that most Linux machines do not have. The Playwright image brings them, and playwright run-server serves its browsers to tests outside the container. TEST_BROWSER_ENDPOINT makes launchBrowser() connect to that server instead of starting a browser. The server must have the playwright version of the root package.json catalog. The tests read @k2b/ui from packages/ui/dist, which --browser does not rebuild, so build it first:

bun run --cwd packages/ui build
version=$(bun -p 'require("./package.json").workspaces.catalog.playwright')
docker run --detach --name playwright-webkit --network host --init --ipc=host \
  --user pwuser --workdir /home/pwuser "mcr.microsoft.com/playwright:v$version-noble" \
  npx -y "playwright@$version" run-server --port 3333 --host 127.0.0.1

TEST_BROWSER=webkit TEST_BROWSER_ENDPOINT=ws://127.0.0.1:3333/ \
  bun run test --browser --filter packages/ui

docker rm --force playwright-webkit

Host networking lets the browser reach the servers that tests start on 127.0.0.1. On a Mac, Playwright's WebKit runs without the image: install it with ./packages/ui/node_modules/.bin/playwright install webkit and leave TEST_BROWSER_ENDPOINT unset.

When a WebKit page stops answering

WebKit on Linux plays media through GStreamer. Stopping a <video> or <audio> element while its source still waits for data can deadlock the page: the main thread waits in a GStreamer state change for the source thread, which waits for the main thread. The page never answers again, however long a test waits. It is a race, not a slowdown: in CI, the other tests of the suite ran at their usual speed around the one that hung. An island that discards a server-rendered <video src> when it mounts stops it this way, so media gets its source only in the browser, as Islands describes. This deadlock made the Spaces item detail video test time out in CI.

Apart from this deadlock, WebKit in the Playwright image stalled the main thread of a fresh page for 15 to 100 seconds, but only on a development machine with 24 CPUs and a load average of 180 or more. Run WebKit tests on a machine with spare cores.

The Spaces item detail suite gives its page 15 seconds to become interactive. When that fails, the error reports whether the page's main thread still answers a short probe, the status of each island module, page errors, failed requests, and how busy the host's CPUs were, how much time the hypervisor took from them, and how often a task waited for one. A main thread that answers while an island module is missing, answered with an error, or did not mount points to a product or test bug. A main thread that does not answer is blocked by a script, a media deadlock, or, on saturated CPUs only, load. Its backtrace shows which. While the page hangs, print the main thread of every WebKit page process from a container that shares the browser container's processes:

docker run --rm --network host --pid=container:playwright-webkit \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  "mcr.microsoft.com/playwright:v$version-noble" bash -c '
  apt-get update -qq && apt-get install -y -qq gdb > /dev/null
  for pid in $(pgrep -x WPEWebProcess); do
    gdb -q -batch -p "$pid" -ex "set sysroot /proc/$pid/root" -ex "thread apply 1 bt 16"
  done'

A main thread in gst_pad_stop_task under gst_element_change_state is the media deadlock.

When a page never fires load

A navigation that commits but never fires load, or never commits, waits for a response that does not finish. The navigation diagnosis lists that response under "Requests still open". When the test server wrote the whole response, look at the sockets on the server's port while the page waits:

A test server can record what it receives and answers with recordTestServer() from packages/ui/test/browser.ts: wrap its fetch with record, call listen with its URL, and stop when it stops. The navigation diagnosis then prints, under each request that is still open, whether the server never received it, received it without answering, or answered it. The Chat harness in packages/chat/src/frontend/ChatWorkspace.browser-fixture.ts does this.

sudo ss -tnoiepm state all '( sport = :PORT or dport = :PORT )'
grep -A1 '^TcpExt' /proc/net/netstat

A browser socket with a receive buffer of about 1 KB (rb in skmem), no unread data, and drops (d), next to a server socket that waits on a 2 KB window, is a collapsed receive queue. Rising PruneCalled, TCPRcvQDrop, and BeyondWindow counters confirm it. On Linux 6.17, the kernel of the CI runners, this happens when the busy browser reads a large page, bundle, or font slowly. The receive buffer then shrinks, and the connection never recovers (#1367). Every CI job that installs dependencies sets an 8 MiB default receive buffer through .github/actions/install. When browser tests stall like this on a development machine, apply the CI setting until the next reboot:

sudo sysctl -w net.ipv4.tcp_rmem="4096 8388608 33554432"

When WebKit reports an internal error

WebKit encountered an internal error is the error that WebKit gives every load still open when its network process ends. The navigation in progress and the requests of the old page then fail together, the next test gets a new network process, and it passes. Offline emulation with context.setOffline(true) gives the same error without a crash. The webkit job in CI records every process that crashes on the runner, also in the Playwright container, with .github/process-crashes.sh. At the end of the job it lists each crash as an error with its time, process name, and signal. A WPENetworkProce crash at the time of the failed test confirms the cause.

For a backtrace, let the kernel write core files to a directory that the Playwright container mounts. Start the container with --ulimit core=-1 -v /tmp/cores:/tmp/cores after this setup:

sudo mkdir -m 1777 -p /tmp/cores
sudo sysctl -w kernel.core_pattern=/tmp/cores/core.%e.%p

Then read a core in the same image. Copy its libraries before installing gdb, because the install can update the C library, and gdb then reads the wrong symbols:

docker run --rm -v /tmp/cores:/cores:ro "mcr.microsoft.com/playwright:v$version-noble" bash -c '
  mkdir -p /orig/usr && cp -a /usr/lib /orig/usr/lib && ln -s usr/lib /orig/lib
  cp -a /lib64 /orig/lib64 && ln -s /ms-playwright /orig/ms-playwright
  apt-get update -qq && apt-get install -y -qq gdb > /dev/null
  for core in /cores/core.WPENetworkProce.*; do
    gdb -q -batch -ex "set sysroot /orig" -ex "bt 30" \
      /ms-playwright/webkit-*/minibrowser-wpe/bin/WPENetworkProcess "$core"
  done'

In #1431, WebKit 26.6 of Playwright 1.63.0 crashed its network process in about one of seven runs of the Notebooks graph test file in CI. Each crash came in a Back from a note page that cancelled the page's font load. The process aborted after it used an object that was already released, mostly in g_object_ref under soup_session_send_async. That is the libsoup 3.6.5 double release of microsoft/playwright#42803. Playwright 1.64.1 bundles libsoup 3.6.6 with the fix.

Engine differences

Write each test so that it checks the same behavior in both engines. Where they legitimately differ, branch on browserName and say why next to the branch:

• Playwright drives WebKit's touch input only as a whole tap. A test that holds or moves a finger, or that changes the default font size, needs Chromium's DevTools protocol. Such a test is skipped in WebKit with test.skipIf(browserName === "webkit") and a comment that names the reason.
• Input.dispatchTouchEvent stamps an event without timestamp when its call arrives, and Chromium flings a touch scroll at the speed it reads from these stamps. On a loaded machine several moves arrive at once and the list flings far further. A test that scrolls by touch and then touches something in the scrolled list stamps its events one 60 Hz frame apart and stamps the lift 100 ms after the last move. Chromium treats a finger that rests 80 ms or longer before it lifts as stopped, so the list stops without a fling. It moves about as far as the finger did, give or take a few pixels, so assert a bound rather than an exact scroll position.
• Input.dispatchTouchEvent returns once the browser has passed the touch on, not once the page has handled it. With the page's main thread busy, the call returns while the page still has to dispatch the touch and its pointer events. Before a test reads what a touch changed, it waits for the page's own record of that touch: a one-shot capture listener for the last event, such as touchend or touchcancel, installed before the touch is sent, then page.waitForFunction for that record. swipe() in signature-input.browser.test.ts does this.
• WebKit matches forced-colors: active under emulation but has no forced colours mode that repaints author colours.
• WebKit does not support reading-flow yet, so focus and VoiceOver keep the source order where a grid or flex layout shows elements in another order. Put elements in the source in the order they are shown, in every layout, so that a focus order test expects the same order in both engines.
• WebKit stores a document.cookie write in its network process asynchronously, so context.cookies() right after a key or click can still return the previous value (#1496). Read cookies that page scripts write with storedCookie(page, name) from packages/ui/test/browser.ts. It waits until the browser store agrees with the document and returns the full cookie with its attributes. The cookie-store check enforces this.
• Playwright's WebKit draws overlay scrollbars that reserve no gutter.
• Playwright's WebKit ignores the charset of a routed response. A page that a test serves names it with <meta charset="utf-8">, as every Cloud page does.
• Chromium's phone emulation widens window.innerWidth to content that is wider than the screen. Measure sideways overflow against document.documentElement.clientWidth.
• Playwright can read and type into a page before its first frame, and WebKit can hold that frame back while a page loads or the machine is busy. A dialog places its initial focus in the next frame after it opens; until then, keys go to the element the browser focused first, which is the content's autofocus element or else its first focusable element. A test that types into a dialog right after it opens first waits for the dialog's content and then for one frame: await page.evaluate(() => new Promise((resolve) => requestAnimationFrame(resolve))). Waiting for the content alone is not a frame: a test that read focus after it failed in WebKit CI (#1456). Where keys must reach the target at once, as in the Notebooks attachment preview, the target carries autofocus, and the test reads focus in the same page.evaluate that opens the dialog, before any frame can pass.
• Neither engine shows a PDF in a frame. Chromium's headless shell reports navigator.pdfViewerEnabled === false, so PdfPreview shows its missing-viewer hint instead of the frame, and an automatic preview requests nothing; Playwright's WebKit reports a viewer but never loads the document, so the loading state stays over the frame. A test therefore defines pdfViewerEnabled on Navigator.prototype in an init script: true where it measures the frame, false where it checks the hint. Where it needs the loaded document, it dispatches the frame's load event itself.
• Fonts differ between machines and engines. A test that expects text to wrap or fit uses text that is clearly too long or clearly short enough.
• Playwright's WebKit on Linux can stop a video for good when play() arrives during a seek or right as it ends. The video reports playing, often then waiting and stalled, and its time no longer advances, although the whole video is buffered. A loaded machine makes it more likely. A media fragment such as #t=0.001 starts a seek too: WebKit starts it once the first frame has loaded, Chromium already with the metadata. VideoPlayer starts a posterless address without a fragment of its own at #t=0.001, and a renewed address at the point where it continues. A test that plays or seeks a video waits until it shows a frame and no seek is under way, !video.seeking && video.readyState >= 2, and then calls play() itself. Played that way, the video did not stall in several hundred runs on a loaded machine; play() from the page's own handlers as the seek ended, or one frame later, stalled in 7 of 520. Because playing also comes before a stall, a test that checks playback waits for ended. A test also lets each seek end before it starts the next one: seeks that overlap can end at the first one's point instead of the last one's. After a renewal, VideoPlayer plays on by itself once the metadata has loaded, in WebKit during its seek; moving that play() to the end of the seek did not measurably make it stall less often. Renewed playback can therefore stall in Linux WebKit, so its test waits for ended in Chromium and only for playing in WebKit.

Replace modules in tests

mock.module replaces a module for the whole Bun process. Without --isolate, bun test runs all files of a suite in one process in file-system order, so a module mock leaks into every file loaded after it and the result depends on the machine. A test file that calls mock.module at the top level therefore needs a package test script with --isolate, or it runs the mocked part in a child process, as packages/notebooks/src/ws-lifecycle.test.ts does. Prefer the child process when --isolate would make the package suite much slower. bun run check fails on a top-level mock.module in a suite that shares one process.

The same isolation rule applies when a test installs a top-level spyOn, imports a module, and restores the spy before running its tests. The imported module can retain the replacement after restoration. If another file imported that module first, the replacement never takes effect. Even a skipped integration suite evaluates its static imports, so it can trigger this failure. The module-mocks check also rejects these import-time spies in shared suites.

Run integration tests

Disposable PostgreSQL containers in CI, nightly, release smoke checks, and the local compose.sync-test.yml and compose.scale-test.yml fixtures keep their data on a tmpfs capped at 4 GiB, with fsync, synchronous_commit, and full_page_writes disabled. Their data is lost when the container stops; they do not exercise crash durability. These settings reduce storage work during tests but do not establish the cause of sporadic slow statements. CI samples the whole cluster's tmpfs usage every 30 seconds during integration tests, including WAL, to check capacity. Keep these settings confined to disposable containers: a test database name does not make a shared development server safe to reconfigure.

Integration tests gate themselves on CLOUD_TEST_* variables through scripts/fixtures/test-infra.ts, which is loaded as a bun test preload:

The object-store suite uses a disposable Garage 2.3 instance. Start only that fixture with docker compose -f compose.test-s3.yml up -d --wait; its Compose file declares the test-only key, secret, bucket, and loopback port. Export those values as the five CLOUD_TEST_S3_* entries above, then run bun run test --integration --filter object-store. Stop it with docker compose -f compose.test-s3.yml down --volumes. CI and nightly do this automatically. Missing S3 configuration skips the suite; an endpoint with missing credentials, region, a bucket without the -test suffix, or unusable storage fails. The suite also needs the database and Valkey test targets. It writes these connection values directly to the encrypted Core settings and restores them afterward; production S3 environment variables do not exist.

The ports are the development stack's defaults. If you moved them with CLOUD_DEV_POSTGRES_PORT, CLOUD_DEV_TEST_VALKEY_PORT, CLOUD_DEV_NATS_PORT, or CLOUD_DEV_CLICKHOUSE_PORT (Change host ports), use the same ports here.

A suite runs when its variables are present and fails loudly when the target is unreachable. When a variable is absent, the suite is skipped and the matching runtime variables (NATS_SERVERS, GOTENBERG_URL, and so on) are removed, because unset is the off switch for those clients. DATABASE_URL and REDIS_URL instead point at a closed loopback port (postgres://127.0.0.1:1/unset, redis://127.0.0.1:1): Bun's default sql and redis handles would otherwise dial localhost, so an ungated test that reaches for them fails with a connection error instead of touching the stack configured in .env.

CLOUD_TEST_CLICKHOUSE_URL is an admin URL, including URL-encoded userinfo, for example http://cloud_ci:cloud_ci@127.0.0.1:8123 for the throwaway CI service. It has no runtime alias: Pulse gets its store connection from app settings. The gate checks /ping within three seconds. An unset target skips ClickHouse suites; a configured but unusable target fails loudly. Diagnostics redact URL passwords for every infrastructure kind.

ClickHouse tests use one pulsetest_<8 hex>_ prefix per process. Generated setup SQL grants the service user only that database wildcard. When loaded with a configured target, scripts/fixtures/test-clickhouse.ts registers its own afterAll hook to remove exactly that prefix's databases, users, settings profiles, roles, quotas and row policies. Cleanup validates the prefix before dropping anything and leaves objects outside it alone. Use a dedicated test instance: checks deliberately probe instance-wide admin privileges and create temporary users. CI and nightly provide ClickHouse 26.9.19.9 for these suites. See development setup for local credentials. Installation tests in pulse/src/migrate.integration.test.ts keep their own disposable Postgres database and run in the integration runner's one-file process, as the previous schema test did.

Run integration tests through bun run test: the infrastructure preload selects the explicit targets and initializes the platform's native SQL and Redis clients. For a direct single-file run, use BUN_OPTIONS=--no-env-file and the same CLOUD_TEST_* targets. tests/integration/test-infra-redis-binding.integration.test.ts proves that the platform Redis client reaches the explicit Valkey target.

bun run test keeps the checkout's .env away from tests. The file configures the development stack, for example APP_URL for a stack served under another address, and its values would replace the defaults the tests expect: the OAuth issuer would become the stack's public address. The runner therefore starts itself with --no-env-file and hands the flag to every test process through BUN_OPTIONS, which a Bun process started by a test inherits with the rest of the environment. Tests see the same configuration as in CI and in worktrees, which have no .env; variables you export in the shell still reach them.

Package scripts receive the fixture preload through the same BUN_OPTIONS, so a Bun process that such a test starts maps the CLOUD_TEST_* targets again. Give that process another database through CLOUD_TEST_DATABASE_URL, not only DATABASE_URL. A DATABASE_URL that names the test database with its own application_name stays unchanged, so a test can find the sessions of the process it started. To run a production script with exactly the variables an operator would set, pass that environment without ...process.env and start the script with --no-env-file.

A direct bun test loads the dotenv files of its working directory. With a CLOUD_TEST_* variable set, the fixture refuses such a process when a file like .env exists there. Start a direct run from that directory with BUN_OPTIONS=--no-env-file bun test …: unlike bun --no-env-file test, the variable also reaches the Bun processes that tests and package scripts start.

Use {checkout} in the database name to give each worktree its own test database. It expands to the checkout directory's name in lowercase, with non-alphanumeric characters replaced by underscores, repeated underscores collapsed, and leading and trailing underscores removed. The slug is shortened so the full PostgreSQL database name fits within 63 bytes. For example, cloud_{checkout}_test becomes cloud_fix_cache_test in a checkout named fix-cache. A URL without the token keeps its existing database name.

The expanded database name must end in _test. Integration tests create and delete rows; the fixture refuses any other name. Never point them at the development database. bun run test --integration connects to the postgres maintenance database on the same server and creates the target database if missing, before probing it and running migrations. The configured PostgreSQL role needs permission to create databases for templates and private fixtures, even when the shared target already exists.

Use the token when running tests from several worktrees: migrations are forward-only, so sharing a database lets one branch's schema changes leak into another branch's tests. Every suite in a checkout shares its target database, and it keeps what earlier runs left behind. A test that asserts on a database-wide listing with a limit, such as the oldest candidates of a background job, runs in a private database from useMigratedDatabase() in scripts/fixtures/test-infra. Call drop() in afterAll or finally.

The bootstrap also prepares an immutable PostgreSQL template with Core and every application's migrations, using the same migration entry point as the shared target. createMigratedDatabase() clones it without binding Bun's default SQL client; useMigratedDatabase() also binds that client before its first query. Mail's integration preload uses the clone before loading its runtime. Ordinary file setup must not replay Core setup after cloning. createDisposableDatabase() and useFreshDatabase() still create empty databases for schema migration tests and suites that build stand-in tables.

Bun cannot stop the body of a test that timed out: its statements keep running, or its transaction keeps holding locks, and every later hook would wait on them. In a private database that the process owns, nothing else runs, so after each test the fixture ends that leftover work: a statement still running for more than 250 ms is cancelled, and a transaction left open is ended. It logs each with its statement and blocking processes, so the report also names what held the timed-out test. useFreshDatabase() and useMigratedDatabase() mark their database as owned; a preload that binds its own private database calls ownTestDatabase(name), as Filesv2's does. The shared test database is never touched this way, because other processes may use it at the same time.

With a database target configured, the preload observes test and per-test hook callbacks before they are registered, including parameterized tests. If the guard cancels or terminates work after a callback timed out, its late database rejection is reported as a guard warning instead of an unhandled error between tests. The original timeout still fails. SQL errors and cancellation during a running callback still fail normally. Private-database suites run serially; concurrent tests must not share a database that the guard owns. tests/integration/test-infra-cancellation.integration.test.ts checks active statements, timed-out hooks, idle transactions, and recovery of the next test.

Templates are keyed by a content hash of workspace runtime sources and assets, the lockfile, workspace definition, and migration fixture code. Runtime inputs are hashed conservatively because migrations import shared helpers and assets; ordinary *.test.*, *.spec.*, and *.test-fixture.* edits do not invalidate them. Other runtime-adjacent fixtures are included conservatively. A PostgreSQL session advisory lock serializes builders across processes and shards. A template is published only after migrations finish and its connections close, and then refuses direct connections. An interrupted build is replaced by the next builder. Existing clones keep their schema when sources change. The runner holds a shared advisory lease until all its files finish, and each file holds its own lease. The last user drops the template; the next run also removes marked templates and builds whose processes died without releasing them. Cleanup takes exclusive advisory locks and skips active runs and builders. Templates are never cloned from the configured target's test data.

Run through bun run test --integration to keep a cold template build outside the file's 30 s hook budget. The runner hashes sources once and forwards that identity to its children; file hooks only acquire a lease and clone the database. A direct file run without that runner may need to build the template first. No timeout extension is needed. The fixture regression in tests/integration/test-database-template.integration.test.ts checks that a migrated private database is ready within three seconds, builders serialize, and a migration source edit produces a new template.

Each integration file must declare the settings its requests and assertions need. For example, call useTestSettings({ "app.url": "http://localhost:3000" }) from @k2b/cloud/services/settings/settings.test-fixture inside the gated suite. It restores stored values and removes previously absent rows after the suite, invalidating the Valkey and identity configuration caches. A suite that chooses values during setup, such as a temporary server's URL, calls preserveTestSettings(keys) before registering its setup hooks. Include every shared setting the suite changes; a fresh process does not reset the database or Valkey between files.

The same rule holds for NATS. The development broker keeps the application streams in the DEV account, which accepts connections without credentials, and bounds tests in the TEST account (Configure local NATS accounts). CLOUD_TEST_NATS_CREDS_FILE names the .creds file of the test identity, .local/nats/test.creds in the checkout that runs the stack, also when the tests run from a worktree. It becomes the runtime NATS_CREDS_FILE; tests never use one from the environment or .env. Tests that open their own connection use connectTestNats() from scripts/fixtures/test-infra; bun run check fails when test code connects without credentials. Both connectTestNats() and the fixture's NATS check refuse a connection that lands in DEV. A broker without accounts, like the one in CI, needs no credentials file.

Valkey separates tests from the stack by instance. Cache keys such as settings:<key> carry no namespace, so a shared Valkey would hand the running stack's cached settings to the tests and the tests' values to the stack. The development stack therefore runs a second Valkey only for tests on port 6380; the one on 6379 belongs to the applications. A database index is no way out, because the fixture refuses one. In CI, the gate's Valkey serves only the tests, so it stays on 6379.

CLOUD_TEST_DATABASE_URL='postgres://postgres:postgres@127.0.0.1:5432/cloud_{checkout}_test' \
CLOUD_TEST_NATS_SERVERS=nats://127.0.0.1:4222 \
CLOUD_TEST_NATS_CREDS_FILE=/path/to/cloud/.local/nats/test.creds \
CLOUD_TEST_VALKEY_URL=redis://127.0.0.1:6380 \
bun run test --integration

--integration first prepares the test database by running Core setup and every application migration (idempotent, a few seconds), then runs only the integration files, one fresh global per file. --shard <n>/<total> splits the suites across parallel jobs, balanced by measured suite time, which is how the gate runs them. --max-minutes <m> fails a run that takes longer than m minutes. --exclude <name> leaves out suites whose name or path matches, and --filter <name> keeps only those. Repeat --filter to run suites matching any of the given names or workspace paths, each suite once:

bun run test --filter packages/core --filter packages/cloud

--exclude removes matches from that union before sharding. Run bun run test --help for all runner options.

Suites that exercise sessions, tokens, or access against a real Core identity authority additionally import scripts/fixtures/authorization-preload right after the gate; it starts the authority only when database and NATS targets are configured. Bun runs file-level afterAll hooks in the order they were registered, so the preload's teardown, which stops the authority and the process Sync, runs before the file's own file-level afterAll. Put teardown that still needs either, such as stopping a live outbox, in an afterAll inside the suite's describe; Bun runs it when the suite ends.

A new integration test needs no configuration of its own: import the appropriate suite gate from scripts/fixtures/test-infra.ts, use the resolved runtime variables, and conventionally name the file *.integration.test.ts. The shared gate import makes the file discoverable; the filename alone does not.

bun:test does not stop a timed-out test body. An integration test ends the work it starts: teardown cancels and awaits every sync, job, or loop the test started before it restores shared stand-ins. A stand-in shared by several tests routes calls to the test that owns them and rejects calls from an ended test; see packages/mail/src/service/remote-changes-fixture.ts.

Keep a test that needs one of these targets out of the Playwright suites (*.browser.test.ts): the nightly browser job sets no CLOUD_TEST_* variables, so the test would only ever be skipped there.

When a package has a test:integration script, bun run test --integration runs that script instead of the package's integration files, so the script must run every one of them. Assistant's runner, for example, gives its artifact service suite disposable PostgreSQL and rsql containers and runs every other Assistant integration file, such as the Code Mode PDF test against Gotenberg, against the CLOUD_TEST_* targets.

Mail's entry point reuses the root runner's infrastructure-gate discovery, including gated files without an .integration suffix. It also retains the explicitly opted-in large-mailbox performance and IMAP connector suites. All selected files run with --isolate, so the Mail preload gives each file its own database clone and Sync runtime; unit and browser files run through the normal test jobs instead. Inspect the selection without starting infrastructure with bun run --cwd packages/mail test:integration --list.

Sync namespaces on the test broker

Integration tests share one NATS JetStream account, locally the TEST account of the development broker, and @k2b/sync keeps them apart only by namespace. JetStream reserves the full size of every stream against the account, 1 GiB per topic, job, and dead-letter stream by default, so each namespace a test leaves behind blocks 1.5 to 2 GiB of the account's limit. The fixture therefore cleans up after every test process:

• Each test process gets its own namespace, test-<8 hex digits>, as SYNC_NAMESPACE. It replaces any value from the environment or .env.
• A test that creates its own Sync instance takes its namespace from testSyncNamespace("<label>"), exported by scripts/fixtures/test-infra. The name is derived from the process namespace, so the test needs no cleanup code of its own. bun run check fails when test code passes createSync any other namespace.
• When the process's tests are done, passed or failed, the fixture deletes the streams of the process namespace and of every namespace derived from it.
• A killed or timed-out process never reaches that cleanup. bun run test therefore first deletes every test namespace whose streams were all created more than an hour ago, longer than any test process runs.

The cleanup waits for all test files of a process only when Bun loads the fixture as a preload: under bun run test, and under a direct bun test started from the repository root or from a package whose bunfig.toml preloads the fixture, like Grids. A direct bun test from another package directory loads the fixture inside its first test file. The cleanup then runs before that file's own afterAll hooks and misses every later file, so what those files leave waits for the sweep of a later bun run test.

The cleanup selects streams only by their sync.namespace metadata and never touches a namespace that does not start with test-, such as dev. Do not give an installation that uses the test broker a test- namespace.

A preload that keeps a Sync instance running until its own afterAll, like Mail's, uses a separate test- namespace and deletes it after draining: the fixture's cleanup runs first, while that instance is still live.

Caller addresses and rate limits

Per-IP rate limits count in the shared test Valkey, across every test file, test process, and concurrent run. A fixed address or a small pool lets unrelated tests spend each other's budget, so a route that allows one request per minute answers 429 at random. rateLimit() counts a request by user only when it carries a Cloud session. Every other request, whether anonymous or authenticated with an API key, an OAuth access token, or a test's stand-in bearer token, counts by address. Without x-forwarded-for, that is the address unknown, which every such request shares.

Give each caller its own address, for example from uniqueCallerAddress() in scripts/fixtures/caller-address, a fresh address in the IPv6 documentation range 2001:db8::/32. Reuse one address only when the test exercises the limit itself.

Fixture short IDs

Raw fixture inserts bypass the service's collision retry. Take short IDs from testShortId() in scripts/fixtures/test-short-id, passing the column's length when it differs from the default six characters. Its counter is deterministic per process; the o mark never appears in readableId() output, so fixture IDs stay apart from service-generated IDs.

Never cut a random UUID or Math.random() to short-ID length: the small space repeats and fails a unique index at random. bun run check flags this with the test-short-ids rule.

Measure the public-site example

bun scripts/lighthouse-public-site.ts builds packages/cloud/examples/public-site in production mode and serves it as http://public-site.localhost:<port>/ on a free port through the gateway's own request handler (host decision, protections, compression) to the app's app.public surface. A fixed snapshot replaces only the domain table and the registry. It then runs Lighthouse 13.6.0 on Playwright's Chromium with mobile emulation and simulated throttling, prints the SEO and performance scores of / and /de/, and exits with 1 below SEO 100 or performance 95. Reports stay in .local/lighthouse-public-site/.

The run uses the integration variables (CLOUD_TEST_DATABASE_URL, CLOUD_TEST_VALKEY_URL, CLOUD_TEST_NATS_SERVERS, and CLOUD_TEST_NATS_CREDS_FILE where needed): the measurement server starts like the gateway process, and its telemetry publishes to its own test namespace, which the script deletes. Run it before a release that changes the gateway's app-domain path or app.public; it is not part of the gate.

Request cache checks

The request-cache suites modify global settings and cache keys, so they run through their own runner with disposable containers. See Verify request caches.

What CI runs

The pull request gate runs bun run check, bun run test, and the integration suites against PostgreSQL 17, NATS JetStream, Valkey, and Gotenberg. Path-filtered jobs, such as the Grids certification, the WebKit browser tests, the documentation checks, and the image boot smoke, run when their inputs changed. The gate runs on the pull request and again in the merge queue, and its result is authoritative.

Unit tests run in nine shards, WebKit tests in eleven, and integration tests in five. Each mode uses its own measured seconds per suite in scripts/test-weights.json, assigning the longest suites first. An unknown suite counts as the median of its table. The browser table contains WebKit measurements; Chromium runs inside the unit suites. Equal weights use the suite name to order suites; the least-loaded shard wins, with suite count and shard index breaking load ties. Integration keeps the Grids exclusion; Grids certification runs in its own path-filtered job. Every integration shard gets fresh service containers, and gate requires the whole matrix to succeed.

The unit, WebKit, integration, and Grids jobs run on the runner that the repository variable CI_HEAVY_RUNNER names, and on ubuntu-24.04 when it is unset. Maintainers set it to a larger GitHub-hosted runner when standard runner capacity is short; that runner is billed per minute. Deleting the variable returns the jobs to standard runners without a pull request.

Each shard prints its predicted and actual time. Its runner stops an active suite and fails with an annotation at the 20-minute budget, before the 30-minute job limit. A shard that finishes above the 15-minute target stays green and gets a warning annotation that asks for a weight refresh. The budget also includes NATS housekeeping and integration database bootstrap. To refresh the weights, run bun scripts/test-weights.ts --run <id> <id> <id> with recent completed runs, including slow or failed runs, using gh-agent authentication; add --mode integration to refresh only integration timings. The script keeps the maximum measurement in the selected runs, including completed suites from failed jobs. It retains previous measurements for suites an aborted job did not finish and removes suites no longer in the checkout. An interrupted suite's elapsed time is only a lower bound, so it does not replace a complete measurement. Later refreshes replace observed weights, so an exceptional slow runner does not increase them permanently. The script plans the current checkout, excluding Grids from integration. It reports the smallest shard count whose predicted slowest shard fits 15 minutes. If one suite alone exceeds that target, split the suite: test:part:<name> scripts in the package become the suites <package> <name> in place of its test script, and together they must run the same files. Commit the refreshed weights and adjust the matrix together; keep the 20-minute execution budget unchanged. Refresh after browser suites grow or timings drift, even when the jobs still pass. An outdated table can balance predicted times while leaving actual times far apart.

The runner tests check that every unit, WebKit, and integration suite is assigned exactly once and each predicted shard fits 15 minutes. They also cover deterministic assignment and distributing unmeasured new files. These checks validate the measured plan rather than a future runner's speed. Importer tests cover failed jobs, unexecuted suites, and replacing old weights when the selected runs are faster. Actual times still depend on the runner: a whole job running several times slower cannot be bounded by historical file weights. Inspect a selection without starting services with bun run test --integration --exclude grids --shard 1/3 --list.

One third of the job limit stays in reserve because setup and GitHub's job finalization count against it. Finalization took up to about 5 minutes during a GitHub Actions incident.

The packed job packs @k2b/cloud, @k2b/ui, @k2b/ssr, and @k2b/editor with npm pack, using the same file list as npm publish. It installs them in a fresh project outside the workspace, typechecks the public source, builds a page with an island using the installed production build script, and starts the app against disposable services. Normal pull requests use the workspace tarballs. On a release pull request, whose diff changes .release-please-manifest.json, it runs with --release, also in the merge queue. Older sibling pins still come from npm. For a published pin equal to the workspace version, it compares the tarballs’ TypeScript files and public package metadata before choosing npm. If Cloud’s own version is already published, the checkout is a snapshot: changed sibling sources use current tarballs. If Cloud’s version is unpublished, changed same-version sibling sources fail with “source changed without a version bump”. Refresh a stale release PR, or land a scoped fix/feat commit for the affected package so release-please bumps it. This is a byte comparison, so shipped comments and compiler output changes also count. Matching published types come from npm, and unpublished sibling pins must match the workspace version. Registry or comparison failures fail the check. Run it locally with Docker and npm available:

bun run --cwd packages/ui build
bun --no-env-file scripts/heavy.ts bun scripts/check-cloud-packed-consumer.ts --release

The release-body job parses the live release PR body with the release-please parser pinned in main.yml, including in the merge queue. It fails if a bumped component is missing, has another version, or swallowed another section. It also tests historical fixtures when the guard, its fixtures, or the parser pin changes. The check job runs bun scripts/check-release-pr-body.ts --message on pull requests and again in the merge queue. It reads the live PR title and body to refuse HTML-like markup in the parts that release-please copies into release notes. After fixing the text, re-run the failed check job to check it again. See Release process.

Jobs of the CI and nightly workflows install workspace dependencies through .github/actions/install. It restores the Bun install cache that main.yml writes for the current bun.lock. With that cache, bun ci needs no registry. scripts/ci-install.ts stops an install that takes over two minutes and retries a stalled install once. Any other install failure fails at once. A pull request that changes bun.lock installs from the registry until it merges. Before the install, the action sets the default TCP receive buffer to 8 MiB, so a slow browser cannot stall a test server's response; see When a page never fires load. The ci-install check rejects direct dependency installs in workflows. Only the release workflow keeps a plain bun ci, because it can finish an older release from a commit without the install action.

The standalone cld test compiles the release target. On Linux x64, Bun downloads that target's runtime into its install cache on first use. The unit job and the nightly ubuntu-next job fetch it in a named step before the tests, so the download runs outside the test's 60 s budget. When the test fails or comes within five seconds of its timeout, it prints a trace of every step and child process: pid, duration, exit status, output sizes, whether each stream ended, and the last output. This shows which step stalled.

Locally, run the typecheck and the focused tests of each package you touched, for example:

bun run --cwd packages/grids typecheck
bun run test --filter packages/grids

The focused run includes the package's behavior tests and its browser tests in Chromium, so it also covers a UI change. When you changed server code, add bun run test --integration --filter packages/grids with the CLOUD_TEST_* targets from Run integration tests. A documentation change runs the checks in Document Cloud core changes. Run the full bun run check and bun run test only when you change the repository checks, the build or test system, or many packages.

The Grids jobs run Gotenberg at debug log level, so the service log at the end of a failed job contains LibreOffice's own error message for a failed PDF/A conversion.

An integration suite that starts its own container needs only Docker and its CLOUD_TEST_* targets, so it runs in the gate as well; the Files stable-reference suite starts a private Filegate this way, and the Assistant artifact suite a disposable PostgreSQL. The integration job pulls such images first through .github/pull-images.sh, with retries, because docker run does not retry a failed pull. When you add a suite like this or change its image, add the image to that pull step in .github/workflows/ci.yml and .github/workflows/nightly.yml. The mirror-images check cannot see which job runs a test file, so it does not report a missing image there; without it, the job pulls a Docker Hub image from Docker Hub.

Tests that read a rendered PDF call Poppler's pdftotext, pdfinfo, pdffonts, and pdfimages from PATH; locally, install Poppler (poppler-utils on Debian and Ubuntu). CI does not install Poppler on the runner: .github/poppler.sh install pulls a Poppler image pinned by digest with the job's other images, starts it without network access, and links the four tools into /usr/local/bin as wrappers that run each call in that container. Installing from the runner's apt mirror hung, and Homebrew failed when its API host formulae.brew.sh timed out. In CI, the tools can open only files under the checkout, the temporary directory, and $RUNNER_TEMP. A test that writes its PDF anywhere else fails there with Couldn't open file, although it passes locally. To move to another Poppler version, bump the image in .github/mirror-images.txt and the mirror reference in that script, as described under Where CI images come from. The image's publisher deletes its date tags after about eight months, so move the pin to a newer tag before that.

Scale acceptance

The nightly scale-acceptance job runs scripts/scale-acceptance.ts. It boots one gateway and two replicas each of Core, Mail, Chat, and Notebooks from compose.prod.yml and the test overlay compose.scale-test.yml, in its own Compose project with its own PostgreSQL, NATS, Valkey, and SMTP sink. It never uses the development stack or .env, and it removes every container, volume, and built image tag when it ends, also after a failure or a signal.

The gateway and all Core replicas start together. Once Core is ready, all replicas of Mail, Chat, and Notebooks start together. After readiness, the harness inspects every application container and requires RestartCount to be zero, reporting the container name and count on failure.

Steps pin requests to one replica through its own published port, except the Mail and Chat steps that explicitly use the gateway:

• a Chat message posted on replica A reaches a live socket on replica B;
• a Notebooks Yjs update sent through replica A reaches a client on replica B;
• one durable outgoing mail is delivered once while two Core replicas run;
• Mail compose opens through the gateway; acquiring again on B with the same editor returns A's lease;
• six Mail autosaves alternate between A and B without lost revisions or recovery copies;
• Mail heartbeats on B keep the lease while eight concurrent saves run on A;
• reloading Mail compose releases on one pod and acquires on the other;
• a Mail takeover on B reaches the old editor live on A and rejects its old token;
• a Mail attachment upload starts on A, resumes and finishes on B, and downloads intact on A;
• sending a Mail draft on B removes its editing lease everywhere;
• Mail A restarts gracefully while editing continues through B and the gateway; every routing gap fails the run, with request samples kept as evidence;
• Chat A stops gracefully and is removed, then a surge adds replacements, which must be ready with RestartCount 0, before the last original Chat replica stops. Gateway requests continue for at least 70 seconds after each stop. Registry gaps fail the run. Upstream-unavailable 502s and network errors during a stop and its next five seconds are counted separately as p4InFlight (RISK P4); every other failure fails the run;
• Core is killed with SIGKILL while it sends a mail, and the other Core replica delivers it exactly once after recovery, which takes about five minutes.

At the end, the run fails unless both replicas of every app are different running containers and each served at least one step, except for the explicitly killed Core and retired Chat containers. Retirement never excuses an unserved replica. The Chat stop and surge scenario adds at least 140 seconds. The suite runs nightly, not on pull requests, because it builds five images. Run it locally with Docker:

bun run --cwd packages/ui build
bun --no-env-file scripts/scale-acceptance.ts

The script takes a heavy slot itself. Add --image-tag <tag> to use published images instead of building them. --replicas 1 is the negative control: every step reaches the only container of each app, and the final check fails because replica B is replica A.

Where CI images come from

No workflow pulls from Docker Hub: its limit for anonymous pulls failed CI runs. Every Docker Hub image that CI, the release workflows, or the image builds use is pinned by digest in .github/mirror-images.txt and comes from the public mirror ghcr.io/k2b-dev/mirror/<name>:<tag>, which serves the same digest:

• Service containers, the docker/setup-buildx-action BuildKit image, the SBOM generator of the release image builds, and the # syntax= and FROM lines of every Dockerfile name the mirror reference, for example ghcr.io/k2b-dev/mirror/postgres:17-alpine@sha256:….
• Steps and test fixtures that start an image by its Docker Hub name, such as docker run nats:2.14.3-alpine, Compose, or the Files Filegate suites, get it from .github/pull-images.sh in the same job. The script pulls the mirrored digest and tags it with the Docker Hub name, so Docker finds the image locally and never contacts Docker Hub. A script may name the mirror reference instead, as the runtime recovery acceptance and .github/poppler.sh do.

The mirror job in ci.yml runs .github/mirror-images.sh before every job that pulls an image. It copies each listed image whose digest the mirror does not serve yet, unchanged, and checks that every listed digest can be pulled without credentials. Consumers pull by digest, so the mirror's tags are only labels. Registry requests are retried, and an image that still fails does not stop the others. The same workflow, mirror-images.yml, also runs weekly and on demand. A new package created by the first copy must be public; if the check reports that it is not, change its visibility in the package settings on GitHub once.

The Core image's DB-IP snapshot is the one mirror entry that is not a Docker Hub image: ghcr.io/k2b-dev/mirror/dbip-lite:<YYYY-MM>. It is referenced by tag and pinned by the gzip checksums in the Dockerfile. .github/mirror-images.sh copies it from DB-IP while DB-IP still serves that month. It holds only data and is pushed for linux/amd64; the Dockerfile takes that manifest with FROM --platform=linux/amd64, so source builds for other architectures can copy the same data. Published release images target linux/amd64 only.

The mirror-images check fails when a workflow or Dockerfile would pull a Docker Hub image directly, including through BuildKit's default SBOM generator, when a mirror reference does not carry the digest from the list, and when a list entry is no longer used.

To add or bump an image:

1. Look up the digest of its manifest list:

   bash

   docker buildx imagetools inspect postgres:17-alpine --format '{{json .Manifest}}' | jq -r .digest

2. Add or change the <name>:<tag>@sha256:<digest> line in .github/mirror-images.txt.

3. Use ghcr.io/k2b-dev/mirror/<name>:<tag>@sha256:<digest> in Dockerfiles, service containers, BuildKit, and scripts, or pass the Docker Hub name to .github/pull-images.sh before a step starts it. Run bun scripts/check.ts mirror-images; it names every reference that still carries the old digest.

The pull request mirrors the new digest in its own mirror job before its other jobs pull it. A pull request from a fork cannot write packages: its mirror job fails on a new digest until a maintainer runs the Mirror images workflow on a branch of this repository with the same list. Dependabot does not update Docker images in this repository; bump them through the list.

To try the copy against a local registry, log in to Docker Hub first: the script copies every listed image, which takes more manifest requests than Docker Hub allows anonymous clients.

docker run --detach --rm --name mirror-registry --publish 127.0.0.1:5000:5000 registry:2
MIRROR=localhost:5000/mirror .github/mirror-images.sh
docker rm --force mirror-registry

gate is the only required status check. It needs every other job in ci.yml and passes only when each one reports success, or skipped for a path-filtered job listed in its MAY_SKIP variable. Any other result fails it and names the job: failure, cancelled, timed_out, a job skipped behind a failed setup, and abandoned from a GitHub Actions outage. A path-filtered job skipped because changes did not succeed fails through the result of changes. The gate fails as well when it receives no job results.

The ci-gate check keeps needs and MAY_SKIP in line with the jobs. A job may have an if: only as a path filter: it needs changes, and its condition reads needs.changes.outputs. scripts/checks/ci-gate.test.ts runs the gate step from ci.yml against these results. Like the runner, it needs jq on PATH.

The gate also builds the production bundle of every application whose package changed, and of every application when packages/cloud, packages/ssr, packages/ui, or another shared input changed. The production build rejects code that development and the tests accept, such as a Bun builtin imported into browser code. Build one application the same way with:

NODE_ENV=production APP_ID=grids bun run packages/cloud/scripts/build.ts

bun run test runs the browser tests in Chromium. The gate's webkit job runs them again in WebKit in eleven shards with bun run test --browser, against the Playwright image as in Run WebKit locally. It runs when a workspace with browser tests or a shared input such as packages/cloud or scripts/ changed. At its end, it lists every process that crashed on the runner; see When WebKit reports an internal error.

CI and nightly start the Playwright server through .github/start-playwright.sh. It installs the exact catalog version with up to three attempts, then starts the server without an npm download. A failed start prints the container logs and recreates the container, with up to three attempts and a 120-second readiness wait each. The ci-runtime-downloads check rejects inline run-server commands in workflow steps; use the helper when adding a job. The standalone CLI test's Bun runtime prefetch also gets three attempts before the tests start. Each download attempt is limited to 120 seconds.

The nightly workflow repeats the integration suites against PostgreSQL 15, the oldest supported version, and runs the longer acceptance checks that are too slow for every pull request. Its browser-smoke job builds the Core, gateway, Accounts, Grids, Mail, and Chat images, boots them against fresh infrastructure, and runs the Grids, Mail, and Chat browser smokes against that stack. It runs independently of the Assistant browser suites, so a failure in one does not hide the other's result.

The Nightly grids job retains complete phase logs, JUnit reports, and verification environment metadata in a nightly-grids-certification-* artifact when certification fails, as the pull request Grids job does. Read these reports before attributing a phase failure to a specific test.

The smokes accept pending legal documents using the version rendered in the consent page's data-legal-version attribute; they must not read island props, whose encoding is internal.

Grids and Chat also receive --app-url for the public origin. When the browser reaches the stack over loopback, their smokes relay the application's session live socket with that public origin and the fixture's session cookie. The application's origin checks still apply. Grids reports only known refusal codes and close metadata; it does not print cookies, frame bodies, or arbitrary close reasons.

Read what a failed suite left running

After an integration suite fails, the runner prints Database activity: next to the suite's output: every statement still running on the test server, how long it has run, what it waits on, and which backends block it. A statement listed there outlived its test.

Such a statement cannot stall the suites after it in the integration and packed jobs: their PostgreSQL sets client_connection_check_interval, so it ends within seconds of its test process and logs connection to client lost. Find its plan in the slow statement log below only when it finished; otherwise the activity line has its text.

Read slow statements from a CI run

The test PostgreSQL in the integration, grids, and packed jobs logs every completed statement that took 250 ms or longer. For queries whose execution took as long, auto_explain also logs the plan, including queries that run inside functions. Use these entries when a test got slower in CI but not locally: they tell a slow database apart from a slow test process.

Find the PostgreSQL log in the job log:

• integration and packed: open the Stop containers step and look for Print service container logs: followed by the postgres container name.
• grids: open the Print the Postgres log and remove containers step and expand the grids-postgres log group.

To search it, download the job log and filter it, for example:

gh run view --job <job-id> --log > job.log
grep -n -A 20 'duration: ' job.log

Each entry starts with the time in UTC, the process ID, the database, and the application name:

• Suites that create their own database show it as <prefix>_<random>_test, such as mail_a931..._test. Suites that use the job's shared database show cloud_ci_test, cloud_packed_test, or grids_test; tell them apart by the statement text and the time. CREATE DATABASE and DROP DATABASE also run on the shared database.
• Most test connections set no application name and show [unknown]. Connections that an application opens through its declaration show cloud:<app>.

A slow query produces a statement entry and a plan entry with the same process ID. They can come in either order: for a prepared query that returns rows, the execute entry and its parameters come first; for an INSERT, UPDATE, or DELETE without RETURNING, the plan comes first.

2026-10-05 13:12:03.456 UTC [812] mail_a931..._test [unknown]: LOG:  duration: 903.530 ms  execute P...: SELECT ... WHERE mc.mailbox_id = $3 ::uuid
2026-10-05 13:12:03.456 UTC [812] mail_a931..._test [unknown]: DETAIL:  Parameters: $1 = '...', $2 = '...', $3 = '...'
2026-10-05 13:12:03.457 UTC [812] mail_a931..._test [unknown]: LOG:  duration: 903.211 ms  plan:
	Query Text: SELECT ... WHERE mc.mailbox_id = $3 ::uuid
	Query Parameters: $1 = '...', $2 = '...', $3 = '...'
	Nested Loop  (cost=0.84..16.90 rows=1 width=28)
	  ...

Not every slow statement has a plan. auto_explain times only the execution of a query, so DDL and other utility statements, such as migrations, CREATE DATABASE, DROP DATABASE, ANALYZE, or COMMIT, and queries that spent their time in parsing or planning show only the statement entry. The PostgreSQL 15 of the grids job has no Query Parameters line in its plans; read the parameters from the statement entry.

Compare the PostgreSQL time at the start of each entry with the GitHub timestamps of the test step's output. Ignore the GitHub timestamps in front of the PostgreSQL lines: the log is printed when the job ends, so they all show the time of that step.

• Entries in the slow window show which statements took the time. The plan shows the chosen strategy with estimated costs; a JIT: section means the query was compiled. The plan has no actual row counts or timings, so reproduce a suspicious plan locally with EXPLAIN (ANALYZE, BUFFERS).
• A statement that fails or is cancelled logs no duration. Look for ERROR: and FATAL: lines in the slow window, such as canceling statement due to statement timeout or terminating connection due to administrator command.
• No duration: and no ERROR: or FATAL: lines in the slow window, while the log still starts with the database initialization, mean that no statement took 250 ms or longer. The time went to the test process, the network, or many shorter statements.
• PostgreSQL logs checkpoints by default. A long checkpoint in the same window can point to slow disk writes on the runner.

A plan can also come from statistics that no longer describe a table. When autovacuum analyzes a table while earlier tests have left it empty, and a test then fills it faster than autovacuum analyzes it again, PostgreSQL plans as if the table were still empty. Such a plan changes with whether autovacuum ran in between. Skewed data or a condition whose selectivity PostgreSQL misjudges can look similar. Two signs in the plan point to stale statistics:

• A scan that runs once over the whole table, such as most Seq Scan nodes, estimates far fewer rows than the test inserted. On the inner side of a nested loop and in a subplan, an estimate counts the rows of one loop, so rows=1 for a lookup by key is normal there.
• An index is used without a condition on its first column, such as an Index Cond on only the second column of a two-column index. Each loop then reads the whole index.

To check a table locally, compare reltuples and relpages in pg_class and last_analyze and last_autoanalyze in pg_stat_user_tables with what the test inserted. A table with reltuples 0 and relpages above 0 is planned as empty however many rows it holds. If EXPLAIN (ANALYZE, BUFFERS) shows a different plan after an ANALYZE of the table, the statistics were the cause. A test that measures a query runs ANALYZE on every table it bulk-loads that the query reads before it measures, as packages/mail/src/service/view-counts.integration.test.ts does.

The log is bounded: Docker keeps two log files of 4 MB for each PostgreSQL container and drops the oldest entries when both are full. A full local integration run writes about 0.7 MB, most of it the text of large migrations that took longer than 250 ms. If the log no longer starts with the database initialization, entries were dropped. Parameter values are cut after 256 bytes.