Deployment requirements

Use this reference before deploying a fresh Cloud installation or adding an application. Choose the apps and the features you intend to use: a ready container does not prove that its mail, AI, storage, or PDF integration works.

This page covers the gateway and every built-in application that a plain docker compose up of production Compose starts, including Pulse. An application behind the unreleased Compose profile has no row until its release; see Add a built-in application. A standalone application can have additional requirements declared by its author. Check the documentation and configuration shipped with the exact release you deploy.

In the supplied Compose files, an app ID such as mail maps to service app-mail; the routing service is named gateway. The shared library, UI package, CLI and desktop development tools are not additional server apps in this service set.

Size Mail for large messages

Each Mail process loads message bodies and extracts attachment text within one processing budget of 256 MiB. Mail estimates the memory of each message or attachment from its size before it starts. Work waits in arrival order until it fits. A message or attachment larger than the budget runs alone. Mail never skips a message because of the budget. Large messages can take longer to appear, and the server answers pages and API requests while they wait. A body that a workflow or a person opens waits in the same order as background work, so it can take longer to load while large messages are processed.

The budget comes on top of the process's own memory. A running Mail process uses about 350–450 MiB. One large message can need more than the budget on its own: about six times its size plus forty times the size of its HTML part. A 40 MiB message with 6 MiB of HTML needs about 500 MiB.

Give Mail a container memory limit of at least 1.5 GiB. With 1 GiB, a burst of large messages can still exceed the limit, and the container is killed and restarted.

While the only Mail replica restarts, Mail pages and API requests return Bad Gateway until it is ready again. Where availability matters, run at least two Mail replicas from the release that contains the registry fix for #1561 and the Mail memory bound from #1571, each with the memory limit above. The number of replicas is your decision; see Scaling and shutdown.

Prepare for browser confirmation of new access

Deploy updated Core and OAuth together: start Core first, then OAuth, as in Start the new release and verify it. OAuth adds the database columns that Core reads for the confirmation, so new OAuth sign-ins fail until updated OAuth has started. New OAuth refresh families require confirmation from a browser sign-in in the last ten minutes, or a terminal Cloud Login approval bound to the exact grant. Existing refresh rotation keeps working without another browser sign-in. The browser flow of cld login always shows consent. Ensure the normal browser sign-in methods remain available for confirmation.

People must create personal API keys and passkeys in the browser. Automation that created personal API keys using OAuth tokens or API keys now receives 403 BROWSER_SIGN_IN_REQUIRED; create those keys in /me/developer instead. An older browser sign-in receives 403 REAUTHENTICATE and requires signing in again. Resource and standalone service-account keys are unchanged.

New browser-confirmed OAuth access sends a New access notice with email and browser delivery recommended. Keep the installation timezone and outgoing mail configured. Failed notice preparation is retried by Core maintenance in batches of at most 100 families; a notice failure does not fail a completed sign-in. See the migration steps.

Upgrade Core and Action callers before applications use typed Action reviews

Applications built with this release may declare Action policies and return per-call reviews or receipts. Upgrade Core and every application that calls Actions, at least Assistant and Capabilities, to this release or newer first. Older versions still discover the Actions through derived approval metadata, but reject the new response shapes. Existing apps need no migration in this release. See Action contract migration.

Prepare the common infrastructure

The repository's compose.yml is local development infrastructure, not a production storage or exposure policy. Its host-published ports, passwords, floating helper-image tags and named volumes are development defaults. compose.prod.yml supplies application services, not Postgres, Valkey, Filegate, Gotenberg, or Geo. Operators must supply the selected dependencies and connect them to the application network. The provided production ingress assumes an existing Traefik network and TLS configuration.

Back up Postgres together with the independent encryption secrets. Also back up Filegate's home/group storage when used. Application attachments in Mail, Notebooks, Grids and Spaces use Postgres; deploying those apps does not itself require Filegate or S3. Notebook S3 snapshots are an optional export, not a replacement for a Cloud database backup. See Secrets and persistent state. If the database uses optional BM25 ranking, the server you restore into must preload pg_textsearch; see Restore a backup without the library.

Releases up to 0.8.0 held session advisory locks, which a transaction pooler leaves behind on its pooled backends and which then reject writes as operation_busy. When upgrading such an installation, release them once by running SELECT pg_advisory_unlock_all() on each affected backend, or restart the pooler so that its server connections are recreated.

There is no universal CPU, memory, disk or database-connection sizing guarantee. Size for your app set, data volume, replicas and workload, and verify headroom with representative traffic before production exposure.

Reserve JetStream storage

JetStream reserves each stream's byte limit on every node that holds one of its replicas as soon as the stream exists, even while it stores almost nothing. At three replicas on three nodes, every node reserves every stream once. Unless max_file_store is set, a NATS server allows 75% of the disk space that is free when it starts. NATS refuses a stream that does not fit with insufficient storage resources available; when that happens during startup, the application does not start. Set max_file_store on every node to a size that its disk actually provides, and compare it with the reservation: the monitoring endpoint /jsz reports reserved_storage and config.max_storage.

Each Sync job, queue, and topic has a work or event stream and a dead-letter stream. Their byte limits come from the declaration:

• A job or queue declared without retention holds 256 messages at its payload limit, at most 1 GiB: 33 MiB at the default 128 KiB. Its dead-letter stream gets the same limit, so it reserves 66 MiB per replica. A typical Cloud job message stores about 500 bytes, so the budget holds tens of thousands of pending jobs. Past it, Sync discards the oldest pending messages, which is why durable state belongs in Postgres, from where recovery submits unfinished work again.
• A job, queue, or topic declared with retention reserves its declared maxBytes twice, unless it sets a smaller deadLetterRetention. The Grids record events and workflow queue, Mail invalidations, Contacts and Spaces events, and the Notebooks snapshot job limit their dead letters to 256 at the payload limit, as Sync does by default; for example, 17.6 MiB for grids:records and 3 MiB for mail:invalidations.
• The live updates of an application use the topic cloud:live:<application ID>, which reserves 64 MiB plus 1 MiB of dead letters. Its sync.owner is cloud, not the application.
• A pump reserves 64 MiB.
• An object store reserves its declared retention.maxBytes once; it has no dead-letter stream.

Per replica, the built-in applications reserve:

Together, that is about 16 GiB per replica for every built-in application. Size max_file_store for the applications you deploy with headroom for standalone applications, and check the live reservation of your account:

curl -s 'http://127.0.0.1:8222/jsz?accounts=true&streams=true&config=true' \
  | jq -r '.account_details[].stream_detail[]? | select(.config.max_bytes > 0)
      | [.config.metadata["sync.owner"], .config.metadata["sync.kind"], .config.metadata["sync.id"], .config.max_bytes] | @tsv'

Releases up to 0.24.0 gave every job and queue without retention 1 GiB per stream, and the dead-letter streams of grids:records, grids:workflow-record-events, mail:invalidations, contact events, item events, and the notebook snapshot job 1 GiB each: about 97 GiB per replica for the same applications. A later release lowers the byte limit of these streams in place when it starts or first uses them, and keeps their pending messages and dead letters. A job or queue stream that already holds more than the new limit keeps its old limit, and the application logs Kept the byte limit of a Sync stream that holds more than its new limit; the job or queue keeps working, and a later start lowers the stream once it holds less. No consumer processes the four topics among them, so their dead-letter streams are empty and always take the new limit.

The change is one-way. A 0.24.0 or older process refuses a lowered stream with a ResourceDriftError on max_bytes: during a rolling upgrade, an older process that restarts or first uses such a job, queue, or topic fails, and after a rollback the applications do not start. Before you roll back, make sure max_file_store on every node fits the old reservation again, then raise every lowered stream of the namespace with a nats context for the Cloud account:

curl -s 'http://127.0.0.1:8222/jsz?accounts=true&streams=true&config=true' \
  | jq -r --arg ns "$SYNC_NAMESPACE" '.account_details[].stream_detail[]?
      | select(.config.metadata["sync.namespace"] == $ns)
      | select((.config.metadata["sync.kind"] | IN("job", "queue"))
          or (.config.metadata["sync.id"] | IN("grids:records", "mail:invalidations",
                "cloud:contacts:events:changes", "cloud:spaces:events:items")))
      | select(.config.max_bytes > 0 and .config.max_bytes < 1073741824) | .name' \
  | while read -r stream; do nats stream edit "$stream" --max-bytes 1073741824 -f; done

The query also selects jobs and queues that declare a retention or deadLetterRetention below 1 GiB themselves. No built-in application did so in 0.24.0; leave out such streams of your own applications, which sync.owner names.

Pass client addresses through the reverse proxy

Applications rate-limit and audit requests by client address, including sign-in and phone pairing. Behind a reverse proxy such as Traefik, the gateway only sees the proxy's address, so the proxy must name the client in X-Forwarded-For, and the gateway must trust the proxy to do so.

GATEWAY_TRUSTED_PROXIES lists the IP addresses or CIDR ranges of trusted proxies. The gateway reads X-Forwarded-For from right to left, skips every trusted proxy, and takes the first other address as the client. It forwards that address as X-Real-IP and as the first entry of X-Forwarded-For, followed by the trusted proxies it passed through. Entries further left came from the client and are dropped. A request from a peer outside the list keeps the peer's own address, whatever its headers claim. A malformed entry ends the search at the last trusted address.

The default trusts loopback and private networks: 127.0.0.0/8, ::1/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and fc00::/7. It covers the supplied compose.prod.yml, where Traefik reaches the gateway over a Docker network and replaces any X-Forwarded-For a client sends. Keep the default there, or set the Traefik network's subnet from docker network inspect traefik to trust only Traefik.

The default trusts every client on a private network as well. Such a client can name any address and escape per-client limits whenever its own X-Forwarded-For reaches the gateway: when it connects to the gateway directly, or when the proxy appends to the client's header instead of replacing it, as nginx does with $proxy_add_x_forwarded_for. If clients can connect from private or IPv6 unique local addresses in such a setup, set GATEWAY_TRUSTED_PROXIES to the proxy's exact address or subnet.

The proxy must replace or append to X-Forwarded-For, never pass a client's header through unchanged. Traefik replaces it by default; do not enable forwardedHeaders.insecure or trust client networks in its forwardedHeaders.trustedIPs. To verify, sign in from two networks: rate limits and audit entries must show different addresses, not the proxy's.

A public proxy or CDN in front of the gateway that is absent from GATEWAY_TRUSTED_PROXIES, such as Cloudflare, makes Core see the proxy's address. Sign-in places and network names will then describe the proxy instead of the person when those details are used; see Sign-in locations. Private-network proxy addresses are ignored by the place lookup, but those proxies still need to be trusted for rate limits.

Assign configuration to the correct service

Inject secrets at runtime, never into image build arguments, browser bundles, or Git. Cloud settings, including provider credentials, are configured through administration; do not invent environment names for settings without a declared environment fallback.

See Runtime configuration for the complete identity configuration, validation behavior and broker-secret rotation. See Identity key operations for signing-key rotation, KEK recovery and revocation. Keep these secrets independent; APP_SECRET is not a signing key or an OAuth broker credential.

Managed Assistant code hosts serve every Assistant turn that runs code, including API clients, cld assistant --detach, and routines (scheduled tasks). They validate CLOUD_CORE_INTERNAL_ORIGIN when a host is created, before launching Chromium. It must be an HTTP(S) origin reachable from the Assistant service. Each host also uses an authenticated ephemeral loopback HTTP listener inside that service; allow local networking without publishing these ports. Binary bodies preserve backpressure across this hop. Eight simultaneous host requests per process are admitted; excess requests receive operation_busy. Size memory for the existing 50 MiB per-file and 250 MiB per-run code budgets plus Chromium; streaming transport does not remove the memory occupied by files loaded for analysis. View files are ordinary chat files under /views and count toward the per-chat file budget; they need no configuration or migration.

Assistant's cloud.chart() draws Studio app, chat app, script and PDF charts through the same options mapping as chat charts. Existing charts keep drawing: the chat's limits on series, points, labels and phone-width fit do not apply to apps, and empty, one-point or partly missing data still draws. Visible changes are a default logical width of 480 instead of 640 (height 280; sparkline remains 160 × 40, and apps without explicit width redraw at their measured width), date axes in the viewer's wall-clock time, and stdlib-only options outside the chart contract, such as padding, being ignored. Renderer errors, such as a fixed domain that does not contain every value, now throw invalid instead of a plain error, and x strings must be ISO dates as documented. Chart groups and the additional chart kinds need no configuration or data migration. Studio and one-off HTML app charts show tooltips on hover, keyboard focus and tap without additional configuration. See Code Mode charts.

Assistant Chromium isolation

Assistant runs headless Chromium in its own container as the unprivileged bun user with --no-sandbox. Both Compose configurations set user: bun, drop all Linux capabilities and enable no-new-privileges; the container keeps the runtime-default seccomp profile. Chromium's inner sandbox stays off, so no custom seccomp profile, unprivileged user namespaces or node preparation is needed, and the service needs neither privileged mode, root, SYS_ADMIN, a Docker socket nor nested containers. Existing custom Assistant deployments must adopt these settings before using Code Mode. The equivalent Kubernetes securityContext is runAsNonRoot: true with a non-root runAsUser, capabilities.drop: [ALL], allowPrivilegeEscalation: false and seccompProfile.type: RuntimeDefault.

Each foreground chat host and scheduled turn has a separate browser process and ephemeral host credential. This is process isolation within one capability-free application container, not a separate VM per user; the residual risk of an unsandboxed renderer inside that container is accepted and tracked in #47, which moves Chromium into a dedicated sidecar. Keep one Assistant replica for now: Code Mode hosts and their approval waits live in the Assistant process, so a poll or an approval that reaches another replica marks the call lost. Lost hosts are not replayed. Studio previews continue running in the user's browser.

Select applications and feature dependencies

Baseline below means Postgres, Valkey, NATS JetStream, APP_SECRET, completed Core schema setup, and a reachable Core for authentication/authority operations. It is a deployment prerequisite, not a claim that every app synchronously probes Core at startup. A feature dependency is required when using that feature, not necessarily to start its container.

Run the checks with an appropriately authorized test account and disposable records. They are acceptance steps, not instructions to send production email or mutate real data without approval.

Platform and operations

Work applications

Directory and utility applications

Configure optional services before testing their features

These are shared service settings, not additional mandatory containers for every app. A feature executes in its owning service: for example, AI provider egress is needed from Core, while Mail needs access to its mailbox providers.

For S3 snapshots, enter credentials in the individual notebook's snapshot configuration, not invented global S3 credential environment keys. Snapshots upload to public storage by default; internal S3-compatible storage requires an operator allowlist as described below. Files (filesv2) uses the Filegate root and lease API of Filegate 6.1 or 7.0; see Files.

Cloud refuses person- and administrator-chosen outbound requests from the Webhook Tester, workflow HTTP steps, assistant file downloads, web push, Mail endpoints, Pulse metrics sources, Notebook S3 snapshots, health webhooks, Geo city search, Assistant rsql, and outgoing mail SMTP and bounce IMAP connections to loopback, private, link-local, shared, and other reserved addresses, including names that resolve there. A failed DNS lookup also refuses the request. An address in the well-known NAT64 prefix 64:ff9b::/96 is judged by the IPv4 address it embeds, so IPv6-only servers with DNS64 still reach public IPv4 hosts. To test an internal service, use an address that is public from the server.

Operators can allow internal administrator-configured endpoints with GATEWAY_OPS_WEBHOOK_ALLOWED_TARGETS in app-gateway-ops (default empty, public targets only), GEO_ALLOWED_TARGETS in app-weather (default geo), ASSISTANT_RSQL_ALLOWED_TARGETS in app-assistant (default rsql), and OUTGOING_MAIL_ALLOWED_TARGETS in app-core (default empty, covering both SMTP and bounce IMAP). Each accepts exact host names, IP addresses and CIDRs; an explicit value replaces the default. Empty Compose passthrough values use the default. Invalid entries, including /0, stop the owning process at startup. The HTTP transports for these endpoints never follow redirects; health webhooks previously followed them. They request identity encoding and reject compressed responses. IMAP tries every validated address. SMTP connects to one validated address per delivery attempt; retries move to the next address, and a test send checks each address in turn on connection failure before sending once. The test send opens a separate connection to verify the selected address and authentication first. Both SMTP and IMAP preserve the original host name for TLS certificate verification.

After this update: Existing installations whose health webhooks, outgoing mail SMTP/IMAP profiles, Geo URL, or rsql URL point at internal hosts other than the defaults fail with a clear reason until the operator sets the corresponding variable. Stored configuration stays. Geo reports city search as unavailable and logs GEO_ALLOWED_TARGETS; mail delivery policy refusals remain retryable until the message deadline. Use narrow entries and retain geo or rsql in an explicit list if the shipped containers must stay reachable.

Pulse operators can allow internal metrics targets with PULSE_SCRAPE_ALLOWED_TARGETS in the Pulse container environment. Use the narrowest entry: an exact host name, a single IP address, or the smallest required CIDR range. A host entry trusts all addresses returned for that exact name. A network entry permits addresses in that range; reserved names such as localhost and .internal names still need an exact host entry. Redirect targets use the same policy. Entries are not limited to a port or path: an entry lets base editors reach every HTTP service on that host or in that range. A /0 entry is refused because it would turn the check off.

After this update, existing metrics sources that point at internal addresses fail with a clear reason until the operator adds their targets to PULSE_SCRAPE_ALLOWED_TARGETS. The source rows remain stored. Scrapes now request uncompressed responses (Accept-Encoding: identity). An endpoint or proxy that compresses the response anyway fails with an unsupported content encoding error; turn off forced compression for /metrics.

Notebooks operators can allow internal S3-compatible storage with NOTEBOOKS_SNAPSHOT_ALLOWED_TARGETS in the Notebooks container environment. Use the narrowest entry: an exact host name, one IP address, or the smallest required CIDR range. Host entries trust all validated addresses for that exact name; reserved names need a host entry. Each entry covers every port and path on the host or range. /0 and other invalid entries stop Notebooks at startup. Failed and empty lookups still refuse an upload, even for listed names.

After this update, existing snapshot configurations that point at internal storage fail with the reason "This S3 endpoint is on an internal network..." and code NOTEBOOK_SNAPSHOT_TARGET_BLOCKED until the operator allows the target. Settings and credentials stay stored. Snapshots now upload with presigned PUT requests; object identities and paths stay the same. Redirects are refused. Transient storage failures (5xx, 429, 408, or connection loss) are retried up to three times per object.

Keep Gotenberg offline

Cloud renders all HTML offline and loads assets only from request files and data: URLs, so it never needs Gotenberg to load a URL or run JavaScript. Run a Gotenberg instance dedicated to Cloud and let it enforce the same:

• start it with --chromium-disable-javascript=true;
• start it with --chromium-allow-list='^file:///tmp/.*', which confines Chromium to local files in Gotenberg's working directory, /tmp. data: URLs keep working. If you use --chromium-deny-list instead, for example to block private address ranges, keep its default ^file:(?!//\/tmp/).* entry, because setting the flag replaces the default;
• give the container no outbound network access: attach it only to a private network through which Cloud services reach it, with no route to other internal services, cloud metadata endpoints, or the internet.

Keep the Gotenberg API reachable only by Cloud services. Its other routes, such as URL conversion, load remote content by design. After changing the flags, render a PDF that contains an image, such as a Grids document with a record image, to confirm that assets still load.

Bring up a fresh installation

1. Select apps and optional features. Generate a private temporary ADMIN_LOGIN_TOKEN and supply it only to Core. Production has no default token and a fresh database has no normal administrator account.
2. Generate and store the independent deployment secrets. Set the public origin, private service addresses, database, Valkey and NATS connections. Give Core and OAuth the same broker secret if OAuth is selected; only Core gets the identity KEK.
3. Start and check persistent infrastructure. Start Core and wait for schema setup, identity initialization and readiness before starting dependent apps. Start the gateway with private app reachability and keep public access restricted during setup.
4. Start the selected apps. OAuth checks Core at startup and requires no credential-creation request. The development and production Compose files wait for Core's readiness healthcheck before starting OAuth. For mandate-backed background integrations, provision the owning app's identity:invoke credential using Background mandates, inject it only into that app, and recreate that app's container to apply environment changes.
5. Open /auth/login?method=admin at the public origin and enter the temporary admin token. Review and accept the displayed legal documents to finish first sign-in. Configure a normal administrator account or FreeIPA administrator group mapping, then verify that sign-in in a separate session. Remove ADMIN_LOGIN_TOKEN from Core and recreate Core before normal operation. Configure optional services through administration using their internal addresses, not localhost from another container. FreeIPA and Files do not import environment values.
6. Verify the selected rows' functional checks before exposing normal traffic. Confirm health, authorization, durable writes, and any required background operation separately.

Provide ClickHouse for Pulse

Provide ClickHouse 26.9 or newer as one node on the internal network, with its own persistent volume. Keep the instance dedicated to Pulse: ClickHouse user management is instance-wide, so the service user can alter any user, including ClickHouse admins. Whoever operates ClickHouse as admin can read all Pulse data.

For the normal read profile, plan at least 10 GB RAM: three concurrent reads at 2 GB, 1 GB for maintenance, and reserve. Choose klein (small) on smaller servers. The setup script also creates groß (large); each read profile has fixed memory, execution, scan and result limits that readers cannot override. This release installs the profiles but does not yet expose read-profile selection for bases.

A Cloud admin opens Administration → Pulse → Data store (/admin/pulse), chooses the service password and requests the setup script. Run that script as the ClickHouse admin once, with clickhouse-client --multiquery or equivalent. It contains a salted password hash only and can be re-run to restore profiles or update the service password. Then enter the address, service user and password on that page, or use:

cld pulse admin store setup-sql --password-file /private/pulse.password
# Run the resulting script as the ClickHouse admin before saving the connection.
cld pulse admin store set --url http://clickhouse:8123 --password-file /private/pulse.password
cld pulse admin store check

The save checks reachability, sign-in, version, scoped rights and profile constraints before replacing the stored settings. Failed checks leave the saved connection unchanged and identify the failed check with a repair hint. Store passwords only through the page, a private file or stdin; never put them in a CLI flag value or URL.

The Cloud admin owns this internal target. Pulse allows exactly the host in the saved or submitted ClickHouse address, without environment configuration. Requests connect directly to checked addresses and never follow redirects. Remote response bodies are never echoed in errors. The answer time can still tell a closed port from an open one, so the Cloud admin role carries reach into the internal network that Cloud can connect to.

Back up all pulse_* databases with ClickHouse BACKUP, together with Cloud Postgres and its encryption keys. Restore them as a consistent set.

In this release, Pulse does not yet store data in ClickHouse. Without a connection, Pulse keeps working as before and both Cloud and Pulse report ready. Later releases require this store. Production Compose does not provision it; the operator supplies it.

Verify readiness and upgrades

During the rollout to this release, and during a rollback to the previous version, replicas of the previous version write no liveness key. While they run, a replica that stops can still delete the shared registry entry once, and surviving replicas restore it within 60 seconds. During a rollback, replicas of this release also replace the previous version's entry with their own on their next refresh until they stop. No configuration or rollout order is required.

Check /_cloud/ready on each service's private origin, then check gateway registration and request an owned route through the public origin. The gateway's own readiness response is not aggregate readiness for all applications. Inspect /admin/gateway/apps and /admin/observability when Gateway operations is installed. Missing optional providers may leave the process ready while individual features remain unavailable.

For existing installations, review Deprecations and migrations before changing replicas. Use the coordinated identity cutover where required; do not infer rolling-upgrade safety from a healthy old container. Mail's current unreleased-alpha schema targets fresh installations, not automatic migration of old alpha automation credentials.

Use Build and deploy for immutable image sets and preflight, Scaling and shutdown for lifecycle behavior, and Troubleshooting for failed routes or dependencies. Plan rollback against both schema and key compatibility; replacing an image does not restore migrated data.

Start replicas of one application together

Replicas of one application can now boot together without setup races causing restarts. A starting replica waits while another runs setup, so its boot time grows by the other replica's setup time. A replica that crashes or is stopped during setup delays the others by at most about 60 seconds while its NATS lease expires.

During the rollout to this release, a replica of the previous version takes no setup lock. Restart old and new replicas of one app one at a time until all run this release. Keep Core first: it creates the shared schemas other applications need.

Prepare Pulse's new data store

This release adds an optional ClickHouse connection and versioned Pulse Postgres migrations. A complete existing Pulse installation is adopted without replaying its baseline; older images must not start against ledger versions they do not recognize. An incomplete Alpha schema still requires an explicit reset and is never upgraded automatically.

Provision and connect ClickHouse before the later releases that move telemetry there. No data moves in this release, and an unset store does not change startup or existing Pulse features.

Upgrade Spaces task deadlines

Spaces adds the nullable spaces.items.deadline_date DATE column and the items_deadline_one_kind check on startup. The check permits at most one of deadline and deadline_date. Existing timestamp deadlines remain timed; there is no backfill. API and capability consumers, such as scripts, agents and integrations, must accept deadline as either YYYY-MM-DD or an ISO instant and can use deadlineAllDay to distinguish them. Deploy the updated Spaces server and its matching browser and CLI assets together. Verify a date-only deadline, a timed deadline, and switching between them after the upgrade.

Upgrade Spaces user claims and assignments

Spaces now assigns a person when they claim a task. On each start, an idempotent backfill adds missing assignee rows for active user claims. It skips claims whose user was deleted and service-account claims. No configuration is required. The backfill logs only when it adds rows.

Assignee replacements that remove an active user claim holder now return 409 unless the caller supplies that holder's claimId, with force when acting for someone else. CLI and capability callers must release first. Releasing your own claim keeps your assignment; forcibly releasing another person's claim removes their assignment. See Implementation tasks and agent handoffs.

Upgrade Mail to several assignees

This Mail release lets a conversation have several assignees and adds mailbox grants that cover only assigned conversations. It needs no configuration change. Its first start creates mail.conversation_assignees, copies every existing assignment into it once, and creates mail.mailbox_assigned_access. Platform administrators now need a mailbox grant like everyone else: an incoming automation authorized by an administrator without one pauses its Spaces authorization on its next run until that person has a grant with Write access and authorizes it again. From then on the new table is the only source of assignments; Mail keeps the previous column mail.conversations.assignee_user_id at the earliest assignee so that an older image still shows one of them.

Roll the Mail replicas over in one go. A replica of the previous version still reads and changes only that column, so an assignment it changes after the copy does not reach the new version, and it treats people with assigned-only access as having no access. Open Mail tabs from before the update cannot assign until they are reloaded. Third-party applications and saved workflows that use the single assignee fields need the changes in Deprecations and migrations.

Rolling Mail back keeps the new tables. The previous version then shows only the earliest assignee and denies people with assigned-only access; assignments it changes are lost when Mail is updated again. After the rollout, assign two people to a test conversation and check that cld --json mail show <conversation-id> lists both.

Mail draft signatures

Mail adds the column mail.drafts.signature_template_id on its next start. No configuration change is needed. Drafts keep their signature in that column instead of in the editable text; see Deprecations and migrations.

Roll the Mail replicas over in one go. A replica of the previous version does not read the column, so a message it sends from a draft that the new version saved goes out without its signature. Rolling Mail back keeps the column and has the same effect for drafts saved after the update. After the rollout, open a new message in a mailbox with a default signature and check that the preview shows the signature below the text.

Mail mailbox logos

On its next start, Mail adds mail.mailbox_logo_versions, mail.mailboxes.logo_version_id, and mail.outbox_submissions.inline_logo_version_id. No configuration change is needed. Roll all Mail replicas together: an older replica ignores the frozen logo in a queued message and sends HTML whose cid: image has no inline part. A rollback has the same limitation for messages queued with logos.

Logos count toward the Mail database share, at most 100 KiB per immutable version. Unused versions are removed hourly; current logos and versions still needed by queued messages or Sent-copy retries are retained. Local sent copies retain their inline images in the deduplicated Mail blob store, independently of logo versions. After upgrading, upload a test logo, preview a signature containing {{ mailbox.logo }}, and schedule it. Check that the local copy shows the logo before provider sync, then send it and check that it arrives as an inline image without a remote image link.

Mail storage for kept conversations

Mail adds the table mail.conversation_keeps and a trigger on mail.conversation_messages on its next start. No configuration change is needed. Replicas of an older Mail image do not know kept conversations and still accept deletes, so update every Mail replica before people rely on Keep.

Mail stores every message it loads in Postgres: the original source, the attachments a second time as decoded files for previews and downloads, and the text for search. Identical files are stored once. Plan the Mail share of the database at up to about twice the size the loaded messages take on the mail server; one message is at most 128 MB. Mail keeps this data when the mail server deletes a message, also without Keep, so keeping a conversation adds only one small row and loads any of its messages that Mail had not loaded yet. Keep changes what people see and what Cloud refuses: the copies stay visible, and Cloud does not delete them. Like all Mail data, they stay in the database and its backups; Mail has no automatic cleanup for message data.

Mail storage has no size cap, with or without Keep: only one message is limited, to 128 MB. Watch the database's disk use and plan growth with the mail volume. The stored sources and attachments are in mail.message_part_chunks, so SELECT pg_size_pretty(pg_total_relation_size('mail.message_part_chunks')) shows the largest share.

Queue Mail sender verification behind provider work

Mail no longer refuses a sender verification with 409 PROVIDER_BUSY while a sync or other provider work holds the mailbox. It keeps the verification as a verify_sender_identity command that runs when it is its turn and ends within 10 minutes. No configuration change is needed. The first Mail start after the update replaces the commands_kind_check constraint of mail.commands so that it allows the new command kind. It does not read the table for this.

During a rolling update, Mail processes of the previous version leave these commands to updated processes. A cld from before this release prints Verified for a verification that still waits, so update cld together with Mail. API clients that verify an identity or set up the default sender must check pendingVerificationCommandId in the response: a 200 response no longer always means that the identity is verified.

Rolling back keeps the wider constraint, and the previous version ignores waiting verifications. After the next update, a verification that waited longer than 10 minutes fails with SENDER_VERIFICATION_TIMEOUT, and the person verifies the identity again.

Index Mail's per-message command lookups

After its first start with this update, Mail builds commands_message_target_idx on mail.commands in the background with CREATE INDEX CONCURRENTLY, so command writes and Mail's start continue. No configuration change is needed. Until the index is valid, the lookups use the previous plan. One Mail process builds an index at a time, including the optional BM25 search index. An interrupted build is replaced on the next start.

Update Assistant approvals for chats and websites

This release lets people remember an approval for one chat and allow a website for one chat or one Studio app. It needs no configuration change. The first Core start adds conversation_id to ai.tool_approval_preferences, replaces the unique constraint ai_tool_approval_preferences_unique with the index ai_tool_approval_preferences_reach, adds two columns to ai.pending_actions, and creates ai.web_addresses for the addresses a chat's searches returned and its read pages link to. Assistant adds two columns to assistant.artifact_agent_approvals. A chat approval and a chat's addresses are deleted with the chat, and an app's website approvals with the app.

Two behaviors change for people. More built-in Actions can be remembered, and web_extract and fetch_file now ask before reading an address that the chat did not supply, which means an address that is not in the person's message, a search result, or a link of a page read earlier. Search results and pages read before the update are not recorded, so reading one of their links in an existing chat asks once. A routine cannot ask, so a routine that reads such an address now fails with a message to run it in the normal chat; give it the address in its prompt instead.

Roll Core and Assistant over together. Before rolling Core back to an earlier release, run the following in the Cloud database. An earlier Core expects the old constraint when it starts, and it would apply a chat approval in every chat:

DELETE FROM ai.tool_approval_preferences WHERE conversation_id IS NOT NULL;
ALTER TABLE ai.tool_approval_preferences
  ADD CONSTRAINT ai_tool_approval_preferences_unique UNIQUE (actor_user_id, tool_name, approval_scope);

After the rollout, ask Assistant in a chat to run code that reads one public JSON URL twice. The first request asks and offers Allow this website for this chat. After you choose it, the second request runs without asking and shows its full URL with Revoke.

Update Studio source writes

code_write now saves one file per call with top-level path and content. File imports moved to the new code_import tool. Results report lines next to length. No configuration change or database migration is required. Stored source, revisions and chats are unchanged; older chat blocks keep their recorded arguments.

Roll Core and Assistant over together. A mixed version rejects the other side's code_write input as invalid, and an older Assistant has no code_import route. Rollback is replacing both images; nothing needs undoing in the database. A turn running during the rollout may get one invalid-input error and then follows the new tool description.

An app build now takes one tool round per source file instead of one round for all files. If a model profile sets a positive maxToolRounds, check that it still covers a typical multi-file build; otherwise the turn stops using tools earlier and answers with the files it saved so far.

After rollout, ask Assistant to build a small app with index.html and app.js. The turn should show one code_write per file, no approval, and each result should list line counts that match the files.

Record approvals as platform requests

This release records every approval that Assistant asks for as one platform approval request. It needs no configuration change. The first Core start creates the schema approvals with the table approvals.requests, adds approval_id to ai.pending_actions and approval_decided_via to ai.tool_calls, and gives every chat approval that is still open a request. Assistant adds approval_id to assistant.capability_calls. Core deletes a request 90 days after it expired.

Update Core before Assistant. Until Core has created the table, Studio code that calls an Action which asks for approval fails. One behavior changes for people: when an Action's review changes between the question and the run, for example because a draft got another recipient, the chat asks again instead of running the Action.

Rolling back keeps the new table and columns, and the previous release ignores them. After the next update, Core gives the chat approvals that are open then a request.

After the rollout, approve one Action in a chat, then check that its request was used and that the decision is in the audit log:

SELECT app_id, action, status, decided_via FROM approvals.requests ORDER BY created_at DESC LIMIT 1;
SELECT action FROM audit.events WHERE target_type = 'approval_request' ORDER BY created_at DESC LIMIT 1;

The request shows consumed and web_session, and the event auth.action.approve.

Decide pending approvals in Cloud

This release adds approval lists and decision channels. Core upgrades the decided_via CHECK constraint of approvals.requests on start. The migration is idempotent. It replaces the existing constraint only when it lacks acp_bridge. No configuration change is required.

Core serves the new /api/approvals list, counts, and decision routes. They fall under Core's registered catch-all / prefix. Core does not declare a separate /api/approvals prefix.

Update Core before Assistant and Mail. Both read the approvals list during SSR through the platform service. Core must migrate the approvals schema before they serve those pages. Core also registers Pending approvals as a user notification. Browser delivery is recommended. Users can change delivery in their settings.

Rolling back keeps the expanded constraint. The older release ignores the new channel value. It does not apply the ACP restriction or serve the new approval lists.

After the rollout, ask Assistant to send a Mail draft. Open My account → Approvals, the Assistant's Approvals view, and the draft. Check that each shows the waiting request. Try to approve a confirm request through a client that sends x-cloud-client: acp. It must return 403 and leave the request open. Approve it in Cloud and check the decision in the audit log.

Approve Actions in Cloud Login

Deploy the updated platform package and Core together. On start, Core adds cloud_login to the decided_via constraint of approvals.requests and adds two nullable timestamps, actions_at and action_wake_at, to auth.app_devices. These migrations are idempotent. No configuration change is required.

Update the Cloud Login image too. A newer Cloud still works with an older Cloud Login server: that server answers Action wake-ups with 400, which Cloud ignores. Existing paired phones keep signing in. They show Actions after the authenticator app updates, without pairing again.

The updated Cloud Login app no longer uses an app PIN. On a phone that still has one, the app asks for it one last time and then opens without it. Its connected Clouds stay paired. If someone forgot the PIN, Reset Cloud Login works as before; they then revoke the old device in Cloud and pair again. Tell users that the phone's own screen lock now protects their approvals.

After the rollout, ask for an Action through cld or an editor bridge. Check that a paired phone shows the request and that its decision continues the waiting chat. See Approve an Action for the signed protocol and push rules.

Rolling back leaves the widened constraint and new columns in place. Older Core ignores the columns and keeps the widened constraint. Existing sign-in commands and pairing keys remain valid.

End approvals when a device is revoked or the person signs out everywhere

Deploy the updated platform package, Core, and Assistant together. On start, Core adds the nullable columns decided_device_id and approver_auth_epoch to approvals.requests. When it adds the epoch column, it fills it once for pending requests with the person's current epoch. Approved requests that are not yet used at the upgrade, and Cloud Login approvals without a recorded device, are treated as ended when used, so the chat asks again. The migration is idempotent. No configuration change is required.

Rolling back keeps the columns; the older release ignores them and does not check revocation when an approval is used. Requests the older release writes keep no epoch, and the updated release treats them as ended. Pending requests leave lists, and decisions on them answer as expired. Unused approvals are cancelled when used, so the person is asked again.

After the rollout, approve an Action on a paired phone, then check that the request records the phone and the epoch:

SELECT status, decided_via, decided_device_id IS NOT NULL AS device, approver_auth_epoch IS NOT NULL AS epoch
FROM approvals.requests ORDER BY created_at DESC LIMIT 1;

The request shows consumed, cloud_login, and true for both checks.

Reuse Agent Skills folders

Update Core and Assistant together for portable Skill frontmatter, relative reference links, and manual-only invocation. No schema migration or new configuration is required. Skills with disable-model-invocation: true stay available for explicit selection but disappear from model discovery; loading by name is refused. Stored and exported links stay relative and are mounted when a Skill loads. Existing absolute links keep working.

Update installed CLI plugins with cld plugins update to get cld assistant skills list, push, and pull. See Reuse shared Skills.

Run Assistant in editors

The Assistant cld plugin adds cld assistant acp (shortcut cld acp), an ACP agent for editors such as T3 Code and Zed. It uses the existing Assistant API and approval rules and needs no configuration or infrastructure change. People who installed the plugin earlier get the command with cld plugins update. See Assistant in editors.

Update browser notifications for grouping and badges

Deploy the updated platform package and Core together to enable grouped browser notifications and app badges. No database migration or configuration change is required.

The browser checks for the new worker when a Cloud page is next loaded or reloaded, and at the latest during a push or notification click once 24 hours have passed since the last check. Its existing skipWaiting() and clients.claim() lifecycle activates the new worker without waiting for tabs to close. Already-shown notifications are unaffected. Older workers ignore the optional group and badge fields until they update; existing ungrouped notifications keep their behavior. Already-queued browser deliveries without these fields remain valid.

Reload a Cloud page first, then verify that two notifications with the same group replace each other on one registered device. Check badges in a browser that supports the Badging API; lack of badge support must not prevent notification display.

Update browser notifications for previews

Deploy the updated platform package and Core together to enable optional browser notification previews. No database migration or configuration change is required. Existing applications send no preview, so their notifications keep the same behavior.

Older service workers ignore preview and show only the title until they update, with the same page-load, push, and click timing described above. Reload a Cloud page, then send a notification with a harmless test preview to a registered device and verify that its text appears below the title. Send another without a preview and verify that it shows only the title. Confirm that notification history contains neither preview. Review the application's preview controls before enabling sensitive content on lock screens or paired devices.

Show app badges in the rail and app grid

The rail and the app grid can show a count on an application's icon when the application declares nav.badge. No database migration or configuration change is required, and no built-in application declares a badge yet, so the shell looks and behaves as before.

Each application renders the shell with its own platform package, so badges appear on the pages of applications built with this release; pages of older images show the same rail without counts. Older releases ignore the field in the registry. Once an application declares a badge, every open, visible tab of a signed-in person sends one GET to that route after the page loads, at most one per minute after that, and one more whenever a page asks for a fresh count after a change. Each tab keeps at most one such request per route in flight and gives up on one after a minute; requests asked for meanwhile share one read after it. Hidden tabs send nothing. Count these requests when sizing the application.

Let widgets declare settings

Widget settings need no configuration change. Core gains POST /api/widgets/v1 for repeated entries with their own settings. GET stays unchanged, so an older Dashboard keeps working during a rolling deploy. See Declare settings.

An application image from before settings existed ignores them. An application that declares settings needs a Core that forwards them; with an older Core, it answers with its no-settings behavior. Deploy Core before a Dashboard that uses settings.

defineApp() now rejects a widget ID that Core could never invoke: longer than 188 characters, or with characters other than lowercase letters, digits, ., _, -, and :. Such a widget only ever showed an error, so an application that declares one now fails at startup instead.

Upgrade the dashboard to one board

This release turns the dashboard into one board of widgets in three sizes; see Board and gallery. It needs no configuration change. Deploy Core and the Dashboard together: the Dashboard asks Core for each widget in a size (widget=<key>@<size>), which an older Core does not understand, so its widgets would show as failed until Core is updated. Applications can follow in any order; an application image from before sizes existed ignores the size and shows its widget in the large size.

The Dashboard's first start adds the nullable column dashboard.user_settings.board. Each person's saved settings are converted once, on their next visit, into the board they saw: their order, a size per widget from its zone and width, widgets they had switched off left out, and their shortcuts kept. The old columns hidden_widgets and widget_layout are emptied for that person as part of the conversion, and the name color in gradient is no longer used. People who never changed a widget follow the new default board. The per-device cookie dashboard_widgets is deleted by the browser on the next visit.

The conversion reads the widgets of the applications registered at that moment. A widget the saved settings name keeps its place while its application is stopped. A person who never moved a widget, and only switched widgets off or changed a zone or width, gets no widget from an application that is restarting at their first visit; they can add it again from the gallery. Update the applications right after Core and the Dashboard, so this window stays short.

Rolling the Dashboard back keeps the board column but shows every widget to everyone whose settings were already converted, because their old settings are empty. After the rollout, open the dashboard, select Edit, move a widget, select Done, and reload: the widget stays where you put it.

Hold back browser notifications during quiet time

Deploy the updated platform package and Core together to enable do not disturb and quiet hours. Core creates the notifications.quiet_times table when it starts, and no configuration is required. Nothing changes for a person until they pause notifications or save quiet hours under My account → Notifications.

From then on, Core's delivery worker drops recommended browser notifications during that person's quiet time instead of postponing them. The delivery becomes suppressed with do_not_disturb or quiet_hours, and its waiting email or deployment-channel fallbacks end with the same code, so no email replaces the push. Required deliveries and email chosen first still go out. Expect these codes in notification history and under Observability → Notifications; they are not delivery failures.

To verify, pause notifications for an hour on a test account with a registered browser, send it a test notification, and check that its history shows the browser delivery held back by do not disturb and no fallback email. Resume, send again, and confirm that the push arrives. An older release ignores the table, so a rollback turns quiet time off and keeps the saved settings.

Enable resource permalinks

Deploy Core and the owning applications together to enable resource permalinks. Core serves /open/<type>/~<encoded-id> and registers /open with the gateway; owners provide permission-aware top-level open destinations in their readers. The ~ prefix is required; links without it return 404. Deploy Core before clients start exposing these links, because older Core images return 404. Updating Core alone does not update Mail's draft destination or other owning-app readers. No configuration or database migration is needed.

During a rolling update, Core accepts an older reader's validated edit link when it has no open. This preserves existing writer links, but cannot repair a destination that requires more access than the reader. Update Mail with Core so readable drafts open the read view for readers and the composer for writers. MCP advertises its consumer refs[].href field in the output schema; owning-app responses retain the strict provider ref schema.

Verify permalinks while signed out, as a writer, as a reader and without access. Sign-in must preserve the return URL, and the destination page must recheck current permissions, including revocation after resolution. Missing access, offline apps and missing destinations show Core's 404 page. Existing readers without links keep working through capability calls. See Resource links.

Bound and expire workflow events

This release limits workflow events and stops workflows.event from growing without bound. It needs no configuration change. Both limits are fixed:

• An emit whose data or context is larger than 16 MiB fails with WORKFLOW_EVENT_TOO_LARGE. Grids answers such a start with a bad-input error. Mail emits every message without its body, and the run reads the body from Mail when a step needs it.
• 30 days after dispatch or dead-lettering, Core deletes the event's deliveries and empties its data, context, and authorization snapshot, once no queued, running, waiting, or needs-attention run references it. The row and its dedupe key stay, so deduplication keeps working. Run details show the event payload as expired.

The first Core start adds the column workflows.event.payload_expired_at. Core then builds two indexes for the retention sweep in the background with CREATE INDEX CONCURRENTLY, so event and run writes continue during the build. The first sweep starts when both indexes are valid and empties old payloads in batches of 1,000 every 6 hours. PostgreSQL reuses the freed space for new rows; the table files shrink only after VACUUM FULL or a similar rewrite. See Size and retention.

Expire routine runs after 90 days

This release expires finished routine runs after 90 days. It needs no configuration change. The first Core start adds the indexes idx_ai_chat_task_occurrences_finished, idx_ai_chat_task_occurrences_turn, and idx_ai_turns_background_finished. The latter uses a plain CREATE INDEX on ai.turns, briefly blocking writes to ai.turns while it builds; allow for a longer build on large installations. The routine recovery tick then deletes completed, failed and stopped runs older than 90 days, their background turns with tool calls, and their run transcripts (ai.task_messages) in batches of 100 per minute. Queued and running runs, finished runs whose turns are still active, and chat result messages stay. The run row holding a one-off task's scheduled time stays, even after completion, until the task gets a new time or is deleted, so that time never runs twice; its turn and transcript expire on the same clock. The same start relaxes ai_memory_learning_runs_evidence_check so a workflow memory learning run keeps its evidence key when its source turn is deleted. Background turns and transcripts of runs whose task was deleted expire on the same 90-day clock, measured from when the turn finished.

An installation with a large backlog drains it over time. PostgreSQL reuses the freed disk space; table files shrink only after VACUUM FULL or a rewrite. See Routine run history.

Assistant approval choices

Always approve is no longer offered on Assistant questions; choices made earlier still apply until the next major release.

Record why routines stop

This release gives every paused routine and every routine that needs its owner a reason. It needs no configuration change. The first Core start adds the columns reason_code, reason_at, title, and locale to ai.chat_tasks, fills the reason and locale of existing routines once, and adds the constraint ai_chat_tasks_reason_state_check. Adding the constraint locks ai.chat_tasks while it checks the existing rows.

Earlier Core and Assistant processes write such states without a reason, which the constraint rejects. Stop them before the new Core starts instead of running both versions side by side. Before rolling Core back to an earlier release, run the following in the Cloud database:

ALTER TABLE ai.chat_tasks DROP CONSTRAINT IF EXISTS ai_chat_tasks_reason_state_check;

See Routine names and stop reasons.

Check the app's review before a routine changes anything

Routines whose calls the app would ask about now stop with Braucht dich / Needs you instead of running. Before each Action with approval: "rememberable", a routine asks the app's review. When the review gives no approval scope, fails, or is missing, the Action does not run, and the routine shows which Action needs the person. It needs no configuration change.

The first Core start adds the column reason_action to ai.chat_tasks and the constraint ai_chat_tasks_reason_action_length_check, which only limits the action name to 200 characters. The change is backward-compatible: an earlier Core can run beside the new one and can be rolled back to without SQL. An earlier Core or Assistant still runs such Actions without the review, so finish the rollout of both before you rely on the stop. See Routine Action reviews.

Approve OAuth sign-ins in Cloud Login

Update Core before OAuth to enable the login_hint device flow. An older Core cannot accept these requests; OAuth returns HTTP 503 temporarily_unavailable. Nothing changes for clients that do not send login_hint.

cld login --user needs all three parts: Core, then OAuth with the login_hint handoff, then Cloud Login with terminal sign-ins. Only Cloud Login on a paired phone shows the request. Without --ttl, such a login lasts user.session.expiry_hours; browser and --device logins without --ttl stay at 90 days. --ttl only shortens a login, the Cloud caps it at user.session.expiry_hours, and cld shows the actual end. Cloud Login shows the requested rights in plain words and approves with a single tap; people who want fewer rights sign in with cld login --scope. When discovery lacks cloud_login_hint_supported, cld refuses --user and prints the matching --device command. When it lacks cloud_login_ttl_supported, cld refuses --ttl and asks to sign in without it. See Upgrade the three components.

End OAuth sign-ins sooner

OAuth logins can request a shorter absolute lifetime with ttl, capped at user.session.expiry_hours and the 90-day default; see Login lifetime. It needs no configuration change. Without ttl or login_hint, logins keep their 90-day refresh family.

Deploy Core with or before OAuth. OAuth now asks Core for access and ID tokens shorter than one hour when a login ends within the hour. An older Core rejects any lifetime other than 3600 seconds with HTTP 400. Such a sign-in or refresh fails with server_error (HTTP 500). OAuth maps only Core's HTTP 403 to invalid_grant. Core also caps every refreshed token at its refresh family's end. OAuth's first start adds the nullable column oauth.device_authorizations.login_lifetime_seconds. A rollback ignores it and issues 90-day logins again.

Upgrade Contacts from before short IDs

If Contacts never finished assigning short IDs, start it once on Cloud v0.45.0 and wait until /_cloud/ready answers ready before upgrading further. An installation that started Contacts on any release from v0.7.0 through v0.45.0 needs no action. See the Contacts short-ID upgrade path.

Discover Commands from navigation apps

The global palette now hides Commands from apps outside a person's navigation. No configuration change is needed. The slash menu offers the same Commands in Open in apps, except those of its own app. Both surfaces require Commands that accept an empty input. The target page still checks access.

Index the messages people sent for Recent

The slash menu's Recent section reads the newest messages a person sent. The first Core start builds the partial index idx_ai_messages_sent_by_person on ai.messages, covering only people's own chat messages. It needs no configuration change. Like the other ai.messages indexes, Core builds it during its migration: writes to ai.messages wait until it is done, so on an installation with a large chat history the first start takes longer. A rollback keeps the index; an earlier Core ignores it.

Coordinate an existing installation's identity upgrade

Use one maintenance window for the gateway, Core, OAuth, applications, and background workers. Replacing images alone is insufficient when the old installation lacks NATS or the identity secrets below. Record the actual image digests and source revisions; an approximate installation date does not identify which migrations apply. The identity checks do not replace the separate Sync, Notebook, or application-schema upgrade checks.

Prepare before stopping services

1. Pull a complete immutable release image set and retain the previous image digests and deployment configuration for recovery. Include independently deployed apps and workers in the inventory; old processes must not keep writing after the migration.
2. Preserve the existing APP_SECRET, database, and saved public app.url. If Core identity keys already exist, preserve their current KEK. If this is the installation's first Core identity release, provision an independent CLOUD_IDENTITY_KEY_ENCRYPTION_KEY for Core; Core creates its signing keys during startup. Do not replace a KEK protecting existing rows with a newly generated value.
3. When running OAuth, supply the same independent CLOUD_OAUTH_BROKER_SECRET to Core and OAuth. OAuth needs no app-credential provisioning. Supply private Core/OAuth origins as described in Runtime configuration, and check NATS, database, cache, private-network, and clock prerequisites above.
4. Quiesce user writes and new background work while old workers finish accepted work. Close Notebook editing connections and prove that their final updates reached durable snapshots before stopping the old snapshot workers. For an existing NATS installation, use the applicable snapshot cutover check. That NATS check does not inspect Redis-backed Sync v5 history; a v5 upgrade needs snapshot coverage established against the old release first.
5. Stop all old application processes after their drain checks pass. Take a consistent recovery backup of Postgres, the applicable broker state, file storage, and the independent deployment secrets. Verify the restore path before allowing new migrations to run. Keep ingress restricted.

Start the new release and verify it

1. Check persistent infrastructure, then start updated Core alone. Wait for migrations, key initialization, and its private /_cloud/ready endpoint. Confirm that the private session and invocation JWKS endpoints respond.
2. Start updated OAuth and the other selected apps, then the gateway. OAuth checks Core's authority before its migrations. Check every service's private readiness and its registered public route; gateway health alone is not enough. The supplied Compose orders OAuth after Core, but does not impose that order on every other application. Core and OAuth must run the same new release; OAuth sign-ins and refreshes fail until both are updated. Update every application built on @k2b/cloud to enforce the live account-epoch check.
3. Sign in again with the installation's normal account method. Check logout and re-login, an authorized app page, global search, a dashboard widget, and a Notebook edit that survives reconnect and reload. If OAuth is used, test an existing client's authorization and refresh flow through the real HTTPS ingress. Old browser sessions and old OAuth access/ID tokens are rejected; client registrations and client secrets are retained. Existing OAuth sign-ins of people end once, including their refresh grants and unredeemed codes. Run cld login again and reconnect MCP clients and other apps. See OAuth sign-ins end once on upgrade for rolling-update behavior.
4. Check startup logs and the JSON metadata repairs. Those repairs run on startup; no blanket manual repair or database reset is needed. Apply each app's separate schema requirements before enabling it. Provision Mail's workload credential only if its background integrations are used; routines without a mandate require owner recreation. Neither is an OAuth startup prerequisite.
5. For the supplied platform Compose, run the read-only fleet check against the new release. Review each app's Sync resources and failures, then reopen normal traffic. Keep backups and retired broker resources through acceptance.

If a startup or functional check fails, keep traffic restricted and inspect the owning service. Do not restart old binaries against the migrated database or flush shared Valkey/NATS state. Restore the coordinated pre-upgrade data, secrets, and old image set if rollback is necessary. See the identity cutover contract for token compatibility and Identity key operations for key recovery.

Upgrade from Sync v5

The Redis-backed Sync v5 runtime cannot read or write Sync v6 state. Finish or explicitly reconcile accepted work before switching all applications together. Notebook updates must be fully snapshotted into Postgres before the cursor schema changes. Follow the repository's Sync v6 migration runbook; backing up Redis alone does not prove that a notebook snapshot includes its last update.

Use the Sync view in Gateway Ops to inspect each application's resources, schedules and dead letters. Requeue and manual schedule runs are administrator actions routed through Core with target-bound invocation credentials. Use the separate NATS view for infrastructure diagnostics. Configure its system-account credentials only on Gateway Ops and set up independent outage monitoring as described in NATS operations.

Upgrade from Sync 6.2.0

Deploy Cloud's pinned Sync 6.4.0 version after stopping every old producer and worker. Old workers can overwrite or delete repaired coalescing claims, so these versions must not share a running fleet. Keep Postgres, Valkey, and NATS data and take backups before starting the new release. If readiness reports a ResourceDriftError, repair only the named resource as the migration runbook describes: a drifted consumer is deleted alone and its stream keeps the retained work; a drifted stream is deleted only where the runbook lists its work as ephemeral or recoverable from Postgres.

First quiesce new work while old workers can finish. Complete the notebook snapshot checks and the FreeIPA backfill checks. Keep retired broker resources through verification. Grids resumes publication from its retained Postgres outbox; historical workflow failures keep their explicit replay path.

Grids requires a fresh schema when replacing an older Grids storage layout; there is no in-place upgrade. See Grids schema reset.

If Sync reports a legacy pending claim without queued input, reconcile that specific job against its application's durable state before clearing its claim and resubmitting it. Do not clear claims in bulk or fabricate lost input. After startup, verify each application's resources, schedules, and dead letters through Core, plus normal domain reads and recovery.

Application authors remove calls to syncOps.registerDeadLetters() and syncOps.registerScheduler(): Cloud discovers native handles automatically. Custom administration clients must include queue, job, or topic in dead-letter mutation paths, between /dead-letters/ and the resource name.

Optional BM25 search ranking

Help publication and native full-text search require the Core-managed Postgres schema. Every supported PostgreSQL version (15 to 17) provides this baseline. Apps renew their Help collection through the existing app heartbeat; Core owns bounded cleanup of expired collections. See In-product Help.

On upgrade, Core adds four columns to help.documents during setup: kind, needs, next_id, and related. Update Core before applications that publish Help. Until Core adds the columns, an updated application that publishes Help fails its first registration and does not start. Every application republishes its Help once after its update because the manifest revision changed. Old collections expire as usual after six minutes without renewal.

Help, Notebooks, Mail, Assistant conversations, AI memories, and AI Skills search with native PostgreSQL full-text search. BM25 ranking with pg_textsearch is optional; it changes the order of matches, not which matches a user may see. This integration was verified with pg_textsearch 1.4.0 and 1.5.1 on PostgreSQL 17. The extension supports PostgreSQL 17 and 18; check its installation instructions against your operator-managed database before enabling it. Read Keep the library loaded while BM25 indexes exist first: enabling BM25 is easy to undo only while the library is still loaded.

1. Install the extension package matching the database major version and architecture.
2. Add pg_textsearch to shared_preload_libraries and restart Postgres during an approved maintenance window.
3. Enable it in the Cloud database with CREATE EXTENSION pg_textsearch.
4. Restart Core, then Notebooks and Mail. Their setup creates the BM25 indexes.

Step 3 can happen without you. On PostgreSQL 17 and later, Core's AI setup and Notebooks setup run CREATE EXTENSION IF NOT EXISTS pg_textsearch when the package is installed. Without the preload this fails and setup continues with native search. With the preload and a database role that may create the extension, it succeeds. A Postgres image that preloads pg_textsearch therefore creates the extension and the AI and Notebooks indexes on the next start, and from then on the database needs the library. Core sets up Help before its AI, so the Help indexes follow on the second Core start, and Mail builds its index on its first start after the extension exists. Run step 3 before restarting to create every index in one pass.

Help and Mail check extension and index availability on every search, so running processes use the indexes once they are ready. Assistant conversations, AI memories, AI Skills, and Notebooks check once per process and use new indexes after Core or Notebooks restarts. If the extension or an index is absent, native search remains active. Ordinary database failures are still reported as errors.

Keep the existing database's storage and major version unchanged when testing BM25 in a separate environment. Verify both search modes with real application articles and the required language; a healthy Postgres container alone does not prove that the optional indexes are usable.

Keep the library loaded while BM25 indexes exist

BM25 is optional when you set it up, not afterwards. Once any index above exists, Postgres rejects every statement on its table unless pg_textsearch is in shared_preload_libraries. Reads, writes, DROP INDEX, and DROP EXTENSION pg_textsearch CASCADE all fail:

ERROR:  pg_textsearch must be loaded via shared_preload_libraries
HINT:  Add 'pg_textsearch' to shared_preload_libraries in postgresql.conf and restart the server.

When the package is missing from the server entirely, the same statements fail with could not access file "$libdir/pg_textsearch", and a server that still lists pg_textsearch in shared_preload_libraries does not start.

Native search cannot take over, because it reads the same tables:

• Core setup stops with this error, so Core does not start. Notebooks setup fails the same way.
• Any application, built-in or not, that publishes Help content the database does not hold yet, for example after an upgrade, does not start: it writes those articles to help.documents when it starts.
• Mail usually starts, but conversation lists, messages, search, and synchronization fail, because they read and write mail.message_contents. An upgrade whose Mail setup changes that table stops Mail setup too.

Treat the preloaded library as part of the database. Every server that runs it needs the package installed and pg_textsearch in shared_preload_libraries:

• streaming replicas and standbys that serve reads or can be promoted;
• any server you restore a backup into;
• the new cluster of a major-version upgrade, which must be PostgreSQL 17 or later;
• a replacement Postgres image or configuration. The postgres:17-alpine image, for example, does not include the package.

Recover a server without the library

1. Install the package matching the server's major version.
2. Add pg_textsearch to shared_preload_libraries and restart Postgres.
3. Restart Core, then the other applications. Processes that kept running may have switched to native ranking and return to BM25 only after a restart.

The tables and BM25 indexes are intact; nothing needs to be rebuilt.

Return to native search

Remove BM25 while the library is still loaded:

1. Stop Core, Notebooks, and Mail, so that their setup cannot recreate the extension or its indexes.

2. In the Cloud database, remove the extension together with every BM25 index:

   sql

   DROP EXTENSION pg_textsearch CASCADE;

   Dropping only the indexes is not enough: while the extension remains, Core, Notebooks, and Mail create them again on their next start. Run the same statement in every other database on this server that has the extension.

3. Remove pg_textsearch from shared_preload_libraries and restart Postgres. Do this before starting Cloud again; otherwise setup can recreate the extension.

4. Start Core, then the other applications. Setup logs that the optional BM25 indexes are unavailable and keeps native search.

Mailboxes set to pg_textsearch then search natively; cld mail operator status --mailbox <mailbox> shows search postgres (fallback).

Restore a backup without the library

With default options, pg_restore into a server without the library restores every table and row, but exits with status 1: creating the extension and each BM25 index fails. The restored database then searches natively. With --single-transaction or --exit-on-error, the restore stops at the extension before any table exists. To restore a custom-format dump without these errors, with or without these options, leave the BM25 objects out:

pg_restore --list cloud.dump | grep -v -E 'pg_textsearch|_bm25_' > restore.list
pg_restore --use-list restore.list --dbname <target-database> cloud.dump

Compare the two lists first: only the extension, its comment, and the indexes in the table above may be missing. To keep BM25, restore into a server that preloads the library instead.

A plain-format dump restored with psql reports the same errors. By default psql continues, restores the data, and exits with status 0; with ON_ERROR_STOP it stops at the extension before any table exists. Check that every error names pg_textsearch or bm25, or restore into a server that preloads the library.

Files capability downloads

The Filegate public origin must be reachable from the requesting user's browser or CLI. See private file lists and downloads for the capability flow, lease expiry, and access checks.

Optional Mail search ranking

Mail searches with native PostgreSQL full-text search on every supported version. With pg_textsearch installed as described in Optional BM25 search ranking, Mail ranks with BM25 once its index mail.message_contents_bm25_idx is valid. Mailboxes set to postgres keep native ranking.

Mail creates the index itself. After its schema setup, every Mail start checks for the extension and the index. If the index is missing or invalid, Mail builds it in the background with CREATE INDEX CONCURRENTLY:

• Mail starts and searches natively while the index builds; mail keeps arriving.
• One Mail process builds at a time, coordinated through a NATS lease. A build that another process or an operator is still running is left alone. Postgres hides which index another database role builds, so while another role builds any index in the same database, Mail logs that and leaves the build to its next start. Every step is a single statement without session state, so the build also works behind a transaction pooler.
• The build has no time limit. With pg_textsearch 1.5.1, a build reacts to a cancel request or statement_timeout only after it has read every message, so a limit would only discard a finished build.
• A build that fails is removed, logged as a warning from mail:migrate, and tried again on the next start. An invalid index left by an interrupted build, or an index of another kind under the same name, is replaced on the next start.

Build time grows with the stored message text. With 92,000 invented messages of about 1 KB of text each on PostgreSQL 17 and pg_textsearch 1.4.0, the build took 18 to 24 seconds and the index used 34 MB. While it runs, the build holds a lock that keeps autovacuum and other schema changes on Mail messages waiting. The current Mail setup does not need that lock once its indexes exist, but an older Mail image that starts during the build waits for it and can fail its setup until the build finishes.

To decide when the build runs, create the index yourself before Mail starts, for example in a maintenance window. Mail then finds the valid index and skips its build:

SET statement_timeout = 0;
CREATE INDEX CONCURRENTLY IF NOT EXISTS message_contents_bm25_idx
  ON mail.message_contents
  USING bm25 ((COALESCE(subject, '') || ' ' || COALESCE(subject, '') || ' ' || COALESCE(plain_text, '')))
  WITH (text_config = 'simple');

Check a mailbox with cld mail status --mailbox <mailbox> --json: search.pgTextsearchInstalled shows the extension, and search.bm25Ready turns true once the index is valid. To return one mailbox to native ranking, run cld mail configure --mailbox <mailbox> --search-backend postgres. To remove BM25 ranking for every mailbox, drop the index:

DROP INDEX CONCURRENTLY IF EXISTS mail.message_contents_bm25_idx;

Mail builds the index again on its next start while the extension stays installed, so set the mailboxes to postgres first if BM25 should stay off. Dropping Mail's index does not remove the need for the preloaded library while the other BM25 indexes exist; to stop using pg_textsearch entirely, follow Return to native search. See Mail search in large mailboxes for how search bounds its work.

Studio HTML self-test runtime

Assistant's existing Chromium host also runs code_check. Chromium launches with site isolation and isolated sandboxed iframes so an app loop cannot freeze the host. Only check mounts use the bundled inspection prelude with axe-core and its own pinned CSP hash; normal app mounts omit inspection code. No separate network asset or service is needed. Existing host admission and execution budgets apply, and each user holds at most 16 disposable check copies at a time. Disposable scopes are removed after checks; the service sweep reclaims scopes left by crashed hosts and queues unreachable database namespaces for deletion. See HTML app checks for data-copy and output limits. No new configuration is required.