Identity key operations

Core is the only private-key authority for platform session, invocation, and OAuth tokens. It stores RSA private JWKs encrypted in PostgreSQL with a Core-only key-encryption key (KEK). Applications verify sessions only from /.well-known/cloud-session-jwks.json and invocations only from /.well-known/cloud-invocation-jwks.json. OAuth keys are published through /.well-known/jwks.json; the validators and key purposes remain mutually exclusive.

If applications share a PostgreSQL credential, the database cannot isolate signing-key rows from a compromised application using that credential. Keep the key-encryption secret exclusive to Core, and use separate database roles when database-level isolation is required.

Normal signing-key rotation

Core performs maintenance for the session, invocation, and OAuth purposes every minute. The target signing lifetime is 30 days. It publishes a pending public key at least ten minutes before activation, keeps the active signer in memory for at most one minute, and leaves retired public keys available until every issued token plus clock and rollout margin has expired. Rotation is protected by a purpose-scoped PostgreSQL transaction lock, so multiple Core instances converge on one active and at most one pending key.

Session issuance extends its signing key's verification deadline in the same transaction that creates the session family. Long configured sessions therefore remain verifiable even if Core stops before the next maintenance pass.

If downtime outlasts the active and pending signing deadlines, Core retires those signers and creates an immediately active replacement during startup. It preserves their existing public verification windows. Like initial startup and emergency replacement, this recovery cannot provide the normal ten-minute prepublication period; verifiers use their bounded unknown-key refresh to load the replacement. A verifier in its refresh cooldown can briefly reject new tokens, but startup does not require another restart or a maintenance timer to recover.

Each purpose-specific JWKS response has its own ETag and a five-minute public cache bound. Unknown kid values trigger the verifier's JWKS refresh behavior. New issuance fails closed when Core cannot load a validated active signer; verification can continue from public material and warm caches.

OAuth access-token verifiers independently cache the compatible /.well-known/jwks.json response for the same five-minute bound. Configure CLOUD_OAUTH_JWKS_ORIGIN with the private OAuth application origin. A warm request verifies locally, then resolves the current client and actor in one PostgreSQL query. It does not query either signing-key table. An unknown kid gets one bounded refresh attempt; a removed or emergency-revoked known key can remain usable only until that local cache expires, at most five minutes.

Every application must be able to reach Core's JWKS before JWT issuance is enabled. Set CLOUD_IDENTITY_JWKS_ORIGIN to the private Core service origin so a cold start or unknown kid does not depend on public DNS, ingress, or hairpin routing. The token issuer remains the public HTTPS Cloud origin; only the public-key transport uses the private address. A warm verifier can tolerate a brief Core outage.

Private HTTP requires a trusted transport network; it is not secure merely because a JWKS contains public keys. Follow the network, Core-first rollout, and clock requirements in Runtime configuration.

Invocation JWTs have a nominal 30-second lifetime and a dedicated two-second clock tolerance. A correctly issued token can therefore remain acceptable for at most 32 seconds after its iat; browser sessions retain their separate 30-second clock tolerance.

Inspect key metadata with the admin endpoint:

GET /api/admin/identity/keys

The response contains lifecycle metadata and the KEK fingerprint, never private key material or token contents.

Rotate the KEK

Generate a new KEK:

openssl rand -hex 32

Then perform a three-phase rolling rewrap:

1. keep the old current KEK and distribute the new value as CLOUD_IDENTITY_NEXT_KEY to every Core replica;
2. after every replica is ready, promote the new value to CLOUD_IDENTITY_KEY_ENCRYPTION_KEY, move the old value to CLOUD_IDENTITY_PREVIOUS_KEY, remove CLOUD_IDENTITY_NEXT_KEY, and roll every replica again;
3. confirm every key reports the new encryptionKeyId;
4. call POST /api/admin/identity/keys/rewrap safely if another idempotent pass is desired;
5. remove CLOUD_IDENTITY_PREVIOUS_KEY and roll every Core replica a final time.

The first phase matters: an old-current replica can decrypt new ciphertext via its pre-distributed next KEK while the second phase is still rolling. Never skip directly to current=new, previous=old when an old Core replica may still be running.

Core decrypts with the current or previous KEK, validates every private/public pair, encrypts only with the current KEK, and updates all rows transactionally. Readiness fails if ciphertext is corrupt, the KEK is wrong, or a pair does not match. Back up the database and the current KEK through the deployment's secret and backup systems before beginning.

Emergency signing-key revocation

Use the authenticated admin endpoint with a durable incident reason:

POST /api/admin/identity/keys/<kid>/revoke
Content-Type: application/json

{"reason":"suspected private-key exposure"}

The key disappears from newly generated JWKS responses and Core immediately creates a replacement signer when the revoked key was active. Already warm verifier and intermediary caches can retain the compromised public key for at most the configured five-minute JWKS cache bound. Treat that interval as part of the incident blast radius. Session-family and user-epoch revocation remains available when all sessions must be invalidated immediately at the database authorization step.

Revoking a session signing key also ends the mobile app's sessions signed with it. Paired phones are not signed out: their device keys stay valid, and each phone gets a new app session from the replacement signer the next time it starts. To sign phones out, remove them or revoke the account's sessions.

Invocation and OAuth issuance confirm the prepared signer against PostgreSQL while holding a shared lock on its active key row. A capability or widget call checks once; Universal Search checks once for the whole target fan-out and reuses that signer. Emergency revocation waits for an already-started signing batch to finish. After the revocation transaction commits, no Core replica can release a new invocation or OAuth token under that kid; a replica with a stale signer cache refreshes once and otherwise fails closed. This guarantee does not invalidate tokens that were already released, so their normal token and verifier-cache windows still apply.

Each issuance has one deadline that covers the whole call: the signer refresh, the identity settings, the wait for a database connection, the key check including lock waits, and signing. Core does not return a token after that deadline. If signing finishes later, Core rolls back its transaction instead of committing a result the caller never receives. The caller fails at the deadline with a TimeoutError that names the step that stalled, for example Identity service did not respond within 5 s (signer refresh).

All callers share one signer refresh and one identity settings load. Each gives up after 30 seconds, and the refresh also ends its own database statements then. Core logs Cloud identity signer refresh failed. While the cached key is still valid, Core keeps signing with it for one more signing-cache window of 60 seconds before it tries again. Without a valid cached key, the next issuance starts a new refresh. Code Mode tools keep retrying a timed-out issuance for about 35 seconds, so one stalled refresh does not end a running or approved call. A database failure may also prevent an audit entry from being saved; inspect service logs when no audit record is available.

Do not rotate APP_SECRET as a substitute. It encrypts settings and credentials and is deliberately not a signing-key or KEK fallback.

Maintainers can measure this guard and the complete search authentication path with the identity performance check. The benchmark reports latency and checks identity I/O budgets.