Observability

Start with gateway health. Then narrow the problem to an application, route, background source, or dependency.

Check the deployment

cld admin status
cld admin apps list

Gateway health shows registered applications, route count, and healthy, degraded, or offline instances.

Use cld admin diagnose for a bounded snapshot of health, logs, telemetry, jobs, Postgres, Valkey, and metrics.

See CLI modules for authentication and output formats.

Read logs

Use structured fields to filter by:

• application source;
• level;
• request ID;
• trace ID;
• route;
• actor or resource identifier when safe.

Do not log secrets, session tokens, authorization headers, prompts, or model output.

See Logging for application APIs.

Trace a request or operation

Request middleware publishes route templates to gateway telemetry. middleware.logger() records 5xx, 429, 401, and 403 responses. Notifications and structured AI create trace spans. Add explicit spans around other application work when you need end-to-end tracing.

Use one trace to answer:

• where time was spent;
• which dependency failed;
• whether a retry ran;
• whether the operation finished or was abandoned.

See Tracing for span APIs.

Diagnose page speed

SSR responses expose Server-Timing durations in milliseconds for auth, settings, runtime, ssr_data, ssr_finalize, and ssr_render. Each phase measures its own work; ssr_render includes HTML serialization and the template. Phases absent from a request remain absent. These timings exclude network transfer and browser work.

Browser collection is off by default. Enable the Core setting observability.web_vitals.enabled temporarily to diagnose page performance. Authenticated SSR pages then report LCP, INP, and CLS to /api/me/web-vitals. Core builds and serves one shared, versioned web-vitals bundle. Applications use the same asset URL and browser cache entry. SSR reads the existing request settings snapshot; it adds no settings lookup for this decision.

Each report creates an entry in the existing logs; collection is intended for controlled measurement windows, not permanent full-traffic analytics. Disabling the setting suppresses the script on subsequent pages and discards reports from already-open pages after the normal shared settings cache invalidation. Server-Timing remains available independently of browser collection.

Open Observability → Telemetry → Browser (/admin/observability/telemetry?view=browser) to view these measurements and switch collection on or off. The switch changes the same global Core setting; both reading and changing it require an administrator. Existing pages need a reload to start collection. Turning collection off leaves retained history visible. The Server view continues to show gateway telemetry separately.

The Browser view offers app, exact route-template, and time-window filters (1 hour to 30 days). It shows p75 and sample counts for each metric, time buckets in side-by-side ChartExplorers, and an integrated paginated route table. The charts share a time axis and inspection cursor for comparison. Lines connect only adjacent buckets; isolated measurements remain visible. ChartExplorer supports point selection and a table view of the same aggregate data. LCP and INP use milliseconds; CLS has no unit. Missing metrics display a dash, and gaps are not interpolated. A loading failure is reported separately from an empty selection. Counts describe metric measurements, not unique users or visits.

Server telemetry also offers synchronized charts for request counts, HTTP error percentage (status 400 or higher), and response duration. Duration shows the request-weighted mean and maximum from existing minute rollups, not a percentile. The overview aligns server-error counts to the log chart's coarser intervals before synchronizing inspection. Missing telemetry buckets remain gaps.

Logs and notification delivery views include a separate chart for uncommon warning or failure outcomes alongside their volume chart. Existing filters apply to both charts. ChartExplorer provides exact counts, formatted units, local table sorting, and copying without loading more data. Postgres and Redis rankings use horizontal bars with a shared scale within each chart; full names remain in inspection and the table view. Redis sampling and Postgres row-estimate notices still apply to these charts.

Aggregation keeps the last valid report per app, route, metric name, and metric ID within the selected window. Buckets use the last report's receipt time in UTC, not navigation start. Overall p75 is calculated from those reports, not from bucket percentiles. Standard log retention can remove older measurements before a query runs; a 30-day selection does not guarantee 30 days of retained history. These self-reported, signed-in samples are operational diagnostics, not a representative public-user benchmark.

The view reads existing logs without another data store or rollup job. Core's normal logging migration installs the metadata decoder used by aggregation; run the updated Core migration before serving the updated Browser view. This supports both object metadata and the JSON-string metadata written by the existing logger. Query cost grows with retained reports in the selected window, so prefer a short window when investigating a high-volume installation.

Find the underlying client-reported diagnostics in logs with source web-vitals. Reports contain the application ID, route template, metric ID, value, navigation type, and server timings. They contain no concrete URL, query string, DOM attribution, page text, or user ID. The endpoint shares the authenticated self-service API's rate limit and accepts at most 4096 bytes per report.

LCP and INP use milliseconds; CLS is unitless. Missing INP means no supported interaction measurement, not zero latency. Reports may arrive when the page is hidden. Keep the latest value per metric ID before aggregating; returning from the back-forward cache produces a new measurement. These are document navigation metrics, not separate measurements for every in-page tab change.

Compare cold and warm visits to the same production build, including its precompressed assets, on a machine without competing builds. Keep browser metrics separate from gateway duration and development-server timings.

Inspect routes and background work

Route telemetry uses the route template, not the concrete URL. This keeps one series for /api/inventory/items/:id.

When no application reported a template (unmatched route, upstream failure, or an app without the platform middleware), the gateway derives one with the shared path redaction. It also replaces numbers with :n, keeps at most eight segments, and reports (other) after 200 distinct derived templates per app.

The Upstream unavailable log carries the same redacted template without the 200-template limit, the error type, and a transport error code such as ConnectionRefused or ENOTFOUND. It never includes the error message or the query.

When a client disconnects before the application answers, the gateway cancels the upstream request and records status 499 with the derived template. Such a request counts toward the HTTP error percentage (status 400 or higher), but not as a server error or an upstream failure, and it writes no log.

Sort by error rate to find unhealthy routes. Sort by requests to find the highest traffic.

For background work, inspect the latest run and then its history. A stuck run is an abandoned span, not proof that a worker is still active. The Sync page lists queue, job, and topic-consumer dead letters. Topic recovery targets the original consumer; historical entries without the original replay metadata remain inspectable but cannot be replayed. Use the separate NATS page for broker nodes, storage, stream replication, and consumer backlog. Follow NATS operations for diagnostics access, recovery, health webhooks, and independent outage monitoring.

Use the dedicated pages for:

• jobs and queues;
• workflow observability;
• notifications.

Inspect backfills

Open Admin > Observability > Jobs and select the Backfills type to find recorded runs over existing data. Narrow by source, inspect the latest outcome, then open its run history. The same read-only checks are available in the CLI:

cld admin jobs list --type backfill --window 7d --json
cld admin jobs runs --source auth:ipa:backfill --window 7d --json
cld admin jobs show <traceId>:<spanId> --json

The source example is Core's FreeIPA expiry backfill. Use the source returned by jobs list for another operation and the run ID returned by jobs runs for jobs show. An empty list can mean no run exists in the selected window.

Pump summaries include dispatched and failed counts. A completed producer can still have unfinished or failed downstream jobs; for FreeIPA, also inspect source auth:ipa:backfill:account and its dead letters. Use the owning app's recovery controls rather than treating a trace as a resumable work item. See Tracing for category selection and automatic pump tracing.

Read capability calls

Use Observability > Capabilities for the single history of dispatched capability calls. The assistant, MCP, HTTP callers, and app-to-app mandates all appear in the same list, so "what did an agent do, on whose behalf, and did it work" is one question rather than one per surface.

The summary reports executions, failures, denials, destructive actions, and average and 95th-percentile duration for the selected range, with per-app and per-capability breakdowns limited to the 50 busiest entries. Filter by app, capability, origin, status, user, destructive-only, and time range; every filter and the keyset cursor stay in the URL. Opening a row shows its request ID, actor, access subject, timing, error code, idempotency key, and the shape of the input and result.

Rows carry correlation and shape, never payloads, and are pruned after 90 days. Correlate an entry with application audit events and request logs through its request ID, and follow an assistant call to Admin > AI > AI Usage for that user. See Capabilities for the recorded fields and status meanings. The same data is available at /api/gateway/capabilities.

Operate AI workloads

Use Admin > AI > AI Usage for bounded 24-hour, 7-day, 30-day, or 90-day product and cost analysis. This is an AI administration surface rather than a general observability page. It reports interactive turns, token and configured credit usage, pricing coverage, model use, generation speed, per-user usage, application capability calls and failures, mid-chat model switches, application-launched chats, message ratings, and background AI failures.

Headline metrics and user/model comparisons combine chat and background inference. Tool calls have separate counts and do not add inference charges. Missing token or price measurements remain unavailable; reported zero values remain zero. Credits follow configured model prices and are not a provider invoice.

Use the four views to inspect totals, compare users and models, read feedback, and open complete stored errors. User, model, provider-model and application filters apply across views. Filter chips apply selections immediately; data notes are expandable. Lists have bounded pagination; range, filters and snapshot time remain in the URL. Charts use UTC buckets with localized dates and zero for inactive intervals. The same reports, filters and run details are available through the admin CLI. See AI usage and feedback for metric definitions, attribution limits, API access and CLI examples.

Retrying or editing a message preserves previously recorded chat token and credit usage. Workflow inference is counted once through structured accounting; workflow task status and retries remain available in the workflow views. Compaction appears as chat-compaction in Background AI.

Historical totals depend on retained source records and may omit unavailable usage. They are an operational cost signal, not an immutable billing ledger.

The report uses durable AI facts and does not expose prompts or private message content. Background records contain task, application, model, usage, duration, repair information, and bounded error metadata. Feedback reasons and comments are visible to administrators.

Monitor:

• queued, running, failed, and attention-needed turns;
• provider latency and errors;
• token usage;
• tool duration, approval, timeout, and failure;
• worker lease recovery;
• conversation file size;
• structured-task repair and failure counts.

Cloud tracing records model and tool metadata. It does not record prompt or output content by default.

Set provider, tool, output, file, and worker limits before production. Test provider failure, stream reconnect, abort, approval denial, and worker restart.

See Chat interface for browser state and shared chat components.

Scrape platform metrics

Use Admin > Observability > Metrics to create a dedicated scrape token. Send it as a Bearer token to /metrics from Pulse or a Prometheus-compatible collector. The token is restricted to metrics; anonymous requests are rejected.

If Pulse reaches /metrics through an internal address, add the exact host to PULSE_SCRAPE_ALLOWED_TARGETS in the Pulse container environment. For example, allow the gateway service name when that is the host in the metrics source URL.

Gateway Ops caches each snapshot for 15 seconds and shares concurrent refreshes. Collectors have bounded response times. A timed-out collector keeps its execution slot until its underlying work finishes; later scrapes report it unavailable instead of starting overlapping work. This limit applies per Gateway Ops process. Other collectors continue to report their results. Check cloud_metrics_collector_success to distinguish a partial scrape from a healthy one, even when the HTTP response succeeds.

Redis metrics contain aggregate keyspace, expiry, and sampling counts. Raw keys and dynamic key-prefix labels are not exported; inspect prefix samples on the administrator-only Redis diagnostics page. The former cloud_redis_prefix_sample_keys series is removed; use cloud_redis_keys_total for total key counts by database.

Alert on user impact

Useful alerts include:

• gateway cannot reach a required application;
• a required application disappears from the registry;
• the orchestrator reports too few healthy replicas;
• route error rate or latency exceeds its threshold;
• background work is failed or stuck;
• Postgres or Valkey is unavailable;
• delivery queues grow without progress.

Alert on sustained conditions. A single failed request is not deployment health.