OpenTelemetry

The zero-cache service embeds the JavaScript OTLP Exporter and can send logs, traces, and metrics to any standard otel collector.

To enable otel, set the following environment variables then run zero-cache as normal:

OTEL_EXPORTER_OTLP_ENDPOINT="<your otel endpoint>"
OTEL_EXPORTER_OTLP_HEADERS="<auth headers from your otel collector>"
OTEL_RESOURCE_ATTRIBUTES="<resource attributes from your otel collector>"
OTEL_NODE_RESOURCE_DETECTORS="env,host,os"

Grafana Cloud Walkthrough

Here are instructions to setup Grafana Cloud, but the setup for other otel collectors should be similar.

Sign up for Grafana Cloud (Free Tier)
Click Connections > Add Connection in the left sidebar
Search for "OpenTelemetry" and select it
Click "Quickstart"
Select "JavaScript"
Create a new token
Copy the environment variables into your .env file or similar
Start zero-cache
Look for logs under "Drilldown" > "Logs" in left sidebar

Distributed Tracing

You can enable end-to-end trace correlation from your frontend through zero-cache to your API server. This allows you to see the full request flow in your tracing UI.

To enable this, provide a getTraceparent callback when creating your Zero client:

import {ZeroProvider} from '@rocicorp/zero/react'
import {propagation, context} from '@opentelemetry/api'
 
function getTraceparent() {
  const carrier: Record<string, string> = {}
  propagation.inject(context.active(), carrier)
  return carrier.traceparent
}
 
return (
  <ZeroProvider
    /* ... other options ... */
    getTraceparent={getTraceparent}
  >
    <App />
  </ZeroProvider>
)

This callback is called before sending WebSocket messages that trigger API server calls (push, changeDesiredQueries, initConnection). The returned W3C traceparent header is forwarded through zero-cache to your API server, where it can be used to continue the trace.

Metrics Reference

🤔Histogram support

zero.server

Metric	Type	Unit	Description
`zero_server_uptime`	Gauge	s	Cumulative uptime, starting from when requests are served
`zero_server_api_requests`	Counter		Calls to user mutate and query APIs, including cleanup and auth-validation operations
`zero_server_api_request_duration`	Histogram	s	End-to-end user API request duration, including retries
`zero_server_api_attempts`	Counter		HTTP fetch attempts made while calling user API endpoints
`zero_server_api_attempt_duration`	Histogram	s	Duration of each API HTTP attempt, excluding retry delays
`zero_server_api_in_flight`	UpDownCounter		API requests currently in flight
`zero_server_startup_duration`	Histogram	s	Time from starting `zero-cache` until it is ready
`zero_server_worker_startup_duration`	Histogram	s	Time from starting a worker until it is ready

zero.replica

Metric	Type	Unit	Description
`zero_replica_db_size`	Gauge	bytes	Size of the replica's main db file (excludes WAL)
`zero_replica_wal_size`	Gauge	bytes	Size of the replica's WAL file
`zero_replica_wal2_size`	Gauge	bytes	Size of the replica's WAL2 file (only if using wal2 mode)
`zero_replica_backup_lag`	Gauge	ms	Time since last litestream backup. Expected to sawtooth from 0 to `ZERO_LITESTREAM_INCREMENTAL_BACKUP_INTERVAL_MINUTES`
`zero_replica_purge_blocked`	Counter		Number of change-log purges blocked because the actual backup state could not be verified or is stale
`zero_replica_litestream_restore_runs`	Counter		Litestream restore runs
`zero_replica_litestream_restore_attempts`	Counter		Litestream restore subprocess attempts
`zero_replica_litestream_restore_db_bytes`	Counter	bytes	SQLite database bytes restored by successful Litestream restores
`zero_replica_litestream_restore_duration`	Histogram	s	Wall-clock duration of Litestream restore runs
`zero_replica_litestream_restore_wait_duration`	Histogram	s	Time spent waiting for replication-manager snapshot status before restoring
`zero_replica_litestream_restore_process_duration`	Histogram	s	Wall-clock duration of Litestream restore subprocesses
`zero_replica_litestream_restore_validation_duration`	Histogram	s	Time spent validating restored replica databases
`zero_replica_litestream_backup_process_runs`	Counter		Litestream backup process exits
`zero_replica_litestream_backup_process_duration`	Histogram	s	Runtime of Litestream backup subprocesses before exit
`zero_replica_litestream_backup_list_duration`	Histogram	s	Time to list the Litestream backup destination
`zero_replica_litestream_backup_verification_duration`	Histogram	s	Time to verify backup state in the destination
`zero_replica_litestream_snapshot_reservation_duration`	Histogram	s	Time snapshot reservations are held while view-syncers restore and subscribe

zero.replication

Metric	Type	Unit	Description
`zero_replication_upstream_lag`	Gauge	ms	Latency from sending a replication report to receiving it in the stream
`zero_replication_replica_lag`	Gauge	ms	Latency from receiving a replication report to it reaching the replica
`zero_replication_total_lag`	Gauge	ms	Measured end-to-end latency of the most recently received replication report; does not grow if reports stop arriving
`zero_replication_last_total_lag`	Gauge	ms	Alias of `zero_replication_total_lag`, retained for dashboards that explicitly use the non-extrapolated metric
`zero_replication_upstream_clock_skew`	Gauge	ms	Estimated offset of the upstream database clock relative to `zero-cache`; positive values mean upstream is ahead
`zero_replication_lag_report_retries`	Counter		Replication lag reports retried because an expected report did not arrive before the next report interval
`zero_replication_events`	Counter		Number of replication events processed
`zero_replication_transactions`	Counter		Count of replicated transactions
`zero_replication_changes`	Counter		Count of replicated changes, including DML and DDL statements
`zero_replication_slot_health`	Gauge	1	One-hot status for the active logical replication slot: `ok`, `unreserved`, `lost`, `missing`, or `unknown`
`zero_replication_slot_retained_wal_bytes`	Gauge	bytes	WAL bytes retained by the active logical replication slot
`zero_replication_slot_safe_wal_bytes`	Gauge	bytes	Remaining WAL capacity before the active logical replication slot is lost; omitted when Postgres reports no value
`zero_replication_initial_sync_runs`	Counter		Number of initial-sync runs
`zero_replication_initial_sync_duration`	Histogram	s	Wall-clock duration of an initial-sync run
`zero_replication_initial_sync_copy_duration`	Histogram	s	Wall-clock duration of the COPY phase for a successful initial-sync run
`zero_replication_initial_sync_copy_other_duration`	Histogram	s	Initial-sync duration excluding SQLite flush and index time for a successful run
`zero_replication_initial_sync_flush_duration`	Histogram	s	Total SQLite flush time for a successful initial-sync run
`zero_replication_initial_sync_index_duration`	Histogram	s	SQLite index creation time for a successful initial-sync run
`zero_replication_initial_sync_rows`	Counter		Rows copied during successful initial-sync runs
`zero_replication_initial_sync_copy_stream`	Counter	bytes	PostgreSQL COPY stream bytes, including failed runs; reported in approximately 8 MiB batches and flushed when the stream ends
`zero_replication_initial_sync_completed_copy_stream`	Counter	bytes	PostgreSQL COPY stream bytes processed during successful initial-sync runs
`zero_replication_initial_sync_copy_chunks`	Counter		PostgreSQL COPY stream chunks processed during initial sync; batched with COPY-stream updates and flushed when the stream ends
`zero_replication_shadow_sync_runs`	Counter		Number of shadow initial-sync runs, labeled by `result`
`zero_replication_shadow_sync_duration`	Histogram	s	Wall-clock duration of a shadow initial-sync run, labeled by `result`
`zero_replication_flow_control_active_subscribers`	Gauge		Active change-stream subscribers receiving live changes
`zero_replication_flow_control_queued_subscribers`	Gauge		Change-stream subscribers waiting for the current transaction to finish before activation
`zero_replication_flow_control_pending_messages`	Gauge		Downstream change-stream messages not yet acknowledged by subscribers
`zero_replication_flow_control_backlog_messages`	Gauge		Live change-stream messages buffered while subscribers catch up
`zero_replication_flow_control_backlog_bytes`	Gauge	bytes	Live change-stream bytes buffered while subscribers catch up
`zero_replication_flow_control_max_backlog_bytes`	Gauge	bytes	Maximum live change-stream bytes buffered by a single subscriber
`zero_replication_flow_control_waits`	Counter		Completed flow-control checkpoints
`zero_replication_flow_control_wait_duration`	Histogram	s	Time replication waits at flow-control checkpoints

zero_replication_total_lag and zero_replication_last_total_lag now report the same latest measured round trip and do not grow when reports stop arriving. Use zero_replication_lag_report_retries to detect a stalled or missing report stream.

zero.sync

Metric	Type	Unit	Description
`zero_sync_max_protocol_version`	Gauge		Highest sync protocol version seen from connecting clients
`zero_sync_active_clients`	UpDownCounter		Number of currently connected sync clients
`zero_sync_active_client_groups`	Gauge		Number of active ViewSyncerService instances in a syncer worker
`zero_sync_queries`	Gauge		Active IVM pipelines across all client groups in a syncer worker
`zero_sync_rows`	Gauge		CVR-tracked rows across all client groups in a syncer worker
`zero_sync_serving_lag`	Gauge	ms	Longest time locally ready replica changes have remained unserved across eligible active client groups
`zero_sync_serving_lag_stats`	Gauge	ms	Distribution of serving lag across eligible active client groups
`zero_sync_serving_lagging_client_groups`	Gauge		Eligible active client groups with locally ready replica changes not yet served to clients
`zero_sync_view_syncer_lag`	Histogram	s	Time from replica changes becoming ready to ViewSyncer output, sampled once per minute per eligible group
`zero_sync_view_syncer_hydration`	Histogram	s	Time from a ViewSyncer query sync requiring hydration until output, per client group
`zero_sync_e2e_serving_lag`	Histogram	s	Completion latency from the upstream transaction commit through ViewSyncer output
`zero_sync_e2e_serving_lag_clamps`	Counter		Negative end-to-end lag observations clamped to zero because the upstream clock was ahead
`zero_sync_lock_wait_time`	Histogram	s	Time spent waiting to acquire the ViewSyncerService lock per operation
`zero_sync_pipeline_resets`	Counter		Count of pipeline resets, labeled by `reason`
`zero_sync_hydration`	Counter		Number of query hydrations
`zero_sync_hydration_time`	Histogram	s	Time to hydrate a query
`zero_sync_advance_time`	Histogram	s	Time to advance all queries for a client group after applying a transaction
`zero_sync_poke_time`	Histogram	s	Time per poke transaction (excludes canceled/noop pokes)
`zero_sync_poke_transactions`	Counter		Count of poke transactions
`zero_sync_poke_rows`	Counter		Count of poked rows
`zero_sync_cvr_load_attempts`	Counter		CVR load attempts
`zero_sync_cvr_load_duration`	Histogram	s	Time to load a CVR
`zero_sync_cvr_flush_attempts`	Counter		CVR flush attempts
`zero_sync_cvr_flush_time`	Histogram	s	Time to flush a CVR transaction
`zero_sync_cvr_rows_flushed`	Counter		Number of changed rows flushed to a CVR
`zero_sync_websocket_open_connections`	UpDownCounter		Open client WebSocket connections
`zero_sync_websocket_connection_attempts`	Counter		Client WebSocket connection attempts
`zero_sync_websocket_connection_successes`	Counter		Client WebSocket connections successfully initialized
`zero_sync_websocket_connection_failures`	Counter		Client WebSocket connection attempts that failed before initialization
`zero_sync_websocket_errors`	Counter		Client WebSocket error events
`zero_sync_ivm_advance_time`	Histogram	s	Time to advance IVM queries in response to a single change
`zero_sync_ivm_conflict_rows_deleted`	Counter		Rows deleted because they conflicted with an added row
`zero_sync_query_transformations`	Counter		Number of query transformations performed
`zero_sync_query_transformation_time`	Histogram	s	Time to transform custom queries via API server
`zero_sync_query_transformation_hash_changes`	Counter		Times a query transformation hash changed
`zero_sync_query_transformation_no_ops`	Counter		Times a query transformation was a no-op
`zero_sync_query_row_set_signature_drifts`	Counter		Unchanged query rehydrations whose row-set signature differs from the CVR, forcing a config-version bump
`zero_sync_query_same_hash_rehydrations_forced_bump`	Counter		Same-hash query rehydrations that force a config-version bump so changed rows are delivered

Serving-lag metrics include only client groups with at least one connected client and a validated background connection context. Retained groups without an eligible connection do not contribute lag.

zero.mutation

Metric	Type	Description
`zero_mutation_crud`	Counter	Number of CRUD mutations processed
`zero_mutation_custom`	Counter	Number of custom mutations processed
`zero_mutation_pushes`	Counter	Number of pushes processed

PreviousConfiguration

NextInspector

For AI assistants (Claude, Codex, Cursor, OpenCode, etc.)

Get Started

Learning Zero

Using Zero

Postgres

Integrations

Deployment

Self-Host

Debugging

Old Stuff

Meta

OpenTelemetry

Grafana Cloud Walkthrough

Distributed Tracing

Metrics Reference

zero.server

zero.replica

zero.replication

zero.sync

zero.mutation

On this page

Self-Host

OpenTelemetry

On this page

Search