Metrics and Health

Prometheus metrics for the Vault API

The Vault API can expose HTTP request and gateway webhook metrics in Prometheus text format on a dedicated listener. Scrape that listener with Prometheus, Grafana, or curl. These series are separate from the Vault operations and Policy Node cwp_* metrics below.

📘

Note:

This scrape is off by default. Enable it by setting telemetry.enable_prom: true in wallet.yaml. When enabled, the metrics server listens on port 9090 by default (configurable via telemetry.prom_port). Metrics are served on this dedicated listener and are not routed on the public API ingress.

The following metrics are exposed on that listener when Prometheus is enabled:

In addition, when Prometheus is enabled the registry includes the standard Go and process collectors (for example go_*, process_*).

Prometheus Metrics for Vault Operations and Policy Nodes

Vault operations (the Core Wallet Platform control plane embedded in the wallet process) and Policy Nodes expose a separate Prometheus endpoint at GET /metrics. All metric names are prefixed with cwp_.

These metrics use the default Prometheus registry for that process. Metric names are registered in both the wallet and Policy Node processes; values are populated only where the corresponding code path runs.

Shared (Wallet orchestrator and Policy Nodes)

Wallet orchestrator (CWP_HTTP_PORT)

Policy Nodes (HealthCheckPort)

Standard Go and process collectors may also appear on these scrapes depending on the process runtime.

Distributed tracing (OpenTelemetry)

The wallet can export traces over OTLP/HTTP to a collector you operate (for example an OpenTelemetry Collector, Grafana Tempo, or Datadog Agent). Tracing is off by default and is independent of Prometheus metrics.

Traces stay inside your infrastructure. The wallet does not send spans to Blockdaemon.

Enable tracing

Set the following in wallet.yaml (see the wallet.yaml reference):

telemetry:
  enable_otlp: true
  service_name: wallet
  otel_endpoint: "https://otel-collector.example.com:4318"
  otel_sample_ratio: 1

Restart the wallet after changing these values. Allow egress from the wallet to the collector. telemetry.enable_prom does not have to be on.

Collector credentials

Do not put exporter API keys or auth headers in wallet.yaml. Set the standard OpenTelemetry environment variable OTEL_EXPORTER_OTLP_HEADERS on the wallet process (for example authorization=Bearer <token>), typically from a secret. Optional OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are also read at process start.

What is traced

The wallet records low-cardinality spans. Request and response bodies, headers, user identifiers, wallet addresses, amounts, and SQL text are not attached.

Policy Nodes do not export traces from this wallet setting.

When a span is active, wallet JSON logs include trace_id and span_id (see Logging). Use those together with correlationId and opID.

Health Status for the Wallet Stack Services

To maintain smooth operations, each service within the deployment provides health endpoints. Use HTTP status codes from those endpoints to drive monitoring and auto-restart.

Wallet (Vault API)

On the wallet API port (typically 8080):

• GET /live: HTTP 200 while the process is accepting HTTP (liveness).
• GET /ready: HTTP 200 when the API server is ready to receive traffic (readiness).

Policy Nodes

When HealthCheckPort is set in policy-node.conf:

• GET /live: HTTP 200 while the process is running (liveness).
• GET /ready: HTTP 200 only after the node has connected to the message broker and subscribed; 503 before then (readiness).

NATS (Message Broker)

On the NATS HTTP monitoring port (http_port in nats-server.conf, typically 8222):

• GET /healthz: HTTP 200 when the NATS server process is healthy. Deployments typically use this for both liveness and readiness probes.

EVM Tracker

When the tracker is started with --health-port (typically 8080):

• GET /livez: HTTP 200 while the process is running (liveness).
• GET /readyz: HTTP 200 when the chain RPC endpoint is reachable; 503 if the RPC client is unavailable or a block-number check fails (readiness).

Recommended practices

1. Probe the endpoints above on a regular interval.
2. Alert when status codes leave the expected success range (Wallet /live or /ready not 200; Policy Node /live not 200; Policy Node /ready stuck on 503; NATS /healthz not 200; EVM Tracker /livez not 200 or /readyz stuck on 503).
3. Auto-restart after repeated failures, with a failure threshold tuned for your environment.
4. Correlate failed probes with component logs (see Logging below; enable debug logging when investigating).

Logging

Institutional Vault components write structured JSON logs to standard output or standard error (one JSON object per line). Collect them with your platform log agent (for example CloudWatch, Azure Log Analytics, or Datadog) and index common fields for search and correlation.

Typical fields include:

OperationID and CorrelationID

• OperationID (opID in logs): The business identifier of a Vault / CWP operation (create account, makeTransaction, and so on). Clients may supply it when starting an operation for idempotency, then use it to poll status, approve, or search. It is the durable key for that operation in the system of record.
• CorrelationID (correlationId in logs and often in API error responses): A tracing identifier that links log lines across components for a single request or workflow. Use it when escalating to support. If no correlation ID is present on the request context, the system may fall back to the OperationID.

When an API call fails, responses may also include an M-prefixed error code (for example M0001, M0025, M0080). See the Troubleshooting Guide for the full M-code catalog, typical HTTP statuses, and resolution steps.

Log verbosity:

• Wallet: log_level in wallet.yaml controls the global level. Valid values:

  Value	Level
  -1	trace
  0	debug
  1	info
  2	warn
  3	error
  4	fatal
  5	panic
  7	disabled

  Default: 1 (info). To adjust verbosity for a single component without
  changing the global level, add a component_log_levels map — for example:

  YAML

  log_level: 1
  component_log_levels:
    canton_poller: 0   # debug this component only
    mpa_listener: 2    # warn this component only

  See the wallet.yaml reference for the full
  list of component names.

• Policy Nodes / Vault control plane: DEFAULT_LOG_LEVEL environment variable (debug, info, warn, error), read at process start.

• EVM Tracker: --verbose / -v flag. When set, logs at info level; when omitted, logs at warn level and above.