Observability

Logs

NAuth uses readable text logs with an info level by default. Set the level to debug when verbose reconciliation logging is needed. Enable JSON when your log pipeline expects structured records.

Helm:

helm upgrade --install nauth oci://ghcr.io/wirelesscar/nauth \
  --namespace nauth \
  --create-namespace \
  --set logging.format=json \
  --set logging.level=info

Binary:

manager --log-format=json --log-level=info

Formats: text, json. Levels: debug, info, warn, error.

Enable verbose logging for local troubleshooting:

helm upgrade --install nauth oci://ghcr.io/wirelesscar/nauth \
  --namespace nauth \
  --create-namespace \
  --set logging.level=debug

Metrics

Enable the controller-runtime /metrics endpoint:

helm upgrade --install nauth oci://ghcr.io/wirelesscar/nauth \
  --namespace nauth \
  --create-namespace \
  --set monitoring.enabled=true

The chart exposes the endpoint through an HTTP Service on port 8080 with the port name http-metrics. The chart configures the operator with --metrics-secure=false, so restrict access to the metrics Service according to your cluster’s security requirements. With the release name and namespace used in the Helm command, the Service is nauth-metrics-service and its cluster DNS name is nauth-metrics-service.nauth.svc:8080.

Core reconcile metric:

controller_runtime_reconcile_total{controller="<controller>",result="<result>"}

Controller labels:

Resource	Controller label	Availability
Account	account	Since v0.1.0
AccountExport	accountexport	Since v0.7.0
AccountImport	accountimport	Since v0.7.0
AccountSigningKey	accountsigningkey	Since v0.7.2
User	user	Since v0.1.0
NatsCluster	natscluster	Since v0.6.1

Grafana queries:

These queries use labels emitted by controller-runtime and do not filter on service. The chart’s service: nauth-metrics-service label selects the Kubernetes Service for Prometheus Operator discovery; a static OpenTelemetry scrape does not add that label to the exported time series. If you scrape multiple NAuth installations, add the target labels supplied by your scrape system, such as job or namespace.

sum by (controller, result) (rate(controller_runtime_reconcile_total{controller=~"account|accountexport|accountimport|accountsigningkey|user|natscluster"}[5m]))

sum by (controller) (increase(controller_runtime_reconcile_total{controller=~"account|accountexport|accountimport|accountsigningkey|user|natscluster"}[15m]))

sum by (controller) (rate(controller_runtime_reconcile_errors_total{controller=~"account|accountexport|accountimport|accountsigningkey|user|natscluster"}[5m]))

Prometheus operator

Enable the chart’s ServiceMonitor and PrometheusRule resources when the Prometheus Operator CRDs are installed:

helm upgrade --install nauth oci://ghcr.io/wirelesscar/nauth \
  --namespace nauth \
  --create-namespace \
  --set monitoring.enabled=true \
  --set monitoring.serviceMonitor.enabled=true

The ServiceMonitor selects the metrics Service using the fixed label service: nauth-metrics-service. The chart’s bundled reconciliation-error alerts cover Account, AccountExport, AccountImport, User, and NatsCluster. AccountSigningKey metrics are exposed, but the chart does not currently include a bundled alert for that controller.

OpenTelemetry Collector scrape example:

The following static target assumes the release name is nauth and the namespace is nauth. If either value differs, use the generated metrics Service name and namespace instead. To find the Service name:

kubectl get service -n nauth \
  -l service=nauth-metrics-service \
  -o jsonpath='{.items[0].metadata.name}{"\n"}'

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: nauth
          metrics_path: /metrics
          static_configs:
            - targets:
                - nauth-metrics-service.nauth.svc:8080

exporters:
  otlp:
    endpoint: grafana-agent.grafana.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]

Account readiness

Since v0.7.5: Account readiness requires a matching, complete NATS Account observation.

NAuth marks an Account as ready only after NATS reports the desired Account JWT as complete through ACCOUNTZ.

A successful JWT upload alone is not sufficient for Ready=True.

Validation workflow

When validation is needed, NAuth performs the following sequence over its system-account connection:

1. Ensure the desired Account JWT is available in NATS, uploading it when necessary.
2. Publish {"acc":"<account-id>","subject":"x"} to $SYS.REQ.ACCOUNT.NSUBS and flush the connection to request runtime Account materialization. NAuth uses x as a placeholder subject required by this API and ignores the subscription count. NAuth does not wait for an application-level response; the flush does not confirm that NATS loaded the Account.
3. Query $SYS.REQ.SERVER.<server-id>.ACCOUNTZ on the same connection.
4. Accept the observation only when the server ID and Account ID match the expected identities and the observed claims hash matches the desired claims hash.

Ready=True requires the accepted observation to also report Complete=true. The materialization request itself is never readiness evidence.

A fresh successful validation is reused until the normal Account validation interval expires, skipping both the materialization request and ACCOUNTZ query. Incomplete or Unknown validation uses a fixed 10-second requeue interval; NAuth does not block reconciliation with a sleep loop.

The NATSAccountComplete condition describes the latest completeness result:

• True: the desired JWT claims hash was observed and NATS reported Complete=true.
• False: NATS reported an incomplete Account or returned a claims hash that does not match the desired state.
• Unknown: NAuth could not establish Account completeness, for example because the connected NATS server does not support the required ACCOUNTZ API or the request was inconclusive.

Ready follows the same NATS result unless another reconciliation requirement reports an explicit failure. Consequently, an inconclusive NATS observation can leave both NATSAccountComplete and Ready as Unknown.

When NATS reports invalid imports, the NATSAccountComplete=False message includes bounded troubleshooting details when available. The message may contain up to five invalid imports and indicate how many additional imports were omitted.

Inspect Account conditions and the last successful NATS observation with:

kubectl get account my-account -n my-team \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}): {.message}{"\n"}{end}'
kubectl get account my-account -n my-team \
  -o jsonpath='{.status.nats}{"\n"}'

The status.nats fields describe the last successful, identity-matched ACCOUNTZ observation, including the observed server ID, observed claims hash, and validation timestamp. NAuth observes the server reached by its current NATS connection and does not query every server in the cluster. See ADR-6 for the observation decision and its trade-offs.

Server-targeted ACCOUNTZ is currently supported from NATS v2.2.0 onward. Older NATS versions may leave Account completeness and Ready as Unknown instead of allowing JWT upload success to produce Ready=True.