Operations - TLS, auth, health, metrics

How to run hopskip-server with TLS, authentication on every surface, and the probe/metrics endpoints cluster infrastructure expects.

Everything here is configured through environment variables, or through a TOML file passed with --config that maps 1:1 onto them (an explicitly set variable wins). The configuration reference lists every setting; hopskip-server check-config validates a deployment without starting it.

TLS

The public gRPC surface (clients + workers) terminates TLS when both of these are set:

Variable	Meaning
HOPSKIP_TLS_CERT_PATH	PEM server certificate (leaf + any intermediates)
HOPSKIP_TLS_KEY_PATH	PEM private key for that certificate
HOPSKIP_TLS_CLIENT_CA_PATH	Optional. PEM CA; when set, every client of this listener must present a certificate signed by it (mTLS)

Setting only one of cert/key is a startup error, as is a client CA without a server identity. Unset means plaintext: fine for hop dev on loopback, disqualifying for anything reachable from a network you don’t trust.

Every Hopskip client process (the CLI, the worker, your own hopskip-client code) reads one shared set of variables when dialing an https:// endpoint:

Variable	Meaning
HOPSKIP_TLS_CA_PATH	PEM CA to trust (private CA / self-signed deployments). Platform roots are used by default
HOPSKIP_TLS_CLIENT_CERT_PATH / HOPSKIP_TLS_CLIENT_KEY_PATH	PEM client certificate + key, presented when the server requires mTLS
HOPSKIP_TLS_DOMAIN	Override the hostname verified against the server certificate

http:// endpoints ignore all TLS variables and connect plaintext.

mTLS for in-cluster callers: the internal listener

The recommended worker↔server posture is mTLS. Requiring it on the public listener works (set HOPSKIP_TLS_CLIENT_CA_PATH and issue every worker a certificate), but a client CA is a property of a listener, not of a caller, so that also requires a certificate from every customer dialing the same address.

HOPSKIP_INTERNAL_BIND opens a second gRPC listener for exactly this. It serves the same API over the same live dispatcher, and it always requires client certificates:

Variable	Meaning
HOPSKIP_INTERNAL_BIND	Second host:port for in-cluster callers. Unset = one listener, as before
HOPSKIP_INTERNAL_TLS_CERT_PATH	PEM certificate for that listener, issued for its in-cluster names
HOPSKIP_INTERNAL_TLS_KEY_PATH	PEM private key for it
HOPSKIP_INTERNAL_TLS_CLIENT_CA_PATH	PEM CA that in-cluster clients’ certificates must be signed by

All three certificate variables are required whenever the bind is set: an internal listener that does not require client certificates is a second public listener, so the server refuses to start rather than open one. The reverse (certificate paths with no bind) is refused too, because the listener an operator configured would silently never exist.

Point workers and other in-cluster clients at that address with HOPSKIP_TLS_CLIENT_CERT_PATH/HOPSKIP_TLS_CLIENT_KEY_PATH set, and leave customers on the public one.

What the certificate establishes is the connection, not the principal. Every RPC on both listeners still carries a bearer token, is still authenticated and authorized the same way, and still lands in the audit trail under the token’s subject. What mTLS adds is that a stolen worker token is useless from outside the cluster, and that network reach alone no longer lets anything open a WorkerStream.

core/hopskip-server/tests/tls_end_to_end.rs is a working example of mTLS on one listener; core/hopskip-server/tests/internal_mtls_end_to_end.rs is the split, and asserts that the public listener stays reachable without a certificate while the internal one refuses.

mTLS between server replicas: the Raft peer transport

The gRPC listeners are not the only socket a replicated deployment opens. The Raft peer transport carries replication: every entry appended to the event log crosses it on the way to a quorum, along with votes, snapshots, and the membership autopilot’s join RPCs. It is a separate protocol on a separate port (HOPSKIP_RAFT_BIND_ADDR, conventionally 7001).

Variable	Meaning
HOPSKIP_RAFT_TLS_CERT_PATH	This node’s peer certificate. Needs both server auth and client auth: every node is both ends
HOPSKIP_RAFT_TLS_KEY_PATH	Its private key
HOPSKIP_RAFT_TLS_CLIENT_CA_PATH	The CA peers are verified against, in both directions
HOPSKIP_RAFT_TLS_PEER_NAME	The name a peer’s certificate must carry. Defaults to hopskip-raft-peer.internal

Unset, the transport is plain sockets, unchanged. Setting the certificate requires the key and the CA too: a peer transport authenticated in one direction is not mutual, and the server refuses to start rather than pretend otherwise.

Three properties worth knowing before turning it on:

• Every node or no node. There is no negotiated mode, deliberately: a “TLS if the other end offers it” transport is one an attacker can decline. A TLS peer and a plaintext peer cannot talk in either direction, so a rolling restart partitions the group along the boundary as it goes. It converges (the un-restarted majority holds quorum, then the restarted nodes re-form), but do it in a quiet window.
• One name for the whole cluster. Every node presents the same certificate and expects the same name, so a handshake proves membership of the deployment rather than which member. That is what Raft can act on: openraft authorizes by its own membership state, not by who opened the socket.
• Renewal needs a restart. The certificate is read once at startup.

core/hopskip-raft-logstore/tests/peer_mtls_cluster.rs is the evidence: a three-node cluster that forms, elects, commits and replicates entirely over mutual TLS, plus a node holding a certificate from another CA that can neither receive replication nor commit anything of its own.

The HTTP surfaces

The console API, SCIM and the ops endpoint terminate TLS with one pair of variables each, plus an optional client CA:

Surface	Certificate	Key	Client CA
Console API	HOPSKIP_CONSOLE_API_TLS_CERT_PATH	..._KEY_PATH	..._CLIENT_CA_PATH
SCIM	HOPSKIP_SCIM_TLS_CERT_PATH	..._KEY_PATH	..._CLIENT_CA_PATH
Ops	HOPSKIP_OPS_TLS_CERT_PATH	..._KEY_PATH	..._CLIENT_CA_PATH

Unset, each is plain HTTP, unchanged. Cert and key must be set together, and a client CA without them is refused.

The client CA is optional here, unlike on the internal gRPC listener, because who calls these is not the deployment’s choice: a kubelet probes the ops endpoint and cannot present a certificate, Prometheus scrapes it from another namespace, an ingress controller reaches SCIM. So the default is encryption plus each surface’s own credential (the console API’s bearer token, SCIM’s HOPSKIP_SCIM_TOKEN), and mTLS is available where the client side can be arranged. Setting a client CA on the ops surface breaks kubelet probes; move them to exec first.

Turning any of these on moves every caller with it, and the callers are outside the process:

• kubelet probes need scheme: HTTPS (they do not verify the certificate).
• A Prometheus scrape needs scheme: https.
• The console’s nginx needs HOPSKIP_CONSOLE_API_SCHEME=https and the CA to verify against.
• An ingress controller needs its backend protocol set to HTTPS for SCIM.

deploy/k8s/components/mtls/ does all four; the annotations differ per provider and are listed there.

core/hopskip-server/tests/http_surface_tls_end_to_end.rs is the proof for the ops endpoint: an https client trusting the CA gets a 200, and a plaintext client against the same port gets nothing.

deploy/k8s/components/mtls/ wires all of this with cert-manager.

Authentication

All gRPC and HTTP surfaces share one authenticator and therefore one credential model: either the HS256 shared-secret (HOPSKIP_STATIC_AUTH_SECRET) or OIDC (HOPSKIP_OIDC_ISSUER + HOPSKIP_OIDC_AUDIENCE):

• Client API (WorkflowClient): bearer token per request, namespace membership enforced per call.
• Worker stream (WorkerStream): the worker presents the same bearer token it uses for nested activity dispatch. Beyond authentication, two opt-in authorization controls: HOPSKIP_WORKER_AUTHZ grants task types to worker principals (subject=pattern[,pattern];..., * wildcards) and refuses a registration declaring an ungranted type (default-deny for uncovered subjects once rules exist); HOPSKIP_WORKER_AUTHZ_NAMESPACES=on only accepts a completion (complete/fail/checkpoint) from a principal whose namespace memberships include the workflow’s namespace. Denials emit authz_denied audit events.
• Result streaming (WorkflowStream): bearer token on the stream open.
• Deploy admin: bearer token with membership of the admin namespace.
• Console API: bearer token on every data-bearing route (/api/health stays open for probes). HOPSKIP_CONSOLE_API_AUTH=off disables this for loopback development; hop dev sets it for its own console. The access-administration routes additionally require the reserved hopskip-acl-admin namespace; see Users and permissions.
• SCIM provisioning (HOPSKIP_SCIM_BIND): a single shared bearer token (HOPSKIP_SCIM_TOKEN), which is what identity providers support. Unset, the endpoint does not bind at all.
• Shard admin (HOPSKIP_SHARD_ADMIN_BIND): shared-token auth via HOPSKIP_SHARD_ADMIN_TOKEN. Set, every request must present it (and a non-loopback bind is permitted); unset, the surface is unauthenticated and the server refuses to bind beyond loopback unless HOPSKIP_SHARD_ADMIN_ALLOW_REMOTE=true is set explicitly. Prefer the token, or loopback + SSH.

OIDC discovery

Set HOPSKIP_OIDC_ISSUER and HOPSKIP_OIDC_AUDIENCE and the server fetches <issuer>/.well-known/openid-configuration at startup to find the JWKS endpoint. Set HOPSKIP_OIDC_JWKS_URL explicitly to skip discovery and pin the endpoint instead.

The discovery document’s own issuer must match the configured one, per OIDC Discovery 1.0 §4.3; a mismatch is refused rather than followed, since otherwise a substituted document could point the server at an attacker’s signing keys. Keys are re-fetched every HOPSKIP_OIDC_JWKS_REFRESH_SECS (default 300s) to follow key rotation.

The server refuses to start without a configured secret. If neither HOPSKIP_STATIC_AUTH_SECRET nor OIDC is configured, hopskip-server exits with an error instead of silently using a publicly known development secret. HOPSKIP_DEV_ALLOW_INSECURE_SECRET=true restores the old behavior for development only. (hop dev is unaffected: it generates a real per-data-dir secret.)

End-to-end payload encryption

TLS protects payloads from outsiders; end-to-end encryption protects them from the party operating the control plane. Payloads are sealed into hopskip.payload.v1.Payload envelopes before they leave the client or worker process and opened only there; the server stores, routes, retries, and meters ciphertext, and holds no payload key. Workflow code observes plaintext exactly as without encryption, so replay, determinism checking, and every guest language work unchanged.

Configuration in brief; the full guide, including key rotation, the Vault/KMS providers, and the semi-managed topology (your workers, your console, our append log and orchestrator), is End-to-end encryption:

• Workers, clients, hop dev: HOPSKIP_PAYLOAD_E2EE_KEY_FILE (a versioned keyring file; rotation is adding a line) or HOPSKIP_PAYLOAD_E2EE_PROVIDER=vault with the standard VAULT_ADDR/ VAULT_TOKEN; AWS/GCP KMS providers are constructed in code.
• Control plane: HOPSKIP_REQUIRE_ENCRYPTED_PAYLOADS=<namespaces|*> refuses plaintext payloads into listed namespaces: a misconfiguration tripwire, not cryptographic verification. The guarantee lives in where the keys are.
• Key loss is data loss. The operator cannot recover what only your keys decrypt; erasing a namespace’s key is a namespace-wide crypto-shred with the log’s hash chain intact.

Users and permissions

Authorization has three layers, evaluated in this order. Any one of them can refuse.

1. The user directory (who exists, and whether their credentials are still accepted), including what your identity provider pushes over SCIM.
2. Namespace membership, the coarse gate: the credential must claim the namespace (or an IdP group must grant it), and an operator’s ceiling can narrow that.
3. Permission grants: which workflow types a member may start within a namespace.

The user directory

A directory record is (subject) -> display name, namespace ceiling, status, durable and replayed at startup. It applies to every authenticated surface, because it wraps the configured authenticator rather than living in any one service. Records arrive from two writers: an operator in the console, and your identity provider over SCIM (below).

The effective namespaces for an enrolled, accepted subject are:

(what the credential claims  ∪  what pushed IdP groups grant)  ∩  the operator's ceiling

• Refusing a credential outright. A disabled subject stops authenticating on the next request: no secret rotation, no waiting for exp. This is the revocation primitive a bearer-token deployment otherwise lacks. There are two independent switches: the operator’s status, and the identity provider’s active flag. Either one being off refuses the credential, and neither clears the other. An emergency lockout typed into the console survives the next sync, and a deactivation upstream survives an edit here. The console shows which party refused, because the two have different fixes.
• Narrowing namespaces. The operator’s ceiling is intersected last, so it can always take away, including from what a pushed group granted. A record with no ceiling (the default for an IdP-provisioned one) narrows nothing; to remove access, disable the subject rather than setting an empty ceiling.
• Widening namespaces is possible for the identity provider alone, through pushed group membership. The console cannot mint membership: an operator surface that could would be a way around your IdP. The first operator token therefore still comes from your provider (or StaticAuthenticator), including the reserved hopskip-acl-admin namespace the access surface requires.

A subject with no record is not narrowed at all, so the directory is adopted one person at a time rather than as a flag day. Removing a record is not the same as disabling one: an un-enrolled subject goes back to authenticating with what its credential claims.

Access control in a replicated cluster

Under raft or sharded, the access-control journals replicate with everything else, so every node has the records. But each node enforces from its own in-memory tables, so having the record is not the same as applying it. Two things follow, and both are operational, not theoretical:

• Propagation is not instant. Each node re-reads the journals every HOPSKIP_ACCESS_SYNC_MS (default 1000ms) and applies whatever another node wrote. That interval is the worst-case delay before a revocation made on one node is enforced on the others: the one number to know here. Setting it to 0 disables the sync entirely, which leaves a node enforcing stale access control until it restarts; don’t, unless you have a specific reason.
• Writes must go to the Raft leader. This is a current limitation of the replicated backend (append has no forward-to-leader yet), and it applies to every admin write, not just these (grants, deployments, schedules, DLQ replays). An admin write sent to a follower fails hard rather than being forwarded. Point your SCIM connector and console at the leader, or at a load balancer that tracks it. (hop cluster is the exception: membership writes go through the multigroup host’s forward-to-leader path, so they may be sent to any node.)

Under memory or disk neither applies: one process is the only writer, so its own tables are authoritative and no sync loop runs.

The journals record changes, not pushes: re-provisioning a user whose details have not moved, or re-applying a Terraform grant that already exists, writes nothing. So an identity provider on an aggressive re-sync schedule costs storage and replication traffic proportional to what changed, not to how often it syncs. Deletes and revocations are always written.

Cluster membership

Under sharded, the cluster knows what it is made of. A node registry inside the placement Raft group records which nodes exist, where to reach them, and what you intend for each (joining, active, draining, left). A separate failure detector reports what is answering right now (alive, suspect, dead, unknown) and is deliberately never authoritative: nothing is ever evicted because a detector said so.

hop cluster status --node hostA:7001     # or --json
hop cluster join   --seed hostA:7001 --node-id 4 --raft-addr hostD:7001
hop cluster drain  --node hostA:7001 --target 3
hop cluster remove --node hostA:7001 --target 3

Every command talks to any node’s peer address; membership writes forward to the placement leader themselves, so there is no leader to find first.

Three things worth knowing before you need them:

• Nodes join by discovery, not by config. Set HOPSKIP_CLUSTER_SEEDS to any live member (host:port) or a DNS name (dns:name:port, on Kubernetes, a headless Service resolves to one address per ready pod). Steady-state membership then comes from the registry, so HOPSKIP_RAFT_MEMBERS is bootstrap-time input only and a cluster whose config has drifted still converges.
• HOPSKIP_RAFT_BOOTSTRAP is idempotent. A node that boots with it against an already-formed cluster does nothing and says so. It is safe to leave set in a manifest.
• Draining is intent, not completion. hop cluster drain returns immediately. The node stops taking new replicas and hands off its group leaderships, but its existing replicas are not moved off automatically; each move costs O(total history) today. hop cluster status reports how many groups it still hosts. remove refuses until that reaches zero.

Auto-scaling the raft backend

The raft backend (a single replicated group, the shape the Kubernetes topology deploys) has no node registry and needs none: openraft’s own membership is the record. With HOPSKIP_CLUSTER_AUTO_SCALE=true it manages itself, and scaling stops being a membership procedure at all:

• Scale up = start a node with a unique id and the shared HOPSKIP_CLUSTER_SEEDS. It joins as a non-voting learner and the leader promotes it once its log has caught up, so an empty node never dilutes quorum while it copies history.
• Scale down = stop a node. After HOPSKIP_CLUSTER_EVICT_AFTER_MS (default five minutes) of silence the leader evicts it, but never below HOPSKIP_CLUSTER_MIN_VOTERS (default 3). A restart inside the window is not a departure; a node evicted during a longer outage rejoins by itself when it returns, even if its own disk still says it is a member.

HOPSKIP_RAFT_MEMBERS becomes optional: the one HOPSKIP_RAFT_BOOTSTRAP=true node forms a single-voter cluster of itself (first asking its seeds whether one already exists, so a bootstrap node that lost its disk rejoins the survivors instead of founding a rival), and everyone else joins through seeds. On Kubernetes this is what makes kubectl scale statefulset/hopskip-server the entire resize procedure.

Choosing a failure detector

HOPSKIP_CLUSTER_FAILURE_DETECTOR is heartbeat (default), gossip, or off.

The default probes every peer directly over the existing peer connections. Detection is immediate: no dissemination delay, no dependency on a third node’s opinion, and at a few dozen nodes the traffic is negligible. gossip runs SWIM instead: worth switching to when the cluster is large enough that probing every peer matters (hundreds of nodes), when it spans a WAN, or for SWIM’s auto-rejoin, where a node wrongly declared down by a transient partition returns on its own. Both report through the same hop cluster status, and the two coexist during a rolling change.

Connecting an identity provider (Okta, Entra ID, …)

The server accepts SCIM 2.0 inbound provisioning (RFC 7643/7644), which is what Okta, Microsoft Entra ID, Google Workspace, JumpCloud and OneLogin all use to push users into an application. Hopskip never calls your provider: there is no API token of theirs to store and no polling loop, and a deactivation lands here when it happens rather than at the next poll.

Variable	Meaning
HOPSKIP_SCIM_BIND	Where the SCIM endpoint listens (host:port). Its own listener, separate from the console API: the provisioning token is directory-admin-grade, so the surface it opens should be reachable by your IdP alone
HOPSKIP_SCIM_TOKEN	The bearer token your provider presents. Required: with no token the endpoint does not bind at all, since an unauthenticated write surface onto the user directory is not a development convenience
HOPSKIP_SCIM_GROUP_PREFIX	Optional. Only pushed groups carrying this prefix map to namespaces, and the prefix is stripped (hopskip-checkout → checkout). Unset, every pushed group maps to the namespace of its own name

In your provider’s application settings, give https://your-host/scim/v2 as the SCIM connector base URL and the token as the bearer credential (“HTTP Header” auth in Okta). In a replicated cluster, point that URL at the Raft leader; see Access control in a replicated cluster. Supported operations: create, read, replace, patch and delete for Users and Groups, with eq filtering and pagination. Bulk, sort, ETag and POST /.search are declared unsupported in ServiceProviderConfig rather than half-implemented.

What maps to what:

SCIM	Hopskip
userName	the record’s subject at creation, and an alias forever after
externalId	the provider’s own id, kept as an alias
displayName / name	display name (never consulted for authorization)
emails	an alias each: every address, not just the primary
active	the identity provider’s switch, not the operator’s, so a console lockout survives a sync
Group.displayName + members	a namespace its members receive

A note on subjects. Several providers’ OIDC sub is opaque (Okta’s is 00u…), and that string is what permission grants are keyed by and what the audit trail shows. Records resolve by externalId and userName as well as by subject, so provisioning works either way; setting HOPSKIP_OIDC_SUBJECT_CLAIM=email (or preferred_username) additionally makes principals readable everywhere. It falls back to sub when the claim is absent.

Permission grants

Worth restating, because the transition surprises people: a namespace with no grants at all enforces nothing; membership alone decides. Recording the first grant in a namespace flips it to fail-closed for everyone in that namespace; anybody without a matching grant is denied from that moment. Revoking the last one returns it to fail-open.

The console access surface

The console (frontend/) serves both from hopskip-server’s console API:

Route	Purpose
GET /api/access/whoami	The caller’s own principal, and whether it may administer access. The one access route with no admin gate
GET /api/access/namespaces	Every namespace either surface knows about, with user/grant counts and whether it enforces
GET|POST /api/users, DELETE /api/users/{subject}	The user directory
GET|POST /api/acl/grants, DELETE /api/acl/grants?namespace=&subject=&workflow_type_pattern=	Permission grants, the same durable table AclAdmin gRPC and the Terraform provider write
GET /api/acl/effective?namespace=&subject=&workflow_type=	Whether that subject would be allowed to start that type, and which of the three layers decided

Every route here (reads included, since the list of who can do what is itself sensitive) requires membership of the reserved hopskip-acl-admin namespace, waived only when HOPSKIP_CONSOLE_API_AUTH=off. The console UI is /users and /permissions; paste an operator token into the box on the Users page and it is sent as Authorization: Bearer on every console API call (kept in that browser’s local storage only).

hop dev runs its console with auth off on loopback, so both pages work there with no token at all.

Safe deploys

Two independent mechanisms, both operator-facing, cover different parts of “did this deploy break anything”; see the CLI reference for the commands.

The compatibility gate reasons about the change itself: it diffs the new build’s control-flow graph against the currently-deployed one and refuses changes that in-flight workflows cannot replay through. Set HOPSKIP_SUSPENSION_TRACKING=true and it also asks this server, at deploy time, how many workflows are currently parked at each await the new build removes, so a removed await with nothing waiting at it deploys with a warning instead of a --force.

Turn that on only once every worker serving those namespaces is new enough to report its suspension points. The index is built from worker reports, and a partially-reported index would answer “zero suspended here” for sites nobody reported. It fails closed rather than fails silently: the server tracks how many live instances it cannot account for, and a deploy against incomplete coverage is refused with that number, not softened. The index is per-process and not journalled, so immediately after a server restart a deploy will report incomplete coverage until workers reach their next suspend boundaries.

Upgrade drains cover the window around the pointer flip, where the fleet is partway between two builds:

hop deploy <hash> --namespace orders --addr $HOPSKIP_SERVER_ADDR --drain

New starts in that namespace are accepted and held in Core for the duration, then dispatched onto the build the deploy installed. Clients see no error. For upgrades that are not a single hop deploy call (rolling a worker fleet, a schema change), hop drain begin|end|status drives the same mechanism directly. A drain gates new starts only: in-flight workflows keep running and completing throughout, including dispatching their next activity.

--drain also does something the coverage paragraph above makes necessary: having held the arrivals, it waits (up to --drain-settle-ms, default 10s) for every already-live instance to report where it is parked, and only then runs the gate. Without that wait, a busy namespace almost always has somebody between boundaries and the consult answers “cannot account for N live instances”: safe, and never useful. A drain is what makes the wait terminate: with new starts held, the in-flight set can only shrink. If the timeout expires (a workflow parked on a long timer or a human approval will not settle), the gate asks against whatever coverage exists and an incomplete answer still blocks; hop suspensions list names who is holding it up.

Both require an operator token with membership in the reserved hopskip-deploy-admin namespace.

Health and metrics

Set HOPSKIP_OPS_BIND (e.g. 127.0.0.1:9464) to serve, on its own address:

• GET /healthz: liveness. 200 while the process serves at all.
• GET /readyz: readiness. 200 only when the dispatcher can be reached; 503 under a wedged/starved dispatcher.
• GET /metrics: Prometheus text format: hopskip_pending_tasks, hopskip_capacity_blocked_tasks, hopskip_worker_connections, hopskip_active_leases, the same numbers the KEDA external scaler serves over gRPC, for deployments scraping instead of scaling.

hop dev binds this at 127.0.0.1:9464 by default (--ops-bind).

The main gRPC bind also serves the standard grpc.health.v1.Health service (compatible with grpc_health_probe and Kubernetes gRPC probes).

Unset HOPSKIP_OPS_BIND means the surface is off: nothing binds a port an operator didn’t ask for.

Ready-made Prometheus alert rules and a Grafana dashboard for these gauges live in deploy/.

Logs and traces

Variable	Meaning
HOPSKIP_LOG_FORMAT	text (default, human-readable) or json (one object per line, for a log shipper)
RUST_LOG	Standard filter directives; defaults to info
HOPSKIP_OTLP_ENDPOINT	OTLP/gRPC collector for span export, e.g. http://collector:4317. Unset means no export
HOPSKIP_OTLP_SERVICE_NAME	Service name in the trace backend (default hopskip-server)

An unrecognized HOPSKIP_LOG_FORMAT is a startup error rather than a silent fallback: asking for JSON and getting prose would otherwise surface only as an empty log pipeline.

Spans are flushed on clean shutdown, so the last seconds before a stop are not lost.

Data lineage

Separate from tracing, and answering a different question: what data a run produced and where it came from. The server can export OpenLineage run events to Marquez, DataHub, or any backend implementing the spec.

Variable	Meaning
HOPSKIP_OPENLINEAGE_ENDPOINT	Base URL of an OpenLineage receiver, e.g. http://marquez:5000. Unset means no export
HOPSKIP_OPENLINEAGE_API_KEY	Bearer token for that endpoint. Unset means no Authorization header
HOPSKIP_OPENLINEAGE_FILE	Append events to this file as newline-delimited JSON; ignored when an endpoint is set
HOPSKIP_OPENLINEAGE_TICK_MS	How often the emitter reads new records (default 5000)
HOPSKIP_OPENLINEAGE_BATCH_SIZE	Records read per partition per tick (default 500)

Off unless one of the two destinations is set. Unlike the visibility projection, this sends data off the machine, so the switch is the configuration rather than a default with an opt-out.

The emitter is a read-side consumer of the event log: it cannot fail a workflow, and a tick holds the dispatcher and log locks only for the read. Neither the decode nor the network round trip happens under them, so an unreachable backend falls behind rather than stalling appends. Delivery is at-least-once with a derived (stable) run id, so a restart re-sends byte-identical events that a conformant backend absorbs.

Requests are bounded (5s connect, 15s total) and consecutive failed ticks back off exponentially to about five minutes, resetting on any successful delivery. Nothing is consumed until it lands, so a backend outage is a backlog rather than a gap: the whole pending set is delivered when it recovers. A openlineage delivery failing; backing off warning names the consecutive-failure count; openlineage delivery recovered marks the other edge.

See the lineage guide for declaring the datasets a workflow reads and writes.

Usage metering

For running Hopskip as a service. Off by default because self-hosted Hopskip is priced per production cluster and server vCPU (pricing). Action counts in the current ledger are operational telemetry; Cloud invoices provider-specific CPU, RAM, storage, and networking instead.

Variable	Meaning
HOPSKIP_METERING	on to meter this node’s log. Unset or anything else means off
HOPSKIP_METERING_TICK_MS	How often new records are folded into usage buckets (default 5000)
HOPSKIP_METERING_BATCH_SIZE	Records read per partition per tick (default 500)
HOPSKIP_METERING_LEDGER_PATH	Directory holding the usage ledger’s hash-chained files (default .hop/usage)
HOPSKIP_METERING_PLAN	Plan for namespaces with no explicit one: developer, standard (default), business, dedicated
HOPSKIP_METERING_EXPORT_PATH	Append flushed usage records to this file as newline-delimited JSON. Unset means no export

Turned on, a background task classifies log records into billable actions per tenant per hour and, once an hour closes, appends one usage record per (namespace, meter, hour) to the usage ledger: an append-only, hash-chained file per namespace, living beside the event log rather than inside it. Metering reads the log and never writes to it. The console serves the result at GET /api/usage and GET /api/usage/{namespace}, each carrying a preview invoice and the ledger’s Merkle root.

Keeping the two apart is on purpose. Billing records outlive workflow history by years, so they must not inherit the log’s retention, restore, or shard-rebalancing lifecycle; and the audit below only means something because the ledger and the log are independent artifacts that can be checked against each other.

The ledger is the record of what was metered; the export is a copy of it. An export that fails is retried on the next flush and every record carries a stable idempotency key, so a failing destination delays an invoice rather than losing usage. A ledger append that fails is more serious: it logs at error level, and the recovery is hop usage --recount, since the log still holds everything the ledger was derived from.

Anyone holding both artifacts can check the result without contacting the server:

hop usage --ledger .hop/usage --log-root .hop/log \
           --namespace acme --verify --recount

--verify checks the ledger’s hash chain (no line altered, removed, or reordered) and an inclusion proof for every record against the recomputed Merkle root. --recount re-derives the action count from the history log using the same classification the meter itself uses: the check the proofs cannot do, since proofs show the ledger was not changed while only a re-derivation shows it was right when written.

The chain is checked on every read, not just under --verify, so a tampered ledger cannot produce a bill at all.

Three meters have ingestion points but no in-process producer yet (storage, egress, and per-tenant compute attribution); they record zero, and the metering snapshot reports how many samples it has seen so “nothing is reporting” is distinguishable from “nothing happened”.

Pushing usage to Stripe

Metering counts; it does not charge anybody. hopskip-billing is the separate scheduled process that carries the ledger to Stripe. It is a Cloud component: a self-hosted deployment is priced per cluster and per vCPU (see pricing) and never needs it.

It runs apart from the server on purpose: hopskip-server executes untrusted tenant Wasm, and nothing in that address space should hold a live payment credential. The dependency graph enforces it: the server does not depend on hopskip-billing, so no configuration can put a Stripe key there.

Variable	Meaning
HOPSKIP_BILLING_DATABASE_URL	Postgres holding the customer map and the push outbox. Required
HOPSKIP_BILLING_STRIPE_KEY	Stripe secret key. Required unless running a dry run
HOPSKIP_METERING_LEDGER_PATH	Usage ledger to read (default .hop/usage)
HOPSKIP_BILLING_INTERVAL_SECS	Seconds between cycles (default 900)
HOPSKIP_BILLING_DRY_RUN	Resolve and classify, send and record nothing
HOPSKIP_BILLING_ONCE	Run one cycle and exit, for a cron-style deployment
HOPSKIP_BILLING_GATEWAY_BIND	Address for the console’s onboarding gateway, e.g. 0.0.0.0:8099. Set with the token or not at all
HOPSKIP_BILLING_GATEWAY_TOKEN	Shared bearer token the console presents. At least 32 characters; openssl rand -hex 32

The onboarding gateway

Setting the two gateway variables gives this process a second job: a small HTTP surface the console calls during signup to create a Stripe customer and open a hosted checkout for a card. It is the same process because it is the same credential - and it is a separate process from the console for exactly the reason above.

Bind it to a cluster-internal address. The token authenticates the console to this listener; somebody holding it can create customers and open checkouts, and cannot charge anybody, refund anybody, read a card, or enumerate the Stripe account. That asymmetry is the point of the split.

Leave both unset and nothing listens, which is the correct posture for a deployment whose console does not collect payment details.

Migrate before starting it

Both of this process’s tables come from db/billing, deployed with sqitch, and it will refuse to start against a database that has not been migrated:

sqitch --chdir db/billing deploy --target "$HOPSKIP_BILLING_DATABASE_URL" --verify

See db/README.md for adding a change and for adopting a database that already has the tables.

hopskip_billing_customer maps a namespace to a Stripe customer. The gateway above writes it when an organization signs up; the push cycle only ever reads it. Writing a row by hand is still the way to map a namespace that was provisioned some other way:

INSERT INTO hopskip_billing_customer (namespace, stripe_customer_id)
VALUES ('acme', 'cus_NciAYcXfLnqBoz');

Set billable = false for an account that is not charged (internal, comped, on trial). That is different from a namespace with no row at all, which is real usage nobody will be billed for, and which the cycle reports by name so it reaches a human.

Start with a dry run against the real ledger; it needs no Stripe key and writes nothing:

HOPSKIP_BILLING_DRY_RUN=1 HOPSKIP_BILLING_ONCE=1 HOPSKIP_BILLING_DATABASE_URL=postgres://... hopskip-billing

Two things that will bite

Usage older than 35 days cannot be billed. Stripe refuses meter events backdated past that, and the ledger keeps records forever. Records that age out are marked expired in the push outbox rather than retried into a permanent failure, so this query is a revenue alarm, not a debug aid:

SELECT namespace, meter, bucket_start_ms, quantity
  FROM hopskip_billing_push_outbox WHERE status = 'expired';

Anything it returns needs a manual invoice item. It also sets the deadline on every other failure here: a deferred record is harmless, but only for five weeks.

Do not rely on Stripe’s identifier to prevent double-billing. It deduplicates within a rolling ~24 hours, aimed at immediate accidental retries. The durable guarantee is the push outbox, keyed by (namespace, meter, hour): a record it has marked accepted is never sent again, however many times the ledger is re-read, restored, or replayed. Stripe’s key covers the one gap the outbox structurally cannot, a crash between Stripe accepting an event and the row being written, and a 409 on the retry is treated as the success it is.

Cycles log at warn when anything needs attention (expired, rejected, or unmapped records) and at info otherwise, so the alert rule is a level filter rather than a message regex. A cycle that fails is retried on the next interval; a rejected credential exits non-zero instead, because every subsequent cycle would fail identically while looking like progress.

Audit log

The server emits a control-plane audit trail, structured events answering “who did (or was refused) what, on which surface”, separate from workflow history (which is already principal-attributed and Merkle-verifiable; see hop audit). What is recorded:

• Authentication denials on every surface (gRPC client/worker/stream/ schedule/DLQ/deploy-admin, console API, SCIM), with the surface and coarse reason. There is no principal to attribute; that is the point.
• Authorization denials: an authenticated principal refused a namespace or the reserved hopskip-deploy-admin namespace, with the RPC name.
• Admin actions: canary start/abort, deployment-breaker config, quarantine replays, SetCurrentDeployment, DLQ replays, schedule create/update/patch/delete, human-task approval decisions and M-of-N gate signatures (per signature, as they arrive), permission grants and revocations (from either the AclAdmin RPC or the console, each tagged with its surface), user directory enrollments/edits/removals (attributed to console_api or scim by which writer made them, including group pushes), and every sharded-admin command (the peer address is recorded in place of a principal, plus whether the shared token check applied).

Audit events are ordinary tracing events on the hopskip_audit target, so they appear in the normal log stream in whatever HOPSKIP_LOG_FORMAT you run. hopskip_audit=info is pinned into the level filter, so RUST_LOG=error does not silently drop the security trail (an explicit RUST_LOG=...,hopskip_audit=off still can).

Variable	Meaning
HOPSKIP_AUDIT_LOG_PATH	Optional. Append-only file additionally receiving every audit event (and only those) as hash-chained JSON lines, a dedicated trail for a SIEM/collector to tail. Unopenable path = startup error

Each JSON line carries kind (authn_denied | authz_denied | admin_action), surface, action, principal, namespace, reason/detail, ts_ms, level, and prev, the SHA-256 of the previous line exactly as written (the first line chains from all zeros; restarts extend the existing chain). That makes the file tamper-evident:

hopskip-server verify-audit-log /var/log/hopskip/audit.jsonl

verifies every line’s prev and names the first altered, removed, or reordered line. What chaining cannot detect is truncation from the end. Ship the file (or its latest line hash) to a collector the server cannot rewrite for that.

What is not here: per-request success events on the data plane. Every accepted client mutation is already durably recorded with its principal in the event history, which is the stronger record.

One authorization note that follows from the audit posture: the sharded-admin surface accepts a shared token via HOPSKIP_SHARD_ADMIN_TOKEN: set, it authenticates every request (and permits a non-loopback bind); unset, the surface stays loopback-only unless explicitly overridden.