End-to-end payload encryption

TLS protects payloads from outsiders. End-to-end encryption protects them from the party operating the control plane, including a hosted Hopskip’s own team and anyone who compromises the control plane’s processes, storage, or backups.

With it enabled, workflow inputs, activity inputs and results, signal/query/update payloads, channel messages, checkpoint values, and their offloaded blobs are sealed before they leave the client or worker process and opened only there. The server stores, routes, retries, and meters ciphertext; no code path in it constructs a decryption key. Workflow code is below the boundary and observes plaintext exactly as it would without encryption, so replay, determinism checking, and every guest language work unchanged, in every SDK.

The envelope

A sealed payload is a serialized hopskip.payload.v1.Payload (the canonical headers + data bundled payload shape) whose headers carry the encryption metadata and whose data is AES-256-GCM ciphertext of another serialized Payload holding the original headers and bytes:

Payload {
  headers: {
    "content-type":            "application/vnd.hopskip.encrypted",
    "hopskip-e2ee-version":       "1",
    "hopskip-e2ee-scope":         "hopskip-namespace:orders",
    "hopskip-e2ee-kek-id":        which key wrapped this payload's key,
    ...nonces, the wrapped data-encryption key...
  },
  data: <ciphertext of Payload { original headers, original bytes }>,
}

Each payload gets a fresh data-encryption key (DEK), wrapped under the scope’s key-encryption key (KEK). The original headers, including the logical type and schema identity, travel inside the ciphertext: a control-plane reader learns “an encrypted payload of N bytes for namespace S”, never “an acme.Order”. Every encryption header is bound into the AEAD as associated data, so a tampered or re-scoped envelope fails to decrypt rather than decrypting wrongly. Two seals of the same payload never produce the same bytes, so the control plane cannot even learn that two payloads are equal.

Keyed replay commitments

Hopskip’s determinism checking records a digest of every command a workflow emits (which activity, with which payload) in durable history. Unkeyed, those digests would let whoever stores them confirm guesses about low-entropy payloads by brute force: hash {"approve": true} and compare. Under end-to-end encryption they are keyed instead. Each run’s start seal mints a random commitment key that travels wrapped inside the input envelope, and workers hash the run’s commands with HMAC-SHA-256 under it. The control plane stores digests it cannot recompute; replay matching is untouched, because both sides of every comparison run in workers holding the key. The wrapped key is bound into the envelope’s authentication, so it cannot be stripped in transit to downgrade a run to guessable hashing.

Keys, rotation, and managed key infrastructure

Keys are scoped per namespace and supplied by a pluggable provider. Pick by how your organization manages keys:

Keyring file (self-managed keys)

# /etc/hopskip/payload-keyring - one version per line, highest seals
v1:6ea27a2d…3d51
v2:8c1b9adb…f1c0

export HOPSKIP_PAYLOAD_E2EE_KEY_FILE=/etc/hopskip/payload-keyring
# dev shorthand, a one-line keyring:
export HOPSKIP_PAYLOAD_E2EE_KEY=<64 hex chars>

Per-namespace keys derive from the master keys (HMAC-SHA256), so every process holding the same file interoperates with no per-namespace distribution step. Rotation is adding a line: generate a new 32-byte key, append it as v<N+1>, roll the file out. New payloads seal under the newest version; everything sealed under any listed version still opens (each envelope records which key wrapped it). Deleting a line is a deliberate crypto-shred of everything sealed under that version.

HashiCorp Vault (Transit engine)

export HOPSKIP_PAYLOAD_E2EE_PROVIDER=vault
export VAULT_ADDR=https://vault.internal:8200
export VAULT_TOKEN_FILE=/var/run/secrets/vault-token
# optional: HOPSKIP_PAYLOAD_E2EE_VAULT_MOUNT (default "transit")
#           HOPSKIP_PAYLOAD_E2EE_VAULT_KEY_PREFIX (default "hopskip-e2ee-")

One Transit key per namespace, created on first use. Key material never enters the Hopskip process: wrap and unwrap are Vault operations, governed by your Vault policies. Rotation is Vault’s own (vault write -f transit/keys/hopskip-e2ee-<namespace>/rotate) and takes effect on the next payload with no file to distribute; old payloads keep decrypting because Vault tokens carry their key version. Deleting the Transit key is a real, Vault-audited crypto-shred of the namespace.

Binaries need the vault feature of hopskip-payload-crypto compiled in; a worker asked for Vault without it refuses to start rather than silently running unencrypted.

AWS KMS / GCP Cloud KMS

The aws and gcp features provide KMS-backed providers (the same ones @sensitive field encryption uses): key material stays inside the KMS, credentials come from each SDK’s standard chain (IRSA, instance roles, Application Default Credentials), and rotation is the KMS’s own. These are constructed in code (PayloadCipher::new) rather than by environment variable, because cloud credential configuration does not reduce to one variable.

Whichever provider: key loss is data loss. The operator cannot recover what only your keys decrypt. That is the feature, and it is also the warning.

Configuration, end to end

Where	Setting
Workers (`hopskip-wasm-worker`, `hopskip-v8-worker`)	`HOPSKIP_PAYLOAD_E2EE_KEY_FILE` (or provider env above); dispatches open before workflow code sees them, results seal before they leave the process
Clients	The same env for `hop run`; `HopskipClient::with_payload_cipher` in the Rust SDK
Schedules	`ScheduleClient::with_payload_cipher`: a schedule’s action input is sealed in the stored definition, so every firing starts its workflow encrypted
`hop dev`	Honors the same env, so an encrypted deployment rehearses on a laptop
Control plane	`HOPSKIP_REQUIRE_ENCRYPTED_PAYLOADS=<ns1,ns2>` (or `*`): refuses plaintext payloads into listed namespaces, on the client RPCs and on worker completions

HOPSKIP_REQUIRE_ENCRYPTED_PAYLOADS is a tripwire, not the guarantee. The server cannot cryptographically verify a seal (it has no keys), but it can loudly refuse a client that meant to encrypt and silently was not configured to, which is the misconfiguration that otherwise surfaces months later as plaintext in the durable log. The guarantee itself lives in where the keys are.

A worker that cannot decrypt a dispatch fails the attempt as retryable (another worker holding the key can serve it); a worker that cannot seal a result fails the attempt rather than completing in plaintext.

The semi-managed topology

End-to-end encryption forces the question “where may plaintext exist?” The deployment answers it. In the semi-managed configuration the hosted control plane owns exactly two things: the append-log cluster and the orchestrator (dispatch, retries, scheduling, metering, visibility over indexed attributes). Everything that touches plaintext runs on your infrastructure:

Workers dial out to the control plane over gRPC/TLS with a bearer credential: outbound 443, no inbound ports, no VPN. Run the standard worker images on your compute with your key configuration. Worker authorization (HOPSKIP_WORKER_AUTHZ, HOPSKIP_WORKER_AUTHZ_NAMESPACES) scopes what each worker principal may serve.
The console is its own deployable (a static frontend behind an oauth2-proxy, the same deploy/k8s/base/console/ unit the hosted stack uses) and should be self-hosted beside your workers, pointed at the hosted control plane’s API with your identity provider. Under E2EE the API can only serve it ciphertext and markers. Self-hosting makes that a topology fact rather than an API promise, and it means the JavaScript your operators run is served by you, not by the party the encryption defends against. Encrypted payloads render as a marker (scope, key id, size); decrypt locally with your keys when you need the contents.
Resume state: point the workers’ shared resume-bundle store at storage you own. A suspended workflow’s memory snapshot contains plaintext state.

What stays visible, by design

Namespaces, workflow type names, instance names, timing, payload sizes, and failure diagnostics remain visible to the control plane, because it cannot schedule, retry, or meter without them. So does anything a workflow explicitly emits as a @searchable/@indexed attribute: emitting one is publishing it to the server’s index. That is the search/privacy lever, per attribute, in your hands. Index what you can afford the operator to see; encrypt the rest and search on what you indexed.

Erasing a namespace’s key (removing its keyring line, deleting its Transit key) is a namespace-wide crypto-shred: the append-only log and its hash chain stay intact and auditable while every payload under that key becomes permanently unreadable.