Provenance & causality

@searchable state (Visibility & search) answers “what does this variable hold right now.” Provenance answers a different question: what wrote it, and what wrote it before that? A workflow’s status field reads "failed"; provenance is what lets you trace back through "charged" → "reserved" → "pending" and see that "charged" came specifically from pending call #7 of charge_card, not merely that charge_card ran at some point.

This is the raw material for a causality graph, not the graph itself. Traced<T> records writes and their upstream attribution; the console’s Causality tab (and GET /api/workflows/{ns}/{wf}/provenance) is what renders them as one.

Rust only, today. Traced<T> exists in the Rust SDK (hopskip_sdk::provenance). The wire envelope and the projection that reads it are language-agnostic, so another SDK could speak it, but only Rust workflows can produce one right now.

Tracing a value back to its source

use hopskip_sdk::{ProvenanceConfig, Traced};
use hopskip_sdk::activity;

let mut status = Traced::new("status", ProvenanceConfig::default(), "init", "pending");

status.set("reserve_inventory", "reserved");

Each write is tagged with what produced it. A plain string is a caller-typed label, useful for a purely computed value with no upstream call to point at. For a write that came from an activity result, tag it with the real invocation instead:

let mut charge = activity::invoke("charge_card", order.total.to_le_bytes().to_vec());
let result = (&mut charge).await;
let source = charge.source(); // the exact pending call that produced `result`

status.set(source, "charged");

(&mut charge).await rather than charge.await keeps the future around afterward so .source() can still be read from it. source carries the activity’s name and the host-assigned id of that specific invocation: not just “a charge_card call happened somewhere,” but which one, distinguishable from every other time this workflow called charge_card.

Sampling is a counter, never a coin flip

A workflow must replay identically from its recorded history, so which writes get intercepted can’t be decided by randomness. That would make the interception decision itself nondeterministic across replay, independently of anything the workflow’s own logic does. ProvenanceConfig instead picks every Nth write via a plain per-instance counter:

Config	Rate	Use for
ProvenanceConfig::default()	1 in 10	The rate you get without thinking about it
ProvenanceConfig::SPARSE	1 in 100	Write volume high enough that 1-in-10 is no longer obviously negligible
ProvenanceConfig::ALWAYS_ON	every write	Establishing an upper bound, or a value you need the complete history of

The first write is always recorded, matching Searchable/Indexed’s “construction always emits” contract. A skipped write costs a counter check, well under 10ns; an emitted one costs the same emit_searchable host round trip @searchable already pays, measured at roughly 0.6-0.8µs. At the write volumes a workflow typically produces (hundreds to a few thousand), even ALWAYS_ON costs low tens of microseconds to about a millisecond total, noise next to a single activity round trip.

Like @searchable and @indexed, this is opt-in per call site: a workflow that never constructs a Traced<T> pays nothing, not even a counter check.

Reading it back

curl $HOPSKIP_SERVER_ADDR/api/workflows/orders/order-42/provenance

{
  "namespace": "orders",
  "workflow": "order-42",
  "writes": [
    {
      "attr_key": "status",
      "source_kind": "tag",
      "source_name": "init",
      "source_pending_id": null,
      "write": 1,
      "sampled_write": 1,
      "value": "pending",
      "sequence": 4
    },
    {
      "attr_key": "status",
      "source_kind": "activity",
      "source_name": "charge_card",
      "source_pending_id": 7,
      "write": 2,
      "sampled_write": 2,
      "value": "charged",
      "sequence": 19
    }
  ]
}

Every write this instance recorded, in emission order, across every Traced<T> key. write is this write’s ordinal among all writes the field saw, sampled or not; sampled_write is its ordinal among only the ones that made it to the host, and the gap between them is how far apart the samples were. sequence is this row’s position in the log, which is what lets you interleave writes to different keys into one timeline.

Namespace scoping, the 403-not-empty-on-a-namespace-you-can’t-read rule, and cluster scatter-gather with a freshness block all work exactly as they do for /state; see Visibility & search. One difference worth knowing: a workflow can have @provenance writes with no flat @searchable attribute at all, so this route resolves its own partition rather than reusing /state’s.

The Causality tab

The console’s workflow detail page renders these writes as a graph: one node per write, edges from each write to the one before it on the same key (“mutated to”), and a second edge in from whatever produced it, a tag node or the specific activity invocation node. Reading top to bottom on a key’s trace is the answer to “what produced this value, and what produced the value before that.”

How it works

Traced<T> is built on the same emit_searchable host import every other annotation in this SDK uses, with no new host call, the same way @indexed extended @searchable without one. A write is rendered as a small header line (key, source, source kind, pending id, the two write counters) followed by the value’s normal scalar encoding, and handed to mark_searchable like any other attribute:

HOPSKIP-PROVENANCE-V1 key=status source=charge_card source_kind=activity pending_id=7 write=2 sampled_write=2
charged

The projector tries this envelope’s magic prefix before falling back to a plain @searchable scalar, so a Traced<T> and an ordinary Searchable can share a key’s storage without colliding. Retention follows the same rule as every other projected row: when a completed workflow passes its namespace’s retention period, its provenance rows are purged along with the rest of its visibility state.

What this is not

It does not track edges between variables (which variable’s value flowed from which other variable’s write), and it does not build a graph on the host side. It records one variable’s own write history and each write’s upstream call. The graph you see in the Causality tab is the frontend assembling that from a single key’s writes, not a host-side dependency tracker. If you need to relate values across keys, that relationship has to be visible in what you tag each write with.