HOPSKIP
Start a run

Concepts

Event store & audit

All workflow state in Hopskip is an append-only event log. This is the Log-Structured Event Store (LSES), and it replaces the Cassandra-or-Postgres-plus-Elasticsearch pair typical of older orchestrators.

Every history is verifiable

Each event carries:

  • the SHA-256 of its predecessor,
  • the OIDC identity of whatever triggered it, and
  • a timestamp (HLC),

forming a Merkle Mountain Range per workflow. You can verify that a run’s history has not been tampered with, and produce a compact cryptographic root:

hop audit default:wf_7b3a91
# fetches the run's history + MMR root over the wire and writes a
# compact inclusion proof file for any record in it

Auditability is a property of the storage format. There is no export pipeline to build and trust.

History stays small

The log records decisions and offsets, not payloads and message floods:

  • Bulk data flows through content-addressable storage (CAS) and is referenced by a DurableRef, not inlined into history.
  • High-frequency messages flow through channels whose partitions hash independently of the workflow’s history partition, so a mailbox flood lands on other shards while history records only offset checkpoints.
  • Memory snapshots bound replay cost.

The result is a hot log that stays small even for workflows that run for years.

Built to scale out

The store scales out by construction rather than as a deployment afterthought:

  • Logical partitions (one per workflow run, plus channel partitions) hash into a 2⁶⁴ key space. A shard owns a contiguous range of that space.
  • There is no fixed shard count anywhere in the system. Shards are inclusive hash ranges that split and merge online.
  • Every routed request carries a shard-map epoch, so the map can change under live traffic, with stale senders rejected and re-resolved.
  • Because chaining, sequencing, and Merkle structure are all per logical partition, a shard split never cuts a workflow’s history.

The hot-tail replication model

Each shard is a Raft group replicating only its recent write-ahead segments (bounded to low single-digit GB), committing appends on a local NVMe quorum for sub-5ms durable acks. Sealed segments upload asynchronously to object storage as Parquet-compatible segments under a per-shard manifest. Object storage is thereafter the authoritative tier, and local copies are cache.

That model is what makes scale-out routine:

  • Replica movement transfers only the hot tail, regardless of history age.
  • Shard splits are metadata operations: children reference parent segments through hash-range filters, compacted lazily.
  • Disaster recovery rebuilds shards from object storage alone.
  • Cold partitions stay queryable in place via DuckDB or Spark, with no rehydration into Core.

The one throughput limit

A single workflow’s history partition is ordered and therefore single-leader, which bounds per-workflow event throughput (target: 5-10k events/sec sustained). The ceiling is inherent, and the pressure valves are architectural: channel partitions hash independently of their workflow’s history partition, so mailbox floods land elsewhere, and reducers absorb commutative load at the channel shard.

No schema migrations

There are no schema migrations against the store. Events are self-describing and tagged with a schema_id. Evolving your workflow’s data shape is a matter of versioning and, where needed, state migration, never an offline migration of the store itself. See Compatibility & versioning.

Visibility is a projection of the log

Because the log is the source of truth, visibility is a projection of it, not a separate database. Core continuously projects the log into an embedded columnar store, and you query live state with SQL. No Elasticsearch. See Visibility & search.