Reference
Architecture
This page is the map. Each subsystem has a dedicated concept page; here they sit together so you can see how the whole composes.
The three commitments
Hopskip is built on three architectural commitments, and the rest of the system follows from them:
- Sandboxed execution: determinism enforced at the runtime boundary, not by convention.
- A log-structured, verifiable event store: audit built into the storage format.
- Push-based dispatch with Core-managed resilience: resilience as declarative server-side policy.
Hopskip Core
Core is a stateless Rust binary (state lives in the log store) forming a Raft-consensus control plane. It performs:
- task scheduling and dispatch,
- policy enforcement (bulkheads, rate limits, FIFO groups, circuit breakers),
- stream and channel brokering, and
- log management.
Because Core is on the path of every task dispatch and every yielded value, it is the natural enforcement point for resilience. The trade is a heavier control plane than a polling matching service, in exchange for server-side resilience and sub-5ms push dispatch.
Core is designed as a library first: a near-pure deterministic state
machine with no I/O and no global state, taking LogStore, Transport, and
Clock traits. A Core unit test can fire timers with zero wall-clock sleeping.
The production binary is a thin shell around it.
The Log-Structured Event Store (LSES)
All workflow state is an append-only, Merkle-chained log:
- Logical partitions (one per run, plus channel partitions) hash into a 2⁶⁴ space; shards own contiguous ranges and split/merge online. There is no fixed shard count.
- Hot-tail replication: each shard is a Raft group replicating recent segments on a local NVMe quorum (sub-5ms acks); sealed segments upload to object storage as Parquet-compatible files, which become the authoritative tier.
- A small placement group owns the shard map and issues epoch-numbered leases that compose with worker fencing tokens.
Execution modes
Workflow code runs in the mode that fits the language; the full account is on the workflow modes page:
| Mode | Substrate | For |
|---|---|---|
| Wasm module | Wasmtime in the universal worker | Rust, Python, Go, Haskell |
| V8 isolate | a real V8 engine in the V8 worker | TypeScript |
| AD-VM | deterministic microVM (forked Firecracker) | JVM, BEAM, Ruby, native binaries |
| Integration | webhook gateway, co-process agent, OCI images | systems Hopskip cannot host |
The Wasm worker is a Rust binary embedding Wasmtime that hosts compiled workflow modules from any of the Wasm languages; TypeScript guests are served by the V8 worker on the same dispatch stream.
Memory snapshots
Because the runtime owns memory, it can serialize execution state at any await.
One mechanism replaces ContinueAsNew, bounds
replay cost, and enables time-travel debugging, production forking, and live
state patching. In the Wasm mode a snapshot is linear memory plus exported
globals; in the V8 mode it is the isolate’s heap; in the AD-VM it is a
full VM image.
Dispatch
Workers hold persistent bidirectional gRPC streams. Core pushes tasks with sub-5ms target latency; workers signal readiness and flow control on the same stream. Backpressure is native, and resilience patterns are declarative Core policy applied without redeploying workflows.
Visibility
Core continuously projects the log into an embedded store (SQLite).
Variables tagged @searchable are extracted at each checkpoint into
structured rows, and operators query live state
with no Elasticsearch and no separate cluster. Cluster-wide queries
scatter-gather across owning nodes.
The clock rule
Workflow-visible time is logical event time: the HLC of the resuming event, sourced only from Core. Every time-touching feature routes through it, which is what makes replay and snapshots deterministic.
What Hopskip rejects
- Universal transparent idempotency. Intercepting arbitrary traffic to auto-memoize side effects cannot distinguish “committed” from “rolled back.” Idempotency is a declarative primitive instead.
- Per-language AST tooling. All structural analysis targets compiled Wasm IR and WIT metadata, never source ASTs.
Multi-region (experimental)
The target is active-active: workflows execute against their local region’s Core with local-latency writes, logs replicate via Merkle-tree anti-entropy, and failover reroutes partitions via topology-aware routing. Whether branching workflow histories can be meaningfully CRDT-merged is an open question; the conservative fallback is per-partition single-writer leases with fast failover.