Skip to content

Recover from faults and replay turns

Akter keeps a faulty actor serving. The host refuses a cyclic synchronous wait at once and quarantines a poison envelope after 10 attempts. It records every turn so that you can rewind an actor and replay a trace.

Every host serves these features⁠

akter dev and Akter Cloud run akterd. akterd serves every feature on this page, so the operator commands work against both.

The in-memory host from @akterjs/effect/testing serves every feature too. InMemoryRpc.serve exposes its operator RPCs, so the operator commands work against an app that a test or a story serves.

The two hosts differ in the facts below.

FactakterdIn-memory host
Where a caller passes a wait chainThe akter-wait-chain request headerThe chain argument of InMemoryHost.waited
Turns that the flight recorder keepsAt least the newest 1,024 write turns of each actorEvery write turn
Largest recording of one turn256 KiBNo limit
Trace ID32 lowercase hexadecimal digitstrc_ and a six-digit number

A cyclic wait fails at once⁠

A synchronous call can carry a wait chain: the actors that wait on it, in order. When the chain already holds the call’s target, the host refuses the call with CyclicActorWaitError within 1 ms. The caller that makes the synchronous call builds the chain, and the host only checks it.

  • chain lists the cycle as actor addresses, such as ["Hop/a", "Hop/b", "Hop/a"].
  • The target keeps serving other calls. Nothing waits on a timeout.
  • The refusal is a client failure, so the caller handles it like ActorUnavailableError.
  • The host admits a call that carries no wait chain.

Pass a wait chain to akterd⁠

To pass a wait chain to akterd, send the akter-wait-chain header with the call to /rpc. The header holds a JSON array with one actor and key object per waiting actor, in order.

Text
akter-wait-chain: [{"actor":"Hop","key":"a"},{"actor":"Hop","key":"b"}]

A poison envelope is quarantined⁠

A message, a timer, a schedule fire, or a job result whose handler fails outside its declared errors is retried. Backoff starts at 100 ms and doubles up to 30 seconds. After 10 failed attempts, the host quarantines the envelope.

  • The envelope keeps its payload, its last error, its attempt count, and its trace.
  • The actor’s later commands and messages still commit. A quarantined envelope never blocks the actor.
  • A quarantined envelope never runs again on its own.
  • The host keeps the envelope in the akter_quarantine table of the actor’s own database. The host owns that table.

Supervise children with onChildFault⁠

To hear about a child’s quarantined envelope, declare onChildFault on the parent. The host then runs the named command with an Akter.ChildFault.

  1. Declare a command that lists Akter.System in callers and takes Akter.ChildFault as its input.
  2. Name that command in onChildFault.
TypeScript
import { Schema } from "effect"
import { Akter } from "@akterjs/effect/contract"

const Clerk = Akter.caller("user", Schema.Struct({ id: Akter.PrincipalId }));

const Watching = Schema.TaggedStruct("Watching", { faults: Schema.Array(Schema.String) });

const Depot = Akter.make("Depot", {
  callers: [Clerk, Akter.System],
  commands: {
    childFaulted: Akter.command({
      callers: [Akter.System],
      errors: [],
      from: [Watching],
      input: Akter.ChildFault,
      output: Schema.Void,
      to: [Watching],
    }),
  },
  initial: { input: Schema.Struct({}), to: Watching },
  key: Schema.String,
  onChildFault: Akter.onChildFault({ command: "childFaulted" }),
  parent: Akter.Root,
  queries: {},
  states: [Watching],
  tables: [],
});

Akter.ChildFault holds the fields below.

FieldHolds
childThe child’s actor and key.
memberThe child’s command that faulted.
attemptsThe failed attempts before the host quarantined the envelope.
envelopeThe command ID of the quarantined envelope, such as timer:<id> for a timer.
messageThe last error.

The host runs the command on the parent actor that has the child’s key. An actor without onChildFault takes no signal. The host still quarantines the child’s envelope.

The host retries a failed onChildFault delivery like any other envelope. After 10 failed attempts, the host quarantines the delivery in the parent with the kind childFault. The host then reports that quarantine to the parent’s own parent in the same way.

akterd refuses a child fault for a parent that was never created with ActorNotCreatedError and does not retry it. Create the parent before its children.

List quarantined envelopes⁠

The operator commands reach the app at --url. Without --url, they read AKTER_APP_URL.

To list an actor’s quarantined envelopes, run akter quarantine list.

Terminal
akter quarantine list --actor Parcel

The command prints each envelope with its key, kind, member, payload, attempts, last error, and trace. The kind is message, timer, jobResult, or childFault.

akter dev admits an operator command from a user caller or an apiKey caller. Akter Cloud admits an operator command only from an apiKey caller. Any other caller fails with UnauthenticatedError.

Rewind an actor to an earlier LSN⁠

Each commit of an actor gets the next log sequence number (LSN). To restore an actor’s state and SQL rows from an earlier LSN, run akter actors rewind.

Terminal
akter actors rewind --actor Parcel --key parcel-1 --to-lsn 12
  • The rewind commits as the next LSN, so the actor’s history stays append-only. A later rewind can return to any recorded LSN, including one before the rewind.
  • The rewind restores the actor’s state, its SQL rows, its timers, and its recorded replies from that LSN.
  • The rewind leaves the actor’s outbox, quarantined envelopes, files, and jobs as they are. A message that the actor already sent stays sent.

The host refuses a rewind with RewindRefusedError. The actor then keeps its current state and rows. The reason field names the cause.

ReasonCause
notCommittedThe LSN was never committed, or the flight recorder no longer holds it.
filesDeclaredThe actor declares files.
jobRunningA job of the actor is running.
historyConflictThe actor’s tables changed after that LSN, or a recorded row change no longer applies.

akterd keeps at least the newest 1,024 recorded turns of each actor. A rewind to an LSN that the recorder no longer holds fails with notCommitted.

Replay a trace⁠

A turn reads time only from its clock and randomness only from its seed. The host records each write turn’s invocation, host calls, host replies, and outcome under its trace. To re-run every turn of a trace against those replies, run akter replay with the trace ID.

Terminal
akter replay 0af7651916cd43dd8448eb211c80319c --step
  • akter quarantine list prints the trace ID of each quarantined envelope.
  • --step prints each replayed turn as its own JSON line, with its LSN, clock, host call count, and verdict.
  • The command ends with a summary of the matched and diverged turns.
  • A turn that diverges from its recorded outcome makes the command exit 1. A job step is skipped, because its journal replays it.

What akterd records for a replay⁠

akterd keeps each recording in the akter_turns table of the actor’s own database. The host owns that table.

  • A recording holds the turn’s invocation, its host calls and replies, its outcome, and an undo log of its row changes.
  • akterd keeps at least the newest 1,024 recordings of each actor, and at most 1,087.
  • akterd keeps no recording of a turn whose recording exceeds 256 KiB, so akter replay leaves that turn out.
  • akterd searches every actor database of the host for the turns of a trace.