Recover from faults and replay turns
Akter keeps a faulty actor serving. The host refuses a cyclic synchronous wait at once and quarantines a poison envelope after 10 attempts. It records every turn so that you can rewind an actor and replay a trace.
Every host serves these features
akter dev and Akter Cloud run akterd. akterd serves every feature on this page, so the operator commands work against both.
The in-memory host from @akterjs/effect/testing serves every feature too. InMemoryRpc.serve exposes its operator RPCs, so the operator commands work against an app that a test or a story serves.
The two hosts differ in the facts below.
| Fact | akterd | In-memory host |
|---|---|---|
| Where a caller passes a wait chain | The akter-wait-chain request header | The chain argument of InMemoryHost.waited |
| Turns that the flight recorder keeps | At least the newest 1,024 write turns of each actor | Every write turn |
| Largest recording of one turn | 256 KiB | No limit |
| Trace ID | 32 lowercase hexadecimal digits | trc_ and a six-digit number |
A cyclic wait fails at once
A synchronous call can carry a wait chain: the actors that wait on it, in order. When the chain already holds the call’s target, the host refuses the call with CyclicActorWaitError within 1 ms. The caller that makes the synchronous call builds the chain, and the host only checks it.
chainlists the cycle as actor addresses, such as["Hop/a", "Hop/b", "Hop/a"].- The target keeps serving other calls. Nothing waits on a timeout.
- The refusal is a client failure, so the caller handles it like
ActorUnavailableError. - The host admits a call that carries no wait chain.
Pass a wait chain to akterd
To pass a wait chain to akterd, send the akter-wait-chain header with the call to /rpc. The header holds a JSON array with one actor and key object per waiting actor, in order.
akter-wait-chain: [{"actor":"Hop","key":"a"},{"actor":"Hop","key":"b"}]A poison envelope is quarantined
A message, a timer, a schedule fire, or a job result whose handler fails outside its declared errors is retried. Backoff starts at 100 ms and doubles up to 30 seconds. After 10 failed attempts, the host quarantines the envelope.
- The envelope keeps its payload, its last error, its attempt count, and its trace.
- The actor’s later commands and messages still commit. A quarantined envelope never blocks the actor.
- A quarantined envelope never runs again on its own.
- The host keeps the envelope in the
akter_quarantinetable of the actor’s own database. The host owns that table.
Supervise children with onChildFault
To hear about a child’s quarantined envelope, declare onChildFault on the parent. The host then runs the named command with an Akter.ChildFault.
- Declare a command that lists
Akter.Systemincallersand takesAkter.ChildFaultas its input. - Name that command in
onChildFault.
import { Schema } from "effect"
import { Akter } from "@akterjs/effect/contract"
const Clerk = Akter.caller("user", Schema.Struct({ id: Akter.PrincipalId }));
const Watching = Schema.TaggedStruct("Watching", { faults: Schema.Array(Schema.String) });
const Depot = Akter.make("Depot", {
callers: [Clerk, Akter.System],
commands: {
childFaulted: Akter.command({
callers: [Akter.System],
errors: [],
from: [Watching],
input: Akter.ChildFault,
output: Schema.Void,
to: [Watching],
}),
},
initial: { input: Schema.Struct({}), to: Watching },
key: Schema.String,
onChildFault: Akter.onChildFault({ command: "childFaulted" }),
parent: Akter.Root,
queries: {},
states: [Watching],
tables: [],
});Akter.ChildFault holds the fields below.
| Field | Holds |
|---|---|
child | The child’s actor and key. |
member | The child’s command that faulted. |
attempts | The failed attempts before the host quarantined the envelope. |
envelope | The command ID of the quarantined envelope, such as timer:<id> for a timer. |
message | The last error. |
The host runs the command on the parent actor that has the child’s key. An actor without onChildFault takes no signal. The host still quarantines the child’s envelope.
The host retries a failed onChildFault delivery like any other envelope. After 10 failed attempts, the host quarantines the delivery in the parent with the kind childFault. The host then reports that quarantine to the parent’s own parent in the same way.
akterd refuses a child fault for a parent that was never created with ActorNotCreatedError and does not retry it. Create the parent before its children.
List quarantined envelopes
The operator commands reach the app at --url. Without --url, they read AKTER_APP_URL.
To list an actor’s quarantined envelopes, run akter quarantine list.
akter quarantine list --actor ParcelThe command prints each envelope with its key, kind, member, payload, attempts, last error, and trace. The kind is message, timer, jobResult, or childFault.
akter dev admits an operator command from a user caller or an apiKey caller. Akter Cloud admits an operator command only from an apiKey caller. Any other caller fails with UnauthenticatedError.
Rewind an actor to an earlier LSN
Each commit of an actor gets the next log sequence number (LSN). To restore an actor’s state and SQL rows from an earlier LSN, run akter actors rewind.
akter actors rewind --actor Parcel --key parcel-1 --to-lsn 12- The rewind commits as the next LSN, so the actor’s history stays append-only. A later rewind can return to any recorded LSN, including one before the rewind.
- The rewind restores the actor’s state, its SQL rows, its timers, and its recorded replies from that LSN.
- The rewind leaves the actor’s outbox, quarantined envelopes, files, and jobs as they are. A message that the actor already sent stays sent.
The host refuses a rewind with RewindRefusedError. The actor then keeps its current state and rows. The reason field names the cause.
| Reason | Cause |
|---|---|
notCommitted | The LSN was never committed, or the flight recorder no longer holds it. |
filesDeclared | The actor declares files. |
jobRunning | A job of the actor is running. |
historyConflict | The actor’s tables changed after that LSN, or a recorded row change no longer applies. |
akterd keeps at least the newest 1,024 recorded turns of each actor. A rewind to an LSN that the recorder no longer holds fails with notCommitted.
Replay a trace
A turn reads time only from its clock and randomness only from its seed. The host records each write turn’s invocation, host calls, host replies, and outcome under its trace. To re-run every turn of a trace against those replies, run akter replay with the trace ID.
akter replay 0af7651916cd43dd8448eb211c80319c --stepakter quarantine listprints the trace ID of each quarantined envelope.--stepprints each replayed turn as its own JSON line, with its LSN, clock, host call count, and verdict.- The command ends with a summary of the matched and diverged turns.
- A turn that diverges from its recorded outcome makes the command exit 1. A job step is skipped, because its journal replays it.
What akterd records for a replay
akterd keeps each recording in the akter_turns table of the actor’s own database. The host owns that table.
- A recording holds the turn’s invocation, its host calls and replies, its outcome, and an undo log of its row changes.
akterdkeeps at least the newest 1,024 recordings of each actor, and at most 1,087.akterdkeeps no recording of a turn whose recording exceeds 256 KiB, soakter replayleaves that turn out.akterdsearches every actor database of the host for the turns of a trace.