Which backend runs your workflows: the in-process memory driver, the SQL driver on SQLite, Postgres or MySQL 8, the Redis driver, or the Hatchet engine. Your workflow code is the same on all four; the choice decides what survives a restart, how many machines can share the work, what you have to operate, and which features are available. The core refuses at start-up what a driver cannot do, so a wrong choice shows up as an error naming the missing capability, not as silently lost work.
| memory | sqlstore | redis | hatchet | |
|---|---|---|---|---|
| Package | github.com/ubgo/workflow/memory (core) |
github.com/ubgo/workflow/sqlstore (own module) |
github.com/ubgo/workflow/redis (own module) |
github.com/ubgo/workflow/hatchet (own module) |
Survives a restart (CapDurable) |
no: everything is lost on exit | yes: every run, step result, wait, sleep and schedule is a row | yes: runs, schedules and leases are Redis keys (with AOF, up to a second can be lost if Redis itself crashes) | yes: the engine holds it |
| Machines that can share the work | one process | SQLite: processes on one machine; Postgres or MySQL: any number of machines | any number of machines on one Redis | any number of workers |
| You operate | nothing | the database you already run | a Redis you already run, set to noeviction |
a Hatchet engine, which itself runs on Postgres |
| Capabilities | all but CapDurable |
all but CapClosure |
all but CapClosure and CapQueueLimitGlobal: wait steps, concurrency and rate keys, named queues with weights, pauses, run-finished events, chains, retry and bulk cancel, schedule conditions and tick events, tenants, results, snooze, heartbeats, run timeouts, node identity, worker scaling |
Durable, DelayedStart, WaitForEvent, DurableSleep, ConcurrencyKey, RateLimit, IdempotencyKey, Cron, Cancel, Queue, RunQuery, RetryRun, CancelAll, ScheduleList, Drain, ScheduleTrigger, ScheduleHistory, EnqueueOnly, Tenant, TickTime, DueQuery, Result, CronSeconds, KeyFIFO, DrainReport (pauses with Config.API) |
| Missing there | durability; closures are in-process only | CapClosure (a Go function cannot be stored; a function by name runs everywhere) |
CapClosure (a Go function cannot be stored; a function by name runs everywhere), CapQueueLimitGlobal (no cluster-wide queue cap; Workers is per client) |
CapRunFinished (run ends are delivered best-effort), CapSnooze, CapHeartbeat, CapRunTimeout, CapNodeIdentity (no per-attempt worker history), CapScheduleActive, CapScheduleKeep, queue weights, CapRetryJitter, CapRetryDelay, misfire policies, CapCronZone (crons in UTC only), CapRunStates (no retrying or discarded status), CapDiscardExhausted, CapQueueLimitGlobal (no count of a queue across workers), CapChain (a run's shape is fixed at registration), CapWorkerScale (slots fixed per worker process), CapClosure, CapLabelPrefix (metadata matched by exact value) |
| Keys for concurrency and rate limits | Go functions (workflow.KeyFunc) |
Go functions | Go functions | CEL expressions (workflow.KeyExpr) |
| Enqueue inside your own SQL transaction | no | Client.EnqueueTx |
through the outbox | through the outbox |
Durable memo journal and shared leaky bucket for Memo and UpstreamLimit |
in-process only | JournalStore, BucketStore (tables) |
JournalStore, BucketStore (keys) |
in-process only; bring your own store |
| Batches | per job | per job; durable batch store (Client.BatchStore) |
per job, completion by status polling | native bulk triggers (batch.GroupRunner), an opt-in coordinator run |
| Run history with the recorder | every end recorded; sqlrecorder or MemoryStore |
every end recorded; sqlrecorder in the same database |
every end recorded; no Redis store, use sqlrecorder |
run ends best effort: run the Reconciler; the record outlives the engine's retention |
| Guide | memory | sqlstore | redis | hatchet |
| Example | quickstart | sqlite-worker, postgres-cluster | redis-worker | hatchet-worker |
The capability rows are printed by the capabilities example from the drivers themselves; the capability reference has the full matrix.
- If it is a test of your app code, use
workflowtest.NewFake(workflowtest.FakeLike(prod)), which runs nothing and refuses what production would (Test app code with the Fake). Use the memory driver with a fake clock when the test must run the steps. - If losing queued work on a crash or deploy is acceptable (a cache warm-up, a best-effort notification, a CLI tool, local development), use memory.
- If you have one server and want durability with nothing new to run, use sqlstore on a SQLite file. Several processes on that machine can share it.
- If you run several machines and already have Postgres or MySQL 8, use sqlstore on it. Workers claim steps with
FOR UPDATE SKIP LOCKED, a cron tick fires once across all of them, and a crashed worker's step is taken over when its lease expires. - If a job must be enqueued if and only if your own transaction commits, use sqlstore with
EnqueueTxwhen the queue is the same database, or the outbox in front of any driver otherwise (Enqueue, EnqueueTx or the outbox). - If you already run Redis and need durable jobs, cron and workflows, use the redis driver: runs are claimed whole under a lease, writes are fenced, a tick fires once across clients, and it has everything the SQL driver has except closures and transactional enqueue (The Redis driver).
- If you already run Hatchet, or want its dashboard and engine-side scheduling, use the hatchet driver. Write concurrency and rate-limit keys as CEL (
workflow.KeyExpr), keep crons in UTC, and do not rely on misfire policies or retry jitter, which it refuses. - If one backend must not be a single point of failure for enqueues, put
failover.New(primary, secondary)in front of two drivers; every job then needs an idempotency key (Fail over to a second backend). - If people must see who started each run and what each step did, past the driver's own retention, add the run recorder on any driver: module
recorder, plusrecorder/sqlrecorderto keep the history in Postgres, MySQL 8 or SQLite. On Hatchet run itsReconcilertoo. - If you start on one and expect to move, write against
workflow.Clientonly, keep keys as Go functions unless you are on Hatchet, and run your workflows through aFakeLiketest for each target driver.
- The memory driver accepts a workflow that sleeps for a week: durability is never inferred from a workflow, so nothing refuses it. Choosing memory is choosing to lose that sleep on exit.
- The Redis driver refuses closures (
EnqueueClosure) and expression keys (KeyExpr), and cannot enqueue inside your SQL transaction; use functions by name, Go key functions and the outbox there. - SQLite is durable on one machine, not shared across machines. A network file system does not change that.
sqlstore.Newnever creates tables, andStartnever migrates: runClient.Migratefrom your deploy first (Run the migrations).- Every sqlstore client sharing a database must use the same cron parser; a different table prefix is a different, separate queue.
- On Hatchet,
OnRunFinishedis best-effort: a restarted client replays only ends of runs created withinConfig.RunFinishedCatchUp. Features that rely on run ends (UniqueUntilFinished, batch callbacks through the hook) keep their fallback there: a TTL, a poller or Hatchet's ownBatchSource. - Hatchet crons run in UTC and a tick missed while the engine was down is dropped. A schedule that must run in a local zone or catch up belongs on sqlstore.
- A key written as a Go function is refused by Hatchet at
Register, and a CEL expression is refused by memory, sqlstore and Redis atStart. Moving between them means rewriting keys.