Every feature of the library on one page, grouped by area, with the package that provides it, the guide section that explains it and the runnable example that shows it. Use it to answer "does it do X, and where is that written down?" without reading the guides in order.
The list is built from the code: the exported types, functions and options of every package in every module (workflow, clock, cron, memory, middleware, batch, failover, workflowtest, events, hybrid, and the separate modules admin, cli, hatchet, otel, recorder with recorder/recordertest, recorder/sqlrecorder, redis, sentry, sqlstore with sqlstore/outbox).
An Example link goes to a program under examples/ (or a Go example test) that uses the feature's API and is run by go test; a dash means no example shows the feature yet, and the guide section is the place to start. A Package is a path inside github.com/ubgo/workflow, where workflow is the root package, unless it names another module.
- Core contract (18)
- Workflows and steps (21)
- Events (5)
- Jobs (60)
- Schedules (36)
- Drivers (31)
- Middleware (16)
- Batches (19)
- Reliability (26)
- Observability (25)
- Testing (15)
- Operations (22)
- Admin (13)
- Extending (3)
The root package workflow: one client interface for every backend, workflows of steps, validation, capabilities, errors and plugins. Standard library only.
| Feature | Package | Guide | Example |
|---|---|---|---|
One Client interface for every backend (Register, Start, Enqueue, Cancel, Status, Signal, Schedule, Unschedule, Capabilities) |
workflow |
client › The Client interface | quickstart, first-workflow |
Nothing runs until Start(ctx); cancelling that context stops every worker |
workflow |
client › Start and stop | quickstart, first-workflow |
Specs are validated before anything runs (Validate, ValidateSchedule, ErrInvalid) |
workflow |
client › Validation | capabilities |
Capabilities a driver declares (Caps, Has, Missing, every Cap* constant) |
workflow |
capabilities › What a driver declares | capabilities |
Check a workflow or job against a driver before running it (Requires, RequiresJob, CheckCaps) |
workflow |
capabilities › Check a workflow before you run it | capabilities |
Check a schedule against a driver (RequiresSchedule, CheckScheduleCaps) |
workflow |
capabilities › Check a schedule | capabilities |
Refusal instead of silent emulation (ErrUnsupported, Unsupported, UnsupportedError) |
workflow |
capabilities › Refusals | capabilities |
Sentinel errors (ErrNotFound, ErrNotStarted, ErrAlreadyStarted, ErrDuplicate, ErrInvalid, ErrCancelled) |
workflow |
errors › Sentinel errors | capabilities |
A shared marker for backend outages (ErrUnavailable, Unavailable, IsUnavailable) |
workflow |
errors › Backend outages | capabilities |
Job labels and the reserved prefixes (ValidateLabels, ReservedLabelPrefixes, ReservedLabel, LabelPrefixInternal, LabelPrefixBatch, LabelPrefixOTel) |
workflow |
labels › Reserved prefixes | capabilities |
Plugins: step middleware, enqueue middleware and run-finished observers behind one name (Plugin, Use, ErrDuplicatePlugin) |
workflow |
plugins › Add a plugin | custom-plugin |
The optional plugin interfaces (StepMiddlewareProvider, EnqueueMiddlewareProvider, RunFinishedProvider) |
workflow |
writing-a-plugin › What a plugin can provide | custom-plugin |
Injected time (clock.Clock, clock.Real, clock.Timer) |
clock |
time › The clock interface | custom-driver |
Run errors: a failed run's error names run, workflow, step and attempt (RunError, RunErrorOf, Describe) |
workflow |
errors › Run errors | lifecycle-events |
Timeout sentinels (ErrTimeout, ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout, ErrRunTimeout, IsTimeout) |
workflow |
errors › Timeouts | — |
Trigger source: what started a run, set by the app or derived from the library's labels (WithSource, SourceOf, Source, ValidateSource, LibrarySources, LabelSource, LabelSourceRef, SourceSchedule, SourceScheduleManual, SourceChild, SourceBatch, SourceEvent, SourceExternal, MaxSourceLen, MaxSourceRefLen) |
workflow |
trigger-source › Name the source of a job | trigger-source |
Step events: a step's timeline handed to every bound sink (RecordEvent, RecordEventLevel, StepEvent, EventAttr, StepEventSink, StepEventFunc, WithStepEventSink, HasStepEventSink, MaxStepEventMessage, MaxStepEventAttrs, MaxStepEventValue) |
workflow |
step-events › Record events from a step | step-events |
Run outcome: whether the work succeeded, apart from the status (SetOutcome, Outcome, Outcomes, OutcomeSuccess, OutcomePartial, OutcomeError, OutcomeSkipped, DefaultOutcome, OutcomeSetter, OutcomeFunc, WithOutcomeSetter, MaxOutcomeNote) |
workflow |
run-outcome › Set it from a step | step-events |
Multi-step workflows: steps in a graph, typed handlers, waits for events and durable sleeps.
| Feature | Package | Guide | Example |
|---|---|---|---|
Workflows of named steps (Workflow, Step, Handler) |
workflow |
steps › Define a workflow | quickstart, first-workflow, signals-and-sleep |
Typed JSON handlers (Define) |
workflow |
steps › Typed handlers with Define | quickstart, first-workflow, signals-and-sleep |
Steps that depend on other steps, with their parents' outputs (Step.After, StepInput) |
workflow |
steps › Steps that depend on other steps | first-workflow, signals-and-sleep |
What a step knows about its run (StepInfo, StepInfoFrom, WithStepInfo) |
workflow |
steps › What a step knows about its run | first-workflow, signals-and-sleep |
Step and workflow timeouts (Step.Timeout, Workflow.Timeout) |
workflow |
steps › Timeouts | signals-and-sleep |
A handler that runs when a run fails (Workflow.OnFailure) |
workflow |
steps › Run a handler when a run fails | first-workflow, signals-and-sleep |
Wait for an event (Step.Wait, WaitSpec, Client.Signal, Event, CapWaitForEvent) |
workflow |
signals › Wait for an event | signals-and-sleep |
Match an event to its run by key, fixed or taken from the job (WaitSpec.Key, WaitSpec.KeyFrom, ResolveKey) |
workflow |
signals › Match the event to the run | signals-and-sleep |
Give up waiting after a deadline or a timeout (WaitSpec.Deadline, WaitSpec.Timeout) |
workflow |
signals › Give up after a deadline | signals-and-sleep |
Durable sleep inside a workflow (Step.Sleep, StepSleep, CapDurableSleep) |
workflow |
sleep › Sleep inside a workflow | signals-and-sleep |
A chain of jobs, each started when the one before it succeeds (Job.Chain, NextInChain, CloneJob, MaxChainJobs, CapChain) |
workflow |
chains › Enqueue a job with a chain | chain-edits |
Edit the rest of the chain from a running step, applied only when the attempt succeeds (PrependToChain, AppendToChain, ChainEditor, WithChainEditor, ChainEditorFrom, ChainEdits) |
workflow |
chains › Edit the chain from a running step | chain-edits |
Memoise work inside a step: the journal plugin, step middleware and store (Journal, JournalKeepFailed, JournalOption, WithJournal, JournalStore, MemoryJournal, NewMemoryJournal, ErrNoJournal) |
workflow |
memoisation › Turn on the journal | flow-memo |
Run a named piece of work once per run, its result returned on later attempts (Memo) |
workflow |
memoisation › Memoise a call with Memo | flow-memo |
One memoised call per item, only failed items rerun (MapEach, MapConcurrency, MapOption) |
workflow |
memoisation › Run a call per item with MapEach | flow-memo |
A bounded loop whose finished iterations replay from the journal (Loop, ErrLoopLimit) |
workflow |
memoisation › Loop until done with Loop | flow-memo |
An export or import in chunks with progress, resumed at the first unfinished chunk (ProcessChunks, ChunkOptions, ChunkProgress) |
workflow |
memoisation › Process rows in chunks with progress | flow-memo |
A value kept for the whole run, read by any later step (SetRunData, RunData) |
workflow |
memoisation › Share a value across steps | flow-memo |
An exclusive choice of one branch, expanded into ordinary steps (Choice, ChoiceSpec, Branch, ChoiceDecideSuffix) |
workflow |
branching › Build a choice | flow-memo |
Child runs started from a step, never twice on a retry (Spawn, SpawnOption, SpawnAllowDuplicates, LabelParentRun, LabelParentStep) |
workflow |
children › Start children with Spawn | flow-children |
Wait for child runs without holding a worker and join their results (AwaitChild, AwaitChildren, ChildOptions, ChildOptions.Parent, DefaultChildPoll) |
workflow |
children › Wait for children with AwaitChildren | flow-children |
The events package: publish and subscribe on any client, and signed inbound webhooks.
| Feature | Package | Guide | Example |
|---|---|---|---|
Publish one event to a run per subscription, deduplicated by event ID (events.New, Bus.Subscribe, Bus.Publish, Subscription, Message, Published, Delivery, WithIDFunc, AllowDuplicates, LabelEvent, LabelEventID, LabelSubscription) |
events |
publish-subscribe › Publish an event | events-webhooks |
Deliver a published event to runs waiting on it (events.WithSignal) |
events |
publish-subscribe › Deliver to waiting runs | — |
A closed-by-default webhook handler that verifies and enqueues (events.NewWebhook, WebhookConfig, DefaultWebhookMaxBody) |
events |
webhooks › Accept a signed webhook | events-webhooks |
HMAC signing and verification with key rotation (events.HMAC, Encoding, EncodingHex, EncodingBase64, Verifier, ErrSignature) |
events |
webhooks › HMAC signatures | events-webhooks |
Standard Webhooks signing with replay refusal (events.NewStandardWebhooks, StandardWebhooks, StandardWebhooksTolerance, HeaderWebhookID, HeaderWebhookTimestamp, HeaderWebhookSignature) |
events |
webhooks › Standard Webhooks | events-webhooks |
Enqueueing work and controlling how it runs: delays, keys, retries, concurrency, rate limits, cancel and status.
| Feature | Package | Guide | Example |
|---|---|---|---|
Enqueue a job (Job: Workflow, Payload, Labels; RunRef) |
workflow |
enqueue › Enqueue a job | quickstart |
Priorities (Job.Priority) |
workflow |
enqueue › Priority | concurrency |
Delayed start (Job.RunAt, ScheduledID, CapDelayedStart) |
workflow |
delays › Run a job later | delayed-jobs |
Idempotency keys: one run per key (Job.IdempotencyKey, CapIdempotencyKey) |
workflow |
idempotency › Idempotency keys | idempotency |
Require, honour or ignore keys per workflow (IdempotencyPolicy: IdempotencyOptional, IdempotencyRequired, IdempotencyOff) |
workflow |
idempotency › Require or ignore keys | idempotency |
Retry policy with exponential backoff (RetryPolicy, Backoff, Max, NoRetryOn) |
workflow |
retries › Retry policy | retries, first-workflow |
Retries per step (Step.Retry, EffectiveRetries, RetriesNone) |
workflow |
retries › Retries per step | retries |
Jitter on the backoff (Backoff.Jitter, Delay, JitteredDelay, CapRetryJitter) |
workflow |
retries › Backoff and jitter | retries |
A retry delay named by the error (RetryPolicy.DelayFor, RetryAfter, RetryAfterDelay, NextDelay, CapRetryDelay) |
workflow |
retries › Retry after a delay the error names | retries |
Errors that must not retry (Permanent, IsPermanent) |
workflow |
retries › Errors that must not retry | retries, first-workflow |
Cancel a run (Client.Cancel, CapCancel, ErrCancelled) |
workflow |
cancel-and-status › Cancel a run | delayed-jobs |
Read a run's status (Client.Status, Status, IsTerminal, RunID) |
workflow |
cancel-and-status › Read a run's status | delayed-jobs |
Limit concurrent runs per key (Concurrency, KeyExpr, KeyFunc, MaxRuns, CapConcurrencyKey) |
workflow |
concurrency › Limit runs per key | concurrency |
What happens to the run over the limit (ConcurrencyQueueNewest, ConcurrencyDropNewest, ConcurrencyCancelNewest, ConcurrencyCancelInProgress, ConcurrencyRoundRobin) |
workflow |
concurrency › Strategies | concurrency |
Rate limits per key (RateLimit, CapRateLimit) |
workflow |
concurrency › Rate limits | concurrency |
Named queues: a workflow's queue and a run sent elsewhere (Workflow.Queue, Job.Queue, DefaultQueue, EffectiveQueue, CapQueue, CapQueueOverride) |
workflow |
queues › Give a workflow its queue | named-queues, sql-queue-workers, hatchet-manage |
Workers and priority per queue (QueueConfig, memory.WithQueues, sqlstore.WithQueues, hatchet.Config.Queues) |
workflow |
queues › Decide how workers serve queues | named-queues, sql-queue-workers, hatchet-manage |
Cap a queue across every worker (QueueConfig.Limit, CheckQueueLimits, CapQueueLimitGlobal) |
workflow, memory, sqlstore |
queues › Cap a queue across every worker | queue-limit |
Pause and resume a workflow, a queue or everything (Pauser, PauseTarget, PauseWorkflow, PauseQueue, PauseAll, CheckPause, CapPauseWorkflow, CapPauseQueue) |
workflow |
manage-runs › Pause and resume, queues › Pause a queue | manage-runs, named-queues, sql-queue-workers |
Retries per dispatch (Job.Retry, EffectiveRetriesFor, CapJobRetry) |
workflow |
retries › Per-dispatch retries | retry-limits |
Which retry policy a step gets: job over step over workflow over default (EffectiveRetriesFor, DefaultRetryMax) |
workflow |
retries › Which policy a step gets | discard-exhausted |
A retry deadline (RetryPolicy.Until, CapRetryUntil) |
workflow |
retries › A deadline: Until | retry-limits |
Cap real errors apart from releases (RetryPolicy.MaxErrors, Attempts, RetryPolicy.Next, CapRetryMaxErrors) |
workflow |
retries › Errors, not releases: MaxErrors | retry-limits |
Drop a run instead of failing it (Discard, IsDiscarded, DiscardedError, ErrDiscarded, CapDiscard) |
workflow |
retries › Drop the run: Discard | retry-limits |
Discard a run that ran out of retries instead of failing it (RetryPolicy.DiscardOnExhaust, RetryPolicy.ExhaustedError, CapDiscardExhausted) |
workflow |
retries › Quiet giving up: DiscardOnExhaust | discard-exhausted |
List runs by filter, or one run by its ref (RunQuerier, RunQuery, RunQuery.Ref, RunInfo, RunPage, CapRunQuery) |
workflow |
manage-runs › List runs | manage-runs, sql-queue-workers, hatchet-manage |
Search runs by label, exactly or by a value prefix, and "has the key" (RunQuery.Labels, RunQuery.LabelPrefixes, CapLabelPrefix, CheckRunQuery) |
workflow |
manage-runs › Search by label | label-search |
An indexed label search on the SQL driver, and its backfill for older runs (migration 005_run_labels, sqlstore.Client.IndexLabels) |
github.com/ubgo/workflow/sqlstore |
manage-runs › The label index on the SQL driver | label-search |
Re-run a failed run from the step that failed (RunRetrier, CapRetryRun, ErrWrongState) |
workflow |
manage-runs › Re-run a failed run | manage-runs, sql-queue-workers, hatchet-manage |
Forget a finished run (RunDeleter, CapDeleteRun) |
workflow |
manage-runs › Forget a finished run | manage-runs, sql-queue-workers |
Cancel every run a filter matches (BulkCanceller, ValidateCancelQuery, CapCancelAll) |
workflow |
manage-runs › Cancel every waiting run | manage-runs |
Each failed attempt kept on the run with its error, time and stack, bounded per run (RunInfo.Errors, AttemptError, StackTracer, StackOf, NewAttemptError, AppendAttemptError, DefaultErrorHistory, MaxErrorStack, WithErrorHistory on each driver, CapErrorHistory) |
workflow |
manage-runs › Error history | run-repair |
Force a run to an end from outside: complete or fail it (RunForcer, ForceComplete, ForceFail, ErrForced, ForcedFailure, CanForceComplete, CanForceFail, CapForceRun) |
workflow |
manage-runs › Force a run to an end | run-repair |
Run a workflow inline, with no client or workers (RunInline, InlineMiddleware, InlineSleep, InlineNow, InlineEvents) |
workflow |
run-inline › Run it | run-inline |
Retrying and discarded run states (StatusRetrying, StatusDiscarded, RunInfo.NextAttempt, RunInfo.NextAttemptAt, EndStatus, Status.Valid, CapRunStates) |
workflow |
run-states › Retrying, run-states › Discarded | run-states |
The run state machine as data (Transition, Transitions, Status.TransitionsFrom, Status.CanTransitionTo) |
workflow |
run-states › The state machine | run-states |
Hold a request's enqueues and send them only when it succeeds (Deferred, NewDeferred, Flush, Discard, WithDeferred, DeferredFrom, DeferEnqueue, ErrDeferredClosed) |
workflow |
deferred › Hold enqueues made under a Deferred, deferred › Without HTTP | deferred-dispatch |
HTTP middleware that flushes on success and drops on an error status or a panic (DeferHTTP, DeferOptions, DeferSucceeded) |
workflow |
deferred › Wrap your HTTP handler | deferred-dispatch |
Run a Go function as an in-process job (ClosureJob, EnqueueClosure, ClosureEnqueuer, ValidateClosure, ClosureWorkflow, ClosureStep, LabelClosure, CapClosure) |
workflow |
closures › Enqueue a closure | closures |
Durable functions by name: a Go function registered under a name and enqueued with a JSON argument, on every driver (NewFunc, Func, RegisterFunc, FuncDefinition, Func.Enqueue, Func.Job, FuncJob, FuncStep, FuncQueue, FuncRetry, FuncTimeout, FuncCatch, FuncConfig, FuncOption) |
workflow |
functions › Define the function, functions › Enqueue by name alone | functions |
A leaky-bucket, cost-based limit on calls to an outside API, one bucket per key (UpstreamLimit, NewUpstreamLimit, BucketSpec, Take, TakeOrSnooze, Wait, ErrThrottled) |
workflow |
upstream-limits › Take before each call | upstream-limits |
Follow the upstream's reported level, and a bucket store of your own (UpstreamLimit.Observe, BucketStore, MemoryBuckets) |
workflow |
upstream-limits › Follow the upstream's own count | upstream-limits |
One idempotency key per scope per calendar period (OncePerPeriod, PeriodKey, Period, PeriodDay, PeriodWeek, PeriodMonth, PeriodQuarter, PeriodYear) |
workflow |
idempotency › Once per period | upstream-limits |
Typed jobs (JobOf) |
workflow |
payloads › Build a job from a struct | typed-payloads |
Decode payloads and results (DecodePayload) |
workflow |
payloads › Decode a payload or a result | typed-payloads |
Payload size limits (PayloadTooLargeError, ErrPayloadTooLarge, PayloadLimiter, CheckPayload, middleware.MaxPayload, memory.WithPayloadLimit) |
workflow, middleware, memory |
payloads › Cap the payload size | typed-payloads |
Payload versioning (LabelPayloadVersion, Upcaster, Upcast, PayloadVersionOf, DefineVersioned, middleware.PayloadVersion) |
workflow, middleware |
payloads › Change a payload's shape between releases | typed-payloads |
Progress (ReportProgress, Progress, MaxProgressData, ValidateProgress, RunInfo.Progress, LifecycleProgress) |
workflow |
progress › Report progress from a step | results-progress |
Heartbeats (Heartbeat, Heartbeater, Step.HeartbeatTimeout, CapHeartbeat) |
workflow |
progress › Catch a hung step | snooze-heartbeat |
Results (ResultReader, RunInfo.Output, RunOutput, Sinks, CapResult) |
workflow |
results › What a run's result is | results-progress |
Await a run (Await, AwaitResult, AwaitOptions, DefaultAwaitPoll, RunFailedError) |
workflow |
results › Wait for a result | results-progress |
Run timeout (Workflow.RunTimeout, Workflow.OnTimeout, FailureHandler, CapRunTimeout) |
workflow |
timeouts › Cap a whole run | snooze-heartbeat |
Timeout errors (ErrTimeout, ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout, ErrRunTimeout, TimeoutError, IsTimeout) |
workflow |
timeouts › Tell timeouts apart | snooze-heartbeat |
Tenants (Job.Tenant, ValidateTenant, RunQuery.Tenant, CapTenant) |
workflow |
tenants › List one tenant's runs | results-progress |
Snooze (Snooze, SnoozeDelay, ErrSnoozed, MaxSnooze, CapSnooze) |
workflow |
retries › Snooze a step that is not ready | snooze-heartbeat |
Runs due soon (RunQuery.DueFrom, RunQuery.DueTo, RunInfo.RunAt, CapDueQuery) |
workflow |
manage runs › List the runs due soon | run-reschedule |
Reschedule a delayed run (RunRescheduler) |
workflow |
manage runs › Reschedule a delayed run | run-reschedule |
Cancel one step (StepCanceller, ErrStepCancelled) |
workflow |
manage runs › Cancel one step | run-reschedule |
Weighted queues (QueueConfig.Weight, PickWeighted) |
workflow |
queues › Share a worker between queues by weight | queue-weights |
FIFO per key (Concurrency.FIFO, CapKeyFIFO) |
workflow |
concurrency › Keep one key's runs in order | queue-weights |
Cron on a workflow or at run time, in a time zone, with a policy for missed ticks.
| Feature | Package | Guide | Example |
|---|---|---|---|
Cron declared on a workflow (Workflow.Cron, CapCron) |
workflow |
cron › Cron on a workflow | cron-schedules |
Schedules created and removed at run time; a named schedule is updated in place (Schedule, Client.Schedule, Client.Unschedule, ScheduleID) |
workflow |
cron › Schedules at run time | cron-schedules |
Own cron parser: five fields, names, steps, macros and @every (cron.Parse, cron.Schedule) |
cron |
cron › Cron syntax | cron-schedules |
| Day of month and day of week together (the Vixie rule) | cron |
cron › Day of month and day of week | cron-schedules |
Parse errors that name the field (cron.ParseError) |
cron |
cron › Parse errors | cron-schedules |
Another cron dialect (cron.Parser, memory.WithCronParser, sqlstore.WithCronParser, workflowtest.FakeCronParser) |
cron |
cron › Use another cron dialect | cron-schedules |
| One fire per tick across every process sharing a database | sqlstore |
cron › One fire per tick across nodes | postgres-cluster, cron-schedules |
Schedules in a time zone (Workflow.CronLocation, Schedule.Location, CapCronZone, IsUTC) |
workflow |
time-zones › Schedules in a time zone | cron-schedules |
| Daylight saving time: one run per day for fixed times, real time for periodic ones | cron |
time-zones › Daylight saving time | cron-schedules |
Misfire policies for missed ticks (Misfire, MisfireSkip, MisfireFireOnce, MisfireFireAll, Within, Max, MisfireGrace) |
workflow |
misfire › Misfire policies | cron-schedules |
Compute missed ticks in your own driver (Misfire.Due, TickSource) |
workflow |
misfire › Compute missed ticks yourself | cron-schedules |
Build an expression from a named frequency (cron.Builder, DailyAt, EveryMinutes, WeeklyOn, Quarterly, ErrBuilder) |
cron |
cron › Build an expression from a frequency | cron-builder |
Time-of-day windows (Schedule.Windows, Workflow.CronWindows, Window, Windows.Allows, CapScheduleWindow) |
workflow |
conditions › Limit ticks to time windows | schedule-hooks |
A condition asked at every tick (Workflow.ScheduleWhen, CapScheduleWhen) |
workflow |
conditions › Decide in code with ScheduleWhen | schedule-hooks |
Every tick reported: fired, skipped with a reason, misfired with a count (TickNotifier, Tick, TickObserver, FiredLabels, LabelSchedule, LabelScheduleName, CapTickEvents) |
workflow |
hooks › Watch every tick | schedule-hooks |
Hooks per schedule (AttachScheduleHooks, ScheduleHooks: Fired, Skipped, OnSuccess, OnFailure, After) |
workflow |
hooks › Hooks for one schedule | schedule-hooks |
Heartbeat pings from schedule hooks (Pinger, NewPinger, HTTPPing, PingFunc, WithPingTimeout, WithPingLogger) |
middleware |
hooks › Ping a heartbeat service | schedule-hooks |
List schedules with their next fire (ScheduleLister, ScheduleInfo, CapScheduleList) |
workflow |
manage › List schedules | schedule-hooks, hatchet-manage |
Pause and resume a schedule; a paused tick never fires (SchedulePauser, ScheduleTarget, CapSchedulePause) |
workflow |
manage › Pause a schedule | schedule-hooks |
| Group schedules that share options, with a Go helper | workflow |
manage › Group schedules | schedule-groups |
Skip a tick while the schedule's previous run is unfinished, across a fleet (Schedule.Overlap, Workflow.CronOverlap, OverlapPolicy, OverlapAllow, OverlapSkip, Blocks, TickReasonOverlap, CapOverlapSkip) |
workflow |
overlap › Skip a tick while the previous run is unfinished | schedule-overlap |
Run a schedule now with its input and labels, obeying its overlap policy (ScheduleTriggerer, TriggerSchedule, ErrOverlap, OverlapRefusal, LabelTrigger, TriggeredLabels, Tick.Manual, CapScheduleTrigger) |
workflow |
overlap › Trigger a schedule now | schedule-overlap |
A schedule's last fire, the run it started and that run's status (ScheduleInfo.Last, ScheduleLast, ScheduleInfo.Overlap, CapScheduleHistory) |
workflow |
overlap › Read the last fire | schedule-overlap |
Capture what a task logs and returns and send it to a file, a writer or your own function such as a mailer (middleware.CaptureOutput, OutputLogger, Output, OutputRecord, OutputSink, OutputSinkFunc, WriterSink, FileSink, FileReplace, FileAppend, FormatOutput, CaptureResult, CaptureOnlyFailures, CaptureLevel, CaptureTee, CaptureOnly, CaptureOnSinkError, CaptureClock) |
middleware |
output › Choose a sink and wrap the steps, output › Options | task-output |
Keep captured output with the run in the database, shared by every process and served to the admin (sqlstore.OutputStore, Client.OutputStore, NewOutputStore, OutputLimits, StoredOutput, RunOutput, PruneBefore, ErrOutputNoRun, OutputMigrationOwner, admin.OutputSourceFunc) |
github.com/ubgo/workflow/sqlstore |
output › Keep output in the database | output-store |
A seconds field: an opt-in six-field dialect with seconds first (cron.ParseSeconds, Schedule.Seconds, Workflow.CronSeconds, ParseCron, CapCronSeconds) |
cron, workflow |
cron › A seconds field | schedule-specs |
Structured calendar specs compiled to a five-field expression (cron.Calendar, ErrCalendar) |
cron |
cron › Calendar specs | schedule-specs |
The tick a fired run stands for (LabelTickAt, StepInfo.TickAt, TickTime, FiredLabelsAt, CapTickTime) |
workflow |
cron › Which tick a run is for | — |
Load a zone by name, failing at boot on an empty, host-local or unknown zone (cron.LoadZone, ErrZone) |
cron |
time-zones › Load a zone by name | schedule-specs |
An embedded zone database for images without one (cron/tzdata) |
cron/tzdata |
time-zones › Load a zone by name | — |
The next runs in the schedule's zone and in UTC (NextRuns, NextRunsQuery, NextRun, MaxNextRuns) |
workflow |
time-zones › Show the next runs in two zones | schedule-specs |
A start and end for a schedule (Active, Schedule.Active, Workflow.CronActive, NextActive, CapScheduleActive) |
workflow |
conditions › An active period | schedule-specs |
Environment gating and combined conditions (InEnvironment, AllWhen) |
workflow |
conditions › Run only in some environments | schedule-specs |
Keep only the newest finished runs of a schedule (Schedule.KeepRuns, Workflow.CronKeepRuns, CapScheduleKeep) |
workflow |
manage › Keep the newest runs | — |
A schedule stored as versioned JSON (Schedule.MarshalJSON, UnmarshalSchedule, ScheduleJSONVersion, ErrScheduleJSON) |
workflow |
manage › Store a schedule as JSON | schedule-specs |
An in-process cron on every node (LocalCron, NewLocalCron, LocalCronFunc, WithLocalCronClock, WithLocalCronParser, WithLocalCronErrors, ErrLocalCronStarted) |
workflow |
local-cron › Run a function on every node | schedule-specs |
The backends behind Client: in-process memory, SQL databases and Hatchet.
| Feature | Package | Guide | Example |
|---|---|---|---|
In-memory driver (memory.New, Stop) |
memory |
memory › Create the client | quickstart, sqlite-worker |
Memory driver options (WithWorkers, WithClock, WithLogger, WithMiddleware, WithJitterSource) |
memory |
memory › Options | sqlite-worker |
What the memory driver can and cannot do (every capability but CapDurable) |
memory |
memory › What it can do | sqlite-worker |
SQL driver on database/sql for SQLite, Postgres and MySQL 8 (sqlstore.New) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Create the client | sqlite-worker |
| Pick a database and a driver for it | github.com/ubgo/workflow/sqlstore |
sqlstore › Pick a database | sqlite-worker |
Workers, polling and an immediate wake-up (WithWorkers, WithPollInterval, Client.Wake) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Workers and polling | sqlite-worker |
Wake idle workers on other machines as soon as work is due, with polling as the fallback (WithNotifier, Notifier, sqlstore/pgnotify on Postgres LISTEN/NOTIFY) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Wake workers at once | worker-wakeup |
A client that only enqueues: web nodes enqueue and read status, worker nodes run (sqlstore.WithEnqueueOnly, CapEnqueueOnly) |
github.com/ubgo/workflow/sqlstore |
deploy › Web nodes that only enqueue | enqueue-only |
Leases, crashed workers and poison steps (WithLease, WithLeaseClock, WithMaxReclaims, ErrAbandoned) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Leases and crashed workers | sqlite-worker |
How database errors are sorted (WithErrorClassifier, DefaultErrorClassifier, ErrorClass, WithTxAttempts) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Database errors | sqlite-worker |
Your own run and schedule IDs (WithIDGenerator, IDKind, DefaultIDGenerator) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Run IDs | sqlite-worker |
Other options (WithClock, WithLogger, WithMiddleware, WithJitterSource, DB) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Options | sqlite-worker |
Hatchet driver (hatchet.New, Config) |
github.com/ubgo/workflow/hatchet |
hatchet › Create the client | hatchet-worker |
Hatchet configuration (Namespace, WorkerName, Slots, StepTimeout, IdempotencyMaxHold, RegisterTimeout, VisibilityLag, FailedSettle, Middleware, Logger, Clock) |
github.com/ubgo/workflow/hatchet |
hatchet › Configuration | hatchet-worker |
What the Hatchet driver can do and what it refuses at Register |
github.com/ubgo/workflow/hatchet |
hatchet › What it can do and what it refuses | hatchet-worker |
| Known Hatchet engine quirks | github.com/ubgo/workflow/hatchet |
hatchet › Known engine quirks | hatchet-worker |
Reach the Hatchet SDK for what the interface does not cover (Client.Native) |
github.com/ubgo/workflow/hatchet |
hatchet › Reach the Hatchet SDK | hatchet-worker |
Memory driver queues, run retention and stop when idle (memory.WithQueues, WithRunRetention, RunRetention, DefaultRunRetention, WithStopWhenIdle) |
memory |
memory › Options | named-queues, manage-runs, graceful-drain |
SQL driver queues, durable pauses and run admin (WithQueues, Runs, Retry, Delete, CancelAll, Pause, PauseSchedule, Schedules, OnTick, Drain, WithStopWhenIdle, EnqueueMany; migration 003_queues_admin) |
github.com/ubgo/workflow/sqlstore |
sqlstore › Queues, pauses and run admin | sql-queue-workers |
Hatchet queues, run list, retry, schedule list and drain (Config.Queues, Runs, Retry, CancelAll, Schedules, Drain) |
github.com/ubgo/workflow/hatchet |
hatchet › Queues, run management and drain | hatchet-manage |
Hatchet workflow and schedule pause through the REST API (Config.API, NewAPI) |
github.com/ubgo/workflow/hatchet |
hatchet › Pause a workflow or a schedule | hatchet-pause |
Hatchet trigger-now, last fire from the engine's run list, and an enqueue-only client (TriggerSchedule, Config.ScheduleHistoryWindow, DefaultScheduleHistoryWindow, Config.EnqueueOnly) |
github.com/ubgo/workflow/hatchet |
hatchet › Run a schedule now, read its last fire, enqueue from web nodes | - |
Redis driver on go-redis v9: durable runs, delays, retries, sleeps, cron once across clients, idempotency keys (redis.New, WithPrefix) |
github.com/ubgo/workflow/redis |
redis › Create the client | redis-worker |
| What the Redis driver declares and refuses | github.com/ubgo/workflow/redis |
redis › What it can do and what it refuses | redis-worker |
Whole-run claims under a renewed lease, fenced writes, reclaim after a crash, clean-stop requeue (WithLease, WithLeaseClock, WithMaxReclaims, ErrAbandoned) |
github.com/ubgo/workflow/redis |
redis › How work moves | - |
| Redis schedules: one fire per tick across clients, named upsert, overlap, trigger-now and last fire | github.com/ubgo/workflow/redis |
redis › Schedules | redis-worker |
Redis web nodes, run list, delete, force, error history and retention (WithEnqueueOnly, Runs, Delete, ForceComplete, ForceFail, WithErrorHistory, WithRetention) |
github.com/ubgo/workflow/redis |
redis › Web nodes and operations | redis-worker |
Other Redis options (WithClock, WithPollInterval, WithWorkers, WithIDFunc, WithLogger, WithMiddleware, WithCronParser, WithJitterSource) |
github.com/ubgo/workflow/redis |
redis › Options | - |
Hybrid placement: schedules on one client, runs on another (hybrid.New) |
hybrid |
hybrid › Split schedules from runs | hybrid-placement |
| Redis at SQL parity: waits, concurrency keys, rate limits, queues, pauses, run-finished, chains, schedule conditions | redis |
redis › What it can do and what it refuses | redis-worker |
Redis shared stores (UniqueStore, DebounceStore, JournalStore, BucketStore) |
redis |
redis › Shared stores | — |
Cross-cutting behaviour around every step and every enqueue, in the middleware package.
| Feature | Package | Guide | Example |
|---|---|---|---|
Step middleware (workflow.Middleware, Chain, each driver's WithMiddleware) |
middleware |
step-middleware › Wrap every step | middleware-stack |
Recover from panics (Recover, ErrPanic) |
middleware |
step-middleware › Recover from panics | middleware-stack |
Time out a step (Timeout) |
middleware |
step-middleware › Time out a step | middleware-stack |
Fail without retrying on chosen errors (FailOn) |
middleware |
step-middleware › Fail on chosen errors | middleware-stack |
Skip a step (Skip) |
middleware |
step-middleware › Skip a step | middleware-stack |
Throttle a failing dependency with a circuit breaker (ThrottleErrors, ThrottleConfig, CircuitStore, CircuitPolicy, MemoryCircuit, ErrThrottled) |
middleware |
step-middleware › Throttle a failing dependency | middleware-stack |
Enqueue middleware (workflow.EnqueueMiddleware, ChainEnqueue, WrapClient) |
middleware |
enqueue-middleware › Wrap every enqueue | middleware-stack |
Route jobs to another workflow or add labels (Route, RouteRule) |
middleware |
enqueue-middleware › Route jobs | middleware-stack |
Payload codecs (Codec, EncodeEnqueue, DecodeStep, WithCodec, ErrDecode) |
middleware |
codecs › Transform payloads | encrypted-payloads |
Encrypt payloads, with key rotation (Encrypt) |
middleware |
codecs › Encrypt payloads | encrypted-payloads |
| Write your own middleware | workflow |
writing-middleware › Write step middleware | custom-plugin |
Release a step for later without running it (Release, ErrReleased) |
middleware |
step-middleware › Release a step for later | release-jobs |
Unique until a step of the run starts (UniqueUntilStarted, ReleaseUniqueOnStart) |
middleware |
enqueue-middleware › Unique until a step starts | release-jobs |
Wrappers forward every optional client interface (workflow.Forward, WrapClient, WithCodec) |
workflow |
client › The Client interface | client-wrapper |
Refuse a job without a trigger source, or with one not on an allow-list (RequireSource) |
middleware |
enqueue-middleware › Require a trigger source | trigger-source, run-recorder |
Step events in a step's captured output (CaptureOutput as a workflow.StepEventSink) |
middleware |
step-middleware › Capture step events | step-events |
Groups of jobs with callbacks and progress, on any client, in the batch package.
| Feature | Package | Guide | Example |
|---|---|---|---|
Dispatch a batch with Then, Catch and Finally callbacks (batch.New, Manager.Dispatch, Batch, AllowFailures) |
batch |
batches › Dispatch a batch | batches |
Progress of a batch (Manager.Progress, Progress, Percent, Processed) |
batch |
batches › Track progress | batches |
Cancel a batch (Manager.Cancel) |
batch |
batches › Cancel a batch | batches |
Labels on a batch, copied onto its callbacks (Batch.Labels, LabelBatch, LabelIndex) |
batch |
batches › Labels on a batch | batches |
Your own batch IDs (WithIDFunc) |
batch |
batches › Batch IDs | batches |
How a batch learns a job ended (Source, HookSource, PollSource, WithCompletionSource, WithStatusPoll, Manager.Attach, Manager.RunFinished) |
batch |
completion › How a batch learns a job ended | batches |
A label-based completion source on Hatchet (hatchet.Client.BatchSource, LabelSource) |
github.com/ubgo/workflow/hatchet |
completion › On Hatchet | batches |
Callbacks that fail to enqueue are retried (WithCallbackRetry, WithManualCallbackRetry, Manager.RetryCallbacks, PendingCallback) |
batch |
completion › Callbacks that fail to enqueue | batches |
Where batch state lives (Store, MemoryStore, State, JobState) |
batch |
durability › Where batch state lives | batch-durable |
A durable batch store on SQL (sqlstore.Client.BatchStore, BatchStore) |
github.com/ubgo/workflow/sqlstore |
durability › A durable store | batch-durable |
Resume a dispatch whose process died (WithResumableDispatch, WithDispatchLease, Manager.ResumeDispatches, RunFinder) |
batch |
durability › Resume a dispatch that died | batch-durable |
Start a batch as one native group (GroupRunner, hatchet.Client.RunGroup, chunking by GroupChunkRuns and GroupChunkBytes) |
batch |
native-groups › Native group runs | batches |
The Hatchet batch coordinator (GroupSpawner, Config.BatchCoordinator, SpawnGroup, CancelGroup) |
github.com/ubgo/workflow/hatchet |
native-groups › The Hatchet batch coordinator | batches |
Before and progress callbacks (Batch.Before, Batch.Progress) |
batch |
lifecycle › Report the start and every job end | batch-lifecycle |
Add jobs to a running batch (Manager.Add, ErrClosed, ErrDispatching) |
batch |
lifecycle › Add jobs from inside the batch | batch-lifecycle |
Retry a finished batch's failed jobs (Manager.RetryFailed, WithRetryableJobs, LabelAttempt, JobLabels) |
batch |
lifecycle › Retry the failed jobs | batch-lifecycle |
Prune old batches (Manager.Prune, PruneFilter, Pruner, sqlstore.BatchStore.Prune) |
batch |
lifecycle › Prune old batches | batch-lifecycle |
Wait for a batch inside a workflow (Manager.AwaitStep, AwaitPoll, AwaitPolls, AwaitAllowFailures, ErrBatchPending, ErrBatchFailed) |
batch |
await › Build the await step | batch-lifecycle |
Bulk enqueue without a batch (EnqueueMany, ManyEnqueuer; one transaction on sqlstore) |
batch |
await › Start many jobs without a batch | batch-lifecycle |
Not losing work and not doing it twice: unique jobs, debounce, claim check, transactional enqueue, the outbox and failover.
| Feature | Package | Guide | Example |
|---|---|---|---|
Unique jobs for a time (Unique, UniqueStore, MemoryUnique, UniqueKeyLabel) |
middleware |
unique-jobs › Unique for a time | unique-jobs |
Unique until the run finishes (UniqueUntilFinished, UniqueOwnerLabel) |
middleware |
unique-jobs › Unique until the run finishes | unique-jobs |
Unique by workflow and arguments, with the fields, queue and labels that count (UniqueByArgs, ArgsFields, ArgsQueue, ArgsLabels, ArgsKeyPrefix, DefaultArgsKeyPrefix) |
middleware |
unique-jobs › Unique by workflow and arguments | unique-jobs |
A unique store shared by every process, usable without the SQL driver (sqlstore.UniqueStore, Client.UniqueStore, NewUniqueStore, PruneExpired) |
github.com/ubgo/workflow/sqlstore |
unique-jobs › A store shared by every process | unique-jobs |
Debounce a burst of jobs into one run (Debounce, DebounceStore, DebounceBurst, MemoryDebounce, WithDebounceStore) |
middleware |
debounce › Debounce a burst | debounce |
Debounce state shared by every process (sqlstore.DebounceStore, Client.DebounceStore, NewDebounceStore) |
github.com/ubgo/workflow/sqlstore |
debounce › A store shared by every process | debounce |
Claim check: big payloads and step outputs go to a blob store (NewClaimCheck, ClaimCheck.Enqueue, ClaimCheck.Step, BlobStore, MemoryBlobStore, WithClaimKeys, ClaimCheckLabel) |
middleware |
claim-check › Move big payloads out of the queue | claim-check |
Delete a run's blobs when it finishes (ClaimCheck.CleanupOnFinish, BlobPrefixDeleter, ErrBlobNotFound) |
middleware |
claim-check › Delete blobs when the run finishes | claim-check |
Enqueue inside your own transaction (sqlstore.Client.EnqueueTx) |
github.com/ubgo/workflow/sqlstore |
transactional-enqueue › Enqueue inside your transaction | transactional-enqueue |
Transactional outbox for any client (outbox.New, Outbox.Write, Outbox.Migrate) |
github.com/ubgo/workflow/sqlstore/outbox |
outbox › Write jobs inside your transaction | outbox |
The relay that delivers outbox entries (NewRelay, Relay.Start, Stop, Notify, Flush, WithBatchSize, WithLease, WithPollInterval, WithEnqueueTimeout) |
github.com/ubgo/workflow/sqlstore/outbox |
outbox › Relay entries to any client | outbox |
Failed and dead outbox entries (WithBackoff, WithMaxAttempts, WithOnDead, ListDead, DeadQuery, Requeue) |
github.com/ubgo/workflow/sqlstore/outbox |
outbox › Failed and dead entries | outbox |
A relay target without idempotency keys (AllowDuplicates) |
github.com/ubgo/workflow/sqlstore/outbox |
outbox › Targets without idempotency keys | outbox |
Fail over to a second backend (failover.New) |
failover |
failover › Fail over to a second backend | failover |
Decide what counts as an outage (Classifier, DefaultClassifier, StrictClassifier, WithClassifier) |
failover |
failover › Decide what counts as an outage | failover |
Idempotency keys required, since a job can reach both backends (ErrKeyRequired, AllowWithoutKey) |
failover |
failover › Why a key is required | failover |
| Run references that name their backend; capabilities are the intersection | failover |
failover › Run references and capabilities | failover |
Schedules on the primary only, or on both (WithSchedulesOnBoth) |
failover |
failover › Schedules | failover-schedules |
Skip the primary for a while after repeated failovers (WithCoolOff, WithClock) |
failover |
failover › Cool-off | failover |
Watch failovers and cool-offs (WithObserver, Event, EventKind) |
failover |
failover › Watch failovers | failover |
Query, retry, cancel and pause runs through a failover client (failover.Client: Runs, Retry, Delete, CancelAll, Pause, the schedule family) |
failover |
failover › Query, retry, cancel and pause runs | failover-manage |
Refuse a job carrying a credential in its payload or labels (SecretGuard, SecretDetector, NewSecretDetector, SecretRules, SecretFinding, ErrSecretInPayload) |
workflow |
secrets › Refuse credentials at enqueue | secret-guard |
Name a secret in the payload and resolve it in the step (SecretRef, SecretResolver, ResolveSecret, ErrNoSecretRef) |
workflow |
secrets › Pass a SecretRef instead | secret-guard |
Unique by tagged fields (middleware.ArgsTagged, UniqueTag) |
middleware |
unique jobs › Fields tagged on the payload type | unique-states |
Unique by holder state (middleware.UniqueWith, UniqueConfig.States, UniqueHolders) |
middleware |
unique jobs › Unique while the run waits to start | unique-states |
Return the existing run (UniqueConfig.ReturnExisting, WithEnqueueOutcome, EnqueueOutcome, MarkDuplicate, MarkInserted) |
workflow, middleware |
unique jobs › Return the existing run | unique-states |
Knowing what ran: the run-finished hook, logs, traces, metrics and a run history the app owns.
| Feature | Package | Guide | Example |
|---|---|---|---|
React when a run finishes (RunNotifier, OnRunFinished, RunEvent, RunFinishedFunc, CapRunFinished) |
workflow |
run-finished › React when a run finishes | run-finished |
Run ends on Hatchet, best effort (RunFinishedPoll, RunFinishedWindow, RunFinishedCatchUp) |
github.com/ubgo/workflow/hatchet |
run-finished › On Hatchet | run-finished |
Alert on final failures, deduplicated and rate limited, to a sink the app writes (Alerts, AlertPlugin, Alerter, AlerterFunc, Alert, AlertConfig, AlertLimiter, MemoryAlertLimiter, AlertLimits, DefaultAlertDedup) |
middleware |
failure-alerts › Deduplication and rate limits | failure-alerts |
Report failures to an error tracker, final and per attempt, never the payload (workflow.Reporter, ReporterFunc, Failure, FailureOf, RunEvent.Step, middleware.Report, ReportFailures, ReportAttempts, ReportConfig) |
workflow, middleware |
failure-alerts › Add them as plugins | failure-alerts |
Send failures to Sentry, labels only on opt-in (sentry.New, WithLabels, WithAllLabels, WithLevels, WithFingerprint, WithEvent, DefaultFingerprint) |
github.com/ubgo/workflow/sentry |
failure-alerts › Report to Sentry | sentry-reporting |
Log every step (Logging, WithLogLevel, StepLogEvent, StepPhase) |
middleware |
logging › Log every step | middleware-stack |
Driver and component logs (WithLogger on memory, sqlstore, batch and outbox; hatchet.Config.Logger) |
several | logging › Driver logs | middleware-stack |
OpenTelemetry spans and metrics for steps and enqueues (otel.New, Instrumentation.Step, Instrumentation.Enqueue, WithTracerProvider, WithMeterProvider) |
github.com/ubgo/workflow/otel |
opentelemetry › Add tracing and metrics | opentelemetry |
Trace context carried from enqueue to step in job labels (WithPropagator, LabelPrefix) |
github.com/ubgo/workflow/otel |
opentelemetry › Trace context from enqueue to step | opentelemetry |
Parent, link or new root for step spans (WithParenting, ParentChild, ParentLink, ParentNewRootLink) |
github.com/ubgo/workflow/otel |
opentelemetry › Parent or link | opentelemetry |
Span names (WithStepSpanName, WithEnqueueSpanName) |
github.com/ubgo/workflow/otel |
opentelemetry › Span names | opentelemetry |
Job labels on spans, opt-in (WithLabelFilter, WithAllLabels, LabelAttributePrefix) |
github.com/ubgo/workflow/otel |
opentelemetry › Job labels on spans | opentelemetry |
Metrics and their names (MetricStepDuration, MetricStepOutcomes, MetricEnqueueCount, Outcome, WithMetricLabels, WithClock) |
github.com/ubgo/workflow/otel |
opentelemetry › Metrics | opentelemetry |
Queue attribute, queue wait histogram and queue depth gauge (AttrQueue, MetricQueueWait, QueueWaitBuckets, MetricQueueDepth, ObserveQueueDepth, QueueDepthFunc; StepInfo.Queue, StepInfo.ReadyAt) |
github.com/ubgo/workflow/otel |
opentelemetry › Queue wait and queue depth | opentelemetry, admin |
Count run ends by status, and a discarded step outcome (Instrumentation.RunFinished, MetricRunsFinished, AttrRunStatus, OutcomeDiscarded) |
github.com/ubgo/workflow/otel |
opentelemetry › Count run ends | opentelemetry |
Lifecycle events: inserted, started, retrying, snoozed, progress, finished (LifecycleNotifier, OnLifecycle, LifecycleEvent, LifecycleKind) |
workflow, memory, redis |
lifecycle › Watch the lifecycle feed | lifecycle-events |
Stuck-attempt reports, detection only (middleware.Stuck, StuckConfig, StuckEvent) |
middleware |
stuck-jobs › Report attempts that run too long | lifecycle-events |
A run history the app owns on any driver: source, input, every step attempt with its events, outcome, result, timings (recorder.New, Recorder.Plugin, RunRecord, StepRecord, EventRecord, MemoryStore, WithClock, WithDefaultSource, WithResultReader, WithOnError, WithWriteTimeout, WithName) |
github.com/ubgo/workflow/recorder |
run-recorder › Wire it in | run-recorder |
Record runs from web nodes that only enqueue (Recorder.EnqueuePlugin) |
github.com/ubgo/workflow/recorder |
run-recorder › Web nodes that only enqueue | recorder-split |
Read and filter the history, and point the admin at it (Store.Run, Store.Records, Query with Sources, Outcomes, Search, OnlyStale; admin.WithRunQuerier, QueryCaps) |
github.com/ubgo/workflow/recorder |
run-recorder › Read the history | run-recorder |
Redact and bound what the history stores (Redactor, JSONPaths, PerWorkflow, Chain, WithRedactor, WithMaxPayload, WithStepBodies, WithErrorHistory) |
github.com/ubgo/workflow/recorder |
run-recorder › Redaction and payload bounds | run-recorder |
Two-stage retention of the history with an archive hook (Retention, RetentionRule, PruneEvery, Pruner) |
github.com/ubgo/workflow/recorder |
run-recorder › Retention | — |
Close the gaps the seams leave: stale and missing runs, backfill of runs started past the recorder (NewReconciler, Reconciler.Pass, Reconciler.Run, ReconcileReport, WithStaleAfter, WithBackfillWindow, WithBackfillQuery) |
github.com/ubgo/workflow/recorder |
run-recorder › Close the gaps with the Reconciler | run-recorder |
The recorder's SQL store on Postgres, MySQL 8 and SQLite (sqlrecorder.New, Postgres, MySQL, SQLite, Store.Migrate, Store.MigrationScripts, WithTablePrefix, WithRetention) |
github.com/ubgo/workflow/recorder/sqlrecorder |
run-recorder › The SQL store | run-recorder, recorder-split |
Bring your own history store and prove it (recorder.Store, the Apply* merge rules, recordertest.RunStore, recordertest.RunDriver) |
github.com/ubgo/workflow/recorder |
run-recorder › Bring your own store | — |
Testing the code that uses the library, and the drivers themselves.
| Feature | Package | Guide | Example |
|---|---|---|---|
A fake client that runs nothing and refuses what a real driver refuses (workflowtest.NewFake) |
workflowtest |
fake › Test app code with the Fake | example_fake_test.go, app-tests |
Fluent assertions (AssertEnqueued, AssertEnqueuedTimes, AssertNotEnqueued, AssertNothingEnqueued, AssertScheduled, AssertSignalled, AssertCancelled) |
workflowtest |
fake › Assertions | example_fake_test.go, app-tests |
Job matchers (HasKey, HasLabel, HasPayload, HasPayloadJSON, HasPriority, HasRunAt, Where) |
workflowtest |
fake › Match jobs | example_fake_test.go, app-tests |
Inspect and finish recorded runs (Enqueued, Jobs, Signals, Schedules, Cancels, Finish) |
workflowtest |
fake › Inspect what was recorded | example_fake_test.go, app-tests |
Test operator actions on the Fake: trigger a schedule, force a run (Fake.TriggerSchedule, ForceComplete, ForceFail, Triggers, Forced, FakeForce, AssertTriggered, AssertForced, FakeSchedule.LastRun) |
workflowtest |
fake › Operator actions | schedule-overlap |
A fake that behaves like your production driver (FakeLike, FakeCaps, FakeKeyExpr, FakeCronParser) |
workflowtest |
fake › Behave like your production driver | app-tests |
Record the enqueues of a real client (NewRecorder, Recorder) |
workflowtest |
recorder › Record a real client | app-tests |
Drive time in tests (clock.NewFake, Advance, Set, BlockUntil, Pending) |
clock |
fake-clock › Drive time in tests | app-tests |
The conformance suite every driver runs (workflowtest.Run, Harness, Instance, Instance.Restart, Instance.EnqueueOnly) |
workflowtest |
writing-a-driver › Run the conformance suite | custom-driver |
Batch and chain assertions (AssertBatched, BatchSize, BatchHasJob, BatchWhere, AssertBatchCount, AssertNothingBatched, AssertChained) |
workflowtest |
batches-and-chains › Assert a batch was dispatched, batches-and-chains › Assert a chain | batch-tests |
Fake some workflows and run the others for real (FakeExcept, FakeOnly) |
workflowtest |
batches-and-chains › Fake some workflows and run the others | batch-tests |
Test one step's outcome (RunStep, StepOutcome: succeeded, failed, permanent, discarded, released) |
workflowtest |
run-step › Run the handler and check the outcome | batch-tests |
Queue, retry, pause and delete assertions (AssertEnqueuedOn, HasQueue, HasRetry, AssertRetried, AssertDeleted, AssertPaused, AssertNotPaused) |
workflowtest |
operations › Assert the queue and the retry budget, operations › Assert pauses, retries and deletes | batch-tests |
Test job chains and closures (HasChain, StepChain, StepCaps, StepOutcome.Chain, ChainNames, Fake.AssertClosureEnqueued, Closures, RunClosures) |
workflowtest |
batches-and-chains › Assert a chain, fake › Inspect what was recorded | chain-edits, closures |
Run everything now in tests: delays, backoffs, snoozes and sleeps end at once (memory.Client.RunAll, RunAllReport, ErrRunAllStalled) |
memory |
run-all › Run everything now | run-all |
Running the SQL driver in production: schema, retention, deployment and tuning.
| Feature | Package | Guide | Example |
|---|---|---|---|
Versioned migrations on a ledger, under the engine's lock (Client.Migrate, MigrationOwner) |
github.com/ubgo/workflow/sqlstore |
migrations › Run the migrations | migrations-export |
Each store's own migration list (UniqueMigrationOwner, DebounceMigrationOwner, outbox.MigrationOwner) |
github.com/ubgo/workflow/sqlstore |
migrations › Each store's migration list | migrations-export |
A newer schema is refused, not overwritten (ErrSchemaTooNew, SchemaTooNewError) |
github.com/ubgo/workflow/sqlstore |
migrations › Rolling back the code | migrations-export |
Ship the SQL through goose, Atlas, Flyway or a DBA (MigrationScripts, MigrationScript) |
github.com/ubgo/workflow/sqlstore |
migrations › Ship the SQL through your own tool | migrations-export |
Table names and a Postgres schema (WithTablePrefix, WithSchema) |
github.com/ubgo/workflow/sqlstore |
migrations › Table names and schemas | migrations-export |
Keep finished runs for a while (WithRetention, Retention, KeepLast, PerWorkflow, Undelivered) |
github.com/ubgo/workflow/sqlstore |
retention › Keep finished runs for a while | retention |
Prune on your own schedule (Client.Prune, PruneResult, WithoutAutoPrune, WithPruneInterval, WithPruneChunk) |
github.com/ubgo/workflow/sqlstore |
retention › Prune on your own schedule | retention |
Archive runs before they are deleted (WithArchive, ArchiveFunc, FinishedRun) |
github.com/ubgo/workflow/sqlstore |
retention › Archive before deleting | retention |
Housekeeping of cron ticks and expired locks (WithTickRetention, WithHousekeepingEvery) |
github.com/ubgo/workflow/sqlstore |
tuning › Housekeeping | retention |
| Run several worker processes on one database | github.com/ubgo/workflow/sqlstore |
deploy › Run several workers | postgres-cluster |
Shut down cleanly (Stop, the Start context) |
several | deploy › Shut down cleanly | postgres-cluster |
Hatchet tunings for big batches and slow engines (GroupChunkRuns, GroupChunkBytes, BatchCoordinatorTimeout, BatchCoordinatorRetry, BatchCoordinatorMaxInput) |
github.com/ubgo/workflow/hatchet |
tuning › Hatchet tunings | hatchet-worker |
Stop gracefully: drain in-flight steps, or stop when idle (Drainer, Drain, Done, CapDrain, memory.WithStopWhenIdle, sqlstore.WithStopWhenIdle) |
workflow |
client › Stop gracefully | graceful-drain, sql-queue-workers, hatchet-manage |
Say what a drain cut off (ReportingDrainer, DrainWithReport, DrainReport, DrainCut, DrainDisposition, DrainRequeued, DrainCancelled, DrainEngineOwned, DrainOutcome, DrainTracker, CapDrainReport) |
workflow, memory, sqlstore, redis, hatchet |
deploy › Give running steps a grace period | graceful-drain-report |
A drain with a grace period, as a hook for any phased shutdown manager (DrainContext, DrainTimeout, DrainTimeoutReport, ShutdownHook, ShutdownHookReport) |
workflow |
deploy › Give running steps a grace period | graceful-drain-report |
Move workers between queues as their load changes (Balancer, BalancerFunc, ByDepth, ByWait, Proportional, BalanceConfig, QueueBounds) |
workflow |
balancing › Pick a strategy, balancing › Set bounds and a step size | worker-balancing |
Read queue loads and set worker counts while running, on an interval (WorkerScaler, QueueLoad, BalanceLoop, DefaultBalanceEvery, CapWorkerScale) |
workflow |
balancing › Run the balance loop | worker-balancing |
Recycle a worker after a number of jobs, a time or a memory size, so its supervisor starts a fresh process (Recycle, ValidateRecycle, HeapObjectBytes, DefaultRecycleCheck, ErrRecycle, RecycleError, RecycleReason, RecycleWatch, NewRecycleWatch, StopCauser, Serve, memory.WithRecycle, sqlstore.WithRecycle, hatchet.Config.Recycle) |
workflow |
recycling › Set the limits, recycling › Serve until the worker stops | worker-recycling |
Node identity per attempt (StepInfo.Node, AttemptError.Node, RunInfo.AttemptedBy, WithNodeID, DefaultNodeID, CapNodeIdentity) |
workflow, memory, redis |
nodes › See which node ran each attempt | node-health |
Online node registry (NodeLister, NodeInfo, NodeMode) |
workflow, memory, redis |
nodes › List the online nodes | node-health |
Health and readiness probes (HealthChecker, HealthReport, HealthCheck, CheckHealth) |
workflow, memory, redis |
nodes › Health and readiness probes | node-health |
Scheduler-only and worker-only processes (redis.WithSchedulerOnly, redis.WithoutScheduler) |
redis |
nodes › Scheduler-only and worker-only processes | node-health |
The operator's surface in the separate admin module (standard library and the core only): Go operations, a JSON API and queue alerts over any driver.
| Feature | Package | Guide | Example |
|---|---|---|---|
Run and queue operations over any driver, each refused when the driver lacks it (admin.New, Runs, Run, Failures, Retry, Delete, Cancel, CancelAll, Pause, Resume, Paused, Schedules, Support, Supports, Requirements) |
github.com/ubgo/workflow/admin |
operations › Wrap the client | admin |
Queue depth and oldest wait per queue (QueueStats, QueueDepths, WithMaxScan) |
github.com/ubgo/workflow/admin |
operations › Queue depth and oldest wait | admin |
A closed-by-default admin JSON API with typed login and authorization hooks (NewHandler, Auth, Identify, Authorize, Request, Action, WithPayloads) |
github.com/ubgo/workflow/admin |
http-api › Say who may do what, http-api › Routes | admin |
Queue depth and wait alerts, once on crossing and once on resolve (Monitor, NewMonitor, Threshold, Alerter, AlerterFunc, Alert, WithInterval, WithRepeat) |
github.com/ubgo/workflow/admin |
alerts › Set thresholds and an alerter | admin |
Discarded runs listed apart from failures, and a scaling client's workers per queue (Discarded, GET /discarded, QueueStat.Workers, QueueStat.Running, QueueReport.Workers) |
github.com/ubgo/workflow/admin |
operations › Find runs, operations › Queue depth and oldest wait | run-states |
A web UI for queues, runs, failures, discarded runs and schedules, served by the admin handler behind the same login (ActionUI, the ui/ routes; strict Content-Security-Policy, no third-party code) |
github.com/ubgo/workflow/admin |
ui › Accept the session cookie in Identify, ui › The screens | admin-ui |
Run counts and numbered pages, label prefix filters, and schedule pause and resume over HTTP (Admin.CountRuns, RunCount, count=1, label=key=prefix*, Admin.PauseSchedule, Admin.ResumeSchedule, OpSchedulePause, ActionPauseSchedule, ActionResumeSchedule) |
github.com/ubgo/workflow/admin |
http-api › Routes, operations › Find runs | admin-ui |
Inspect and trigger a schedule, force a run to an end, and read captured output (Admin.Schedule, TriggerSchedule, ForceComplete, ForceFail, MaxForceReason, OpScheduleTrigger, OpForce, WithOutputs, OutputSource, OutputSourceFunc, MemoryOutputs, RunOutput) |
github.com/ubgo/workflow/admin |
operations › Schedules, operations › Captured output | output-store |
Admin routes and screens for schedule detail and Run now, force complete and fail, per-attempt errors with stacks on opt-in, and captured output (GET /schedules/{id}, POST /schedules/{id}/trigger, POST /runs/{id}/complete, POST /runs/{id}/fail, GET /runs/{id}/output, WithStacks, Resource.Reason) |
github.com/ubgo/workflow/admin |
http-api › Routes, ui › The screens | - |
Bulk retry and delete by filter, with dry run (RetryAll, DeleteAll, PreviewCancelAll, BulkOptions, BulkResult, WithMaxBulk) |
admin |
operations › Act on many runs at once | admin-bulk |
Run-end charts (Metrics, MetricsSource, MetricsSourceFunc, WithMetrics, MemoryMetrics) |
admin |
operations › Run-end history for charts | admin-bulk |
Nodes, health, reschedule and cancel a step from the admin (Nodes, Health, Reschedule, CancelStep) |
admin |
operations › Nodes and health | admin-cli |
The workflow command line client (cli.Run, cli.Options, cli.Main) |
cli |
cli › Commands | admin-cli |
Your own drivers, stores, plugins and middleware.
| Feature | Package | Guide | Example |
|---|---|---|---|
| Write a driver | workflow |
writing-a-driver › What a driver implements | custom-driver |
| Bring your own store for unique locks, debounce state, circuits, blobs or batches | middleware, batch |
custom-stores › Bring your own store | custom-plugin |
| Write a plugin | workflow |
writing-a-plugin › Write a plugin | custom-plugin |