Skip to content

fix(backup): gate retries on last attempt, add bounded upload backoff - #108

Merged
digizeph merged 2 commits into
mainfrom
fix/backup-retry-loop
Aug 30, 2026
Merged

digizeph merged 2 commits into
mainfrom
fix/backup-retry-loop

Conversation

@digizeph

Copy link
Copy Markdown
Member

Fixes #107.

The periodic backup gate advanced last_backup_time only on success, so once an S3 upload failed, every update tick started a new attempt that re-uploaded the entire file from byte zero. With a multi-gigabyte database this loops indefinitely and never succeeds.

Changes:

  • The serve loop now tracks the last backup attempt (initial and periodic paths), so a failed attempt waits for a full BGPKIT_BROKER_BACKUP_INTERVAL_HOURS window before the next try.
  • The S3 upload step retries up to 3 attempts with exponential backoff (2s initial, doubling) before giving up for the interval window, logging each retry.
  • Failure logs now state explicitly that the next attempt waits for the interval to elapse.
  • CHANGELOG entry under Unreleased.

Not addressed here (possible follow-ups from #107): compressing the backup before upload, resumable multipart upload, and a failure-side heartbeat.

Fixes #107.

The periodic backup gate advanced only on success, so a failed S3
upload re-ran the full-file backup on every update tick. Track the
last attempt instead, and retry the upload step up to 3 times with
exponential backoff before waiting out the interval window.
@digizeph
digizeph enabled auto-merge (rebase) August 30, 2026 04:43
@digizeph
digizeph merged commit 59fd0d1 into main Aug 30, 2026
1 check passed
@digizeph
digizeph deleted the fix/backup-retry-loop branch August 30, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Periodic S3 backup: failed uploads retry in full with no backoff, and the backup interval never advances on failure

1 participant