Skip to content

Warden skips the on-call process #57

Description

@ecv

Summary

Warden, the production patrol skill, lives only in the infra repo and records what it finds in a monthly infra issue that the on-call rotation never reads.

Engineering runs on-call and incidents from its own repo, with a weekly on-call log, an incident issue template with a UTC timeline, and a label for alerts nobody tracks yet. Warden follows none of it, so a patrol's findings stay invisible to the next engineer on shift.

One pass moves warden into the plugin and lines it up with those practices.

Acceptance criteria

  • A session in any Datum repo runs warden from the plugin, the infra copy is gone, the monthly Alerts Seen issue is closed out with no successor, and the infra runbooks and alerting standards that point at it point at the on-call log instead
  • Warden reads recent on-call log comments and open incident issues before its first pass, then keeps one shift comment on the current week's log, edited in place, covering notable events, issues and PRs touched, and handoff
  • An alert with no tracking issue gets an engineering issue with the monitoring label after a duplicate check, and rule hygiene notes go to the rule owner's repo instead of the shift comment
  • A suspected incident produces a draft from the incident template with a UTC timeline, and warden stops for a person to declare it

The infra side of this lands as a companion infra PR in the same pass.

Related to https://github.com/datum-cloud/engineering/blob/main/docs/oncall.md
Related to https://github.com/datum-cloud/engineering/blob/main/docs/incidents.md

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions