[ifnull]

Problem

The add-on was being killed and restarted in a loop on a 2 GB Raspberry Pi, a
few minutes after each start. The add-on log gave no reason at all — normal
startup, db_refreshed, then the log simply begins again:

{"event": "service_starting", ...04:08:16.988}
{"event": "journal_opened",   ...04:08:17.195}
{"tracks": 19962, "event": "journal_warm_loaded", ...04:08:17.386}
{"aircraft": 616410, "event": "db_refreshed", ...04:08:27.973}
[ha-airspace] rendering config from add-on options    <- restart, nothing between

No service_stopping, no service_crashed, no traceback. Host log had the
answer:

kernel: oom-kill:...,global_oom,task=ha-airspace
kernel: Out of memory: Killed process 122326 (ha-airspace) anon-rss:608804kB
supervisor.apps.app: App Airspace exited with non-zero exit code 137

Root cause

DatabaseLoader.refresh_once held up to three full copies of the ~620k-row
merged database simultaneously: parse mictronics into its own dict, copy it
entry-by-entry into merged via dict(entry), then parse adsbexchange into a
third dict while both were still reachable.

Two things made this hard to diagnose:

• Supervisor sets an add-on's container mem_limit to total system RAM, so
  the cgroup limit never trips. ha apps stats showed memory_percent: 23%
  moments before death. The kernel fires a global OOM instead and picks this
  process as the largest target.
• SIGKILL runs none of the service's exit paths, so both an OOM and a segfault
  in a C extension produce byte-identical evidence: nothing.

Changes

fix(databases) — both parsers take an optional accumulator and merge
per-key; the loader parses every enabled source in ascending priority straight
into one dict. Per-key merge (not whole-entry replace) is what preserves the
precedence contract: ADSBex overwrites the fields it has, absent keys still
fall through to Mictronics. Repeated string values (type, model, ownop)
reuse one object per distinct value via a local dedup table; reg is left
alone since tail numbers are ~unique.

feat(cli) — faulthandler enabled at startup, SIGUSR1 for on-demand
all-thread stack dumps. Exists only because diagnosing this was slower than it
should have been. It can't catch SIGKILL, and that asymmetry is the point: with
faulthandler on, no dump narrows the cause to an external kill and sends you
to the host log rather than into the source.

fix(receivers) — a receiver on a dead URL logged one warning per poll,
forever: 22,159 identical lines over 18h in the log that prompted this, every
one a journald write to the SD card this project targets. Failures up to the
unhealthy threshold still log individually (that window diagnoses a blip); past
it the interval doubles to a capped ~1200 (≈1h at a 3s poll), each line
carrying suppressed_since_last. Adds receiver_recovered, which didn't exist
and which the throttle would otherwise have made actively misleading.

chore(scripts) — memwatch.sh, because the obvious one-liners for
watching RSS are both wrong in ways that produce plausible numbers. Details in
the commit message; both were hit while verifying this.

Verification

Peak RSS for a full refresh, measured against the live reference DBs
(616,504 entries):

	RSS
Field OOM kill (old code)	608 MB
Parsers in isolation, new code	253 MB
Full service, new code, in-container equivalent	~282 MB

Full service flat at 314 MB (282 MB excluding the uv run wrapper, which the
container doesn't have) for 2.5 minutes past the DB load — confirming there is
no second leak behind the loader, consistent with the add-on dying 14s after
db_refreshed rather than climbing slowly.

786 tests pass, ruff check / ruff format --check / mypy --strict clean,
coverage 96%.

Notes

• Nothing HA-facing changed — no MQTT payloads, no discovery, no config schema.
• publish.yml fires only on v* tags, so merging doesn't reach any device.
• Not the end state. ~282 MB is dominated by the 616k-entry dict sitting
  resident forever; a SQLite-backed DatabaseStore would take that to near zero
  and is the real fix for a 2 GB Pi. The # TODO(phase-2a-slice-2) disk cache
  is where it belongs. This stops the bleeding.
• Separately noticed, not fixed: discovery unique_ids are hardcoded
  airspace_* and don't incorporate base_topic, so two instances against one
  broker fight over the same HA entities.

🤖 Generated with Claude Code