fix: deduplicate PostgreSQL latest-file upserts

[digizeph]

Summary

• retain only the newest item per PostgreSQL (collector_id, data_type) latest-file key before issuing the batched upsert
• prevent PostgreSQL from rejecting crawler batches that contain multiple observations for the same latest-file key

Failure mode

The PostgreSQL backend used one INSERT ... ON CONFLICT DO UPDATE statement for each 1,000-item batch. A crawl can contain multiple rows with the same (collector_id, data_type), and PostgreSQL rejects an upsert that would update its target row twice in one statement.

Validation

• cargo build --verbose
• cargo build --features cli --verbose
• cargo test --no-default-features --verbose
• cargo clippy --all-features -- -D warnings
• live Terrier PostgreSQL-backed Broker updater: initial crawl completed with 8,995 inserts; API health returned OK

[Copilot]

Pull request overview

Deduplicates PostgreSQL latest-file upserts to prevent conflicting updates within a batch.

Changes:

• Groups latest files by collector and data type.
• Retains only the newest observation before batched upserts.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.