Skip to content

Assume naive datetimes are UTC in the msgpack and cbor2 converters - #775

Open
onk3sh wants to merge 3 commits into
python-attrs:mainfrom
onk3sh:fix/naive-datetime-timestamps
Open

Assume naive datetimes are UTC in the msgpack and cbor2 converters#775
onk3sh wants to merge 3 commits into
python-attrs:mainfrom
onk3sh:fix/naive-datetime-timestamps

Conversation

@onk3sh

@onk3sh onk3sh commented Sep 1, 2026

Copy link
Copy Markdown

Reported in #774.

The msgpack and cbor2 converters unstructure datetime with datetime.timestamp(). On a naive datetime that reads the value as local time, so the number written to the wire depends on the timezone of the machine doing the unstructuring:

TZ dumps output loads returns
UTC cb41daa362b2000000 2026-08-25 12:30:00+00:00
Asia/Tokyo cb41daa3430e000000 2026-08-25 03:30:00+00:00
America/Toronto cb41daa370c2000000 2026-08-25 16:30:00+00:00

An aware datetime is the control: it yields cb41daa362b2000000 in all three timezones.

The change

The structure hooks already declare the wire contract — datetime.fromtimestamp(v, timezone.utc) — so the unstructure hooks now pin naive datetimes to UTC before converting, and the two ends agree. The date hook sitting immediately below the datetime hook in msgpack.py already did exactly this with time(tzinfo=timezone.utc); this brings the datetime hook in line with it.

The shared helper lives next to validate_datetime in preconf/__init__.py, since both converters need it and both already import from there.

A naive input still comes back aware, because a bare float has nowhere to record awareness. What changes is that the value survives and the payload no longer depends on the host.

Note on the alternative

Raising on naive input instead of assuming UTC is the stricter option, and I would understand preferring it — it refuses to guess rather than guessing well. I went with the UTC assumption because it keeps working code working and because the structure hook had already committed to UTC as the format's meaning. Happy to switch it if you would rather have the exception.

Tests

tests/test_preconf.py never generated naive datetimes: the shared strategy pins timezones=just(timezone.utc) (L155-158), and the strategy that would produce them is passed include_datetimes=False at all seven of its call sites.

The regression test sets TZ explicitly rather than relying on the host, because on a UTC machine — including the ubuntu CI runners — a naive-datetime assertion passes whether or not the bug is present. It restores the previous TZ and calls tzset() again on the way out, and skips where tzset is unavailable.

Verified locally: the two new tests fail on unpatched source (assert 1787628600.0 == 1787661000.0, the 9-hour JST offset) and pass with the change. Full suite is 992 passed, 15 xfailed. ruff check and ruff format --check clean under the pinned lint group.

Onkesh Bansal added 2 commits September 1, 2026 09:13
`datetime.timestamp` interprets a naive datetime as local time, so the
value these two converters wrote to the wire depended on the timezone of
the machine unstructuring it. The same object serialized in two
timezones produced two different payloads, and neither round-tripped
back to the input.

The structure hooks already read timestamps back as UTC, so the
unstructure hooks now pin naive datetimes to UTC before converting,
which makes the two ends agree and the output machine-independent. This
is what the neighbouring `date` hook in the msgpack converter already
did.

The preconf tests never exercised naive datetimes: the shared strategy
pins `timezones=just(timezone.utc)`, and the strategy that would
generate them is disabled at every call site. The regression test
therefore drives the timezone explicitly, since it would otherwise pass
vacuously on a UTC host.
@codspeed-hq

codspeed-hq Bot commented Sep 1, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 64 untouched benchmarks


Comparing onk3sh:fix/naive-datetime-timestamps (739d17c) with main (893351d)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (d4ff793) during the generation of this report, so 893351d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@Tinche

Tinche commented Sep 1, 2026

Copy link
Copy Markdown
Member

This is technically a backwards-compatibility break, but maybe it's warranted. We should probaby document how to restore the old behavior in migrations.md. Also don't forget to add the appropriate :: versionchanged clauses.

…ehavior

Add a migrations.md entry for the msgpack and cbor2 converters, with the
recipe for restoring `datetime.timestamp` on a converter, and versionchanged
clauses in the preconf docs for both converters.

Also replace the branchy timezone helper in the regression test with a
monkeypatch fixture, so the teardown path is exercised and the test file
stays at 100% coverage.
@onk3sh

onk3sh commented Sep 1, 2026

Copy link
Copy Markdown
Author

Pushed 739d17c with the docs. On the compat break:

It is a break, and I think it's warranted because the old behavior isn't a behavior anyone can depend on. The value it produced depended on the TZ of the machine doing the unstructuring, so the same naive datetime serialized to different bytes on different hosts:

>>> converter.dumps(datetime(2026, 8, 25, 12, 30)).hex()   # msgpack, before

TZ=UTC               cb41daa362b2000000
TZ=Asia/Tokyo        cb41daa3430e000000
TZ=America/Toronto   cb41daa370c2000000

And the round-trip inside cattrs was already wrong: the structure hook reads timestamps back as UTC, so structure(unstructure(naive)) only returned the original instant when the host happened to be on UTC. Everywhere else it silently shifted by the local offset. So the break moves users from a host-dependent value to a stable one, and the only people it can affect are those who were relying on local-time semantics — which they can still get with the one-liner in the migration note.

The other two:

  • migrations.md — new ## NEXT section with the restore recipe (converter.register_unstructure_hook(datetime, datetime.timestamp)). I checked it under TZ=Asia/Tokyo: it reproduces the old float and the old wire bytes exactly.
  • versionchanged — added to both the msgpack and cbor2 sections of docs/preconf.md, tagged NEXT, alongside a line stating the new assumption.

The same commit also fixes the coverage failure. The timezone helper in the test had a branch that never ran, which tripped fail-under=100 since tests is in source_pkgs; it's now a monkeypatch fixture with no branch, and tests/test_preconf.py is back at 100%.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants