Skip to content

Fix out-of-range Python timestamps with pandas 3.1 - #19248

Merged
jt2594838 merged 3 commits into
apache:masterfrom
hongzhi-gao:fix/python-dataframe-date-range
Nov 13, 2026
Merged

jt2594838 merged 3 commits into
apache:masterfrom
hongzhi-gao:fix/python-dataframe-date-range

Conversation

@hongzhi-gao

@hongzhi-gao hongzhi-gao commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Python client CI with pandas 3.1.1 fails when reading extreme TIMESTAMP values in both tree and table models. For example, 9223372036854775807 milliseconds raises OutOfBoundsDatetime in convert_to_timestamp(), and the fallback repeats the same failing numeric constructor. This affects Field formatting and query results converted to a DataFrame.

pandas 3.1 defaults numeric datetime input to microsecond resolution, so rescaling extreme millisecond values overflows.

Change

Construct fallback timestamps from a NumPy datetime64 scalar at the original precision, preserving the exact integer epoch value. Convert timezones from UTC and retain the existing naive-timestamp fallback when timezone conversion cannot represent the date. The normal conversion path is unchanged.

Before: the maximum millisecond value raises OutOfBoundsDatetime. After: it remains exactly 9223372036854775807 milliseconds and can be formatted as 292278994-08-17T07:12:55.807 when the naive fallback is required.

The diff only changes rpc_utils.py; no dependency pinning or server changes.

Regression coverage

A compact parameterized test covers -9223372036854775807 and 9223372036854775807 milliseconds with UTC and Asia/Shanghai timezones. It asserts the exact integer epoch value after conversion, catching overflow and timezone-induced shifts. The minimum int64 value itself is reserved for NaT.

Validation

  • On pandas 3.1.1, the four new cases fail with the original helper and pass with the fix.
  • python -m pytest tests/unit -q: 54 passed on pandas 3.1.1 (Python 3.11.14).
  • python -m pytest tests/unit/test_rpc_utils.py -q: 4 passed on pandas 2.2.3 (Python 3.12.3).
  • Black formatting and git diff --check: passed.
  • Earlier validation of the same production fix: the existing test_insert_use_tablet integration test and a full Python suite with the earlier, larger regression set passed (152 tests). The full integration suite was not rerun for this test-only update.

Local integration validation reused the existing Apache IoTDB iotdb:dev Docker image. Local Python versions differ from CI's Python 3.15.2; that runtime is covered by remote CI.

ts = pd.Timestamp(time, unit=precision).tz_localize(timezone)
except NotImplementedError:
return pd.Timestamp(time, unit=precision)
ts = (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep focused regression coverage in the final diff

The second commit removes tests/unit/test_rpc_utils.py, so this overflow path has no regression test in the PR. Please retain at least a compact test for extreme millisecond values under pandas 3.1, including a non-UTC timezone and signed-boundary values; the current Python CI can pass without exercising the inputs that triggered this failure.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@jt2594838
jt2594838 merged commit b92ca10 into apache:master Nov 13, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants