Skip to content

Fix skipped lines for headerless csvstack stdin after a file - #1371

Open
pauloflp wants to merge 2 commits into
wireservice:masterfrom
pauloflp:fix/headerless-stdin-grouping-20261027
Open

pauloflp wants to merge 2 commits into
wireservice:masterfrom
pauloflp:fix/headerless-stdin-grouping-20261027

Conversation

@pauloflp

@pauloflp pauloflp commented Oct 27, 2026 •

Copy link
Copy Markdown

With -H --skip-lines, csvstack only inspects the first input to determine its column count. If a named file comes before -, stdin has not been inspected, but the second pass still assumes its leading lines were already skipped. For example, cat examples/test_skip_lines.csv | csvstack -H --skip-lines 3 examples/test_skip_lines.csv - incorrectly includes stdin's skipped lines in the output.

Track whether stdin was inspected before deciding to skip its leading lines. The regression checks stdin in both input positions, so lines are skipped exactly once. Includes an Unreleased changelog entry.

Related: #1370 addresses grouping of the buffered first stdin row. This PR is limited to the separate skipped-lines bug and does not include that grouping change.

Validation on Linux / Python 3.12.14:

  • The new regression fails on the unchanged upstream source when stdin follows a file, and passes with this fix in both positions.
  • csvstack suite: 30 passed, 2 skipped, 2 subtests passed.
  • Full suite: 351 passed, 4 skipped, 33 subtests passed; one existing failure in TestCSVStat.test_decimal_format (9748.346 vs 9,748.346), reproduced unchanged on clean upstream base eefcd0ef72747ebf7e9086987ff267291ad877fa.
  • Changed-file flake8/isort and git diff --check pass.

AI-assisted implementation and local verification.

@pauloflp pauloflp changed the title Fix grouping and skipped lines for headerless csvstack stdin Fix skipped lines for headerless csvstack stdin after a file Oct 27, 2026

@ub3dqy ub3dqy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified 92f5b02 on Windows / Python 3.14.2 with 18 real-pipe checks: stdin first/middle/last, headed and headerless input, and 0/1/3 skipped lines, including quoted commas and multiline fields. The base produces incorrect output in four headerless cases with stdin after a file; this head passes all 18. The csvstack suite also passes: 32 tests and 2 subtests.

The full suite has the same 20 existing failures on both revisions, with 335/336 passing tests respectively. No additional failure was introduced in that comparison.

Dima Mak / ub3dqy

@pauloflp

Copy link
Copy Markdown
Author

Dima Mak / ub3dqy

Thanks for running the independent Windows/Python 3.14.2 checks and comparing against the upstream base.

The real-pipe coverage across stdin positions, header modes, and skipped-line counts is especially useful. Great to see all 18 cases passing with this fix and no additional full-suite failures compared with the base.

Appreciate the detailed verification!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants