Skip to content

Fix MS-ZIP blocks that end with short codes or copy from 32 KiB back - #7

Merged
mnadareski merged 2 commits into
SabreTools:mainfrom
gmipf:fix-mszip-decoding
Oct 26, 2026
Merged

mnadareski merged 2 commits into
SabreTools:mainfrom
gmipf:fix-mszip-decoding

Conversation

@gmipf

@gmipf gmipf commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Fixes #1.

Two problems in the inflater made MS-ZIP blocks decompress wrong. Each commit fixes one and adds a test that fails without it.

Short codes at the end of the input. InflateCodes waited for as many bits as its lookup table uses before it decoded a code. An MS-ZIP block is a deflate stream on its own, so when a block ended with fewer bits than that, its last match was not decoded and the block came out short. This is what #1 reports: block 43495 of AgainstRome1.cab decodes to 32638 bytes and block 7 of folder 177 of Data2.cab to 32743 bytes, the same numbers as in the issue. Since 4dc1378 the missing end is filled with zeros, which only gives the right data when the block ends in zeros. The short blocks of AgainstRome1.cab and Group2.cab do, so main extracts those two correctly. The one in Data2.cab does not, so _86261BAE7A6B4C45A5D691D3320D5993 from the issue still comes out wrong on main. When no more input is there, the code is now looked up with the bits that are left and used if it is not longer than those.

A full 32 KiB history. SetDictionary kept only the last 32767 bytes of the previous block, so a match at the start of a block that copies from 32768 bytes back read a byte that was not in the window. None of the three disc cabinets below has such a match. Generated blocks with them decode the same in zlib, cabextract and 7-Zip, and wrong on main.

Measured on Linux with the cabinets from the issue, taken from the dumps of GameStar 07-2008 (Germany) (DVD-XL), Test Drive Unlimited (Germany) (En,Fr,De,Es,It) (Rerelease) and GameStar 09-2008 (Germany) (DVD-XL):

  • Every MS-ZIP block of AgainstRome1.cab, the Data.cab set and Group2.cab (45124, 230187 and 14497 blocks) now decodes to the same bytes as zlib. On main, 6 of them come out short.
  • ExtractionTool from SabreTools.Serialization main extracts the same files as cabextract with this branch. On main, only _86261BAE7A6B4C45A5D691D3320D5993 differs.

The split block in the spanned Data.cab set, which threw the exception in the issue, is already joined correctly on SabreTools.Serialization main.

Prepared with AI assistance (Claude Opus 5.5) and reviewed before submission.

gmipf and others added 2 commits October 20, 2026 12:13
InflateCodes waited for as many bits as its lookup table uses before
it decoded a literal, length or distance code. When the input ended
with fewer bits than that, the last codes were not decoded, even though
they were complete. An MS-ZIP block is a deflate stream on its own, so
its last match was lost and the block came out short (SabreTools#1).

When no more input is there, the code is now looked up with the bits
that are left and used if it is not longer than those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetDictionary kept only the last 32767 bytes of a 32 KiB dictionary,
because the window needs one free byte between its read and write
positions. MS-ZIP sets the previous 32 KiB block as the dictionary,
and a match at the start of a block can copy from 32768 bytes back,
which then read a byte that was not in the window.

A dictionary that fills the window now starts with the read and write
positions at 0, so the first byte written replaces the oldest byte of
the dictionary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@mnadareski mnadareski left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest that this DEFLATE code isn't originally mine (as is already documented) and MS-ZIP is now the only thing that uses it. So if this fixes MS-ZIP, then it's all good by me. I'd 100% prefer to use zlib, but that's a while off.

@mnadareski
mnadareski merged commit e818a32 into SabreTools:main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MSZIP decompression issues

2 participants