Repository navigation
Diacritics and header encoding #85
Description
Activity
- pinned this issue
on Jan 29, 2026 This will need to be integrated into the dataset upgrade fix since the internal API use run directly from datasetd.
This is related to issue 80 too.
This might be more complicated than the URL or header being ASCII and not UTF-8. Browsers send the UTF-8 correctly. That is because host names in IPv6 if not IPv4 can be UTF-8 (I get RSS feeds from host names that are in Japanese characters). When I was doing RegExp to create a validator function for clpid and clgid I ran into some really odd behaviors with some of our clpid and they might be the same ones that present a problem on saving the data. Needs much more investigation.
Once we have a spec for clpid and clgid I can normalize the problematic identifiers and this will solve our problem. It's not the header ASCII encoding issue I thought it was.
- unpinned this issue
on Apr 13, 2026 - pinned this issue
on Sep 25, 2026 Root cause confirmed, not the header-charset-declaration issue from the original comments: it is that HTTP header field values are restricted to the ByteString/Latin-1 range (RFC 7230 3.2.6), and every save handler (people, groups, journals, subjects, thesis_option, funders, doi_prefix) was putting the raw clpid/clgid/etc. straight into the redirect
Locationheader.Two failure modes were present:
- An identifier with any code point outside Latin-1 (0-255) made the
Responseconstructor throw outright when the handler tried to build the redirect -- a hard save failure. - An identifier whose diacritics happen to sit inside Latin-1 (most common Western European accents) did not throw, but went out as a raw byte that is not valid UTF-8 -- silently wrong on the wire, which is why this was not showing up as a reproducible complaint recently.
Fixed in be4c33f by
encodeURIComponent()-ing the identifier before it becomes a header value, across all seven files that build a redirect Location. Verified this round-trips correctly back through the existingpathIdentifier()/decodeURIdecode path. Covered by a newlocation_header_test.ts. Full suite (184+7 tests) still green.The separate clpid/clgid validator regex bug (rejects
COVID-19,S2I-style identifiers with digits) turned out to be unrelated dead code and is filed as #114.Will ship in the next release.
- An identifier with any code point outside Latin-1 (0-255) made the
- added a commit that references this issue
on Sep 28, 2026 - unpinned this issue
on Sep 28, 2026
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
While the body sent from the browser can be UTF-8 the header has to be Latin1 to preserve the diacritics, RDM ran into this issue. I need to investigate and confirm this is why I cannot save the clpid records that have diacritics in the string.