Skip to content

Remove unsupported brotli from Accept-Encoding header - #124

Open
nuxnik wants to merge 1 commit into
opsdisk:masterfrom
nuxnik:master
Open

nuxnik wants to merge 1 commit into
opsdisk:masterfrom
nuxnik:master

Conversation

@nuxnik

@nuxnik nuxnik commented Oct 7, 2026 •

Copy link
Copy Markdown

Problem

./ghdb_scraper.py failed with JSONDecodeError: Expecting value: line 1 column 1 (char 0) despite the request returning HTTP 200.

Root cause

The request headers set Accept-Encoding: deflate, gzip, br. Exploit-DB honored br (brotli) and compressed the JSON response with it. Python has no built-in brotli support (the optional brotli package is not installed), so the response body was never decompressed, and response.json() received raw compressed bytes.

Change

ghdb_scraper.py:62 — Accept-Encoding changed from deflate, gzip, br to deflate, gzip. Both formats are decompressed natively by Python, so the server always returns a body the scraper can parse. No new dependencies required.

Verification

  • ./ghdb_scraper.py → Total Google dorks retrieved: 7944
  • ./ghdb_scraper.py -i -s -j → 14 category files, all_google_dorks.json, and all_google_dorks.txt written to dorks/

@opsdisk

opsdisk commented Oct 9, 2026

Copy link
Copy Markdown
Owner

@nuxnik Appreciate the PR! Give me a few days to dig deeper into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants