0xj4f · Nov 6, 2026 · Nov 6, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,40 @@ Version numbers are produced at release time from the GitHub Actions
 `MAJOR_VERSION` and `MINOR_VERSION` variables; the patch component
 auto-increments. See [README — Release process](README.md) for details.
 
+## [v1.3.4]
+
+### Added
+- **Baked secrets are mined without `--spa`.** Inline `<script>` blocks (incl. `__NEXT_DATA__` / `__NUXT__` and `--browser` rendered pages) and JS files the crawler fetches are now scanned. Results land in a new top-level `secrets` list, deduplicated and tagged with a `source` (`bundle` / `sourcemap` / `inline` / `js`); `spa.secrets` is unchanged. Toggle: `extract.secrets`.
+- **`--spa` scans source-map original source for secrets**, not just the minified bundle.
+- **More secret patterns:** OpenAI, Anthropic, SendGrid, Twilio, Mailgun, GitHub fine-grained PATs, Google OAuth client secrets, Azure storage `AccountKey`, Stripe test/restricted keys, private-key blocks, JWTs.
+
+### Changed
+- **Generic API-key matching is wider:** any casing of `apiKey` / `api_key` / `api-key` / `x-api-key` / `apikey`, quoted object keys and header calls (`setRequestHeader("X-API-Key", …)`), and values containing `.` `/` `+` `=`. `secret` also matches `clientSecret` / `secret_key`. Obvious placeholders (`YOUR_API_KEY`, `xxxx`, `example`) are skipped.
+- The spider now keeps `application/javascript` response bodies. They are only used for the secret scan; link extraction and the HTML extractors still skip them.
+- The "Baked Secrets" table is shown whenever secrets are found, not only under the SPA section, and gains a Source column.
+- **Rate-limited responses are retried.** Every request to the target retries `429` (and `503` carrying `Retry-After`) up to 3 times, honouring `Retry-After` (capped at 10 s). Pages still rate-limited are left out of the analysis with a warning. Module: `utils/http.py`.
+- `framework_recon.fetch_errors` lists framework probes whose request failed.
+
+### Fixed
+- **A rate limiter's error page was treated as the app.** On a target that answers bursts with `429` (nginx `limit_req`), the 429 page became the landing HTML, so SPA/framework detection flipped from run to run, and crawled 429 pages fed forms, comments and fingerprinting.
+- **The root page was crawled twice** (`http://host` and `http://host/`), duplicating findings and inflating fingerprint confidence. URLs with an empty path now normalise to `/`.
+- **Reports differed between identical scans.** Seeds were de-duplicated through a `set` and pages analysed in fetch-completion order; both are now ordered, so "first wins" fields and list order are stable.
+- **The crawler followed redirects off the target.** A redirect to another host is now recorded but not followed, so that host's forms, emails and links are no longer reported as the target's. Links on a redirected page now resolve against the page it landed on.
+- **Pages beyond `max_depth` blocked later discovery.** A link first seen too deep was marked as queued and then dropped, so it was never crawled even when a shallower page linked it.
+- **`max_pages` was overshot** by up to the concurrency level.
+- **Sitemap parsing could recurse without end** when a sitemap index child failed (the child fell back to `/sitemap.xml`, the index itself). Children are fetched once, in scope only, nested at most 3 levels; `Sitemap:` URLs from robots.txt are now used.
+- **robots.txt pattern rules were crawled literally** (`/admin/*.php$`); only the literal prefix is seeded now.
+- **A 401/403 wall flooded "exists but protected".** Common-path, fingerprint and framework probes now drop 401/403 responses that match the answer to a random bogus path. The common-path probe also reports `303` redirects.
+- **Failed framework probes were counted as findings** (e.g. a timeout on `/.env` printed as `[CRITICAL] [0] /.env` and counted in the summary).
+- **Session-cookie false positives.** The session hint was matched against the whole `Set-Cookie` line (`theme=dark; Path=/authors` → "auth"), and flags by substring. Now the cookie name and the parsed attributes are used.
+- **Binary interesting files and header-only evidence were skipped.** A `/backup.sql` served as `application/octet-stream`, or an empty `401` with `WWW-Authenticate: Basic`, had no text body and was dropped before analysis.
+- **Source-map URL with a query string** became `app.js?v=3.map`; it is now `app.js.map?v=3`.
+- **Parameterised SPA routes** (`/users/:id`, `/posts/[slug]`, `/docs/*`) were crawled as literal URLs; they stay in `spa.routes` but are no longer seeded.
+
+### Performance
+- Phase 3 parses each page's HTML once and shares it across the form, comment and script extractors (it was parsed 3–4 times).
+- Fingerprint signature regexes are compiled once instead of per page.
+
 ## [1.3.3] - 2026-10-19
 
 ### Added

diff --git a/README.md b/README.md
@@ -64,6 +64,7 @@ Single-pass recon against **one target domain**. Signal-only — it maps and det
 - 🤖 **Parses** robots.txt, sitemap.xml, security.txt
 - 🔐 **Authenticated crawling** — bearer token (`--token`) or any header (`-H`), origin-scoped and redacted
 - 🧬 **Framework CVE signal-recon** (`--framework-recon`) — Laravel, Django, Flask, Spring Boot, Next.js, Voyager, Innoshop; maps exposures to CVEs with a "verify manually" disposition (never claims "vulnerable")
+- 🔑 **Baked-secret mining** — API keys, tokens, JWTs and private keys in inline scripts, crawled JS, `--spa` bundles and source maps (redacted)
 - 🪲 **Stack-trace mining** — flags debug-mode error pages, leaked filesystem paths, framework versions, secrets/DB-URIs
 - 📦 **SPA bundle mining** (`--spa`) — follows every Vite/ES-module chunk, source maps, client-side routes (incl. route-constant objects), RPC methods (WebSocket RPC clients such as Hydra), baked secrets for React/Vue/Angular/Next/Nuxt; concrete API paths it mines are fetched (GET) during the crawl
 - 🌐 **Browser mode** (`--browser`, optional extra) — renders pages in headless Chromium: rendered forms, client-side redirects, live XHR/fetch/WebSocket calls, third-party hosts; logged-in SPAs via `--browser-storage`. Passive: never clicks, blocks non-GET
@@ -415,6 +416,7 @@ extract:
   comments: true
   emails: true
   js_endpoints: true
+  secrets: true
 
 fingerprint:
   enabled: true

diff --git a/config.yaml b/config.yaml
@@ -26,6 +26,7 @@ extract:
   comments: true                      # HTML/JS comments
   emails: true
   js_endpoints: true                  # Extract URLs from JavaScript
+  secrets: true                       # Baked secrets in inline + crawled JS (no --spa needed)
   metadata: true                      # Meta tags, headers
 
 # Fingerprinting

diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
@@ -79,7 +79,10 @@ flowchart LR
     spider --> auth
     spider --> validator["utils/validator.py<br/>scope/normalize"]
     spider --> parser
-    recon --> baseline["utils/baseline.py<br/>404 + redirect baseline"]
+    recon --> baseline["utils/baseline.py<br/>404 / redirect / 401-403 wall baseline"]
+    crawler --> http["utils/http.py<br/>429-retrying client · Set-Cookie parsing"]
+    spider --> http
+    recon --> http
     fingerprint --> baseline
     framework --> baseline
     spa --> extractor
@@ -135,6 +138,8 @@ Every top-level key in the saved JSON, and the phase that fills it:
 |---|---|---|
 | **404 content baseline** | catch-all routers (SPA dev servers) that 200 every path → phantom path hits | `utils/baseline.py` |
 | **Redirect baseline** | apps that 302 every path to a login (GitLab) → flood of false probe "findings" | `Baseline.is_catchall_redirect` |
+| **401/403 wall baseline** | a WAF / auth wall answering every path with the same 401/403 → "exists but protected" for every probe | `Baseline.is_noise` (shared by all three path probers) |
+| **Rate-limit retry** | a `429` error page used as the landing HTML, a probe result or a crawled page | `utils/http.py` `RetryTransport` |
 | **Content-confirm gate** | a bare 200 at `/console` / `/admin/` masquerading as the real product | `framework_recon` `confirm_any` |
 | **Conflict resolution** | two incompatible techs both firing (Laravel vs Rails on shared `<meta csrf-token>`) | `fingerprint.resolve_conflicts` |
 | **Product-specific signatures** | generic paths/headers (`/admin/`, `x-powered-by: PHP`) cross-tagging frameworks | `signatures/fingerprints.yaml` |

diff --git a/docs/features.md b/docs/features.md
@@ -12,10 +12,12 @@ Complete feature reference for **shatterpoint**.
 |---------|-------------|
 | **Async I/O** | Built on `httpx` + `asyncio` for non-blocking concurrent requests |
 | **Concurrency control** | Configurable semaphore (default: 15 concurrent requests) |
-| **Scope enforcement** | Strict single-domain — never leaves the target `netloc` |
-| **Redirect tracking** | Manual redirect handling up to N hops (default: 3), logs the full chain |
+| **Scope enforcement** | Strict single-domain — never leaves the target `netloc`, including on redirects: a hop to another host is recorded in the chain but not followed |
+| **Redirect tracking** | Manual redirect handling up to N hops (default: 3), logs the full chain; links on a redirected page resolve against where it landed |
 | **Depth limiting** | Configurable max crawl depth from seed URL |
-| **Page cap** | Safety limit on total pages crawled (default: 500) |
+| **Page cap** | Safety limit on total pages crawled (default: 500), exact even under concurrency |
+| **Rate-limit tolerance** | `429` (and `503` with `Retry-After`) is retried up to 3× honouring `Retry-After` (capped at 10 s). Pages still rate-limited are left out of the analysis with a warning, so a limiter's error page is never reported as app content |
+| **Stable output** | URLs are normalised (`http://h` = `http://h/`) and pages are analysed in URL order, so repeat scans of an unchanged target give the same report |
 | **Polite delay** | Configurable inter-request delay (default: 0.1s) |
 | **Self-signed certs** | TLS verification disabled — works with OSCP lab targets |
 | **Custom User-Agent** | Defaults to Firefox on Linux, fully configurable |
@@ -59,7 +61,13 @@ Complete feature reference for **shatterpoint**.
 
 ### Scripts
 - External script sources with integrity hashes
-- Inline script content (fed to JS endpoint extractor)
+- Inline script content (fed to JS endpoint extractor and secret scanner)
+
+### Baked Secrets
+- Curated regexes run over inline `<script>` blocks and every JS file the crawler fetches (no `--spa` needed), plus `--spa` bundles and source-map original source
+- Provider keys: AWS, Google/Firebase, Stripe (live, test, restricted), Slack, GitHub (classic + fine-grained), OpenAI, Anthropic, SendGrid, Twilio, Mailgun, Google OAuth client secret, Azure storage `AccountKey`, private-key blocks, JWTs
+- Generic assignments: `apiKey` / `api_key` / `x-api-key` / `apikey` (any casing, quoted keys, header calls such as `setRequestHeader("X-API-Key", …)`), `secret` / `clientSecret` / `secret_key`, `token`; obvious placeholders (`YOUR_API_KEY`, `xxxx`, `example`) are skipped
+- Values are redacted to `first4…last4` before they reach the report
 
 ### Meta Tags
 - All `<meta>` attributes (name, property, http-equiv, content, charset)
@@ -104,11 +112,11 @@ Custom signatures can be added to `signatures/fingerprints.yaml`.
 ### robots.txt
 - Fetches and parses `robots.txt`
 - Extracts `Disallow`, `Allow`, and `Sitemap` directives
-- Disallowed paths added as crawl seeds
+- Disallowed paths added as crawl seeds (pattern rules are cut at the first `*` / `$`, so `/admin/*.php$` seeds `/admin/`)
 
 ### sitemap.xml
-- Parses `sitemap.xml`, `sitemap_index.xml`, `.gz` variants
-- Handles XML namespaces and sitemap indexes (recursive)
+- Tries the `Sitemap:` URLs from robots.txt first, then `sitemap.xml`, `sitemap_index.xml`, `.gz` variants
+- Handles XML namespaces and sitemap indexes (in-scope children only, each fetched once, nesting capped at 3 levels)
 - Regex fallback for malformed XML
 - Discovered URLs added as crawl seeds
 
@@ -125,13 +133,15 @@ Custom signatures can be added to `signatures/fingerprints.yaml`.
 - **CMS files**: `/xmlrpc.php`, `/CHANGELOG.txt`, `/README.txt`
 - **Dev tools**: `/actuator`, `/console`, `/_debug`, `/_profiler`
 
-Responses categorized as: **200 OK**, **Redirect** (with location), **Protected** (401/403).
+Responses categorized as: **200 OK**, **Redirect** (301/302/303/307/308, with location), **Protected** (401/403).
+Anything that matches how the target answers a random bogus path (soft-404 200, "everything → login"
+redirect, a 401/403 wall in front of every path) is dropped as noise.
 
 ### Authentication Detection
 Passive identification only — no attacks:
 - **HTTP Auth**: Basic, Digest, NTLM, Kerberos/Negotiate, Bearer
 - **Login forms**: Password field detection with CSRF token check
-- **Session cookies**: Name pattern matching with flag analysis (HttpOnly, Secure, SameSite)
+- **Session cookies**: Cookie-name pattern matching (`session`, `sid`, `auth`, `token`, …) with flags read from the parsed attributes (HttpOnly, Secure, SameSite)
 
 Security headers (CSP, HSTS, X-Content-Type-Options, X-XSS-Protection, Permissions-Policy,
 X-Frame-Options, Referrer-Policy, COOP/COEP/CORP) are reported **separately** from auth

diff --git a/docs/schema.md b/docs/schema.md
@@ -26,6 +26,7 @@ The output report is a single JSON object saved to `./output/recon_{domain}_{tim
   "security_txt": { ... },
   "common_paths": [ ... ],
   "all_urls": [ ... ],
+  "secrets": [ ... ],
   "attack_surface": { ... }
 }
 ```
@@ -227,6 +228,25 @@ For protected paths:
 }
 ```
 
+### `secrets[]`
+Baked secrets found in JS: `--spa` bundles and source maps, plus inline `<script>` blocks and JS files the crawler fetched (always on; `extract.secrets: false` disables the crawl-side scan). Deduplicated on `type` + `value_redacted`; raw values are never stored. `spa.secrets` still holds the `--spa` subset.
+```json
+{
+  "type": "GENERIC_API_KEY",
+  "value_redacted": "sk3y…9f2a",
+  "bundle": "main-abc123.js",
+  "url": "http://10.10.10.1/assets/main-abc123.js",
+  "source": "bundle"
+}
+```
+`source` is one of `bundle` (`--spa` bundle), `sourcemap` (original source from a `.map`), `inline` (inline `<script>`, incl. `__NEXT_DATA__` / `__NUXT__` and `--browser` rendered pages), `js` (JS file fetched by the crawler).
+
+### `framework_recon.fetch_errors[]`
+Framework probes whose request failed (timeout, connection error). They prove nothing either way, so they are listed here instead of in `framework_recon.probes` and are not counted in `framework_recon.summary`.
+```json
+{ "path": "/.env", "error": "timeout" }
+```
+
 ### `attack_surface`
 ```json
 {