Skip to content

perf(stats): serve 5chan stats on their own and keep threads only for popular boards - #47

Merged
tomcasaburi merged 1 commit into
masterfrom
codex/feature/5chan-stats-endpoint
Oct 21, 2026
Merged

tomcasaburi merged 1 commit into
masterfrom
codex/feature/5chan-stats-endpoint

Conversation

@tomcasaburi

@tomcasaburi tomcasaburi commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

The 5chan home page downloads the whole boards document (180 KB gzipped) before either of its boxes can render. On a slow path to Cloudflare that download alone took 1.3–5 s, and past 5 s 5chan gives up and loads the boards over P2P instead (9–15 s).

  • GET /api/v1/5chan/stats: the same boards without threads, about 8 KB gzipped, for the Stats box.
  • GET /api/v1/5chan/boards: threads only for the 20 worksafe and 20 NSFW boards whose threads score highest; every other board keeps its stats with threads: []. 5chan picks from the threads of the 16 most popular boards it shows, and its content filter shows worksafe boards (the default), NSFW boards or both, so each group's top 20 covers every choice. On the live data this is about 125 KB gzipped (33 of 63 boards keep a thread).
  • The monitor now keeps each 5chan directory's features.safeForWork and writes it per board (null when undeclared), so the server can rank the two groups apart. The monitor's file is unchanged otherwise; it still holds every board's thread, which it reads back after a restart.

Both endpoints share the boards caching (max-age=60, Cloudflare stale-while-revalidate=600), and the existing Cloudflare cache rule already covers /api/v1/5chan/.

5chan's client change lands in bitsocialnet/5chan#1265, which reads /stats for the Stats box. This needs to deploy first.

Verification: yarn workspace @bitsocial/stats-web test (29 pass), monitor node --test (33 pass), yarn lint (no new warnings), yarn build:stats-web.

Summary by CodeRabbit

  • New Features
    • Added the GET /api/v1/5chan/stats endpoint, providing board statistics without thread data.
    • Added optional work-safe classification to 5chan board data.
    • Limited retained thread details to the 20 most popular work-safe and 20 most popular NSFW boards.
    • Preserved statistics for other boards while omitting their thread lists.
  • Documentation
    • Updated API documentation with the new endpoint, fields, response behavior, caching, and payload details.

… popular boards

The 5chan home page's Stats box can read GET /api/v1/5chan/stats, the boards without their threads (8 KB gzipped instead of 180 KB). The boards document keeps threads only for the 20 worksafe and 20 NSFW boards whose threads score highest, which still covers the 16-board pool the Popular Threads box picks from under each content filter, and drops to about 125 KB gzipped. The monitor now records each board's safeForWork from its directory so the server can rank the two groups apart.
@vercel

vercel Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
bitsocial-chain Ready Ready Preview Oct 21, 2026 6:22pm UTC
bitsocial-web Ready Ready Preview Oct 21, 2026 6:22pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
bitsocial-docs Skipped Skipped Oct 21, 2026 6:22pm UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T18:23:59.489915Z 80d9f9e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 178b6ee1-00b5-42fe-8019-b4dadd5e6ce3
📥 Commits

Reviewing files that changed from the base of the PR and between 7ed0e69 and 80d9f9e.

📒 Files selected for processing: 11
  • stats/monitor/README.md
  • stats/monitor/config.js
  • stats/monitor/lib/client-sources.js
  • stats/monitor/lib/client-sources.test.js
  • stats/monitor/lib/five-chan-boards.js
  • stats/monitor/lib/five-chan-boards.test.js
  • stats/web/README.md
  • stats/web/server/five-chan-boards.mjs
  • stats/web/server/five-chan-boards.test.mjs
  • stats/web/server/server.mjs
  • stats/web/server/server.test.mjs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The monitor now includes board safety metadata in 5chan board documents. The web server generates separate boards and stats snapshots, filters retained thread data, and serves both snapshots through API routes.

Changes: 5chan API

Layer / File(s) Summary
Normalize and serialize board safety metadata
stats/monitor/lib/client-sources.js, stats/monitor/lib/client-sources.test.js, stats/monitor/lib/five-chan-boards.js, stats/monitor/lib/five-chan-boards.test.js, stats/monitor/README.md, stats/monitor/config.js
The monitor preserves boolean safeForWork values and serializes missing values as null. Tests cover boolean, missing, and non-boolean inputs. Monitor endpoint references now include /api/v1/5chan/stats.
Build boards and stats snapshots
stats/web/server/five-chan-boards.mjs, stats/web/server/five-chan-boards.test.mjs, stats/web/README.md
The web server generates a threadless stats snapshot and limits retained threads to the top 20 worksafe and NSFW boards by thread popularity score. Tests cover ranking and thread removal. The README documents both endpoints and the filtering behavior.
Route boards and stats requests
stats/web/server/server.mjs, stats/web/server/server.test.mjs
The server maps both API paths to their snapshot keys and applies shared cache and public-read headers. Tests cover the stats response and reject extra path segments.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant handleApi
  participant handleFiveChanDocument
  participant SnapshotStore
  Client->>handleApi: Request boards or stats path
  handleApi->>handleFiveChanDocument: Dispatch mapped snapshot key
  handleFiveChanDocument->>SnapshotStore: Retrieve snapshot
  SnapshotStore->>handleFiveChanDocument: Return snapshot
  handleFiveChanDocument->>Client: Return document and headers
Loading
Merge Risk: ⚪ Minimal · up to 80d9f

The new stats endpoint and reduced boards response are ready for normal merge checks; no unresolved issue was identified.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly summarizes the two primary changes: serving 5chan statistics separately and retaining threads only for popular boards.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 9…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 9 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude

claude Bot commented Oct 21, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@tomcasaburi
tomcasaburi merged commit f3cc314 into master Oct 21, 2026
7 checks passed
@tomcasaburi
tomcasaburi deleted the codex/feature/5chan-stats-endpoint branch October 21, 2026 20:18

This branch was successfully deployed

2 active deployments
Preview – bitsocial-web — 80d9f9e7 Deployed Oct 21, 2026 by vercel[bot]
Preview – bitsocial-chain — 80d9f9e7 Deployed Oct 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant