Skip to content

chore(seo): serve a real robots.txt and noindex the app shell - #833

Merged
Xavier-Charles merged 1 commit into
devfrom
chore/seo-app-shell-noindex
Sep 13, 2026
Merged

Xavier-Charles merged 1 commit into
devfrom
chore/seo-app-shell-noindex

Conversation

@Xavier-Charles

Copy link
Copy Markdown
Contributor

app.alphaday.com has no robots.txt at any path — /robots.txt returns the SPA shell as text/html with HTTP 200, and so does /sitemap.xml and every other URL. All of them are the same 3,252-byte contentless document, which makes each route a duplicate of the others and of the landing page for the same board at alphaday.com/{slug}.

Two changes:

  • public/robots.txt, so the host actually has one. It allows crawling deliberately. Disallow blocks crawling rather than indexing, so a blocked crawler never reads the noindex below, and Google still lists blocked URLs it finds linked elsewhere as bare entries with no snippet. Crawling has to stay open for the directive to be readable. Once Search Console shows these URLs dropped out, a Disallow can be added to save crawl budget.

  • A static noindex, follow in index.html. Static rather than injected after hydration because the model crawlers this most matters for do not execute JavaScript, so a post-render directive would be invisible to them. follow keeps links back to alphaday.com as a signal. No canonical accompanies it: noindex plus canonical is a conflicting instruction, and the obvious target does not exist for every board that has a dashboard here.

Verified in the build output: dist/robots.txt is emitted and the meta survives into dist/index.html. The PWA workbox config already excludes index.html from precaching, so no stale service worker holds the old shell.

Closes the last outstanding Phase 0 item in the SEO plan (home repo, docs/seo-strategy.md §5.9).

app.alphaday.com has no robots.txt at any path — /robots.txt returns the SPA
shell as text/html with HTTP 200, and so does /sitemap.xml and every other
URL. All of them are the same 3,252-byte contentless document, which makes
each route a duplicate of the others and of the landing page for the same
board at alphaday.com/{slug}.

Two changes:

- public/robots.txt, so the host actually has one. It allows crawling
  deliberately. Disallow blocks crawling rather than indexing, so a blocked
  crawler never reads the noindex below, and Google still lists blocked URLs
  it finds linked elsewhere as bare entries with no snippet. Crawling has to
  stay open for the directive to be readable. Once Search Console shows these
  URLs dropped out, a Disallow can be added to save crawl budget.

- A static `noindex, follow` in index.html. Static rather than injected after
  hydration because the model crawlers this most matters for do not execute
  JavaScript, so a post-render directive would be invisible to them. `follow`
  keeps links back to alphaday.com as a signal. No canonical accompanies it:
  noindex plus canonical is a conflicting instruction, and the obvious target
  does not exist for every board that has a dashboard here.

Verified in the build output: dist/robots.txt is emitted and the meta survives
into dist/index.html. The PWA workbox config already excludes index.html from
precaching, so no stale service worker holds the old shell.

Closes the last outstanding Phase 0 item in the SEO plan (home repo,
docs/seo-strategy.md §5.9).
@Xavier-Charles
Xavier-Charles marked this pull request as draft September 9, 2026 14:23
@Xavier-Charles
Xavier-Charles marked this pull request as ready for review September 13, 2026 16:01
@Xavier-Charles
Xavier-Charles merged commit bbe6a60 into dev Sep 13, 2026
1 check passed
@Xavier-Charles
Xavier-Charles deleted the chore/seo-app-shell-noindex branch September 13, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants