chore(seo): serve a real robots.txt and noindex the app shell - #833
Merged
Merged
Conversation
app.alphaday.com has no robots.txt at any path — /robots.txt returns the SPA
shell as text/html with HTTP 200, and so does /sitemap.xml and every other
URL. All of them are the same 3,252-byte contentless document, which makes
each route a duplicate of the others and of the landing page for the same
board at alphaday.com/{slug}.
Two changes:
- public/robots.txt, so the host actually has one. It allows crawling
deliberately. Disallow blocks crawling rather than indexing, so a blocked
crawler never reads the noindex below, and Google still lists blocked URLs
it finds linked elsewhere as bare entries with no snippet. Crawling has to
stay open for the directive to be readable. Once Search Console shows these
URLs dropped out, a Disallow can be added to save crawl budget.
- A static `noindex, follow` in index.html. Static rather than injected after
hydration because the model crawlers this most matters for do not execute
JavaScript, so a post-render directive would be invisible to them. `follow`
keeps links back to alphaday.com as a signal. No canonical accompanies it:
noindex plus canonical is a conflicting instruction, and the obvious target
does not exist for every board that has a dashboard here.
Verified in the build output: dist/robots.txt is emitted and the meta survives
into dist/index.html. The PWA workbox config already excludes index.html from
precaching, so no stale service worker holds the old shell.
Closes the last outstanding Phase 0 item in the SEO plan (home repo,
docs/seo-strategy.md §5.9).
elcharitas
approved these changes
Sep 4, 2026
Xavier-Charles
marked this pull request as draft
September 9, 2026 14:23
Xavier-Charles
marked this pull request as ready for review
September 13, 2026 16:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
app.alphaday.com has no robots.txt at any path — /robots.txt returns the SPA shell as text/html with HTTP 200, and so does /sitemap.xml and every other URL. All of them are the same 3,252-byte contentless document, which makes each route a duplicate of the others and of the landing page for the same board at alphaday.com/{slug}.
Two changes:
public/robots.txt, so the host actually has one. It allows crawling deliberately. Disallow blocks crawling rather than indexing, so a blocked crawler never reads the noindex below, and Google still lists blocked URLs it finds linked elsewhere as bare entries with no snippet. Crawling has to stay open for the directive to be readable. Once Search Console shows these URLs dropped out, a Disallow can be added to save crawl budget.
A static
noindex, followin index.html. Static rather than injected after hydration because the model crawlers this most matters for do not execute JavaScript, so a post-render directive would be invisible to them.followkeeps links back to alphaday.com as a signal. No canonical accompanies it: noindex plus canonical is a conflicting instruction, and the obvious target does not exist for every board that has a dashboard here.Verified in the build output: dist/robots.txt is emitted and the meta survives into dist/index.html. The PWA workbox config already excludes index.html from precaching, so no stale service worker holds the old shell.
Closes the last outstanding Phase 0 item in the SEO plan (home repo, docs/seo-strategy.md §5.9).