This project supports qualitative research on how open-source communities treat AI-generated contributions. Researchers copy project AI policies into this repository, then apply a shared codebook to that text: passage-level tags on specific lines, and overall codes for the policy as a whole.
Coding happens on GitHub, not in a custom app. Policy files live under projects/<slug>/. Coders open a pull request and leave review comments on the Files tab, using codebook hashtags (for example #disclosure/required). Maintainer tooling fetches those comments, turns them into JSON, and publishes a static site where anyone can browse, filter, and follow evidence links. The site never writes back to the repo—GitHub remains the editor of record.
Helpers on the site speed up the awkward parts: a tag picker stages the exact hashtag strings for paste into review comments, and a coding view walks passage evidence and emits a paste-ready coding: YAML block for project.yaml. Exact tag strings always come from the codebook (via the picker), not from memory.
Live site: https://www.pyopensci.org/open-research-tagging/
How to add a project: below
How to tag on a PR: codebook/COMMENTING.md
How the pieces fit: docs/ARCHITECTURE.md
- On a new branch (not
main), create a new folder with the name of the project as the folder name. - Copy the
project.yamlfile inprojects/_template/intoprojects/<slug>/. - Copy project files (READMEs, Contributing Guides, AI policy docs, etc.) into the project folder. The site lists them automatically; you can fill descriptions later on the coding view (Copy YAML).
- Open a pull request. This will trigger a GitHub action that re-deploys the site with your project listed. Check the projects page to confirm.
- The project should now appear on the website. You can finish filling out the project metadata (provenance info, descriptions for the files) via the coding view on the site. Note that the site does not save data itself - instead, you'll use the "copy yaml" feature at the bottom of the page to get the updated yaml, and use that to write over the project.yaml file in github. (You'll repeat this process later when you do project-level coding, or whenever you update the files or other metadata.)
There are two types of tagging/coding you'll want to do.
Comment tags. These mark a specific passage (a line or range in a policy file) with codebook hashtags such as #disclosure/required. Use the tag picker to stage and copy the exact strings, then paste them into a Files-tab comment on the PR (details: COMMENTING.md). After you comment, someone needs to run Actions → Update comment data (or wait for the daily run) so the site picks them up. They show up in the comment browser (filter by project) and as evidence in the coding view.
Project-level coding. These are the overall codes for the whole policy (stance, scope, accountability, and so on) stored under coding: in project.yaml — not as hashtags on lines. Open the coding view for your project slug, walk the comment evidence, edit provenance and file descriptions, fill overall codes, then Copy YAML and paste over projects/<slug>/project.yaml on the PR.
| Path | Role |
|---|---|
projects/<slug>/ |
Policy files + project.yaml (provenance + overall coding:) |
projects/_template/ |
Starter for a new project slug |
codebook/ |
codebook.yaml + COMMENTING guide |
site/ |
Static Pages site (index, projects, tags, coding + data/) |
src/open_research_tagging/ |
Pipeline package (fetch, process, tag-check, emit) |
.github/workflows/ |
Tests, tag-check, comment refresh, Pages deploy |
GitHub is the editor. The site is read-only.
| Layer | Where | Helper |
|---|---|---|
| Passage | Hashtags on Files-tab review comments | Tag picker |
| Overall | coding: in project.yaml |
Coding view |
Typical coder path: tag picker → paste on the Files tab → maintainers refresh site data → coding view → Copy YAML over project.yaml. PR checklist: .github/PULL_REQUEST_TEMPLATE.md.
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
playwright install chromium # once; needed for site integration tests
pytestcd site && python3 -m http.server 8000| URL | Page |
|---|---|
| http://localhost:8000/ | Comment browser |
| http://localhost:8000/projects.html | Projects / overall codes |
| http://localhost:8000/tags.html | Tag picker |
| http://localhost:8000/coding.html?project=pyopensci | Coding view |
Committed site/data/*.json is an offline fixture snapshot for local preview. Public Pages gets live JSON built in Actions (not committed to main). Playwright uses a temp copy of site/ filled from raw_comments.fixture.json. Details: ARCHITECTURE → Fixture vs live site JSON.
| Workflow | When | What |
|---|---|---|
Tests (test.yml) |
Push to main, PRs |
pytest (incl. Playwright) |
| Check comment tags | Files-tab comment create/edit; PR open/sync | Soft bot reply on bad tags; PR scan hard-fails malformed only |
| Update comment data | Daily cron + manual | Fetch → process (incl. open-PR projects) → deploy Pages (no git commit of JSON) |
| Deploy Pages | site/ / projects/ / codebook/ (etc.) on main, or manual |
Rebuild live JSON in CI → publish site/ (no git commit) |
While the codebook is still changing, tag-check hard-fails the PR check only on malformed tags (e.g. bare #Design, unparseable #…). Well-formed but unknown ids (e.g. #stance/not-in-codebook-yet) are still reported (and can get a reply / needs_tag_review on the site) but do not fail the status check. Reply jobs use --soft (always green). Prefer Add single comment or Submit the review so replies can attach — see COMMENTING.md.
Production host is the pyOpenSci org repo. One-time setup (already done for production):
- Settings → Pages → Build and deployment → Source: GitHub Actions.
- Push workflows on
main(or merge a PR that adds.github/workflows/). - Run Actions → Update comment data → Run workflow once (or wait for the daily cron) so Pages gets live comments (JSON is built in CI, not committed).
- Smoke-check: comments browser, projects, tag picker, coding view.
Org production URL: https://www.pyopensci.org/open-research-tagging/
Personal/test staging (optional): https://shaunagm.github.io/open-research-tagging/