For: Engineers extending, debugging, or contributing to the GCUID-POC codebase. Not for: End users (see User Guidebook), operators (see SysAdmin Guidebook), or design reviewers (see Architect Guidebook).
This guidebook walks the codebase so you can make changes confidently.
| # | Section | What's covered |
|---|---|---|
| 1.1 | Who this is for | Audience model, prerequisites, what you should already know |
| 1.2 | How the project is organized | Top-level directory map, workspace model, build artifacts |
| 1.3 | Setting up a local dev environment | Dependencies, start-dev.sh/start-dev.bat/start-dev.ps1, port allocation |
| 1.4 | The core packages | @gcuid/core deep dive: source API, connector API, sqlite-source-adapter |
| 1.5 | The API layer | Express routes in staging/api/routes/, shared lib in staging/api/lib/ |
| 1.6 | The Web UI layer | React + Vite, pages and components, the API client |
| 1.7 | The connector system | Plan B B3 architecture, how connectors proxy to sources |
| 1.8 | Library & dependency catalog | Custom modules, third-party deps (better-sqlite3, express, xstate, winston), and why each was chosen |
| 1.9 | Database schema | registry.db and source DBs, db-paths resolver |
| 1.10 | Adding a new connector | Worked example: clone conn-hr to add a new department source |
| 1.11 | Adding a new API endpoint | Worked example: extending /api/hr/employees |
| 1.12 | Testing and debugging | Running smokes, jest, reading logs, the invariants smoke |
| 1.13 | Build, package, ship | npm run build, workspace boundaries |
This guidebook is for engineers who need to modify, debug, or extend the GCUID-POC codebase. That includes:
- Engineers adding a new connector (department source)
- Engineers adding a new API endpoint
- Engineers investigating a bug in the lookup flow
- Engineers porting the local-only stack to a deployment target
- Engineers maintaining this POC and shipping fixes or new features
- TypeScript / Node.js: comfortable with ES modules, async/await, and the Node ecosystem. The codebase targets Node 18+ (uses native
fetch). - SQL: comfortable reading and writing SQLite queries. The runtime uses
better-sqlite3. - React: basic component model, hooks, JSX. The Web UI is Vite + React + TypeScript.
- Express: familiar with middleware-based routing. The API layer is Express.
- Git: the project uses a multi-commit cadence (one phase = one commit) so you can review and bisect at the phase boundary.
- The federal identity domain. The Architect Guidebook explains the GCUID domain model in plain language; you can read it if a particular identity-state transition or attribute-mapping decision looks puzzling.
- The previous proprietary source. This is the public POC fork; the canonical doc for "what's in the original that's not here" is the README's "Public POC release" notice.
Each section is self-contained. If you're adding a connector, you can read §1.7 (the connector system) and §1.9 (adding a new connector) without reading §1.5 (the API layer) end-to-end. The "See also" links at the bottom of each section point at the canonical reference docs for the topic — when this guidebook says "see the architecture doc," follow the link rather than re-deriving the content.
The companion to this guidebook is the Tester Guidebook — once you've made a change, that's where to find the recipes for verifying it (smokes, jest, the invariants smoke).
GCUID-POC is a TypeScript monorepo with a single workspace pattern. The top-level layout is:
E:\GitHub\GCUID-POC\
├── README.md # the project's front door
├── package.json # workspace root (workspaces: ["packages/*"])
├── tsconfig.json # base TS config; per-package overrides via tsconfig.json
├── start-dev.sh # canonical entry point — POSIX
├── start-dev.bat # canonical entry point — Windows cmd
├── start-dev.ps1 # canonical entry point — Windows PowerShell
├── stop-dev.sh # canonical stop — POSIX
├── stop-dev.bat # canonical stop — Windows cmd
├── run-smokes.sh # canonical smoke runner — POSIX
├── run-smokes.bat # canonical smoke runner — Windows cmd
├── run-smokes.ps1 # canonical smoke runner — Windows PowerShell
├── scripts/ # cross-platform dispatchers + smoke tests
│ ├── run-dev.js # dispatch start-dev / stop-dev by OS
│ ├── run-smokes.js # orchestrator for smokes
│ └── smoke-*.ts # the 5 smokes
├── docs/ # project-level reference docs + guidebooks
│ └── guidebook-*.md # the 5 guidebooks (this one, plus 4 siblings)
├── packages/ # npm workspaces
│ └── core/ # the @gcuid/core library
│ ├── src/ # TS source
│ ├── dist/ # build output (consumed by staging/)
│ └── (legacy historical artifacts preserved but not part of the active codebase)
├── staging/ # the live runtime
│ ├── api/ # Express server (port 5182)
│ ├── web-ui/ # Vite + React UI (port 5181)
│ ├── scripts/ # start-* and seed-data launchers
│ ├── config/ # JSON configs (connectors, sources, seed)
│ ├── data/ # runtime SQLite files (registry.db, logs.db, sources/*.db)
│ ├── logs/ # per-service log files (when running)
│ ├── __tests__/ # jest test suites
│ ├── package.json # staging's package.json — npm scripts live here
│ ├── STARTUP-GUIDE.md # canonical operational doc
│ └── USE-CASES.md # the demo personas and end-to-end flows
├── (root scripts: start-dev, stop-dev, run-smokes + their .bat/.ps1 variants)
The root package.json declares workspaces: ["packages/*"] — this is a flat npm workspace. The actual runtime (staging/) is NOT a workspace package; it has its own package.json and is treated as a sibling project. This is intentional:
packages/coreis a library that the staging runtime imports (the stagingtsconfig.jsonpoints atpackages/core/dist/for compiled JS types)staging/is the application — it has its own deps, scripts, and build
The npm scripts you use day-to-day (npm run start-all, npm run stop-all, npm run smoke) are at the staging package level, not the workspace root. The workspace root only runs npm run build, npm run test, npm run clean — all of which delegate to the workspace packages.
packages/core/dist/is regenerated bynpm run buildat the workspace root (which runstsc -p tsconfig.jsonfor each workspace package). The staging runtime imports compiled JS from this directory.staging/api/has no compile step —tsxruns the TS source directly.staging/web-ui/dist/is regenerated by Vite'snpm run build(not run by default; only for production-bundle testing).
| Looking for… | Look in… |
|---|---|
| The HTTP API server | staging/api/server.ts (entry point) |
| API routes | staging/api/routes/<resource>.ts |
| API shared logic | staging/api/lib/<topic>.ts |
| Connector code | packages/core/src/connector-api.ts (compiled to packages/core/dist/) |
| Source-server code | packages/core/src/source-api.ts + sqlite-source-adapter.ts |
| Web UI pages | staging/web-ui/src/pages/<Page>.tsx |
| Web UI components | staging/web-ui/src/components/<Component>.tsx |
| Database schema (DDL) | staging/data/schema/*.sql |
| Connector configs | staging/config/connectors/conn-<id>.json |
| Source configs | staging/config/sources/src-<id>.json |
| Smoke tests | scripts/smoke-*.ts |
| Unit tests | staging/**/__tests__/*.test.ts |
The codebase may contain historical artifacts (legacy directories and files preserved from earlier iterations). These are not part of the active POC — they're kept for reference only. The active POC uses only the directories shown in the tree above.
If you encounter unfamiliar files or directories while exploring the codebase, they're likely legacy artifacts and can be ignored. The active POC scope is the directories documented in this section.
This section is the canonical recipe for getting the project from "I just cloned the repo" to "the stack is running and I can hit the Web UI."
- Node.js 18+ (the API uses native
fetch, AbortSignal.timeout, and other Node 18 features). Node 20 LTS recommended. - npm 9+ (workspaces + overrides support).
- A POSIX-ish shell for running the shell scripts. On Windows, use git-bash (which Hermes provides out of the box) for any
bashcommands. PowerShell and cmd are also supported for the canonical ops scripts viastart-dev.ps1/start-dev.bat. - SQLite CLI (optional, only for manual DB inspection). The runtime uses
better-sqlite3which ships its own native binding.
git clone https://github.com/phi-beta/GCUID-POC.git
cd GCUID-POC
cd staging
npm install
cd ..The staging/npm install is the only install step — it pulls better-sqlite3, express, react, vite, tsx, and friends. The root package.json has no runtime deps (its devDependencies are just typescript for workspace builds).
From the repo root:
./start-dev.sh # POSIX / git-bash
# or
./start-dev.bat # Windows cmd
# or
./start-dev.ps1 # Windows PowerShellWhat this does (in order):
- Starts the API server on
:5182(registers 4 source_systems, runs the seed migration frome5c3b8cif needed, binds to port) - Starts the four source servers on
:5188-5191(each one self-bootstraps its DB if missing, then binds to its port) - Starts the four connector servers on
:5183-5186(each loads its connector JSON config, binds to its port, and registers with the API's/api/connectorscatalog) - Starts the Web UI on
:5181(Vite dev server with hot reload)
The whole startup takes about 8-12 seconds. When it's ready you'll see logs in your terminal saying "✅ Source server running at http://localhost:5188" (etc.) and "Local: http://localhost:5181/" from Vite.
After startup, hit these URLs in a browser:
| URL | What you should see |
|---|---|
http://localhost:5181/ |
The Web UI dashboard (4 source servers, all healthy) |
http://localhost:5182/health |
{"status":"ok"} JSON |
http://localhost:5182/api/system/dashboard |
Full system state including the 4 source-server health entries |
http://localhost:5188/health |
HR source server health (unauthenticated /health probe) |
Or run the orchestrator's invariants smoke from a separate terminal:
./run-smokes.shExpected: === smoke-invariants-runtime: 39/39 assertions passed === and === smoke-cs-4.1-startup-cycle: 20/20 assertions passed ===.
| Port | Service | Auth |
|---|---|---|
| 5181 | Web UI (Vite dev server) | None |
| 5182 | API server (Express) | Bearer token for /api/admin/* and /api/connectors/* |
| 5183 | HR connector | staging_hr_token_12345 |
| 5184 | LDAP connector | staging_ldap_token_12345 |
| 5185 | Engineering connector | staging_eng_token_12345 |
| 5186 | Finance connector | staging_fin_token_12345 |
| 5187 | reserved for ea-mcp-bridge (do not use) |
— |
| 5188 | HR source server | gcuid_source_hr_token_12345 |
| 5189 | LDAP source server | gcuid_source_ldap_token_12345 |
| 5190 | Engineering source server | gcuid_source_engineering_token_12345 |
| 5191 | Finance source server | gcuid_source_finance_token_12345 |
| 11434 | (often) Ollama if you have it — irrelevant to GCUID | — |
If you're behind a port conflict, the canonical port registry lives at E:\github\.dev-ports.json (outside this repo, at e:\github\). Update it if you remap.
./stop-dev.sh # POSIX / git-bash
# or
./stop-dev.bat # Windows cmdThis iterates through the canonical port list and force-kills whatever is listening. It does not gracefully shut down — the API's WAL mode ensures SQLite consistency on next start, but you may see "Address already in use" warnings on the next ./start-dev.sh if a previous instance didn't fully release its port. Re-run stop-dev and try again.
"EADDRINUSE :5182" — port 5182 is held by a previous API instance. Run ./stop-dev.sh and try again. On Windows, occasionally taskkill /F is needed if a tsx.cmd shim is holding the port.
"Cannot find module 'better-sqlite3'" — npm install in staging/ was skipped. Run it.
"Module 'foo' not found" at startup — the staged build of @gcuid/core is out of sync with the source. Run npm run build at the repo root to recompile packages/core/dist/, then restart.
The Web UI starts but shows "Unreachable" for all sources — the source servers didn't bind. Check the start-dev.sh/start-dev.bat console output for EADDRINUSE or seed errors. (Per-source log files are not produced — all sources log to the parent terminal.) See the SysAdmin Guidebook §3.9 Troubleshooting for the recipe.
Web UI hot reload doesn't pick up changes — Vite's file watcher sometimes misses files added after start. Restart ./start-dev.sh if you add a new file under staging/web-ui/src/.
- 9 node processes (1 API + 4 connectors + 4 sources); the Web UI is Vite's dev server, also a node process
- 6 SQLite databases under
staging/data/(registry.db,logs.db, andsources/conn-{hr,ldap,engineering,finance}.db) - 9 PID files under
staging/data/(.api.pid,.connectors.pid,.web.pid,.sources.pid,.source-*.pid) — used bystop-devto identify and kill processes. These are.gitignored. - Log files under
staging/logs/(when running) — useful for post-mortem
Now that the stack is up, you're ready to make changes. The next sections walk you through the codebase:
- §1.4 — the
@gcuid/corelibrary - §1.5 — the API layer
- §1.6 — the Web UI layer
- §1.7 — the connector system (Plan B B3)
When you change code, the Vite dev server hot-reloads the Web UI on file save; the API server does not — restart ./start-dev.sh after any change to staging/api/ to pick up the change. Connector / source server code lives in packages/core/dist/ (compiled from packages/core/src/); run npm run build at the repo root then restart.
This section is the deep dive into packages/core/ — the @gcuid/core library that the staging runtime loads from.
The staging runtime uses only two compiled entry points from the core library:
| Staging caller | Imports from @gcuid/core/dist/ |
Purpose |
|---|---|---|
staging/scripts/start-sources-all.ts |
SourceAPIServer, createSqliteSourceProvider, SourceConfig (via index.js) |
Each of the 4 source servers |
staging/scripts/start-connectors.ts |
ConnectorAPIServer, SourceAPIClient |
Each of the 4 connector servers |
That's it. Every other file in packages/core/src/ is library-only — it's available for other consumers but not wired into the staging runtime. The README's "What this repository contains" table calls this out: packages/core/ provides connector-api.ts and mock-sources.ts (plus everything else that compiles) but the staging runtime only loads from dist/ for the symbols listed above.
packages/core/src/connector-api.ts (~17 KB / ~470 lines) defines the connector HTTP server. Each of the 4 connector processes loads this. Its surface:
GET /health — unauthenticated, returns { status: 'ok', sourceId, ... }
GET /info — authenticated, returns connector metadata
POST /search — authenticated, body: { sourceId, attributes }
GET /records/:id — authenticated, returns the entity record
PATCH /record/:id — authenticated, mutates one attribute field
GET /records?since=… — authenticated, returns delta since timestamp
The connector server is intentionally thin — its main responsibility is HTTP proxying to its upstream source server (the SourceAPIClient does the actual HTTP call to :5188-5191).
packages/core/src/source-api.ts (~15 KB / ~417 lines) defines the source HTTP server and its typed client. Each of the 4 source-server processes loads SourceAPIServer with a SourceDataProvider (constructed via createSqliteSourceProvider) and a SourceConfig (loaded from staging/config/sources/src-<id>.json).
The source server's surface mirrors the connector's (intentionally — they're the same wire format):
GET /health — unauthenticated, returns { sourceId, status, recordCount, latencyMs }
GET /info — authenticated
POST /search — body: { attributes }
GET /records/:id — singular record
PATCH /records/:id — mutate one field
GET /records?since=… — delta
SourceAPIClient is the typed client used by the connector server when it needs to fetch from the upstream. Same symbols, opposite role.
packages/core/src/sqlite-source-adapter.ts (~5.8 KB / ~206 lines) is the SQLite-backed implementation of SourceDataProvider. It opens a SQLite file (e.g. staging/data/sources/conn-hr.db) and exposes the record operations the source server needs. This is what makes the source server pluggable: the SourceAPIServer doesn't know it's reading SQLite; it just calls the SourceDataProvider interface.
The other ~30 source files in packages/core/src/ are library exports that nothing in staging consumes today. They include:
| File | What it provides | Used by staging? |
|---|---|---|
registry.ts |
The identity registry's storage layer | ❌ No — staging has its own staging/api/lib/db-*.ts |
identity-registry.ts, identity-state-machine.ts |
Identity CRUD + state transitions | ❌ No |
fuzzy-matching.ts, enhanced-matching-engine.ts |
Fuzzy attribute matching | ❌ No — staging uses exact-match lookups |
alias.ts, crockford-base32.ts |
GCUID encoding (V1/V2 format) | ❌ No — staging generates IDs differently |
data-normalization.ts |
Attribute normalization for matching | ❌ No |
push-client.ts |
Push-cache client helper | ❌ No |
migrate-to-v2.ts |
Schema migration tooling | ❌ No |
examples-*.ts, *-examples.ts, authoritative-source-examples.ts |
Demo / example code | ❌ No |
cli-new.ts |
archived (see commit 750486c) |
❌ No — placeholder, never built out |
connector-test-utils.ts, connector-integration-tests.ts |
Test helpers for connector developers | ❌ No — not exercised by staging smoke tests |
These files are part of the public surface of @gcuid/core but they're not used by this POC's staging runtime. If you're looking at the code and wondering "why isn't this wired up?" — it's because the staging POC was deliberately scoped down to demonstrate the connector-source separation (Plan B B3), not the full matching/registry/encoding stack. The README's "Public POC release" notice is the high-level explanation; see the Architect Guidebook §4.8 Trade-offs taken for why each was deferred.
The monorepo has 4 workspace packages declared in package.json at the root, not just @gcuid/core. The other three are smaller libraries that no staging code currently imports but are still built as part of npm run build --workspaces:
| Package | Source files | What it provides | Used by staging? |
|---|---|---|---|
@gcuid/core |
32 .ts files in packages/core/src/ |
The runtime library — see §1.4 above | ✅ Yes (via dist/) |
@gcuid/federation |
1 .ts file (packages/federation/src/index.ts, 78 lines, 2.2 KB) |
FederationManager class — a federated-entity coordinator. Exports FederatedEntity, FederationAdapter interfaces and the FederationManager class. The class itself is a thin shell that delegates to registered adapters (no adapters ship today) |
❌ No — declared but no consumers |
@gcuid/attributes |
2 .ts files (packages/attributes/src/index.ts 7 KB / 274 lines + examples.ts 11 KB) |
AttributeManager class — attribute-level CRUD with Attribute and AttributeQuery types. Wraps registry lookups with attribute-specific helpers |
❌ No — declared but no consumers |
@gcuid/api |
1 .ts file (packages/api/src/index.ts, 73 lines, 2 KB) |
GCUIDAPI class — a thin wrapper that composes FederationManager + AttributeManager + registry + matching into a single API object. Has the dependency graph: depends on @gcuid/core, @gcuid/federation, @gcuid/attributes |
❌ No — declared but no consumers |
These three packages are declared in the workspace graph and built by npm run build but not imported by anything in staging/. They exist as the "next layer up" that would compose @gcuid/core's primitives into a higher-level API. Whether they're "needed" depends on what you want to build next:
- If you want a single-process library that wires up
GCUIDManager+IdentityRegistry+FederationManager+AttributeManagerinto a unified object — that's what@gcuid/apiis for. The staging POC skipped it because the staging runtime is HTTP-first (every operation crosses a process boundary to enforce the connector→source split). - If you want federation across multiple identity registries — that's what
@gcuid/federationis for. The staging POC skipped it because all 4 source DBs are local SQLite files, not separate registries. - If you want attribute-level ergonomics over the registry — that's what
@gcuid/attributesis for. The staging POC uses direct SQL viastaging/api/lib/connector-resolver.tsinstead.
When you change @gcuid/core source code, the build order is: core → federation (depends on core) → attributes (depends on core + federation) → api (depends on core + federation + attributes). npm run build --workspaces handles this automatically. None of federation, attributes, or api are imported by staging, so changes to those packages are non-load-bearing for the running stack.
Beyond the 2 wired-in modules (§1.4 above), @gcuid/core/src/ contains 30 more source files spanning registries, identity lifecycle, fuzzy matching, encoding, examples, and CLI stubs. Below is the complete catalog with the major algorithms explained.
| File | Size | Exports | What it does |
|---|---|---|---|
index.ts |
25 KB / 928 lines | GCUIDManager, Identity, IdentityV2, EntityType, CoreIdType, GCUIDFormatVersion, ~20 types |
The barrel + main GCUIDManager class (V1 + V2 generation, parsing, validation). V2 is urn:gcuid:{26-char Crockford core}; V1 is the legacy 5-segment form. Also defines GCUID_V1_REGEX and GCUID_V2_REGEX for parse/validate. |
alias.ts |
9 KB / ~310 lines | generateAlias, validateAlias |
Human-friendly alias generation for GCUIDs (e.g. urn:gcuid:abc123 → @alice@hr). V1 only; V2 makes aliases optional. |
crockford-base32.ts |
6 KB / 231 lines | CROCKFORD_ALPHABET, uuidToCrockford, ulidToCrockford, isValidCrockford |
The Crockford Base32 encoder — converts 128-bit UUID/ULID cores into 26-character base32 strings (omits I/L/O/U to avoid confusion). Used by GCUIDManager for V2 generation. |
logger.ts |
6 KB / ~225 lines | logger, logDebug, logError |
The core's winston-based logger. The staging API uses its own logger (staging/api/lib/logging/logger.ts) rather than this one. |
| File | Size | Exports | What it does |
|---|---|---|---|
state-machine.ts |
6.3 KB / 224 lines | IdentityState, StateTransition, validateTransition, getValidTransitions, canTransition, gcuidStateMachine |
The base identity state machine (xstate-based). Defines states (provisional, active, under_review, suspended, retired, archived) and the allowed transitions. validateTransition(state, event) returns the next state or throws. |
identity-state-machine.ts |
11 KB / 319 lines | IdentityEntityState enum, state machine |
A second state machine for identity entities (separate from IdentityState). Likely the per-entity lifecycle (provisioning → linking → active → purged). |
identity-registry.ts |
14 KB / ~440 lines | IdentityRegistry, RegistryEntry, RegistryQuery, RegistryStats |
The storage layer for identities. Wraps better-sqlite3 with type-safe queries. Staging doesn't use this — staging has its own staging/api/lib/db-*.ts registry. |
registry.ts |
16 KB / ~520 lines | IdentityOrchestrator, CentralizedMatcher, plus others |
The orchestration layer — composes registry + matching + state-machine into a higher-level API. Staging doesn't use this directly. |
identity-orchestration.ts |
18 KB / 568 lines | DuplicateDetectionResult, IdentityOrchestrator |
Multi-step orchestration (provision → match → link → activate). Includes duplicate detection logic (find identities with overlapping attributes). |
migrate-to-v2.ts |
13 KB / ~430 lines | migration CLI | Tooling to migrate V1 GCUIDs to V2 format. Run via npm run migrate -w @gcuid/core. |
| File | Size | Algorithm |
|---|---|---|
fuzzy-matching.ts |
16 KB / 535 lines | Levenshtein-based fuzzy match with attribute weighting. The exported MatchResult interface scores candidates by string similarity (Levenshtein distance normalized to 0..1) + per-field weights from matching-config.ts. Returns top-N candidates above a configurable threshold. |
enhanced-matching-engine.ts |
16 KB / 559 lines | Enhanced match — combines fuzzy + exact + phonetic + abbreviation-expansion. Returns EnhancedMatchResult with score breakdown. Designed for typo-tolerant search. |
data-normalization.ts |
12 KB / 452 lines | Normalization pipeline — strips diacritics, lowercases, removes punctuation, normalizes whitespace, handles common abbreviations ("St" → "Street"). Input to both matchers. Exports RecordNormalizer + NormalizedRecord. |
matching-config.ts |
11 KB / 431 lines | Matching config — AttributeMatchingStrategy per attribute (which algorithm, what threshold, what field weight). Default config plus per-attribute overrides. |
authoritative-source.ts |
22 KB / 868 lines | Source schema definition — AuthoritativeSourceSchema describes the table/columns/keys for an authoritative source, plus transformation helpers. The schema model that connector-api.ts consumes at runtime. |
| File | Size | Exports |
|---|---|---|
push-client.ts |
5.9 KB / 172 lines | PushConnectorClient, PushConnectorClientConfig, IngestRecord, IngestResult — the typed client a source uses to push records to a connector. Used in the push-cache access mode (sources notify connectors of changes rather than being polled). |
connector.ts |
4.3 KB / ~135 lines | The base connector class that connector-api.ts extends. Defines the lifecycle hooks (onSearch, onGet, onUpdate, onSync). |
authoritative-source.d.ts |
9 KB / ~225 lines | TypeScript declaration file for authoritative-source.ts. |
connector-test-utils.ts |
7.2 KB / ~240 lines | Test helpers for connector developers (mock SourceAPIClient, fixture builders). Not currently exercised by staging smoke tests. |
connector-integration-tests.ts |
11 KB / ~360 lines | Integration tests for connectors. Not exercised by staging smoke tests (depends on test fixtures from earlier iterations). |
| File | Size | Purpose |
|---|---|---|
examples.ts |
9.2 KB / ~290 lines | Working examples of GCUIDManager, IdentityRegistry, basic identity flow. Run via npm run example -w @gcuid/core. |
examples-v2.ts |
15 KB / ~470 lines | V2-specific examples — V2 generation, parsing, alias flow. Run via npm run example:v2 -w @gcuid/core. |
identity-examples.ts |
12 KB / ~380 lines | Identity lifecycle examples (provision, link, retire). |
registry-examples.ts |
0.4 KB | Stub (1-line re-export). |
orchestration-examples.ts |
12 KB / ~410 lines | Orchestration examples (multi-step flows). |
authoritative-source-examples.ts |
27 KB / ~870 lines | The biggest example file — walks through authoring an authoritative-source schema + connector + matching config end-to-end. |
cli-new.ts |
0.4 KB | CLI stub (only #!/usr/bin/env node). |
The following source files exist in packages/core/src/ but are not exercised by the active POC:
mock-sources.ts(and its compiled.js+.d.ts+MOCK-SOURCES.md) — earlier-iteration mock-source code superseded by Plan B B3; not importedcli.ts(legacy 2,486-line CLI) — not imported; superseded bystaging/cli/enhanced-matching-examples.ts,integration-demo.ts,test-mock-sources.ts— example/test files that depended on out-of-scope symbols; not imported
These files are kept in the source tree but not part of the active POC scope. Treat them as read-only reference — don't add new imports of them. If you need to bring any of them into the active POC, copy the relevant logic into the appropriate active file rather than importing directly.
The @gcuid/core library is a full identity-platform kit: registry + matching + state-machine + encoding + orchestration + push-cache. The staging POC wires up only the connector/source layer because that's the architectural decision being demonstrated. The other components are kept as a public surface so future deployments can compose them differently (single-process, federated, attribute-centric, etc.) without re-deriving them.
For the staging POC's scope, treat the unwired modules as read-only reference: don't add staging imports of them (it'd violate the connector-source boundary). If you need to add a new algorithm or storage layer to the library, edit the source and rebuild — the staged compilation chain will surface type errors immediately.
The packages/core/dist/ directory is regenerated by:
# from the repo root
npm run buildThis runs tsc -p tsconfig.json for @gcuid/core and @gcuid/federation (the only workspace packages). The output is JS + d.ts files in dist/ that staging imports via require('../../packages/core/dist/connector-api.js').
When you change a file in packages/core/src/:
- Run
npm run buildat the repo root (orcd packages/core && npm run build) - Restart
./start-dev.shso the connectors/sources reload the new compiled JS
The connectors and source servers load their compiled core at startup; there's no hot-reload for compiled modules.
The codebase contains some legacy source files that exist in packages/core/src/ but are not part of the active POC. All of these were removed from the live source in earlier cleanup commits but are mentioned here for historical reference:
mock-sources.ts— earlier-iteration "fake source" code that the connector used to call directly before Plan B B3 split source servers into separate processes. Superseded by the active source-server architecture (§1.7). Archived in commit14acdda(sub-phase 6.1).mock-sources.{js,d.ts,d.ts.map}— pre-built artifacts of the above. Archived in14acdda.cli.ts— a 2,486-line legacy CLI for@gcuid/core(NOT used by staging; staging has its ownstaging/cli/). Archived in14acdda.cli-new.ts— placeholder stub for a planned replacement CLI; never built out. Archived in750486c.connector-test-utils.ts,connector-integration-tests.ts— test scaffolding not wired into jest. Archived in750486c.enhanced-matching-examples.ts,integration-demo.ts,test-mock-sources.ts— example code that depended on out-of-scope symbols. Archived in14acdda.MOCK-SOURCES.md— documentation that went withmock-sources.ts. Archived in14acdda.__tests__/mock-sources.factories.test.ts— test file for the now-out-of-scope factory functions. Archived in14acdda.
If you need any of the archived logic, git checkout <commit>^ -- <file> brings it back into the working tree. The archived versions are preserved in git history (commits 14acdda, 750486c).
| Change | Steps |
|---|---|
| Add a new HTTP route to the connector | Edit packages/core/src/connector-api.ts, run npm run build, restart ./start-dev.sh |
| Add a new HTTP route to the source server | Edit packages/core/src/source-api.ts, run npm run build, restart |
| Change the SQLite schema or add a column | Edit packages/core/src/sqlite-source-adapter.ts, update staging/data/schema/*.sql, regenerate seed (tsx staging/scripts/seed-data.ts --source=<name>), restart |
| Add a new type or interface | Edit packages/core/src/*.ts, run npm run build to emit dist/*.d.ts so staging's TypeScript sees the new types |
The hot-reload boundaries: Vite picks up staging/web-ui/src/ changes; tsx runs staging/api/ and staging/scripts/ directly (no compile step — restart picks them up); packages/core/ requires npm run build then restart.
This section is the walkthrough of staging/api/ — the Express server that fronts every backend.
server.ts (~375 lines) is where every route gets registered. The interesting parts:
- Port: defaults to
5182; override withPORT=<n> node .... - Base path:
process.env.BASE_PATH— optional URL prefix (empty in local mode). - App base URL:
process.env.APP_BASE_URL— for absolute URLs in responses (e.g.https://example.com). - Bootstrap admin: if
GCUID_BOOTSTRAP_ADMIN_EMAILis set, ensures that email exists instaging/config/users.jsonwith roleadminat startup. Idempotent. - DB path: defaults to local SQLite; can be redirected via
GCUID_DB_PATHfor a different storage location (loopback process, separate machine, etc.). - CORS + JSON body parsing: open by default (
cors()with no options);express.json()parses bodies up to the default limit. - Static UI: serves
staging/web-ui/dist/if it exists (production build); the dev mode is Vite on:5181which proxies/api/*to:5182.
The route registrations (from server.ts:231-248):
app.use('/api/user', userRoutes); // User Portal (public lookup)
app.use('/api/hr', hrRoutes); // HR System (authoritative source)
app.use('/api/ldap', ldapRoutes); // LDAP/AD (authoritative source)
app.use('/api/engineering', engineeringRoutes); // Engineering Department
app.use('/api/finance', financeRoutes); // Finance Department
app.use('/api/gcuid', adminRoutes); // GCUID Portal (central coordination)
app.use('/api/caes', caesRoutes); // CAES (Common Attribute Exchange Service)
app.use('/api/flags', flagsRoutes); // PoC feature flags
app.use('/api/log-levels', logLevelsRoutes); // Runtime log level settings
app.use('/api/logs', logsRoutes); // Log viewer (read-only)
app.use('/api/connectors', connectorsRoutes); // Connector configurations
app.use('/api/system', systemDashboardRoutes); // System dashboard + log mgmt
app.use('/api/bulk', bulkRoutes); // Bulk import/export
app.use('/api/hooks', hooksRoutes); // Event-driven hooks (subscribe/poll/verify)
app.use('/api/gcuid-request', gcuidRequestRoutes); // Portal GCUID request (preview + submit)
app.use('/api/tools', toolsRoutes); // System tools (GCUID purge, etc.)
app.use('/api/ux-config', uxConfigRoutes); // UX appearance configuration
app.use('/api/policies', policiesRoutes); // Application access policiesThat's 18 route groups at 19 source files (one of those files — caes.ts — also exports upsertConnectorMappings for the seed-migration flow). The error handler is registered last so all routes bubble through it.
| Route file | Size | Purpose |
|---|---|---|
routes/user.ts |
78 KB | Self-registration, profile lookup, identity lifecycle, attribute updates — the bulk of the end-user surface |
routes/admin.ts |
125 KB | The biggest route file; manages identities, source systems, connectors, admin operations, the seed migration (e5c3b8c's seedSourceSystems) |
routes/caes.ts |
98 KB | Common Attribute Exchange Service — query, bulk fetch, attribute mappings, upsertConnectorMappings |
routes/connectors.ts |
42 KB | The connector-config API: GET /:id/ping, GET /:id/health, GET /:id/sync, CRUD on connector configs |
routes/system-dashboard.ts |
46 KB | The dashboard payload: connector health, source-server health, log-management endpoints |
routes/gcuid-request.ts |
38 KB | The user-facing GCUID-request flow: preview-then-submit, source-link bundling |
routes/engineering.ts, routes/finance.ts, routes/hr.ts, routes/ldap.ts |
~30 KB each | Per-system portals (CRUD on source records, connector-info proxy) |
routes/policies.ts |
29 KB | Application access policies (RBAC, attribute-access control) |
routes/bulk.ts |
20 KB | Bulk import/export (CSV/JSON) |
routes/hooks.ts |
25 KB | Event-driven hooks (subscribe/poll/verify) |
routes/ux-config.ts |
35 KB | UX appearance config (theme tokens, runtime overrides) |
routes/tools.ts |
9 KB | System tools (GCUID purge, etc.) |
routes/logs.ts |
10 KB | Log viewer (read-only — queries logs.db) |
routes/log-levels.ts |
6 KB | Runtime log level configuration |
routes/flags.ts |
7 KB | PoC feature flags |
Some route files are larger than others because they encode domain logic that didn't fit in lib/. When you add a new endpoint, prefer placing it in the existing route file that already owns the resource, not in lib/.
Below is the complete endpoint inventory, grouped by URL prefix. Each row shows method + path. Read-endpoints (GET) are the most numerous; mutation endpoints (POST/PUT/PATCH/DELETE) are usually per-resource.
DELETE /api/gcuid/source-mappings/{gcuid}/{id}DELETE /api/gcuid/source-systems/{id}GET /api/gcuid/aliases/{gcuid}GET /api/gcuid/auditGET /api/gcuid/audit/{auditGcuid}GET /api/gcuid/detect-duplicatesGET /api/gcuid/healthGET /api/gcuid/identifier-state/{gcuid}GET /api/gcuid/listGET /api/gcuid/searchGET /api/gcuid/source-mappingsGET /api/gcuid/source-mappings/{gcuid}GET /api/gcuid/source-systemsGET /api/gcuid/statsGET /api/gcuid/subject/{gcuid}GET /api/gcuid/view/{gcuid}PATCH /api/gcuid/aliases/{gcuid}/{id}/disablePATCH /api/gcuid/aliases/{gcuid}/{id}/enablePATCH /api/gcuid/attributes/{gcuid}POST /api/gcuid/aliases/{gcuid}POST /api/gcuid/aliases/{gcuid}/derive-emailPOST /api/gcuid/approve/{gcuid}POST /api/gcuid/archive/{gcuid}POST /api/gcuid/clear-suspension/{gcuid}POST /api/gcuid/flag-review/{gcuid}POST /api/gcuid/identifier-state/{gcuid}/bindPOST /api/gcuid/identifier-state/{gcuid}/nullifyPOST /api/gcuid/identifier-state/{gcuid}/supersedePOST /api/gcuid/mergePOST /api/gcuid/registerPOST /api/gcuid/rehire/{gcuid}POST /api/gcuid/retire/{gcuid}POST /api/gcuid/source-mappings/{gcuid}POST /api/gcuid/source-systemsPOST /api/gcuid/suspend/{gcuid}PUT /api/gcuid/source-systems/{id}
DELETE /api/caes/attribute-mappings/{id}GET /api/caes/attribute-mappingsGET /api/caes/healthGET /api/caes/match-configsGET /api/caes/schemaGET /api/caes/self-service-attributesGET /api/caes/statsPOST /api/caes/attribute-mappingsPOST /api/caes/matchPOST /api/caes/match-and-linkPOST /api/caes/match-configsPOST /api/caes/resolvePUT /api/caes/attribute-mappings/{id}PUT /api/caes/attribute-mappings/{id}/authoritativePUT /api/caes/match-configs/{sourceSystem}
DELETE /api/engineering/employees/{id}GET /api/engineering/connector/healthGET /api/engineering/coverage-gapsGET /api/engineering/employeesGET /api/engineering/employees/{id}GET /api/engineering/searchGET /api/engineering/statsPATCH /api/engineering/employees/{id}/linkagePOST /api/engineering/employeesPOST /api/engineering/request-gcuidPOST /api/engineering/request-gcuid/confirmPUT /api/engineering/employees/{id}
DELETE /api/finance/employees/{id}GET /api/finance/connector/healthGET /api/finance/coverage-gapsGET /api/finance/employeesGET /api/finance/employees/{id}GET /api/finance/searchGET /api/finance/statsPATCH /api/finance/employees/{id}/linkagePOST /api/finance/employeesPOST /api/finance/request-gcuidPOST /api/finance/request-gcuid/confirmPUT /api/finance/employees/{id}
GET /api/user/aliases/checkGET /api/user/aliases/{gcuid}GET /api/user/attributes/{gcuid}GET /api/user/history/{gcuid}GET /api/user/lookup/{id}GET /api/user/searchGET /api/user/verify/{gcuid}PATCH /api/user/aliases/{gcuid}/{id}/disablePATCH /api/user/aliases/{gcuid}/{id}/enablePATCH /api/user/attributes/{gcuid}POST /api/user/aliases/{gcuid}POST /api/user/self-register
DELETE /api/hr/employees/{id}GET /api/hr/connector/healthGET /api/hr/connector/infoGET /api/hr/coverage-gapsGET /api/hr/employeesGET /api/hr/employees/{id}GET /api/hr/searchGET /api/hr/statsPOST /api/hr/employeesPUT /api/hr/employees/{id}
DELETE /api/ldap/users/{username}GET /api/ldap/connector/healthGET /api/ldap/connector/infoGET /api/ldap/coverage-gapsGET /api/ldap/searchGET /api/ldap/statsGET /api/ldap/usersGET /api/ldap/users/{username}POST /api/ldap/usersPUT /api/ldap/users/{username}
GET /api/connectors/GET /api/connectors/{id}GET /api/connectors/{id}/healthGET /api/connectors/{id}/pingPATCH /api/connectors/{id}POST /api/connectors/{id}/eventPOST /api/connectors/{id}/ingestPOST /api/connectors/{id}/registerPOST /api/connectors/{id}/sync
DELETE /api/system/databases/{dbId}/tables/{tableName}/rows/{pkValue}DELETE /api/system/logs/{system}GET /api/system/dashboardGET /api/system/databasesGET /api/system/databases/{dbId}/tables/{tableName}GET /api/system/healthGET /api/system/policyGET /api/system/statsGET /api/system/versions
DELETE /api/ux-config/theme-library/{id}GET /api/ux-config/GET /api/ux-config/theme-libraryPATCH /api/ux-config/theme-library/{id}PATCH /api/ux-config/{scopeId}POST /api/ux-config/theme-libraryPUT /api/ux-config/PUT /api/ux-config/theme-library/{id}
DELETE /api/hooks/subscriptions/{id}GET /api/hooks/eventsGET /api/hooks/subscriptionsGET /api/hooks/subscriptions/{id}PATCH /api/hooks/subscriptions/{id}POST /api/hooks/subscriptionsPOST /api/hooks/verify
DELETE /api/policies/{appId}GET /api/policies/GET /api/policies/{appId}POST /api/policies/validatePOST /api/policies/validate-gcuidPUT /api/policies/{appId}
POST /api/gcuid-request/previewPOST /api/gcuid-request/scan-linkagesPOST /api/gcuid-request/submit
GET /api/logs/GET /api/logs/statsGET /api/logs/{log_id}
GET /api/bulk/exportPOST /api/bulk/import
GET /api/flags/PATCH /api/flags/
GET /api/log-levels/PATCH /api/log-levels/
GET /api/tools/gcuid/searchPOST /api/tools/gcuid/delete
The lib/ directory holds the cross-route shared code. The interesting files:
| File | Size | What it does |
|---|---|---|
connector-resolver.ts |
19 KB | The lookup pipeline: takes a (gcuid, attribute) request, walks the connector chain, returns the resolved attribute. Handles the direct-DB fallback path (when the GCUID_DB_PATH points at a non-loopback location, the resolver can short-circuit the source HTTP call). The e5c3b8c migration lives in admin.ts but the resolver is what calls it |
source-scan.ts |
35 KB | Walks the connector/source/registry tree to discover what's running and how it relates. Used by system-dashboard.ts to construct the dashboard payload |
logging/log-events.ts |
58 KB | The exhaustive log-event catalog: every event ID, every level, every category. Used by the logger middleware and the log viewer |
logging/log-types.ts |
18 KB | The TypeScript types for log events |
logging/logger.ts |
21 KB | The logger itself (writes to logs.db) |
dept-cache.ts |
11 KB | The attribute-cache layer (push-cache / scheduled-pull modes use this for read-side caching) |
resolve-identifier.ts |
6 KB | Identifier resolution helpers |
apply-transformation.ts |
8 KB | Transformation helpers for connector attribute mapping |
db-paths.ts |
2.4 KB | The canonical DB-path resolver (REGISTRY_DB_PATH, LOGS_DB_PATH, SOURCES_DIR, connectorDbPath) |
version.ts |
1.6 KB | Build version constants |
The db-paths.ts module is the central DB-path resolver. Every SQLite file in the system is accessed via a path returned by this module:
// from staging/api/lib/db-paths.ts
export const DB_BASE: string = efsBase ?? path.join(process.cwd(), 'data');
export const REGISTRY_DB_PATH: string = path.join(DB_BASE, 'registry.db');
export const LOGS_DB_PATH: string = path.join(DB_BASE, 'logs.db');
export const SOURCES_DIR: string = path.join(DB_BASE, 'sources');
export function connectorDbPath(connectorId: string): string {
return path.join(SOURCES_DIR, `${connectorId}.db`);
}If you add a new SQLite file, add a getter here rather than hard-coding the path. The dbPath variable honours the GCUID_DB_PATH environment variable — point it at a different storage location to switch.
The API uses Bearer token auth for admin/connector routes. The token verification lives in each route file (or its middleware). The canonical tokens:
gcuid_admin_token_demo— for/api/admin/*,/api/connectors/*, etc.staging_hr_token_12345— for the HR connectorstaging_ldap_token_12345— for the LDAP connectorstaging_eng_token_12345— for the Engineering connectorstaging_fin_token_12345— for the Finance connectorgcuid_source_<id>_token_<n>— for the source servers (set instaging/config/sources/src-<id>.json)
The token comparison is constant-time in some routes, plain === in others. Don't change auth comparisons without understanding the security implications — see the Architect Guidebook §4.10 Open questions for the local-only stance on this.
For a typical lookup (GET /api/gcuid/<id>):
- Express receives the request, runs
cors()andexpress.json()middleware - The
loggermiddleware records the request tologs.db adminRoutes(/api/gcuid) matches/api/gcuid/<id>via its router- The route handler reads
source_systemsfromregistry.dbto find the connector_id and access_mode - If the access mode is
push-cacheorscheduled-pull, it returns from the attribute cache (viadept-cache.ts) - Otherwise, it calls
ConnectorAPIClient(frompackages/core/dist/connector-api.js) which proxies to the connector server on:5183-5186 - The connector proxies to the source server on
:5188-5191(viaSourceAPIClient) - The source server reads from
data/sources/conn-<id>.dband returns the record - The connector transforms the record per its
attributeMappingsconfig and returns to the API - The API optionally writes the attribute to the cache and returns the response to the client
Each step is logged via the structured logger. The full event chain is in logs.db and visible via the Web UI's log viewer (or GET /api/logs).
Add a new route to an existing resource: edit the corresponding file in routes/, restart ./start-dev.sh (the API doesn't hot-reload). The new route is available at the registered path; no other registration needed.
Add a new route group (e.g. /api/audit-log): create routes/audit-log.ts, export a Router, add app.use('/api/audit-log', auditLogRoutes); in server.ts. Restart.
Add a new DB column: edit the table-creation DDL in admin.ts (or wherever the table is created), update the type definitions, update the seed migration if needed. Restart.
Change auth: search the route file for Authorization / Bearer / token-comparison logic. Be conservative — auth changes are high-risk.
Add a new shared helper: put it in lib/ if it's used by 2+ routes; put it in the route file if it's used by only 1.
- §1.4 — what
@gcuid/coreprovides - §1.7 — the connector system
- §1.9 — adding a new connector (worked example)
- §1.10 — adding a new API endpoint (worked example)
- Architect Guidebook §4.6 Access modes — what
dynamic-pull/scheduled-pull/push-cachemean - SysAdmin Guidebook §3.5 Authentication tokens — where the tokens are configured
The gcuid-staging CLI is the command-line operator for the staging registry. It runs against the same SQLite files that the API server uses, so it shares data state with the live HTTP API.
# Via npm script (works from staging/)
npm run cli -- <command>
# Or directly
npx tsx cli/index.ts <command>The CLI resolves paths relative to the current working directory, so run it from staging/ (or anywhere whose data/ subtree mirrors the staging SQLite layout).
Three persona-flavoured subcommand groups:
| Subcommand | Persona | Read-only? | Writes? |
|---|---|---|---|
user |
End user / auditor | Yes | No |
dept |
HR / LDAP / Engineering / Finance operator | Yes (most), No (data) | dept data is read-only |
admin |
Central GCUID registry admin | Yes (most) | approve / suspend / retire mutate the registry |
The CLI is wired against data/registry.db (central GCUID registry: identities, state_history, source_mappings, audit_log) and data/sources/conn-hr.db (HR connector database: employees table). The other connector databases (LDAP, Engineering, Finance) are read by the API but not by this CLI — use the API at /api/ldap/*, /api/engineering/*, /api/finance/* or the web UI for those.
user lookup <id>— Look up by employee ID, email, or GCUID. Tries each shape in order (identities → source_mappings → email prefix) and returns the first match. Options:-f, --format <table|json>.user search <query>— Substring search acrossidentities(name, email, entity_id). Returns up to 50 results.user history <gcuid>— State-transition history for an identity.user verify <gcuid>— Validate the GCUID format and check whether it is currentlyactive.
Example: npm run cli -- user lookup E11001 returns a table with GCUID, entity ID, state, email, name, source, created/updated timestamps.
admin list [--state <state>] [--limit <n>] [--format <table|json>]— List identities, optionally filtered by state (provisional,active,under_review,suspended,retired,archived). Default limit: 50.admin view <gcuid> [--with-history]— Detailed identity view.admin approve <gcuid> [--reason <reason>]— Transitionprovisional/under_review→active. Default reason:"Administrator approval".admin suspend <gcuid>— Transitionactive→suspended(also permitted fromunder_review).admin retire <gcuid>— Transition →retired. Tombstones the identifier token. (Allowed fromactive,suspended,under_review.)admin detect-duplicates— Scan the registry for potential duplicate groups. Reports exactentity_idmatches and fuzzy matches based on source mappings.admin search <query>— Substring search across allidentitiesfields, including JSON metadata.admin audit <gcuid>— Complete audit trail (creation + every state transition).admin stats— Comprehensive registry statistics: counts by state, by entity type, by source system, integration coverage.admin health— Health check: validates registry file, GCUID formats, checks for orphaned history rows.
The dept subcommands operate on the HR connector's SQLite file (data/sources/conn-hr.db). The cross-connector coverage report in dept data stats also reads the central registry.
dept data list [--limit <n>] [--format <table|json>]— List all employees in the HR connector.dept data view <id> [--with-gcuid]— One employee record.--with-gcuidincludes integration status (looks up in central registry).dept data search <query> [--limit <n>] [--with-gcuid]— Substring search across HR employee fields (name, email, department). Default limit: 20.dept data stats— HR data statistics: total employees, count by department, integration rate (joins HR employee ids againstidentities.entity_id).dept connector status— Connector status: registry online, HR DB online, integration rate. Notes that the live HTTP connector service must be started vianpm run start-connectors.dept connector test— Run 4 connectivity tests (registry DB on disk, HR DB on disk, can query each).dept connector info— Connector configuration: type, protocol, DB paths, working directory.dept monitor dashboard— Top-line operational dashboard: total employees, total GCUIDs, counts by state, integration coverage.dept monitor stats— 24-hour activity stats: identity-state counts (with percentages) and transition volume.
| Operation | CLI | HTTP API |
|---|---|---|
| Identity lookup | user lookup <id> |
GET /api/user/lookup/{id} |
| Identity history | user history <gcuid> |
GET /api/user/history/{gcuid} |
| GCUID verify | user verify <gcuid> |
GET /api/user/verify/{gcuid} |
| Search identities | admin search <q> |
GET /api/gcuid/search |
| Approve identity | admin approve <gcuid> |
POST /api/gcuid/approve/{gcuid} |
| Duplicate check | admin detect-duplicates |
GET /api/gcuid/detect-duplicates |
| Registry stats | admin stats |
GET /api/gcuid/stats |
Mutating CLI commands (approve, suspend, retire) write directly to data/registry.db — the next API call sees the change because they share the same SQLite file. The CLI does not currently update the audit_log table the same way the API does; use the API for full audit compliance.
- CLI — fastest for read-only inspection, scripted operations, or ops automation that doesn't need the full audit log.
- HTTP API — use for any state mutation that needs the full audit trail and the per-system logging (
data/logs.db). - Web UI — use for ad-hoc human review and demonstrations; same data, but visualised.
staging/cli/index.ts— entry point, command registrationstaging/cli/commands/user.ts—usersubcommandsstaging/cli/commands/dept.ts—deptsubcommandsstaging/cli/commands/admin.ts—adminsubcommands
This section is the walkthrough of staging/web-ui/ — the React + Vite frontend that surfaces every API endpoint.
- React 18 with TypeScript (strict mode)
- Vite 5 dev server + bundler (configured in
staging/web-ui/vite.config.ts) - React Router v6 for client-side routing (
App.tsxdeclares the route map) - React Query for server-state caching (used per-page; not centralized)
- No component library — every component is hand-rolled. There is no Material-UI / Tailwind / Ant Design.
staging/web-ui/src/
├── App.tsx (5.4 KB, the route map)
├── main.tsx (1.5 KB, the React root + providers)
├── api/
│ └── client.ts (2,036 lines, the only file that talks to the API)
├── components/ (14 .tsx files, shared UI)
├── config/
│ └── portals.ts (8 KB, route → portal metadata table)
├── context/ (3 .tsx files: theme, datetime, portal mode)
├── data/
│ └── connectorThemePresets.ts (57 KB, theme presets for connectors)
├── hooks/ (6 .ts files: feature flags, pagination, sortable tables, theme)
├── pages/ (28 .tsx files, one per route)
├── theme/ (navTheme.ts, useScopeNavTheme.ts)
└── utils/
└── environment.ts (1 KB, NODE_ENV helpers)
Total: 48 .tsx files, 1 .ts file in api/ across pages/ + components/ + context/ + hooks/ + theme/ + utils/. The biggest files are the page-level data grids (GCUIDPortal.tsx is 164 KB, CAESPortal.tsx is 122 KB, TestingPortal.tsx is 100 KB) — they encode their own table state, columns, and filters rather than using a shared table primitive.
client.ts is the only file in the Web UI that talks to the API. Every other component/page imports from api/client.ts instead of calling fetch directly. The pattern:
// example usage from any page
import { api } from '../api/client';
const result = await api.listConnectors();The client exposes one class (api) with ~80 methods, organized by resource:
| Method group | Example methods |
|---|---|
| Identity | getIdentity(gcuid), updateIdentity(gcuid, attrs) |
| Source systems | listSourceSystems(), getSourceSystem(id) |
| Connectors | listConnectors(), getConnector(id), pingConnector(id), healthConnector(id) |
| GCUID request | previewGcuidRequest(req), submitGcuidRequest(req) |
| Bulk | importBulk(payload), exportBulk(query) |
| Logs | listLogs(filter), getLog(id) |
| Tools | purgeGcuid(gcuid), resetConnector(id) |
| UX config | getUXConfig(), updateUXConfig(...) |
| Policies | listPolicies(), getPolicy(id) |
| System | getSystemDashboard() |
The client adds the bearer-token header from localStorage (or session state) and constructs URLs against /api/... — the Vite dev proxy on :5181 rewrites those to http://localhost:5182 (the API). In production, vite build produces a static bundle in dist/ and the API serves it (see server.ts WEB_UI_DIST static handler).
The full route map (extracted from App.tsx):
| Path | Page component | Purpose |
|---|---|---|
/ |
PortalIndexPage.tsx |
Landing page — index of portals |
/user |
UserPortal.tsx |
User lookup by attribute |
/register |
RegistrationPortal.tsx |
New-identity self-registration |
/helpdesk |
HelpDeskPortal.tsx |
Internal staff lookup |
/helpdesk/register |
HelpDeskRegistrationPortal.tsx |
Staff-assisted registration |
/hr |
HRPortal.tsx |
HR system records |
/ldap |
LDAPPortal.tsx |
LDAP/AD records |
/engineering |
EngineeringPortal.tsx |
Engineering dept records |
/finance |
FinancePortal.tsx |
Finance dept records |
/gcuid |
GCUIDPortal.tsx |
Central GCUID lookup + admin |
/caes |
CAESPortal.tsx |
Common Attribute Exchange Service |
/sources |
SourceConnectorPortal.tsx |
Source-connector browse |
/options |
OptionsPortal.tsx |
User options / preferences |
/logs |
LogsPortal.tsx |
System log viewer |
/dashboard |
DashboardPortal.tsx |
Aggregate dashboard |
/testing |
TestingPortal.tsx |
API testing surface (curl-equivalent) |
/databases |
DatabasesPortal.tsx |
Database browser (read-only) |
/tools |
ToolsPortal.tsx |
System tools (purge, reset) |
/ux |
UXPortal.tsx |
UX theme admin |
/system |
SystemPortal.tsx |
System admin (services, processes) |
/apps / /apps/:appId |
CorporateAppPortal.tsx |
Corporate applications registry |
/policies |
PolicyPortal.tsx |
Application access policies |
/about |
AboutPortal.tsx |
About / license / version |
27 routes across 24 distinct page components (PortalIndexPage.tsx covers /; the rest are 1:1 with their path). CorporateAppPortal.tsx is the only page with a parameter — /apps/:appId selects which corporate app to show.
The 14 components in components/:
| Component | Size | Used by |
|---|---|---|
Layout.tsx |
31 KB | App shell — top bar, side nav, portal switcher (used by every page) |
ConnectorIntegrationTab.tsx |
50 KB | Connector detail view (sources/options/testing pages) |
ConnectorSequenceDiagram.tsx |
12 KB | Mermaid sequence diagrams for connector flows |
ConnectorStatusPanel.tsx |
12 KB | Per-connector health badge + ping/health buttons |
ConnectorCacheBar.tsx |
8 KB | Cache controls (purge, refresh) for connectors |
GcuidRequestPanel.tsx |
15 KB | The user-facing GCUID-request form (preview-then-submit) |
StateDiagram.tsx |
10 KB | State machine visualizer for identity state transitions |
UXThemeContext (provider) |
7.6 KB | Theme tokens (light/dark + custom themes) |
PortalLayout.tsx |
6.3 KB | Per-portal layout wrapper (top-of-page header + portal-specific nav) |
navTheme.ts (data, not component) |
6.8 KB | Theme tokens for nav/topbar |
useScopeNavTheme |
1 KB | Hook for scoped theme overrides |
StateBadge.tsx |
3.2 KB | Identity-state badge (provisional, active, etc.) |
Pagination.tsx |
4.2 KB | Cursor-based pagination control |
SortTh.tsx |
3.0 KB | Sortable table header |
ErrorMessage.tsx |
1.6 KB | Error banner component |
LoadingSpinner.tsx |
1.0 KB | Loading spinner |
CopyButton.tsx |
2.9 KB | Copy-to-clipboard button |
The Web UI uses React Query for server state (cached, invalidated on mutation, refetched on focus). Theme and portal mode use React Context (UXThemeContext, PortalModeContext). Date/time formatting uses DateTimeFormatContext. There is no Redux, Zustand, or other global store.
Pagination, sorting, and filtering are page-local state managed via useState + custom hooks (usePagination, useSortableTable, useFeatureFlags). Cross-page state survives via React Query's cache (keyed by query string).
staging/web-ui/vite.config.ts configures the dev proxy:
server: {
port: 5181,
proxy: {
'/api': { target: 'http://localhost:5182', changeOrigin: true }
}
}When you curl http://localhost:5181/api/connectors in dev mode, Vite forwards it to :5182. In production (vite build), the proxy is gone; the API serves the static dist/ files and routes /api/* itself.
staging/web-ui/src/— Vite hot-reloads on every save. No restart needed.staging/api/— Restart./start-dev.shafter any change (API doesn't hot-reload).packages/core/—npm run buildthen restart (compile step required).
Add a new page: create pages/MyPage.tsx, register a route in App.tsx (<Route path="my-page" element={<MyPage />} />), add a metadata entry in config/portals.ts if it should show in the portal nav.
Add a new API client method: edit api/client.ts, add a method to the api class, type the request/response. TypeScript will surface the new method in every page that imports api.
Add a new component: create components/MyComponent.tsx, import it where needed. No central registry.
Add a new hook: create hooks/useMyHook.ts, export the hook function. Hooks are conventional useXxx names.
- §1.5 — the API layer that the client calls
- §1.7 — the connector system surfaced in the UI
- §1.10 — extending the Web UI (worked example)
- User Guidebook §2.4 The Web UI tour — what end users see
This section is the developer-facing walkthrough of the connector system — the 4 connector processes and the 4 source-server processes they proxy to. This is the central architectural decision of the POC.
Where this lives in design docs: The why and the alternatives considered are in Architect Guidebook §4.5 Plan B B3. The access-mode taxonomy is in Architect Guidebook §4.6 Access modes. This section is the how to work with it companion.
The POC runs 4 connector processes (one per authoritative source) and 4 source-server processes (one per source data store):
| Connector process | Port | Source it proxies | Source-server process | Source-server port | Backing DB |
|---|---|---|---|---|---|
| HR | 5183 | HR system | src-hr | 5188 | data/sources/conn-hr.db |
| LDAP | 5184 | LDAP/AD | src-ldap | 5189 | data/sources/conn-ldap.db |
| Engineering | 5185 | Eng department | src-engineering | 5190 | data/sources/conn-engineering.db |
| Finance | 5186 | Finance department | src-finance | 5191 | data/sources/conn-finance.db |
Both processes are started by start-dev.sh (the canonical local stack script). The orchestration scripts (staging/scripts/start-connectors.ts and staging/scripts/start-sources-all.ts) read the per-system configs and spawn them with the right port + token + DB path.
For a request to GET /api/gcuid/<id>:
┌────────┐ HTTP ┌────────┐ HTTP ┌────────┐ HTTP ┌────────┐ SQLite ┌────────┐
│ │ :5182 │ │ :5183 │ │ :5188 │ │ read │ │
│ API │─────────▶│ HR │─────────▶│ src-hr │────────▶│ Source │────────▶│ conn- │
│ server │ │ connector│ │ server │ │ Data │ │ hr.db │
│ │ │ │◀─────────│ │◀────────│ Provider│◀────────│ │
└────────┘ └────────┘ response└────────┘ JSON └────────┘ └────────┘
That's 4 hops: API → connector → source server → SQLite. Each hop is HTTP. Each is independently restartable.
Each connector + source is configured by two files (in staging/config/connectors/ and staging/config/sources/):
Connector config (conn-hr.json):
{
"id": "conn-hr",
"label": "HR System Connector",
"connectorServer": {
"port": 5183,
"authToken": "staging_hr_token_12345"
},
"upstreamSource": {
"baseUrl": "http://localhost:5188",
"authToken": "gcuid_source_hr_token_9999"
},
"attributeMappings": {
"primaryKeyField": "employee_id",
"tables": { "employees": { "alias": "person", "primaryKey": "employee_id" } }
},
"accessMode": "dynamic-pull"
}Source config (src-hr.json):
{
"id": "src-hr",
"label": "HR Authoritative Source",
"sourceServer": {
"port": 5188,
"authToken": "gcuid_source_hr_token_9999"
},
"dataProvider": {
"type": "sqlite",
"dbPath": "./data/sources/conn-hr.db",
"table": "employees",
"primaryKey": "employee_id"
}
}The authToken on the source side (gcuid_source_*) must match the upstreamSource.authToken on the connector side — that's the bearer-token handshake between them. The connector token (staging_hr_*) is separate and authenticates connector-to-API calls.
This is the canonical "extend the POC with a new authoritative source" workflow. Worked example: adding conn-parking + src-parking.
- Connector port: next free in
:5183-5186(5187 is reserved forea-mcp-bridge) - Source-server port: next free in
:5188-5191 - Connector token:
staging_parking_token_<n> - Source token:
gcuid_source_parking_token_<n>
Create staging/config/sources/src-parking.json with the schema above. Adjust dbPath / table / primaryKey to match your source data.
Create staging/config/connectors/conn-parking.json. The upstreamSource block must reference the source-server's port + the matching token.
Either:
- Manual: write a SQLite file at
staging/data/sources/conn-parking.dbwith the schema your data provider expects (seestaging/data/schema/source-employees.sqlfor the HR template), and seed rows. - Seeded: write a seed script
staging/scripts/seed-data.ts --source=parkingthat callsseedSourceIfEmpty('conn-parking', ...). See §1.7 for the seed-file format.
The registry (data/registry.db table source_systems) needs a row for conn-parking. Add it via the seed migration in admin.ts:2414-2428. Pattern:
db.prepare(`
INSERT OR IGNORE INTO source_systems (system_id, display_name, system_type, api_url, api_secret)
VALUES ('conn-parking', 'Parking System', 'department', 'http://localhost:5183', 'staging_parking_token_<n>')
`).run();Idempotent — safe to re-run. Authority for any specific attribute (e.g. "HR is canonical for person.employment.employee_id") is not set here — it's a property of the attribute_mappings row, not the source-system row. To declare a source authoritative for an attribute, use the CAES service:
PUT /api/caes/attribute-mappings/<mapping_id>/authoritative
This sets is_authoritative=1 on the chosen mapping and atomically clears it on every other mapping for the same standard_attribute (the CAES route implementation at staging/api/routes/caes.ts:1046-1071 wraps both updates in a transaction).
Both staging/scripts/start-connectors.ts and staging/scripts/start-sources-all.ts discover configs from the config directories. They iterate over the files in staging/config/connectors/*.json and staging/config/sources/*.json and spawn one process per config. Adding the JSON is enough.
This picks up the new files in both directories, spawns the new connector + source processes, and the dashboard reflects them on next refresh.
curl http://localhost:5182/api/connectors/conn-parking/ping
# → { reachable: true, base_url: "http://localhost:5183", ... }
curl -H "Authorization: Bearer gcuid_source_parking_token_<n>" \
http://localhost:5188/health
# → { sourceId: "src-parking", status: "ok", recordCount: <n>, latencyMs: <ms> }If you want a portal page for the new source (e.g. /parking/lookup), copy one of the per-system route files (routes/hr.ts is the smallest), rename, and register it in server.ts. Skip if the new source is read-only via /api/gcuid/<id>.
The connector doesn't just proxy records — it transforms them per the attributeMappings config. This is the policy layer: the source DB might use employee_id as the primary key; the GCUID system uses gcuid. The connector maps between the two.
The mapping is per-table (or per-record-set). Each table can declare:
alias— the logical name the API exposes (e.g.person)primaryKey— the source-side key fieldattributeMap— a per-field transformation spec
The transformation lives in packages/core/src/connector-api.ts — see applyTransformations() and mapAttribute(). If you need a new transformation type, add it there and rebuild @gcuid/core.
Cause: token mismatch. The connector's upstreamSource.authToken doesn't match the source's sourceServer.authToken. Check both configs.
Diagnostic: curl -H "Authorization: Bearer <connector-token>" http://localhost:5188/health returns 200 (the connector token shouldn't grant access; if it does, the configs are misaligned).
Cause: missing/wrong Authorization header on the source-server call. The SourceAPIClient adds the bearer token automatically; if you bypass it (e.g. via curl), you need to pass it manually.
Cause: self-bootstrap not run, or seed-data script failed silently. Re-run:
tsx staging/scripts/seed-data.ts --source=parkingThe seedSourceIfEmpty() helper in start-sources-all.ts is supposed to call this automatically on startup; if the DB exists but is empty, it skips seeding (assumes you cleared it deliberately).
Cause: usually a startup error — port already in use, bad config (JSON syntax), or DB path missing. Check logs.db (SELECT * FROM log_events WHERE component = 'connector' ORDER BY timestamp DESC LIMIT 20) or the Web UI log viewer.
Cause: the source server's dataProvider table name or primary key doesn't match the source config. The /health endpoint doesn't actually query the table (it returns metadata); only /search / /records/:id exercise the DB.
The original POC design (Plan A) had the connector calling the source DB directly — no separate source server. Plan B B3 split that into:
- A1 — Direct DB call, same process (kept the source DB in-process)
- B1 — Direct DB call, separate process (file-system access across processes)
- B3 — HTTP API call, separate process (the chosen path)
B3 was chosen because:
- Process isolation: a bug in the connector can't corrupt the source DB file.
- Network boundary: the connector can be deployed to a different host than the source DB (future-proof for cloud).
- Independent restart: the source DB can be re-seeded without restarting the connector.
- Testable in isolation: each source server can be smoke-tested directly via
curlwithout spinning up a connector.
The trade-off is latency: an extra HTTP hop adds ~5ms per request. For the POC this is acceptable; in production you'd cache frequently-fetched attributes.
- §1.4 — the connector server type (
ConnectorAPIServer) - §1.5 — the API layer (calls into the connector)
- Architect Guidebook §4.5 Plan B B3 — why B3 was chosen
- Architect Guidebook §4.6 Access modes — the access-mode taxonomy
- SysAdmin Guidebook §3.4 Logs and diagnostics — log events for connector failures
This section is the definitive catalog of every external library and dependency the POC actually uses, why each was chosen, and where to find its call sites. It's the answer to "what does this codebase depend on, and what's it for?".
Two layers: §1.4 covers the custom modules built in-repo (under
packages/). This section (§1.8) covers the third-party dependencies the custom modules + the staging runtime depend on.
The POC pulls in dependencies from two layers of package.json files: the workspace-package level (packages/core/package.json declares what @gcuid/core needs) and the staging-runtime level (staging/api/package.json + staging/web-ui/package.json declare what the running services need).
| Package | Version | Used by source files | What it does |
|---|---|---|---|
better-sqlite3 |
^12.6.2 | connector-api.ts, registry.ts, identity-registry.ts, sqlite-source-adapter.ts (and any file that opens a .db) |
Synchronous SQLite — the entire storage layer. Chosen for its synchronous API (no callback/Promise overhead) and predictable performance under load. The staging API and the source servers both use new Database(path) + prepared statements. |
express |
^5.2.1 | connector-api.ts (the connector HTTP server) |
The connector HTTP server. The source servers use Node's built-in http module directly (smaller surface, no Express dependency). The staging API uses Express 5 too (separate dependency at the staging layer). |
winston |
^3.19.0 | logger.ts (and indirectly staging/api/lib/logging/logger.ts) |
Structured logger with levels, transports, and JSON formatting. The core library uses winston; the staging API has its own wrapper (staging/api/lib/logging/) that writes to logs.db. |
xstate |
^5.25.1 | state-machine.ts, identity-state-machine.ts |
State-machine library — declarative state definitions + transition guards. The identity-state-machines in core are xstate machines. |
convict |
^6.2.4 | config.ts |
Config validation — schema-driven config with type-checking and defaults. Used by core's config layer; staging has its own JSON-only config approach. |
cli-table3 |
^0.6.5 | (declared for example CLIs / legacy cli.ts) |
Pretty-printed tables for CLI output. Not currently used by any wired-in code; declared for the example/CLI surface. |
commander |
^14.0.2 | (declared for example CLIs / legacy cli.ts) |
CLI argument parser. Not currently used by any wired-in code; declared for the example/CLI surface. |
ulid |
^3.0.2 | index.ts (GCUIDManager can generate from a ULID core) |
ULID generator — lexicographically-sortable unique IDs. Used as one of the two core-ID sources (the other is UUID v4). |
uuid |
^13.0.0 | index.ts (v4 import) |
UUID v4 generator — the other core-ID source. |
@types/express |
^5.0.6 | (TypeScript types only — no runtime impact) | Express types for the connector server. |
| Package | Version | Notes |
|---|---|---|
@gcuid/core |
^0.1.0 | Workspace dependency. The federation layer composes core primitives. |
| Package | Version | Notes |
|---|---|---|
@gcuid/core |
^0.1.0 | Workspace dependency. |
@gcuid/federation |
^0.1.0 | Workspace dependency. The attribute layer uses federation internally. |
| Package | Version | Notes |
|---|---|---|
@gcuid/core |
^0.1.0 | Workspace dependency. |
@gcuid/federation |
^0.1.0 | Workspace dependency. |
@gcuid/attributes |
^0.1.0 | Workspace dependency. |
The staging API is its own Node project. Its dependencies are separate from @gcuid/core's. The staging runtime depends on:
| Package | Where used | What it does |
|---|---|---|
express |
staging/api/server.ts |
The HTTP server. |
cors |
staging/api/server.ts:179 |
Open CORS (any origin) for local dev. |
better-sqlite3 |
Every route file that opens registry.db or logs.db |
The DB driver. |
(No Express middleware beyond cors and express.json()) |
No helmet, no express-rate-limit, no body-parser. The POC keeps the middleware stack minimal. |
The staging API uses Node built-ins for everything else: crypto for hashing, fs for file ops, path for path resolution, url for fileURLToPath. No lodash, no axios, no moment.
The Web UI pulls in:
| Package | Where used | What it does |
|---|---|---|
react, react-dom |
Every page/component | The UI framework (React 18). |
react-router-dom |
App.tsx, Layout.tsx |
Client-side routing. |
@tanstack/react-query |
Every page that fetches data | Server-state caching. |
vite, @vitejs/plugin-react |
vite.config.ts, dev server |
The dev server + bundler. |
typescript |
tsconfig.json | TypeScript compilation. |
There is no state-management library beyond React Query + Context — no Redux, no Zustand, no Jotai. There is no UI component library — no Material-UI, no Tailwind, no Ant Design. There is no date library beyond Intl.DateTimeFormat (native browser API) wrapped in useDateTimeFormat.
Each runtime dep was chosen for a specific reason. The rationale table:
| Package | Why this and not an alternative |
|---|---|
better-sqlite3 |
Synchronous API = simpler code (no async/await ceremony) + faster for small queries. The staging runtime is single-process and local-only; the async-overhead tax isn't worth it. Alternatives (sqlite3, node-sqlite3) are async-callback based. |
express |
The connector server needs HTTP routing; Express is the de-facto Node standard. Alternatives (Koa, Fastify, raw http) were considered — Express won on familiarity + ecosystem. |
winston |
Structured logging with multiple transports (console, file, HTTP) out-of-the-box. Alternatives (pino, bunyan) are faster but winston's API is more approachable for the POC. |
xstate |
Declarative state machines with visualizer support. Alternatives (hand-rolled switch statements) don't give you the static-analysis benefits (xstate machines are inspectable at runtime via inspect()). |
convict |
Schema-driven config validation. Alternatives (hand-rolled JSON load + check) lose the type-safety and the env-var interpolation. |
ulid + uuid |
Two different ID formats — ULID is time-sortable (good for log/event ordering), UUID v4 is the universal default. The CoreIdType enum lets you pick either. |
cli-table3 + commander |
Declared for the example/CLI surface (legacy cli.ts + the examples-*.ts files). Not currently exercised by staging; carried forward so the example scripts continue to work. |
react + react-dom |
React 18's concurrent features (useTransition, Suspense) are useful for the data-grid pages with their heavy filter state. |
react-router-dom |
The de-facto React router; familiar API. |
@tanstack/react-query |
Caching + invalidation without writing fetch wrappers. |
vite |
Fast dev server + simple config. No webpack config to maintain. |
The POC deliberately avoids:
- No
dotenv— config is via JSON files (staging/config/) and env vars read directly viaprocess.env. - No
helmet/ noexpress-rate-limit/ nocsurf— the API is local-only. - No
axios— the API client uses nativefetch(withAuthorizationheaders constructed manually). - No
lodash— small utility code is hand-rolled; the project doesn't need lodash's breadth. - No date library (
moment,date-fns,luxon) —Intl.DateTimeFormatcovers the needs;useDateTimeFormatwraps it. - No ORM (Prisma, TypeORM, Drizzle, Knex) — direct
better-sqlite3prepared statements throughout. The schema is small enough that an ORM would add overhead without value. - No test framework beyond Jest (declared in
@gcuid/core/package.json) — the staging layer uses hand-rolledscripts/smoke-*.tssmoke tests instead of adding Jest to the staging runtime.
For @gcuid/core (or any workspace package):
cd packages/core
npm install --save <package>
npm run build # rebuilds dist/
# staging/api now needs the dep too:
cd ../../staging/api
npm install --save <package>For staging/api:
cd staging/api
npm install --save <package>
# restart ./start-dev.shFor staging/web-ui:
cd staging/web-ui
npm install --save <package>
# Vite hot-reloads; if Vite fails to pick up, restart ./start-dev.shAvoid adding deps for things you could do with Node built-ins. The POC has a deliberate "small deps" stance; each new dep adds attack surface, install time, and maintenance burden.
To see what a particular package actually depends on at runtime:
# from the staging runtime perspective:
grep -rE "require\(|from '" /e/GitHub/GCUID-POC/staging/api/ \
--include='*.ts' | grep -vE "from '\./|from '\.\./|require\('\./|require\('\.\./"
# from the core library perspective:
grep -rE "^import .* from ['\"]" /e/GitHub/GCUID-POC/packages/core/src/ \
| sed -E "s/.*from ['\"]([^'\"]*)['\"].*/\1/" | grep -vE "^\./|^\.\./" | sort -uThe first command lists every non-relative import in staging. The second lists every non-relative import in @gcuid/core/src/. If a dependency is declared in package.json but doesn't appear in either output, it's declared-but-unused — a candidate for removal if the intent was purely to enable the example scripts.
- §1.4 — the custom modules (
@gcuid/core,@gcuid/federation,@gcuid/attributes,@gcuid/api) - §1.5 — where Express + better-sqlite3 are used in the staging API
- §1.6 — where React + Vite + React Query are used in the Web UI
- §1.9 — the database schema (what better-sqlite3 opens)
- SysAdmin Guidebook §3.2 Dependency versions — how to bump a dep
- Architect Guidebook §4.8 Trade-offs taken — why the "small deps" stance
This section is the canonical reference for the SQLite databases the POC uses. The DBs fall into three groups: the registry (central identity coordination), the per-source DBs (one per authoritative source), and auxiliary DBs (logs, hooks, policies, UX config).
| DB file | Backing module | Owner | Purpose |
|---|---|---|---|
data/registry.db |
staging/api/routes/admin.ts (DDL at L2340-2400) |
API (read/write) | The central identity registry + source-system catalog + attribute cache. All four client processes (HR, LDAP, Engineering, Finance) ultimately resolve through this. |
data/logs.db |
staging/api/lib/logging/log-events.ts + logger.ts |
API (write), UI (read) | The structured log storage. Every API request, connector health check, and source-server ping lands here. |
data/sources/conn-<id>.db |
packages/core/src/sqlite-source-adapter.ts |
Source server (read/write) | One per authoritative source: conn-hr.db, conn-ldap.db, conn-engineering.db, conn-finance.db. Holds the raw employee/identity records. |
| Hooks/policies/flags/log-levels/ux-config | staging/api/routes/{hooks,policies,flags,log-levels,ux-config}.ts |
API | Smaller auxiliary tables. Often co-located in registry.db as separate tables (default) — see db-paths.ts if they need to be split. |
In practice, most auxiliary DBs are co-located in registry.db as separate tables (since they're all under the same SQLite file by default). The db-paths.ts resolver controls where each table lives; check that module if you need to split.
registry.db holds the schema that ties everything together. The biggest table is source_systems:
-- staging/api/routes/admin.ts:2341-2365
CREATE TABLE IF NOT EXISTS source_systems (
id INTEGER PRIMARY KEY AUTOINCREMENT,
system_id TEXT UNIQUE NOT NULL,
display_name TEXT NOT NULL,
system_type TEXT NOT NULL CHECK(system_type IN ('authoritative_source','department')),
db_path TEXT,
api_port INTEGER,
api_url TEXT,
api_secret TEXT,
description TEXT,
is_active INTEGER NOT NULL DEFAULT 1,
connector_id TEXT,
access_mode TEXT NOT NULL DEFAULT 'dynamic-pull'
CHECK(access_mode IN ('dynamic-pull','push-cache','scheduled-pull')),
cache_ttl_seconds INTEGER NOT NULL DEFAULT 300,
pull_schedule TEXT,
allow_inbound INTEGER NOT NULL DEFAULT 0,
last_sync_at TEXT,
cache_stale_policy TEXT NOT NULL DEFAULT 'serve-stale',
identifier_mode TEXT NOT NULL DEFAULT 'gcuid'
CHECK(identifier_mode IN ('gcuid','alias')),
created_at TEXT DEFAULT (datetime('now')),
updated_at TEXT DEFAULT (datetime('now'))
)This is the table the API's resolver consults to decide where to route a lookup request. Critical columns:
system_id— the connector ID (e.g.conn-hr,conn-ldap). Stable identifier — never changes after seed.access_mode— which fetching strategy the connector uses (dynamic-pull,push-cache,scheduled-pull).api_url+api_secret— the URL + bearer-token the API uses to call the connector. This is the column with the pre-port-remap bug (fixed ine5c3b8c) — see §1.12.connector_id— currently a free-text field; the JSON config instaging/config/connectors/is the source of truth for runtime wiring.
Other key registry tables:
| Table | DDL location | Purpose |
|---|---|---|
identities |
staging/scripts/seed-data.ts:758-768 |
The central GCUID identity table. One row per identity. Columns: gcuid (PK), entity_type, entity_id, state, identifier_state (V1/V2), created_at, updated_at, metadata (JSON). |
state_history |
staging/scripts/seed-data.ts:769-779 |
The identity-state-transition audit log. One row per state change. Indexed on gcuid. |
identifier_state_history |
staging/api/routes/admin.ts:1989-2002 |
A separate audit table for identifier-state transitions (V1↔V2). Tracks gcuid, from_state, to_state, transition, changed_by, reason, timestamp. |
aliases |
staging/scripts/seed-data.ts:781-792 |
GCUID aliases. One row per alias. Indexed on gcuid and alias_value for fast lookup-by-alias. |
source_mappings |
staging/scripts/seed-data.ts:793-803 |
The map between a GCUID and its source-system record (e.g. gcuid=urn:gcuid:abc123 → conn-hr/employee_id=42). Indexed on gcuid and (source_system, source_id). |
attribute_mappings |
staging/api/routes/caes.ts:59 |
CAES-side attribute mappings. Used by the Common Attribute Exchange Service. |
match_configs |
staging/api/routes/caes.ts:1861 |
Per-source matching configuration (which attributes to match on, what algorithm, what threshold). |
attribute_cache |
staging/api/routes/admin.ts:2387 |
The cache for push-cache and scheduled-pull access modes. Stores (gcuid, attribute_name) → resolved value with TTL. |
hook_subscriptions + hook_events |
staging/api/routes/hooks.ts:60-83 |
The hook subscription registry + emitted-event log. Indexed on event_type, emitted_at, gcuid, subscriber_id. |
app_policies |
staging/api/routes/policies.ts:70 |
Application-level access policies. app_id is PK. |
config (multiple variants) |
flags.ts, log-levels.ts, ux-config.ts |
Generic (key TEXT PK, value TEXT, updated_at) for runtime overrides. Three separate tables by design (different access patterns, different consumers). |
Each connector points at one source DB. The schemas are per-source — HR's employees table may differ from Engineering's. They're seeded by staging/scripts/seed-data.ts (the self-bootstrapping helper).
-- staging/scripts/seed-data.ts:901-918
CREATE TABLE employees (
employee_id TEXT PRIMARY KEY,
gcuid TEXT UNIQUE,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
department TEXT,
job_title TEXT,
manager_gcuid TEXT,
hire_date TEXT,
termination_date TEXT,
-- + extensions per HR config
);
CREATE INDEX idx_employees_dept ON employees(department);
CREATE INDEX idx_employees_email ON employees(email);The employee_id is the primary key that the connector's attributeMappings.primaryKeyField refers to. The gcuid is the linkage back to the registry. The source DB doesn't enforce the gcuid linkage — that's a registry concern; the source DB just stores the gcuid as an opaque string.
LDAP uses a different schema (the LDAP/AD model):
-- users table (LDAP-style)
CREATE TABLE users (
-- username, email, full_name, groups, etc.
);
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_dept ON users(department);Engineering's employees table extends HR's with engineering-specific fields:
CREATE TABLE employees (
-- base fields (employee_id, gcuid, first_name, last_name, email)
hr_employee_id TEXT, -- FK back to HR's employee_id
clearance_level TEXT,
project_codes TEXT, -- comma-separated
...
);
CREATE INDEX idx_eng_email ON employees(email);
CREATE INDEX idx_eng_hr_id ON employees(hr_employee_id);
CREATE INDEX idx_eng_gcuid ON employees(gcuid);Finance's employees table also extends HR's, with finance-specific fields:
CREATE TABLE employees (
-- base fields
hr_employee_id TEXT,
clearance_level TEXT,
cost_center TEXT,
...
);
CREATE INDEX idx_fin_email ON employees(email);
CREATE INDEX idx_fin_hr_id ON employees(hr_employee_id);
CREATE INDEX idx_fin_gcuid ON employees(gcuid);
CREATE INDEX idx_fin_clearance ON employees(clearance_level);The registry schema is idempotent — every table is CREATE TABLE IF NOT EXISTS, every index is CREATE INDEX IF NOT EXISTS. This means:
- Fresh DB → tables created on first API startup
- Existing DB → no-op (preserves data)
- Schema change → add a new
CREATE TABLE IF NOT EXISTSblock; the migration runs on next startup
Column additions are wrapped in try/catch with explicit PRAGMA table_info(...) checks (the pattern at admin.ts:2006-2007). The try/catch + warning-on-error pattern means a read-only bundled DB doesn't prevent module load — it just warns.
The seed migration in admin.ts:2414-2428 initializes source_systems.api_url to the canonical port (5183-5186). The original migration used WHERE api_url IS NULL OR api_url = '' — which is exclusion-based, not validation-based. It caught fresh-DB state but not stale-but-non-empty state from a prior port assignment (the pre-remap ports 8101/8102/8201/8202).
The fix (commit e5c3b8c) extended each UPDATE to also catch the known stale values:
UPDATE source_systems SET api_url = 'http://localhost:5183' WHERE system_id = 'conn-hr'
AND (api_url IS NULL OR api_url = '' OR api_url = 'http://localhost:8101')General rule for any future migration that updates a value with a known-default: enumerate the stale values that pre-date the migration, don't rely on IS NULL OR = '' alone. The runtime-invariant smoke (scripts/smoke-invariants-runtime.ts) catches this class of bug — see §1.12.
Every SQLite file path goes through staging/api/lib/db-paths.ts:
// from staging/api/lib/db-paths.ts
export const DB_BASE: string = efsBase ?? path.join(process.cwd(), 'data');
export const REGISTRY_DB_PATH: string = path.join(DB_BASE, 'registry.db');
export const LOGS_DB_PATH: string = path.join(DB_BASE, 'logs.db');
export const SOURCES_DIR: string = path.join(DB_BASE, 'sources');
export function connectorDbPath(connectorId: string): string {
return path.join(SOURCES_DIR, `${connectorId}.db`);
}Always use these getters in your code rather than hard-coding './data/registry.db' or similar. The path resolver honours GCUID_DB_PATH to switch storage locations automatically; hard-coded paths break this.
For interactive inspection, use sqlite3 (CLI) or the Web UI's DatabasesPortal:
# from /e/GitHub/GCUID-POC
sqlite3 staging/data/registry.db "SELECT * FROM source_systems;"
sqlite3 staging/data/registry.db ".schema"
sqlite3 staging/data/sources/conn-hr.db "SELECT COUNT(*) FROM employees;"For modifications in code, every route uses better-sqlite3 (synchronous). The pattern is db.prepare(sql).get(...) for single rows, db.prepare(sql).all(...) for lists, db.prepare(sql).run(...) for mutations.
The smoke test smoke-cs-3.1-self-bootstrap.ts does NOT touch the registry — it touches the source DBs. Source DBs are regenerated by seed-data.ts --source=<name> if they go missing. The registry (registry.db) is the canonical state for the API; if it gets corrupted, see the SysAdmin Guidebook §3.5 Database management for the reset procedure.
- §1.4 — where
better-sqlite3is used (storage layer) - §1.5 — how routes open the DB
- §1.7 — how connectors read from
data/sources/conn-<id>.db - §1.11 — the runtime-invariant smoke that catches pre-port-remap style bugs
- SysAdmin Guidebook §3.5 Database administration — backup/restore, migrations
- Architect Guidebook §4.7 Data model — why this schema shape
This section is a step-by-step walkthrough of adding a new connector + source pair. The 9-step recipe is in §1.7 (the connector system); this section re-presents it as a worked example with the concrete files you'll create/modify.
Add a new authoritative source conn-it-helpdesk (IT helpdesk) backed by SQLite, port 5183 (the original HR port) — wait, that port is taken. Let me pick a fresh scenario: add conn-facilities (Facilities department) on connector port 5192, source-server port 5193, with a new SQLite DB data/sources/conn-facilities.db.
Note: 5192/5193 are outside the standard 5183-5186 / 5188-5191 allocation. Real additions stay inside the allocation; the example just illustrates the mechanics with unused port numbers.
| What | Value |
|---|---|
| Connector ID | conn-facilities |
| Connector port | 5192 |
| Source ID | src-facilities |
| Source-server port | 5193 |
| Connector token | staging_facilities_token_7777 |
| Source token | gcuid_source_facilities_token_8888 |
| Backing DB | data/sources/conn-facilities.db |
| Primary table | facilities_employees |
| Primary key | facility_employee_id |
Check ports are free: netstat -an | grep 5192 (Windows) or lsof -iTCP:5192 -sTCP:LISTEN (macOS/Linux). If a port is taken, pick the next free one and update the dev-ports registry (.dev-ports.json at the repo root — see SysAdmin Guidebook §3.7 Port management).
Create staging/config/sources/src-facilities.json:
{
"id": "src-facilities",
"label": "Facilities Authoritative Source",
"sourceServer": {
"port": 5193,
"authToken": "gcuid_source_facilities_token_8888"
},
"dataProvider": {
"type": "sqlite",
"dbPath": "./data/sources/conn-facilities.db",
"table": "facilities_employees",
"primaryKey": "facility_employee_id"
}
}Validate: node -e "JSON.parse(require('fs').readFileSync('staging/config/sources/src-facilities.json'))" should print nothing if the JSON is valid.
Create staging/config/connectors/conn-facilities.json:
{
"id": "conn-facilities",
"label": "Facilities Connector",
"connectorServer": {
"port": 5192,
"authToken": "staging_facilities_token_7777"
},
"upstreamSource": {
"baseUrl": "http://localhost:5193",
"authToken": "gcuid_source_facilities_token_8888"
},
"attributeMappings": {
"primaryKeyField": "facility_employee_id",
"tables": {
"facilities_employees": {
"alias": "facility_person",
"primaryKey": "facility_employee_id"
}
}
},
"accessMode": "dynamic-pull"
}The upstreamSource.authToken must match the source's sourceServer.authToken. The connectorServer.authToken is separate (the API authenticates with it).
Either write the schema yourself:
sqlite3 staging/data/sources/conn-facilities.db <<'SQL'
CREATE TABLE facilities_employees (
facility_employee_id TEXT PRIMARY KEY,
gcuid TEXT UNIQUE,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
department TEXT,
building TEXT,
floor TEXT,
office_number TEXT,
manager_gcuid TEXT,
hire_date TEXT
);
CREATE INDEX idx_fac_email ON facilities_employees(email);
CREATE INDEX idx_fac_building ON facilities_employees(building);
SQLOr extend staging/scripts/seed-data.ts with a seedFacilities() function and call tsx staging/scripts/seed-data.ts --source=facilities. The second approach is what HR/LDAP/Engineering/Finance use; it auto-runs via seedSourceIfEmpty() when the source server starts up.
Open staging/api/routes/admin.ts and find the seed migration block around line 2414-2428. Add an UPDATE for conn-facilities:
db.prepare(`
INSERT OR IGNORE INTO source_systems
(system_id, display_name, system_type, api_url, api_secret)
VALUES
('conn-facilities', 'Facilities Department', 'authoritative_source',
'http://localhost:5192', 'staging_facilities_token_7777')
`).run();The INSERT OR IGNORE makes this idempotent — re-running won't error if the row already exists. Important: the api_url here is the connector's URL (port 5192), not the source-server's URL (port 5193).
To mark a specific attribute authoritative for this connector (e.g. make conn-facilities the canonical source for person.security.clearance_level), call CAES after the row exists:
PUT /api/caes/attribute-mappings/<mapping_id>/authoritative
where <mapping_id> is the attribute_mappings.id for the (standard_attribute=person.security.clearance_level, source_system=conn-facilities) row. This sets is_authoritative=1 on that mapping and clears the flag on any other mapping for the same standard_attribute.
staging/scripts/start-connectors.ts and staging/scripts/start-sources-all.ts iterate over the config directories at runtime — new connector/source JSON files are picked up automatically on the next start-dev.sh invocation. Verify the all-launcher finds them with:
ls staging/config/connectors/conn-facilities.json
ls staging/config/sources/src-facilities.json(Replace facilities with whatever your system name is.)
./start-dev.sh stop # or ./stop-dev.sh
./start-dev.sh # or ./run-dev.jsThis picks up the new config files and spawns the new connector + source-server processes. Check the logs for "Facilities Connector bound to 5192" and "src-facilities bound to 5193".
# Connector ping (uses the API's connector-ping route)
curl http://localhost:5182/api/connectors/conn-facilities/ping
# → { reachable: true, base_url: "http://localhost:5192", status: 200, latency: <ms> }
# Source-server health (uses the source token directly)
curl -H "Authorization: Bearer gcuid_source_facilities_token_8888" \
http://localhost:5193/health
# → { sourceId: "src-facilities", status: "ok", recordCount: <n>, latencyMs: <ms> }
# Source-server info (requires auth)
curl -H "Authorization: Bearer gcuid_source_facilities_token_8888" \
http://localhost:5193/info
# → { sourceId: "src-facilities", dbPath: "...", table: "facilities_employees", primaryKey: "..." }
# Lookup via the connector (uses the connector token)
curl -H "Authorization: Bearer staging_facilities_token_7777" \
"http://localhost:5192/records/fac-employee-001"
# → { ... the record ... }
# End-to-end via the API (uses the admin token)
curl -H "Authorization: Bearer gcuid_admin_token_demo" \
"http://localhost:5182/api/connectors/conn-facilities/records/fac-employee-001"
# → { ... same record ... }If any of these return errors, see §1.12 (testing and debugging) for the diagnostic flow.
If users need a dedicated page for Facilities (not just lookups via /api/gcuid), create staging/api/routes/facilities.ts modeled on routes/hr.ts. Register it in staging/api/server.ts:
app.use('/api/facilities', facilitiesRoutes); // Facilities PortalCreate the corresponding page in staging/web-ui/src/pages/FacilitiesPortal.tsx and register a route in App.tsx. This step is optional — many sources are usable via /api/gcuid/<id> without a dedicated portal.
| Symptom | Likely cause |
|---|---|
| Connector doesn't appear in dashboard | JSON syntax error in conn-facilities.json — validate with node -e "JSON.parse(...)" |
ping returns reachable: false |
Token mismatch between connector's upstreamSource.authToken and source's sourceServer.authToken |
ping returns 502 |
Source server isn't running — check start-dev.sh output for it spawning successfully |
Lookups return null |
Source DB is empty — run tsx staging/scripts/seed-data.ts --source=facilities |
Lookups return 500 |
The source server's dataProvider.table doesn't match the actual table name in the SQLite DB |
Registry doesn't have the conn-facilities row |
Migration didn't run — restart ./start-dev.sh (migrations fire on API startup) |
- §1.7 — the connector system architecture (the 9-step recipe lives here too)
- §1.11 — the runtime-invariant smoke catches missing-from-registry bugs automatically
- SysAdmin Guidebook §3.4 Logs and diagnostics — where to find per-process logs
This section is a step-by-step walkthrough of adding a new API endpoint. The example: extend the HR Portal with GET /api/hr/employees/by-department/:dept — a server-side filter that returns all HR employees in a given department.
Add a new route that joins employees (from data/sources/conn-hr.db) with the registry's identities table to return a list of {gcuid, first_name, last_name, email, department, job_title, identity_state} for a given department. Useful for "show me all the HR people in Finance" queries.
The endpoint reads from HR's source DB and the registry. The natural home is staging/api/routes/hr.ts (HR Portal) — it's an HR-flavoured query. If you wanted to filter across all sources, you'd put it in routes/admin.ts (the GCUID Portal).
Open staging/api/routes/hr.ts and add a new handler:
/**
* GET /api/hr/employees/by-department/:dept
*
* Returns all employees in the given department, joined with the
* registry's identities table to expose the GCUID and identity state.
*/
router.get('/employees/by-department/:dept', (req: Request, res: Response, next: NextFunction) => {
try {
const dept = req.params.dept;
if (!dept || dept.length > 64) {
throw createError(400, 'Department name must be 1-64 chars');
}
// Open both DBs (synchronous, better-sqlite3)
const hrDb = new Database(HR_DB, { readonly: true });
const regDb = new Database(REGISTRY_DB, { readonly: true });
try {
// Query: employees in this department + their registry identity
const rows = hrDb.prepare(`
SELECT
e.employee_id, e.gcuid, e.first_name, e.last_name,
e.email, e.department, e.job_title,
i.identifier_state, i.state AS identity_state
FROM employees e
LEFT JOIN identities i ON e.gcuid = i.gcuid
WHERE e.department = ?
ORDER BY e.last_name, e.first_name
`).all(dept);
logHR(HR_EVENTS.EMPLOYEE_DEPARTMENT_QUERY, {
department: dept,
resultCount: rows.length
});
res.json({
department: dept,
count: rows.length,
employees: rows
});
} finally {
hrDb.close();
regDb.close();
}
} catch (err) {
next(err);
}
});Conventions used (matching the rest of routes/hr.ts):
readonly: trueon both DBs — we're not mutating anythingtry/finallywith explicitdb.close()—better-sqlite3is sync but connections still need closing- The
logHR(HR_EVENTS.EMPLOYEE_DEPARTMENT_QUERY, {...})call adds a structured log entry (the log event catalog lives instaging/api/lib/logging/log-events.ts; add a new event ID if the existing one doesn't fit) createError(400, ...)throws a typed error that the global error handler converts to a 400 response- The handler is wrapped in
try/catchcallingnext(err)so the global error handler inmiddleware/error.tssees it
If HR_EVENTS.EMPLOYEE_DEPARTMENT_QUERY doesn't exist, add it to staging/api/lib/logging/log-events.ts. Find the HR_EVENTS block and add:
export const HR_EVENTS = {
// ... existing events ...
EMPLOYEE_DEPARTMENT_QUERY: { id: 'HR-DEPT-QUERY-001', level: 'info', category: 'hr' },
} as const;The id is human-readable (matches the catalog numbering convention); level is one of info/warn/error/debug; category is the broad bucket (used for filtering in the log viewer).
./start-dev.sh stop
./start-dev.shThe API doesn't hot-reload — you need a restart for new routes. (The Web UI hot-reloads automatically, but we're adding an API route, not a UI route.)
# Direct curl
curl http://localhost:5182/api/hr/employees/by-department/Engineering
# → { department: "Engineering", count: 4, employees: [...] }
# Wrong department
curl -i http://localhost:5182/api/hr/employees/by-department/XYZ
# → 404 (no employees in XYZ)
# Bad input (empty after URL decode)
curl -i "http://localhost:5182/api/hr/employees/by-department/"
# → 404 (Express doesn't match the empty path)For browser-accessible testing, the Web UI's TestingPortal page is the canonical UI. Add a button or form there if you want users to trigger it from the UI.
If the Web UI is going to call this endpoint, add a method to staging/web-ui/src/api/client.ts:
async listHrEmployeesByDepartment(department: string): Promise<{
department: string;
count: number;
employees: Array<{...}>;
}> {
return this.request(`/api/hr/employees/by-department/${encodeURIComponent(department)}`);
}Then call it from your page/component:
import { api } from '../api/client';
const result = await api.listHrEmployeesByDepartment('Engineering');The encodeURIComponent is important — department names with spaces, slashes, or non-ASCII need URL encoding.
Some routes check the bearer token. The pattern (from routes/admin.ts):
const ADMIN_TOKEN = 'gcuid_admin_token_demo';
function requireAdmin(req: Request, res: Response, next: NextFunction) {
const auth = req.headers.authorization;
if (auth !== `Bearer ${ADMIN_TOKEN}`) {
return res.status(401).json({ error: 'unauthorized' });
}
next();
}
// Use it on specific routes
router.get('/admin/something', requireAdmin, (req, res) => { ... });For the HR Portal's department query, read-only is the default — no auth check needed (matches the other HR routes). If you need admin-only, add requireAdmin as middleware on the specific route.
If you're adding a brand-new resource (e.g. /api/audit-log doesn't exist yet), create staging/api/routes/audit-log.ts:
import { Router, Request, Response, NextFunction } from 'express';
const router = Router();
router.get('/', (req, res) => { /* list */ });
router.get('/:id', (req, res) => { /* get one */ });
export const auditLogRoutes = router;Then register it in staging/api/server.ts:
import { auditLogRoutes } from './routes/audit-log.js';
// ...
app.use('/api/audit-log', auditLogRoutes);Restart, and /api/audit-log is live.
| Mistake | Symptom | Fix |
|---|---|---|
| Forgot to restart the API | New route returns 404 | ./start-dev.sh stop && ./start-dev.sh |
Used app.use(...) instead of router.get(...) |
All routes inside that file return 404 | Use the exported router object |
Forgot try/finally around Database |
Process exits with "too many open files" under load | Always db.close() in finally |
| Typo in route path | Browser console shows 404 on the API path | Check the registration in server.ts AND the handler's path string |
Used ' in a SQL string |
SQLite throws a syntax error | Use ? placeholders with .all(arg) or template literals for identifiers |
| Returning raw DB rows | UI gets Date objects as strings (timestamps) |
Cast dates to ISO strings before returning |
- §1.5 — the API layer (where new routes plug in)
- §1.6 — the API client (where new client methods go)
- §1.9 — the database schema (what tables to query)
- §1.12 — testing and debugging the new endpoint
- SysAdmin Guidebook §3.4 Logs and diagnostics — where your log event lands
This section is the operational reference for testing changes and diagnosing failures. It covers the smoke test surface (smoke tests, unit tests, runtime-invariant checks), the debugging workflow (where to look when something goes wrong), and the diagnostic patterns the project uses (PID-scoped cleanup, log-first debugging, config-file vs. state).
The POC has three distinct testing layers, each with a different scope:
| Layer | What it tests | How to run | When to use |
|---|---|---|---|
| Unit tests (Jest) | Functions and modules in packages/core/src/ (e.g. Crockford encoding, state-machine transitions, fuzzy matching) |
npm test (at repo root) — runs Jest across the workspace |
After changing any @gcuid/core function. Catches logic bugs. |
Smoke tests (scripts/smoke-*.ts) |
End-to-end behavior — source-server liveness, connector API, the runtime invariants (registry data + UI-equivalent API) | ./run-smokes.sh (or npm run smoke) — runs the default 2/5 smokes |
After changing anything that touches the running stack. Catches wiring bugs. |
Runtime invariants (scripts/smoke-invariants-runtime.ts) |
Cross-layer consistency: registry data matches connector configs matches port registry; pings succeed against the live stack | ./run-smokes.sh invariants (or included in default npm run smoke) |
After any DB migration, port remap, or config change. Catches pre-port-remap-class bugs. |
The Jest tests live in packages/core/__tests__/ and packages/core/src/**/__tests__/. The current count:
- 21 tests pass
- 4 tests fail (
hr.test.ts— relative path issues, pre-existing)
Run from the repo root: npm test. Or from inside packages/core: jest.
The Jest config is in packages/core/jest.config.js. Tests use the standard Jest pattern (describe + it + expect).
Five smoke scripts under scripts/:
| Smoke | What it tests | Runs by default? |
|---|---|---|
smoke-cs-1.1-hr-source.ts |
Spawns an HR connector on port 5183 from scratch, verifies it serves a record | ❌ No (needs port isolation — run with npm run smoke:all) |
smoke-cs-2.1-all-sources.ts |
Spawns all 4 connectors + all 4 sources, verifies the full topology | ❌ No (needs port isolation) |
smoke-cs-3.1-self-bootstrap.ts |
Mutates the source DBs to verify seedSourceIfEmpty() reseeds them on next startup |
❌ No (mutates state) |
smoke-cs-4.1-startup-cycle.ts |
Structural smoke — verifies the startup scripts discover configs and the orchestration pipeline works, without actually spawning processes | ✅ Yes (default orchestrator) |
smoke-invariants-runtime.ts |
The 39-assertion runtime invariant check (see below) | ✅ Yes (default orchestrator) |
Run via:
# Default (PID-safe, runs against the live stack)
./run-smokes.sh
# or
npm run smoke
# List available smokes
./run-smokes.sh --list
# or
npm run smoke:list
# Run everything including port-conflicting smokes
./run-smokes.sh --all
# or
npm run smoke:all
# Run a specific smoke
./run-smokes.sh invariantsThe orchestrator is scripts/run-smokes.js. It discovers every scripts/smoke-*.ts automatically, runs them in sequence, and aggregates pass/fail. Exit code is non-zero if any smoke fails.
Three smokes are excluded by default (the STACK_CONFLICTING constant in run-smokes.js): cs-1.1, cs-2.1, cs-3.1. They conflict with the user's running stack on the same ports or mutate state. Use npm run smoke:all to run them, but only when you're OK with the stack being down/mutated.
scripts/smoke-invariants-runtime.ts is the smoke that catches pre-port-remap-class bugs. It has 39 assertions across 3 sections:
Section A — Stack liveness (9 assertions)
TCP-probes each service port: API (5182), 4 connectors (5183-5186), 4 source servers (5188-5191). If any port is closed, the smoke fails with port 5183 not listening.
Section B — Registry data integrity (8 assertions)
For each connector, reads registry.db and connector.json, then asserts:
source_systems.api_urlmatchesconnector.json.connectorServer.portsource_systems.api_secretmatchesconnector.json.connectorServer.authTokenconnector.json.connectorServer.portis in.dev-ports.json(the port registry)- The source config exists at
staging/config/sources/src-<id>.json - The source config's
portis also in.dev-ports.json - The source config's
authTokenmatches the connector'supstreamSource.authToken
Section C — UI-equivalent probe (4 + 4 = 8 assertions)
For each connector, calls:
GET /api/connectors/<id>/ping— assertsreachable=true,base_urlmatches expected portGET /api/connectors/<id>/health— assertsstatus=healthy, noerrorfield
Total: 39 assertions that catch the class of bug where the code is right but the data/config is stale.
How it caught the pre-port-remap bug: when source_systems.api_url was http://localhost:8101 (the pre-remap value) instead of http://localhost:5183, the smoke failed at the Section B assertion that api_url must match the connector's port (5183), with a clear expected 5183, got 8101 message.
The smokes do NOT kill processes by port. They kill by PID-scoped taskkill:
// from scripts/smoke-cs-3.1-self-bootstrap.ts
function cleanupProcesses(childPids: number[]) {
for (const pid of childPids) {
spawnSync('taskkill', ['/T', '/F', '/PID', String(pid)]);
}
}Why PID-scoped, not port-based? Port-based cleanup (e.g. Get-NetTCPConnection | Where-Object LocalPort -in 5183-5186 | Stop-Process) would kill ANY process listening on those ports — including the user's own running stack. The smoke isn't supposed to fight the user's stack; it should only clean up its own children.
The default orchestrator excludes smokes that need port isolation (cs-1.1, cs-2.1, cs-3.1) so the default run is observation-only.
When something doesn't work, follow this diagnostic ladder:
# Quick check
curl http://localhost:5182/health
# or (more detailed)
./run-smokes.sh invariantsIf the API isn't responding, check data/logs.db (the API's log table). If the API process isn't running at all, check ./start-dev.sh output and look for spawn errors.
For a connector/source, probe its port directly:
# Connector
curl http://localhost:5183/health
# Source server
curl -H "Authorization: Bearer gcuid_source_hr_token_***" \
http://localhost:5188/healthIf the port is closed but the API can see it, the connector died after startup. Check data/logs.db for the connector's last log entry.
For pre-port-remap-class bugs:
sqlite3 staging/data/registry.db "SELECT system_id, api_url, api_port, api_secret FROM source_systems;"Compare with the JSON in staging/config/connectors/conn-<id>.json. They should match.
Trace a single request via the logs:
sqlite3 staging/data/logs.db "SELECT timestamp, component, event_id, level, message FROM log_events ORDER BY timestamp DESC LIMIT 30;"Look for the request in the log — which component handled it, which event ID, what was the response.
Test the API directly with curl (skipping the UI):
curl http://localhost:5182/api/connectors/conn-hr/pingIf the API works but the UI doesn't, the issue is in staging/web-ui/src/api/client.ts or the page's component. Open the browser dev tools Network tab and check the actual request/response.
React Query caches aggressively. Force a refresh:
- Browser: Cmd-Shift-R (hard reload)
- React Query:
queryClient.invalidateQueries(...)in the dev console
If the API returns fresh data but the UI shows stale, the React Query cache key is wrong. See §1.6 for the query-key pattern.
Three log sources to know about:
| Source | What it logs | How to query |
|---|---|---|
data/logs.db (log_events table) |
Every API request, every connector health check, every source-server event | sqlite3 staging/data/logs.db "SELECT * FROM log_events ORDER BY timestamp DESC LIMIT 30;" |
| Per-process stdout | The connector/source-server processes log to stdout (captured by start-connectors.ts/start-sources-all.ts PID files). |
cat staging/data/logs/.connectors.log or similar (paths are in the start script config) |
logs.db via the Web UI |
The log viewer (/logs page) queries log_events with filters |
Open http://localhost:5181/logs |
The structured log event catalog is staging/api/lib/logging/log-events.ts. Every event ID maps to a category + level. When debugging, search by event category first (hr, connector, source, api), then by event ID, then by message text.
| Symptom | Likely cause | Fix |
|---|---|---|
| Stack says it's up but API returns 502 | Source server isn't bound on its port | Check start-dev.sh output; restart |
ping returns reachable: false base_url: 8101 |
Pre-port-remap-style stale port in source_systems.api_url |
Run ./run-smokes.sh invariants to confirm; restart API to fire the migration |
| API returns 200 but UI shows old data | React Query cache key mismatch | Hard-reload browser; check query key in dev tools |
| New route returns 404 | Forgot to restart API | ./start-dev.sh stop && ./start-dev.sh |
| Source DB is empty | seedSourceIfEmpty() skipped (DB existed but was empty) |
Manually run tsx staging/scripts/seed-data.ts --source=<id> |
| Connector process dies silently | Startup error (port in use, bad config, missing DB) | Check data/logs.db for the component's startup log line |
npm run build fails with TS errors in packages/core/ |
Type mismatch between staged TS and staged code | Run npm run build --workspace=@gcuid/core to see the specific errors |
Create scripts/smoke-<topic>.ts:
#!/usr/bin/env -S npx tsx
import assert from 'node:assert/strict';
async function smokeTest() {
// ... your assertions ...
assert.equal(actual, expected, 'message');
console.log(' ✓ assertion passed');
}
smokeTest().then(() => {
console.log('PASS: smoke-<topic>');
}).catch((err) => {
console.error('FAIL:', err.message);
process.exit(1);
});The orchestrator (scripts/run-smokes.js) picks it up automatically. If your smoke needs port isolation or mutates state, add it to the STACK_CONFLICTING array (it'll be excluded from default runs).
- §1.5 — how to add a new API endpoint to test
- §1.10 — how to add a new connector (the worked example for what to smoke)
- §1.11 — how to add a new API endpoint (the worked example for what to smoke)
- Tester Guidebook — the test authoring reference
- SysAdmin Guidebook §3.4 Logs and diagnostics — log event catalog
- SysAdmin Guidebook §3.5 Database administration — debugging DB issues
This section is the build pipeline reference — how the workspace packages compile, how the staging runtime assembles, and how the Web UI bundles for production.
The repo root has a package.json with workspace-aware scripts:
{
"scripts": {
"build": "npm run build --workspaces",
"test": "npm run test --workspaces",
"clean": "npm run clean --workspaces && rm -rf node_modules",
"smoke": "node scripts/run-smokes.js",
"smoke:list": "node scripts/run-smokes.js --list",
"smoke:all": "node scripts/run-smokes.js --all"
}
}npm run build --workspaces builds each package in dependency order:
@gcuid/core(no internal deps) →packages/core/dist/@gcuid/federation(depends on core) →packages/federation/dist/@gcuid/attributes(depends on core + federation) →packages/attributes/dist/@gcuid/api(depends on core + federation + attributes) →packages/api/dist/
The staging API imports packages/core/dist/connector-api.js and packages/core/dist/source-api.js directly via require() paths, so it picks up the rebuilt code on next start.
cd staging/web-ui
npm run build
# → staging/web-ui/dist/ (static HTML/JS/CSS)The Vite build produces a static bundle. The staging API serves this directly when NODE_ENV=production:
// from staging/api/server.ts (production mode block)
if (WEB_UI_DIST_EXISTS) {
app.use(express.static(WEB_UI_DIST));
app.get('*', (req, res) => res.sendFile(path.join(WEB_UI_DIST, 'index.html')));
}In dev mode (./start-dev.sh), Vite serves on :5181 with the /api proxy to :5182 — no static bundle needed.
staging/api/— TypeScript files run viatsx(no compile step in dev). For production you'dtscthem intodist/. The current POC skips this since it's local-only.staging/cli/— Separate CLI executable. Builds vianpm run build(its ownpackage.json).staging/web-ui/— Vite bundles todist/onnpm run build.staging/scripts/— Run viatsxdirectly. No build step.
# Just @gcuid/core
npm run build --workspace=@gcuid/core
# Just @gcuid/attributes
npm run build --workspace=@gcuid/attributesUseful when you've only changed one package and don't want to rebuild the whole workspace.
If a build is in a bad state:
# Clean all workspace packages
npm run clean --workspaces
# Clean one
npm run clean --workspace=@gcuid/core
# Nuclear option: blow away node_modules and reinstall
rm -rf node_modules packages/*/node_modules
npm installThe POC is local-only by design. The README's "local-only" notice is the binding constraint.
What this means in practice:
- Don't add Dockerfile or
deploy.ymlfiles — they imply production deployment and conflict with the local-only stance. - Don't add HTTPS termination code — the local stack runs HTTP only.
- Don't add production middleware (helmet, rate-limit, csrf) — the API is local-only.
- Do keep the dev scripts working —
./start-dev.sh,./run-smokes.sh,./stop-dev.share the canonical local workflow.
| If you're adding... | Put it in... | Why |
|---|---|---|
| A new identifier format | packages/core/src/index.ts (GCUIDManager) or packages/core/src/crockford-base32.ts (encoding) |
These are library code; multiple consumers may want it |
| A new connector HTTP route | packages/core/src/connector-api.ts |
It's the connector HTTP server's surface |
| A new source HTTP route | packages/core/src/source-api.ts |
Same, for sources |
| A new staging API endpoint | staging/api/routes/<resource>.ts (existing or new file) + register in server.ts |
The staging layer is the consumer-facing API |
| A new page | staging/web-ui/src/pages/<Name>Portal.tsx + register route in App.tsx |
Each page is its own file |
| A new operational script | start-dev.sh (start) / stop-dev.sh (stop) / run-smokes.sh (smokes) / scripts/smoke-<topic>.ts (smoke) |
Canonical script layout |
| A new docs file | docs/<topic>.md |
Operational docs live in docs/, code-adjacent docs can live next to the code |
- §1.4 — the workspace packages and how they compile
- §1.5 — the staging runtime (where staging imports
dist/from) - §1.8 — the third-party dependency catalog
- §1.12 — how to test the build (smokes + invariants)
- SysAdmin Guidebook §3.1 Local install — operational counterpart
- Architect Guidebook §4.8 Why local-only — the local-only rationale
- User Guidebook — what end users see
- SysAdmin Guidebook — operating the stack
- Architect Guidebook — design intent and trade-offs
- Tester Guidebook — test surface and how to add tests