This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
flAPI is a SQL-to-API framework that automatically generates REST APIs and AI-compatible tools from SQL templates and YAML configurations. Instead of writing backend code to expose data, analysts write SQL queries and endpoint configurations, and flAPI handles all boilerplate (HTTP server, parameter validation, caching, auth, rate-limiting, MCP protocol).
Core Value Proposition:
- No backend coding: REST API emerges from SQL + YAML configuration
- Data analyst-friendly: Uses SQL (familiar skill) instead of requiring backend developers
- Multi-protocol: REST and MCP (Model Context Protocol) equally supported
- Built-in intelligence: Validators, caching (DuckLake), authentication, rate-limiting
- DuckDB-powered: Access 50+ data sources (BigQuery, Postgres, S3, Snowflake, etc.)
Key Characteristics:
- Written in modern C++20 with zero runtime dependencies (single-binary deployment)
- Supports Linux (x86/ARM64), macOS (Intel/Apple Silicon), and Windows
- Declarative API philosophy: logic lives in YAML/SQL, not compiled code
- Single binary deployment with built-in DuckDB 1.5.5
Start at docs/README.md — the task-first documentation index.
Architecture and design (the "how it works" tree):
- docs/spec/ARCHITECTURE.md - System architecture overview with component diagrams
- docs/spec/DESIGN_DECISIONS.md - Rationale for key design choices
- docs/spec/REQUEST_LIFECYCLE.md - End-to-end request flow with sequence diagrams
- docs/spec/components/ - Component-level documentation:
- config-system.md - Configuration, copy-on-write endpoints
- query-execution.md - SQL templates and DuckDB
- caching.md - DuckLake caching system
- mcp-protocol.md - MCP server, SEP-414 trace context
- security.md - Auth, validation, telemetry egress
- observability.md - Request identity, spans, capture tiers
Task-first guides (docs/guides/): getting-started, rest-endpoints, mcp-tools,
caching, authentication, cloud-storage, yaml-includes, self-packaging,
serverless, claude-code.
flAPI ships a Claude Code plugin from this repo (.claude-plugin/marketplace.json
-> plugins/flapi/): 10 skills (skills/flapi-*/SKILL.md), 7 slash commands
(commands/), an MCP server entry (.mcp.json) and a validate-and-reload hook
(hooks/). Load it in-repo with claude --plugin-dir ./plugins/flapi.
- Update the skill in the same PR as the feature. A new config key, CLI flag, template variable or behaviour change is not done until the skill that teaches it says so.
make check-skills(CI:cli-integrationjob) checks every YAML key against the parser, everyflapi/flapiicommand and option against--help, every{{cache.*}}variable, the version stamp, and runs every tagged example through--validate-configand a live server. Tag runnable examples (# flapi: endpoint group=g file=x.yaml,-- flapi: template ...,# flapi: check GET /x -> 200); seescripts/check_skills.py.- Version:
plugin.json, every SKILL.mdversion:and**flAPI:** v...banner equal the newest## vX.Y.Zheading in CHANGELOG.md. Bump them in the release-prep commit. - Deprecated or wrong facts the skills must not teach are listed in
scripts/skill_snippets/forbidden.txt.
Reference:
- docs/CONFIG_REFERENCE.md - Configuration file format
- docs/CLI_REFERENCE.md - CLI commands
- docs/MCP_REFERENCE.md - MCP protocol details
- docs/CONFIG_SERVICE_API_REFERENCE.md - Runtime configuration API
- docs/OBSERVABILITY.md - Tracing, audit, log correlation
docs/archive/ holds completed plans and shipped design notes. Not maintained
— never cite it as current behaviour.
# Build both debug and release versions
make all
# Build specific configuration
make debug # Debug version (faster compilation, more debugging info)
make release # Optimized release build
# Clean all build artifacts
make clean
# Run with example configuration
make run-debug # Run debug binary with examples/flapi.yaml
make run-release # Run release binary with examples/flapi.yaml# Run all C++ unit tests
make test
# Run integration tests (requires Python + tavern/pytest)
make integration-test # All integration tests
make integration-test-rest # REST API tests (Tavern)
make integration-test-mcp # MCP protocol tests
make integration-test-ducklake # Cache/DuckLake tests
make integration-test-ci # Integration tests with server management
# Run all tests (unit + integration)
make test-all
# Setup Python integration test environment
make integration-test-setup # Install Python dependencies# Run a single C++ test using ctest
cd build/release && ctest -V -R "test_name_pattern"
# Run a single Python/Tavern test
cd test/integration && pytest test_name.tavern.yaml -v
# Run a specific Python test function
cd test/integration && pytest test_mcp_integration.py::test_function_name -v# Full development cycle
make debug # Fast iterative builds
make run-debug # Start server with examples
# Edit code and rebuild: make debug
# Before committing
make test-all # Run full test suite
make clean # Clean artifactsRequest Processing Pipeline:
HTTP Request → APIServer (Crow)
↓
[Middleware: CORS, RateLimit, Auth]
↓
RequestHandler: Parse params → Validate → Check cache
↓
QueryExecutor: Render SQL template (Mustache) → Execute on DuckDB
↓
CacheManager: Check/update cache (if configured)
↓
Response: JSON serialization → HTTP response
MCP Protocol Integration:
MCP Client → MCPRouteHandlers (JSON-RPC)
├→ initialize, list_tools, call_tool, list_resources, read_resource
└→ MCPToolHandler: Convert REST endpoints to MCP tools
Configuration Loading:
flapi.yaml (main config)
├→ connections: Define data sources
├→ duckdb: Engine settings
├→ endpoints: Directory with endpoint configs
└→ sqls/endpoint_name.yaml
├→ url-path, request validators
├→ template-source: SQL file with Mustache syntax
├→ cache configuration (TTL, refresh strategy)
└→ auth/mcp tool definitions
Layer 1: User/Config Layer
- YAML endpoint configurations
- SQL templates with Mustache syntax
- User HTTP/MCP requests
Layer 2: Request Processing
- Parameter extraction and validation
- Request routing to endpoint handler
- Template expansion with user parameters
Layer 3: Query Execution
- Mustache template to SQL rendering
- DuckDB query execution
- Result formatting
Layer 4: Response Handling
- JSON serialization
- HTTP response formatting
- Optional caching (DuckLake)
| Component | Location | Purpose |
|---|---|---|
| APIServer | src/api_server.cpp |
HTTP server (Crow framework), routing, middleware |
| ConfigManager | src/config_*.cpp |
YAML parsing, endpoint discovery, validation |
| DatabaseManager | src/database_manager.cpp |
DuckDB connection pooling, extension management |
| RequestHandler | src/request_handler.cpp |
HTTP request processing, parameter validation |
| QueryExecutor | src/query_executor.cpp |
SQL template rendering and execution |
| CacheManager | src/cache_manager.cpp |
DuckLake caching: TTL, refresh, materialization |
| AuthMiddleware | src/auth_middleware.cpp |
JWT/Basic auth validation |
| RateLimitMiddleware | src/rate_limit_middleware.cpp |
Request rate limiting |
| MCPToolHandler | src/mcp_*.cpp |
Model Context Protocol server implementation |
A connection defines access to a data source (file, database, cloud storage, API).
Structure:
connections:
my-data:
properties:
path: './data/customers.parquet' # File-based
# OR
host: 'db.example.com' # Database
port: '5432'
database: 'mydb'
user: 'read_user'
password: '{{env.DB_PASSWORD}}' # Environment variable (whitelisted under template:)Key Points:
- Properties are custom per connection type (file path vs. database credentials)
- Environment variables supported via
{{env.VAR_NAME}}(must matchtemplate.environment-whitelistin flapi.yaml; an unlisted one stops startup; empty whitelist = none;${VAR}is NOT supported - only{{env.VAR}}, with optional spaces inside the braces) - Loaded once at startup, reused for all requests
- Available in templates as
conn.property_name
An endpoint is a callable API operation (REST or MCP).
Structure:
url-path: /customers
method: GET # GET, POST, PUT, DELETE, PATCH
request:
- field-name: id
field-in: query # or path, body, header
required: false
validators:
- type: int
min: 1
template-source: customers.sql # SQL file name
connection: [my-data] # Connection(s) to useRequest Lifecycle:
- HTTP request arrives with parameters
- Parameters extracted per
field-in(query/path/body/header) - Validators applied (fail fast with 400 if invalid)
- Template expanded with validated parameters
- SQL executed against DuckDB
- Results formatted as JSON
- Response sent
Templates are Mustache files that generate SQL from request parameters.
Available Variables:
params.*- Request parameters (query, path, body, header)conn.*- Connection propertiescache.*- Cache metadata (if cache enabled)env.*- Whitelisted environment variables
Key Rule: Triple vs. Double Braces
- Triple braces
{{{ }}}for strings: Renders the raw value (no HTML entity escaping). Use inside single-quoted SQL string literals. - Double braces
{{ }}for numbers/identifiers: HTML-escapes the value (turns<into<etc.), which is harmless but not SQL-aware — use only where the value is a number or known-safe identifier.
Security note: Neither form performs SQL-specific escaping. Mustache does not understand SQL string literals, quote-doubling, or comment syntax. Defense against injection comes from the
RequestValidator(typed fields, regex/range/enum checks) and disciplined template authoring (quote string params, parameterise numerics). When in doubt, add a stricter validator rather than relying on rendering.
Example Template:
SELECT * FROM read_parquet('{{{ conn.path }}}')
WHERE 1=1
{{#params.id}}
AND customer_id = {{{ params.id }}}
{{/params.id}}
{{#params.min_price}}
AND price >= {{ params.min_price }}
{{/params.min_price}}
ORDER BY created_at DESCPagination owns
limit/offset. GET endpoints are paginated by default (with-pagination: true, 100 rows per page, body{next, total_count, data}), and?limit=/?offset=are reserved for it. Never writeLIMIT {{ params.limit }}in a template: it applies before flAPI's offset, so?limit=2&offset=2returnsdata: [].
Validators enforce input constraints before SQL execution.
Common Types:
validators:
- type: int
min: 1
max: 999999
- type: string
min-length: 1
max-length: 200
regex: "^[a-zA-Z0-9_]+$"
- type: email
- type: uuid
- type: enum
allowedValues: ["active", "inactive", "pending"]
- type: date
min: "2020-01-01"
max: "2025-12-31"Security Strategy (Defense in Depth):
- Validators: First line (whitelist validation)
- Quoting discipline in the template: triple braces inside single-quoted SQL literals. Note this is NOT escaping — see the security note in §3.
- Never trust user input even with both layers
DuckLake is a table versioning system for snapshot-based caching.
Cache Modes:
- Full Refresh: Recreate entire table each refresh (simple, slow for large tables)
- Incremental Append: Only add new rows (fast for append-only data)
- Incremental Merge: Insert/update/delete handling (complex, handles all changes)
Configuration:
cache:
enabled: true
table: customers_cache
schedule: "6h" # How often to refresh
primary-key: [id] # For merge mode
cursor:
column: updated_at # Tracks changes
type: timestampCache Template Variables:
{{cache.table}}- Cache table name{{cache.schema}}- Schema name{{cache.previousSnapshotTimestamp}}- Last refresh time{{cache.snapshotTimestamp}}- This refresh time ({{cache.currentSnapshotTimestamp}}does not exist and renders empty)
Snapshot Lifecycle:
Time 0: Cache initialized with full refresh
→ Run populate SQL
→ Store all rows, create snapshot v1
Time 6h: Schedule triggers
→ Run populate SQL (incremental)
→ MERGE changed rows into cache
→ Create snapshot v2 (only changes stored)
Time-travel: Query any previous snapshot
→ SELECT * FROM cache AS OF TIMESTAMP '...'
Endpoint Configuration Structure (sqls/endpoint_name.yaml):
# REST endpoint definition
url-path: /customers # URL path
method: GET # HTTP method
# Request parameters
request:
- field-name: id
field-in: query # query, path, header, body
required: false # the type comes from the validator; `field-type` has no effect (startup warns)
validators:
- type: int
min: 1
# SQL execution
template-source: customers.sql # Relative to sqls/ dir
connection: [data-source-name] # From connections in flapi.yaml
# Caching (optional)
cache:
enabled: true
schedule: 6h # how often to refresh
# Mode is INFERRED: no primary-key/cursor = full refresh;
# cursor only = append; primary-key + cursor = merge.
table: customers_cache # Cache table name
# MCP tool definition (optional)
mcp-tool:
description: Get customer information
input_schema:
type: object
properties:
id:
type: integer
description: Customer ID
# Authorization (optional). Without this block the top-level `auth:` of
# flapi.yaml applies; this block REPLACES it (no merge); `auth: {enabled: false}`
# opts this endpoint out.
auth:
enabled: true
type: basic # basic | bearer (alias: jwt) | oidc
roles: [admin, user] # caller needs one of them, else 403; empty = any authenticated callerSQL Template with Mustache (sqls/endpoint_name.sql):
SELECT * FROM read_parquet('{{{ conn.path }}}')
WHERE 1=1
{{#params.id}}
AND customer_id = {{ params.id }}
{{/params.id}}
{{#params.status}}
AND status = '{{{ params.status }}}'
{{/params.status}}Template variables available:
params.*: Query/path/body parameters from requestconn.*: Connection properties (paths, credentials) of the endpoint's first connectionauth.*: Authentication context (auth.username,auth.roles,auth.email,auth.type,auth.authenticated)context.conn.*/context.auth.*do not exist and render as empty strings
The same flapi binary that serves the API can also fold an entire
config tree into itself, producing a self-contained executable
deployable via scp.
# Pack a config tree into a new bundled binary
flapi pack --in ./examples --out flapi-prod
# Inspect what's bundled
./flapi-prod info
# Extract the bundle for debugging
./flapi-prod unpack --to /tmp/extracted
# Run it -- serves the bundled config from any cwd
cd /tmp && ./flapi-prodHow it works:
- A ZIP archive is appended after the executable (or, on macOS,
written into a reserved
__FLAPI/__bundleMach-O segment that was allocated at link time -- 16 MiB default, knobFLAPI_RESERVED_BUNDLE_MIB). Mach-O is re-codesigned so the output is notarisable. - At startup,
bundle_locatoreither reverse-scans the EOCD signature from EOF (Linux / Windows) or probes the reserved section (macOS). On hit, entries are decompressed once into a sharedArchiveEntries. EmbeddedArchiveFileProvider(implementsIFileProvider) serves config / SQL templates from that map.FileProviderFactorydispatches non-remote paths to it when a bundle is present.- For SQL templates that use
read_csv()/read_parquet(), anEmbeddedFileSystemis registered with DuckDB on theembed://scheme, soread_csv('embed://data/cities.csv')resolves to the same in-memory bytes.
Extensions can go in the bundle. flapi pack --with-extensions installs
the config's DuckDB extensions (host platform) and embeds them under
.flapi/extensions/. At startup they are extracted once to
$TMPDIR/flapi-extensions/ext-<hash>/, used as extension_directory, and
offline extensions are on (FLAPI_EXTENSIONS_OFFLINE=0 overrides).
Secrets never go in the bundle. pack refuses files matching
*.env, secrets/*, *.pem, *.key by default. Credentials come
from environment variables at runtime (AWS_*, GOOGLE_*, AZURE_*,
FLAPI_CONFIG_SERVICE_TOKEN, {{env.VARNAME}} interpolation in
YAML). See DESIGN_DECISIONS §9
for the rationale and CLI_REFERENCE §3
for full subcommand options.
Reproducibility. Set SOURCE_DATE_EPOCH (epoch seconds) before
flapi pack and the output is bit-identical across runs.
Use WHERE 1=1 with conditional sections to safely build dynamic queries:
SELECT * FROM table
WHERE 1=1
{{#params.filter1}}
AND column1 = '{{{ params.filter1 }}}'
{{/params.filter1}}
{{#params.filter2}}
AND column2 = {{{ params.filter2 }}}
{{/params.filter2}}
ORDER BY created_at DESCWhy This Works:
WHERE 1=1allows adding any number of AND conditions- Each condition wrapped in conditional section (only rendered if parameter exists)
- No
LIMIT: default pagination adds it (see the warning in §3) - Triple braces for strings, double for numbers
- Injection defence is the validator (plus prepared-statement binding for a bare
{{ params.x }}outside sections); neither brace form escapes SQL
Conditional Rendering:
{{#params.name}}
-- Only rendered if params.name exists and is truthy
AND name = '{{{ params.name }}}'
{{/params.name}}
{{^params.name}}
-- Rendered if params.name does NOT exist
AND name IS NULL
{{/params.name}}Default Values:
AND region = '{{#params.region}}{{{ params.region }}}{{/params.region}}{{^params.region}}EU{{/params.region}}'
-- Use params.region if provided, otherwise default to EU(Not for limit/offset: those are reserved for pagination.)
Connection Properties:
SELECT * FROM read_parquet('{{{ conn.path }}}')
-- Access connection properties defined in flapi.yamlCache Metadata (Incremental Refresh):
INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}
SELECT * FROM source
{{#cache.previousSnapshotTimestamp}}
WHERE updated_at > TIMESTAMP '{{{cache.previousSnapshotTimestamp}}}'
{{/cache.previousSnapshotTimestamp}}
-- Only refresh rows changed since last snapshotNever include Claude Code attribution in commit messages.
Do NOT add lines like:
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Rationale: Work is owned and authored by the human developer, even when AI assistance is used. AI is a tool, not a co-author.
Commit messages should follow this format:
feat|fix|chore|docs|test: Brief description (imperative mood)
- Bullet point explaining what changed
- Why it changed (if not obvious)
- Any relevant issue references (#123)
Examples:
feat: Add VFS support for cloud storage configuration
- Enable S3, GCS, Azure paths in config files
- Implement PathSchemeUtils for scheme detection
- Add LocalFileProvider and DuckDBVFSProvider
chore: Update documentation for reference structure
- Consolidate authentication references
- Add cross-reference sections to all docs
- Create REFERENCE_MAP.md for navigation
fix: Correct template expansion in DuckDB queries
- Use triple braces for string parameters
- Fix null handling in optional fields
- Closes #42
File and Naming:
- Files:
lowercase_with_underscores.cppand.hpp - Classes:
PascalCase - Functions:
PascalCase - Variables:
snake_case - Constants:
SCREAMING_SNAKE_CASE - Member variables: prefix with
_(e.g.,_userId)
Modern C++ Practices:
- Use
std::unique_ptrovershared_ptr; usenew/deleteonly in exceptional cases - Always use
constand references where appropriate - Use
constreferences for non-trivial objects:const std::vector<T>& - Avoid namespace imports:
std::string, notusing std - Use C++11 range-based for loops:
for (const auto& item : items) - Use
std::optional,std::variantfor type-safe alternatives
Class Layout:
class MyClass {
public:
MyClass();
int public_variable;
public:
void MyMethod();
private:
void PrivateMethod();
private:
int _private_variable;
};Important Rules:
- All functions in
src/directory should be in theduckdbnamespace - Use
overridewhen overriding virtual methods - Use
[u]int(8|16|32|64)_tinstead ofint,long,uint - Use
idx_tinstead ofsize_tfor offsets/indices/counts - Always use braces for
ifstatements and loops (no single-line statements) - Use exceptions only for exceptional situations that terminate query execution
- Validate inputs at function boundaries
- Use
D_ASSERTfor programmer errors, never for user input
Location: cli/ directory
Important Rules:
- Both CLI and VSCode extension should share a common API client
- Both should communicate with flAPI server via ConfigService (never directly via files)
- All interactions use the same authentication tokens and URL resolution logic
CLI Commands (cli/src/commands/):
config/: Configuration managementendpoint/: Endpoint testing and introspectiontemplate/: SQL template validationcache/: Cache inspection and managementserve/: Local development server
Framework: Catch2 (test/cpp/)
Test files: src/components/*_test.cpp
Running tests:
make test # Run all unit tests
cd build/release && ctest -V -R pattern # Run specific testTest patterns:
- Use Catch2
TEST_CASEandSECTIONmacros - Mock dependencies with test fixtures
- Test both success and error cases
Framework: pytest + Tavern (test/integration/)
Python Environment: Always use uv for Python virtual environments:
cd test/integration
uv venv # Create virtual environment
uv pip install -e . # Install dependencies from pyproject.toml
source .venv/bin/activate # Activate environmentTest types:
*.tavern.yaml: REST API tests using Tavern specification*.py: Custom Python tests (MCP, async, complex scenarios)
Running tests:
# Manual test runs (after uv environment setup)
cd test/integration
source .venv/bin/activate
pytest test_customers.tavern.yaml -v # Single test file
pytest test_mcp_integration.py::test_name -v # Single test function
# Or use make targets (handles environment automatically)
make integration-test-rest # Tavern tests
make integration-test-mcp # MCP protocol tests
make integration-test-ci # Full suite with server managementTest server lifecycle:
- Tests assume server running on
http://localhost:8080 - Use
integration-test-citarget for automatic server startup/shutdown - See
test/integration/conftest.pyfor fixtures
- Embedded in-process OLAP database (v1.5.5)
- Extensions for external data sources: BigQuery, Postgres, Iceberg, Parquet, CSV
- Query execution with result caching
Database Operations:
- Defined in
DatabaseManager(src/database_manager.cpp) - Connection pooling to external sources
- Extension loading and configuration
- Transaction support with ACID compliance
Cache Configuration in Endpoint YAML:
cache:
enabled: true
schedule: 6h # how often to refresh
# Mode is INFERRED from primary-key/cursor - there is no `refresh:` key.
table: cache_table_name # Where to store cached results
template-file: cache_populate.sql # The populate SQL (there is no refresh_query key)Cache Refresh Lifecycle:
- HeartbeatWorker checks
cache.schedule(the scheduler is on by default;ducklake.scheduler.enabled: falseopts out). There is no refresh at boot beyond warm-up: a cache is due onescheduleafter its last refresh - On refresh trigger: run the
template-fileverbatim, all statements in one transaction (a failure changes nothing) - Full refresh: the template must write the table itself, e.g.
CREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} AS SELECT ...(a bareSELECTstatement is refused before anything runs, and a script that leaves no cache table fails and rolls back) - Incremental:
INSERT INTO ...(append) or a single-actionMERGE(merge), behind a{{^cache.previousSnapshotTimestamp}}bootstrap that creates the table on the first run - Requests always read the cache table; a GET never triggers a refresh. A cache whose first build failed answers
503 {"error":"cache_build_failed"} - Full rebuild on demand:
flapii cache refresh <path> --full/?full=true/flapi cache refresh --fulldrops the table and rebuilds it with the watermark erased
Materialized Results:
- Cache tables live in the DuckLake catalog (
ducklake.metadata-path), not induckdb.db_path - Data is written as Parquet files under
ducklake.data-path - Fast serving from local storage (millisecond latency)
# Project metadata
project-name: my-project
project-description: Description
version: 1.0.0
# SQL templates location
template:
path: ./sqls
# Data source connections
connections:
data-source-name:
type: postgres | snowflake | bigquery | local # Optional, auto-detected
properties:
# Type-specific: path, host, database, api_key, etc.
path: ./data/customers.parquet
# DuckDB engine settings
duckdb:
db_path: ./flapi_cache.db # Local cache database
threads: 4 # Query parallelism
max_memory: 2GB # Memory limit
extensions: # Optional: explicit extension config
- name: json
- name: postgres
# Logging (top level, not nested under `server:`)
log-level: info # debug, info, warning (or warn), error; anything else stops startup
log-format: text # text | json
# MCP server (its own top-level block). No port of its own: MCP is served on
# the HTTP port, at /mcp/jsonrpc.
mcp:
enabled: true
# Global rate limiting (optional). Note the underscore here and the HYPHEN
# in the per-endpoint `rate-limit:` block - they genuinely differ.
rate_limit:
enabled: true
max: 100 # requests per interval
interval: 60 # seconds
# Tracing (optional, off by default, read once at startup)
tracing:
enabled: falseDo not trust this snippet over the code. It is illustrative; the authoritative key list is docs/CONFIG_REFERENCE.md, and the parser in
src/config_manager.cppis the ground truth. Before documenting any key, grep for a reader of it, not just a parser.
# Configuration
FLAPI_CONFIG=path/to/flapi.yaml # Config file path
FLAPI_LOG_LEVEL=debug|info|warning|warn|error
FLAPI_PORT=8080 # HTTP port (fallback for --port)
FLAPI_HOST=0.0.0.0 # Bind address (fallback for --host)
# Authentication
JWT_SECRET=your-secret-key # JWT signing key
AWS_REGION=us-east-1 # For AWS Secrets Manager
# Development
FLAPI_CROSS_COMPILE=arm64 # Cross-compile target (Linux only)The flapii CLI tool communicates with the running flAPI server via REST APIs. These management APIs allow runtime configuration changes without restarting.
Endpoint Management:
GET /api/v1/_config/endpoints # List all endpoints
GET /api/v1/_config/endpoints/{path} # Get one endpoint
POST /api/v1/_config/endpoints # Create new endpoint
PUT /api/v1/_config/endpoints/{path} # Update endpoint
DELETE /api/v1/_config/endpoints/{path} # Delete endpointTemplate Management:
GET /api/v1/_config/endpoints/{path}/template # Get template content
PUT /api/v1/_config/endpoints/{path}/template # Update template
POST /api/v1/_config/endpoints/{path}/template/expand # Expand templateCache Management:
GET /api/v1/_config/endpoints/{path}/cache # Get cache config
PUT /api/v1/_config/endpoints/{path}/cache # Update cache
POST /api/v1/_config/endpoints/{path}/cache/refresh # Force refreshSchema Introspection:
GET /api/v1/_schema # Get database schema
GET /api/v1/_schema/connections # List connections
POST /api/v1/_schema/refresh # Refresh schema cacheServer Health:
POST /api/v1/_ping # Check server status# List all endpoints
flapii endpoints list
# Get specific endpoint config
flapii endpoints get /customers
# Test template expansion
flapii templates expand /customers --params '{"id":"123"}'
# Refresh cache for endpoint
flapii cache refresh /customers
# View cache configuration
flapii cache get /customers
# Validate endpoint YAML before creating
flapii endpoints validate sqls/new_endpoint.yaml
# Create endpoint from YAML file
flapii endpoints create sqls/new_endpoint.yamlUser runs: flapii endpoints list
↓
CLI Command (cli/src/commands/endpoints/list.ts)
↓
Makes HTTP call: GET http://localhost:8080/api/v1/_config/endpoints
↓
ConfigService API Handler (src/config_service.cpp)
↓
Returns endpoint list as JSON
↓
CLI formats as table/JSON and displays
The CLI provides a project initialization command that scaffolds a new flapi project with all necessary directories and example files.
Quick Start:
# Initialize in current directory
flapii project init
# Create new project directory
flapii project init my-api-project
# Force overwrite existing files
flapii project init my-api --forceWhat Gets Created:
project-directory/
├── flapi.yaml # Main configuration file
├── sqls/ # Endpoint definitions and SQL templates
│ ├── sample.yaml # Example endpoint config
│ └── sample.sql # Example SQL template with Mustache
├── data/ # Data files directory (add your Parquet, CSV, etc.)
├── common/ # Reusable configuration templates
│ ├── auth.yaml # Authentication template
│ └── rate-limit.yaml # Rate limiting template
└── .gitignore # Git ignore rules for flapi projects
Command Options:
# Skip validation after setup
flapii project init my-api --skip-validation
# Only create directories and flapi.yaml (no examples)
flapii project init my-api --no-examples
# Force overwrite if files already exist
flapii project init my-api --force
# Advanced mode (additional templates, Phase 3)
flapii project init my-api --advancedDetailed Walkthrough:
-
Run initialization:
flapii project init my-first-api Setting up flapi project: my-first-api ✅ Created directories ✅ Created flapi.yaml ✅ Created sample endpoint ✅ Created .gitignore ✅ Created reusable configs ✓ Configuration is valid
-
Edit
flapi.yamlto add your database connections:connections: my-database: properties: host: localhost port: 5432 user: '{{env.DB_USER}}' password: '{{env.DB_PASSWORD}}'
-
Create endpoint YAML in
sqls/directory -
Write SQL templates in
sqls/ -
Run server:
./flapi -c my-first-api/flapi.yaml
- Create endpoint YAML (
sqls/my_endpoint.yaml):
url-path: /my-endpoint
method: GET
request:
- field-name: param1
field-in: query
validators:
- type: string
max-length: 100
template-source: my_endpoint.sql
connection: [data-source]- Create SQL template (
sqls/my_endpoint.sql):
SELECT * FROM table
WHERE column = {{ params.param1 }} -- bound as a prepared parameter: no quotes- Test locally:
make run-debug
# Test: curl http://localhost:8080/my-endpoint?param1=value- Add integration test (
test/integration/test_my_endpoint.tavern.yaml):
test_name: Test my endpoint
stages:
- name: Get data
request:
url: http://localhost:8080/my-endpoint?param1=value
method: GET
response:
status_code: 200The CLI includes an interactive endpoint creation wizard with optional AI assistance powered by Google Gemini. This lets you generate endpoint configurations from natural language descriptions.
Quick Start:
# Interactive wizard with manual mode (no AI)
flapii endpoints wizard
# AI-assisted endpoint creation
flapii endpoints wizard --ai
# Save configuration to file instead of creating endpoint
flapii endpoints wizard --output-file endpoint-config.yaml
# Preview without making changes
flapii endpoints wizard --dry-run
# Skip validation (useful for batch processing)
flapii endpoints wizard --skip-validationUsing AI Generation:
- Run
flapii endpoints wizard --ai - Provide API key when prompted (stored in
~/.flapi/config.json) - Describe your endpoint in natural language:
"Create a GET endpoint to fetch active customers with pagination. Should filter by status and region. Cache for 5 minutes." - AI generates endpoint configuration with parameters, validators, and cache settings
- Review the generated config and choose to:
- Accept and save immediately
- Edit configuration (modify name, path, parameters, cache settings)
- Regenerate with a different description
- Fall back to manual mode
- Cancel
AI Authentication:
- Set
FLAPI_GEMINI_KEYenvironment variable, OR - Provide API key when first prompted (saved securely to
~/.flapi/config.json) - Get a free API key: https://aistudio.google.com/app/apikey
Manual Mode:
If you prefer not to use AI or it's not available:
flapii endpoints wizard # No --ai flag
# Wizard prompts for:
# - Endpoint name and URL path
# - HTTP method (GET/POST/PUT/DELETE)
# - Parameters (name, type, location: query/path/body)
# - Cache settings (TTL)Saving to File:
Generate configuration files without creating endpoints:
# Output as YAML (for version control/review)
flapii endpoints wizard --output-file my-endpoint.yaml
# Review file before creating
cat my-endpoint.yaml
flapii endpoints create --file my-endpoint.yaml- Add cache section to endpoint YAML:
cache:
enabled: true
schedule: 6h
table: my_endpoint_cache-
Create materialized view or cache table definition in DuckDB
-
Test cache refresh:
make run-debug
# Monitor DuckDB cache: SELECT * FROM my_endpoint_cache;- Templates use Mustache syntax:
{{ variable_name }} - Available variables:
params.*- Request parametersconn.*- Connection propertiesauth.*- Auth context
- Test template rendering locally before deployment
- Check server logs:
make run-debug
# Look for error messages in output- Enable verbose logging:
./build/debug/flapi --config examples/flapi.yaml --log-level debug- Test endpoint directly:
curl -v http://localhost:8080/endpoint?param=value- Check configuration validation:
# Review endpoint YAML for required fields
# Check SQL template for Mustache syntax errorsTest how Mustache templates render before deploying:
# Expand template with parameters
flapii templates expand /customers --params '{"id":"123"}'
# See generated SQL output
# Check for SQL syntax errorsWhen parameter validation is rejecting valid input:
# List endpoint config with validators
flapii endpoints get /endpoint | jq '.request[0].validators'
# Test request
curl -v "http://localhost:8080/endpoint?param=value"
# Check response for validation error details
# Adjust validators in endpoint configDebug cache refresh and hit rates:
# Force cache refresh
flapii cache refresh /endpoint
# Check cache status
flapii cache get /endpoint
# View cache configuration
flapii cache get /endpoint | jq '.cache'
# Monitor cache tables
# SELECT * FROM endpoint_cache_table;- Add type to validator enum in
src/include/validators.hpp - Implement validation logic in
src/validators.cpp - Add YAML schema documentation
- Write unit tests in
test/unit/validators/ - Write integration tests in
test/integration/ - Update CLI help text
flAPI can use any DuckDB extension. Extensions are auto-loaded when needed:
# In connection config, use DuckDB-specific syntax
connections:
bigquery-data:
properties:
project_id: 'my-project'
dataset: 'my_dataset'
# Extensions automatically loaded: postgres_scanner, httpfs, bigquery, json, etc.Common extensions:
postgres_scanner- Query Postgreshttpfs- Read S3, GCS, Azurejson- JSON functionsiceberg- Apache Iceberg tablesdelta- Delta Lake tables
Key Features:
- C++20 project-wide, with one deliberate exception: DuckDB's subdirectory is
built at C++17. abseil (via opentelemetry-cpp) exports
cxx_std_20, and mixing standards across the link produced a real segfault; but C++20 removedstd::uncaught_exception(), which DuckDB calls behind a__cplusplusguard that MSVC defeats.CMakeLists.txtsaves and restoresCMAKE_CXX_STANDARDaroundadd_subdirectory(duckdb). Do not "simplify" this — seedocs/spec/DESIGN_DECISIONS.md § 10c.scripts/check_cxx_standard_uniform.shguards the rest. - vcpkg integration for consistent dependency management
- Platform-specific configurations (Windows, macOS, Linux/ARM64)
- Cross-compilation support
- Sanitizer support for debug builds (ASAN, UBSAN)
Build Targets:
flapi-lib: Static library with core functionalityflapi: Executable binaryintegration_tests: Python integration test target- Catch2 tests: C++ unit tests
| Library | Purpose |
|---|---|
| DuckDB 1.5.5 | In-process OLAP database |
| Crow | C++ web framework |
| OpenSSL | Encryption/security |
| jwt-cpp | JWT authentication |
| yaml-cpp | YAML configuration parsing |
| argparse | CLI argument parsing |
| fmt | String formatting |
| AWS SDK | AWS Secrets Manager integration |
Linux/macOS: vcpkg (automatic setup via CMake) Windows: vcpkg with x64-windows-static-md triplet
Add new validator types beyond the built-in ones (int, string, email, uuid, enum, date, time).
How to Add:
- Add type to validator enum in
src/include/validators.hpp - Implement validation logic in
src/validators.cpp - Use in endpoint config:
validators: - type: custom_type option1: value1
flAPI supports all DuckDB extensions. They auto-load when needed.
Supported Extensions:
postgres_scanner- Query Postgres databaseshttpfs- Read from S3, GCS, Azure, HTTPjson- JSON functions and operatorsiceberg- Apache Iceberg tablesdelta- Delta Lake tablescsv- CSV file scanninghttpfs- Cloud storage access- And 50+ more
Usage: Just reference the data source in your template, extension loads automatically.
Currently supports Basic and JWT auth. Can be extended for:
- OAuth2/OIDC
- API keys
- Custom token validation
Configure in endpoint:
auth:
type: jwt
token-key: '{{env.JWT_SECRET}}'
issuer: "example.com"Default format is JSON. Can extend to support:
- CSV
- Parquet
- Arrow
- XML
Reference in endpoint or query parameter.
Beyond cache refresh, can extend scheduler for:
- Custom background jobs
- Data synchronization
- Reporting
- Maintenance tasks
Configure with cron expressions or interval schedules.
| Operation | Time |
|---|---|
| Parameter validation | 1-5ms |
| Template expansion (Mustache) | 2-10ms |
| Simple SELECT (cached) | 5-50ms |
| Complex query (not cached) | 100-1000ms |
| Network roundtrip | 1-10ms |
| Total for simple GET | 10-70ms |
- Base server: ~50MB
- Per connection: ~5-20MB
- Cache table (1M rows): ~100-500MB
- DuckDB buffer pool: Configurable, default 1GB
- Concurrent connections: 1000+
- Endpoints: 1000+
- Parameters per endpoint: No practical limit (~20 recommended)
- Request rate: 1000+ req/sec (single-threaded DuckDB)
- DuckDB performs aggressive optimization on queries
- Use
EXPLAINto analyze query plans:EXPLAIN SELECT ... - Filters in WHERE clauses are pushed down to source systems
- Joins between parquet/CSV and external sources are optimized
- Use full refresh (declare neither
primary-keynorcursor) for small datasets - Use incremental append (declare a
cursor) for large append-only data - Set
cache.schedulebased on data freshness requirements (there is no TTL) - Monitor cache hit rates in logs
- Configure
duckdb.max_memorybased on available system memory - DuckDB spillsover to disk when memory limit reached
- Use
duckdb.threadsto control parallelism
make docker # Build Docker image
# Image includes pre-built flAPI binary
# Port: 8080 serves both REST and MCP (/mcp/jsonrpc)
# Published tags: :latest and :v<version> from releases only,
# :main and :sha-<commit> from main; pull requests push nothing.make release # Build optimized release binary
./build/release/flapi --config flapi.yamlThe binary is statically linked with all dependencies and can be deployed to any Linux/macOS/Windows system without additional runtime requirements.
Both the flapi server and the flapii CLI ship inside a single PyPI
package, flapi-io. Installing it provides both console commands:
pip install flapi-io # installs both `flapi` (server) and `flapii` (CLI)The flapi_io wheel bundles both binaries (flapi_io/bin/flapi and
flapi_io/bin/flapii) and declares console-script entry points for each
(flapi = flapi_io:main_flapi, flapii = flapi_io:main_flapii). Wheels are
built automatically during release using
bin-to-wheel.
Package name on PyPI:
flapi-io— the only published package ("flapi" was taken on PyPI); carries both the server and the CLI. There is no separateflapiiPyPI package.
Versioning: v* tags (e.g., v0.5.0). Tag push triggers the full release pipeline.
| Component | Platform | Build method | Artifact name |
|---|---|---|---|
| flapi (server) | Linux x86_64 | Docker cross-compile | flapi-linux-amd64 |
| flapi (server) | Linux ARM64 | Docker cross-compile | flapi-linux-arm64 |
| flapi (server) | macOS ARM64 | Native (macos-latest) | flapi-macos-arm64 |
| flapi (server) | Windows x64 | MSVC (windows-latest) | flapi-windows-amd64 |
| flapii (CLI) | Linux x86_64 | bun build --compile |
flapii-linux-amd64 |
| flapii (CLI) | Linux ARM64 | bun build --compile |
flapii-linux-arm64 |
| flapii (CLI) | macOS ARM64 | bun build --compile |
flapii-macos-arm64 |
| flapii (CLI) | Windows x64 | bun build --compile |
flapii-windows-amd64 |
- Ensure all CI builds pass on
main(gh run list)- In the release-prep commit, after rolling
## Unreleasedinto## v{version}, stamp the Claude Code plugin:python3 scripts/check_skills.py --set-version {version}(rewrites plugin.json, every SKILL.mdversion:and banner), thenpython3 scripts/check_skills.py --static-onlymust pass.
- In the release-prep commit, after rolling
- Tag and push:
git tag v{version} && git push origin v{version} .github/workflows/build.yamltriggers onv*tag push (there is no separaterelease.yaml):- Downloads build artifacts (4 flapi server + 4 flapii CLI binaries)
- Creates archive assets for the flapi server only (
.tar.gzfor Unix,.zipfor Windows) - Builds 4
flapi_ioPython wheels viabin-to-wheel, each bundling both the flapi and flapii binaries for that platform - Creates the GitHub Release with the 4 server archives, the wheels and
server.json. The flapii binaries are not attached as standalone release assets; they ship only inside theflapi-iowheels (theflapii-*names in the table above are CI artifact names) - Publishes the
flapi-iowheels to PyPI via trusted publishing (OIDC)
Uses OIDC trusted publishing — no API tokens needed. Configuration:
| Package | GitHub Environment | PyPI Pending Publisher |
|---|---|---|
flapi-io |
pypi-flapi-io |
DataZooDE/flapi, build.yaml, pypi-flapi-io |
Only flapi-io is published (it carries both the server and the CLI), so a single trusted-publisher environment is needed.
Each release produces 4 flapi_io wheels (one per platform), each bundling both the flapi and flapii binaries:
| Package | Platform | Wheel platform tag |
|---|---|---|
flapi-io |
Linux x86_64 | manylinux_2_17_x86_64 |
flapi-io |
Linux ARM64 | manylinux_2_17_aarch64 |
flapi-io |
macOS ARM64 | macosx_11_0_arm64 |
flapi-io |
Windows x64 | win_amd64 |
The flapii CLI is a TypeScript project (Commander.js) compiled to standalone binaries using Bun's --compile flag. The flapii-build job in build.yaml runs a matrix build across 4 targets:
bun build --compile --target=bun-linux-x64 src/index.ts --outfile flapii
bun build --compile --target=bun-linux-arm64 src/index.ts --outfile flapii
bun build --compile --target=bun-darwin-arm64 src/index.ts --outfile flapii
bun build --compile --target=bun-windows-x64 src/index.ts --outfile flapii.exeEach produces a self-contained binary with no runtime dependencies (Bun runtime is embedded).
Problem: CMake finds wrong DuckDB version
- Solution:
rm -rf build/and rebuild
Problem: vcpkg dependencies not found (macOS/Windows)
- Solution: Ensure
VCPKG_ROOTenvironment variable is set
Problem: Cross-compilation failure (Linux ARM64)
- Solution: Set
FLAPI_CROSS_COMPILE=arm64and use proper toolchain
Problem: GET /my-endpoint returns 404 Not Found
Causes:
- Endpoint YAML not found in sqls/ directory
url-pathdoesn't match request path- Endpoint not loaded during startup
Solution:
# List all registered endpoints
flapii endpoints list
# Get specific endpoint
flapii endpoints get /my-endpoint
# Check YAML file exists
ls -la sqls/my_endpoint.yaml
# Review url-path in config
flapii endpoints get /my-endpoint | jq '.url-path'
# Reload endpoint if already created
flapii endpoints reload /my-endpointProblem: 400 Bad Request: Template error or 500 Internal Server Error
Causes:
- Mustache syntax error (unmatched braces)
- Variable not in context (typo in variable name)
- Invalid SQL generated
Solution:
# Test template expansion with parameters
flapii templates expand /endpoint --params '{"id":"123"}'
# Check syntax
flapii endpoints validate sqls/endpoint.yaml
# Enable debug logging
flapii config log-level set debug
# Check error message in logs
./build/debug/flapi --config flapi.yaml --log-level debugProblem: 400 Bad Request: Invalid parameter: field - validation failed
Causes:
- Validator type mismatch (expecting int, got string)
- Constraints too strict (min/max bounds)
- Regex pattern doesn't match valid input
Solution:
# Check validator configuration
flapii endpoints get /endpoint | jq '.request[0].validators'
# Test with curl (no validators)
curl -v "http://localhost:8080/endpoint?id=test"
# Review validator config - may need to:
# - Change type
# - Loosen min/max bounds
# - Fix regex pattern
# - Make parameter optional (required: false)Problem: Cache data is stale, refresh not triggering
Causes:
- Scheduler opted out (
ducklake.scheduler.enabled: false; startup warns) - Template error in cache populate SQL (a bare
SELECTis refused: the template must write the table) - DuckLake not initialized
- Cache table doesn't exist
Solution:
# Check cache configuration
flapii cache get /endpoint
# Refresh now, or drop and rebuild from scratch
flapii cache refresh /endpoint
flapii cache refresh /endpoint --full
# Refresh history: mode, status, duration, rows, snapshot
flapii cache audit /endpoint -o table
# Check cache status
flapii cache list
# Validate cache populate template
cat sqls/cache-populate.sql
# Enable debug logging to see refresh attempts
flapii config log-level debug
# Check if cache table exists
# SELECT * FROM cache_table_name;Problem: DuckDB using lots of memory, slow queries
Causes:
- Large query result set without LIMIT
- No LIMIT clause in template
- Cache accumulating snapshots
- Memory limit not configured
Solution:
# In flapi.yaml, set memory limit
duckdb:
max_memory: "4GB"
# In endpoint config, set reasonable LIMIT
request:
- field-name: limit
validators:
- type: int
max: 10000# Expire old snapshots per retention and delete their files (path is required)
flapii cache gc /endpoint
# Check retention policy
flapii cache get /endpoint | jq '.cache.retention'
# Monitor memory in debug logs
flapii config log-level debugProblem: 500 Internal Server Error or connection refused
Causes:
- Wrong connection credentials
- Database not accessible from network
- Missing environment variables
- Connection not whitelisted
Solution:
# List connections
flapii schema connections
# Test connection
flapii schema refresh --connection connection-name
# Check credentials
echo $DB_PASSWORD # Should not be empty
# Verify connection config in flapi.yaml
cat flapi.yaml | grep -A 10 "connections:"
# Check environment variable whitelist
cat flapi.yaml | grep -A 5 "environment-whitelist:"
# Enable debug logging
flapii config log-level debug
./build/debug/flapi --config flapi.yamlProblem: 401 Unauthorized, 403 Forbidden or JWT validation errors
Causes:
- Missing or invalid JWT secret
- Token expired or malformed
- Auth not enabled for endpoint (or inherited from the top-level
auth:when the endpoint has no block) - Wrong authentication type
403: authenticated, but the principal holds none of the endpoint'sauth.roles
Solution:
# Check if auth is enabled for endpoint (`inherited: true` = the top-level auth applies)
flapii endpoints get /endpoint | jq '.auth'
# Verify JWT secret is set
echo $JWT_SECRET # Must not be empty
# Check token format
# Should be: Authorization: Bearer <token>
# Verify auth configuration
cat flapi.yaml | grep -A 10 "auth:"
# Test without auth (temporary)
# Set `auth: {enabled: false}` on the endpoint (removing the block inherits the top-level auth)Problem: 400 Bad Request or 500 Internal Server Error. A 500 body carries only a request_id; grep the server log for it to see the SQL error
Causes:
- Invalid SQL generated from template
- Mustache variables containing invalid SQL
- Triple braces not used for strings
- Missing connections
Solution:
# Expand template to see generated SQL
flapii templates expand /endpoint --params '{"id":"123"}'
# Check generated SQL syntax
# Copy output and run in DuckDB
# Fix template:
# - Use triple braces for strings: {{{ }}}
# - Use double braces for numbers: {{ }}
# - Verify variable names match
# Test with EXPLAIN
# EXPLAIN SELECT ... (in generated SQL)After making code changes, update the relevant documentation:
| Change Type | Documents to Update |
|---|---|
| New component or major refactor | docs/spec/ARCHITECTURE.md, relevant component doc |
| API endpoint changes | docs/CONFIG_SERVICE_API_REFERENCE.md |
| Configuration options | docs/CONFIG_REFERENCE.md |
| MCP protocol changes | docs/MCP_REFERENCE.md |
| CLI command changes | docs/CLI_REFERENCE.md |
| Design pattern changes | docs/spec/DESIGN_DECISIONS.md |
| Request flow changes | docs/spec/REQUEST_LIFECYCLE.md |
| Config system changes | docs/spec/components/config-system.md |
| Query/DuckDB changes | docs/spec/components/query-execution.md |
| Cache system changes | docs/spec/components/caching.md |
| MCP implementation changes | docs/spec/components/mcp-protocol.md |
| Auth/security changes | docs/spec/components/security.md |
| Tracing/audit/logging changes | docs/OBSERVABILITY.md, docs/spec/components/observability.md |
| A new user-facing capability | a guide in docs/guides/, linked from docs/README.md |
Before completing work that modifies code:
- If architecture changed → update
docs/spec/ARCHITECTURE.md - If new design decision → add to
docs/spec/DESIGN_DECISIONS.md - Every config key you document, grep for a reader in
src/, not just a parser. Three keys once shipped documented and parsed but never read; operators could set them and nothing happened. - If request flow changed → update
docs/spec/REQUEST_LIFECYCLE.md - If component internals changed → update relevant
docs/spec/components/*.md - If user-facing API changed → update relevant reference doc in
docs/
- Use Mermaid diagrams for visual architecture
- Include source file references (e.g.,
src/config_manager.cpp:123) - Keep explanations concise and code-focused
- Update "Last updated" timestamps when making significant changes
When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.
MANDATORY WORKFLOW:
- File issues for remaining work - Create issues for anything that needs follow-up
- Run quality gates (if code changed) - Tests, linters, builds
- Update issue status - Close finished work, update in-progress items
- PUSH TO REMOTE - This is MANDATORY:
git pull --rebase bd sync git push git status # MUST show "up to date with origin" - Clean up - Clear stashes, prune remote branches
- Verify - All changes committed AND pushed
- Hand off - Provide context for next session
CRITICAL RULES:
- Work is NOT complete until
git pushsucceeds - NEVER stop before pushing - that leaves work stranded locally
- NEVER say "ready to push when you are" - YOU must push
- If push fails, resolve and retry until it succeeds
When working on GitHub issues, use feature branches and pull requests to enable code review and maintain a clean main branch history.
Workflow Steps:
-
Create feature branch from main:
git checkout main git pull origin main git checkout -b feature/gh-<issue-number>-<short-description> # Example: git checkout -b feature/gh-9-arrow-streaming
-
Create or link beads epic (optional but recommended):
bd create --title="<GitHub Issue Title>" --type=epic --priority=1 # Add external link to GitHub issue bd update <beads-id> --notes="GitHub: https://github.com/DataZooDE/flapi/issues/<N>"
-
Work and commit on feature branch:
- Make changes, run tests
- Commit with descriptive messages
- Reference GitHub issue:
git commit -m "feat: Add Arrow streaming support (#9)"
-
Create pull request:
git push -u origin feature/gh-<issue-number>-<short-description> gh pr create --title "..." --body "Closes #<issue-number>..."
-
After PR merge:
git checkout main git pull origin main git branch -d feature/gh-<issue-number>-<short-description> bd close <beads-epic-id> bd sync
Branch Naming Convention:
feature/gh-<N>-<description>- For GitHub issue workfix/gh-<N>-<description>- For bug fixeschore/gh-<N>-<description>- For maintenance tasks
PR Description Template:
## Summary
<1-3 bullet points>
## Test plan
- [ ] Tests added/updated
- [ ] Manual verification steps
Closes #<issue-number>Integration with Beads:
- Beads tracks local work breakdown and dependencies
- GitHub issues track public feature requests and bugs
- Link them via notes field:
bd update <id> --notes="GitHub: #N" - Close beads epic after PR merges
Beads (bd) is a git-backed issue tracker used in this project. This section documents operational lessons learned, particularly for multi-repo configurations.
# Common commands
bd ready # Find issues ready to work (no blockers)
bd list --status=open # All open issues
bd show <id> # View issue details
bd create --title="..." --type=task --priority=2 # Create issue
bd update <id> --status=in_progress # Start work
bd close <id> # Complete issue
bd sync # Sync with git remoteWhen a user has multiple beads workspaces (e.g., ~/.beads-planning for personal planning alongside project repos), bd create may route issues to the wrong repository.
Symptom:
bd create --title="Fix bug" --type=bug
Error: database not initialized: issue_prefix config is missing
This happens even when the current project has beads properly configured.
Diagnosis - Use --verbose:
bd --verbose create --title="Fix bug" --type=bug
# Output shows: DEBUG: Routing to target repo: /Users/jr/.beads-planning
# ^^^ Wrong repo! Should be current projectSolution - Use --repo flag:
# Explicitly specify the repository path
bd create --repo=/Users/jr/Projects/datazoo/flapi --title="Fix bug" --type=bug --priority=1| Command Type | Behavior | Examples |
|---|---|---|
| Read commands | Respect current directory | bd list, bd show, bd config get |
| Write commands | May use multi-repo routing | bd create - can route to wrong repo |
Read commands work fine from the project directory. Write commands like bd create use multi-repo routing logic that may send issues to a different repository.
issue-prefixstored in database takes precedence- Config file setting in
.beads/config.yamlmay not be picked up by daemon - Use
bd config set issue-prefix <value>to set in database
Check current prefix:
bd config get issue-prefixWhen bd create fails or routes incorrectly:
-
Check routing with verbose mode:
bd --verbose create --title="Test" --type=task -
Use explicit repo path:
bd create --repo=$(pwd) --title="Issue title" --type=task --priority=2
-
Verify issue prefix is configured:
bd config get issue-prefix # If missing, set it: bd config set issue-prefix flapi
-
Restart daemon if needed:
# Kill any running beads processes pkill -f beads # Retry command bd create --title="..." --type=task
-
Check .beads directory exists:
ls -la .beads/ # Should contain: config.yaml, issues.db, etc.
- Always verify routing first when working in multi-repo environments
- Use
--repoflag if you have multiple beads workspaces - Check with
bd listafter creating to confirm issue landed in correct repo - Run
bd syncat session end to push changes to remote
IMPORTANT: This project uses bd (beads) for ALL issue tracking. Do NOT use markdown TODOs, task lists, or other tracking methods.
- Dependency-aware: Track blockers and relationships between issues
- Git-friendly: Auto-syncs to JSONL for version control
- Agent-optimized: JSON output, ready work detection, discovered-from links
- Prevents duplicate tracking systems and confusion
Check for ready work:
bd ready --jsonCreate new issues:
bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json
bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --jsonClaim and update:
bd update bd-42 --status in_progress --json
bd update bd-42 --priority 1 --jsonComplete work:
bd close bd-42 --reason "Completed" --jsonbug- Something brokenfeature- New functionalitytask- Work item (tests, docs, refactoring)epic- Large feature with subtaskschore- Maintenance (dependencies, tooling)
0- Critical (security, data loss, broken builds)1- High (major features, important bugs)2- Medium (default, nice-to-have)3- Low (polish, optimization)4- Backlog (future ideas)
- Check ready work:
bd readyshows unblocked issues - Claim your task:
bd update <id> --status in_progress - Work on it: Implement, test, document
- Discover new work? Create linked issue:
bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:<parent-id>
- Complete:
bd close <id> --reason "Done"
bd automatically syncs with git:
- Exports to
.beads/issues.jsonlafter changes (5s debounce) - Imports from JSONL when newer (e.g., after
git pull) - No manual export/import needed!
- ✅ Use bd for ALL task tracking
- ✅ Always use
--jsonflag for programmatic use - ✅ Link discovered work with
discovered-fromdependencies - ✅ Check
bd readybefore asking "what should I work on?" - ❌ Do NOT create markdown TODO lists
- ❌ Do NOT use external issue trackers
- ❌ Do NOT duplicate tracking systems
For more details, see README.md and docs/QUICKSTART.md.
This project uses beads_viewer for issue tracking. Issues are stored in .beads/ and tracked in git.
# View issues (launches TUI - avoid in automated sessions)
bv
# CLI commands for agents (use these instead)
bd ready # Show issues ready to work (no blockers)
bd list --status=open # All open issues
bd show <id> # Full issue details with dependencies
bd create --title="..." --type=task --priority=2
bd update <id> --status=in_progress
bd close <id> --reason="Completed"
bd close <id1> <id2> # Close multiple issues at once
bd sync # Commit and push changes- Start: Run
bd readyto find actionable work - Claim: Use
bd update <id> --status=in_progress - Work: Implement the task
- Complete: Use
bd close <id> - Sync: Always run
bd syncat session end
- Dependencies: Issues can block other issues.
bd readyshows only unblocked work. - Priority: P0=critical, P1=high, P2=medium, P3=low, P4=backlog (use numbers, not words)
- Types: task, bug, feature, epic, question, docs
- Blocking:
bd dep add <issue> <depends-on>to add dependencies
Before ending any session, run this checklist:
git status # Check what changed
git add <files> # Stage code changes
bd sync # Commit beads changes
git commit -m "..." # Commit code
bd sync # Commit any new beads changes
git push # Push to remote- Check
bd readyat session start to find available work - Update status as you work (in_progress → closed)
- Create new issues with
bd createwhen you discover tasks - Use descriptive titles and set appropriate priority/type
- Always
bd syncbefore ending session