This is the in-depth reference for cloud storage: every provider, its authentication, and its failure modes.
If you just want to read data from a bucket, start with guides/cloud-storage.md — it is two pages and links back here for the detail.
Known limitation: a remote
template.path(ans3://,gs://,az://orhttps://directory) does not work yet. flAPI lists the template directory on the local filesystem, so startup fails withTemplate path does not exist or is not a directory. Keep the endpoint YAML and SQL templates on local disk (or bundle them withflapi pack). Data files and DuckLake data paths on cloud storage do work.
flAPI supports loading configuration and SQL template files from remote storage via DuckDB's Virtual File System (VFS). This enables:
- Serverless deployments: Run flAPI on AWS Lambda, Google Cloud Run, or Azure Functions without volume mounts
- Centralized configuration: Store all endpoint definitions in a single S3 bucket or GCS bucket
- GitOps workflows: Deploy configuration changes by updating files in cloud storage
- Multi-environment setups: Use different storage paths for dev/staging/prod
The simplest way to use remote configuration is via HTTPS:
flapi --config https://raw.githubusercontent.com/myorg/configs/main/flapi.yamlFor S3, set AWS credentials and use an S3 URL:
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_REGION=us-east-1
flapi --config s3://my-bucket/configs/flapi.yamlFor Google Cloud Storage:
flapi --config gs://my-bucket/configs/flapi.yamlflAPI does not use GOOGLE_APPLICATION_CREDENTIALS or GOOGLE_CLOUD_PROJECT
for gs:// (startup logs a warning when they are set), and DuckDB's httpfs
does not read service-account files. The config file is read before any
connection init runs, so loading it from gs:// works only for an object
that is readable without credentials. For a private bucket, keep the config in
S3 or pack it into the binary; data in a private GCS
bucket is read with an HMAC key, see Google Cloud Storage.
| Backend | URL Scheme | Authentication |
|---|---|---|
| Amazon S3 | s3:// |
AWS credentials (env vars or IAM role) |
| S3-compatible | s3://, s3a://, s3n:// |
Same as S3 |
| Google Cloud Storage | gs:// |
HMAC key in a DuckDB secret (TYPE gcs); GOOGLE_* variables are not used |
| Azure Blob Storage | az://, abfs:// |
Storage account key or managed identity |
| HTTPS | https:// |
None (public) or HTTP auth headers |
| HTTP | http:// |
None (not recommended for production) |
| Local filesystem | file:// or plain path |
Filesystem permissions |
Load templates from a web server or GitHub raw URLs:
# flapi.yaml
project-name: HTTPS Example
project-description: Templates served via HTTPS
template:
path: https://raw.githubusercontent.com/myorg/templates/main/sqls/
connections:
local-data:
properties:
path: ./data/customers.parquetFull configuration with S3 for both config and templates:
# flapi.yaml (can be hosted at s3://my-bucket/configs/flapi.yaml)
project-name: S3 Cloud API
project-description: Fully cloud-native configuration
template:
path: s3://my-bucket/templates/
connections:
s3-data:
init: |
INSTALL httpfs;
LOAD httpfs;
SET s3_region='us-east-1';
properties:
bucket: my-data-bucket
base_path: s3://my-data-bucket/data/
duckdb:
access_mode: READ_ONLYUse local config with remote templates:
# flapi.yaml (local)
project-name: Mixed Storage API
project-description: Local config, remote templates
template:
path: https://templates.example.com/sqls/
connections:
local-parquet:
properties:
path: ./data/customers.parquetSpecify remote templates per-endpoint:
# sqls/customers.yaml
url-path: /customers
method: GET
# Remote template - full URL overrides template.path
template-source: https://templates.example.com/customers.sql
connection:
- local-dataOption 1: Environment Variables
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_REGION=us-east-1
# Optional: for temporary credentials
export AWS_SESSION_TOKEN=your-session-tokenOption 2: IAM Role (EC2/ECS/Lambda)
When running on AWS infrastructure, flAPI automatically uses the instance's IAM role. No environment variables needed.
Option 3: AWS Profile
export AWS_PROFILE=my-profileDuckDB's httpfs reads gs:// through Google's S3-interoperability API, which
takes an HMAC key (Cloud Storage → Settings → Interoperability). It does not
use service-account key files, Application Default Credentials or an attached
service account, and flAPI does not configure GCS from GOOGLE_APPLICATION_CREDENTIALS
or GOOGLE_CLOUD_PROJECT: when either is set, startup logs a warning saying so.
Create the secret in a connection's init, with the values from whitelisted
environment variables:
template:
path: ./sqls
environment-whitelist: ['^GCS_HMAC_.*']
connections:
events:
init: |
INSTALL httpfs;
LOAD httpfs;
CREATE OR REPLACE SECRET gcs_hmac (
TYPE gcs,
KEY_ID '{{env.GCS_HMAC_KEY_ID}}',
SECRET '{{env.GCS_HMAC_SECRET}}'
);
properties:
path: 'gs://my-bucket/events/*.parquet'Option 1: Storage Account Key
export AZURE_STORAGE_ACCOUNT=mystorageaccount
export AZURE_STORAGE_KEY=your-storage-keyOption 2: Connection String
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;..."Option 3: Managed Identity
When running on Azure infrastructure, uses the managed identity automatically.
FROM public.ecr.aws/lambda/provided:al2
COPY flapi /var/task/
COPY bootstrap /var/task/
# Config loaded from S3 at runtime
ENV FLAPI_CONFIG=s3://my-bucket/configs/flapi.yaml
ENV AWS_REGION=us-east-1
CMD ["flapi"]# bootstrap script
#!/bin/sh
exec /var/task/flapi --config "$FLAPI_CONFIG"FROM gcr.io/distroless/cc
COPY flapi /flapi
# Config loaded from GCS at runtime. The object must be readable without
# credentials (see "Loading Config from GCS"); otherwise pack the config in.
ENV FLAPI_CONFIG=gs://my-bucket/configs/flapi.yaml
ENTRYPOINT ["/flapi", "--config", "${FLAPI_CONFIG}"]# container-app.yaml
properties:
configuration:
secrets:
- name: azure-storage-key
value: your-storage-key
template:
containers:
- name: flapi
image: myregistry.azurecr.io/flapi:latest
env:
- name: FLAPI_CONFIG
value: az://my-container/configs/flapi.yaml
- name: AZURE_STORAGE_ACCOUNT
value: mystorageaccount
- name: AZURE_STORAGE_KEY
secretRef: azure-storage-keyAlways use https:// for remote configuration in production. HTTP URLs are supported but transmit configuration (potentially containing secrets) in plain text.
Use least-privilege IAM policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::my-bucket/configs/*", "arn:aws:s3:::my-bucket/templates/*"]
}
]
}Use environment variables for sensitive values:
# Good - secrets from environment
auth:
jwt-secret: '{{env.JWT_SECRET}}'
connections:
postgres:
properties:
password: '{{env.DB_PASSWORD}}'For S3, enable SSE:
aws s3 cp flapi.yaml s3://my-bucket/configs/ --sse AES256For AWS, configure VPC endpoints to access S3 without public internet:
aws ec2 create-vpc-endpoint \
--vpc-id vpc-xxx \
--service-name com.amazonaws.us-east-1.s3 \
--route-table-ids rtb-xxx-
Verify credentials are set correctly:
echo $AWS_ACCESS_KEY_ID echo $AWS_REGION
-
Test access with AWS CLI:
aws s3 ls s3://my-bucket/configs/
-
Check IAM permissions allow
s3:GetObject
-
Verify the full URL is correct:
curl -I https://example.com/templates/customers.sql
-
Check for typos in bucket names or paths
-
Ensure the file exists in the remote location
Remote configuration loading adds network latency to startup. To minimize impact:
- Use regional endpoints (same region as your compute)
- Keep configuration files small
- Consider caching (future feature)
When using remote template.path, relative paths in template-source are resolved against it:
# flapi.yaml
template:
path: s3://my-bucket/templates/
# sqls/customers.yaml
template-source: customers.sql
# Resolves to: s3://my-bucket/templates/customers.sqlTo use a different location, specify a full URL:
template-source: https://other-server.com/templates/customers.sql- Documentation index - Start here; organised by what you are trying to do
- Configuration Reference - Configuration file format (§ 2.11 for VFS configuration)
- CLI Reference - Server command-line options including
--config - Technical Design - VFS abstraction implementation details (historical design note, not maintained)