Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,22 @@
# AMCONNECT_SEED=true
# AMCONNECT_HARVEST=true

# ---- Users and login (see README "Users and access") ------------------------
# SUPERADMIN_EMAIL=admin@example.org
# SUPERADMIN_PASSWORD=admin12345
# GOOGLE_CLIENT_ID=
# GOOGLE_CLIENT_SECRET=
# MAIL_FROM=no-reply@amconnect.local
# client secrets shared between Keycloak and the apps
# PLATFORM_CLIENT_SECRET=dev-platform-secret
# PLATFORM_ADMIN_CLIENT_SECRET=dev-platform-admin-secret
# CKAN_CLIENT_SECRET=dev-ckan-secret
# AUTH_SECRET=dev-auth-secret-change-me

# ---- Admin accounts --------------------------------------------------------
# CKAN_SYSADMIN_NAME=admin
# CKAN_SYSADMIN_PASSWORD=admin12345
# CKAN_SYSADMIN_EMAIL=admin@example.org
# CKAN_SYSADMIN_EMAIL=ckan-admin@example.org
# GEOSERVER_ADMIN_USER=admin
# GEOSERVER_ADMIN_PASSWORD=geoserver
# KEYCLOAK_ADMIN=admin
Expand Down
1 change: 1 addition & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
c75935a8f3cd40e71423b86c31f8b749fa09e081 # ruff and Prettier formatting
73 changes: 73 additions & 0 deletions .github/workflows/quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Code checks: the same `make` targets developers run, on the runner instead of compose.
name: quality

on:
pull_request:
types: [opened, synchronize, reopened, edited]
push:
branches: [main]

permissions:
contents: read

jobs:
python:
runs-on: ubuntu-latest
services:
db:
image: postgis/postgis:16-3.5
env:
POSTGRES_USER: platform
POSTGRES_PASSWORD: platform
POSTGRES_DB: platform
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U platform" --health-interval 5s --health-retries 20
env:
BACKEND: uv run --directory backend
DATABASE_URL: postgresql+psycopg://platform:platform@localhost/platform
KEYCLOAK_INTERNAL_URL: http://keycloak.invalid
KEYCLOAK_ISSUER: http://keycloak.invalid/realms/amconnect
PLATFORM_CLIENT_SECRET: test
PLATFORM_ADMIN_CLIENT_SECRET: test
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0 # diff-cover compares with the base branch
- uses: astral-sh/setup-uv@v10.3.3
- run: make check-python
- run: make test
- name: Coverage of changed backend lines (80% minimum)
if: github.event_name == 'pull_request'
working-directory: backend
run: >-
uv run diff-cover coverage.xml
--compare-branch=origin/${{ github.base_ref }} --fail-under=80

frontend:
runs-on: ubuntu-latest
env:
FRONTEND: pnpm --dir frontend
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 26
- run: npm install -g pnpm@12
- run: pnpm --dir frontend install --frozen-lockfile
- run: make check-frontend

docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- run: make docs
- name: Prose lint of the PR description (reports only)
if: github.event_name == 'pull_request'
continue-on-error: true
env:
BODY: ${{ github.event.pull_request.body }}
run: |
printf '%s\n' "$BODY" > pr-description.md
docker run --rm -v "$PWD":/docs -w /docs jdkato/vale:v3.25.0 pr-description.md

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,7 @@ __pycache__/
.idea/
*.swp
.DS_Store

# test reports
.coverage
coverage.xml
7 changes: 7 additions & 0 deletions .vale.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Prose lint for Markdown and PR descriptions (`make docs`). Reports only, never blocks.
StylesPath = .vale/styles
MinAlertLevel = suggestion
Vocab = AMConnect

[*.md]
BasedOnStyles = Vale, Akvo
36 changes: 36 additions & 0 deletions .vale/styles/Akvo/AIWords.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Words and phrases that make text read as machine-written. Seeded from the
# no-ai-slop and humanizer skill lists; add more as reviewers spot them.
extends: existence
message: "'%s' reads as filler; say the plain thing instead."
level: warning
ignorecase: true
tokens:
- delve
- robust
- seamless(ly)?
- leverag(e|es|ed|ing)
- utiliz(e|es|ed|ing)
- comprehensive
- cutting-edge
- game-changer
- tapestry
- testament
- pivotal
- crucial
- streamlin(e|es|ed|ing)
- unlock(s|ed|ing)?
- empower(s|ed|ing)?
- elevat(e|es|ed|ing)
- foster(s|ed|ing)?
- holistic
- synerg(y|ies)
- vibrant
- boasts
- furthermore
- moreover
- it's worth noting
- in today's
- at its core
- plays a (crucial|key|vital) role
- rest assured
- navigat(e|ing) the complexities
6 changes: 6 additions & 0 deletions .vale/styles/Akvo/EmDash.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
extends: existence
message: "Em dash: a comma, colon or full stop usually reads better."
level: suggestion
nonword: true
tokens:
- "—"
7 changes: 7 additions & 0 deletions .vale/styles/config/vocabularies/AMConnect/accept.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
AMConnect
SQLModel
uv
Mailpit
pgAdmin
pycsw
[Cc]yclomatic
1 change: 1 addition & 0 deletions AGENTS.md
36 changes: 36 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# AMConnect

Monorepo: `frontend/` (Next.js, TypeScript), `backend/` (FastAPI, SQLModel, Alembic),
`ckan/` and `geoserver/` (Data Service), `keycloak/` (realm config). See README.md.

## Done

- `make check` passes (lint, format, types; needs `docker compose up -d`).
- `make test` passes. A bug fix comes with a test that failed before the fix.
- `make format` applies the formatters.

## Limits (backend and frontend)

Short units are easier to review and change.

- Cyclomatic complexity 10, nesting depth 3: fewer paths to test.
- 50 lines per function, 400 per module: one screen, one purpose.
- 5 parameters: more means the function does too much.
- Types on every exported or public function; no `any`, `Any`, `@ts-ignore` or bare
`# type: ignore`.

## Tests

- Every test asserts a result.
- Mock only external boundaries: Keycloak's HTTP API, SMTP, time. Not our own code.
- At least 80% of a PR's changed backend lines must be covered (CI, `diff-cover`).

## Writing

- Comments are short and say why, not what. Prefer clear names.
- Plain words, no invented shorthand. Vale (`make docs`) flags AI-sounding words.

## PRs

- One concern per PR; refactors apart from behavior changes.
- Over 400 changed lines: propose a split.
35 changes: 35 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Code checks. Locally they run in the compose containers (`docker compose up -d` first);
# CI runs the same targets on the runner by overriding the prefixes below.
BACKEND ?= docker compose exec -T backend uv run
FRONTEND ?= docker compose exec -T frontend pnpm
RUFF ?= uvx ruff@0.17.3
VALE ?= docker run --rm -v "$(CURDIR)":/docs -w /docs jdkato/vale:v3.25.0

.PHONY: check check-python check-frontend docs test format

## done means this passes
check: check-python check-frontend docs

check-python:
$(RUFF) check .
$(RUFF) format --check .
$(BACKEND) pyrefly check

check-frontend:
$(FRONTEND) exec eslint .
$(FRONTEND) exec prettier --check .
$(FRONTEND) exec next typegen # route types such as LayoutProps, normally made by next dev
$(FRONTEND) exec tsc --noEmit

## prose lint; reports only, never fails the build
docs:
-$(VALE) $(shell git ls-files '*.md')

test:
$(BACKEND) pytest --cov --cov-report=term-missing --cov-report=xml

## apply formatters and import sorting
format:
$(RUFF) check --select I --fix .
$(RUFF) format .
$(FRONTEND) exec prettier --write .
73 changes: 64 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ AMConnect platform and data service.
| Database | `db/` | One PostGIS server, one database per service |

The CKAN and GeoServer parts come from the
[poc-amconnect](https://github.com/dedenbangkit/poc-amconnect) proof of concept.
[`poc-amconnect`](https://github.com/dedenbangkit/poc-amconnect) proof of concept.

## Run

Expand All @@ -26,25 +26,71 @@ Reset everything with `docker compose down -v`.

| Service | URL | Login |
|---|---|---|
| Frontend | http://localhost:3000 | |
| Frontend | http://localhost:3000 | super admin `admin@example.org` / `admin12345` |
| Backend | http://localhost:8000/docs, `/health` | |
| CKAN | http://localhost:5000 | `admin` / `admin12345` |
| CKAN | http://localhost:5000 | Keycloak users, or local sysadmin `admin` / `admin12345` |
| GeoServer | http://localhost:8080/geoserver | `admin` / `geoserver` |
| Keycloak | http://localhost:8180 | `admin` / `admin` |
| Keycloak admin console (developers only) | http://localhost:8180/admin | `admin` / `admin` |
| Mailpit (outgoing mail) | http://localhost:8025 | |
| pgAdmin | http://localhost:5050 | `dev@akvo.org` / `password` |
| Postgres | `localhost:5432` | `postgres` / `password` |

Databases (user = password = name): `ckan`, `keycloak`, `platform`, `gis`.
They are created by `db/init/20-databases.sql` on the first start of an empty volume only.

## Users and access

Keycloak holds every account; users never see a Keycloak page. Platform and CKAN show
their own login forms and check passwords with Keycloak server-side. Google login only
redirects through Keycloak to Google.

| Realm role | Grants |
|---|---|
| `super-admin` | can grant `user-manager` |
| `user-manager` | can invite users (Platform `/invite`), granting only access they hold |
| `ckan-access` | can log in to CKAN; without it CKAN refuses the login and creates no account |
| `geoserver-access` | can read `amconnect` GeoServer layers with `Authorization: Bearer <access token>` |

Invitations are emailed (see Mailpit); the link lets the invitee set name and password,
or continue with Google. Until the invitee does, inviting the same email again replaces
the invitation. Passwords need 8 characters and must not be the email; repeated failed
logins lock the account for a while. The super admin comes from `SUPERADMIN_EMAIL` /
`SUPERADMIN_PASSWORD` in `.env`.

Keycloak config lives in `keycloak/realm-amconnect.json`. The `keycloak-config` service
([keycloak-config-cli](https://github.com/adorsys/keycloak-config-cli)) applies it to the
running Keycloak on every `docker compose up -d`: edit the file, run `up`, done. Users are
kept, and the super admin's password is set only when that user is first created.

* Values come from the environment as `$(env:NAME)`, not `${NAME}`.
* Something removed from the file is deleted only if keycloak-config-cli created it.
* Renaming a role is a delete plus create: everyone loses it. Moving existing users to a
new role needs its own step (e.g. a backend migration calling Keycloak's Admin API).

GeoServer's Keycloak filter and layer rule are written by `geoserver/entrypoint.sh` at
every start.

Check the whole flow against the running stack (with demo data loaded, see below):

```bash
./scripts/e2e-issue-1.sh
```

### Google login

1. In Google Cloud console, create an OAuth client (Web application) with redirect URI
`http://localhost:8180/realms/amconnect/broker/google/endpoint`.
2. Put `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` in `.env` and `docker compose up -d`. The "Continue with Google" buttons appear.

Google logs in only an invited email; anyone else sees an AMConnect-styled Keycloak page
asking for an invitation.

## Backend

Migrations run on container start (`alembic upgrade head`). Code reloads on save.

```bash
docker compose exec backend pytest
docker compose exec backend alembic revision --autogenerate -m "add thing"
docker compose exec backend ruff check .
docker compose exec -u "$(id -u):$(id -g)" backend alembic revision --autogenerate -m "add thing"
```

Import new SQLModel table modules in `backend/alembic/env.py` so autogenerate sees them.
Expand All @@ -55,10 +101,19 @@ Add dependencies on the host with `uv add <pkg>` in `backend/`, then

`pnpm dev` runs in the container with hot reload.

## Checks

With the stack up, from the repo root:

```bash
docker compose exec frontend pnpm lint
make check # lint, format check, types (ruff, Pyrefly, ESLint, Prettier, tsc), Vale
make test # backend tests with coverage
make format # apply ruff and Prettier
```

CI (`.github/workflows/quality.yml`) runs the same targets. It also requires 80% coverage
of a PR's changed backend lines. Limits and conventions are in `CLAUDE.md`.

## Demo data

Example layers (`demo/geoserver-data`) and two mock harvest sources (`demo/mock-ckan`,
Expand All @@ -77,4 +132,4 @@ a known ckanext-harvest / CKAN 2.11 issue; the harvest still finishes.

## License

[AGPL-3.0](LICENSE), inherited from poc-amconnect.
[AGPL-3.0](LICENSE), inherited from `poc-amconnect`.
2 changes: 1 addition & 1 deletion backend/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
ENV UV_PROJECT_ENVIRONMENT=/venv UV_LINK_MODE=copy PATH=/venv/bin:$PATH
ENV UV_PROJECT_ENVIRONMENT=/venv UV_LINK_MODE=copy PATH=/venv/bin:$PATH PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen
Expand Down
8 changes: 8 additions & 0 deletions backend/alembic.ini
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,14 @@ path_separator = os
# ruff.executable = ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME

hooks = ruff_fix, ruff_format
ruff_fix.type = exec
ruff_fix.executable = ruff
ruff_fix.options = check --fix REVISION_SCRIPT_FILENAME
ruff_format.type = exec
ruff_format.executable = ruff
ruff_format.options = format REVISION_SCRIPT_FILENAME

# Logging configuration. This is also consumed by the user-maintained
# env.py script only.
[loggers]
Expand Down
2 changes: 1 addition & 1 deletion backend/alembic/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from sqlalchemy import engine_from_config, pool
from sqlmodel import SQLModel

import app.models # noqa: F401 (registers tables on SQLModel.metadata)
from alembic import context

# this is the Alembic Config object, which provides
Expand All @@ -16,7 +17,6 @@
if config.config_file_name is not None:
fileConfig(config.config_file_name)

# import SQLModel table modules here so autogenerate sees them
target_metadata = SQLModel.metadata

# other values from the config, defined by the needs of env.py,
Expand Down
Loading
Loading