xspct_db

xspct_db is an async user-data query service for mail filtering. It puts a single HTTP API in front of LDAP directories, MySQL databases, and static YAML data, so mail filters and other clients can look up users and their settings without implementing each backend's protocol, connection pooling, or caching themselves.

Rspamd is a primary integration: xspct_db can turn user data into a settings response for the current message. The same service also exposes single-user and batch lookup endpoints for clients that need the underlying data rather than Rspamd settings.

Overview

In a mail system, account information and filtering preferences may live in different places. A directory might hold mailbox addresses and aliases, while a database holds per-user policy and YAML supplies local overrides. xspct_db brings these sources together through configurable queries and returns a combined user object.

• Combine data sources. Configure multiple LDAP, MySQL, or YAML queries, translate backend fields into a common structure, and merge their results. Queries can use earlier results with use_result, or run in parallel phases where configured.
• Resolve addresses consistently. Rewrite relay addresses or SASL realms to a canonical mailbox before lookup. Optional domain and pattern prefilters exclude addresses before cache or backend access; wildcard domain queries provide a configurable fallback when a regular lookup finds no user.
• Build Rspamd settings. The /v1/rspamd-settings endpoint evaluates configurable rules against the queried user data to produce message-specific settings. Alias mapping and reject-level translation help adapt existing account preferences to Rspamd without duplicating that logic in each client.
• Reuse lookup results. An in-process object cache serves repeated lookups, with optional Redis as a shared second layer. Positive and negative results have configurable lifetimes. A separate response cache stores complete batch and Rspamd settings responses, avoiding repeated querying and serialization.
• Bound client waiting time. When request timeouts are enabled, separate foreground and background slot pools limit concurrent work. A query that exceeds its timeout returns 504 and can continue in the background to warm the cache; exhausted foreground capacity returns 503.
• Operate as a standalone service. The aiohttp server supports API-key authentication, optional TLS, JSON and optional msgpack responses, Prometheus metrics, and latency tracing. Backends and optional integrations are selected through a single YAML configuration file.

The service supplies data and settings; it does not scan message content or replace Rspamd's filtering engine. Clients remain responsible for applying the returned information to their mail-processing decisions.

Installation

Requires Python 3.10 or newer.

pip install "xspct_db @ git+https://github.com/HeinleinSupport/xspct_db.git"                   # core (aiohttp + PyYAML)
pip install "xspct_db[ldap] @ git+https://github.com/HeinleinSupport/xspct_db.git"          # + bonsai LDAP support
pip install "xspct_db[mysql] @ git+https://github.com/HeinleinSupport/xspct_db.git"         # + aiomysql support
pip install "xspct_db[redis] @ git+https://github.com/HeinleinSupport/xspct_db.git"         # + Redis caching
pip install "xspct_db[msgpack] @ git+https://github.com/HeinleinSupport/xspct_db.git"       # + msgpack body encoding
pip install "xspct_db[uvloop] @ git+https://github.com/HeinleinSupport/xspct_db.git"        # + uvloop event loop
pip install "xspct_db[all] @ git+https://github.com/HeinleinSupport/xspct_db.git"           # all optional backends
pip install "xspct_db[all,dev] @ git+https://github.com/HeinleinSupport/xspct_db.git"       # + dev/test dependencies
pip install "xspct_db[all,dev,docs] @ git+https://github.com/HeinleinSupport/xspct_db.git"  # + Sphinx documentation

Usage

xspct-db /etc/xspct-db.yml
# or
python -m xspct_db /etc/xspct-db.yml

Configuration is a single YAML file. All keys are optional; see docs/guide/configuration.md for the full reference, including examples for multiple databases with result merging and chained queries with use_result.

HTTP API

Method	Path	Auth	Description
GET	/	—	Health / liveness check
GET	/ping	—	Ping → Pong
GET	/metrics	optional	Prometheus metrics
GET	/v1/query/{user}	API key	Single user lookup
POST	/v1/query-json	API key	Batch user lookup
POST	/v1/rspamd-settings	API key	Rspamd settings blob

Legacy path prefixes (/query/v1/{user}, /query-json/v1, /rspamd-settings/v1) are also accepted for backwards compatibility.

Authentication uses the X-Api-Key header (configurable). See docs/guide/api.md for request/response details and all exposed metrics.

Backends

Backend	Extra	Description
yaml	—	Static data from the config file
dummy	—	No-op backend (returns the username as-is)
delay	—	Artificial-delay backend for testing
ldap	[ldap]	LDAP via bonsai with connection pooling
mysql	[mysql]	MySQL via aiomysql with connection pooling

Caching

xspct_db has three independent cache layers, all using TTLCache from cachetools:

Layer	Config key	Scope
L1 object cache	xspct_db_local_cache	Per-user lookup; zero latency, in-process
L2 object cache	xspct_db_redis_cache	Per-user lookup; shared across workers via Redis
Response cache	xspct_db_response_cache	Full serialised response body for POST /v1/query-json and POST /v1/rspamd-settings (JSON or msgpack, cached separately)

On a GET /v1/query/{user} request, lookups flow: L1 → L2 (Redis) → backend. On a POST request, the response cache is checked first; on a miss the backend is queried and the serialised response is stored for reuse.

Wildcard Domain Query

Any query can be configured with wildcard_domain_query: true. When a user address is not found by the regular lookup, xspct_db re-runs that query using a wildcard key derived from the address (default: use the full domain part → user@example.com → @example.com). The fallback result is returned under the original address in the response. When multiple queries enable wildcard fallback, each query derives its wildcard key from its own wildcard_key_pattern / wildcard_key_replacement settings.

The key derivation is configurable per query via wildcard_key_pattern and wildcard_key_replacement (Python re.sub syntax). See docs/guide/configuration.md for the full reference and examples.

Address Rewrite Rules

xspct_db_rewrite_rules rewrites incoming addresses before the prefilter, object cache lookup, and backend queries run. This is useful for relay domains, SASL realms, or other alias forms that should map to a canonical mailbox.

Rules are evaluated in order, and the first rule that actually changes the address wins. The response is still keyed under the original address received from the client, and both the original and canonical forms are registered as cache aliases.

xspct_db_rewrite_rules:
  - pattern: '^(.+)@relay\\.example\\.com$'
    replacement: '\\1@example.com'

See docs/guide/configuration.md for the full reference and examples.

Concurrency

When xspct_db_request_timeout is greater than 0, every query endpoint is guarded by two asyncio.Semaphore instances:

• Foreground slots (xspct_db_foreground_slots, default 30) — limit concurrent client-blocking queries. When all slots are busy a new request immediately receives 503 Service Overloaded.
• Background slots (xspct_db_background_slots, default 5) — when a query exceeds the timeout the client receives 504 Request Timeout but the backend task is promoted to a background slot so it can complete and warm the cache for subsequent requests. If no background slot is free the task is cancelled.

See docs/guide/configuration.md for all options.

Development

git clone https://github.com/HeinleinSupport/xspct_db
cd xspct_db
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov

# Build documentation
pip install -e ".[docs]"
python -m sphinx -b html docs docs/_build/html

License

European Union Public Licence v. 1.2 (EUPL-1.2) — see LICENSES/EUPL-1.2.txt.

The logos and app icons in docs/_static are licensed separately under Creative Commons Attribution-NoDerivatives 4.0 International, copyright 2026 Heinlein Support GmbH.

Changelog

See CHANGELOG.md.