RISM Matomo Log Importer

The JSON format used by our nginx server was not compatible with the existing Matomo logs importer, and was a bit of a mess modifying it to support it. The format we use is compatible with our Grafana log analyzer.

The primary importer is now a Go CLI. It preserves the existing command-line arguments from the Python script:

go run ./cmd/import_logs --config config-muscat.toml --dry-run /path/to/log.json

or, after building:

go build -o import_logs ./cmd/import_logs
./import_logs --config config-muscat.toml /path/to/log.json

If you run from cron or another working directory, pass the bots file explicitly:

./import_logs --config /opt/log-analysis/config.toml --bots-file /opt/log-analysis/bots.json /var/log/nginx/access.log.1

It supports plain, gzip, and bzip2 logs; config-driven filtering; CIDR exclusions; custom dimensions; Matomo batching; and bot detection based on the Matomo device-detector regex list.

Separate Bot Statistics

By default, detected bots are excluded from the primary Matomo site. To upload them to a separate site on the same Matomo instance while processing each log only once, configure bot routing:

[bot_routing]
enabled = true
idsite = "2"

Bot routing reuses the URL, authentication token, proxy, and batch size from [matomo]. It requires [exclude].bots = true. Path, extension, and CIDR exclusions still discard matching bot requests, so the bot site contains only traffic that would otherwise be eligible for statistics. The --report output labels these hits as Bot routed when this mode is enabled. Bot-site hits include Matomo's bots=1 tracking override, which makes Matomo retain detected bot user agents; the main site never receives this override.

To backfill only the bot site from an existing log, use --only-bots:

./import_logs --config config.toml --only-bots /var/log/nginx/access.log.1

This mode requires enabled bot routing. It still reads and classifies every record, but it skips all real-site uploads and submits only eligible bot hits. In reports, real hits are labelled Real skipped.

Bot Regex Data

The importer reads bot regex data from bots.json. Refresh the dataset with:

go run ./cmd/create_bots

Or build the generator binary first:

go build -o create_bots ./cmd/create_bots
./create_bots --output bots.json

The generated bots.json is a structured file that separates exact literal bot strings from regex-only patterns so the importer can use a fast literal matcher before falling back to regexp2.