langid.cpp

warning: this is claude slop, unreviewed, and untested. probably fine as it uses solid methodology from transcribe.cpp

Spoken language identification on ggml. A small C library (langid_*) plus a CLI: give it 16 kHz mono audio, get a ranked list of languages back. Sibling of transcribe.cpp, sharing its vendored ggml, build mechanics, and numerical-validation methodology.

The first (and so far only) model family is ecapa_tdnn: SpeechBrain's ECAPA-TDNN trained on VoxLingua107, 107 languages, 21.3M parameters, Apache-2.0. See docs/model.md.

Status: pre-alpha. No releases, no tags, no published packages, and no published GGUFs. The engine is implemented and validated (see docs/validation.md), but everything about distribution is still local: you build it and you convert the model yourself. The API is 0.x and may break between minor versions.

Build

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

macOS and Linux. Metal is on by default on Apple silicon; -DLANGID_VULKAN=ON and -DLANGID_CUDA=ON opt into the other GPU backends.

There are four build lanes, differing only in where ggml comes from and how it is packaged. The first is the default and the other three are opt-in:

# 1. static liblangid, ggml linked in — self-contained langid-cli (default)
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release

# 2. shared liblangid + shared ggml beside it — the FFI/packaging posture
cmake -B build-shared -G Ninja -DLANGID_BUILD_SHARED=ON

# 3. + ggml backends as loadable plugins (a missing GPU driver is skipped,
#    not fatal); the host calls langid_init_backends(dir)
cmake -B build-dl -G Ninja -DLANGID_BUILD_SHARED=ON -DLANGID_GGML_BACKEND_DL=ON

# 4. link an ALREADY-INSTALLED ggml — one ggml for several engines
cmake -B build-sys -G Ninja -DLANGID_USE_SYSTEM_GGML=ON -DCMAKE_PREFIX_PATH=/opt/ggml

docs/distribution.md has the lane table, what each one installs, and the langid-link.json keys a non-CMake consumer reads.

Get the model

GGUFs are not committed to this repo and are not published anywhere yet. Convert from the upstream checkpoint (about 400 MB of downloads, once):

uv run --project scripts/envs/ecapa_tdnn \
  scripts/convert-ecapa_tdnn.py speechbrain/lang-id-voxlingua107-ecapa \
  --outtype f32

That writes models/lang-id-voxlingua107-ecapa/lang-id-voxlingua107-ecapa-F32.gguf (81 MiB). --outtype f16 (41 MiB) and --outtype q8_0 (23 MiB) produce the smaller production variants. F32 is the reference dtype every parity gate runs against.

Run

$ build/bin/langid-cli -m models/lang-id-voxlingua107-ecapa/lang-id-voxlingua107-ecapa-F32.gguf \
    samples/fleurs-ja.wav --top 3
1  ja  Japanese  prob=1.0000  prob_open=1.0000  logit=18.360
2  ko  Korean  prob=0.0000  prob_open=0.0000  logit=3.972
3  ha  Hausa  prob=0.0000  prob_open=0.0000  logit=2.318
label: ja
timings: load=46.8 ms  mel=2.4 ms  encode=25.2 ms  total=28.7 ms  wall_median=28.7 ms

--allow restricts the decision to the languages a user actually selected, which is where most of the accuracy comes from (docs/usage.md). --json emits one object per run:

$ build/bin/langid-cli -m .../lang-id-voxlingua107-ecapa-F32.gguf \
    samples/fleurs-de.wav --allow en,de,fr --top 3 --json
{"label":"de","label_index":18,"candidates":[{"index":18,"code":"de","name":"German","prob":0.999998,"prob_open":0.996940,"logit":15.881246},{"index":20,"code":"en","name":"English","prob":0.000002,"prob_open":0.000002,"logit":2.751277},{"index":28,"code":"fr","name":"French","prob":0.000000,"prob_open":0.000000,"logit":-2.309476}],"n_allowed":3,"allowed_mass":0.996942,"audio_ms":11160.000,"timings":{"load_ms":50.244,"mel_ms":2.639,"encode_ms":24.511,"total_ms":28.265,"wall_median_ms":28.266},"backend":"MTL0","version":"0.1.0"}

prob is the softmax over the allowed set, prob_open the softmax over all 107 labels, and allowed_mass the open-set probability the allowed set captured — a low value means the speech is probably not in the user's selection at all.

Audio must be 16 kHz mono. Resample first: ffmpeg -i in.mp3 -ar 16000 -ac 1 out.wav.

Using the C API

#include <langid.h>
#include <stdio.h>

int identify(const char * gguf, const float * pcm, int n_samples) {
    struct langid_model * model = NULL;
    if (langid_model_load(gguf, NULL, &model) != LANGID_OK) return 1;

    struct langid_context * ctx = NULL;
    if (langid_context_init(model, NULL, &ctx) != LANGID_OK) { langid_model_free(model); return 1; }

    const char * allowed[] = { "en", "de", "ja" };
    struct langid_identify_params ip;
    langid_identify_params_init(&ip);
    ip.allowed = allowed;
    ip.n_allowed = 3;
    ip.top_k = 3;

    int rc = 1;
    if (langid_identify(ctx, pcm, n_samples, &ip) == LANGID_OK) {
        struct langid_result res;
        langid_result_init(&res);
        langid_get_result(ctx, &res);
        for (int i = 0; i < res.n_candidates; i++) {
            struct langid_candidate c;
            langid_candidate_init(&c);
            langid_get_candidate(ctx, i, &c);
            printf("%s %s %.4f\n", c.code, c.name, (double) c.prob);
        }
        printf("allowed_mass %.4f over %.0f ms\n", (double) res.allowed_mass, (double) res.audio_ms);
        rc = 0;
    }
    langid_context_free(ctx);
    langid_model_free(model);
    return rc;
}

pcm is mono float32 in [-1, 1] at 16 kHz; at least 500 ms is required. Params structs must be initialised with their _init() function — {0} is rejected. Read include/langid.h for the threading and lifetime rules; the short version is that a model is shareable and immutable, a context is used by one thread at a time, and only one compute may be in flight per model in 0.x.

Validation

uv run scripts/validate.py all --family ecapa_tdnn

Runs the SpeechBrain reference dumper, the C++ engine, and a per-tensor comparison across 8 fixture clips. docs/validation.md explains what is proven and how; docs/accuracy.md has the FLEURS accuracy tables.

Docs

File	What
docs/usage.md	How to use it well: how much audio, allowed, thresholds, weak pairs
docs/model.md	Model card: provenance, architecture, the 107 labels, GGUF variants
docs/accuracy.md	Measured accuracy on FLEURS, and the C++ vs reference agreement gate
docs/latency.md	Measured latency by dtype, backend and thread count
docs/validation.md	How correctness is proven, and how to re-run each gate
docs/distribution.md	Install layout, the link manifest, the ggml pin policy
docs/environment-variables.md	Every environment variable the library and scripts read
docs/adding-a-family.md	Adding a second language-ID model family
docs/for-transcribe-cpp-users.md	Pairing langid.cpp with transcribe.cpp

notes/PLAN.md (implementation spec) and notes/DECISIONS.md (decision log) are maintainer-local and not in the repository yet; CONTRIBUTING.md the contribution policy and review gates.

License

MIT (see LICENSE). Vendored third-party components and their licenses are listed in THIRD-PARTY-LICENSES.md. The model weights are Apache-2.0 and are not distributed here.