Skip to content

Add dkg-rag-pipeline: RAG pipeline using DKG V10 memory as retrieval source - #29

Open
kiyeps wants to merge 8 commits into
OriginTrail:mainfrom
kiyeps:add/dkg-rag-pipeline
Open

kiyeps wants to merge 8 commits into
OriginTrail:mainfrom
kiyeps:add/dkg-rag-pipeline

Conversation

@kiyeps

@kiyeps kiyeps commented Aug 22, 2026 •

Copy link
Copy Markdown

Summary

Adds integrations/dkg-rag-pipeline.json — a registry entry for DKG RAG Pipeline, using DKG V10 Working/Shared Memory as a vector-searchable retrieval source with provenance citations.

Repo: https://github.com/kiyeps/dkg-rag-pipeline (commit pinned in entry)

What it does

  • Ingest: chunk documents, write each chunk as a provenance-tagged KA to DKG WM (source, chunk index, embedding ref).
  • Index: local cosine-similarity index over chunk embeddings (persisted JSON, no external vector DB).
  • Retrieve: embed query, rank chunks, return top-k with DKG KA UALs as citation.
  • Generate: feed retrieved chunks to a pluggable LLM with DKG provenance as grounding.
  • Local embedding fallback (character n-gram TF) — runs with zero external API keys; optional OpenAI-compatible provider via env.

Why it fits

Directly serves the bounty's LLM-Wiki / Autoresearch theme: WM/SWM as the retrieval source for grounded generation, with every answer citing its verifiable DKG source.

Round 1 scope compliance

  • WM + SWM only. No vm/publish tool. No endorsement/voting tools.
  • Secret redaction before any literal leaves the process.

Verification

  • Schema validation: 0 errors / 0 warnings
  • 35 unit tests passing (node:test)
  • Live smoke test: chunks written to WM, retrieved via SPARQL (8 bindings, Base Sepolia testnet)
  • Quads {subject,predicate,object}, no per-quad graph field
  • Subjects reject urn:dkg:file:* / urn:dkg:extraction:*

Submission metadata — final update (2026-09-08)

Tag: cfi-dkgv10-r1

Pinned commit: 69c24ef6f5165f84c2760b9302e895ac4bed3753 (v0.1.2)

Contributor attestation (§8a.8): I attest that this submission is my own original work (or properly licensed third-party work), and that it contains no intentional backdoors, no hidden remote-execution paths, and no undeclared network egress. — @kiyeps

Design brief: https://github.com/kiyeps/dkg-rag-pipeline/blob/main/docs/DESIGN.md

Demo: https://github.com/kiyeps/dkg-rag-pipeline/blob/main/docs/demo-transcript.txt — recorded against a live local DKG V10 node (v10.0.15, Base Sepolia testnet, context graph openclaw-curator). Real runtime interaction: ingest → query returning the chunk with its UAL citation. No mock node, no scripted transcript.

Security notes (§8): Secret redaction before any literal leaves the process. The optional HTTP service is hardened: binds 127.0.0.1 by default; remote exposure requires explicit DKG_RAG_HOST plus a bearer token (DKG_RAG_TOKEN) — requests without the token are rejected 401; CORS does not reflect arbitrary origins by default; request bodies are capped at 1 MiB (DKG_RAG_MAX_BODY) and oversized requests are rejected 413 before buffering. A self-review pass found and fixed these issues pre-submission (loopback bind, body cap, CORS default); no other credential, write-authority, or egress surface is exposed. Egress: local node only (http://127.0.0.1:9200), plus an optional user-configured embedding provider endpoint.

Maintenance commitment: Maintained by @kiyeps; support window ≥ 6 months post-merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant