Skip to content

Latest commit

 

History

History
118 lines (85 loc) · 3.87 KB

external_dependencies.md

File metadata and controls

118 lines (85 loc) · 3.87 KB

External Dependencies

MosaicLeaks keeps large local inputs out of the repo. Bring your private task files, BrowseComp-Plus corpus files, and dense embedding shards through paths, dataset names, or environment variables.

Local Private Documents

The generation pipeline reads local private documents from chunks_local.jsonl. Each chunk row includes:

  • chunk_id
  • doc_id
  • source_type
  • text
  • offsets
  • meta.task_id

Recommended metadata:

  • meta.company_name
  • meta.file_name
  • meta.relative_path
  • meta.extension

For paper replication, build this file from a DRBench-style task root with folders named like DR0001, DR0002, and so on. Each task may contain:

  • task-local files to chunk. The chunker has dedicated readers for PDF, DOC, DOCX, XLS, XLSX, and JSONL, and falls back to text decoding for other files.
  • context.json with company_info.name
  • config/eval.json with optional enterprise-fact seed QA pairs

For other local corpora, use mosaicleaks.data_prep.chunk_local_folder. The manifest form is recommended when you have more than one task:

{"path": "/path/to/file.pdf", "task_id": "task_001", "company_name": "Example Company"}

Manifest fields:

  • path: file path, absolute or relative to the manifest file
  • task_id: group ID used for chain construction and local retrieval
  • company_name: optional label used for metadata and split grouping
  • relative_path: optional stable path for generated doc_id values

Manual private QA seeds can also be supplied as JSONL and imported into secret_inventory.jsonl:

{"doc_id": "local/task_001/file.pdf", "question": "What was the internal launch date?", "answer": "Q3 2026", "secret_type": "dates"}

The importer checks that the referenced document exists and that the answer appears in the document text, unless --no-require-answer-in-doc is passed.

BrowseComp-Plus

We use BrowseComp-Plus as the public-web corpus for generation patterns that include web hops. Dataset generation reads prepared web chunks for bridge discovery:

chunks_web.jsonl

Each chunk row includes chunk_id, doc_id, text, offsets, and optional metadata. The retrieval helper can additionally use the full corpus in two ways:

  • --browsecomp-corpus Tevatron/browsecomp-plus-corpus, loaded through the datasets package.
  • --browsecomp-corpus /path/to/corpus.jsonl, where each row has docid, url, and text.

Dense BrowseComp retrieval, including the retrieval helper, uses embedding shards compatible with the MosaicLeaks BrowseCompHybridSearcher:

corpus.shard00000_of_000NN.pkl
corpus.shard00001_of_000NN.pkl
...

Each shard stores the dense vectors and lookup IDs used to map search results back to BrowseComp document IDs. The pickle payload is a two-item tuple: (embeddings, lookup), where embeddings is a 2D float array and lookup contains the corresponding BrowseComp document IDs in row order. Use the same embedding model for search queries that was used to create those shards.

Curated Task URLs

The generation pipeline can supplement BrowseComp-Plus with small curated web files derived from task URLs:

  • docs_curated_web.jsonl: fetched URL metadata and full text
  • chunks_curated_web.jsonl: paragraph/heading-aware chunks

These files are optional. When present, the helper and generator restrict their use by task ID so they do not become a general web source.

OpenAI-Compatible Endpoints

All LLM and embedding calls use OpenAI-compatible APIs:

  • chat completions for generation, validation, filtering, and rewording
  • embeddings for dense local and BrowseComp retrieval

StepFun can be used as the default provider by setting:

export MOSAICLEAKS_LLM_BASE_URL=http://your-endpoint:8000/v1
export MOSAICLEAKS_LLM_MODEL=step-3.5-flash

Equivalent local vLLM, OpenAI, OpenRouter, or other compatible endpoints work when they implement the same API shape.