Sequential experiment selection under uncertainty and limited budgets.
ECHO is a computational laboratory for one decision problem:
Given what I already measured, what experiment should I run next?
It is research software, not a product and not an autonomous scientist. Policies never see the hidden law. Evaluation is multi-metric, multi-seed, and records failures. The answer is allowed to be no, or only under conditions X.
experiment → observation → belief update → next experiment
Accuracy on a held-out set, a single benchmark score, or Bayesian optimization of a scalar do not automatically answer a scientific design question. ECHO isolates that question: under a fixed budget, which sequential policy recovers a hidden mechanism — and where does it fail?
The same loop runs on built-in synthetic worlds or on your function / CSV table.
| You bring | ECHO provides |
|---|---|
| A hidden response (f(x)), or a table of candidate experiments | Paired-seed comparison of policies |
| A budget and a noise level | GP posterior (+ optional hypothesis posterior) |
| Optional custom acquisition | RMSE, (P(H_{\mathrm{true}})), SHD, coverage, cost, failures, figures, LaTeX |
Full laboratory guide: docs/using_echo.md.
The Unix command echo is a shell builtin. Use python -m echo or echolab.
git clone https://github.com/chetx27/echo.git
cd echo
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytestRequires Python 3.9+ · NumPy · SciPy · scikit-learn · Matplotlib · PyYAML.
from echo.lab import compare_policies
def f(X):
return X[:, 0] ** 2 + 0.3 * X[:, 1]
compare_policies(
f,
dim=2,
algorithms=["random", "uncertainty", "echo_v0"],
budget=15,
n_candidates=400,
n_seeds=5,
name="my_system",
)python -m echo list
python -m echo compare --config configs/example_oscillator.yaml --jobs 4
python -m echo analyze --run results/example_oscillator
python -m echo benchOutputs (per run): results/<name>/summary.json, metrics.csv, table.tex, report.md, failures/, and figures under figures/<name>/. Interrupted runs resume. --jobs N parallelizes seeds.
- Worlds. Hidden synthetic laws, a Python callable, or a CSV lookup table. Candidates are finite. Noise is seed-paired so two policies querying the same index see the same (y).
- Belief. Exact RBF Gaussian process (misspecified for most worlds). When the world exposes a model class list, a posterior over parametric hypotheses is maintained from Gaussian marginal likelihoods.
- Policies. Random, greedy, uncertainty, diversity, expected improvement, GP-UCB, Thompson, local information gain, ECHO V0 (global expected knowledge change on a probe set), hypothesis discrimination, falsification, cost wrappers, open-loop ECHO.
- Isolation.
DecisionStatehas no hidden formula, no (\theta), no test set. Ground truth is evaluator-only. - Claims protocol. Pre-specified question, paired Wilcoxon tests, 95% CIs, seed-level failure files.
summary.jsonis the record; prose reports copy it.
No language model is used at any layer.
| World | Scientific target | Config |
|---|---|---|
| Nonlinear surface | Reconstruct (3x_1 + 2x_2^2 - 4\sin(x_3)) | configs/first_experiment.yaml |
| Competing hypotheses | Identify quadratic vs linear / sinusoid | configs/experiment2_hypotheses.yaml |
| Falsification | Disagreement with the leading class | configs/experiment3_falsification.yaml |
| Cost-aware design | Heterogeneous (x_1) costs | configs/experiment4_cost.yaml |
| Unseen form | Same policies, unused (f) | configs/experiment5_generalization.yaml |
| Causal SCM | Graph recovery after (\mathrm{do}(A,B)) | configs/experiment_causal.yaml |
| Multimodal | Three mechanisms in (x_1) | configs/experiment6_multimodal.yaml |
| Anomaly box | Compact structured violation of a linear law | configs/experiment7_anomaly.yaml |
python -m echo list environments · python -m echo list algorithms · python -m echo bench
ECHO-Bench is a local task index, not a community leaderboard.
Numbers are from results/*/summary.json. If this table disagrees with the JSON, the JSON wins. Full write-ups: docs/reports/.
| Study | Primary finding |
|---|---|
| Nonlinear | Expected improvement is the wrong objective for reconstructing (f). ECHO V0 ties uncertainty on RMSE ((p=0.52)). |
| Hypotheses | (P(H_{\mathrm{true}})) saturates for every policy, including random. The class list is too easy at this budget. |
| Falsification | Same ceiling. Open-loop ECHO wrecks surface RMSE (0.64 vs ~0.07). |
| Cost | Cost wrappers change spend, not identification. |
| Unseen form | EI/random ranking holds. ECHO V0 vs uncertainty is no longer a tie ((p=0.036), 19/30). One environment — not a general win. |
| Causal | Structural Hamming distance does not stably rank sequential designs. |
| Multimodal | All methods visit all three regions. Diversity is best on region RMSE. |
| Anomaly | Random finds the box more often than uncertainty / ECHO V0. Method-class failure for structured incompleteness. |
Reproduce a paper config (resumes completed trajectories):
python -m echo compare --config configs/first_experiment.yaml --jobs 8
python -m echo analyze --run results/first_experimentSmoke (CI, not claims): configs/smoke.yaml, configs/smoke_hypotheses.yaml.
| Document | Contents |
|---|---|
docs/using_echo.md |
Plug in a function, CSV, or custom acquisition |
docs/methodology.md |
Estimators, metrics, statistics, limitations |
docs/experiments.md |
Configs, commands, how to read outputs |
docs/research_questions.md |
Pre-specified questions |
docs/research_log.md |
Dated runs, including failures |
docs/literature_review.md |
Working bibliography |
docs/decisions/ |
Design decisions |
papers/draft/ |
No manuscript yet — do not invent one |
In scope. Sequential design, Gaussian-process and hypothesis-aware acquisition, synthetic and user-supplied systems, honest multi-seed evaluation.
Out of scope. Language models, wet-lab hardware control, a published community benchmark, a claim that ECHO is a better scientist.
A result in this repository is research only if it has a stated question, baselines, seeds, uncertainty, failure cases, and limitations.
There is no paper yet. Cite software version 0.2.0. Metadata: CITATION.cff. When a preprint exists, that file will be updated.
BibTeX
@software{echo2026,
title = {ECHO: sequential experiment selection under uncertainty},
author = {Chethana, G.},
year = {2026},
version = {0.2.0},
url = {https://github.com/chetx27/echo},
note = {Research software. No peer-reviewed publication at this version.}
}APA
Chethana, G. (2026). ECHO: sequential experiment selection under uncertainty (Version 0.2.0) [Computer software]. https://github.com/chetx27/echo
MIT. See LICENSE.