- Predictor and behavior: inspect the scoring rule, explain one output, or measure predictive performance. Removal-based explanations distinguish output contributions from loss-based importance; these targets need different comparisons.
- Training data: influence methods estimate how training observations affect predictions or loss. Koh and Liang approximate parameter responses; deletion-and-retraining is a separate validation target.
- Representation and computation: probes test decodable information; causal abstractions test proposed computations through aligned interventions. Accessible information alone does not establish its use.
- Decision changes: recourse concerns feasible actions and their downstream effects. Intervening on model activations tests the program; intervening on people or physical systems requires assumptions about the world.
- Sparse scoring systems: learn small integer coefficients with sparsity and application-specific constraints. RiskSLIM — Ustun and Rudin (2019). The score is the predictor itself.
- Rule lists: ordered conditions followed by a default prediction. CORELS (2017/2018) certifies optimal regularized training risk over its supplied rule candidates when optimization completes.
- Additive models and interactions: GAMs sum feature functions on a link/score scale; GA²M (2013) adds pairwise terms. EBMs learn them by boosting. Additivity on that scale need not imply additive probability changes.
- Neural additive models: jointly train one neural network per input and sum their outputs on the score/link scale. Agarwal et al. (2020/2021). The original architecture exposes feature functions without explicit cross-feature interaction terms.
- RETAIN: uses reverse-time, two-level attention and computes visit/variable contributions with attention held fixed. Choi et al. (2016); code. Interpretation relies on this architecture's contribution calculation.
- Removal-based explanations: Covert et al. (JMLR 2021) connect SHAP, LIME, permutation tests and other methods through three choices: how features are removed, which behavior is measured, and how removal effects are summarized.
- Partial dependence (PDP): averages model predictions while varying selected inputs and retaining other observed values. Correlated inputs can produce unrealistic combinations. Definition and implementation.
- Individual conditional expectation (ICE): plots a separate prediction curve for each observation, exposing heterogeneity hidden by average partial dependence. Goldstein et al. (2013/2015).
- Accumulated local effects (ALE): accumulates locally averaged prediction differences, reducing extrapolation caused by dependent inputs. Apley and Zhu (2016/2020). Feature effects describe the fitted model, not causal effects.
- Permutation importance: measures a fitted model's score change after shuffling one input. Results depend on the model, score, evaluation data, and correlated substitutes. Definitions and limitations.
- Conditional subgroup importance: forms interpretable subgroups with trees, then permutes within them to approximate conditional feature importance; also adapts PDPs. Molnar et al. (2020/2023).
- Dependence and extrapolation: breaking feature relationships can make importance and effect plots reflect behavior in sparsely observed regions. Hooker and Mentch (2019) analyze this failure and alternatives.
- SAGE: assigns Shapley values to features using their contribution to expected predictive loss reduction. Covert et al. (NeurIPS 2020). This measures predictive power rather than averaging local prediction attributions.
- Model class reliance: ranges over feature importance across a specified class of sufficiently accurate predictors. Fisher et al. (JMLR 2019). Different models can rely on different inputs; the class and performance tolerance define the range.
- SHAP: decomposes an individual prediction into additive Shapley contributions. Lundberg and Lee (2017); code. Aggregation summarizes contributions over chosen observations; it does not turn them into loss-based importance.
- TreeSHAP: computes tree-model attributions under specified assumptions about missing or dependent features. Lundberg et al. (2018); TreeExplainer. Background data and perturbation mode affect the explanation.
- Shapley-value choices: the value function and treatment of absent features define the attribution problem; different constructions can disagree. Sundararajan and Najmi (ICML 2020).
- Duplicate-input example: let X2=X1 be a fair bit and f(x)=x1. At (1,1), conditional Shapley gives (0.25,0.25); marginal replacement gives (0.5,0). Game definitions change whether an unused but informative input receives credit.
- Integrated Gradients: integrates input gradients along a straight path from a baseline to the observation. Sundararajan, Taly, Yan (ICML 2017). Attributions sum to the output difference; the baseline is part of the explanation.
- DeepLIFT: propagates activation differences relative to reference activations to assign input contributions. Shrikumar, Greenside, Kundaje (ICML 2017); code.
- Layer-wise relevance propagation (LRP): redistributes a prediction backward through layers using relevance-propagation rules. Bach et al. (2015). The propagation rule determines the decomposition.
- Grad-CAM: weights convolutional feature maps with target gradients to produce a coarse spatial attribution map. Selvaraju et al. (2016/2017). A heatmap's visual appeal does not establish faithfulness.
- Attribution guarantees: IG completeness constrains a sum; Shapley axioms characterize an allocation for a specified game. Neither property alone identifies a unique internal computation.
- LIME: fits an interpretable surrogate around one prediction using weighted perturbed samples. Ribeiro, Singh, Guestrin (2016); code. Sampling, neighborhood weights, and surrogate capacity shape the result.
- MeLIME: extends LIME with data-distribution-aware sampling and alternative local interpretable models. Botari et al. (2020, arXiv). Its changes address the perturbation and surrogate-fitting stages.
- Anchors: searches for rules that preserve a prediction with high probability under a specified perturbation distribution. Ribeiro, Singh, Guestrin (AAAI 2018); code. Precision depends on that distribution.
- Abductive explanations: find input assignments sufficient to preserve a prediction for every completion in the encoded domain. Ignatiev et al. (2018). The guarantee requires a model/domain encoding and a sound constraint solver.
- Minimality and scope: subset-minimal means no assignment can be dropped; minimum-cardinality means fewest assignments. Unlike Anchors’ perturbation-distribution precision, formal sufficiency quantifies over all admissible completions.
- Counterfactual explanations: search for nearby input changes that produce a desired model output. Wachter, Mittelstadt, Russell (2017/2018). A changed prediction alone does not establish an actionable intervention.
- Diverse counterfactuals: DiCE balances proximity and diversity to offer several ways to change a prediction. Mothilal, Sharma, Tan (2019/2020); code. Feature constraints specify permitted changes.
- Actionable recourse: optimizes changes to actionable inputs for linear classifiers. Ustun et al. (2018/2019). Karimi et al. (2020/2021) distinguish input edits from interventions with downstream causal effects.
- Influence functions: approximate how upweighting a training example changes a test prediction or loss using derivatives of the fitted model. Koh and Liang (ICML 2017). Deep-network approximations require validation.
- TracIn: estimates training-example influence using gradient inner products across saved training checkpoints. Pruthi et al. (NeurIPS 2020). This differs from a single-checkpoint Hessian approximation.
- Datamodels — 2022. Predict a target example’s model output from which training examples were included. Ilyas et al. fit these surrogates using many subset-trained models, enabling dataset counterfactuals; the training algorithm and subset distribution define the target.
- TRAK — 2023. Approximates training-data effects using randomly projected model gradients and a kernel-based calculation. Evaluated against retraining-based prediction changes across vision and language tasks; these scores remain approximations to dataset interventions.
- TCAV: learns concept directions from examples and measures prediction sensitivity through directional derivatives. Kim et al. (ICML 2018). Results depend on concept examples and the chosen representation layer.
- Concept bottleneck models: predict annotated concepts before predicting the target; editing predicted concept values changes downstream predictions. Koh et al. (ICML 2020). These are interventions inside the model.
- Concept leakage: soft concept outputs can encode task information beyond their labels, even with sequential training. Mahinpei et al. (2021, preprint) demonstrate this on synthetic and MNIST tasks; concept supervision does not ensure semantic isolation.
- ProtoPNet: predicts by matching image regions to learned prototypes associated with training-image patches. This Looks Like That (2018/2019). Prototype similarity forms part of the predictor, rather than a separate local surrogate.
- Probing with controls: test what a representation makes decodable while checking what the probe itself can memorize. Hewitt and Liang (2019). Decodability alone does not establish that the original model uses that information.
- Visual representations: feature visualization optimizes inputs for activation; Network Dissection measures unit alignment with labeled concepts (2017). Activation patterns and semantic alignment do not by themselves test causal use.
- Causal abstraction: aligns neural representations with variables in a proposed causal model and tests corresponding interchange interventions. Geiger et al. (2021). This evaluates a specified computational hypothesis, not causal relations in the world.
- Superposition: controlled toy models store sparse features in overlapping directions, yielding polysemantic neurons. Elhage et al. (2022). This motivates sparse dictionaries; it does not establish that a learned dictionary recovers a real network’s true features.
- Sparse autoencoders: reconstruct activations as sparse combinations of learned directions, seeking features distributed across neurons. Towards Monosemanticity (2023, technical report); scaling and evaluation (2024).
- Feature absorption: a child concept can absorb a parent concept in an SAE, causing the parent latent to miss examples. Chanin et al. (2024). Larger dictionaries or greater sparsity need not resolve this failure.
- Native feature baselines: feed-forward key/value features compete with SAEs and transcoders on the tested interpretability measures. Ye et al. (NeurIPS 2025). Dictionary widths and available checkpoints constrain the comparison.
- SAE seed sensitivity: different initializations on the same model/data yield different feature dictionaries. Paulo and Belrose (2025). Overlap depends on architecture and matching; one fitted SAE is not a unique feature inventory.
- Neurons versus transcoders: many late-layer transcoder activations align with MLP neurons. Shabalin and Piotrowski (2026 preprint). Sparsified neurons need a learned linear output correction for competitive replacement KL divergence; transcoders better reconstruct outputs.
- Neuron comparison scope: neurons match transcoders on top-activation sort tests but fare worse across broader activation quantiles; transcoders better detect fine-grained concepts such as bigrams. Comparisons (2026), mainly English data.
- Activation patching: replaces internal activations between runs to test their effect on model behavior. Zhang and Nanda (2023/2024) show that corruption choices and evaluation metrics can change localization results.
- Transformer circuit framework: decomposes attention computation into query–key and output–value paths, including interactions across layers. Elhage et al. (2021, technical report). The initial analysis studies small attention-only models.
- Circuit discovery case study: identifies attention-head roles in GPT-2 small's indirect-object identification task and tests faithfulness, completeness, and minimality. Wang et al. (2022/2023). Findings concern that model and task.
- ACDC: removes computational edges through activation-patching tests while seeking to retain a specified behavior. Conmy et al. (2023). The recovered circuit depends on the dataset, intervention and metric.
- Sparse Feature Circuits / SHIFT: humans select task-irrelevant SAE circuit features for ablation, then retrain the classifier head. Marks et al. (2024/ICLR 2025) improve profession classification on gender-balanced biographies in their case study.
- Circuit tracing: builds prompt-specific attribution graphs using transcoders that approximate model components. Ameisen et al. (2025, technical report). Reconstruction error and frozen attention limit the original method.
- Attention-path extensions: traces feature interactions through attention query–key computations omitted by the original circuit-tracing method. Tracing Attention Computation (2025, technical report).
- Neuron circuits: MLP activations with RelP attribution yield sparse circuits competitive with SAE circuits on tested tasks. Arora et al. (2026). Activation site and attribution rule both matter; sparsity does not establish monosemanticity.
- MIB: circuit localization. Four tasks and five models; attribution and mask-optimization methods lead on the benchmark’s circuit metrics. Mueller et al. (ICML 2025). Scores trade behavioral recovery against circuit size under specified ablations.
- MIB: causal-variable localization. Supervised distributed alignment search leads; tested SAE features do not improve over neuron coordinates. Results (2025). This compares task-aligned intervention effects, not general feature readability.
- Circuit-score misranking: Geng et al. (2026, preprint v2) find intervention scores favoring equally sized circuits with worse held-out behavioral agreement. Restoring intact-model signals reverses selected failures; mechanism identity remains unverified.
- SAEBench: evaluates sparse autoencoders across eight metrics spanning interpretation, disentanglement, and downstream uses; reconstruction gains need not improve other scores. Karvonen et al. (ICML 2025); code.
- SAEBench reliability: Chanin’s audit (2026, preprint v1) tests reseed noise, synthetic ground-truth agreement and training-trajectory discrimination. Targeted Probe Perturbation and Spurious Correlation Removal fail multiple tests at canonical settings.
- Mechanism validation protocol: record checkpoint, task, intervention, reference behavior and metric; separate hypothesis-discovery examples from held-out tests. MIB supplies task datasets and intervention-based comparisons.
- Evaluation objectives: distinguish application-grounded, human-grounded, and functionally grounded evaluation. Doshi-Velez and Kim (2017, position paper). A proxy score and usefulness to a person answer different questions.
- Human usefulness: transparent, few-feature apartment-price models improved prediction simulation, but not advice-following; transparency sometimes impaired error correction. Poursabzi-Sangdeh et al. (CHI 2021), 3,800 participants; effects depended on presentation.
- Sources of disagreement: separate explainer approximation noise, changes from model fitting, different attribution games, and different accurate models. Changing the target is distinct from estimating the same target unreliably.
- Faithfulness versus plausibility: an explanation's agreement with model behavior differs from whether a person finds it convincing. Jacovi and Goldberg (2020) distinguish these evaluation targets.
- Saliency sanity checks: randomize model parameters or training labels to test whether explanations depend on learned behavior. Adebayo et al. (2018). Some tested methods remained visually similar after randomization.
- Infidelity and sensitivity: compare attributed changes with output changes under perturbations, and measure explanation variation. Yeh et al. (2019). A constant explanation can minimize sensitivity without being informative.
- Remove-and-retrain (ROAR): masks highly ranked inputs and retrains to assess retained predictive information. Hooker et al. (2018/2019). This tests a different question from perturbing a fixed predictor.
- Out-of-distribution perturbations: masking can make feature-importance scores and evaluation metrics reflect behavior outside the training distribution. Hase et al. (2021). Perturbation design is part of the evaluation.
- Adversarial explanations: classifiers can behave differently on real and perturbed inputs, concealing biased predictions from LIME/SHAP in the studied attacks. Slack et al. (2019/2020).
- Influence-estimation failures: network depth, regularization, and test-point choice affect approximation accuracy in the investigated neural networks. Basu, Pope, Feizi (2020/2021). Reported influence is not an exact retraining result.
- Attention as explanation: Jain and Wallace find alternative attention weights with similar predictions; Wiegreffe and Pinter challenge the inference and propose additional tests (2019). The result depends on architecture and evaluation.
- Chain-of-thought faithfulness: models can use prompt hints or biases without reporting them in their explanations. Turpin et al. (2023); Chen et al. (2025, arXiv) test reasoning models. Findings are conditional on the tested tasks and hints.
- Interpretation pitfalls: feature dependence, interactions, poor generalization, uncertainty, and unjustified causal readings can invalidate conclusions. Molnar et al. (2020); Limitations of Interpretable ML Methods (2020).
- Interpretable models: InterpretML provides Explainable Boosting Machines and post-hoc methods; framework paper (2019).
- Feature effects and attribution: scikit-learn for PDP/ICE and permutation importance; SHAP, LIME, ELI5 for model explanations.
- Neural-network attribution: Captum provides input, neuron, and layer attribution plus evaluation metrics for PyTorch; paper (2020). DeepLIFT supplies reference-based propagation.
- Explanation toolkits: AIX360 and Alibi collect multiple explanation methods; DiCE generates counterfactual examples.
- Mechanistic analysis: TransformerLens exposes activations and interventions; SAELens trains and analyzes sparse autoencoders; circuit-tracer constructs attribution graphs and supports interventions.
- Explanation evaluation: Quantus collects metrics for neural-network explanations; paper (2022). Metric assumptions and perturbation choices remain part of the comparison.
- TSViz: combines input/filter influence, filter clustering, and inverse optimization to inspect convolutional time-series models. Siddiqui et al. (2018).
- Interpretable Machine Learning — Molnar. Interpretable models, feature effects, attribution, examples, and limitations.
- Explanatory Model Analysis (2021) — Biecek and Burzykowski. Model diagnostics and local/global explanations, with R and Python examples.
- Interpretable models versus post-hoc explanations: Rudin argues for directly interpretable predictors in high-stakes settings where suitable models can be built. Perspective (2018/2019).
- Already Clean: Goodfire research essay (2026) argues that better visualizations and concept descriptions explain apparent polysemanticity. Its neuron examples motivate direct baselines; broad cleanliness claims remain preliminary.
- Glucose forecasting: applies RETAIN to longitudinal glucose prediction and examines its explanations. De Bois et al. (2020). Evidence concerns the studied type-2 diabetes cohort.
Feature selection · Feature extraction · Causal inference · Neural networks · Ensemble learning · Models.