OW Preprint Analysis

Analysis code and data for the opioid withdrawal (OW) phenotyping preprint. The study uses automated home-cage behavioral phenotyping (JABS pipeline via Nextflow) to characterize morphine withdrawal across multiple inbred mouse strains and dose levels.

---

Project Overview

Raw per-video behavioral features (JABS pose-based behaviors, gait, morphometrics, fecal boli counts) are exported from a Nextflow pipeline and processed through four stages:

1. Nextflow postprocessing — QC screening, merging, and final curation of raw feature files into a single analysis-ready dataset.
2. Imputation — NA filtering, median imputation within strain/treatment groups, and delta computation (withdrawal minus baseline).
3. Dimensionality reduction — sparse PCA (sPCA) and sparse PLS-DA (sPLS-DA) to identify behavioral phenotypes that separate treatment conditions and strains.
4. Ordinal regression — regularized cumulative link models (CLM) with subject-grouped k-fold cross-validation and Ranked Probability Score evaluation to characterize the dose-response structure of withdrawal behavior.

---

Manuscript Methods Text

METHODS_sPCA.md, METHODS_sPLS-DA.md, METHODS_sPLSDA_performance.md, and METHODS_OrdinalRegression.md (repo root) are manuscript-ready methods write-ups for each analysis stage, verified against the current code and the generated result tables under results/preprint_figures/. They are written for the preprint's methods section, not for developers — see src/README.md and src/nextflow_postprocess_src/README.md for pipeline and code documentation.

---

Directory Structure

OW_preprint_analysis/
├── README.md                          ← this file
├── metadata.csv                       ← per-mouse experimental metadata (strain, sex, dose, cohort)
├── videos_to_exclude.txt              ← NetworkFilenames manually flagged for exclusion after QC review
├── OW_final_nextflow_dataset.csv      ← analysis-ready dataset (output of postprocessing pipeline)
├── features_removed_from_curated_dataset.csv  ← log of features excluded and the reason for each
│
├── Nextflow_Output/                   ← raw per-cohort Nextflow output (inputs to postprocessing)
│   ├── OW_Pilot/, OW_V1-5/, OW_WS1-2/, OW_WS3-4/, OW_WS5/, OW_All/, OW_corner_corrected/, All_groups/
│   │                                  ← one folder per data-intake batch; names grow as cohorts are added
│   └── final_nextflow_feature_data/  ← merged intermediate files produced by the postprocessing pipeline
│
├── src/                               ← analysis scripts (see src/README.md for details)
│   ├── utils.R                        ← shared functions sourced by all analysis scripts
│   ├── OW_imputation_sPCA_LDA.R       ← main imputation + sPCA / sPLS-DA pipeline
│   ├── OW_dose_response.R             ← dose-response ordinal regression (CLM)
│   ├── OW_strain_delta_figs.R         ← strain-level delta figures and feature selection
│   ├── create_summary_table.R         ← builds manuscript summary/supplemental tables from the outputs above
│   ├── verify_cross_model.R           ← standalone structural checks for the cross-model validation matrix
│   ├── nextflow_postprocess_src/      ← Nextflow postprocessing pipeline (run first; see below)
│   └── Legacy/                        ← earlier exploratory scripts; not part of primary pipeline
│
├── results/                           ← per-strain imputed CSVs and analysis outputs
│   ├── *_imputed.csv                  ← median-imputed feature data per strain (all timepoints)
│   ├── *_imputed_completed.csv        ← imputed data with synthesised missing timepoints added
│   ├── all_mice_imputed*.csv          ← pooled-strain equivalents of the above
│   ├── all_mouse_imputed_completed_deltas.csv  ← withdrawal-minus-baseline delta values, all strains
│   ├── dose_response_*.csv            ← dose-response subsets for strains included in that design
│   └── preprint_figures/              ← final PDFs for manuscript figures (gitignored)
│
└── qc/                                ← QC logs and figures generated by the postprocessing pipeline
    ├── nextflow_qc_logs/              ← qc_all.csv, qc_failed.csv (per-video pass/fail metrics)
    ├── missing_or_dup_data/           ← CSVs/spreadsheets flagging missing or duplicated recordings
    └── qc_figs/                       ← QC figures: fecal boli trends, scatter-plot outliers,
                                           z-score boxplots by feature class

---

Key Input Files

File	Description
metadata.csv	One row per mouse; columns include MouseID, strain, sex, treatment dose, and cohort. Joined to feature data during postprocessing.
videos_to_exclude.txt	NetworkFilenames (one per line) excluded after manual review of qc/nextflow_qc_logs/qc_failed.csv and QC figures.
OW_final_nextflow_dataset.csv	Final curated wide-format dataset with one row per mouse per video session. Direct input to src/OW_imputation_sPCA_LDA.R and src/OW_dose_response.R.
features_removed_from_curated_dataset.csv	Audit log of every feature column removed during postprocessing and the reason (e.g., zero variance, high false-positive rate, manual exclusion).

---

Nextflow Postprocessing Pipeline (src/nextflow_postprocess_src/)

Run these scripts sequentially before any analysis. Each script reads outputs from the previous one. Full documentation is in src/nextflow_postprocess_src/README.md.

Step	Script	Purpose
1	NextFlow_Output_QC_Postprocess_1.R	Reads raw Nextflow output (features, gait, morphometrics, fecal boli, QC reports); applies QC thresholds; flags duplicates and missing recordings; produces intermediate CSVs and fecal boli QC figures for manual review.
2	merge_all_dataframes_2.R	Merges QC-passed feature files across all cohorts with metadata; checks for cross-file discrepancies; outputs merged_nextflow_dataset.csv.
3	final_dataframe_QC_3.R	Final curation: removes zero-variance and manually excluded features; generates z-score outlier figures; outputs OW_final_nextflow_dataset.csv and features_removed_from_curated_dataset.csv.

corr_heatmaps_4.R (pairwise/partial feature correlation heatmaps) was removed from the pipeline; the three scripts above are the current postprocessing sequence.

---

Analysis Scripts (src/)

The three analysis scripts below (plus create_summary_table.R, which runs last) set setwd() to the project root and source src/utils.R for shared functions. Run them from the project root after the postprocessing pipeline is complete.

src/OW_imputation_sPCA_LDA.R — Main analysis

Reads OW_final_nextflow_dataset.csv, applies NA filtering and median imputation within strain/treatment groups, synthesises missing timepoints (complete_and_impute), and computes withdrawal-minus-baseline delta values. Then runs sPCA and sPLS-DA:

• All mice combined
• Per strain (8 strains, after excluding C3H/HeJ and SWR/J)
• By pre-defined locomotor activity group (Low / High)
• A cross-group performance matrix (how well does each strain's model generalise to others?)

Outputs per-strain imputed CSVs to results/, scatter plots, loading tables, and confusion matrices.

src/OW_dose_response.R — Dose-response analysis

Ordinal regression only — this script contains no sPCA or sPLS-DA; all dimensionality-reduction models live in OW_imputation_sPCA_LDA.R. Treats dose as a ranked outcome (Naive < 10-50_1/day < 20-100_2/day) for the four strains in that design (129S1/SvImJ, BALB/cJ, C57BL/6J, DBA/2J) and fits a regularized cumulative link model (CLM, lasso penalty via ordinalNet) per strain plus one pooled model:

• Subject-grouped 10-fold cross-validation (mouse-level, never split across folds) to guard against overfitting
• Ranked Probability Score (RPS) skill score with bootstrap 95% CI to evaluate ordinal predictive accuracy
• A 4×4 cross-strain generalisation matrix (how well does a CLM trained on one strain predict dose in another?)
• Negated linear predictor (eta') as a continuous withdrawal severity score per mouse
• Latent variable threshold diagrams visualising the CLM dose boundaries
• Projection of two held-out C57BL/6J cohorts (Acute, NAL) onto the fitted latent axis, without refitting

src/OW_strain_delta_figs.R — Strain delta figures

Reads the pooled delta dataset (all_mouse_imputed_completed_deltas.csv). Identifies the top 100 most variable and most directionally consistent features across strains (using select_consistent_features). Outputs large survey PDFs for manual feature selection, then generates fixed-panel publication figures for the 6 features selected for the paper. Includes Kruskal-Wallis strain-effect tests for each.

src/create_summary_table.R — Manuscript summary tables

Runs last. Reads the CSVs already written by the three scripts above — no model is refit or re-tuned here — and assembles the sample-size, model-parameter, non-zero-loading, and OR-performance tables used in the manuscript and its supplement, plus a feature definitions glossary built from a hand-curated feature guide (Nextflow_features_measures.xlsx). Writes both CSV and copy-paste-ready LaTeX (.txt) versions via write_latex_table() (utils.R §10).

src/verify_cross_model.R — Cross-model validation checks

Not part of the pipeline: a standalone, fast (seconds) structural check on the cross-group performance matrix in OW_imputation_sPCA_LDA.R. Asserts that comparison_group() reproduces CROSS_MODEL_MAP.csv exactly, that no cell shares mice between its train and test frames, and that every training frame is large enough to tune. Source OW_imputation_sPCA_LDA.R through the cross-model helper definitions, then source this script. Run it after touching anything in that section.

src/utils.R — Shared utility library

Sourced by all four scripts above (the three analysis scripts plus create_summary_table.R). Contains:

Section	Contents
1. Color Palettes & Shapes	Named colour vectors and point-shape vectors for treatment groups, dose levels, and strains
2. Theme Functions	theme_strain_deltas, theme_sPCA, theme_sPLSDA — ggplot2 themes for each figure type
3. Plot Export	print_fixed_panel_size() — exports ggplots as PDFs with fixed panel dimensions
4. Data Quality & Imputation	make_na_summary, filter_na_summary, median_imputation_by_strain_and_tx, complete_and_impute
5. Feature Selection	select_consistent_features — directional consistency filter + signal-to-noise ranking
6. Dimensionality Reduction	tune_spca_model (expensive keepX/ncomp search, results cached to CSV) plus sPCA_by_group and sPLSDA_by_group — two independent models over the same feature block, each returning figures and data frames
7. Cross-Validation	binary_metrics, assign_folds_by_group, assign_folds_stratified_by_group, kfold_splsda — the subject-grouped k-fold CV actually used by every sPLS-DA model and the cross-group matrix. loocv/loocv_multiclass (leave-one-out) are also defined here but are legacy reference implementations not called by any current script.
8. Statistical Utilities	bootstrap_rps_skill — paired bootstrap CI for the RPS skill score
9. Visualization Utilities	plot_or_latent_diagram — CLM latent variable threshold diagram
10. LaTeX Table Export	write_latex_table and helpers — turns a data frame into a copy-paste-ready booktabs table; used by OW_strain_delta_figs.R and create_summary_table.R

---

Results Directory (results/)

Populated by running the analysis scripts. Intermediate imputed datasets are committed to the repo; final figures are gitignored (results/preprint_figures/).

File pattern	Produced by	Contents
*_imputed.csv	OW_imputation_sPCA_LDA.R	Median-imputed features for each strain (all recorded timepoints)
*_imputed_completed.csv	OW_imputation_sPCA_LDA.R	Same, with synthesised rows for missing timepoints
all_mouse_imputed_completed_deltas.csv	OW_imputation_sPCA_LDA.R	Withdrawal-minus-baseline delta values across all strains
*dose_response_imputed.csv	OW_dose_response.R	Imputed data for the dose-response strains
dose_response_all_mice_imputed.csv	OW_dose_response.R	Pooled dose-response imputed dataset
preprint_figures/tables/*.csv / .txt	create_summary_table.R	Manuscript sample-size, model-parameter, non-zero-loading, and OR-performance tables (CSV + LaTeX)

---

QC Directory (qc/)

Generated entirely by the postprocessing pipeline. Intended for manual review between pipeline steps — no analysis scripts read from this directory.

Path	Contents
qc/nextflow_qc_logs/qc_all.csv	Per-video QC metrics (clip length, tracklet count, pose coverage) for every recording
qc/nextflow_qc_logs/qc_failed.csv	Subset of the above that failed one or more QC thresholds
qc/missing_or_dup_data/	CSVs and an Excel file flagging recordings present in one data file but absent in another, or with duplicate rows
qc/qc_figs/	Fecal boli trends, scatter-plot outlier figures, and z-score boxplots by feature class (gait, JABS, morphometrics)