Repository navigation
Add figures for similarity, drift and signal landscape; fix drift denominators - #2
Merged
Merged
Conversation
…ominators The figure set was thin and repetitive: seven of twelve charts were bars, and the three most interesting analyses -- similarity, drift, duplicates -- had no figure at all. There was also no volcano plot, the standard pharmacovigilance view, despite the data being right there. Adds three, each chosen by what the reader has to do with it rather than by variety: - signal_landscape: evidence against shrunk effect size, log-log. Emphasis rather than categorical -- the job is to find the handful of pairs top-right, and colouring five screening rules would bury exactly those. - class_similarity: between-class mean similarity as a sequential heatmap. This shows the validation instead of asserting it; the diagonal is visibly darkest. One off-diagonal is real: TNF inhibitors and bisphosphonates sit well above other cross-class pairs, sharing a rheumatology population. - drift_trajectories: what a flagged term looks like beside a stable one, so the step change reads as a shape rather than a count. Palette replaced with validated values. The slot order is the CVD-safety mechanism, not decoration: slots 1-3 clear the all-pairs separation and normal-vision floors in both light and dark modes. Drawing the drift figure exposed a defect in the drift stage itself. A few records carry an FDA receipt date outside the real collection window -- the earliest lands in 1999, in a quarter holding exactly one report -- so a term in that report had a 100% share and a step ratio near 10^5. Common, entirely stable terms (DIARRHOEA, HEADACHE, NAUSEA) were being flagged as vocabulary drift. With a denominator floor: flagged terms 3,100 -> 2,334 (42% -> 32%), mentions affected 24% -> 16%. Roughly 766 terms were false positives. Fixing that reintroduced an earlier bug, which the tests caught: the corpus window moved onto the trajectory table, which only contains quarters where a term appeared, so a late-onset term again defined its own window and its onset became invisible. The window now comes from the quarter denominators, which exist regardless of which terms appear. The hardcoded floor also made the detector return nothing on any corpus smaller than itself; it is now a parameter. Report page held to 1.5 MB: a Vega-Lite spec embeds its data, and the volcano at 12,000 points was 2.5 MB on its own. 142 tests.
… data Figures did not render. The page emitted only Vega-Lite specs and relied on three CDN scripts plus JavaScript to draw anything, and the embed's failure handler logged to the console -- so offline, behind a proxy, or with the CDN blocked, the reader got blank space and no error. The page now ships the PNG as what renders, and upgrades to the interactive chart only if Vega loads. Fifteen img tags where there were none. Rewrite of the prose after an external critique. The report's main character was the pipeline, and the data kept interrupting its account of how it improved on a 2020 notebook. That comparison ran through nine passages and is now gone entirely; what it occupied is spent on named drugs, reactions and counts. Specific changes: - Abstract led with development history. Now leads with findings: the two most frequent reaction entries are not clinical events; ATORVASTATIN and ATORVASTATIN CALCIUM held 275,333 and 269,328 reports as separate substances. - The top-signal section said "the pairs with the largest EB05 are not adverse reactions" and then described them abstractly. It now names them with counts and expectations -- technetium tetrofosmin with radioisotope scan abnormal, 257 against an expected 0.02; Proplex T with hepatitis C, reflecting plasma-derived contamination. Dropped the claim that expected pharmacology excludes harm. - Dechallenge used the wrong denominator. Among suspect drug records, the population the question is about, it is 0.248% (103,665 of 41,799,712), not the 0.147% across all records including concomitants. Removed the claim that the remainder "is association" -- absent fields are not evidence against involvement. - Drift said terms "look like vocabulary change rather than epidemiology" and called the estimate a floor. An abrupt change does not identify its cause and an unvalidated detector does not establish a bound. Now: flagged trajectories account for 16% of mentions, cause unverified. - Duplicates invented a bound between 2.7% and 9.4%. With 70% of cases never examined and unknown matcher precision, neither figure bounds anything. - Adjustment claimed confounding "does not create signals but does modify them". Threshold survival does not support either half. Now states what was measured. - Limitations contradicted the report: it claimed class interpretation was "not attempted" two sections after attempting it, and pointed to a section "below" that is above. Also cut: "the honest reading", "the practical lesson of the whole exercise", "that is the shrinkage doing its job", and the positive-control victory lap. Nine reference pairs met every criterion; one unflagged comparison pair does not establish specificity, and it no longer says otherwise. 142 tests.
A reader noticed lines appearing to run backwards in the drift figure. They were terms being connected across quarters in which they do not exist: three of the twelve plotted are present in only 29-37 of 90 quarters, so the line mark drew straight segments across years of absence, and where two crossed they read as a trajectory doubling back. Gaps now render as gaps. With gaps visible, every line broke at the same point -- including the stable ones, which should not happen. 2018Q1 held 371 reports where its neighbours hold roughly 300,000 and 376,000, because its DEMO table had parsed to zero rows. FDA reissued that quarter and ships its demographics as DEMO18Q1_new.txt. The member matcher required an exact stem and found nothing. DRUG, REAC and the rest parsed normally, but every child table joins to DEMO by record_id and dedup keys off it, so one unmatched filename removed the entire quarter from a corpus this report describes as complete. Recovered: 412,702 raw records, 319,031 distinct cases, 11,053 additional scored pairs. Positive controls remain 9/9 on all five rules. Two fixes. The matcher accepts a trailing _qualifier, preferring the bare stem where both exist. More importantly, a missing DEMO now raises instead of returning zero: harmonize reported "DEMO: 0" for an entire pipeline run and nothing looked at it, which is the same silent-zero shape as the gndr_cod mapping, the RxNav encoding, and the two mis-scoped metrics. Notable that no test found this. The download verified, the checksums matched, the stage exited clean, and the quarter count was right. A chart that looked wrong found it. 147 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds figures for the analyses that had none, rewrites the report prose, and fixes a quarter that was missing from the corpus.
Figures
Seven of twelve charts were bars, and similarity, drift and duplicates had no figure at all. Adds three: a signal landscape (evidence vs effect size), a drug-class similarity heatmap, and drift trajectories.
JS issue.
2018Q1 was missing
FDA reissued that quarter as
DEMO18Q1_new.txtRecovered 412,702 raw records / 319,031 distinct cases. A missing DEMO now raises instead of returning zero.
Report
Rewritten after external review. Cut the running comparison with the old notebook (nine passages), the overclaiming, and methodology narration. Replaced abstractions with named pairs and counts. Fixed the dechallenge denominator — 0.248% among suspect drug records, not 0.147% across all records — and removed invented bounds on the duplicate rate.
Numbers
20.6M distinct cases · 2.01M scored pairs · 688,879 at EB05 ≥ 2 · positive controls 9/9