NeuroQuery audits of search outputs: Supplementary Materials

Date: October 22, 2025 NeuroQuery ASD vs Without-ASD queries: Corpus Stability & Drift Methods: • Title only corpora built from two NeuroQuery query outputs (no hand curation). • Tokenization: lowercase, hyphen to space, punctuation stripped, light stopwords (kept ASD markers: autism/asd/autistic/spectrum). • Entropy: Shannon H over group token frequencies (higher means broader vocabulary used). • Overlap & Leakage: jaccard of group token sets and leakage = share of titles with ASD markers; two-proportion z-test between groups; baseline = overall marker rate. • Semantic Separation: TF–IDF vectors (idf=log((N+1)/(df+1))+1), group centroids, cosine distance; significance via 1,000-label-shuffle permutation test. • Directional Drift: KL divergence D(PǁQ) on smoothed token distributions (α=1e-6); row→column measures how well the column explains the row. Overall results summary (we can use this in our manuscript): Using only the titles from the two NeuroQuery outputs (found in Neuroquery_asd_DOI.docx file), we tested for cross condition contamination and corpus stability using the “Children with ASD” and “Children without ASD” corpora. The queries results were effectively the same (Figure 1. Shannon Entropy by Group): extremely high token overlap (Figure 2. Jaccard Overlap of Token Sets (2×2) ≈ 0.97), 100% ASD marker presence in both groups although they are different clinical conitions (Figure 3. Leakage of ASD Markers by Group), no semantic separation (cosine ≈ 0.005, p=1.0) (Figure 4. TF–IDF Centroid Cosine Distance), and only very small, asymmetric KL differences (Figure 5. Directional KL Divergence). To summarize, the “without ASD” query failed to isolate a non-ASD related cohort; it returned a corpus that is almost indistinguishable from the ASD corpus yet it produced different fMRI scans for each condition. This is a concrete example of cross condition contamination and query instability in NeuroQuery under these settings. Date: October 22, 2025 Figure 1. Shannon Entropy by Group (bits). ! Figure 2. Jaccard Overlap of Token Sets (2×2). ! Date: October 22, 2025 Figure 3. Leakage of ASD Markers by Group; dashed line = overall baseline. ! Figure 4. TF–IDF Centroid Cosine Distance with permutation p-value. ! Figure 5. Directional KL Divergence D(PǁQ). Date: October 22, 2025 ! Summary of report (title-only analysis): ! o Input file: Neuroquery_asd_DOI.docx (using titles only & no abstracts). o Groups detected: With ASD (n=30 titles), Without ASD (n=30 titles). o Entropy (bits): With=6.43, Without=6.48. o Jaccard overlap (token sets): With vs Without = 0.97. o ASD-marker leakage: With=100.0% (hits=30), Without=100.0% (hits=30); baseline=100.0%. o Leakage z-test (With vs Without): z=0.00, p=1.000. o TF–IDF centroid cosine distance: 0.01 (permutation p=1.000). o Directional KL: With→Without=0.08 & Without→With=0.22. !