Why it fails
Dataset identity is the loudest thing in the signal
Turn the same features to a different question — which dataset is this? — and a classifier answers far more accurately than it predicts disease. Site and acquisition dominate the feature space, which is exactly why the biomarker does not travel.
ROC-AUC · dataset identity vs. disease
Predict which dataset (logistic regression)
0.95
Predict which dataset (random forest)
0.89
Predict disease within a cohort
0.72
Predict disease across cohorts
0.67
This is a result about dataset shift, not brain biology — the kind of negative result that keeps a field honest.
Which features the model leans on
Top standardized coefficients on the development cohort. Cross-dataset agreement of feature importance is Pearson r = +0.04 — near zero: the model relies on different features in each cohort.
| feature | mean coef | sign consistency |
|---|---|---|
| log_power_beta__occipital__iqr | -0.691 | 1.00 |
| abs_power_theta__central__iqr | -0.610 | 1.00 |
| rel_power_beta__occipital__iqr | +0.601 | 1.00 |
| log_power_theta__frontal__iqr | +0.597 | 1.00 |
| rel_power_theta__occipital__iqr | +0.588 | 1.00 |
| paf__parietal__iqr | -0.531 | 1.00 |
| hjorth_complexity__occipital__median | +0.506 | 1.00 |
| paf__occipital__iqr | -0.505 | 1.00 |
| abs_power_beta__temporal__iqr | +0.446 | 1.00 |
| log_power_alpha__frontal__iqr | +0.432 | 1.00 |