How AI is advancing patient subgroup identification in oncology
Over the last two decades, oncology drug development has been through a transition. The assumption has been that if patients shared a diagnosis, such as lung or breast cancer, then they would require the same treatment. Increasingly, oncology drugs are now designed based on individual drivers of disease at the molecular level. This has resulted in more therapies built around a cancer’s specific biological signature, tested in patients selected for that signature, rather than for a diagnosis alone.
I’ve watched this transition play out firsthand in my career, which spans working on the first generation of CAR-T therapies for acute leukaemia, to helping bring advanced therapies from concept to commercialisation on the biotechnology side. It is now common practice to pair diagnostics with targeted oncology treatments in order to identify patient subgroups, a practice that has allowed us to create increasingly personalised oncology drugs. Even so, patients who share a diagnosis and a molecular target can still respond very differently to the same therapy.
Now that identifying patient subgroups is central to oncology drug development, it also plays a larger role in determining the success of clinical trials. Across pharma, mitigating the high risk of late-stage clinical trials has become an industry-wide priority. About 70% of drugs that complete Phase 2 fail to advance to Phase 3 and, among the small number that do, more than half still fail to win regulatory approval.
With the advent of AI, it’s now possible to enrich clinical trials for patient subgroups more precisely than ever before. Regulators are establishing frameworks for these approaches: FDA guidance addresses predictive enrichment in clinical trials, while more recent draft guidance outlines a risk-based framework for assessing the credibility of AI models used to support regulatory decision-making. The challenges that remain are determining how AI can best serve patient subgroup identification and figuring out how to build standard industry workflows around it.
Making the most of clinical trial data
Early research shows that AI has the potential to surface insights from clinical trial data that can help identify patient subgroups. One study built a machine learning model to stratify acute respiratory distress syndrome patients and found it could define three distinct subpopulations by outcome, rapid death, and spontaneous recovery. The researchers showed that this type of stratification could improve statistical power and potentially reduce the number of patients required. In another study, researchers were able to use a machine learning model to find a biomarker that predicted which cancer patients would respond to immunotherapy, based on data from completed trials. When the model was tested on an independent dataset, it correctly identified responders and non-responders.
AI can also enrich trial recruitment. A study of patient health records and machine learning used the patterns it found to redesign trial eligibility rules, aiming to enrol a population that better reflects who actually has the disease, instead of defaulting to the same restrictive criteria used in earlier trials.
As oncology datasets become increasingly multimodal, the next generation of patient stratification may depend on more than identifying a single predictive biomarker. AI creates an opportunity to evaluate combinations of molecular, clinical, imaging, and treatment characteristics that may collectively define clinically meaningful differences in response. Because this approach considers many characteristics at once, confirming that a subgroup holds up in independent data becomes an important step before it shapes the design of a future trial.
Why subgroup discovery requires purpose-built AI
Clinical trial subgroup discovery presents a specific analytical challenge: datasets may be comparatively small, highly heterogeneous, and contain many more candidate relationships than can be tested reliably. Research has shown that both conventional and AI-enabled approaches can generate unstable findings when the analytical design does not adequately control overfitting or separate discovery from validation. For example, one study found that when patient data was given to nine different LLMs in a spreadsheet form, even the strongest model tested struggled to hit acceptable accuracy on structured data queries, and performance got worse as the dataset or the question got more complex.
Even statistical models face challenges in analysing trial data. Standard regression techniques often overfit results based on patients in the current pool of data, a risk that increases when the number of variables considerably exceeds the number of patients. A study of both foundation models and LLMs found that the models couldn’t recover meaningful subgroup information from three real, completed clinical trial datasets. Meanwhile, a methodology designed specifically for heterogeneous clinical trial data was successful at identifying subgroup variables, subsequently improving the performance of eight statistical models across all three datasets. Key to its success was the ability to focus analysis on patients for whom the data supported a meaningful subgroup assignment, improving downstream model performance.
Using AI to better inform the next phase
While many of the published examples of what AI can pull from clinical trial data are retrospective, the greater opportunity is to apply these insights prospectively, using what is learned in Phase 2 to make better decisions before Phase 3 begins. For an AI-derived patient profile to become useful in drug development, sponsors need to understand which variables define it, determine whether the pattern is reproducible in independent data, and assess whether it can be prospectively incorporated into a future study. A credible analysis must also be capable of concluding that the available data does not support a stable subgroup.
Once a Phase 3 trial is already underway, it’s often too late to alter any of the variables that could meaningfully strengthen the study signal. But insights pulled from an early-stage trial have the potential to shape everything from recruitment strategies to trial design, better identifying non-responders, placebo responders, and patients with or without key biomarkers. Identifying more clearly defined responder populations prepares sponsors for the reimbursement stage of taking successful drugs to market, where they must convince payers, not just regulators, that a drug is effective.
While many clinical trial AI developers are focused on using the technology for drug discovery, the more urgent problem in oncology and across pharma is reducing the risks of late-stage trial failure. The next advancement will come from applying the right methods at the right stage of development and ensuring that the identified subgroups are interpretable, independently reproducible, biologically coherent, operationally measurable, and capable of prospective evaluation. Used this way, AI can help sponsors make better-informed decisions before committing patients, time, and capital to Phase 3, design more informative trials, and connect the right patients with the therapies most likely to benefit them.
About the author
Dr Panteli Theocharous, PhD, FRCPath, is a globally recognised clinical-development executive, translational scientist, and biotechnology entrepreneur with more than 35 years of experience across the pharmaceutical, biotechnology, and CRO sectors. He serves as chief therapeutics & clinical strategy officer and president of Europe at Worldwide Clinical Trials, and as fractional chief medical officer at NetraMark. His career spans oncology, haematology, rare disease, precision medicine, and cell and gene therapy, from pioneering work in advanced therapeutics to global clinical and commercial strategy. He is recognised for translating complex science, emerging technologies, and patient-level insights into rigorous development strategies that can accelerate the delivery of transformative therapies.
