Why drugs fail: the unrelenting challenge of finding a new drug

Jordan Lane, co-founder and CSO for Ignota Labs, breaks down the scientific and structural barriers that contribute to high drug failure rates and explores how artificial intelligence (AI) may help to address them.

Discovering a new drug is often cited as one of the most challenging feats of engineering ever conceived. The reason lies in the extraordinary complexity of the human body. It’s an intricate network of specialised organs, each with unique functions, filled with specialised cells that operate like miniature cities. Within each cell, there are networks and feedback loops, every role performed by a different protein – a molecular machine in its own right.

These proteins don’t merely perform static tasks; they engage in dynamic interactions governed by electrodynamics, fine-tuned actions, and communication channels that can only exist within the delicate balance of cellular environments. The level of precision required to interact with these processes – to alter just one protein’s behaviour without disrupting countless others - is almost unimaginably high.

Diseases, particularly complex ones like cancer, add another layer of difficulty. They wield one of nature’s most potent weapons: evolution. Over generations, cancers evolve, “testing” thousands of genetic mutations until the right set produces growth, survival, or resistance traits that give them an advantage. It’s a trial-and-error process, ruthlessly effective over time. Against this biological force, drug discovery is an attempt to interrupt or manipulate highly evolved systems – many of which have adapted specifically to evade our best efforts.

In early drug development altering a molecule’s solubility, for example, can affect its absorption, while increasing binding affinity may introduce unforeseen interactions elsewhere in the body. Every step in this complex chain requires meticulous tuning, as even the smallest change can cascade into unintended consequences, making the journey from lab to treatment an intense and uncertain process. In late drug development, the complexity of the patient's disease biology can throw up unforeseen, unforeseeable challenges.

The question posed: Why do drugs fail? My response: given the complexity, how has anything ever been approved?! Drug discovery is a game of assumptions and models of models, and it is hoped that everything will translate at a certain point. Drugs fail at all points in this journey when there is a mismatch in those things.

For small biotechs and early discovery teams, drug development is often derailed by a series of obstacles collectively known as the “valley of death”. This framework – around for decades – describes the formidable gap between laboratory discovery and clinical application where countless promising compounds fall short of making it to market, including 56% of which fail due to safety failures.

This “valley” captures both scientific and financial hurdles, with companies frequently caught in a cycle of limited funding and technical setbacks. To complicate matters, structural challenges such as misaligned incentives, investor pressures, and regulatory limitations further entrench these scientific bottlenecks. And once a molecule’s chemical structure is locked in, options for adapting its formulation, dose, or other characteristics are limited, making it even harder to address any emerging issues in the later stages of development.

Translating from discovery to clinical application

Predictive gaps in human response

One of the most significant scientific challenges in drug development is accurately predicting how a compound will behave in humans based on preclinical findings. For all drug types, translating lab or animal model data to human biology is imperfect. Toxicity profiles, off-target effects, and efficacy seen in animal models frequently fail to carry over into clinical settings, leading to high attrition rates.

When drugs do fail, it’s most often in preclinical animal testing or early Phase 1 human trials, where preliminary safety and tolerability are assessed. In many cases, clinical observations are limited to organ-level insights, which reveal broad toxicological responses without pinpointing the exact molecular pathways involved.

This lack of granularity leaves project teams with few tools to understand and resolve failures effectively, making it difficult to salvage or reformulate the compound. As a result, when preclinical studies uncover adverse effects, teams are often left with a partial picture that limits their ability to address the underlying issues.

lady looking at computer

Locked-in chemistry: Limited flexibility beyond early development

Once a compound’s chemical structure is finalised, the options for adjusting its behaviour are restricted. Beyond dose, formulation, or delivery adjustments, there is limited flexibility compared to modifying the compound itself.

Any potential safety or efficacy issues embedded within the original structure become “locked in” as the compound moves through development, making later-stage adaptations difficult, if not impossible. For small molecules, this rigidity creates a development environment where minor issues identified early on can become significant hurdles later.

Regulatory limitations and funding misalignment

Science-first gaps in safety validation

Despite rigorous safety standards, regulatory requirements in early development stages address only a fraction of a compound’s potential safety risks. The FDA, EMA, and other agencies focus on specific endpoints, but this leaves many possible risks unidentified. While these frameworks establish essential safety guidelines, they don’t cover all potential biological complexities or interactions, leading to cases where compounds advance through early trials without fully validated safety profiles.

The regulatory structure inadvertently reinforces an efficacy-first mentality. As drug developers focus on meeting regulatory milestones to push compounds forward, broader safety validation is often deprioritised. Consequently, some safety concerns that might emerge with more comprehensive testing are missed in early trials, creating an environment where efficacy typically outweighs safety.

female lab researcher

Misaligned incentives between discovery and development

Beyond regulatory constraints, funding gaps compound the scientific and safety issues in small molecule development. Early discovery and preclinical work are typically funded by academic institutions, spin-outs, or small biotechs, working with constrained budgets. Advancing a drug through clinical trials, however, requires a significant increase in capital – resources that are often only available from Big Pharma or substantial venture capital investment.

This handoff from initial discovery teams to large funders creates a structural misalignment. The researchers responsible for developing the drug often lack the resources to support its clinical testing. Venture capitalists and Big Pharma, stepping in at later stages, tend to prioritise efficacy signals over comprehensive safety data, seeking faster returns.

For these investors, demonstrating efficacy is often sufficient to justify additional funding or acquisition, with early safety assessments seen as secondary. Consequently, efficacy-first metrics are favoured, cementing early-stage scientific issues into the development process, and making them harder to address as trials progress.

The role of AI in reframing drug discovery

AI’s “start-from-the-start” approach: Building a stronger foundation

AI has emerged as a powerful tool in drug discovery, tackling entrenched challenges like target identification, compound interaction modelling, and screening efficiency. The “start-from-the-start” approach offers a new foundation for drug development, focusing on early refinements to improve later outcomes.

However, this approach has limitations: the ultimate proof of success – clinical validation – is years, if not decades, away. While there have been companies with AI to aid target validation or speed up the hit-to-lead section of drug discovery, they often operate within a vacuum, disconnected from the complex realities of human biology that surface only in clinical trials.

tablet showing data

Forward-looking considerations

As the pharmaceutical industry continues to face high failure rates in small molecule development, a fundamental shift in approach may be necessary. Addressing the valley of death, realigning funding priorities, and leveraging the capabilities of next-generation AI are promising strategies. However, progress will require collaboration across stakeholders – researchers, investors, and regulators alike.

Funding sources, particularly venture capital, may benefit from rethinking their priorities to incentivise early safety assessment alongside efficacy. Regulators, too, might consider expanding frameworks to encourage more comprehensive early testing. While AI brings new opportunities, it’s essential to approach it with a balanced outlook, recognising that true progress will come from both technological advancements and a cultural shift towards patient-first, holistic drug development.

Dr Jordan Lane

About the author

Dr Jordan Lane is co-founder and CSO for Ignota Labs. He has extensive experience in the AI drug discovery space and is driven by a profound commitment to revolutionising drug discovery; integrating artificial intelligence with bioscience to enhance drug safety and efficacy.

At Ignota Labs, Dr Lane leads innovation, leveraging his extensive background in biochemistry, genetics, and AI to pioneer transformative treatments in global patient care. His journey began at the University of Nottingham, where he earned a BSc in Biochemistry and Genetics, followed directly by a PhD focused on Alzheimer's Disease.

Sign up

Supercharge your pharma insights: Sign up to pharmaphorum's newsletter for daily updates, weekly roundups, and in-depth analysis across all industry sectors.

Click on either of the images below for more articles from this edition of Deep Dive: AI 2024