International benchmarking and NICE

The National Institute for Health and Care Excellence (NICE) is arguably the most well known Health Technology Assessment (HTA) agency in the world and maybe even – overall – the best in its field. But can NICE still learn from others?

International perspectives on NICE

Following their invited review of the Health Technology Appraisal (HTA) programme at NICE, the World Health Organisation (WHO) said in 2003 that: “NICE has developed a well-deserved reputation for innovation and methodological developments that represent an important model for technology appraisals internationally”.

There’s no doubt that other countries – and the myriad stakeholder groups from local patient groups to industry to Governments within them – are still very interested in what NICE does and says.

Closer to home, the first Triennial Review of NICE – one of the ways that the Government provides robust challenge to arms length agencies – suggests that NICE should look at the work of international HTA agencies. The review recommended that NICE investigate the possibility of benchmarking functions with international comparators.

This recommendation was made in the context of exploring opportunities for efficiencies – but it also reflected concerns from some international stakeholders about NICE’s willingness to engage formally in international committees. One stakeholder had also suggested that NICE appeared unwilling to adapt or consider new methodologies.

Can NICE be compared to any international peers?

NICE was required to report one year on how it has approached the recommendations from the Triennial Review. The report, available in NICE Board papers for July 2016, says that “there are no direct international comparators for the totality of NICE’s work.” It adds that there are “no obvious benchmarking metrics that could be applied to either the sum or the outputs from individual programmes”.

Their solution? To undertake a review of the recent literature and to report to the Board in November 2016.

The real question: what to benchmark against?

Ignoring the full range of what NICE does (an impressive list of activities) and focusing just on HTA creates a challenge: there isn’t  a consensus on what is good HTA. Maybe there will never be in the real world of vested interests, messy politics and genuine and well-intentioned debate about methods and processes for HTA.

Some have tried to help shape the quality of HTA. For example, Michael Drummond and colleagues set out key principles for the improved conduct of health technology assessments for resource allocation decisions in 2008. The fifteen principles cover the structure of HTA programmes, methods, processes for conducting HTA and use of HTA in decision making. These have been used to test out agreement and application by agencies as well as the state of HTA research methods. Industry too has some principles for HTA, with some overlap with those of Drummond and colleagues.

This means that there is somewhere to start from in terms of what NICE could be benchmarked against, at least in terms of HTA. There is work too that delves more deeply into some of the principles Drummond and colleagues set out. For example, Drummond and colleagues recommend in principle 10: those conducting HTAs should actively engage all key stakeholder groups. HTAi’s Patient and Citizen Involvement sub-group have set out value and quality standards for patient involvement in HTA which build on the spirit of principle 10. The Scottish medicines Consortium (SMC) has said it is committed to achieving these. Presumably to know whether it is or not, some further work will develop a way to benchmark against those value and quality standards.

The next question: how to practically benchmark NICE?

Next is just how NICE could practically do benchmarking. There’s help on this too. Building on the principles for improved conduct of HTA the question of benchmarking HTA has also been considered including practical audit questions.

It’s not just an academic pursuit either: consultancies like CRA International have also drawn on principles to compare HTA agencies and see whether they’ve improved over time. Metrics used by CRA include the length of time for the review by the HTA agency and the time between regulatory approval to HTA recommendation where you can see which agency is the fastest – or slowest – leaving aside how they reached their recommendation or what the recommendation is itself.

CRA analysis suggests NICE compares highly overall on their adapted principles for HTA versus other HTA agencies, but also that they have worsened over time too. For example, NICE did not score as highly in 2013 for identifying areas in which the evidence on an intervention could be most usefully developed in the future, compared to 2011. CIRS also have HTA performance metrics.

NICE has some of the data that would be needed for benchmarking. For example, if speed was to be benchmarked with other agencies, NICE includes speed of production in their balanced scorecard so must collate the data. NICE being slower than another HTA agency doesn’t give a clear steer to action, but it prompts reflection of why that is, and what the trade-offs might be to enable NICE to produce their recommendations more quickly.

NICE desire to benchmark where possible

Elsewhere in the same Board papers, NICE say that they are aiming to “promote a culture of continuous improvement within the organisation and uphold the ambition to remain a world-renowned organisation, benchmarking where possible its systems and processes and outcomes against best players internationally”.

This is taking some time though: NICE had aimed to identify what international benchmarking is possible by September 2016. They’ll also identify ten publications that can go to the Board that “assess and provide an opinion on one or more aspects of NICE’s work”. A report to the Board should become available by November 2016.

It’s not clear how NICE can avoid international benchmarking, yet at the same time, benchmark themselves internationally. Plus shouldn’t NICE set itself the same tests for comprehensiveness that it sets for others: why not a systematic review? That would avoid the challenge of cherry picking – of the many many articles out there – comparing NICE with other HTA agencies.

A need to look outward

NICE needs to ensure that it looks outward – as well as inward – at a time when it needs to be more efficient as its funding shrinks, and to ensure that it serves patients and citizens of the UK as best as it can. NICE may be a world leading HTA agency, but shouldn’t also be a world learning HTA agency too?