Measuring outcomes in CME
Rebecca Aris interviews Suzanne Murray and Thomas Kellner
AXDEV Group International and AXDEV Europe
Suzanne Murray and Thomas Kellner share their thoughts on the current CME landscape in Europe.
As our medical communications month continues we speak with Suzanne Murray, founder of AXDEV Group International, and Thomas Kellner, Managing Partner at AXDEV Europe. Both are experts in the field of optimizing the performance of professionals, teams, and organizations, and evaluating the impact of initiatives.
Suzanne and Thomas share their thoughts here on the current CME landscape in Europe, and what they think needs to change to make medical education more effective.
Interview summary
RA: How would you describe the current landscape in CME in Europe?
SM &, TK: The current Continuing Medical Education (CME) landscape in Europe is a fast evolving one. Multiple factors account for this rapid evolution.
First, CME in Europe is tightly linked to healthcare systems that are fast evolving themselves due to various trends, such as the emergence of – or emphasis placed on – inter-professional collaboration, team-based delivery of care, patient-centric approaches, and telemedicine to name a few.
In parallel, the medical field has to adapt to the increasing multiculturalism in society, to an aging population, and in many markets, to the increasing costs of care in a period of austerity and economical difficulties. All of these issues impact CME by creating new educational needs.
Secondly, European CME is continuously evolving, and is integrating and adapting learnings from American models on many aspects, such as the integration of technology, adaptation of competency-based models and transition from traditional CME to Continuing Professional Development (CPD) and Performance Improvement (PI). This creates a pressure to evolve faster, trying to meet and even surpass the standards set by other regions.
Thirdly, there is increasing pressure for CME to become evidence-based, addressing needs that have been demonstrated through rigorous methods, and for its outcomes to be more diligently assessed. The goal is to be able to demonstrate what type of education works, and what does not. This creates additional pressure for change for European CME professionals, who need to be guided through this evolution.
RA: What needs to change in order to make medical education more effective?
SM &, TK: Medical education needs to evolve and become more pro-active. Currently, programs are designed in reaction to needs that are assessed, and with a focus on knowledge improvement. In order to optimize CME, it is fundamental to ensure that education, is rounded in evidence-based medicine and disease management, while taking into consideration the context and system in which healthcare professionals and teams function and apply learnings in their clinical decision-making. From a business standpoint, this would allow pharmaceutical companies to gain a competitive edge, and would further validate their willingness to partner with governments and other healthcare stakeholders in the goal of improving clinical performance and efficiencies. Although this ideal CME might be a long way out, there are many smaller and immediate changes that could happen to make CME more effective. For instance, more time and energy should be invested in the use of technology in CME, specifically for team-based learning. However, it should be cautioned that technology can impact only certain components of learning, as documented in educational research and educational psychology. In addition, changes must come from healthcare professionals themselves, the literature highlights that physicians are still resistant to attend competency-based education, or to involve themselves in Performance Improvement initiatives. We can assume, and based on some literature, that this may be due to a shift in actual culture in the discipline of medicine, and to new ways of approaching and implementing accountability in the field of medicine. Moreover, all stakeholders involved in the field of CME, such as academic providers, speciality societies and medical associations, colleges, licensing bodies, and pharmaceutical companies, should be knowledgeable about adult learning principles and educational psychology to ensure that return on educational investment is being achieved. We are referring here to investment in financial resources, as well as investments in time and in human resources, in the deployment of educational programming and curriculum.
 ,
"...CME in Europe is tightly linked to healthcare systems that are fast evolving themselves..."
 ,
RA: Do you think guidance is needed around the area of outcomes evaluation in CME and if so why?
SM &, TK: In general, most CME outcomes evaluations are limited to assessing satisfaction, and are not looking at how education impacts patient health outcomes. Satisfaction, as a measure, is important from a business perspective, as pharmaceutical companies want their customers to select their educational programs over their competitors’. However, given that the primary objective of education should be for the learner to learn, and more importantly to apply learnings to practice, outcomes evaluation should focus on measuring the impact of education on clinical reasoning as well as behaviours in clinical settings. By doing so, outcomes evaluation serves to validate the relevance and impact of education that the healthcare professional attended. Guidance is needed to ensure that outcomes evaluation is appropriate in relation to an initiative’s learning objectives, and to ensure it reaches a sufficient level to be able to identify what works and what does not, what impacts practice and what does not, and most importantly, the causality behind such outcomes.
RA: What in your opinion is the best way to measure outcomes in CME?
SM &, TK: We do not think there is one best way to measure outcomes in CME. We believe that evaluations need to be closely adapted to programs. Therefore, there are possibly as many best ways as there are programs. We see outcomes evaluation as a three-dimensional matrix (see figure). On one side, you have the breadth: how large of an assessment do you need (satisfaction, knowledge, competencies, all the way up to performance and patient outcomes). On the other side, you have the learning objectives, which summarize the outcomes that the program is trying to achieve. The third dimension represents the depth of your evaluation, a component which is most often neglected. The depth of evaluation responds to questions such as: Are you satisfied with simply assessing the “what” (i.e., did the learner acquire new knowledge, was the newly acquired knowledge transferred into practice), or do you want to know the “why” (i.e., why the learner did or did not acquire new knowledge, why the newly acquired knowledge was or was not transferred into practice)? Do you need to assess “how” the program attempted to reach its objectives (i.e., respectful of adult learning principles and CME best practices)? So to answer your question, the best way to measure outcomes in CME is to go as broad and as in-depth as you need to, and as realistically as you can, for each learning objective. To decide which outcome measures are most appropriate, one needs to consider which levels an educational activity will affect through its various learning objectives, and how relevant it is to understand the “why” and the “how” for each learning objective. When designing outcomes evaluation, it is important to ensure these questions are answered prior to designing evaluations and collecting data.
 ,
"Medical education needs to evolve and become more pro-active"
 ,
Figure 1: * Moore DE, Green JS and Gallis HA (2009) Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions 29(1):1-15. © 2012 AXDEV Group Inc.
 ,
RA: How best can we measure satisfaction, knowledge uptake and intent to maintain or change behaviour in CME participants?
SM &, TK: We do not think there is a best way to measure these variables. However, it is safe to say that there is a more appropriate way for each specific context. As mentioned previously, evaluation has to be linked to the educational program. Therefore, outcomes evaluations should be as different as programs are.
Generally, when trying to achieve depth in outcomes evaluation, quantitative evaluations generally reach their limit quite rapidly. This is why we favour mixed-method protocols, where we can combine the analytic power of quantitative and the in-depth richness of qualitative evaluations. It allows us to go beyond the “what”, and identify the causalities, or in other words “why” specific components of a program worked or did not work.
RA: How important is it to define learning objectives in CME activities?
SM &, TK: We should start off by saying it is very important to define learning objectives in CME activities, given it is the basis of adult education theory. Clear learning objectives allow the learner to identify what should be achieved by selecting the program. More importantly, learning objectives are utilized as a guide to develop program evaluation metrics.
In addition, learning objectives need to be tied to needs that are properly identified, and need to be S.M.A.R.T. (specific, measurable, attainable, relevant and time-bound). These characteristics are essential to make both the program content and the outcomes evaluation as targeted and precise as possible, making optimal use of program attendees’ time, and of the grantor’s investment.
RA: What do you think CME will look like in ten years time?
SM &, TK: We are confident that CME will have taken many steps towards the ideal CME that we partially described earlier. In other words, we know that CME will continue to evolve towards Continuing Professional Development and Performance Improvement, and to be recognized as a critical component to improving clinical performance in a worldwide evolving healthcare system. Technology will certainly be better integrated, in ways that would maximize return on educational investment for funders. CME will also be more personalized, addressing the need of the individual physician, as well as their teams, rather than the needs of the majority. In that regard, Performance Improvement initiatives will definitely play an increased role, as we are already observing the beginning of this evolution in Europe and in North America, as well as in emerging markets, such as in Asia.
Medical education will look at the art of medicine more globally, where competencies such as communication and collaboration are an integral part of one’s ability to provide care to patients.
RA: Thank you for your time.
About the interviewees:
Suzanne Murray:
During Suzanne’s early career years in senior health and research management at the McGill Centre for Studies in Aging and the Montreal General Hospital, she observed that an important part of the challenges faced in healthcare were at the organisational and system levels. So, in 1997, she founded AXDEV Group International, a Performance Improvement organization, trusting that she could make an impact.
As CEO and co-Founder of AXDEV, she is dedicated to offer solutions, credible research and evidence to decision-makers and leaders in the healthcare environment. Her role is to explore, collaborate and support international initiatives that aim at improving systems, as well as clinical team and professional functioning in healthcare delivery settings.
Thomas Kellner:
Thomas holds a medical degree from the University of Munich.
He started his career as an e-strategy consultant in a new media company, where he rose to leader of the healthcare division.
He joined MSD where he developed multi-channel marketing, and managed an education platform for healthcare professionals. Thomas held various international positions over the 10 years he spent at Merck, including leader of the global medical education strategy.
Thomas recently founded AXDEV Europe as Managing Partner, a subsidiary of AXDEV Group International.
What do you think is the best way to measure outcomes in CME?
 ,
