Continuing Medical Education (CME) has established a rough spectrum of potential outcomes that ranges from merely taking up the doctors' time to saving the world (only a slight hyperbole). The definitive characterization of the spectrum is in Moore's 7 Levels of CME Outcomes Measurements (reference below), and pharmacy education (ACPE) appears to be adopting the same evaluation techniques (reference below).
Part of The Challenge
One part of the challenge is that it is difficult enough to establish level 3, which indicates that the participant actually learned the material. With both live and distance education (i.e., any non-proctored scenario), there is ample opportunity for someone to look up answers for knowledge-based examinations. Case-based (i.e., application-based) exams reduce the cheat factor and imply some degree of level 4 (competence). However, the increase in difficulty generally results in a decrease in satisfaction since the participants have a harder time.
The more difficult part is getting participants to help with meaningful data-gathering for higher levels when only level 1 (participation) is necessary to receive credit for many (especially live) courses. Grant providers want you to prove that participants learn the materials, apply the knowledge, and master it in such a way that the general community benefits. And you have to prove it with participants who get no measurable benefit from doing more than just keeping a seat warm -- other than being better at their jobs, of course.
When dealing with specialists, the challenge is broader because you can only address cases that come in front of you. So as you deal with special cases during an educational session, it might only affect 1-2 cases per participant. That adds up to a major impact for a large audience, but it is difficult to compute the impact on an individual participant when they might go months without seeing a relevant case. To see a dramatic impact on a complete population or to see a major impact on a practitioner's effectiveness, you almost have to focus on general interest items rather than specialty knowledge.
Finally, suppose you actually have the funding and willing participants necessary to gather improvement data. Even then, it is difficult to establish a link back to a specific educational product. Any medical professional would have received numerous points of contact within months of your educational activity. They read. They interact with colleagues. They attend other activities. While the research may be sufficient for grant applications, the inability to isolate the key variable places a very large question mark over any data collected.
There is no perfect solution to this quandry. Satisfying all levels would be very expensive to implement and to measure. In the end, you just have to play the game, ask questions that resemble one of the target levels, and attempt to do follow-up to get long-term data. It is not perfect. And it is unavoidable.
- Moore's 7 Levels of CME Outcomes Measurements | Trusted.MD Network
- Outcomes Levels, Data Source, and Process | The Institute for Continuing Health Care Education
- CME Outcomes and Measures
- National Institute for Quality Improvement and Education (NIQIE)
- NIQIE Presentation: Performance-Based Learning in Practice
- PRIME® Science of CME: Needs Assessment, Activity Development, and Outcomes Assessment
- ACPE: Integrating Outcomes Assessment