Evaluation of the MMSF Collaborative
Executive summary
The Measurement and Monitoring of Safety Framework (MMSF) consists of five dimensions, and a series of prompting key questions, that guide users to comprehensively and conceptually view safety. These five dimensions and related questions address: past harm, reliability, sensitivity to operations, anticipation and preparedness, and integration and learning.
In October 2018, the Canadian Patient Safety Institute (CPSI), now Healthcare Excellence Canada, launched a patient safety improvement project under the leadership of Maryanne D'Arpino of CPSI (Executive Lead) and Dr. G. Ross Baker at the University of Toronto (Academic Lead). This program, a learning collaborative with expert faculty and mentorship, aimed to enable the implementation of the MMSF amongst 11 teams from seven provinces across Canada over an 18-month period with the aim of each team developing a more comprehensive approach to safety and the delivery of safer care. This report presents findings from an evaluation study funded by CPSI that aimed to examine the effectiveness of this Collaborative.
Download the evaluation report
Methods
This study used a qualitative approach based on interview, observation and documentary data collection methods. In-depth semi-structured interviews were conducted with team members. One-day site visit observations were conducted at five sites; observations of learning sessions were conducted and relevant documents were collected. Thirty-six team members participated in the interviews. A total of 29 hours was spent at site visits; in addition to approximately 33 hours in learning session 3, the closing congress and all-team virtual meetings.
Key findings
- Overall participants provided positive feedback about the in-person learning sessions. They particularly valued the expert presenters, multifaceted approaches used to teach the MMSF, and the structure created for learning within and between participating teams.
- While some participants felt 'overwhelmed' at the amount of information in the first learning session, the majority felt positive about the framework from the outset. The first learning session set in place the need for a shift in thinking about safety from an absence of harm to presence of safety, to thinking about changing culture, and that it would take time to understand and implement the MMSF.
- The coaching by CPSI senior program managers played key roles in participants' understanding and implementation of the MMSF. The coaches were responsive and accessible between site visits. They provided ongoing education and support; and were instrumental in providing the feedback necessary for ongoing implementation of the MMSF. Some participants would have wanted more coaching and more clarity about coaching and team accountability expectations.
- Team members used a range of teaching strategies and methods to support the implementation of MMSF into practice. These included teaching about the framework to groups of stakeholders (e.g. healthcare providers and senior leadership).
- Quality improvement (QI) consultants, physicians and boards began teaching about the framework by integrating its language into day- to- day communication and using it to discuss specific safety or patient care issues. Team members made decisions about how to teach the framework to stakeholder groups, taking into consideration issues such as availability, number of people involved, professional roles and interest. There were different perceptions about the effectiveness of teaching the MMSF and whether it’s necessary to teach the framework itself or if it’s sufficient to teach and implement tools and processes informed by the framework.
- Teams were encouraged to focus on MMSF implementation strategies that were context specific and allowed for the integration of the framework into the daily clinical and administrative work of the units or targeted areas. Teams consequently used a variety of strategies. These included the use of the MMSF to inform the following processes and activities: safety huddles, health care processes, safety incidents and reports; meetings; communication; patient and family focused initiatives; and board and senior leadership level activities. Each strategy had success in targeting different stakeholders and effecting change in different ways.
- The MMSF teams consisted of individuals with varied professional backgrounds and roles at local, regional and provincial levels. This variability allowed for sharing of diverse perspectives and multiple avenues to teach, implement and spread the MMSF. However, variability in engagement with the collaborative and movement out and into the teams over the 18 months were challenges. A small number of teams had patient, family and board representation who were seen to bring valuable perspectives to the team and its work. Physicians were a more difficult group to engage.
- The majority of participants were supportive of wider spread of the MMSF yet there was variability in their opportunities for spread beyond their implementation site(s). While a small number remained focused at the original site of implementation, the other teams demonstrated varying levels of spread: unplanned spread; planned individual or team efforts which led to pockets of uptake in the organization or region; planned and coordinated widespread efforts to spread the MMSF across an organization and region. Challenges to spread included limited dedicated resources, uncertain authority to influence spread, the need for alignment with wider-level processes and frameworks and healthcare organizational and regional restructuring.
- The majority of participants reported positive impacts from MMSF implementation. These included changes in thinking about safety which impacted on behaviours and practices; healthcare staff engagement in prevention, identification and management of safety issues; patient, resident and family engagement in safety; and improvements in healthcare processes
Conclusions and implications
The MMSF collaborative was successful in teaching the teams about the MMSF and coaching them to implement the framework in their local settings. Participants perceived the MMSF work as having positive impacts on stakeholder groups' knowledge and behaviours and on healthcare processes and patient outcomes. These findings support further education and implementation of the MMSF; however, these efforts would need to address the facilitators and challenges identified in this report to ensure a more systematic and comprehensive spread throughout healthcare organizations and regions.
