No Two Systems Are the Same: Paramedic Perceptions of Contemporary System Performance Using Prehospital Quality Indicators

Introduction In recent years, researchers have identified two new models of paramedicine within the Anglo-American paramedic system known as the Directive and Professionally Autonomous paramedic systems. The research team now seek to compare paramedic perception of system performance between the two systems using prehospital quality indicators. Methods Paramedics employed within Anglo-American paramedic systems undertook a survey evaluating their experience and perception of system performance against a set of modified prehospital quality indicators. Data were collected using a survey combining single-choice questions with matrix multiple-choice questions. Key results were cross-tabulated with demographic (informant) and system factors to compare performance between the two new paramedic systems. Results The survey indicated a substantial difference in perceived clinical and operational performance between the Professionally Autonomous and Directive paramedic systems, with the Professionally Autonomous paramedic system performing consistently better in all 11 prehospital quality indicator domains. Conclusion The results of this survey are a vital step in helping paramedics, health leaders, and academics understand the complex relationship between paramedic system design and system performance, and, for the first time, provides empirical evidence upon which to make a conscious decision to adopt one system or the other.


Introduction
As the paramedic profession matures and evolves, the journey toward professionalization has taken many different paths across the world. These paths have varied depending on each country's, and even local regions, socio-political environment. What is clear is that no two paramedic systems are the same. Yet, until recently, the age-old method of classifying paramedic systems and understanding their structure, function and performance, had not changed in 70 years [1].
In the past two decades, notable work by researchers such as Dick and O'Meara et. al. has sought to explore the divergence in structure and characteristics between paramedic systems. Their work has found that there is an immense difference between the way one system is structured and operates compared to another and that this is influenced by a myriad of complex micro and macro factors, including system leadership, professionalization status, and higher education to name a few [2][3][4][5].
A 2020 scoping review, published by this research team, identified two new systems of practice within the Anglo-American paramedic system. They were named the Directive and Professionally Autonomous paramedic systems [6]. Identifying what structures, characteristics, processes, and cultural factors contribute to system performance was the first part of our research series, work that was grounded by a conceptual framework [7]. We subsequently sought to define the two systems through expert consensus in a Delphi study [8]. This provided us with an understanding of the individual characteristics of each system, and importantly, validated the findings to ensure that there was confluence in the basis of our future research.
Identifying new systems of practice is important. However, understanding how each system performs when compared to prehospital quality indicators is equally important for establishing an argument for change toward a system that is better suited to the complex and changing health needs of its community.

Objective
The aim of this study was to compare paramedic perception of the Professionally Autonomous and Directive paramedic system performance using prehospital quality indicators.

Study design
A cross-sectional questionnaire was developed to measure and compare performance between the Professionally Autonomous and Directive paramedic systems.

Prehospital quality indicator construction
To assist with selecting the prehospital quality indicators for the survey, a set of criteria for inclusion and exclusion ( Table 1) was created. These criteria helped to ensure the indicators used were supported by evidence, easily measurable, reflective of important aspects of care in paramedicine and finally were able to inform real change. The authors then reviewed contemporary research, including works by the Pre-Hospital Outcomes for Evidence-Based Evaluation [9][10][11][12] to help guide the development of the prehospital quality indicators list, with the final list ( Table 2) being heavily influenced by a recent scoping review conducted by Pap et al. [13][14], which identified commonly utilized prehospital quality indicators across modern paramedic systems.

Inclusion Criteria Exclusion Criteria
(1) Prehospital quality indicator supported by scientific evidence or best practice.
(2) Reflective of important aspects of care in contemporary paramedic settings (e.g., safety, effectiveness, patient-centeredness). (3) Actionable and capable of driving quality improvement.
(1) Lack of scientific evidence or best practices to support the quality indicator. (2) Overlap with other quality indicators or measures already in use.
(2) Not feasible to collect or report on in a reliable or timely manner.  To make up for this shortfall in openly available data, each indicator was adapted from established checklists into questions that measured paramedic perception of system performance and administered as part of an online questionnaire to informant paramedics.

Survey design
Data were collected using a questionnaire combining single-choice questions with matrix multiple-choice questions. The survey tool was designed to collect information in support of the research question using a constructivist approach and was designed and deployed through the Qualtrics© platform (Qualtrics, Provo, UT, USA). Distribution was primarily through social media and was open for responses over a 30-day period.
The first section of the questionnaire established which system type the paramedic worked in based on previously established definitions [8]. The second section of the questionnaire measured the perceived performance of each system using respondent knowledge of their own paramedic service. To minimize reviewer judgment and confirmation bias [17], we transformed criteria from checklists into data points and questions.
Throughout the questionnaire, respondents were asked to determine if a skill or action was considered "common practice." Common practice was defined as something that is performed often and is considered normal practice.
A pilot was undertaken using experts to assess the functionality of the online survey platform as well as the clarity of questions. Additionally, the content validity index of the questions was tested using the expert panel to assess inter-rater agreement. An expert was defined as a (1) paramedic or physician in the field of paramedicine that held a senior leadership position within a medium to large Anglo-American paramedic system or (2) a senior academic who had studied and published on the topic of paramedic leadership, governance and/or paramedic system design. "Senior leadership position' was defined as Director level or above, whilst a 'medium size paramedic system' was defined as more than 250 full-time equivalent salaried employees [8].
A total of 11 participants consented to participate in the pilot study. Each expert panel member evaluated each item (questions) for content validity on a four-point scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant). The item level content validity index (I-CVI) (Appendices) was calculated as the number of experts giving a rating of either 3 or 4, divided by the number of experts [18,19]. Questions with an I-CVI rating below 0.80 were excluded from the final survey (appendix 1) [20]. Pilot data were not included in the analysis, although participants were invited to complete the final version of the survey. Following the review of the pilot data, nine questions were removed and two new questions related to patient registries were added owing to numerous requests from pilot participants.

Participants
Following the completion of a sample size calculation (Cochrans formula) [21], the target population was determined to be 375 respondents using a confidence level of 95%. The inclusion criteria for this survey were that participants were employed as a paramedic or an emergency medical technician working in an Anglo-American paramedic system in an English-speaking developed county. This included Australia, New Zealand, the United States, Canada, the United Kingdom, and the Republic of Ireland. Paramedics were excluded if they had a qualification less than basic life support or if they worked as a volunteer, as they were more likely to be considered a non-medically trained first responder such as a first aid-qualified driver.

Ethics
The study received human research ethics committee approval from the Monash University Human Research Ethics Committee and ascribed project number 32646. All participants provided written informed consent prior to commencing the online survey.

Demographics
There were 475 survey responses (Table 3), with 57.5% of respondents indicating that they worked in a Directive paramedic system, compared to 42.5% who worked in a Professionally Autonomous system. Most respondents were from the United States (35.6%), Canada (22.5%), Australia (18.1%) and the United Kingdom (18.1%). The survey identified a gender difference (Χ2=26.5, p<0.001) between systems, with 52.9% of Professionally Autonomous system participants being female as compared with only 30.7% in the Directive paramedic system group. The legal basis for practice varied amongst respondents however showed that those from a Directive paramedic system were more likely to be licensed (65.9%) as compared to respondents from the Professionally Autonomous paramedic system who were more likely to be registered (84.6%). Finally, level of practice also showed significant variance, with respondents from Directive paramedic systems indicating that they practised at an Emergency Medical Technician or Primary Care Paramedic scope (47.9%) compared to only 7.4% of paramedics from Professionally Autonomous paramedic systems whose most common level of practice was the Advanced Care Paramedic scope (39.6%).

Prehospital quality indicators
The questionnaire results section has been structured into prehospital quality indicator domains as viewed by respondents in the questionnaire ( Table 4).

Prehospital quality indicator 1: cardiac arrest (Q1-4)
Respondents in Professionally Autonomous services indicated that callers requesting assistance for a suspected cardiac arrest were offered instructions in chest-compression-only cardio-pulmonary resuscitation 97.5% of the time compared to 68.5% in Directive systems. Similarly, 98.5% of respondents in Professionally Autonomous services indicated it is common practice that a patient will be identified to be in cardiac arrest by the emergency medical call-taker before the first resource arrives on the scene compared to 69.6% of respondents in Directive systems. Significantly more (80.6%) respondents from a Professionally Autonomous paramedic system reported their organization contributed to a cardiac arrest registry, compared to 42.4% of respondents from a Directive paramedic system.

Prehospital quality indicator 2: acute coronary syndrome (Q5-12)
Almost all respondents (97.5%) from Professionally Autonomous services stated they had a documented clinical care pathway that details the care and transport provided to patients with signs and symptoms suggestive of acute coronary syndrome. Similarly, as common, 97.5% of Professionally Autonomous ambulances are equipped with a 12-lead ECG-capable monitor/defibrillator compared to only 56.7% of Directive service ambulances. With respect to medication administration, respondents indicated that patients presenting with signs and symptoms suggestive of acute coronary syndrome and normoxaemia (SpO2 >94%) were not administered supplementary oxygen in 97.5% of events within Professionally Autonomous systems, compared to 63.3% of respondents from Directive paramedic systems. Aspirin administration in acute coronary syndrome was reported by 99% of respondents from a Professionally Autonomous paramedic system and 98% receive analgesia as part of their treatment, compared to 70.7% and 54.6%, respectively, of Directive system clinicians.

Prehospital quality indicator 3: delivery of care in end of life (Q13)
Respondents from Professionally Autonomous paramedic systems indicated that they had higher access to clinical care pathways that detail the delivery of end-of-life (palliative) care (86.1%) compared to 32.9% of Directive paramedic service respondents.

Prehospital quality indicator 4: delivery of care in trauma (Q14-17)
As an initial measure of a service's research and development, 79.2% of respondents from Professionally Autonomous services stated their service participates in a trauma registry while 37.3% from Directive services stated the same. There was a significant difference between service models in the delivery of tranexamic acid, where a patient with recent (≤3 hours) traumatic injury resulting in ongoing hemorrhage or acute traumatic coagulopathy receives the medication in 89.1% of Professionally service responses compared to 38.7% of Directive service responses.

Prehospital quality indicator 5: meaningful change in pain score (Q18-19)
An overwhelming number of respondents (99.5%) from Professionally Autonomous systems stated it is common practice that an adult patient has their pain intensity measured using the 0-10 verbal numerical rating scale, compared to 66.6% of respondents from a Directive system. Moreover, whilst 97.5% of respondents from Professionally Autonomous services reported that their paramedic service administers analgesic agent(s) to patients reporting mild (2-3/10), moderate (4-6/10) or severe (7-10/10) pain unless contraindicated or refused, only 50.1% of respondents from the Directive services did the same.

Prehospital quality indicator 6: clinical resource and resource management (Q20)
Respondents in Professionally Autonomous systems reported higher rates of access, enabling them to consult with a senior paramedic clinician via telephone when treating a patient (89.1%) compared to 27.4% of respondents in the Directive group.

Prehospital quality indicator 7: proportion of calls closed with telephone advice or managed without transport to the emergency department(Q21)
Respondents in the Professionally Autonomous group shared that they operate a secondary triage service in 80.6% of cases compared to 27.4% of respondents in the Directive group. This represents an approximately 60% difference between both groups, which was among the most significant across all survey questions.

Prehospital quality indicator 8: proportion of decisions to leave a patient at the scene ('hear and treat' and 'see and treat') (Q22-25)
When asked whether it is common practice that a patient who meets service-defined treat-and-discharge or treat-and-refer criteria is not transported in their paramedic services, 90.5% of respondents from Professionally Autonomous services stated "yes" compared to 32.9% from Directive services. In alignment with the non-transport pathway availability, 87.6% of Professionally Autonomous service respondents had a policy/clinical practice guideline that describes the treat-and-refer arrangements for patients not conveyed to a health care facility, as compared to 29.6% of Directive service respondents. Finally, patients who are not conveyed to a healthcare facility receive a clinical follow-up within 48 hours in 57.4% of Professionally Autonomous system responses while this occurs in 7.3% of all Directive system responses.

Prehospital quality indicator 9: patient safety, satisfaction and quality improvement (Q26-28)
Clinicians from Professionally Autonomous systems reported their paramedic service has a dedicated patient safety reporting system in 88.1% of responses. Likewise, most respondents from Professionally Autonomous systems (81.6%) indicated that their paramedic service collects and analyses data pertaining to patient experience and satisfaction for the purpose of quality improvement. Lastly, a far higher proportion of Professionally Autonomous system paramedics (88.6%) reported they had a quality improvement program that includes quality assessment/measurement, control and improvement in comparison to the Directive system cohort (50.1%).

Prehospital quality indicator 10: employee welfare (Q29-31)
Amongst participants from Directive paramedic services, 51.6% shared that their employer schedules paramedics to work shifts 12 hours or longer in duration with only 7.4% of Professionally Autonomous systems doing the same. Professionally Autonomous cohort respondents (90.1%) shared that their paramedic service provides mental health programs to its paramedics compared to 47.9% in Directive services.

Prehospital quality indicator 11: overall satisfaction
Two additional survey questions were dedicated to overall employee satisfaction ( Table 5). Satisfaction was determined to be higher in the Professionally Autonomous cohort. Overall satisfaction with their services' clinical care standard followed a similar pattern, with 87.1% of participants in a Professionally Autonomous system stating they were "satisfied" or "very satisfied" with the care they deliver compared to 39.5% of Directive system respondents. There was a significant difference between group responses, with Professionally Autonomous respondents more satisfied with leadership (z=9.81, p<0.001) and care delivery (z=11.25, p<0.001) within their service.

Discussion
The results of this study portray paramedic perception of the performance of the two systems of practice under investigation and have highlighted significant disparity with respect to the perceived performance of each system.

Staff and demographics
The respondent demographics ( Table 2) are a vital step in validating the results of a previously published Delphi study [8] and confirms several previously held assumptions. First, it validates that the Professionally Autonomous paramedic system is largely found in Australia, New Zealand and the UK, with the majority of the workforce indicating they hold a registration issued by a regulatory board and practice at an advanced clinical level, i.e. Specialist Paramedic. Second, it confirms that most respondents who identified as working within a Directive paramedic system are overwhelmingly located in the US and Canada, are licensed rather than registered and practice at a basic life support level (emergency medical technician and primary care paramedic) reflecting the significant differences in education between the two systems [22][23][24]. The third point worth noting is that those who identified themselves as operating within a Professionally Autonomous paramedic system tended to be younger, a fact that has been substantiated by the yearly demographics report of the Council of Ambulance Authorities [25]. Finally, the results confirm gender balance between the two systems was consistent with previous evidence indicating Professionally Autonomous systems have made great strides in achieving gender equity in the workforce [25].

Perceptions of care standards
This survey assessed five common clinical domains as a means of assessing respondents' perception of the quality of care provided within their system. With respect to out-of-hospital cardiac arrest systems identified to be Directive in nature, they were less likely to offer instruction to callers in chest-compressiononly cardiopulmonary resuscitation and to identify a patient to be in cardiac arrest prior to the arrival of the first ambulance. This is surprising given the strong advocacy by the US-based Global Resuscitation Alliance [26] whose aim is to help to improve survival in out-of-hospital cardiac arrests through their 10step community-based program. One hypothesis may be that the Global Resuscitation Alliance is more impactful in larger jurisdictional systems such as those found in Australia, New Zealand, the United Kingdom and parts of Canada when compared to the United States, a country that lacks a nationally standardized system or scope of practice [27]. As such, the United States has thousands of paramedic systems that must conform to their respective state's and local jurisdiction's specific legislation, regulations, medical protocols, scopes of practice, standards of care, policies, procedures and requirements [27]. Similarly, it was surprising that Directive respondents, mostly based in the United States, indicated their system did not contribute to a cardiac arrest registry given the availability of a national free-to-access Resuscitation Outcomes Consortium Epistry [28]. Similar national registries exist in the United Kingdom, Australia and New Zealand where respondents indicated better uptake by their systems [29,30].
Care of acute coronary syndrome patients showed a similar difference with far fewer respondents from Directive systems indicating they were equipped with a 12-lead electrocardiogram capable monitor/defibrillator, which also directly correlated with their ability to perform a 12-lead ECG on patients presenting with signs and symptoms of acute coronary syndrome as well as providing a referral to a centre capable of providing primary percutaneous coronary intervention. This is likely linked back to the reduced scope of practice driven by a workforce that is predominantly trained in a technical institute. Cash et al. report that 86% of the 790 education programs in the US award a certificate, with the remaining 14% awarding either a diploma, associate degree or bachelor's degree [23]. This contrasts with the Professionally Autonomous paramedic system where a degree to entry practice is mandatory for all new paramedic applicants resulting in the baseline qualification of advanced care paramedic (or equivalent) [22,31,32]. The same drivers likely influence the results related to the administration of analgesia to patients experiencing pain and the various trauma-related interventions.
In contrast, in what is considered a truly novel approach in expanding the paramedic-led delivery of healthcare beyond emergency health response, respondents from the Professionally Autonomous system reported that they had greater access to care pathways for end-of-life patients (86.1%). However, in balancing the discussion, the utilisation of paramedics within palliative care contexts is not without challenges. In Australia, for example, higher transport rates have been reported, with approximately threequarters of the 4348 palliative care patients who contacted a state ambulance service in a 12-month period being conveyed to a hospital [33]. The core of the issue may lie in a lack of confidence and competence in managing palliative patients due to a perceived aperture in paramedic education across both systems [34].

Clinical advice and consultation
Similarly, many respondents in the Professionally Autonomous paramedic system (89.1%) indicated they had access to a paramedic-led clinical advice and consultation service with only 27.4% of respondents employed in a Directive system indicating they had access to the same resource. A recent study by Armour et al. [35], in which paramedic-led telehealth consultations were studied in British Columbia, Canada, noted that paramedics felt that their peers were uniquely suited to provide clinical consultations and advice in a way physicians were not based on their lived experience of out-of-hospital care. It's likely that the professionalization and self-regulation of paramedics combined with an undergraduate and postgraduate education have helped Professionally Autonomous paramedic services recognise the role paramedics have to play in delivering teleconsultations. Furthermore, with no uniform requirements for paramedics in the United States to undertake a higher education degree in order to enter practice, an argument for the irrelevance of the physician-led clinical advice model is hard to make. A question worth asking is whether the concept of medical direction has a role to play in hindering the inconsistent education of paramedic service providers in North America.

Virtual and integrated care
Comparisons between the Professionally Autonomous and Directive systems indicated that Professionally Autonomous paramedic systems offered a higher rate of telehealth (virtual care) services (80.6% vs 20.5%) as well as provided their workforce with a suite of non-conveyance options, including 'hear and treat' and 'see and treat' (90.5% vs 32.9%). Additionally, these same systems provided a higher rate of patient follow-up, a service directly correlated to the higher number of patients discharged by paramedics without further conveyance. There are many enablers that contribute to the successful delivery of telehealth services and alternative care pathways, most important of which is a paramedic system that is highly integrated into the local health network in which there is an equal funding model for conveyance vs. non-conveyance decisions [36]. Funding is a key barrier in Directive paramedic systems, particularly in the United States where paramedic regulations have historically only allowed payment for emergency ground ambulance services when individuals are transported to hospital [37]. Most patients who call an ambulance are therefore only transported to a hospital emergency department, even when a lower-acuity destination or no need for transportation may more appropriately meet an individual's clinical care needs [38]. However, this model is now starting to shift, with the recent introduction of the Emergency Triage, Treat, and Transport innovation program by the US Government [37]. This program aims to change the funding model for paramedic systems away from incentivising conveyance to a hospital toward equal funding access for patients who are either (1) conveyed to alternate care centres, such as urgent primary care centres, (2) discharged with care and advice at home or (3) assessed using a telehealth model of care. This radical shift in policy is now transforming the way in which patients within Directive paramedic systems receive care with hundreds of providers already switching to telehealth models of care [39].

Clinical governance functions
One of the commonly touted benefits of medical direction in paramedicine is the ability to help integrate public safety-facing systems better into the healthcare system through governance functions provided by Medical Directors [40]. Yet, this survey identified that more than half of respondents who practice in a Directive paramedic system do not have a patient safety reporting system (55.6%) or quality improvement program (57.5%) within their service. A stark contrast to the Professionally Autonomous paramedic system, which is typically more fully integrated into the health care network and whose respondents indicated above 80% inclusion of the above governance functions. These results validate the recent findings of a literature review exploring the relationship between medical oversight and clinical governance, which concluded that there was little high-quality evidence supporting the effectiveness of "medical direction" model as a proxy for clinical quality and safety in paramedic services. Rather, clinical governance is more effective when modelled as a systems approach with shared responsibility for quality and safety [5].

Employee welfare initiatives
Directive paramedic systems again rated poorly with respect to employee welfare initiatives with over half of the respondents surveyed (51.6%), indicating they performed shifts greater than 12 hours in duration. The literature on this topic is highly developed and notes that for outcomes considered critical or important to paramedics, shifts <12 hours in duration are more favourable than shifts ≥12 hours [41,42]. Even more concerning was the absence of mental health support for paramedics in some systems, with 49.4% of respondents working in a Directive paramedic system noting they had no access to these supports, in contrast to 92.0% of paramedics in Professionally Autonomous paramedic systems identified that they did.
There is an abundance of literature linking sleep, shift work and mental health supports together when caring for employee welfare [43,44], and the evidence overwhelmingly supports a reduction in depression, anxiety and improved well-being when employee welfare initiatives are implemented and utilized [45].

Limitations
This survey carries the risk of response bias from participants who seek to both exaggerate and/or understate the performance of their system [46]. Second, there can be a difference between perceptions of performance and performance itself, and this can lead to perception bias as a factor in the results of this study. This was addressed in the design of the questionnaire by ensuring that questions were not anchored to any system identifiers, however, ultimately the indicators are based on actions rather than outcomes. Third, the recruitment of participants through social media led to the presence of selection bias in the recruitment strategy. Fourth, the exclusion of volunteers as participants may have resulted in selection bias. Volunteers were excluded as participants, as there is often a lack of consistency in the definition and role of "paramedic" worldwide, and many volunteers are not medically trained first responders. Although we acknowledge that there are exceptions to this, we believed that excluding volunteers would lead to more accurate and clearer data for our research. Finally, whilst the demographic results have helped to validate previous definitions of the two systems, we found a natural cross-over in demographic information. For example, some participants who identified working in a Professionally Autonomous system indicate they are licensed rather than registered. Whilst human error can potentially account for some results, these results have also helped us to validate a previously held belief that many systems are hybrid in nature, a factor that was not accounted for as part of the questionnaire process.

Conclusions
This research is a vital step in helping paramedics and leaders understand the complex relationship between paramedic system design and system performance, and for the first time, provides the evidence upon which to base an argument for change. The perceptions of paramedics across 11 prehospital quality indicators provide evidence indicating the Professionally Autonomous paramedic system outperforms the Directive system in all areas that were studied. Future research needs to explore a knowledge-to-action framework that supports systems to transition from a Directive model of care to that of the better performing Professionally Autonomous paramedic system as well as the development of consistent reporting frameworks for comparison and analysis of system performance.  All participants provided written informed consent prior to commencing the online survey. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.