Evaluating Peer Review of Palliative Radiation Plans at a Canadian Tertiary Care Cancer Center

Introduction: Peer review (PR) of palliative-intent radiation plans is an important but understudied component of quality assurance. This retrospective review aims to improve our understanding of palliative PR by examining the characteristics of reviewed plans and peer feedback along with the associated time burden of two different types of PR processes. Methods: This single-institution, quality assurance project assessed palliative PR between 2018 and 2020. Initially, the PR involved a multi-disciplinary team PR. Subsequently, it transitioned to independent PR by a single physician. Characteristics of reviewed plans and feedback on PR were captured and abstracted. Time requirements of PR were based on self-reported estimates and attendance records. Results: A total of 1942 cases were reviewed, representing 85.7% (1942/2266) of all palliative-intent plans between 2018 and 2020. A total of 41.1% (n=799) were simple (2D/3D) radiation plans while 56.0% (n=1087) were complex (volumetric modulated arc therapy (VMAT) or tomotherapy) plans. Approximately one-third (30.4%, n=590) of all plans were stereotactic treatments. The rate of any peer feedback was 2.3% (n=45), while the rate of a specific recommended or implemented change was 1.2% (n=24) and 0.9% (n=18), respectively. PR before the start of treatment was associated with more frequent recommended (p=0.005) and implemented changes (p=0.008). Most other factors, including plan complexity and use of stereotactic radiation, were not predictive in this analysis. Comparing the independent versus team PR approach, there was no significant difference in recommended or implemented changes. The mean±standard deviation (SD) staff time required per plan reviewed was 36±6 and 37±6 minutes, including 21±6 and 10±6 minutes of physician time, for team and independent PR, respectively. Conclusion: This work highlights the high frequency of complex and stereotactic radiation in the palliative setting, along with the importance of timely PR and the potential benefit of reviewing even simple, 2D/3D radiation plans. Additionally, from a process perspective, our work showed that independent PR may require less dedicated physician time.


Introduction
Quality assurance (QA) programs are an important part of ensuring the best possible clinical care for patients receiving radiation and, a robust QA program includes peer review (PR) of radiation treatment plans.In the context of radiation therapy, PR involves a review of the treatment plan by at least one qualified radiation oncologist (RO) besides the treating physician [1].This is a crucial step in the treatment planning process as appropriate PR can help to identify plan errors, improve standardization of practice, as well as promote continued education among staff [1][2][3][4][5].
The process of PR can be quite varied between different institutions [6].It can take a variety of forms, including one-on-one PR, multi-disciplinary team PR, and inter-institutional PR [7].During one-on-one PR, subsequently referred to as independent PR, a single RO other than the treating physician independently reviews the treatment plan.During multi-disciplinary team PR, subsequently referred to as team PR, professionals from various specialties, such as radiation oncology, medical physics, dosimetry, and radiation therapy, meet to review and discuss radiation plans [7].For inter-institutional PR, plan review involves various professionals at different institutions [7].Differences in efficacy and resource utilization between the different PR models are not well understood.
Much of the literature on PR of radiation treatment plans has focused on curative intent treatment.A systematic review of the literature, which included mainly curative intent treatment plans, estimated that PR led to clinical change in approximately one in nine of all reviewed plans; however, there was a wide range in the rate of clinical change [1].The data for PR of specifically palliative-intent plans is more limited but also shows notable variation in rates of recommended or implemented changes.The reported rates of recommended changes following palliative PR range from 2.1% to 28% and the rate of implemented changes range from 1.3% to 6.1% [8][9][10][11].Given the scarcity of the literature, we performed a retrospective, singleinstitution analysis to evaluate the characteristics of reviewed plans as well as the outcomes of PR, including the rate and type of peer feedback along with any factors influencing error identification in palliative-intent radiation plans.Additionally, during the time frame of this study, it transitioned from team PR with weekly meetings to independent PR.This offered the opportunity to compare different PR processes, including the associated time burden of each.

Captured data
This single-institution, QA project assessed palliative QA review at a Canadian academic cancer center (London Regional Cancer Program, London, Ontario, Canada), between September 2018 and October 2020.Relevant data were captured by a palliative clinical specialist radiation therapist (pCSRT).Captured data included patients' age and gender, treating RO, primary diagnosis, treatment site, treatment status, radiation prescription, treatment technique, and additional free-text clinical notes (including prior radiation and systemic therapy treatment details).Data were compiled and retrospectively grouped into standardized categories for 1) diagnosis, 2) treatment site, 3) plan complexity, and 4) the use of stereotactic radiation.Stereotactic radiation was defined during the initial data collection by the pCSRT based on a high dose per fraction.Free-text clinical notes had been captured at the time of the initial treatment plan review by our pCSRT.These were retrospectively reviewed to identify patients with previous radiation and, based on the prior site of treatment, were categorized based on the presence of potential overlap with the treatment plan being peer-reviewed.

Team peer review
From late September 2018 to early March 2020, palliative QA used a team PR approach with in-person rounds, scheduled for one hour each week, and led by the pCSRT with attendance of ROs, medical physicists, as well as RO residents and other learners.Contours, fields, plans, supplemental notes, and images were reviewed by the group, as available.The pCSRT prospectively captured which plans were reviewed and recorded peer feedback, if any, on each plan.Peer feedback was retrospectively categorized as either 1) queries (questions about the plan that were not a specific recommendation) or 2) change recommendations (major or minor).Classification of major versus minor change recommendations was based on definitions by Rouette et al. [12].A major change was "a change requiring repeat planning and/or having a foreseeable effect on treatment toxicity or cancer outcomes" [12].A minor change "did not meet the criteria for a 'major' change and did not lead to significant repeat treatment planning" [12].Categorization was performed retrospectively by two independent reviewers (SG and AM), with discrepancies settled by a third reviewer (GR).Reviewers were blinded to the treating physician and the type of review (independent versus team PR).
Of note, during the team PR period, queries were not recorded if the treating RO was present to address the question during rounds.Attendance of ROs was recorded through a sign-in sheet.This information, along with the frequency of queries recorded when the treating RO was not present, was used to estimate the number of unrecorded queries during team PR.

Independent peer review
Starting in mid-March 2020 and extending until October 2020, the QA process was changed to independent PR due to the COVID-19 pandemic and the associated need for physical distancing.Initially, familiarity with virtual meeting software was limited, and asynchronous, offline QA was more accessible and more convenient in a time when staffing scarcity was a concern.During this independent PR period, each plan was assigned to a single RO for independent review, as during the team PR period, the pCSRT kept detailed records including the same patient and treatment data as was recorded during team PR along with a record of any peer feedback.

Time estimates
RO attendance and hours spent on PR were determined from weekly attendance sheets, with one pCSRT and one medical physicist present at each meeting.When calculating time per peer feedback, the extrapolated number of queries was used.For the independent PR period, RO PR time was based on the number of plans reviewed and the staff survey estimation of the minutes required to review a palliative plan.Preparation and follow-up time for pCSRT were added based on the self-reported time dedicated to PR per week.

Statistical analysis
Descriptive statistics were generated, stratified by cohort (team versus independent PR), and compared using the chi-square test, Fisher's exact test, independent two-sample t-test, or Wilcoxon rank sum test, as appropriate.Univariable and multivariable logistic regression were performed for recommend changes and applied changes for all eligible variables for all patients and analyzed per plan.All eligible variables were incorporated into a multivariable regression model and sequentially removed using backward elimination techniques until all remaining covariates had p-values < 0.05 with the exception of the cohort (retained in all models).All statistical analyses were performed using SAS version 9.4 software (SAS Institute, Cary, North Carolina, United States), using two-sided statistical testing at the 0.05 significance level.

Reviewed plans
In total, 1942 cases were reviewed, representing 85.7% (1942/2266) of all palliative plans during the study period.Details of the reviewed plans are summarized in Table 1.The unreviewed plans were not included in this study.The most common primaries were lung, breast, and prostate cancer.The most common treatment sites were brain, spine, and bone (non-spinal, non-pelvic).A total of 41.1% (n=799) of the plans were categorized as simple (2D/3D) radiation plans, and 56.0%(n=1087) were complex (volumetric modulated arc therapy (VMAT) or tomotherapy) plans.Approximately one-third (30.4%, n=590) of all plans were stereotactic treatments.Nearly half (47.5%, n=922) of the plans were reviewed before treatment started, 42.9% (n=833) were reviewed while treatment progressed, and 4.2% (n=82) were reviewed after the completion of treatment.Of the 82 plans that were completed prior to PR, the majority (n=54, 65.9%) were single-fraction treatments.

Characteristic
All palliative plans (n=1942)  Of the 1942 reviewed cases, 1657 (85.3%) underwent team PR, and 285 (14.7%) underwent independent PR.There were significant baseline differences in treatment plan characteristics including diagnosis, treatment site, and plan complexity (see Appendix 1).When transitioning to independent PR, complex plans were prioritized for review.During the independent PR period, 95.8% (n=273) of the reviewed plans were complex and 58.9% (n=168) were stereotactic, compared to 49.1% (n=814) and 25.5% (n=422), respectively, during the team PR period (p < 0.001).
Finally, within team PR only, the presence of the treating RO was also recorded (i.e., was the treating physician present during team PR of their radiation plan?).Over half of the time a plan was reviewed (57.9%), the treating RO was present during the review.
Comparing team PR and independent PR, the rates of peer feedback were 2.5% (n=41/1657) and 1.4% (n=4/285), respectively (p=0.267).Accounting for the estimated number of unrecorded queries that occurred during team PR (refer to Team peer review in the Materials & Methods section), the rate rose to 3.6% (60/1657), which trended toward significant when compared to independent PR (p=0.053)(Figure 1).Rates of major or minor change recommendations were 1.3% (n=22) and 0.7% (n=2) for team and independent PR, respectively (p=0.145).Rates of implemented changes were 1.0% (n=17) and 0.4% (n=1) and an average of one change was implemented per 97 and 285 plans reviewed for team and independent PR, respectively (p = 0.50).Peer feedback includes all queries and recommended changes.For team PR, the unrecorded queries were estimated as described.

Predictors of feedback
On multivariable logistic regression, the frequency of recommended changes (major and minor) was impacted by the presence (versus absence) of the treating RO at the time of review (odds ratio (OR): 0.31; 95% confidence interval (CI): 0.12-0.75;p=0.010), and whether the patient had started treatment (OR: 3.80; 95% CI: 1.49-9.67;p=0.005;  p-values < 0.05 shown in bold. 1 Only includes prior radiation that was potentially relevant to the treatment plan being reviewed. OR: odds ratio; CI: confidence interval; PR: peer review; NR: not reported; VMAT: volumetric modulated arc therapy Rate of feedback of PR was similar between simple, 2D/3D versus VMAT or tomotherapy radiation plans as well as between stereotactic versus non-stereotactic radiation plans with no significant difference in the rate of recommended or implemented change (Table 2).Other examined factors, including the type of plan review (team versus independent PR), demographics, site, and previous radiation, were also not significantly predictive of recommended or implemented changes.Primary diagnoses of breast or prostate cancer were both associated with higher rates of implemented, but not recommended changes.Additionally, as a sensitivity analysis, univariable logistic regression was repeated for team PR plans only to examine the effect of the number of ROs present.The frequency of recommended or implemented changes was not significantly affected by the number of ROs present during PR (see Appendix 2).

Time dedicated to palliative PR
The mean number of ROs present during the weekly team PR meeting was 8.1±2.4 (range: 2-14).The mean weekly total time requirement of all oncology staff to participate in team PR for palliative plans was estimated at 14.1±2.4hours per week, including 8.1±2.4 hours of total physician time per week.The mean time per plan reviewed was 35.7±6.0 minutes.The estimated mean time per independent PR plan reviewed was similar to team PR at 37.1±6.4minutes; however, the burden on physician time was less (20.5±6.0 minutes versus 9.7±6.4minutes per plan reviewed for team versus independent PR).Further examination of the effect of the number of ROs present as well as the effect of differing rates of recommended or implemented change reported in prior studies is further examined in Appendix 3.

Discussion
Our study provides a detailed retrospective examination of palliative PR at a large academic center.We report on relevant demographic data, including the high frequency of complex and stereotactic treatment techniques used in the palliative setting.Additionally, we present an analysis of predictors of recommended and implemented changes in palliative radiation treatment following PR along with a novel comparison of the peer feedback and time required for two different PR approaches.
Comparing our data to the available literature, the rate of peer feedback for team PR of palliative plans was similar to that found by for a large number of plans that underwent both 'early' team PR within 48 hours of CT simulation as well as 'late' team PR after treatment planning was completed [10].They also found a high rate of recommended changes at 28% (143/508) [10].The wide variation in the rates of recommended and implemented changes is also seen in the curative-intent PR literature.A pooled analysis of PR of 11,491 predominantly radical plans showed a weighted mean change rate of 10.8%, with a wide variance (3.0%-100%) across different series [1].
Interpreting rates of peer feedback can be challenging and we lack a good benchmark metric for the rate of radiation plan peer feedback, especially in the palliative setting.The striking ranges may be attributable to multiple factors, including a lack of formalized and consistent change definitions, inconsistencies in recording recommendations, group culture regarding peer critique, relative consistency in clinical practice between oncologists, different PR models, and retrospective data collection.Despite the high importance placed on palliative PR in our institution, the relatively low peer feedback rate observed in our cohort may have been influenced by any of these factors.Additionally, not all peer feedback necessarily leads to a change in treatment and institutional practice can vary on how this is handled.Within our own institution, the implementation of queries or recommended changes is left to the discretion of the treating physician.
Reasons behind why recommendations were not implemented were not consistently tracked within our retrospective data; however, we did find that major recommendations were more likely to be implemented compared to minor recommendations or queries.Within our analysis, we used the definition of major versus minor recommendation suggested by Rouette et al. [12].In their work, a major change was "a change requiring repeat planning and/or having a foreseeable effect on treatment toxicity or cancer outcomes", while a minor change "did not meet the criteria for a 'major' change and did not lead to significant repeat treatment planning" [12].They did not use queries within their work, which we defined here as questions about the plan that were not a specific recommendation.There is a lack of standardization within the literature of definitions of these categories and we suggest that the definitions we use here are a reasonable starting point.Standardization of definitions, along with prospective tracking of recommendations and treating physician's responses to peer feedback with well-defined categorization across institutions would help to improve comparability and overall PR quality moving forward.
The time required for PR is a commonly cited barrier to broader implementation [13].As such, analyzing and optimizing process efficiency is of interest, especially in the setting of growing stresses on healthcare resources.Independent PR required approximately half as much physician time per plan reviewed as team PR (20.5±6.0 minutes and 9.7±6.4minutes, respectively).Additionally, our study did not find a significant difference in peer feedback rates between independent PR when compared to team PR.While there may have been a trend toward a higher rate of peer feedback when including queries, there was no difference in recommended or implemented changes.This suggests that while a team-based approach may generate more discussion, this may not necessarily translate into actual changes in the treatment plan.Additionally, subsequent analysis within the team PR model did not identify a clear relationship between the number of ROs present and the likelihood of recommended or implemented changes.However, given that we cannot know the ideal or optimal rate of peer feedback, we cannot truly comment on the comparative efficacy of team PR versus independent PR.
Nevertheless, taken together, these findings raise the potential for future work looking at the optimal number of physicians involved in team PR as well as further comparisons of the outcome of team versus independent PR, ideally in a prospective manner and with a large sample.An Independent review of identical plans using different PR processes would allow a better assessment of the efficacy of each method.Additionally, as a potential aspect of this future work, we note that the rapid acceleration of artificial intelligence in radiation oncology may also represent an exciting horizon to increase the efficacy and throughput of PR processes, with the potential for augmentation or replacement in some settings [14].However, the lack of defined guidelines, subjective target volumes, and complexity in cases of reirradiation will require careful evaluation of large language models to assess efficacy.Further, while striving for efficiency remains crucial, we must also recognize the less measurable benefits of PR that may not be met as well by certain PR processes such as independent PR, including inter-disciplinary discussions, learning for trainees, and continued education for physicians.
In evaluating outcomes of PR, our analysis identified several predictors for peer feedback on multivariable logistic regression.The timing of PR was found to be a significant predictor, with radiation plans reviewed prior to treatment initiation having significantly higher rates of peer feedback.This is consistent with prior work in the non-palliative setting [10,15].Early PR provides a better opportunity to address concerns and adapt the treatment plan.However, palliative treatments tend to have fewer fractions and start quickly to manage symptoms, making PR prior to the start of treatment difficult and decreasing the opportunity for change.While many institutions, including ours, meet on a weekly basis for team PR [6], more frequent PR could decrease this problem.Thompson et al. routinely reviewed palliative plans twice a week with additional ad-hoc sessions scheduled to review urgent plans, and, using this approach, 96.4% of plans were reviewed before or on the day of the start of treatment [8].Alternatively, a hybrid model, wherein urgent cases are reviewed by independent PR after approval by the treating physician as opposed to waiting for weekly team PR, may reduce the number of cases being reviewed after the start of treatment, while minimizing the additional time burden on physicians of the frequent PR required to ensure cases are reviewed prior to the start of treatment.
The finding that the absence of the treating RO was predictive of a change recommendation can be interpreted in different ways.It is possible that their presence dissuaded criticism, or that they provided rationale, negating the need for formal change recommendation.The presence of the treating RO did not significantly affect the rate of implemented changes, which supports the latter interpretation.However, sample size may also play a role in this given that there were more recommended changes than implemented changes.Nevertheless, we also note that in the relatively anonymized independent PR model, the rate of recommendations did not increase.This decreases the likelihood that 'social intimidation' plays a significant role, but is clouded by the potential lack of rigor in a single plan reviewer.Verification would require a prospective, narrative review.
Interestingly, neither plan complexity nor stereotactic technique was found to be predictive of peer feedback, suggesting potential value in reviewing even 2D/3D radiation plans.This is consistent with prior work by Walburn et al., where the radiation treatment technique (3D-CRT (conformal radiation therapy) versus IMRT (intensity-modulated radiation therapy) versus SBRT (stereotactic body radiation therapy) versus electron versus other) was not found to be a significant predictor of change recommendation (p = 0.71) [10].Given the potential impact on institutional PR practice, this would be worth further exploration within a larger data set.However, it should also be acknowledged that while the similar rate of peer feedback for stereotactic versus non-stereotactic plans could reflect a true similarity in the rate of issues, it does also raise the possibility that there was not always sufficient expertise in general palliative team PR when reviewing more specialized cases.Given the rising complexity of palliative radiation and the increasing proportion of stereotactic treatments [16,17], it is important to consider the expertise present for palliative PR.One change our institution has made in our PR process subsequent to this study is that brain stereotactic cases are now reviewed in a separate set of specialized PR rounds.
Finally, we note that primary cancer types, specifically breast and prostate, were identified as statistically significant predictors of a completed change in the treatment plan.The clinical implications and rationale for this are unclear.This is potentially attributable to small sample size, unrecognized confounders, practice patterns of treating physicians in those sites, or the comparatively favorable prognosis of these disease sites warranting a lower threshold for implementing potentially meaningful alterations.
The findings of our work must be considered with respect to its limitations.This is a single-institution study with a limited sample size.The relatively small sample size of the independent PR period represents an important limitation of this analysis.The study is also subject to the inherent biases that may be present in retrospective studies.Being a retrospective review, the collection of peer feedback also represents a limitation.As stated, queries were not recorded if they were addressed by the treating physician during team PR.To address this, we calculated an estimate of these 'unrecorded' queries, which is subject to error (refer to Peer review in the Materials & Methods section).In contrast to queries, recommendations were always recorded, and based on our analysis, the absence of the treating physician was associated with a higher rate of recommendation.Therefore, the method used to account for 'unrecorded' queries could potentially have led to an overestimation of the 'unrecorded' queries.If this is the case, then the true rate of queries should lie in between the adjusted and unadjusted rate (i.e., between 2.5% and 3.6%).A prospective, narrative evaluation would help to further characterize these issues.
Nevertheless, despite the methodologic limitations, this analysis provides novel, hypothesis-generating data on an important topic.Palliative radiotherapy is increasing in complexity with more conformal treatments, increasing rates of re-treatment, and increasing the utilization of radiation therapy.While the rate of change was low, major changes were recommended in varying plan complexities, indications, and treatment modalities.Having the right individuals in the right environment at the right time will improve plan quality, departmental collaboration, and learning.Continued efforts to define the 'right' mix, and to optimize the PR process for all radiation plans are needed to ensure value for all stakeholders.Ideally, PR models should be compared in a prospective, randomized manner to help reduce bias and subjectivity inherent in the published works to date.

Conclusions
QA of palliative radiation remains an understudied area of the literature.Our work demonstrates the high frequency of complex and stereotactic radiation in the palliative setting and examines predictors of peer feedback within the setting of palliative QA along with the potential benefit of reviewing even simple, 2D/3D radiation plans.Timely PR, before the start of radiation is strongly predictive of peer feedback; however, as seen in our work, this can be challenging for palliative radiation.Further work looking into palliative QA, especially into comparisons of different QA processes, may help to ensure timely PR while still seeking to minimize the time burden of QA on physicians.Our work showed that independent PR can be less burdensome in terms of physician time, but additional study is needed comparing team versus independent PR processes, especially in terms of the rigor and efficacy of peer review.Time estimates are provided based on the mean ± standard deviation (SD) as well as the minimum and maximum number of ROs present at team PR during the study period.

FIGURE 1 :
FIGURE 1: Rate of peer feedback following peer review (PR) over the entire study period (team and independent PR combined) as well as for team and independent PR individually.

TABLE 1 : Baseline characteristics for all reviewed radiation treatment plans during both team and independent PR periods.
SD: standard deviation; VMAT: volumetric modulated arc therapy; PR: peer review

Table 2
). Change recommendations were less likely if the treating RO was present during the plan review, but the rate of implemented change was not affected.Plans that had not yet started were more likely to have recommended and implemented changes (OR: 4.58; 95% CI: 1.49-14.11;p=0.008).Patients with either a breast (OR: 4.33; 95% CI: 1.48-12.69;p=0.008) or prostate diagnosis (OR: 4.64; 95% CI: 1.36-15.76;p=0.014) were more likely to have an implemented change, but the rate of recommended change was not affected.

TABLE 2 : Univariable and multivariable logistic regression models for recommended and implemented changes.
Thompson et al., who reported a rate of recommended and implemented changes of 2.1% (29/1413) and 1.3% (19/1413), respectively, following team PR [8].Other publications have reported higher rates of peer feedback in palliative plans.Mitchell et al. reported a rate of implemented changes of 6.1% (7/115) and Qureshi et al. reported a 15% (3/20) rate of recommended changes following team PR, albeit both on very small samples [9,11].Another publication, Walburn et al. reported rates of peer feedback

TABLE 3 : Baseline characteristics for all reviewed radiation treatment plans by type of peer review (PR) (team PR versus independent PR).
For 'primary diagnosis' and 'treated site' only, categories representing > 2% or > 5%, respectively, of reviewed plans are shown.2Onlyincludes prior radiation that was potentially relevant to the treatment plan being reviewed.

TABLE 4 : Univariable logistic regression of the effect of the number of treating radiation oncologists present during team peer review (PR) on rates of recommended and implemented changes.
OR: odds ratio; CI: confidence interval; NR: not reported