Rebecca Walwyn

It is recognized that treatment‐related clustering should be allowed for in the sample size and analyses of individually‐randomized parallel‐arm trials that evaluate therapist‐delivered interventions such as psychotherapy. Here, interventions are a treatment factor, but therapists are not. If the aim of a trial is to separate effects of therapists from those of interventions, we propose that interventions and therapists should be regarded as two potentially interacting treatment factors (one fixed and one random) with a factorial structure. We consider the specific design where each therapist delivers each intervention (crossed therapist‐intervention design), and the resulting therapist‐intervention combinations are randomized to patients. We adopt a classical Design of Experiments (DoE) approach to propose a family of orthogonal factorial designs and their associated data analyses, which allow for therapist learning and center too. We set out the associated data analyses using ANOVA and regression and report the results of a small simulation study conducted to explore the performance of the proposed randomization methods in estimating the intervention effect and its standard error, the between‐therapist variance and the between‐therapist variance in the intervention effect. We conclude that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions and outline areas for further methodological research.

Background: Self-harm is common in adolescents and a major public health concern. Evidence for effective interventions is lacking. An individual participant data meta-analysis has potential to provide more reliable estimates of the effects of therapeutic interventions than conventional meta-analyses and to explore which treatments are best suited to certain groups. Methods: A systematic review and individual participant data meta-analysis of randomised controlled trials of therapeutic interventions to reduce repeat self-harm in adolescents with a history of self-harm and who had presented to clinical services. We searched Cochrane Library, EMBASE, trial registers and other databases for randomised controlled trials published in January 2022. Eligible randomised controlled trials compared any therapeutic intervention against a control, aimed to reduce self-harm in adolescents (11–18 years old), with past self-harm presenting to clinical services, and collected outcome data on self-harm or suicide attempts. Interventions reviewed were grouped into nine categories: cognitive–behavioural therapy; dialectical behaviour therapy; family therapy; group therapy; mentalisation based, psychodynamic, cognitive analytic therapy; multisystemic therapy; problem-solving, psychoeducation, support; postcards, tokens, documents (postcards/tokens); and other single session, brief interventions. Control interventions were all either treatment as usual or enhanced treatment as usual and were not usually well described. There were no ‘no treatment’ controls except in the postcard/document/token studies. Primary outcome was repetition of self-harm at 12 months. Other outcomes included repetition of self-harm at other time points, overall mental health, depressive symptoms, thoughts of suicide, quality of life and death. Two-stage random-effects individual participant data meta-analyses were conducted overall and by intervention, and to examine interaction between treatment received and participant characteristics. Secondary analyses incorporated aggregate data from randomised controlled trials without individual participant data. Metaregression explored moderating study effects. Results: We identified 39 eligible studies, from 10 countries, where we sought individual participant data (18 studies with full sample eligibility, 21 with partial sample eligibility). We obtained individual participant data from 26 studies of 3448 eligible participants. We used published data from a further seven studies where individual participant data were not available for a combined individual participant data aggregate data meta-analysis (698 participants). For our primary outcome, repetition of self-harm, only six studies were rated as low risk of bias. There was no evidence that intervention/s were more or less effective than controls at preventing repeat self-harm by 12 months using individual participant data (odds ratios 1.06, 95% confidence interval 0.86 to 1.31) or individual participant data + aggregate data (odds ratios 1.02, 95% confidence interval 0.82 to 1.27) and no evidence of heterogeneity of treatment effects on study and treatment factors. We found no evidence that intervention was more or less effective than control for secondary outcomes, except general psychopathology and suicidal ideation at 12 and 6 months, respectively. Across all interventions, participants with multiple prior self-harm episodes showed evidence of improved treatment effect on self-harm repetition 6–12 months after randomisation [odds ratios 0.33 (95% confidence interval 0.12 to 0.94), studies = 9, n = 1771]. Modest evidence suggesting differential treatment effects based on participants’ age, gender, self-harm method, and anxiety levels are noted. Limitations: A significant limitation was missing individual participant data where authors were unable to share data; we offset this by including published data in secondary individual participant data plus aggregate meta-analysis. A wide range of interventions were evaluated and lacked replication. There was variability in the definitions and timings of outcomes, measures used for data collection, and available moderator data, with little consistency across studies. Conclusions: More attention needs to be paid to seeking appropriate consent from study participants for data-sharing. We found no evidence that any therapeutic intervention (overall or by intervention) was more or less effective than control for reducing repeat self-harm. We are therefore unable to recommend any specific intervention to prevent repetition of self-harm in adolescents. We observed evidence and trends indicating more effective interventions within specific subgroups. Analysis was constrained due to scarcity of data concerning common baseline characteristics, outcomes, and follow-up lengths. We recommend efficient, adaptive platform trial designs to tackle research questions and ascertain the most effective interventions for different groups, covering available treatments. Funding: This synopsis presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number 17/117/11.

Background: Adherence to adjuvant endocrine therapy (AET) in women with breast cancer is poor. Multicomponent intervention packages are needed to address adherence barriers. Optimizing these packages prior to definitive evaluation can increase their effectiveness, affordability, scalability, and efficiency. Purpose: To pilot procedures for an optimization-randomized controlled trial (O-RCT) of the 'Refining and Optimizing Strategies to support Endocrine Therapy Adherence' (ROSETA) intervention. Methods: This was a multisite individually randomized external pilot trial using a 24-1 fractional factorial design (ISRCTN10487576). Breast cancer survivors prescribed AET were recruited from 5 hospitals and randomized to one of 8 conditions, each comprising a combination of 4 intervention components set to “on” or “off” (SMS messages, information leaflet, guided self-help, and self-management website). We set criteria to inform the decision to progress to an O-RCT for consent rate, component adherence, and availability of outcome measures, with predefined cutoffs for “green” (proceed), “amber” (minor changes), and “red” (major changes). Results: Among 141 eligible patients, 54 (38.3%) consented (green range). At least 50.0% of participants adhered to the minimum threshold set for each intervention component (green range). Data for one of the 3 medication adherence measures were available (amber range). Most (86.8%) participants were satisfied with their trial experience. Exploratory analysis indicated some evidence of a negative main effect of the information leaflet on medication adherence (adjusted mean difference = 0.088, 95% CI, 0.018, 0.158). Conclusions: Progression to a fully powered O-RCT of the ROSETA intervention package is feasible, but review of medication adherence measures is required.

Background: Four main types of cluster-randomised trial (CRT) are well known: parallel-group (PG), factorial, stepped-wedge and crossover designs. This established typology relates to how clusters are exposed to intervention(s) or control(s) during the trial. Published guidance is lacking on how to link design features to how individuals within clusters may be exposed and measured. Thus, the aim of this paper was to develop a classification system for different types of cluster membership in CRTs, focussing on PG designs and building on our experiences of delivering a care home trial. Methods: The classification system was developed in seven stages: i) a scoping review was conducted to explore use of open-cohort PG-CRTs in a range of settings; ii) a version of the classification system was developed, using the stepped-wedge CRT typology; iii) this was tested using a sample of published trials from the scoping review; iv) a second version was developed, reviewed and further amendments made to aid clarity; v) 15 trialists with experience of CRTs in a range of settings provided feedback in a one-day, face-to-face user engagement workshop; vi) a wider group of 39 trialists completed an online survey, providing examples and additional feedback; and vii) all authors reviewed and approved the final version. Results: Six types of cluster membership in PG-CRTs are proposed: the closed-cohort and cross-sectional designs already established, a new-admission-continuous-recruitment, open-cohort with discrete-recruitment, open-cohort with continuous-recruitment, and a non-standard closed-cohort design. The final classification system is made up of six core design features and five additional design considerations. Diagrams of each type of cluster membership are introduced and used to illustrate examples. Conclusions: Implications of distinctions between the six types of cluster membership for the statistical analysis requires further research. CONSORT guidance needs updating to include specific guidance on reporting the type of cluster membership alongside the description of how design features apply to clusters. Further methodological research is required into both the statistical and the practical implications of adopting previously unlabelled but frequently used types of cluster membership.

Background: Self-harm is common in adolescents and a major public health concern. Evidence for effective interventions is lacking. An individual patient data meta-analysis has the potential to provide more reliable estimates of the effects of therapeutic interventions for self-harm than conventional meta-analyses, to explore which treatments are best suited to certain groups. Method: A systematic review and individual patient data meta-analysis of randomised controlled trials of therapeutic interventions to reduce repeat self-harm in adolescents who had a history of self-harm and presented to clinical services. Primary outcome was repetition of self-harm. The methods employed for searches, study screening and selection, and risk of bias assessment are described, with an overview of the outputs of the searching, selection and quality assessment processes. Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidance is followed. Results: We identified a total 39 eligible studies, from 10 countries, where we sought Individual Patient Data (IPD), of which the full sample of participants were eligible in 18 studies and a partial sample of participants were eligible in 21 studies. We obtained IPD from 26 studies of 3448 eligible participants. For our primary outcome, repetition of self-harm, only 6 studies were rated as low risk of bias with 10 rated as high risk (although 2 of these were for secondary outcomes only). Conclusions: Obtaining individual patient data for meta-analyses is possible but very time-consuming, despite clear guidance from funding bodies that researchers should share their data appropriately. More attention needs to be paid to seeking appropriate consent from study participants for (pseudo) anonymised data-sharing and institutions need to collaborate on agreeing template data-sharing agreements. Researchers and funders need to consider issues of research design more carefully. Our next step is to analyse all the data we have collected to see if it will tell us more about how we might prevent repetition of self-harm in young people. Funding: This article presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number 17/117/11.

Two accepted designs exist for parallel-group cluster-randomised trials (CRTs). Closed-cohort designs follow the same individuals over time with a single recruitment period before randomisation, but face challenges in settings with high attrition. (Repeated) cross-sectional designs recruit at one or more timepoints before and/or after randomisation, collecting data from different individuals present in the cluster at these timepoints, but are unsuitable for assessment of individual change over time. An ‘open-cohort’ design allows individual follow-up with recruitment before and after cluster-randomisation, but little literature exists on acceptability to inform their use in CRTs. To document the views and experiences of expert trialists to identify: Qualitative consultation via two expert workshops including triallists (n = 24) who had worked on CRTs over a range of settings. Workshop transcripts were analysed using Descriptive Thematic Analysis utilising inductive and deductive coding. Two central organising concepts were developed. Design and conduct challenges with established CRT designs confirmed that current CRT designs are unable to deal with many of the complex research and intervention circumstances found in some trial settings (e.g. care homes). Perceptions of potential benefits and barriers of open cohort designs included themes on: approaches to recruitment; data collection; analysis; minimising/investigating the impact of bias; and how open-cohort designs might address or present CRT design challenges. Open-cohort designs were felt to provide a solution for some of the challenges current CRT designs present in some settings. Open-cohort CRT designs hold promise for addressing the challenges associated with standard CRT designs. Research is needed to provide clarity around definition and guidance on application.

Background: Adherence to adjuvant endocrine therapy (AET) is low in women with breast cancer. Negative beliefs about the necessity of AET and high concerns are barriers to adherence. Purpose: To use the multiphase optimization strategy to optimize the content of an information leaflet intervention, to change AET beliefs. Methods: We conducted an online screening experiment using a 25 factorial design to optimize the leaflet. The leaflet had five components, each with two levels: (i) diagrams about AET mechanisms (on/off); (ii) infographics displaying AET benefits (enhanced/basic); (iii) AET side effects (enhanced/basic); (iv) answers to AET concerns (on/off); (v) breast cancer survivor (patient) input: quotes and photographs (on/off). Healthy adult women (n = 1,604), recruited via a market research company, were randomized to 1 of 32 experimental conditions, which determined the levels of components received. Participants completed the Beliefs about Medicines Questionnaire before and after viewing the leaflet. Results: There was a significant main effect of patient input on beliefs about medication (β = 0.063, p < .001). There was one significant synergistic two-way interaction between diagrams and benefits (β = 0.047, p = .006), and one antagonistic two-way interaction between diagrams and side effects (β = −0.029, p = .093). There was a synergistic three-way interaction between diagrams, concerns, and patient input (β = 0.029, p = .085), and an antagonistic four-way interaction between diagrams, benefits, side effects, and concerns (β = −0.038, p = .024). In a stepped approach, we screened in four components and screened out the side effects component. Conclusions: The optimized leaflet did not contain enhanced AET side effect information. Factorial experiments are efficient and effective for refining the content of information leaflet interventions.

Background: National clinical audit programmes aim to improve patient care by reviewing performance against explicit standards and directing action towards areas not meeting those standards. Their impact can be improved by (1) optimising feedback content and format, (2) strengthening audit cycles and (3) embedding randomised trials evaluating different ways of delivering feedback. Objectives: The objectives were to (1) develop and evaluate the effects of modifications to feedback on recipient responses, (2) identify ways of strengthening feedback cycles for two national audits and (3) explore opportunities, costs and benefits of national audit participation in a programme of trials. Design: An online fractional factorial screening experiment (objective 1) and qualitative interviews (objectives 2 and 3). Setting and participants: Participants were clinicians and managers involved in five national clinical audits – the National Comparative Audit of Blood Transfusions, the Paediatric Intensive Care Audit Network, the Myocardial Ischaemia National Audit Project, the Trauma Audit & Research Network and the National Diabetes Audit – (objective 1); and clinicians, members of the public and researchers (objectives 2 and 3). Interventions: We selected and developed six online feedback modifications through three rounds of user testing. We randomised participants to one of 32 combinations of the following recommended specific actions: comparators reinforcing desired behaviour change; multimodal feedback; minimised extraneous cognitive load for feedback recipients; short, actionable messages followed by optional detail; and incorporating ‘the patient voice’ (objective 1). Main outcome measures: The outcomes were intended actions, including enactment of audit standards (primary outcome), comprehension, user experience and engagement (objective 1). Results: For objective 1, the primary analysis included 638 randomised participants, of whom 566 completed the outcome questionnaire. No modification independently increased intended enactment of audit standards. Minimised cognitive load improved comprehension (+0.1; p = 0.014) and plans to bring audit findings to colleagues’ attention (+0.13, on a –3 to +3 scale; p = 0.016). We observed important cumulative synergistic and antagonistic interactions between modifications, participant role and national audit. The analysis in objective 2 included 19 interviews assessing the Trauma Audit Research Network and the National Diabetes Audit. The identified ways of strengthening audit cycles included making performance data easier to understand and guiding action planning. The analysis in objective 3 identified four conditions for effective collaboration from 31 interviews: compromise – recognising capacity and constraints; logistics – enabling data sharing, audit quality and funding; leadership – engaging local stakeholders; and relationships – agreeing shared priorities and needs. The perceived benefits of collaboration outweighed the risks. Limitations: The online experiment assessed intended enactment as a predictor of actual clinical behaviour. Interviews and surveys were subject to social desirability bias. Conclusions: National audit impacts may be enhanced by strengthening all aspects of feedback cycles, particularly effective feedback, and considering how different ways of reinforcing feedback act together. Future work: Embedded randomised trials evaluating different ways of delivering feedback within national clinical audits are acceptable and may offer efficient, evidence-based and cumulative improvements in outcomes. Trial registration: This trial is registered as ISRCTN41584028. Funding details: This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 15. See the NIHR Journals Library website for further project information.

Background National audits aim to reduce variations in quality by stimulating quality improvement. However, varying provider engagement with audit data means that this is not being realised. Aim The aim of the study was to develop and evaluate a quality dashboard (i.e. QualDash) to support clinical teams’ and managers’ use of national audit data. Design The study was a realist evaluation and biography of artefacts study. Setting The study involved five NHS acute trusts. Methods and results In phase 1, we developed a theory of national audits through interviews. Data use was supported by data access, audit staff skilled to produce data visualisations, data timeliness and quality, and the importance of perceived metrics. Data were mainly used by clinical teams. Organisational-level staff questioned the legitimacy of national audits. In phase 2, QualDash was co-designed and the QualDash theory was developed. QualDash provides interactive customisable visualisations to enable the exploration of relationships between variables. Locating QualDash on site servers gave users control of data upload frequency. In phase 3, we developed an adoption strategy through focus groups. ‘Champions’, awareness-raising through e-bulletins and demonstrations, and quick reference tools were agreed. In phase 4, we tested the QualDash theory using a mixed-methods evaluation. Constraints on use were metric configurations that did not match users’ expectations, affecting champions’ willingness to promote QualDash, and limited computing resources. Easy customisability supported use. The greatest use was where data use was previously constrained. In these contexts, report preparation time was reduced and efforts to improve data quality were supported, although the interrupted time series analysis did not show improved data quality. Twenty-three questionnaires were returned, revealing positive perceptions of ease of use and usefulness. In phase 5, the feasibility of conducting a cluster randomised controlled trial of QualDash was assessed. Interviews were undertaken to understand how QualDash could be revised to support a region-wide Gold Command. Requirements included multiple real-time data sources and functionality to help to identify priorities. Conclusions Audits seeking to widen engagement may find the following strategies beneficial: involving a range of professional groups in choosing metrics; real-time reporting; presenting ‘headline’ metrics important to organisational-level staff; using routinely collected clinical data to populate data fields; and dashboards that help staff to explore and report audit data. Those designing dashboards may find it beneficial to include the following: ‘at a glance’ visualisation of key metrics; visualisations configured in line with existing visualisations that teams use, with clear labelling; functionality that supports the creation of reports and presentations; the ability to explore relationships between variables and drill down to look at subgroups; and low requirements for computing resources. Organisations introducing a dashboard may find the following strategies beneficial: clinical champion to promote use; testing with real data by audit staff; establishing routines for integrating use into work practices; involving audit staff in adoption activities; and allowing customisation. Limitations The COVID-19 pandemic stopped phase 4 data collection, limiting our ability to further test and refine the QualDash theory. Questionnaire results should be treated with caution because of the small, possibly biased, sample. Control sites for the interrupted time series analysis were not possible because of research and development delays. One intervention site did not submit data. Limited uptake meant that assessing the impact on more measures was not appropriate. Future work The extent to which national audit dashboards are used and the strategies national audits use to encourage uptake, a realist review of the impact of dashboards, and rigorous evaluations of the impact of dashboards and the effectiveness of adoption strategies should be explored. Study registration This study is registered as ISRCTN18289782. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 12. See the NIHR Journals Library website for further project information.

Background: Blood transfusion is a common but costly treatment. Repeated national audits in the UK suggest that up to one-fifth of transfusions are unnecessary when judged against recommendations for good clinical practice. Audit and feedback seeks to improve patient care and outcomes by comparing clinical care against explicit standards. It is widely used internationally in quality improvement. Audit and feedback generally has modest but variable effects on patient care. A considerable scope exists to improve the impact that audit and feedback has, particularly through head-to-head trials comparing different ways of delivering feedback. Objectives: The AFFINITIE (Development & Evaluation of Audit and Feedback INterventions to Increase evidence-based Transfusion practIcE) programme aimed to design and evaluate enhanced feedback interventions, within a national blood transfusion audit programme, to promote evidence-based guidance and reduce the unnecessary use of blood components. We developed, piloted and refined two feedback interventions, ‘enhanced content’ and ‘enhanced follow-on’ (workstream 1), evaluated the effectiveness and cost-effectiveness of the two feedback interventions compared with standard feedback practice (workstream 2), examined intervention fidelity and contextual influences (workstream 3) and developed general implementation recommendations and tools for other audit and feedback programmes (workstream 4). Design: Interviews, observations and documentary analysis in four purposively sampled hospitals explored contemporary practice and opportunities for strengthening feedback. We developed two interventions: ‘enhanced content’, to improve the clarity and utility of feedback reports, and ‘enhanced follow-on’, to help hospital staff with action-planning (workstream 1). We conducted two linked 2 × 2 factorial cross-sectional cluster-randomised trials within transfusion audits for major surgery and haematological oncology, respectively (workstream 2). We randomised hospital clusters (the organisational level at which hospital transfusion teams operate) to enhanced or standard content or enhanced or standard follow-on. Outcome assessment was masked to assignment. Decision-analytic modelling evaluated the costs, benefits and cost-effectiveness of the feedback interventions in both trials from the perspective of the NHS. A parallel process evaluation used semistructured interviews, documentary analyses and web analytics to assess the fidelity of delivery, receipt and enactment and to identify contextual influences (workstream 3). We explored ways of improving the impact of national audits with their representatives (workstream 4). Setting and participants: All NHS hospital trusts and health boards participating in the National Comparative Audit of Blood Transfusions were invited to take part. Among 189 hospital trusts and health boards screened, 152 hospital clusters participated in the surgical audit. Among 187 hospital trusts and health boards screened, 141 hospital clusters participated in the haematology audit. Interventions: ‘Enhanced content’ aimed to ensure that the content and format of feedback reports were consistent with behaviour change theory and evidence. ‘Enhanced follow-on’ comprised a web-based toolkit and telephone support to facilitate local dissemination, planning and response to feedback. Main outcome measures: Proportions of acceptable transfusions, based on existing evidence and guidance and algorithmically derived from national audit data. Data sources: Trial primary outcomes were derived from manually collected, patient-level audit data. Secondary outcomes included routinely collected data for blood transfusion. Results: With regard to the transfusions in the major surgery audit, 135 (89%) hospital clusters participated from 152 invited. We randomised 69 and 66 clusters to enhanced and standard content, respectively, and 68 and 67 clusters to enhanced and standard follow-on, respectively. We analysed a total of 2222 patient outcomes at 12 months in 54 and 58 (enhanced and standard content, respectively) and 54 and 58 (enhanced and standard follow-on, respectively) hospital clusters. With regard to the haematology audit, 134 hospital clusters (95%) participated from 141 invited. We randomised 66 and 68 clusters to enhanced and standard content, respectively, and 67 clusters to both enhanced and standard follow-on. We analysed a total of 3859 patient outcomes at 12 months in 61 and 61 (enhanced and standard content, respectively) and 63 and 59 (enhanced and standard follow-on) hospital clusters. We found no effect of either of the enhanced feedback interventions in either trial across all outcomes. Incremental enhanced intervention costs ranged from £18 to £248 per site. The enhanced feedback interventions were dominated by the standard intervention in cost-effectiveness analyses. The interventions were delivered as designed and intended, but subsequent local engagement was low. Although the enhancements were generally acceptable, doubts about the credibility of the blood transfusion audits undermined the case for change. Limitations: Limitations included the number of participating clusters; loss to follow-up of trial clusters, reducing statistical power and validity; incomplete audit and cost data contributing to outcome measures; participant self-selection; reporting; missing data related to additional staff activity generated in response to receiving feedback; and recall biases in the process evaluation interviews. Conclusions: The enhanced feedback interventions were acceptable to recipients but were more costly and no more effective than standard feedback in reducing unnecessary use of blood components, and, therefore, should not be recommended on economic grounds. Future work: We have demonstrated the feasibility of embedding ambitious large-scale rigorous research within national audit programmes. Further head-to-head comparisons of different feedback interventions are needed in these programmes to identify cost-effective ways of increasing the impact of the interventions. Trial registration: This trial is registered as ISRCTN15490813. Funding: This project was funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research programme and will be published in full in Programme Grants for Applied Research; Vol. 10, No. 2. See the NIHR Journals Library website for further project information.