Justin Keen

Background: National clinical audit programmes aim to improve patient care by reviewing performance against explicit standards and directing action towards areas not meeting those standards. Their impact can be improved by (1) optimising feedback content and format, (2) strengthening audit cycles and (3) embedding randomised trials evaluating different ways of delivering feedback. Objectives: The objectives were to (1) develop and evaluate the effects of modifications to feedback on recipient responses, (2) identify ways of strengthening feedback cycles for two national audits and (3) explore opportunities, costs and benefits of national audit participation in a programme of trials. Design: An online fractional factorial screening experiment (objective 1) and qualitative interviews (objectives 2 and 3). Setting and participants: Participants were clinicians and managers involved in five national clinical audits – the National Comparative Audit of Blood Transfusions, the Paediatric Intensive Care Audit Network, the Myocardial Ischaemia National Audit Project, the Trauma Audit & Research Network and the National Diabetes Audit – (objective 1); and clinicians, members of the public and researchers (objectives 2 and 3). Interventions: We selected and developed six online feedback modifications through three rounds of user testing. We randomised participants to one of 32 combinations of the following recommended specific actions: comparators reinforcing desired behaviour change; multimodal feedback; minimised extraneous cognitive load for feedback recipients; short, actionable messages followed by optional detail; and incorporating ‘the patient voice’ (objective 1). Main outcome measures: The outcomes were intended actions, including enactment of audit standards (primary outcome), comprehension, user experience and engagement (objective 1). Results: For objective 1, the primary analysis included 638 randomised participants, of whom 566 completed the outcome questionnaire. No modification independently increased intended enactment of audit standards. Minimised cognitive load improved comprehension (+0.1; p = 0.014) and plans to bring audit findings to colleagues’ attention (+0.13, on a –3 to +3 scale; p = 0.016). We observed important cumulative synergistic and antagonistic interactions between modifications, participant role and national audit. The analysis in objective 2 included 19 interviews assessing the Trauma Audit Research Network and the National Diabetes Audit. The identified ways of strengthening audit cycles included making performance data easier to understand and guiding action planning. The analysis in objective 3 identified four conditions for effective collaboration from 31 interviews: compromise – recognising capacity and constraints; logistics – enabling data sharing, audit quality and funding; leadership – engaging local stakeholders; and relationships – agreeing shared priorities and needs. The perceived benefits of collaboration outweighed the risks. Limitations: The online experiment assessed intended enactment as a predictor of actual clinical behaviour. Interviews and surveys were subject to social desirability bias. Conclusions: National audit impacts may be enhanced by strengthening all aspects of feedback cycles, particularly effective feedback, and considering how different ways of reinforcing feedback act together. Future work: Embedded randomised trials evaluating different ways of delivering feedback within national clinical audits are acceptable and may offer efficient, evidence-based and cumulative improvements in outcomes. Trial registration: This trial is registered as ISRCTN41584028. Funding details: This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 15. See the NIHR Journals Library website for further project information.

There is a widespread belief that machine learning tools can be used to improve decision-making in health and social care. At the same time, there are concerns that they pose threats to privacy and confidentiality. Policy makers therefore need to develop governance arrangements that balance benefits and risks associated with the new tools. This article traces the history of developments of information infrastructures for secondary uses of personal datasets, including routine reporting of activity and service planning, in health and social care. The developments provide broad context for a study of the governance implications of new tools for the analysis of health and social care datasets. We find that machine learning tools can increase the capacity to make inferences about the people represented in datasets, although the potential is limited by the poor quality of routine data, and the methods and results are difficult to explain to other stakeholders. We argue that current local governance arrangements are piecemeal, but at the same time reinforce centralisation of the capacity to make inferences about individuals and populations. They do not provide adequate oversight, or accountability to the patients and clients represented in datasets.

Objective Health services in many countries are investing in interorganisational networks, linking patients’ records held in different organisations across a city or region. The aim of the systematic review was to establish how, why and in what circumstances these networks improve patient safety, fail to do so, or increase safety risks, for people living at home. Design Realist synthesis, drawing on both quantitative and qualitative evidence, and including consultation with stakeholders in nominal groups and semistructured interviews. Eligibility criteria The coordination of services for older people living at home, and medicine reconciliation for older patients returning home from hospital. Information sources 17 sources including Medline, Embase, CINAHL, Cochrane Library, Web of Science, ACM Digital Library, and Applied Social Sciences Index and Abstracts. Outcomes Changes in patients’ clinical risks. Results We did not find any detailed accounts of the sequences of events that policymakers and others believe will lead from the deployment of interoperable networks to improved patient safety. We were, though, able to identify a substantial number of theory fragments, and these were used to develop programme theories. There is good evidence that there are problems with the coordination of services in general, and the reconciliation of medication lists in particular, and it indicates that most problems are social and organisational in nature. There is also good evidence that doctors and other professionals find interoperable networks difficult to use. There was limited high-quality evidence about safety-related outcomes associated with the deployment of interoperable networks. Conclusions Empirical evidence does not currently justify claims about the beneficial effects of interoperable networks on patient safety. There appears to be a mismatch between technology-driven assumptions about the effects of networks and the sociotechnical nature of coordination problems. PROSPERO registration number CRD42017073004.

Background: Interoperable networks connect information technology systems of different organisations, allowing professionals in one organisation to access patient data held in another one. Health policy-makers in many countries believe that they will improve the co-ordination of services and, hence, the quality of services and patient safety. To the best of our knowledge, there have not been any previous systematic reviews of the effects of these networks on patient safety. Objectives: The aim of the study was to establish how, why and in what circumstances interoperable information technology networks improved patient safety, failed to do so or increased safety risks. The objectives of the study were to (1) identify programme theories and prioritise theories to review; (2) search systematically for evidence to test the theories; (3) undertake quality appraisal, and use included texts to support, refine or reject programme theories; (4) synthesise the findings; and (5) disseminate the findings to a range of audiences. Design: Realist synthesis, including consultation with stakeholders in nominal groups and semistructured interviews. Settings and participants: Following a stakeholder prioritisation process, several domains were reviewed: older people living at home requiring co-ordinated care, at-risk children living at home and medicines reconciliation services for any patients living at home. The effects of networks on services in health economies were also investigated. Intervention: An interoperable network that linked at least two organisations, including a maximum of one hospital, in a city or region. Outcomes: Increase, reduction or no change in patients’ risks, such as a change in the risk of taking an inappropriate medication. Results: We did not find any detailed accounts of the ways in which interoperable networks are intended to work and improve patient safety. Theory fragments were identified and used to develop programme and mid-range theories. There is good evidence that there are problems with the co-ordination of services in each of the domains studied. The implicit hypothesis about interoperable networks is that they help to solve co-ordination problems, but evidence across the domains showed that professionals found interoperable networks difficult to use. There is insufficient evidence about the effectiveness of interoperable networks to allow us to establish how and why they affect patient safety. Limitations: The lack of evidence about patient-specific measures of effectiveness meant that we were not able to determine ‘what works’, nor any variations in what works, when interoperable networks are deployed and used by health and social care professionals. Conclusions: There is a dearth of evidence about the effects of interoperable networks on patient safety. It is not clear if the networks are associated with safer treatment and care, have no effects or increase clinical risks. Future work: Possible future research includes primary studies of the effectiveness of interoperable networks, of economies of scope and scale and, more generally, on the value of information infrastructures. Study registration: This study is registered as PROSPERO CRD42017073004. Funding: This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 40. See the NIHR Journals Library website for further project information.

Introduction National audits are used to monitor care quality and safety and are anticipated to reduce unexplained variations in quality by stimulating quality improvement (QI). However, variation within and between providers in the extent of engagement with national audits means that the potential for national audit data to inform QI is not being realised. This study will undertake a feasibility evaluation of QualDash, a quality dashboard designed to support clinical teams and managers to explore data from two national audits, the Myocardial Ischaemia National Audit Project (MINAP) and the Paediatric Intensive Care Audit Network (PICANet). Methods and analysis Realist evaluation, which involves building, testing and refining theories of how an intervention works, provides an overall framework for this feasibility study. Realist hypotheses that describe how, in what contexts, and why QualDash is expected to provide benefit will be tested across five hospitals. A controlled interrupted time series analysis, using key MINAP and PICANet measures, will provide preliminary evidence of the impact of QualDash, while ethnographic observations and interviews over 12 months will provide initial insight into contexts and mechanisms that lead to those impacts. Feasibility outcomes include the extent to which MINAP and PICANet data are used, data completeness in the audits, and the extent to which participants perceive QualDash to be useful and express the intention to continue using it after the study period. Ethics and dissemination The study has been approved by the University of Leeds School of Healthcare Research Ethics Committee. Study results will provide an initial understanding of how, in what contexts, and why quality dashboards lead to improvements in care quality. These will be disseminated to academic audiences, study participants, hospital IT departments and national audits. If the results show a trial is feasible, we will disseminate the QualDash software through a stepped wedge cluster randomised trial.

Background: There is a widespread belief that information technologies will improve diagnosis, treatment and care. Evidence about their effectiveness in health care is, however, mixed. It is not clear why this is the case, given the remarkable advances in hardware and software over the last 20 years. This review focuses on interoperable information technologies, which governments are currently advocating and funding. These link organisations across a health economy, with a view to enabling health and care professionals to coordinate their work with one another and to access patient data wherever it is stored. Given the mixed evidence about information technologies in general, and current policies and funding, there is a need to establish the value of investments in this class of system. The aim of this review is to establish how, why and in what circumstances interoperable systems affect patient safety. Methods: A realist synthesis will be undertaken, to understand how and why inter-organisational systems reduce patients’ clinical risks, or fail to do so. The review will follow the steps in most published realist syntheses, including (1) clarifying the scope of the review and identifying candidate programme and mid-range theories to evaluate, (2) searching for evidence, (3) appraising primary studies in terms of their rigour and relevance and extracting evidence, (4) synthesising evidence, (5) identifying recommendations, based on assessment of the extent to which findings can be generalised to other settings. Discussion: The findings of this realist synthesis will shed light on how and why an important class of systems, that span organisations in a health economy, will contribute to changes in patients’ clinical risks. We anticipate that the findings will be generalizable, in two ways. First, a refined mid-range theory will contribute to our understanding of the underlying mechanisms that, for a range of information technologies, lead to changes in clinical practices and hence patients’ risks (or not). Second, many governments are funding and implementing cross-organisational IT networks. The findings can inform policies on their design and implementation. Systematic review registration: PROSPERO CRD42017073004

Policy makers and health system managers in many countries are advocating the deployment of inter-operable health information technology systems, spanning organisations in a health economy, believing that they will be clinically effective. The case for investments has not, however, been made to date. This paper presents early results from a systematic review of the effects of inter-operable systems on patient safety. The review uses the realist synthesis method, which focuses on evidence about the decisions and actions that link interventions and outcomes, as well as the evidence about those outcomes. The evidence base is sufficient to identify plausible arguments for investments in inter-operable systems. This said, there is limited empirical evidence about each of the steps in the sequences of events. We comment on implications for the design of sustainable socio-technical solutions. We suggest that current gaps in the evidence base are in areas where informatics field methods can make a valuable contribution to our understanding of the role of inter-operable systems in patient safety.

Objective To assess developments over time in the capture, curation and use of quality and safety information in managing hospital services. Setting Four acute National Health Service hospitals in England. Participants 111.5 hours of observation of hospital board and directorate meetings, and 72 hours of ward observations. 86 interviews with board level and middle managers and with ward managers and staff. Results There were substantial improvements in the quantity and quality of data produced for boards and middle managers between 2013 and 2016, starting from a low base. All four hospitals deployed data warehouses, repositories where datasets from otherwise disparate departmental systems could be managed. Three of them deployed real-time ward management systems, which were used extensively by nurses and other staff. Conclusions The findings, particularly relating to the deployment of real-time ward management systems, are a corrective to the many negative accounts of information technology implementations. The hospital information infrastructures were elements in a wider move, away from a reliance on individual professionals exercising judgements and towards team-based and data-driven approaches to the active management of risks. They were not, though, using their fine-grained data to develop ultrasafe working practices.

Background In the UK managers from multiple organisations are commonly tasked with collectively devising and implementing local health and wellbeing policies as a way of addressing increasing demand for healthcare. This requires them to create knowledge together but relatively little is known about how this occurs. This paper reports the results of research into how managers collectively create knowledge in order to address local health and wellbeing challenges. Methods We undertook a case study in three sites in England. Using statistical network modelling we identified clusters of actors and interviewed managers from heterogeneous clusters about their collective activities. We used interview and documentary data to construct accounts of collective knowledge creation. Findings Managers simultaneously work across stable bureaucratic networks and temporary taskforces in order to create and implement local health and wellbeing policy. They collectively create knowledge by enacting networks of relationships which enable them to share and build on routines and discourses and to reach out for new evidence, perspectives and skills. When creating knowledge, managers' ability to draw on and harmonise alternative programmes of action and their willingness to collectively negotiate is more important than their managerial status or position. Conclusions Managers should be encouraged to examine and discuss their alternative programmes of action and to see these as a catalyst for rather than barrier to collectively creating and implementing local health and wellbeing policies, and should be supported and valued for their ability to harmonise conflicting programmes of action.

Background: There have been concerns about the quality and safety of NHS hospital services since the turn of the millennium. This study investigated the progress that acute NHS hospital trusts have made in developing and using technology infrastructures to enable them to monitor quality and safety following the publication in 2013 of the second Francis report on the scandal at Mid Staffordshire NHS Foundation Trust (The Mid Staffordshire NHS Foundation Trust Public Inquiry. Chaired by Sir Robert Francis QC.Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry. HC 898. London: The Stationery Office; 2013).Methods: A telephone survey of 15 acute NHS trusts in the Yorkshire and the Humber region, and a review of board papers of all acute NHS trusts in England for January 2015, were undertaken. The telephone survey was used to identify trusts for a larger field study, which was undertaken in four acute NHS trusts between April 2015 and September 2016. The methods included the direct observation of the use of whiteboards and other technologies on two wards in each trust, an observation of board quality committees, semistructured interviews and an analysis of the quality and safety data in board papers. Published sources about national and local agencies were reviewed to identify the trust quality and safety data that these agencies accessed and used. An interview programme was also undertaken with those organisations. The Biography of Artefacts approach was used to analyse the data.Findings: The data and technology infrastructures within trusts had developed over many years. The overall design had been substantially determined by national agencies, and was geared to data processing: capturing and validating data for submission to national agencies. Trust boards had taken advantage of these data and used them to provide assurance about quality and safety. Less positively, the infrastructures were fragmented, with different technologies used to handle different quality and safety data. Real-time management systems on wards, including electronic whiteboards and mobile devices, were used and valued by nurses and other staff. The systems support the proactive management of clinical risks. These developments have occurred within a broad context, with trusts focusing on improving the quality and safety of services and publishing far more data on their performance than they did just 3 years earlier. Trust-level data suggest that quality and safety improved at all four trusts between 2013 and 2016. Our findings indicate that the technology infrastructures contributed to these improvements. There remains considerable scope to rationalise those infrastructures.Limitations: The four trusts in the main study were, in part, purposively selected, and deliberately biased towards sites that had made progress with designing and deploying real-time ward management systems. This limits the generalisability of the study. The study focused more on the work of nurses and nurse managers, and has relatively little to say about the experiences of doctors or allied health professionals.Future work: Future research might focus on the effects of mobile technologies and electronic whiteboards on acute wards, the value of current national data returns, and the uses and value of trust data warehouses.Funding: The National Institute for Health Research Health Services and Delivery Research programme.