Meredith Broussard

A high-level précis of the Synthesis Report can be found in the Summary for Policymakers Recommendations for a Global AI Auditing Framework: Summary of Standards and Features. The growing integration of artificial intelligence (AI) into critical sectors of society, from healthcare to education, has the potential to support widespread social transformation and progress. However, AI systems also have the power to perpetuate biases, deepen inequalities, and cause environmental harm. Accurately evaluating the risks and benefits of an AI system requires a careful audit. Current approaches to auditing, however, rarely involve independent auditors, provide sufficient evidence, or account for global impacts. Policymakers urgently need a comprehensive global framework for AI audits that validates genuine benefits and risks. The IPIE’s Scientific Panel on Global Standards for AI Audits set out to independently establish global, cross-disciplinary scientific consensus on what makes an audit effective and trustworthy. The Panel consisted of 16 experts from computer science, the social sciences, and the humanities, with expertise spanning generative AI systems, algorithmic auditing, indigenous data sovereignty, data journalism, and the relationship between civil society, human rights, and AI.

This Summary for Policymakers provides a high-level précis of the Synthesis Report Towards A Global AI Auditing Framework: Assessment and Recommendations. The growing integration of artificial intelligence (AI) into critical sectors of society, from healthcare to education, has the potential to support widespread social transformation and progress. However, AI systems also have the power to perpetuate biases, deepen inequalities, and cause environmental harm. Accurately evaluating the risks and benefits of an AI system requires a careful audit. Current approaches to auditing, however, rarely involve independent auditors, provide sufficient evidence, or account for global impacts. Policymakers urgently need a comprehensive global framework for AI audits that validates genuine benefits and risks. The IPIE’s Scientific Panel on Global Standards for AI Audits set out to independently establish global, cross-disciplinary scientific consensus on what makes an audit effective and trustworthy. The Panel consisted of 16 experts from computer science, the social sciences, and the humanities, with expertise spanning generative AI systems, algorithmic auditing, indigenous data sovereignty, data journalism, and the relationship between civil society, human rights, and AI.

This Synthesis Report is a literature review outlining the regulatory, industry, and academic approaches to AI audits. We review 78 articles published in peer-reviewed journals and as preprints, 21 documents from industry associations and standard-setting organizations, and national policy documents and regulations from 20 countries. Based on this review, we identify three key takeaways about the landscape of AI auditing: 1. To accurately assess the potential risks, and impacts of AI systems, we need a trustworthy audit ecosystem with complementary approaches from internal, external, and community auditors. 2. Auditors need better access to data and audit artifacts from the developers and deployers of AI systems. Comprehensive auditability requires documentation and disclosure of an AI system’s model and data components, associated risks and impacts, and easily understandable explanations of its outcomes. 3. Given that the development and use of AI systems impacts communities across the world, audit regimes must account for their global effects. Most existing audits have been conducted in North America, Europe, and other regions of the ‘global north’, with their results typically published in English and focused on effects within these regions. The impacts of AI systems, however, include shifts in social and environmental conditions beyond the immediate development or application contexts of a system.

Important works of data journalism are disappearing from the web because they are too technologically complex to be captured or archived by libraries or web archiving technologies. Research based on journalism depends on the existence of news archives. For the benefit of future scholars, it is imperative that libraries and newsrooms solve this problem. This research contends that dynamic web archiving of data journalism will require a new, emulation-based approach to capturing these works. This new approach in turn necessitates new web archiving tools and workflows to enable collaborative collection of the projects, because unlike in print-based archiving, the process will depend on detailed technical information sharing among stakeholders. Toward this end, this article summarizes the results of a questionnaire that described the most common frameworks, database technologies, and programming languages used to build 76 complex works of data journalism published between 2008 and 2017, as well the ways these works are being maintained and stored. This information can inform the development of emulation-based archiving tools to capture and preserve these stories using methods that would fit within the workflow of news organizations. This research is a first step toward devising an automated solution for long-term preservation of data journalism projects.

Born-digital news content is increasingly becoming the format of the first draft of history. Archiving and preserving this history is of paramount importance to the future of scholarly research, but many technical, legal, financial, and logistical challenges stand in the way of these efforts. This is especially true for news applications, or custom-built websites that comprise some of the most sophisticated journalism stories today, such as the “Dollars for Docs” project by ProPublica. Many news applications are standalone pieces of software that query a database, and this significant subset of apps cannot be archived in the same way as text-based news stories, or fully captured by web archiving tools such as Archive-It. As such, they are currently disappearing. This paper will outline the various challenges facing the archiving and preservation of born-digital news applications, as well as outline suggestions for how to approach this important work.

Currently the digital archives of newspapers are not archiving news apps, the interactive database-driven, multimedia projects. Because of the multiple elements required to access a news app, conversion of the dynamic news app into static HTML pages is one possible avenue for future archiving.

Applied research in computational journalism offers opportunities for scholars to engage with data and platforms by developing original code-based scholarship and research methods. However, scholars who write code need a place to publish about it. This paper suggests new article forms and academic publishing conventions that will allow the field of computational journalism to expand. It explores logistical details associated with sharing and publishing code, and describes reproducible research methods in use in other disciplines.