Research Analysis
Empowering Participatory AI Audits for a Responsible Future
Our research demonstrates the critical role of non-expert stakeholders in auditing predictive AI. By enabling end-users and decision subjects to participate, we uncover deeper insights and drive more accountable AI development.
Key Impact Metrics from Participatory Auditing
Stakeholders identified crucial areas where their involvement significantly enhanced audit outcomes.
0
Domain Experts Identified
0
New Harms Uncovered
0
Early Intervention Points
0
Prototypes Tested
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Involvement
Impact Discovery
Tool Support
Stakeholder Involvement in AI Auditing
Our findings underscore the value of involving diverse stakeholders, including end-users and decision subjects without AI expertise, in the AI auditing process. This approach moves beyond internal, technical audits to a more comprehensive, external assessment, highlighting the need for varied perspectives to uncover blind spots and ensure equitable outcomes. The most frequently identified auditors included domain experts (N=26), general public (N=2), and internal/independent AI experts (N=8/N=3). Participants emphasized auditing across the entire AI development pipeline, not just at deployment, with a strong focus on data preparation and ethical considerations.
Uncovering AI Impacts: Beyond Harms
Participants were keen to identify both positive and negative impacts of AI applications. While traditional auditing often focuses on harms, our co-design workshops revealed a strong desire to also document 'Positive Impact on Care' (N=6). Common negative concerns included 'Misuse' (N=4) of AI outputs, 'Negative Impact on Care' (N=3) due to overreliance, and 'Societal Biases' (N=3). Participants also identified 'Application Limitations' (N=3) and 'Privacy' (N=2) issues. The use of taxonomies (like Weidinger et al. [62]) was found to be a useful 'thinking tool' but not restrictive, allowing for expansion and classification of positive events.
Designing Tools for Participatory Auditing
The research highlights critical requirements for designing effective participatory AI auditing tools. Such tools must provide AI application information in an accessible format for non-experts, support the assessment of both positive and negative impacts, and offer taxonomies as flexible thinking aids rather than rigid labels. A key challenge identified was participants' difficulty in defining metrics to measure impacts, underscoring the need for structured, step-by-step guidance in metric creation. Visual components for performance analysis were also requested, along with mechanisms for recording audit outcomes, notes, and feedback, including a 'Conditional' outcome option for situations with insufficient evidence.
26
Domain Experts Requested as Auditors
Participatory AI Auditing Process Flow
Impact Discovery
→
Standards Identification
→
Performance Analysis
→
Audit Communication & Advocacy
| Aspect | Traditional Auditing | Participatory Auditing |
|---|---|---|
| Auditors |
|
|
| Scope |
|
|
| Timing |
|
|
| Focus |
|
|
Case Study: Health AI Applications
Our co-design workshops utilized two health-related predictive AI applications: SPARRA (Scottish Patients at Risk of Re-Admission and Admission) and SAM (School Attachment Monitor). Participants identified a strong desire to audit these systems across the entire development lifecycle. For SPARRA, patients highlighted concerns about 'Inaccurate scores for patients with low resource access' and 'Exploitation of data by unauthorised third parties'. For SAM, teachers and parents raised issues like 'Stigmatisation of families' due to labeling and 'Overreliance on results without context'. These real-world examples demonstrated the value of non-expert insights in uncovering impacts often overlooked by AI experts, emphasizing the need for tools to support their auditing process.
9
Co-design Workshops Conducted
Assess Your AI Audit Readiness & ROI
Estimate the potential return on investment and resource savings from implementing a comprehensive participatory AI auditing framework.
Roadmap to Responsible AI Auditing
A phased approach to integrating participatory AI auditing into your enterprise.
Phase 1: Discovery & Scoping
Define AI applications, identify key stakeholders, and understand current audit practices. Establish initial impact assessment criteria with participatory input.
Phase 2: Tool & Process Customization
Adapt or develop participatory auditing tools based on stakeholder needs. Design bespoke metric creation and impact measurement processes.
Phase 3: Pilot & Feedback Integration
Run pilot audits with diverse stakeholder groups. Collect feedback on tools and processes, iterate and refine for optimal usability and effectiveness.
Phase 4: Scaled Rollout & Continuous Improvement
Implement participatory auditing across relevant AI applications. Establish continuous monitoring and improvement loops for sustained responsible AI.
Ready to Transform Your AI Accountability?
Join leading enterprises in building a transparent and responsible AI future with our expert-led participatory auditing strategies.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat