AI Adoption Case Study: learn how Holistic AI’s AI governance platform enables enterprises to adopt and scale AI confidently while regularly monitoring risk

Learn how Holistic AI's AI governance platform and five-point criteria.

techUK’s AI adoption collection of case studies showcases examples of how organisations are Putting AI into Action, either through the adoption of AI models within their organisations, or by developing AI tools that can be leveraged by others.

By shining a light on these use cases, techUK hopes to demonstrate examples of best practice from across sectors and from organisations of all sizes.

Holistic AI is an AI Governance platform headquartered in the U.S., with offices in Palo Alto and London. We offer end-to-end solutions to test and validate the safety, robustness, and trustworthiness of algorithms deployed across various sectors.

1. Challenge:

Holistic AI’s AI governance platform enables enterprises to adopt and scale AI confidently while regularly monitoring risk. Through a Software as a Service (SaaS) platform, the company conducts independent evaluations of Enterprise AI systems and offers solutions on risk management, audits, and regulatory compliance.

Each AI system is evaluated against a five-point criteria derived from Koshiyama et al (2021):

Bias: Risks of the AI system generating biased outputs due to improper training data (training bias), inappropriate context (transfer-context bias) and inadequate inferential capabilities (inference bias).
Efficacy: Risk of the AI system underperforming relative to its use case.
Robustness: Risks of the AI system failing in instances of adversarial attacks.
Privacy: Risks associated with the AI system leaking sensitive information or personal data.
Explainability: Risks of the AI system generating arbitrary decisions, with its outputs not understandable to developers, deployers and users.

Each of these risk verticals do not occur in siloes and can often be interrelated. A growing field of research strongly emphasizes the trade-offs and interactions that may occur between them. The Holistic AI Governance platform not only allows enterprises to review the performance of its AI systems against the criteria, but also streamlines the decision-making process around navigating trade-offs.

Holistic AI provides a proprietary solution for auditing Language Models through Safeguard, a specialized module dedicated to LM Governance. When specifically evaluating Large Language Models (LLMs), the Holistic AI team uses a robust combination of the following approaches:

Benchmarking: Involves evaluating LLMs against both, academic and internally developed datasets to gauge levels of model bias, hallucinations, personal information leakage, toxicity, explainability and robustness.
Red Teaming: Involves adversarially prompting LLMs to unearth unknown model vulnerabilities through mechanisms like Jailbreaking and debiasing.
Fine Tuning: Involves leveraging high-quality datasets to align models towards safety, helpfulness and harm reduction.
Human oversight: This involves assessment of LLM-generated content by reviewers for its relevance, accuracy, and appropriateness, to identify any discrepancies that need improvement.
Assurance: Benchmarking risk discovery, triage, assessment and mitigation processes to regulation (like the EU AI Act), and standards (such as the NIST’s AI Risk Management Framework (AI RMF) to aid with compliance readiness and assurance.

Through such techniques, enterprises can be confident that their models function as intended, are free from biases, and robust enough to handle real-world data and interactions ethically and effectively.

Enterprises have found success through the continual use of the platform, which allows them to monitor the progress of an AI system’s performance. The approach echoes evaluations from established industries, such as the financial and transportation sectors, which institutionalize regular safety checks on vital systems.

2. Barriers:

The emerging field of AI governance faces several challenges:
Pace of technological development: AI is rapidly developing and the laws, regulations, and standards that govern today’s technology may not be effective in the future. Whether or not the entity is a government or a corporation, established governance practices may be ill-equipped to manage quickly evolving technological change. The Holistic AI governance platform provides a convenient way for enterprises to monitor these changes, but the ability for companies to adapt to them remains a challenge.
Multiple regulatory fronts: Governments and regulators around the world have shown great interest in the possibility of regulating AI and several different approaches have consequently emerged. It is unlikely enterprises will be subject to a singular approach and as a result, AI governance cannot be a ‘one size fits all’ approach. Holistic AI’s governance platform includes a regulation tracker that enterprises can use to stay up to date with these developments. Nonetheless, the burden of compliance is variable and therefore a potential challenge.
Lack of universal standards and measures: The AI assurance and auditing ecosystem currently lacks standardization, which leads to inconsistency in approaches. Institutional support on this topic varies on jurisdiction, which also causes some uncertainty that limits the establishment and execution of such mechanisms.

3. Impact:

Holistic AI has assured over 100 enterprise AI projects that cover more than 20,000 different algorithms. The company has equipped Fortune 500 corporations, SMEs, governments and regulators to safely and responsibly deploy and scale AI systems across a range of applications. Overall, those who have employed the company’s AI governance platform have seen better visibility over AI deployments, reduced risk, better performance and increased trust.

As a result of the sociotechnical approach that many firms now take, AI is not simply viewed as a technological issue but an interdisciplinary one. Holistic AI is proud to have pioneered its uptake among enterprises as an early proponent of such an approach.

Usman Ikhlaq

Programme Manager - Artificial Intelligence, techUK

Usman joined techUK in January 2024 as Programme Manager for Artificial Intelligence.

His role is to help techUK members of all sizes and across all sectors to adopt AI at scale. This includes identifying the barriers to adoption, considering solutions and how best to maximise AI's potential.

Prior to joining techUK, Usman worked as a policy, government affairs and public affairs professional in the advertising sector. He has also worked in sales and marketing and FinTech.

Usman is a graduate of the London School of Economics and Political Science (MSc), BPP Law School (GDL and LLB) and Queen Mary University of London (BA).

When he isn’t working, Usman enjoys spending time with his family and friends. He also has a keen interest in running, reading and travelling.

Email:: [email protected]
LinkedIn:: https://uk.linkedin.com/in/usman-ikhlaq,https://uk.linkedin.com/in/usman-ikhlaq

Read lessmore

techUK - Seizing the AI Opportunity

The UK is a global leader in AI innovation, development and adoption. The economic growth and productivity gain that AI can unlock is vast, but to fully harness this transformative opportunity, immediate action is required. techUK and our members are committed to working with the Government to turn the AI Opportunities Action Plan into reality. Our aim is to ensure the UK seizes the opportunities presented by AI technology and continues to be a world leader in AI development.

techUK runs a full calendar of activities including events, reports, and insights to demonstrate some of the most significant AI opportunities for the UK. Our AI Hub is where you will find details of all upcoming activity. We also send a monthly AI newsletter which you can subscribe to here.