This document defines the explainability, safety, and governance framework for all AI systems developed and deployed by the organisation. It establishes the principles and requirements for ensuring that AI-driven outputs are transparent, understandable, and appropriately governed throughout their lifecycle.
The purpose of this framework is to:
This policy applies to all artificial intelligence (AI) and machine learning (ML) systems that are developed, deployed, integrated, or embedded within the organisation's SaaS platform. It covers any system that produces automated or semi-automated outputs that may influence clinical, operational, or administrative decisions.
All system access is restricted to authenticated users via secure access mechanisms. Role-based access control (RBAC) is enforced across the platform to ensure users can only access data and perform actions aligned with their assigned permissions.
All integrations, data connections, and external system interfaces must be reviewed and approved by the IT or security function prior to activation.
The AI system does not expand, infer, or expose access to data beyond what the user is already authorised to view in the underlying source systems. It operates strictly within existing permission boundaries.
The system processes only the minimum amount of data necessary to fulfil a user's request or query.
Data retrieval is scoped to explicitly permitted data sources and user-authorised records. No additional, unrelated, or excessive personal data is accessed or processed.
Data within the system is regularly synchronised with source systems to ensure accuracy and consistency.
Where data is updated or deleted in the source system, these changes are reflected in the AI system in line with applicable data retention and deletion policies. This ensures compliance with UK GDPR principles of accuracy and storage limitation.
All data is encrypted in transit and at rest using industry-standard encryption protocols.
Sensitive information, including credentials and authentication secrets, is securely stored using appropriate encryption and secret management practices.
The system is hosted within a secured, firewall-protected environment with strict access controls and monitored infrastructure security measures.
The system is built and tested using a combination of synthetic datasets and real-world datasets containing thousands of representative examples.
Test scenarios are designed in collaboration with care professionals to ensure they reflect real clinical and operational contexts, and to support fairness, safety, and domain relevance.
The system is designed to produce outputs grounded in source data through a context engineering approach that ensures responses remain anchored to relevant and verifiable information.
Outputs are generated using retrieved or referenced data and are not intended to express subjective opinion or unsupported assertions. To reduce the risk of hallucination, distortion, and bias, responses are consistently linked to underlying data sources and citations where applicable.
Fairness, bias, and representativeness are assessed through defined evaluation methodologies, including subgroup performance testing across protected characteristics under the Equality Act 2010, and monitoring for disparities in output quality and accuracy. These evaluations are conducted as part of the system's ongoing validation and governance process, with results reviewed at release and during periodic audits.
Care professionals are involved throughout the system lifecycle, including the design of test cases and the review of system outputs.
Where conflicting or inconsistent information is detected, the system clearly highlights discrepancies and presents relevant source data transparently.
The system is designed to support decision-making, not replace it. Final interpretation and decisions remain with qualified human users.
The system is validated through a structured testing approach, including unit tests and scenario-based evaluations using real-world care and operational examples.
All test scenarios are defined and reviewed in collaboration with care professionals to ensure clinical relevance, safety, and practical applicability.
All changes, updates, or new features must pass at least 90% of the defined test scenarios before being released into production.
No deployment is permitted unless all validation and evaluation tests have been fully completed and formally approved.
This requirement is enforced by the Engineering Team under the oversight of the CTO, who holds ultimate responsibility for release readiness. Where evaluations fail, the AI Engineering Team is required to remediate and update the codebase until all tests are successfully passed.
Any identified inaccuracies, defects, or unexpected behaviours are formally logged, tracked, and prioritised for resolution.
Relevant cases are added to the ongoing evaluation and regression test suite to prevent recurrence.
All previously identified issues must pass regression testing before subsequent releases are approved.
The system is continuously monitored to ensure performance, reliability, and issue detection.
Monitoring and incident management are supported through:
All incidents are reviewed, categorised, and used to inform ongoing system improvements.
All AI-generated outputs are produced with the underlying retrieval context attached, enabling transparency and explainability. This includes visibility of the source data used to generate the response, ensuring outputs can be traced back to their originating context and supporting information.
Human oversight is embedded within the system, particularly for high-risk or sensitive use cases.
Users are able to:
The system is designed to support, not replace, human judgement.
The system is designed with capabilities intended to support the exercise of key data subject rights under UK GDPR, including access, rectification, erasure, and restriction of processing, subject to applicable legal, technical, and organisational constraints.
These controls ensure that AI functionality remains aligned with data protection obligations.
The system is built using a Retrieval-Augmented Generation (RAG) architecture. This approach combines structured data retrieval with controlled AI generation to ensure outputs are accurate, traceable, and grounded in source data.
When data is ingested into the system, it is broken down into smaller, structured chunks to enable efficient retrieval.
Data is stored in a format optimised for search and relevance ranking, while preserving the original meaning, context, and structure of the source material.
When a user submits a query, the system analyses and, where necessary, decomposes it into smaller sub-queries.
Only data sources that the user is authorised to access are searched. Retrieved results are then ranked, filtered, and prioritised based on relevance to the query.
Responses are generated using a constrained, low-temperature AI model designed to prioritise accuracy over fluency.
The system:
All responses are grounded in retrieved content and include clear citations to the underlying data.
Explainability is built into the system by default. All AI-generated outputs are designed to be transparent and traceable, with responses including citations or references to the underlying source data used to generate the answer. This ensures users can verify outputs against their originating context and supporting evidence.
All responses are:
The system prioritises deterministic, evidence-based outputs over open-ended or generative responses, ensuring consistency, auditability, and user trust.
The AI system is designed with:
These measures ensure alignment with key UK GDPR principles, including: