The AI Governance Checklist for Clinics and Health Tech

Most healthcare teams I talk to know they should be governing their AI. What stops them isn't willingness — it's not knowing where to start, and a quiet fear that "doing it properly" means a compliance project they don't have time for.

So here's the reframe: you don't need to boil the ocean. You need to start with the inventory. Almost everything else in AI governance depends on first knowing what AI you actually have. This checklist is organised so you can work top to bottom, and even the first section leaves you materially better off than most of your peers.

Work through it in order. If you only get through the first two sections this quarter, that's still a real result.

How do I know what AI I actually have?

You can't govern AI you haven't listed, so the inventory always comes first. AI tools creep into a business quietly — a scheduling automation here, a note-taker there, a CRM feature switched on by a vendor. The point of this section is to make the invisible visible.

• List every AI tool your organisation uses, including ones bundled inside other software
• For each tool, record what personal or patient data goes in
• Record what each tool produces — a score, a recommendation, a summary, a decision
• Note whether the tool makes or merely influences decisions about patients or staff
• Flag which tools touch clinical decisions, triage, or access to care
• Note where each tool sends data, especially anything leaving the UK or EU
• Keep it simple — a spreadsheet is completely fine to start

Who owns each AI system?

Every AI system needs a named owner, because "the tech team" is not accountability. When something goes wrong or a regulator asks a question, you need one person who can answer for each system.

• Assign a single named owner to every tool in your inventory
• Define who signs off before a new AI tool is adopted
• Name who's accountable if an AI system produces a harmful or wrong output
• Decide who reviews AI decisions that affect patients, and how often
• Make sure clinical leadership is involved in owning clinically-facing AI, not just IT
• Write down these responsibilities somewhere findable, not just in someone's head

How do I assess AI risk and impact?

Assess your highest-stakes systems first — the ones that touch clinical decisions or patient data. Risk assessment asks what could go wrong; impact assessment asks who could be harmed and how. In healthcare, the two together are the heart of responsible AI.

• Identify, for each significant tool, what could go wrong and who could be affected
• Prioritise assessment of tools involved in diagnosis, triage or access to care
• Check training and input data for gaps or bias that could disadvantage patient groups
• Complete a Data Protection Impact Assessment where AI processes patient data — this is mandatory, not optional, for high-risk health processing
• Assess whether any tool might qualify as, or sit inside, a medical device
• Reassess when a tool is updated, repurposed, or fed new kinds of data

For a fuller explanation of why healthcare AI usually counts as high-risk under the EU AI Act, see my post on what the EU AI Act means for digital health.

How do I document and control my AI?

Governance you can't evidence doesn't count — so document what you do and control who can change it. This is the section that turns good intentions into something an auditor, buyer or regulator will accept.

• Keep technical documentation for each significant AI system: what it does, how it was built, how it's meant to be used
• Maintain a written AI policy stating how your organisation uses AI and what it won't do
• Put a Data Processing Agreement in place with every AI vendor that handles personal data
• Disclose AI use in your Privacy Notice — name the tools and explain what they do with people's data
• Explain, for automated decisions, how individuals can request human review
• Control who can change AI configurations or switch new features on
• Version your documentation so you can show what was in place and when

If your Privacy Notice hasn't been updated for AI, that overlaps directly with UK data protection duties — I've covered that in your GDPR setup is probably out of date.

How do I keep humans in the loop and monitor performance?

Governance is not a document you write once — deployed AI has to be watched. Models drift, data changes, and a tool that looked accurate at launch can quietly degrade. Human oversight and monitoring are what keep it honest.

• Ensure a human meaningfully reviews AI outputs that affect patients, not a rubber-stamp glance
• Design oversight so it survives time pressure — the busiest moments are when automation bias creeps in
• Monitor accuracy and performance of deployed tools on a regular schedule
• Set thresholds that trigger review — for example, unexpected changes in outputs or error rates
• Capture and act on incidents where an AI tool got something wrong
• Feed what you learn back into your risk assessments and documentation

How do I prepare for scrutiny?

Prepare as though a buyer, auditor or regulator will ask — because increasingly, they will. Procurement questionnaires, investor due diligence and conformity assessments all probe how your AI is governed. Being ready turns those moments from fire drills into formalities, and it's the point of most healthcare AI governance consulting engagements.

• Keep your inventory, policies and assessments current and in one place
• Be able to answer, for any system, "who owns this and how is it governed?" in minutes
• Map your governance to ISO/IEC 42001 so it's recognisable to buyers and auditors
• Decide whether you need to align now and certify later, or certify sooner for a specific deal
• Keep evidence of human oversight and monitoring, not just of intentions
• Review the whole checklist periodically as your AI, your data and the rules all move

If you want the background on the standard itself and why aligning early is the cheaper route, see ISO 42001 explained for healthcare founders.

Start with the inventory

If this list feels like a lot, remember the core message: you don't need all of it done this week. You need the inventory started, an owner for each tool, and honest attention on your highest-risk systems. Everything else builds from there.

That's exactly the sequence I use in my AI implementation and governance work with clinics and health tech teams — begin with what you have, govern the riskiest things first, and formalise toward certification when it counts. Start small, start now, and you'll be further ahead than almost everyone else in the room.