Algorithmic Opacity, Clinical Justification, and the Duty of Candour in AI-Assisted Care

Source: PubMed Central Open Access, NCBI / U.S. National Library of Medicine

Artificial intelligence (AI) systems are becoming increasingly embedded within NHS clinical systems, including areas such as emergency department triage, diagnostic support, and risk stratification. Many high-performing systems rely on complex machine learning architectures that are difficult or even impossible for clinicians to interpret at the point of care. This paper aims to examine the ethical implications of deploying uninterpretable AI systems in high-risk clinical environments. Utilising a constructed, however clinically plausible, vignette of sepsis misclassification, we argue that algorithmic opacity may undermine clinicians' ability to satisfy justificatory obligations within the professional and legal framework of the UK Duty of Candour. We distinguish between ordinary clinical uncertainty and structural opacity arising from system design, arguing that the latter generates a distinctive ethical problem by weakening traceability, accountability, and meaningful explanation after patient harm. We further argue that interpretability should be treated as a safety-relevant ethical requirement in high-risk clinical contexts, rather than as an optional technical feature subordinate to predictive performance alone.