FDA QMSR Design Controls for Radiology AI: From Dataset to Clinical Validation

For radiology AI, clinical validation starts before a final performance report. It begins when the team defines the clinical task, selects its data and decides what evidence would justify the claimed use. A large dataset cannot compensate for an unsuitable reference standard, patient leakage or a study that does not represent the intended workflow.

This practical FDA QMSR design-controls guide follows the evidence from dataset planning to release. It complements a general software-quality process while concentrating on the decisions that make Imaging AI validation reproducible, interpretable and inspection-ready.

Define design controls and the clinical claim

Design controls are the planned activities and records used to translate user needs into a device design and demonstrate that the design meets specified requirements and intended uses. Under QMSR, applicable manufacturers must address design and development within the quality system; FDA’s QMSR overview explains the framework effective February 2, 2026.

For a radiology AI product, write the clinical claim before choosing a benchmark. Specify detection, characterization, prioritization or another task; identify users, modality, anatomy and patient population. Explain whether the output assists a reader, changes a worklist or supports a diagnostic decision. Those distinctions affect endpoints, workflow testing and whether an observer study is appropriate.

The QMSR implementation guide covers broader quality-system preparation. Here, the central question is narrower: can every validation conclusion be traced to the intended use and the data that support it?

Establish dataset provenance and reference standards

Maintain a dataset inventory describing source institutions, collection periods, inclusion criteria, imaging protocols, permissions and handling controls. Document annotation procedures and the qualifications of annotators. Record how disagreements are adjudicated and how uncertain or incomplete cases are handled.

A reference standard must be appropriate to the clinical task. Depending on the claim, it may involve expert review, pathology, follow-up or another justified method. Do not call routine labels “ground truth” without assessing their limitations. Keep model output from inappropriately influencing reference labeling.

FDA’s Good Machine Learning Practice resource points to principles addressing representative data, independent testing and clinically relevant conditions. These principles inform development practice; they do not prescribe one universal sample size or replace device-specific recommendations.

Prevent leakage before selecting the final model

Separate training, tuning and final testing through documented rules. Patient-level separation matters because multiple studies, series or follow-up scans from the same person can create hidden overlap. Examine institutional overlap, duplicate images and preprocessing steps that may transfer information across partitions.

A held-out dataset is not independent if teams repeatedly consult its results to choose models or thresholds. Lock the model and analysis plan before final evaluation, restrict access appropriately and document any changes. If the final test set has been used for development decisions, reassess whether new independent evidence is needed.

External-site testing may strengthen evidence of generalization, but “external” is not a guarantee of representativeness. Record similarities and differences between development sites, evaluation sites and the intended clinical environment.

Use a dataset-to-validation evidence matrix

Assign an accountable owner and acceptance rationale to each gate. Link records by identifiers instead of copying summaries across files. A reviewer should be able to retrieve the exact dataset, model configuration and approved analysis associated with a reported result.

Distinguish three different evidence questions

Software verification tests whether the product implements specified requirements: loading images, preprocessing, routing outputs and responding safely to errors. Analytical performance assesses the algorithm’s behavior on a defined dataset against an appropriate reference. Clinical validation addresses whether performance supports the intended clinical use.

For applicable computer-assisted detection devices, FDA’s September 2022 final CADe clinical performance guidance discusses standalone performance and clinical observer studies. Its scope is CADe devices in radiology 510(k) submissions; do not automatically apply every recommendation to all radiology AI, diagnostic or triage products.

Choose metrics that fit the task, such as sensitivity with false-positive measures for appropriate detection applications. Report uncertainty and justify the study design. A strong standalone result may not establish the effect on reader performance or workflow. Neither retrospective nor prospective evidence is universally sufficient for every claim; resolve important design questions through appropriate FDA engagement.

Evaluate variability and clinically relevant subgroups

Prespecify analyses for relevant age groups, disease presentations, acquisition settings and other characteristics justified by the intended population. Consider scanner vendors, reconstruction methods, slice thickness and protocol variation where they affect the device. Inspect exclusions and missing data rather than reporting only the easiest evaluable cases.

A high pooled result can hide weakness in a small subgroup. Report subgroup denominators and uncertainty, acknowledge underpowered analyses and document follow-up evidence needs. Do not invent minimum subgroup thresholds or imply that one fairness metric proves clinical suitability.

Review the human-device interaction: alert presentation, unavailable results, false reassurance, excessive findings and user interpretation. Validate relevant interfaces and clearly state unsupported conditions in labeling. Keep limitations aligned across the validation report, user information and commercial claims.

Maintain the evidence after a change

Dataset expansion, retraining and threshold changes can alter the original validation basis. Review affected requirements and hazards, compare relevant performance and update evidence before release. Monitoring should define responsibilities for complaints, drift signals and escalation without assuming that every observed difference proves a defect.

An authorized PCCP may support specified modifications under its approved conditions; our PCCP guide helps organize the assessment. For a controlled remediation sequence, use the 90-day QMSR roadmap. Neither resource substitutes for device-specific regulatory decisions.

Validation readiness checklist and download

• The claim, population and clinical role are approved.
• Data provenance and reference-standard limitations are documented.
• Partition independence and leakage checks are complete.
• Model, thresholds and analysis plan were locked before final testing.
• Relevant variability, subgroups and uncertainty are reported.
• Deviations, labeling limitations and release approvals are traceable.

Download the Radiology AI Dataset and Clinical Validation Evidence Matrix.xlsx. Record dataset references, partition checks, endpoints, subgroup analyses and unresolved actions without entering patient-level data. This editable preparation aid is not an FDA form, clinical protocol or certification tool.

Make validation evidence easier to review

Explore MedTech Copilot to connect dataset decisions, risk controls, V&V records and regulatory evidence in a structured workflow. Qualified clinical, statistical and quality reviewers remain responsible for study adequacy and conclusions. Request an Imaging AI demonstration to discuss a non-confidential evidence-organization example.

FDA primary sources

• Quality Management System Regulation: applicable quality-system framework.
• Clinical performance assessment for radiology CADe devices: September 2022 final guidance.
• Good Machine Learning Practice: FDA resource linking the final IMDRF principles.
• Device software submission documentation: June 2023 final guidance.

Confirm current source versions and applicability to the specific clinical claim; implementation examples are not universal FDA requirements.