Guiding Principles for conducting Clinical Trial for Machine Learning-enabled Medical Devices (MLMD) in South Korea (MFDS–HSA)

MFDS (Korea) and HSA (Singapore) jointly released guiding principles (first release: December 11, 2024) to address MLMD-specific challenges in clinical studies and to support efficient market entry while maintaining rigorous expectations for safety and effectiveness. ([MFDS][1]) This article summarizes the key points for sponsors planning clinical trials involving machine learning-enabled medical devices.

Primary reference (official PDF): Guiding Principles for conducting Clinical Trial for Machine Learning-enabled MDs (MFDS–HSA)

1) Scope and key definitions

What is an MLMD?

The guidance uses the term Machine Learning-enabled Medical Device (MLMD) for a medical device that uses machine learning (in part or in whole) to achieve its intended medical purpose, referencing IMDRF key terminology. ([imdrf.org][2])

Compliance baseline still applies

Beyond the principles in the document, sponsors should ensure clinical trials comply with applicable local laws and regulations for medical research, human subjects, and data protection to safeguard participants and protect data integrity and privacy.

2) Clinical trial design: align to intended use and clinical workflow

The guidance emphasizes that trial design determines validity, reliability, and ethical conduct, and it highlights standard core elements:

• Trial configuration (e.g., single-arm, parallel, crossover)
• Statistical hypothesis
• Study population characteristics
• Randomization and blinding strategies
• Control groups
• Primary and secondary endpoints
• Sample size calculation
• Statistical analysis plan

Prospective vs retrospective studies

The document notes that retrospective trials (e.g., using existing datasets) may consider parallel or crossover designs depending on objectives, but retrospective designs typically cannot evaluate factors like usability and unintended consequences within the clinical workflow; additional studies (e.g., usability studies) may be needed to address those limitations.

Performance objective matters

When designing a trial, sponsors should consider the objective and product attributes, including the approach to demonstrating safety and performance/effectiveness (e.g., superiority, equivalence, non-inferiority).

3) Patient and test dataset selection: representativeness, independence, and bias control

Representative study population or testing dataset

The guidance stresses that participants (or test datasets) should be representative of the intended patient population. Clear inclusion/exclusion criteria should reflect intended use/indication and cover factors such as target population, disease groups, disease frequency, gender, and other relevant variables.

Independence from training data

When using retrospective or prospective test datasets, the test data should remain independent from the training datasets used in development.

Sample size and statistical robustness

Adequate sample size and appropriate statistical methods are important for both prospective and retrospective trials, considering disease, purpose, endpoints, statistical power, and other relevant considerations.

Minimizing unwanted bias

The guidance highlights randomization and blinding to reduce unwanted bias in assignment and outcome assessment (including in retrospective and prospective designs).

4) Clinical reference standard and interpretation of clinical data

Choosing a reference standard

A reference standard is an objectively determined benchmark used as an expected result for comparison and assessment. ([imdrf.org][2]) Typically, it should be selected based on established clinical guidelines. Where guidelines are unclear due to a novel use case, the guidance suggests clinical experts (with relevant domain experts) may establish a reference guideline; additional studies may be required to establish novel clinical associations, consistent with broader SaMD clinical evaluation concepts. ([imdrf.org][3])

Handling disagreement among clinical experts

Disagreements in interpreting complex or ambiguous clinical data should be addressed systematically and transparently, documenting:

• areas of disagreement,
• resolution process (e.g., panel review, adjudication, data-driven consensus),
• rationale for final consensus.

Independence to reduce bias

To reduce unwanted bias, the guidance advises that experts determining reference standards should be independent from the clinical investigator.

5) Primary endpoint and results analysis: predefine, justify, and substantiate

The primary endpoint is the main outcome to evaluate effectiveness and safety. Analysis should compare outcomes against predefined acceptance criteria using appropriate statistical methods. The acceptance criteria may be determined by the sponsor, but the sponsor is expected to justify and substantiate how the criteria were established.

Examples of performance indicators for clinically meaningful primary endpoints include:

• Sensitivity, Specificity
• PPV, NPV
• NNT
• AUC

6) Practical sponsor checklist (ready-to-use)

• Define intended use, users, and clinical workflow context; align design and endpoints accordingly.
• Choose prospective vs retrospective strategy; if retrospective, plan additional studies for usability/workflow effects where needed.
• Ensure test datasets are representative and independent from training datasets.
• Plan bias controls (randomization/blinding) and document rationale.
• Select and document the reference standard; pre-plan expert adjudication and consensus processes.
• Predefine acceptance criteria for endpoints and provide a defensible justification.

---

Q&A

Q1. Why do MLMD trials need special attention compared with traditional devices?

Because MLMD performance and clinical value can be sensitive to data representativeness, dataset independence, reference standard construction, and interpretation processes—areas the guidance highlights to maintain validity and limit bias.

Q2. Can I rely only on a retrospective dataset study for an MLMD?

Retrospective studies can be useful, but the guidance cautions they may not assess usability or unintended workflow consequences; sponsors may need additional studies (e.g., usability studies).

Q3. What does “independent test dataset” mean here?

The test dataset used for evaluation (prospective or retrospective) should remain independent from the training datasets used during development, to avoid inflated performance estimates.

Q4. What should I do if clinical experts disagree on ground truth labels?

Use a systematic, transparent process (panel review/adjudication/consensus), and document disagreements, the resolution method, and the rationale for the final consensus.

Q5. Do I have to use established clinical guidelines as the reference standard?

Typically yes, but if there are no clear guidelines due to novelty, experts may establish a reference guideline; additional studies may be needed to validate novel clinical associations.

Q6. Who sets the acceptance criteria for the primary endpoint?

The sponsor can set acceptance criteria but must justify and substantiate how those criteria were established.

---

References

• MFDS page hosting the MFDS–HSA guidance (registration date: 2024-12-11) ([MFDS][1])
• HSA-hosted PDF: Guiding Principles for conducting Clinical Trial for Machine Learning-enabled MDs (MFDS–HSA)
• IMDRF/AIMD WG/N67: Machine Learning-enabled Medical Devices: Key Terms and Definitions ([imdrf.org][2])
• IMDRF/SaMD WG/N41: Software as a Medical Device (SaMD): Clinical Evaluation ([imdrf.org][3])
• Attached PDF provided in this chat: Guiding Principles for conducting Clinical Trial for Machine Learning-enabled MDs_MFDS-HSA.pdf