Study Reveals Privacy Risks in Medical Language Models Through Clinical Framework

Researchers introduced a new privacy evaluation framework specifically designed for medical language models that goes beyond traditional memorization testing to assess realistic threat scenarios. The framework evaluates information leakage across a graded spectrum of adversarial access, from publicly inferable demographics to leaked note fragments, measuring both verbatim memorization of patient-specific text and semantic leakage of sensitive diagnoses. Testing on a language model pretrained on 378,000 clinical notes revealed that routine encounter metadata—including patient names, dates of birth, providers, practices, and visit dates—triggers high rates of verbatim memorization across patient timelines and enables recovery of sensitive diagnoses with high accuracy. However, the researchers also found that exact-match memorization metrics can overstate actual disclosure risk, as 36% of memorized tokens reflect templated documentation rather than unique patient information. The work emphasizes the particular risks associated with training on longitudinal clinical data and provides a practical framework for contextual privacy evaluation of medical language models.