Chapter meetings

Click here to RSVP.

• Free of charge

• Free parking and food will be provided

• Participation in our chapter activities is not restricted to ASA members

Machine learning systems for automated performance assessment in the educational and clinical space are far from new. However, the opportunity for deploying systems with real-world impact has undoubtedly accelerated by the advent of access to powerful large language models (LLMs) in a post-ChatGPT world.  The logical and reasoning capability of the most capable frontier and open-source LLMs (such as GPT4, Gemini, and Mixtral) represent a sea change in what is possible for automated systems across all domains, including medical education. In this talk I will share exciting developments at UT Southwestern Medical Center’s Simulation Center where we are taking the first steps to piloting multi-modal AI systems for automatically grading medical students. In particular, the Objective Structured Clinical Exam or “OSCE” is an important tool and standard approach to measuring medical student competence by participation in a live-action simulated patient encounters with human actors. The post-exam learner note is a vital element of the OSCE and accurate assessment of student performance on this task is labor intensive, requiring text documents to be reviewed by a specially trained human evaluator.  We demonstrate how LLMs enable a new era of automatic, rubric-based grading, without the requirement of any prior training data or labels (i.e. “zero-shot”). I’ll also share our experimental evaluations with local, open-source models that have been fine-tuned for this domain.

Dr. Jamieson is an Assistant Professor at the Lyda Hill Department of Bioinformatics. He has served as a co-leader of the Bioinformatics Core Facility (BICF), responsible for leading campus-wide collaborative research engagements with an emphasis on computational image analysis. In 2020, as the global pandemic emerged, Dr. Jamieson's group led the development of the UTSW COVID-19 forecast model (UTSW Newsroom article). Before returning to academia, he worked in Pharma Services, Operational Excellence, and the MultiOmyx group at Clarient, a GE Healthcare molecular diagnostics company (later sold to NeoGenomics). In 2012, Andrew dove into industry as the first employee and Data Scientist for a Plano, TX-based Big Data Analytics start-up, Oculus360. Andrew completed his BA (Physics), and Ph.D. (Medical Physics) at the University of Chicago under the supervision of Maryellen L. Giger, Ph.D., a pioneer in Breast Cancer Computer-Aided Diagnosis.