Projects

Vision Language Models for Healthcare


Vision-Language Models & Multimodal Foundation Models

Our laboratory advances medical vision-language models (VLMs) and multimodal foundation models that connect medical imaging, clinical language, and patient-level information. Led by Dr. Imon Banerjee, our research spans the development, adaptation, evaluation, and clinical translation of multimodal AI systems across radiology, oncology, and cardiovascular medicine.

Our goal is to move beyond isolated image recognition toward clinically grounded multimodal reasoning—AI systems that can integrate visual evidence with clinical context, communicate their findings, generalize across populations and institutions, and support patient-centered decision-making.


VITAL: Vision-Language Intelligence for Translational and Clinical AI

VITAL represents our broader effort to develop and evaluate clinically meaningful vision-language intelligence. The initiative connects foundation-model development with real-world clinical translation, emphasizing models that can reason across medical images, language, and patient context rather than treating each modality independently.

VITAL research spans the full translational pathway—from multimodal representation learning and domain-specific adaptation to clinical reasoning, external validation, fairness, and responsible deployment.


Domain-Specific Vision-Language Foundation Models

We investigate how large-scale vision-language models can be adapted to specialized medical domains where labeled data are scarce and clinical concepts differ substantially from natural images.

Our work explores knowledge-guided adaptation, contrastive learning, selective sampling, hard-negative mining, transfer learning, and domain-specific representation learning to create more data-efficient and clinically meaningful multimodal models.


Breast Imaging & Cancer Screening

We develop VLMs for screening mammography and breast cancer risk assessment, including breast density estimation, BI-RADS classification, image-text retrieval, and multimodal risk prediction.

This research examines whether domain-specific vision-language representations can improve performance with limited annotations while maintaining generalization across external screening populations.


Cardiovascular Vision-Language Models

Our research extends multimodal foundation models to cardiovascular medicine, including echocardiography and chest radiography.

Projects such as EchoAtlas explore conversational, multi-view vision-language foundation models capable of visual question answering, quantitative interpretation, and clinically grounded reasoning from echocardiography.

Complementary work integrates chest radiographs with physician-authored clinical histories to investigate multimodal approaches for cardiovascular risk prediction.


Multimodal Clinical Reasoning

A central focus of our research is moving from image interpretation to clinical reasoning.

We develop models that combine imaging with clinical narratives, electronic health records, and other patient-specific information. These systems are designed to reason over complementary sources of evidence and generate predictions that more closely reflect how clinicians make decisions in practice.


Trustworthy & Generalizable VLMs

Clinical foundation models require more than high benchmark performance. We study robustness, fairness, external generalization, calibration, transparent reporting, and clinically appropriate evaluation of VLMs.

Through VITAL and related work, we aim to establish rigorous approaches for determining when multimodal foundation models are ready—and when they are not ready—for real-world healthcare use.


Screenshot 2024-08-29 at 4.56.17 PM

1. EchoAtlas: A Conversational, Multi-View Vision-Language Foundation Model for Echocardiography Interpretation and Clinical Reasoning

View Publication PubMed

2. On the Transfer Learning Behavior of Domain-Specific Vision-Language Models in Screening Mammography

View Publication PubMed

3. Knowledge-Grounded Adaptation Strategy for Vision-Language Models: Building a Unique Case-Set for Screening Mammograms for Residents Training

View Publication DBLP

4. VITAL: Rationale and Methodology for Reporting Healthcare Vision-Language Models

Publication link: 

5. Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare

View Publication arXiv

6. Multimodal Artificial Intelligence Models for Radiology

View Publication PubMed

7. Mammo-FM: Breast-Specific Foundational Model for Integrated Mammographic Diagnosis, Prognosis, and Reporting

View Publication ASU Engineering Labs

8. EchoGraph System for Automated Quality Assessment of Echocardiography Reports

https://www.nature.com/articles/s41746-025-02140-w

Large Language Models for clinical and biomedical domains

Domain adaptation – Generic large language models (LLMs), such as GPT-4, are trained on diverse datasets to develop a broad understanding of human language. While powerful across general domains, these models often struggle with the complexity and precision required in specialized fields like clinical medicine. Our project addresses this gap by developing domain-tailored LLMs specifically optimized for the clinical context of prostate and breast cancer.


Risk assessment and mitigation – Despite their transformative impact on natural language processing, public LLMs present notable vulnerabilities given the source of training data is often web-based or crowdsourced, and hence can be manipulated by perpetrators. We delve into the vulnerabilities of clinical LLMs, which is trained on publicly available biomedical literature and clinical notes, in the realm of data poisoning and model editing attacks. We are developing open-source toolkit for identification and recovery of the poisoned LLM.


  1. Development of LLM for Prostate Cancer—The Need for Domain-Tailored Training.” Cancer Detection and Diagnosis: A Handbook of Emerging Technologies (2025). https://www.taylorfrancis.com/chapters/edit/10.1201/9781003449942-48/development-llm-prostate-cancer—the-need-domain-tailored-training-amara-tariq-man-luo-aisha-urooj-avisha-das-jiwoong-jeong-shubham-trivedi-haidar-abdul-muhsin-umer-ghaffar-nathan-yu-bhavik-patel-imon-banerjee

     

  2. “Weakly supervised language models for automated extraction of critical findings from radiology reports.” npj Digital Medicine 8, no. 1 (2025). https://www.nature.com/articles/s41746-025-01522-4
  3. “Exposing Vulnerabilities in Clinical LLMs Through Data Poisoning Attacks: Case Study in Breast Cancer.”  https://pmc.ncbi.nlm.nih.gov/articles/PMC10984073/
  4.  “Evaluating large language models in echocardiography reporting: opportunities and challenges.” European Heart Journal-Digital Health https://pubmed.ncbi.nlm.nih.gov/40395412/
  5.  “Patient-centric Summarization of Radiology Findings using Two-step Training of Large Language Models.” ACM Transactions on Computing for Healthcare https://dl.acm.org/doi/10.1145/3709154

Graph-based fusion for prediction

To successfully perform any clinical prediction task, it is essential to learn effective representations of various data captured during patient encounters and model their interdependencies, including patient demographics, diagnostic codes, and radiologic imaging. Graph convolutional neural networks (GCN) present an intuitive and elegant way of processing multi-modal data presented as a graph structure. We proposed novel GCN model design to understand the relationship between imaging and non-imaging data and by incorporating holistic weighted edge formation based on patient clinical history and demographic information. 

Two graph based AI models are currently in clinical rollout at Mayo Clinic.


Picture1
  1. Graph convolutional network-based fusion model to predict risk of hospital acquired infections. Journal of the American Medical Informatics Association. https://pubmed.ncbi.nlm.nih.gov/37027831/
  2.  “Fusion of imaging and non-imaging data for disease trajectory prediction for coronavirus disease 2019 patients.” Journal of Medical Imaging https://pmc.ncbi.nlm.nih.gov/articles/PMC10306115/
  3. “Adaptable graph neural networks design to support generalizability for clinical event prediction.” Journal of biomedical informatics. https://www.sciencedirect.com/science/article/pii/S1532046425000231
  4. “Graph convolutional network-based fusion model to predict risk of hospital acquired infections.” Journal of the American Medical Informatics Association. https://pubmed.ncbi.nlm.nih.gov/37027831/
  5. “Predicting 30-day all-cause hospital readmission using multimodal spatiotemporal graph neural networks.” IEEE Journal of Biomedical and Health Informatics. https://pmc.ncbi.nlm.nih.gov/articles/PMC11073780/

MUSE: Artificial Intelligence enabled multi-modal sensor platform for at-home health monitoring of patients 

Project 1 — AFRL: In-Sensor AI for Wearable Physiologic Monitoring

The A3I Hub participated in a SEMI and Air Force Research Laboratory (AFRL)-funded research program focused on developing ultra-low-power, in-sensor artificial intelligence for continuous physiologic monitoring. The central objective was to move machine-learning inference from conventional cloud-based platforms, e.g. Apple watch, directly into wearable and sensor-integrated hardware, enabling real-time detection of clinically relevant physiologic abnormalities from patient vital signs.

The research explored digital, analog, and bio-inspired neural computing architectures, including artificial neural networks (ANNs), reservoir computing, and echo-state networks implemented in CMOS. These approaches were designed to substantially reduce the energy and computational requirements of machine-learning inference, supporting continuous operation in resource-constrained wearable devices. Target applications included arrhythmia and sepsis detection from physiologic signals, ECG-based stress detection, cardiovascular disease prediction, and broader at-home health monitoring.

A major technical theme was demonstrating that clinically relevant machine-learning models could be implemented directly in low-power integrated circuits rather than requiring transmission of raw sensor data to an external processor. Representative work included real-time stress detection from wearable ECG using digital machine-learning circuitry; a bio-inspired reservoir computer for ECG analysis; a 7.5-nJ/inference CMOS echo-state network for coronary heart disease prediction; and integrated reservoir-computing hardware aimed at real-time, at-home patient monitoring. Parallel studies of analog neural-network classifiers established circuit architectures and custom activation functions for extremely energy-efficient embedded inference.

Collectively, the AFRL-supported work established an early foundation for the A3I Hub's wearable-sensor research: integrating sensing and AI computation at the edge to create low-power, autonomous systems capable of continuously interpreting physiologic data. This paradigm has potential advantages for wearable healthcare, including reduced communication bandwidth and power consumption, lower inference latency, greater independence from network connectivity, and improved privacy by minimizing transmission of raw physiologic signals.


Project 2 — NIH SCH: MUSE — Multimodal At-Home Monitoring for Acute Kidney Injury Survivors

The A3I Hub is currently leading in an NIH Smart and Connected Health (SCH)-funded project focused on developing MUSE (MUlti-modal SEnsor), an AI-enabled, non-invasive platform for monitoring patients at home following acute kidney injury (AKI). The project addresses an important gap in post-AKI care: survivors face substantial risks of recurrent AKI, progression to chronic kidney disease, cardiovascular complications, hospital readmission, and mortality, yet routine specialist follow-up is difficult to scale and can impose substantial burdens on patients and healthcare systems.

The central concept of MUSE is to replace reliance on frequent in-person assessment with patient-operated multimodal sensing coupled with AI-based risk prediction. The platform combines urinary biomarkers, wearable physiologic measurements, and patient clinical information to identify changes associated with AKI recurrence and alert patients when their estimated risk warrants medical attention. MUSE integrates four urinary biomarkers—urea, creatinine, lactate, and pH—with electrocardiogram (ECG) measurements and electronic medical record (EMR) information. Urinary measurements provide indicators of renal status, while ECG adds cardiovascular information relevant to post-AKI risk. Rather than treating each measurement independently, the system uses multimodal AI to integrate these heterogeneous data streams into a consolidated assessment of the patient's evolving health status.

A smartphone serves as the computational and user-interface hub for the system. Patients capture images of a urine dipstick-based sensor using the smartphone camera, with computer-vision algorithms quantifying the urinary biomarkers. ECG measurements are transferred to the phone using near-field communication (NFC). Patient medical information provides additional clinical context. Computer-vision and AI algorithms running on the smartphone then process and fuse the sensor and clinical data to estimate risk and generate alerts when potentially adverse changes are detected.

The proposed monitoring paradigm calls for patients to collect paired urine and ECG measurements at least twice daily, separated by approximately 6–8 hours, providing longitudinal measurements rather than isolated clinical snapshots. This repeated multimodal sampling creates an opportunity for AI models to detect evolving physiologic changes and potentially identify deterioration before it results in an emergency visit or rehospitalization.

From the perspective of the broader A3I wearable-sensor research program, MUSE represents an important progression from Project 1's in-sensor AI and low-power physiologic inference toward an integrated digital-health system combining wearable sensing, biochemical sensing, computer vision, clinical data, and multimodal AI. The project emphasizes translation into a patient-facing application: the technology is designed to be non-invasive, affordable, easy to operate, and usable without continuous involvement of a clinical expert.

Key innovation: MUSE establishes a framework for multimodal, AI-enabled remote patient monitoring, in which inexpensive biochemical sensing and wearable physiologic measurements are combined with existing clinical information to provide individualized longitudinal risk assessment. If successfully validated, this approach could help identify high-risk AKI survivors who require clinical intervention while reducing unnecessary visits for lower-risk patients, potentially improving post-AKI outcomes while making specialized follow-up more scalable.


Project 3 — Multimodal Wearable Biosensing for CAR-T Toxicity and Cardiotoxicity

The A3I Hub is leading a translational effort to develop and clinically validate a multimodal wearable biosensing platform for continuous monitoring of patients receiving CAR-T therapy, in partnership with Terasaki, Empaticaand Mayo Clinic clinical experts in oncology and cardiology. The project targets two major complications of CAR-T treatment: Cytokine Release Syndrome (CRS) and treatment-associated cardiotoxicity. 

CAR-T therapy has transformed the treatment of hematologic malignancies, but its potentially severe toxicities create a significant monitoring challenge. CRS can develop rapidly and unpredictably, progressing from early inflammatory changes to severe hemodynamic instability within hours. Current surveillance relies largely on intermittent vital signs, laboratory testing, and clinician assessment, making monitoring episodic and often reactive. In particular, inflammatory cytokines such as IL-6 and TNF-α are not continuously measured in routine clinical care, meaning that molecular evidence of immune activation may precede recognition of clinical deterioration. Consequently, interventions such as Tocilizumab may occur only after symptomatic CRS is established.

The proposed platform seeks to shift this paradigm from episodic clinical monitoring to continuous molecular and physiologic surveillance. Wearable and biosensor technologies will capture longitudinal inflammatory biomarker trajectories together with physiologic signals. These multimodal measurements will be integrated with relevant clinical information—including demographics, comorbidities, treatment characteristics, and medications—using AI-based risk models.

A principal research objective is to determine whether dynamic changes in inflammatory biomarkers and physiologic signals can provide an early-warning signature of CRS before overt clinical symptoms emerge. Rather than relying on individual threshold measurements, the AI framework will analyze temporal and multimodal patterns to generate individualized risk estimates. Demonstrating clinically meaningful lead time could enable earlier risk-adapted surveillance and more precisely timed therapeutic intervention.

A complementary objective is early characterization of CAR-T-associated cardiotoxicity. Cardiovascular complications may evolve before overt cardiac dysfunction becomes apparent through conventional clinical evaluation. Continuous longitudinal sensing provides an opportunity to identify subtle physiologic patterns associated with emerging cardiovascular injury and establish a foundation for predictive rather than retrospective cardiac risk assessment.

The translational program combines prospective clinical studies, wearable and biosensor deployment, real-time AI-driven risk modeling, and clinical validation. The Mayo Clinic A3I Hub brings together AI scientists with expert cardiology and oncology investigators, while Empatica contributes expertise in deployable wearable technologies. Beyond demonstrating algorithmic performance, the effort is designed to establish a scalable, production-ready monitoring system suitable for high-risk oncology care.

Key innovation: Project 3 advances the A3I wearable-sensor program from physiologic and biochemical monitoring toward continuous multimodal molecular + physiologic sensing coupled with real-time clinical AI. By integrating inflammatory biomarkers, wearable physiology, and clinical context, the project aims to provide actionable early warning of CRS while simultaneously establishing digital signatures of cardiotoxicity. The broader goal is to transform CAR-T toxicity management from reactive detection of established complications to proactive, individualized risk prediction and intervention.


Project 4 — PerfusionEdge: On-Chip AI for Transplant Organ 

In collaboration with transplant surgeon at Mayo Clinic, ASU and Terasaki institute, PerfusionEdge is an edge-AI platform for continuous, real-time assessment of organ viability during machine perfusion. Building on the team's dual-compartment biochemical monitoring (DCBM) system, which continuously measures glucose, lactate, and pH in perfusate and bile, and SCOPE, a self-supervised AI model for detecting abnormal graft physiology, the project moves AI inference from offline workstations directly into sensing hardware.

PerfusionEdge will deploy SCOPE on ultra-low-power analog in-memory computing chips, enabling nanojoule-scale inference without cloud connectivity. The system will be integrated with DCBM electronics and prospectively evaluated alongside clinical liver perfusion at Mayo Clinic Arizona. The collaboration combines Mayo Clinic's clinical perfusion and AI expertise with ASU's analog AI and system-on-chip design.

Key innovation: integrating continuous biochemical sensing with embedded AI to provide autonomous, real-time organ viability assessment at the point of perfusion, creating a reusable platform for transplantation and other applications requiring continuous physiologic intelligence.

 

Core technical objectives

1. Design a battery-less patch fabricated on a soft, skin-like platform to record ECG, PPG and Temperature, and encode the signal using an in-sensor analog reservoir computer for ultra-low energy consumption.

2. Design a Computer-vision enabled dipstick sensor platform. Develop a single, microfluidic dipstick platform with embedded colorimetric assays for detecting four markers from urine samples through automated computer vision analysis.

3. Train a Deep learning based fusion AI model. Design a deep-learning based representation learning fusion AI model that combines multimodal sensor data (urine samples, ECG, PPG and Temperature) and patient medical record (past co-morbidities and demographics) for predicting risk of adverse events.

4. Smart phone application development. Develop a smartphone application that will: 1) allow user to input their medical record; 2) use built-in camera to capture images of dipstick platform that detects urinary bio-markers; 3) record ECG signal using near-field communication (NFC) protocol in the smartphone; 4) run computer vision and AI algorithms on-board to process sensor data, fuse with EMR for monitoring user health, and send alerts to user.

Screenshot 2025-07-27 at 9.25.15 AM
  1. Bhattacharya, Amartya, Sudarsan Sadasivuni, Chieh-Ju Chao, Pradyumna Agasthi, Chadi Ayoub, David R. Holmes, Reza Arsanjani, Arindam Sanyal, and Imon Banerjee. “Multi-modal fusion model for predicting adverse cardiovascular outcome post percutaneous coronary intervention.” Physiological Measurement 43, no. 12 (2022): 124004.
  2. Sadasivuni, Sudarsan, Sumukh Prashant Bhanushali, Imon Banerjee, and Arindam Sanyal. “In-sensor neural network for high energy efficiency analog-to-information conversion.” Scientific reports 12, no. 1 (2022): 18253.
  3. Sadasivuni, Sudarsan, Monjoy Saha, Neal Bhatia, Imon Banerjee, and Arindam Sanyal. “Fusion of fully integrated analog machine learning classifier with electronic medical records for real-time prediction of sepsis onset.” Scientific reports 12, no. 1 (2022): 5711.
  4. Sikha, Madhu Babu, Lalith Appari, Gurudatt Nanjanagudu Ganesh, Amay Bandodkar, and Imon Banerjee. “Multi-Analyte, Swab-based Automated Wound Monitor with AI.” arXiv preprint arXiv:2506.03188 (2025).

Adversarial and Causal debiasing and adaptation — medical image case-study

Despite the expert-level performance of artificial intelligence (AI) models for various medical imaging tasks, real-world performance failures with disparate outputs for various minority subgroups limit the usefulness of AI in improving patients’ lives. AI has been shown to have a remarkable ability to detect protected attributes of age, sex, and race, while the same models demonstrate bias against historically underserved subgroups of age, sex, and race in disease diagnosis. Therefore, an AI model may take shortcut predictions from these correlations and subsequently generate an outcome that is biased toward certain subgroups even when protected attributes are not explicitly used as inputs into the model. We explore various types of bias from shortcut learning that may occur at different phases of AI model development and develop mitigation technique from preprocessing (data-centric solutions) and during model development (computational solutions) and postprocessing (recalibration of learning).


Domain adaptation – While the kidney segmentation is a required step for various clinical analyses, starting from disease detection, surgical planning, radiation dose optimization, to longitudinal characterization, the model development has largely focused on using a single-phase CT, resulting in significant performance degradation caused by drift in contrast phase appearance. We target three domain shifts to propose a robust kidney segmentation model: (i) contrast to non-contrast, (ii) arterial to venous phase, and (iii) normal functioning kidney to abnormal. Leveraging a traditional segmentation residual network (SegResNet) backbone, we design the domain adaptation for phase resiliency using an auxiliary contrast-phase classifier with confusion loss. 


Causal debiasing – Our project introduces MOSCARD, a novel predictive modeling framework that integrates chest X-rays (CXR) and 12-lead electrocardiograms (ECG) to enhance cardiovascular risk assessment. While CXR provides insights into chronic conditions contributing to MACE, ECG captures real-time cardiac electrical activity and structural abnormalities. By combining these two complementary modalities, MOSCARD delivers a more holistic and accurate risk profile than traditional models based on clinical scores, CT scans, or biomarkers. Unlike much of the existing literature, which often overlooks the role of confounders and underlying causal structures in multimodal data, our approach explicitly integrates causal reasoning by incorporating comorbidities and mitigates bias.

Screenshot 2025-07-27 at 9.08.25 AM

Adversarial debiasing – In our ongoing study, we develop a two-step adversarial debiasing approach with partial learning that can reduce the racial disparity while preserving the performance of the targeted task. The methodology has been evaluated onindependent medical image case-studies – skin cancer, chest X-ray, mammograms, and showed promises in bias reduction while preserving the targeted performance.


  1.  “MOSCARD–Causal Reasoning and De-confounding for Multimodal Opportunistic Screening of Cardiovascular Adverse Events.” MICCAI 2025. https://link.springer.com/chapter/10.1007/978-3-032-04984-1_32
  2. “Mitigating Bias in Opportunistic Screening for MACE with Causal Reasoning.” IEEE Transactions on Artificial Intelligence (2025). https://pubmed.ncbi.nlm.nih.gov/41497309/
  3. “Open-source domain adaptation to handle data shift for volumetric segmentation—use case kidney segmentation.” European Radiology (2025). https://pubmed.ncbi.nlm.nih.gov/40581675/
  4. “Shortcuts” causing bias in radiology artificial intelligence: causes, evaluation and mitigation.” Journal of the American College of Radiology (2023). https://pubmed.ncbi.nlm.nih.gov/37506964/
  5.  “Anomaly Detection using Cascade Variational Autoencoder Coupled with Zero Shot Learning.” In Medical Imaging with Deep Learning.https://openreview.net/pdf?id=VmFdXXpVx8
  6.  “Adversarial Debiasing techniques towards ‘fair’ skin lesion classification.” In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER). https://ieeexplore.ieee.org/document/10123788
  7. “Causal debiasing for unknown bias in histopathology—A colon cancer use case.” PloS one.https://pubmed.ncbi.nlm.nih.gov/39576760/

AI Technique for curating Large Cancer Databases (AI-LAD) – Breast, prostate and pancreas

Despite advances in breast cancer treatment, disparities in outcomes based on age, race/ethnicity, and socioeconomic status persist. Most existing studies are limited by small, localized datasets and focus mainly on insurance and tumor stage, lacking comprehensive long-term data. A key factor in survival—adherence to long-term treatments like endocrine therapy—is often underrepresented due to data limitations.

To address this, a large, diverse, and longitudinal breast cancer database is needed. However, manual curation is impractical due to the complexity and scale of the task. National registries like SEER only track initial treatments and lack ongoing clinical follow-up.

A multidisciplinary team is developing AI-driven natural language processing (NLP) tools to automate the extraction of long-term clinical and patient-centered outcomes from electronic medical records. Their toolset, AI-LAD, is designed to be deployed locally at institutions to parse clinical notes, pathology, and radiology reports, enabling scalable and accurate data curation for breast cancer research.

  1. Gupta, Anupama, Imon Banerjee, and Daniel L. Rubin. “Automatic information extraction from unstructured mammography reports using distributed semantics.” Journal of biomedical informatics 78 (2018): 78-86.
  2. Banerjee, Imon, Selen Bozkurt, Jennifer Lee Caswell-Jin, Allison W. Kurian, and Daniel L. Rubin. “Natural language processing approaches to detect the timeline of metastatic recurrence of breast cancer.” JCO clinical cancer informatics 3 (2019): 1-12.
  3. Banerjee, Imon, Kevin Li, Martin Seneviratne, Michelle Ferrari, Tina Seto, James D. Brooks, Daniel L. Rubin, and Tina Hernandez-Boussard. “Weakly supervised natural language processing for assessing patient-centered outcome following prostate cancer treatment.” JAMIA open 2, no. 1 (2019): 150-159.
  4. Luo, Man, Shubham Trivedi, Allison W. Kurian, Kevin Ward, Theresa HM Keegan, Daniel Rubin, and Imon Banerjee. “Automated Extraction of Patient-Centered Outcomes After Breast Cancer Treatment: An Open-Source Large Language Model–Based Toolkit.” JCO Clinical Cancer Informatics 8 (2024): e2300258.
  5. Chiang, Chia‐Chun, Man Luo, Gina Dumkrieger, Shubham Trivedi, Yi‐Chieh Chen, Chieh‐Ju Chao, Todd J. Schwedt, Abeed Sarker, and Imon Banerjee. “A large language model–based generative natural language processing framework fine‐tuned on clinical notes accurately extracts headache frequency from electronic health records.” Headache: The Journal of Head and Face Pain 64, no. 4 (2024): 400-409.
  6. Tariq, Amara, Madhu Sikha, Allison W. Kurian, Kevin Ward, Theresa HM Keegan, Daniel L. Rubin, and Imon Banerjee. “Open-Source Hybrid Large Language Model Integrated System for Extraction of Breast Cancer Treatment Pathway From Free-Text Clinical Notes.” JCO Clinical Cancer Informatics 9 (2025): e2500002.

Opportunistic screening using multi-modal data

Unstructured medical data analysis and integration of multimodal data (image + EHR) can unlock the large amount of electronic healthcare records (EHR) for clinical event prediction (e.g. ER visits, hospitalization, short-term mortality). Opportunistic screening occurs when healthcare providers use a patient’s existing clinical interactions (e.g., routine visits, imaging for unrelated issues) to screen for conditions like cancer, rather than relying solely on scheduled screening programs. Our research interest is multimodal clinical data integration and predictive modeling to perform opportunistic screening for ASCVD and pancreatic cancer. For example, A 65-year-old patient undergoes a non-gated chest CT for evaluation of chronic cough. The scan is not intended for cardiac assessment, but it incidentally a deep learning model captures the calcium in coronary arteries —a marker of subclinical atherosclerosis, and generates a patient-specific report with CAC images and guideline-based recommendations using prior clinical history.

  • Design a temporal deep learning model for estimating short-term life expectancy of the patients by analyzing free-text clinical notes.
  • Developed a computerized technique for assessing treatment response by analyzing noninvasive 2D/3D scans.
  • Proposed a framework of data analysis tools for the automatic computation of qualitative and quantitative parameters to support effective annotation of patient-specific follow-up data.
  • Developing a longitudinal machine learning approach to predict weight gain/loss in the context of insulin sensitivity and resistance by combining multiple omics data.

Papers/products

1.   “Patient-specific COVID-19 resource utilization prediction using fusion AI model.” NPJ digital medicine 4, no. 1 (2021): 1-9. [link]

2. “Probabilistic Prognostic Estimates of Survival in Metastatic Cancer Patients (PPES-Met) Utilizing Free-Text Clinical Narratives.” [link]

3.  “Assessing treatment response in triple-negative breast cancer from quantitative image analysis in perfusion magnetic resonance imaging.” [link]

4.    “Integrative Personal Omics Profiles during Periods of Weight Gain and Loss.”, [link]

5. “Semantic annotation of 3D anatomical models to support diagnosis and follow-up analysis of musculoskeletal pathologies.” [link]

Anomaly detection for medical images

Traditional anomaly detection methods focus on detecting inter-class variations while medical image novelty identification is inherently an intra-class detection problem. For example, a machine learning model trained with normal chest X-ray and common lung abnormalities, is expected to discover and flag idiopathic pulmonary fibrosis which a rare lung disease and unseen by the model during training. The nuances from intra-class variations and lack of relevant training data in medical image analysis pose great challenges for existing anomaly detection methods. To tackle the challenges, we propose a hybrid model – nonlinear Transformation-based Embedding learn- ing for Novelty Detection (TEND). Without any out-of-distribution training data, TEND performs novelty identification by unsupervised learning of in-distribution embeddings with a vanilla AutoEncoder in the first stage and dis- criminative learning of in-distribution data and the non-linearly transformed counterparts with a binary classifier and a margin-aware objective metric in the second stage. The binary discriminator learns to distinguish the in-distribution data from the generated counterparts and outputs a class probability. The margin-aware objective is optimized jointly to include the in-distribution data in a hypersphere with a pre-defined margin and exclude the unexpected data. Even- tually, the weighted sum of class probability and the distance to margin constitutes the anomaly score.

True Positive (TP, 1st row), True Negative (TN, 2nd row) predictions of TEND 500 on RSNA datasets. d: distance value from the margin learner module, p: probability outputted by the binary discriminator module, s: final score, t: optimal threshold

  1. Guo, Xiaoyuan, Judy Wawira Gichoya, Hari Trivedi, Saptarshi Purkayastha, and Imon Banerjee. “MedShift: Automated Identification of Shift Data for Medical Image Dataset Curation.” IEEE Journal of Biomedical and Health Informatics(2023).
  2. Ramasamy, Gokul, Bhavik N. Patel, and Imon Banerjee. “Anomaly Detection using Cascade Variational Autoencoder Coupled with Zero Shot Learning.” In Medical Imaging with Deep Learning, short paper track. 2023.
  3. Guo, Xiaoyuan, Judy Wawira Gichoya, Saptarshi Purkayastha, and Imon Banerjee. “CVAD: An Anomaly Detector for Medical Images Based on Cascade VAE.” In Workshop on Medical Image Learning with Limited and Noisy Data, pp. 187-196. Cham: Springer Nature Switzerland, 2022.

Quantitative analysis of medical images to support diagnosis.

Interested in developing computational methods that can extract quantitative information from images, integrate diverse clinical and imaging data, enable discovery of image biomarkers, and improve clinical treatment decisions. I am leading several innovative medical image analysis research projects related to cancer diagnosis, e.g. prostate cancer aggressiveness detection, histopathologic subtype classification of brain tumor, prediction of semantic features of bone tumor. I am developing a novel computational framework that can automatically interpret implicit semantic content from multimodal and/or multiparametric radiology images to enable biomedical discovery and to guide physicians in personalized care. I am responsible for the overall design of the framework, and development, execution, verification, and validation of the systems. 

Two selected axial CT images of the chest from two separate patients with positive diagnosis of PE. The left CT scan demonstrates a left lower lobe posterolateral basal segmental artery filling defect consistent with a pulmonary embolism. The CT scan on the right panel demonstrates a small elongated filling defect bridging across the segmental arteries of the right lower lobe consistent with a segmental pulmonary embolism, in addition to surrounding collapse of the right lower lobe. The vision-only model yielded false-negative predictions for both cases, but the fusion model correctly predicted both as positive. [link]

Publications and open-source code:

1. “Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection”, [link]

2. “Transfer Learning on Fused Multiparametric MR Images for Classifying Histopathological Subtypes of Rhabdomyosarcoma”, [link].

3. “Relevance feedback for enhancing content based image retrieval and automatic prediction of semantic image features: Application to bone tumor radiographs.” [link].

4. “Computerized Prediction of Radiological Observations based on Quantitative Feature Analysis: Initial Experience in Liver Lesions” [link].

5.  “Computerized Multiparametric MR image Analysis for Prostate Cancer Aggressiveness-Assessment”.

Fusion of Fully Integrated Analog Machine Learning Classifier with Electronic Medical Records

The objective of this work is to develop a fusion artificial intelligence (AI) model that combines patient electronic medical record (EMR) and physiological sensor data to accurately predict early risk of sepsis and cardiovascular event. The fusion AI model has two components – an on-chip AI model that continuously analyzes patient electrocardiogram (ECG) data and a cloud AI model that combines EMR and prediction scores from on-chip AI model to predict risk score. The on-chip AI model is designed using analog circuits for sepsis prediction with high energy efficiency for integration with resource constrained wearable device. Combination of EMR and sensor physiological data improves prediction performance compared to EMR or physiological data alone, and the late fusion model has an accuracy of 93% in predicting sepsis 4 hours before onset. The key differentiation of this work over existing sepsis prediction literature is the use of single modality patient vital (ECG) and simple demographic information, instead of comprehensive laboratory test results and multiple vital signs. Such simple configuration and high accuracy makes our solution favorable for real-time, at-home use for self-monitoring.

Publications and open-source code:

1. “Recurrent Neural Network Circuit for Automated Detection of Atrial Fibrillation from Raw ECG.” In 2021 IEEE International Symposium on Circuits and Systems (ISCAS), 2021. [link]

2. “Fully Integrated Analog Machine Learning Classifier Using Custom Activation Function for Low Resolution Image Classification,” [link][code]

3.   “Digital Machine Learning Circuit for Real-Time Stress Detection from Wearable ECG Sensor.”  [link][code]

Natural Language Processing on clinical notes

The lack of labeled data creates a data “bottleneck” for developing deep learning models for medical imaging. However, healthcare institutions have millions of imaging studies which are associated with unstructured free text radiology reports that describe imaging features and diagnoses, but there are no reliable methods for leveraging these reports to create structured labels for training deep learning models. Unstructured free text thwarts machine understanding, due to the ambiguity and variations in language among radiologists and healthcare organizations. 

Covid QueryBot

My research is focused in developing methods to extract structured annotations of medical images from radiology reports for training complex deep learning models.

Our method has outperformed many existing NLP algorithms on several radiology report annotation tasks (CT reports, mammography reports, US reports, and X-ray reports), as well as can infer targeted information from heterogeneous clinical notes (e.g., hospital notes, discharge summary, progress notes).

Publications and open-source code:

1. “Weakly supervised temporal model for prediction of breast cancer distant recurrence.” [link][code]

2. “Radiology Report Annotation using Intelligent Word Embeddings: Applied to Multi-institutional Chest CT Cohort,” [link][code]

3.   “Weakly supervised natural language processing for assessing patient-centered outcome following prostate cancer treatment.”  [link][code]

4. “Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification.” [link][code available upon request]

5. A Scalable Machine Learning Approach for Inferring Probabilistic US-LI-RADS Categorization [link][code available upon request]

6. “Development and Use of Natural Language Processing for Identification of Distant Cancer Recurrence and Sites of Distant Recurrence Using Unstructured Electronic Health Record Data.” [link]