SAMed-2: Selective Memory Enhanced Medical Segment Anything Model

Medical Image Segmentation with Memory-Enhanced SAM

Zhiling Yan¹, Sifan Song², Dingjie Song¹, Yiwei Li³, Rong Zhou¹, Weixiang Sun⁴, Zhennong Chen², Sekeun Kim², Hui Ren², Tianming Liu³, Quanzheng Li², Xiang Li², Lifang He¹, Lichao Sun*¹

¹Lehigh University, Bethlehem, PA, USA ²Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA ³University of Georgia, Athens, GA, USA ⁴University of Notre Dame, Notre Dame, IN, USA

Paper Code Dataset Model

Abstract

SAMed-2 is a new foundation model for medical image segmentation built upon the SAM-2 architecture. We introduce a temporal adapter into the image encoder to capture image correlations and a confidence-driven memory mechanism to store high-certainty features for later retrieval. This memory-based strategy counters the pervasive noise in large-scale medical datasets and mitigates catastrophic forgetting when encountering new tasks or modalities.

Method

Figure 1: Workflow of SAMed-2. It integrates a temporal adapter in the image encoder to capture multi-dimensional context and a confidence-driven memory module to store high-certainty features. During inference, the model retrieves these memory features and fuses them with image embeddings via attention.

MedBank Dataset

MedBank is a comprehensive collection of various public medical imaging datasets for training and evaluating medical image segmentation models. It covers diverse modalities and anatomical structures.

17

Datasets

1.2M

Images

20+

Organ Types

7

Modalities

Citation


@article{yan2025samed,
  title={SAMed-2: Selective Memory Enhanced Medical Segment Anything Model},
  author={Yan, Zhiling and Song, Sifan and Song, Dingjie and Li, Yiwei and Zhou, Rong and Sun, Weixiang and Chen, Zhennong and Kim, Sekeun and Ren, Hui and Liu, Tianming and others},
  journal={arXiv preprint arXiv:2507.03698},
  year={2025}
}