AI Research Portal

This paper introduces OFL-SAM2, a prompt-free framework for medical image segmentation (MIS) that adapts the SAM2 model. OFL-SAM2 addresses the need for extensive annotated data and manual prompts in MIS by training a lightweight mapping network to generate discriminative target features from generic image features. This network is updated online during inference, enhancing generalization. An adaptive fusion module integrates these learned features with SAM2's existing memory-attention features to produce accurate segmentations. The authors claim OFL-SAM2 achieves state-of-the-art performance with limited training data on three diverse MIS datasets.

The paper addresses several challenges in adapting SAM2 for medical image segmentation:

Need for extensive annotated medical data and high-quality manual prompts: These are labor-intensive and require expert intervention, hindering efficient segmentation of large datasets or 3D volumes.
Domain gap between natural and medical images: SAM2's zero-shot performance in MIS is suboptimal due to this gap.

A novel prompt-free SAM2 framework for label-efficient MIS, featuring an online few-shot learner that continuously trains a lightweight mapping network to generate discriminative target representations without manual prompts.
An adaptive fusion module that dynamically integrates learned target features with SAM2's memory-attention features to adapt the frozen SAM2 decoder and suppress distractor representations.
Comprehensive evaluations on three MIS datasets (CT, MRI, surgical videos) demonstrating state-of-the-art performance with limited training data.

OFL-SAM2 builds upon the SAM2 architecture by incorporating two key lightweight components: an online few-shot learner and an adaptive fusion module. The framework operates in two phases: offline training and online inference.

Offline Training:

The framework utilizes the image encoder and memory attention module from the pre-trained SAM2.
A lightweight mapping network (T) and an adaptive fusion module (AFM) are introduced.
The few-shot learner is trained using limited annotated samples from the training set to optimize the parameters of the mapping network. The objective is to minimize the squared error between the transformed generic features and the target features derived from memory.
The core SAM2 components (image encoder, memory encoder, mask decoder) are kept frozen during this phase.

Online Inference:

For a query image, the image encoder extracts generic features.
The memory attention module, using features from a memory bank (populated with selected training examples), generates memory-conditioned features (E1).
The trained mapping network transforms the generic features into target features (E2).
The Adaptive Fusion Module (AFM) dynamically fuses E1 and E2 using a learned weight map (W) to produce refined target features (Etar). This fusion aims to balance contributions and suppress distractors.
The frozen mask decoder then uses Etar to predict the segmentation mask.
A quality assessment mechanism evaluates the mask prediction. High-confidence predictions are used to update the mapping network parameters (via the few-shot learner) and populate the memory bank, enabling continuous adaptation to the test sequence.

Key Equations:

Cosine similarity for selecting memory bank candidates: Sim (FQ, Fs;) = (FQ · Fs;) / (||FQ|| ||Fs;||)
Few-shot learner loss: L(T) = (1/2) Σ ||T(Fs) – Ms||² + λ ||T||²
Adaptive Fusion Module feature fusion: Etar = W ⊙ E1 + (1 − W) ⊙ E2
Confidence score for update strategy: Scf = Σi,j Pi,j Gi,j / Σi,j Gi,j

Architecture: OFL-SAM2 builds on SAM2, incorporating a lightweight mapping network (a differentiable linear layer, implemented as a 3x3 convolutional layer) and an adaptive fusion module (AFM). The core SAM2 components (image encoder, memory encoder, mask decoder) are frozen.
Key Equations:
- Cosine similarity for memory bank selection: Sim (FQ, Fs;) = (FQ · Fs;) / (||FQ|| ||Fs;||)
- Few-shot learner loss: L(T) = (1/2) Σ ||T(Fs) – Ms||² + λ ||T||²
- Adaptive Fusion Module feature fusion: Etar = W ⊙ E1 + (1 − W) ⊙ E2, where W = Ge([E1, E2])
- Confidence score for update strategy: Scf = Σi,j Pi,j Gi,j / Σi,j Gi,j
Loss/Objective: The training loss is a combination of Cross-Entropy loss and Dice loss. The few-shot learner minimizes squared error.
Optimization: AdamW optimizer is used. Initial learning rate is 1e-3, decaying by a factor of 0.1 at the 10th and 30th epochs. For the few-shot learner, 10 iterations of optimization are used during training and 5 during inference.
Augmentation: Not stated in the paper.
Data Handling: For Synapse-CT and PROMISE12, 2 or 3 volumes are used as training sets. For Autolaparo, 10% or 5% of video sequences are used. The remaining data is for evaluation. Two to K=2 most similar image-mask pairs from the training set are selected for the memory bank.
Implementation Details: Trained on one NVIDIA RTX 3090 GPU using the SAM2_base model. C=256, D=64 for the mapping network. Update strategy threshold γ = 0.8.

Datasets: Synapse-CT (multi-organ CT), PROMISE12 (prostate MR), Autolaparo (surgical video).
Metrics: Dice score (DSC) and average Hausdorff distance (HD).
Baselines: SAM2 (mask prompt), SAMed, SurgicalSAM, H-SAM, FS-MedSAM2, FATE-SAM2.
Numeric Results:
- Synapse-CT (3 volumes): OFL-SAM2 achieved 82.52% mean DSC and 8.22 HD, outperforming all competitors. With 2 volumes, OFL-SAM2 achieved 81.81% mean DSC and 9.10 HD.
- PROMISE12 (3 volumes): OFL-SAM2 achieved 88.74% mean DSC, surpassing H-SAM by 1.47%. With 2 volumes, OFL-SAM2 achieved 88.07% mean DSC.
- Autolaparo (10% data): OFL-SAM2 achieved 84.41% mean DSC and 11.04 HD. With 5% data, OFL-SAM2 achieved 83.78% mean DSC.
Runtime/Training Cost: The paper states that the entire adaptation requires modest computational overhead as the mapping network and AFM are lightweight, enabling rapid training convergence with limited annotations. Specific training times are not reported.

The paper does not explicitly state any weaknesses or limitations of the proposed OFL-SAM2 framework.

Method evidence: "Crucially, the mapping network supports online parameter update during inference, enhancing the model's generalization across test sequences." (Abstract)
Result evidence: "Extensive experiments on three diverse MIS datasets demonstrate that OFL-SAM2 achieves state-of-the-art performance with limited training data." (Abstract)
Limitation evidence: No explicit paper-reported limitations were found.

OFL-SAM2: Prompt SAM2 with Online Few-shot Learner for Efficient Medical Image Segmentation

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures