AI Research Portal

This paper introduces Hymavi, a novel hybrid network for volumetric medical image segmentation that combines Mamba-based sequence modeling and attention mechanisms in a parallel, multi-view framework. Hymavi leverages the strengths of both approaches to capture fine-grained local details and long-range spatial dependencies. Experiments on ACDC, BraTS2023, and AMOS22 datasets demonstrate Hymavi's effectiveness and generalization capabilities across different segmentation tasks and imaging modalities.

The paper addresses several challenges in volumetric medical image segmentation:

High dimensionality and anatomical diversity: 3D medical images are complex, requiring models to capture both fine anatomical structures and broader spatial patterns across multiple slices.
Limitations of attention-based models: While effective for long-range dependencies, transformer-based models often suffer from high computational demands and quadratic complexity with respect to input size.

Hymavi Architecture: A novel parallel hybrid architecture that integrates Mamba-based sequence modeling and attention mechanisms to jointly capture long-range context and fine-grained detail in 3D medical segmentation.
Multi-View Learning Framework: A novel multi-view learning strategy with intra-view enhancement and inter-view interaction modules to enable more comprehensive and context-aware segmentation across axial, sagittal, and coronal planes.
Empirical Validation: Extensive experiments demonstrating that Hymavi consistently surpasses strong baselines on multiple benchmark datasets while maintaining reasonable computational cost.

Hymavi is a U-shaped encoder-decoder network that incorporates a multi-view learning strategy. The core of the architecture is the Hybrid Mamba-Attention (HYMA) block, which processes features in parallel using Mamba for global context and attention for local spatial interactions.

Architecture: Hymavi consists of a Stem Layer for initial downsampling, an encoder with multiple stages of Intra-view Enhancement (ITA) modules, a decoder with Skip-connection Intra-view Enhancement (SITA) modules, and an Expanding Layer for upsampling.
HYMA Block: This block takes input features and applies RMS normalization and linear projection. It then splits these into query, key, value, and Mamba inputs. The attention stream computes spatially adaptive token-to-token interactions, while the Mamba stream captures global context using bidirectional processing. The outputs of both streams are summed to form the final feature representation.
- Attention Stream: fa = Att(Q, K, V)
- Mamba Stream: fm = SSM(X) = Mamba(Xf) + Mamba(X)
- Hybrid Output: fx = Att(Q, K, V) + SSM(X)
Intra-view Enhancement (ITA) Module: This module focuses on learning intricate feature representations within each view (axial, sagittal, coronal) using the HYMA block. It processes 3D input volumes, extracts spatial features using a 3D ResBlock, and then applies a transformation operator to create view-specific sequences. Multiple ITA modules are stacked in the encoder.
Inter-view Enhancement (ITE) Module: This module integrates features from different views. It uses learnable weight matrices to compute cross-enhanced features between sagittal/coronal views and the axial view, allowing selective emphasis. The final output aggregates original view-specific features and their cross-view enhancements.
Skip-connection Intra-view Enhancement (SITA) Module: Located in the decoder, this module fuses skip-connection features from the encoder with decoder features. It uses cross-attention for encoder-decoder alignment and Mamba for internal decoder dynamics.
Multi-View Fusion: After processing each view independently with ITA modules, their outputs are aggregated using an averaging operation to form a multi-view representation. The ITE module is applied to full-resolution layers to capture inter-view relationships.

Architecture:
- U-shaped encoder-decoder network.
- Includes Stem Layer, Encoder (ITA modules), Decoder (SITA and ITA modules), and Expanding Layer.
- HYMA block integrates Mamba and Attention.
- ITE module for inter-view enhancement.
Key Equations:
- HYMA block output: fx = H(Q, K, V, X) = Att(Q, K, V) + SSM(X)
- Mamba stream: fm = SSM(X) = Mamba(Xf) + Mamba(X)
- ITE module output: fe = fa + f' + fc + fsa + fca
Loss/Objective:
- Combines cross-entropy loss and soft Dice loss.
Optimization:
- Optimizer: SGD.
- Learning rate: 10-2 for ACDC and AMOS22, 10-3 for BraTS2023.
- Weight decay: 3 × 10-5.
- Momentum: 0.99.
Augmentation:
- Additive brightness adjustment, gamma correction, rotation, scaling, mirroring, and elastic deformation.
Data Handling:
- Cropped patch sizes: (14, 160, 160) for ACDC, (128, 128, 128) for BraTS2023 and AMOS22.
- Dataset splits: 70% training, 10% validation, 20% testing.
Implementation Details:
- Implemented in PyTorch.
- Training conducted for approximately 106 patch iterations (approx. 850 epochs).
- Experiments performed on NVIDIA H100 GPUs.

Datasets:
- ACDC (Automated Cardiac Diagnosis Challenge)
- BraTS2023 (Brain Tumor Segmentation Challenge 2023)
- AMOS22 (Abdominal Multi-Organ Segmentation)
Metrics:
- Dice Similarity Coefficient (DSC)
- 95th percentile Hausdorff Distance (HD95)
Baselines:
- SegresNet [22], MedNeXt [25], nnUnet [16], UNETR [13], nnFormer [36], SwinUNETR-V2 [15], EM-Net [5], LKM-UNet [30], SegMamba [31], LeViT-UNet-384s [32], Swin-UNet [4], TransUNet [7], UNETR++ [26], MCCSeg [1].
Numeric Results:
- BraTS2023: Achieved the highest average DSC (91.92%), best performance on tumor core (93.62%) and enhancing tumor (89.59%), and lowest average HD95.
- ACDC: Achieved the highest DSC scores across all cardiac structures (average DSC 92.67%), surpassing baselines.
- AMOS22: Delivered the best overall DSC (87.33%), with consistent gains in difficult organs.
Runtime/Training Cost:
- Model Complexity: Hymavi has more parameters than SegMamba but consumes significantly less memory (approx. 7.4 GB vs. 25 GB at inference for 128^3 patches).
- Speed: Achieves substantially faster inference and training speeds compared to SegMamba.
- Parallel vs. Sequential Fusion: Parallel fusion in Hymavi consistently outperformed sequential fusion strategies (SSM→Att, Att→SSM) by an average DSC gain of +1.95 and +1.45, respectively.

The paper notes that while Hymavi integrates multiple sophisticated mechanisms, its performance with limited training data remains a subject for future exploration.
The authors suggest that future work could focus on developing lightweight parallel variants of Hymavi.

Method evidence: "We propose the HYMA block, which integrates attention mechanisms and Mamba-based sequence modeling in parallel, allowing the model to simultaneously capture fine-grained, content-adaptive interactions and structured global context." (section: 2.1. Hybrid Mamba-Attention Block (HYMA))
Result evidence: "BraTS2023. As shown in Table 1, our method achieves the highest average DSC (91.92%) and delivers the best performance on tumor core (93.62%) and enhancing tumor (89.59%)." (section: 3.4. Results and Comparison)
Limitation evidence: "While Hymavi integrates multiple sophisticated mechanisms, its performance with limited training data remains a subject for future exploration." (section: 4. Conclusion)

Hymavi : A Hybrid Mamba-Attention Network in Multi-View Framework for Volumetric Medical Image Segmentation

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures