AI Research Portal

DeformTrace is a novel framework for Temporal Forgery Localization (TFL) that integrates State Space Models (SSMs) with deformable dynamics and relay mechanisms. It addresses challenges in TFL such as ambiguous boundaries, sparse forgeries, and limited long-range modeling in SSMs. The core idea involves a Deformable Self-SSM (DS-SSM) for dynamic receptive fields, a Relay Token Mechanism to mitigate long-range decay, and a Deformable Cross-SSM (DC-SSM) for cross-sequence interactions. DeformTrace achieves state-of-the-art performance on standard benchmarks with fewer parameters, faster inference, and stronger robustness.

The paper identifies several bottlenecks in existing Temporal Forgery Localization (TFL) methods and State Space Models (SSMs):

Ambiguous Boundaries: Forgery boundaries are often unclear, unlike well-defined boundaries in tasks like Temporal Action Detection (TAD). Standard SSMs use fixed state updates, leading to temporal smoothing that reduces localization precision.
: Most frames in a video are non-forged, causing SSMs' recursive updates to be dominated by non-forgery patterns, which weakens sensitivity to sparse forgeries.

We propose Deformable Self-SSM (DS-SSM), the first to introduce dynamic receptive fields into temporal state space models, improving localization of ambiguous boundaries.
We introduce a Relay Token mechanism that explicitly mitigates the long-range decay of SSMs, which is a key limitation in prior state-space models.
We propose Deformable Cross-SSM (DC-SSM), the first to incorporate cross-sequence interactions into state space modeling, enhancing sensitivity to sparse forgeries.
We integrate these components into a unified hybrid TFL architecture that combines the strengths of Transformers and SSMs. Extensive experiments on standard benchmarks demonstrate state-of-the-art performance with lower model size and faster inference.

DeformTrace adopts a query-based architecture, building upon TadTR, for temporal forgery localization and video-level classification.

Feature Extraction: Video and audio streams are processed by frozen, pretrained visual and audio backbones to extract classification tokens and feature sequences. These are concatenated and linearly projected to form fused features, which are further processed by downsampling layers to obtain multi-scale temporal context.

Encoder: The encoder receives flattened features and outputs a context-enhanced sequence. It comprises a Deformable Self-SSM (DS-SSM), a Deformable Self-Attention (DSA), and a Feed-Forward Network (FFN).

Deformable Self-SSM (DS-SSM): This module integrates a deformable dynamic receptive field mechanism into SSMs. It predicts offsets at each time step to dynamically sample input features, reducing computational overhead while preserving SSMs' low complexity. It uses Deformable Self-Scanning, Forward-Backward SSM (FB SSM), and an FFN. Deformable Self-Scanning allows each token to flexibly sample semantically relevant positions beyond local windows. FB SSM performs bidirectional state-space propagation.
- The normalized reference location pn for the n-th feature in the l-th layer is computed as: pn = ω · 2^(l-1) · (n + 0.5) / (fps · d) where fps is frames per second (default 25), ω is the stride, and d is the video duration in seconds.
Relay Token Mechanism: Learnable global tokens are evenly inserted into the input sequence before state-space updates. These tokens partition the sequence into subspaces, allowing local states to pass information to the relay token, which then broadcasts the aggregated message to other subspaces, forming a sparse sequence-to-token information flow to mitigate long-range decay.

Decoder: The decoder fuses visual and audio classification tokens via an MLP into a global multimodal feature, which forms initial queries. It has M layers, each containing Deformable Cross-SSM, Multi-Head Self-Attention (MHSA), Deformable Cross-Attention (DCA), and FFN.

Deformable Cross-SSM (DC-SSM): This module introduces cross-sequence interactions into deformable state space modeling. Each auxiliary token, representing a potential forgery, serves as a query to retrieve forgery-relevant information from the main stream sequence. This partitions the global state space into query-specific subspaces, reducing non-forgery accumulation and boosting sensitivity to sparse forgeries.

Architecture:
- Visual and audio backbones: Raven's visual and audio encoders, specifically the base model pretrained on VoxCeleb2 and LRS3.
- Feature levels: L = 6 feature levels via 1D convolution with kernel size 3.
- Decoder layers: M = 3 decoder layers.
- Deformable self-/cross-SSM layer sampling: N = 6 points per scale.
- Query tokens: Nq = 60.
- Relay tokens: Nr = 8.
- Input sequence length: T = N₁ = 200 frames (8 seconds at 25 fps).
- Feature dimensionality: C = 256.
Key Equations:
- Normalized reference location: pn = ω · 2^(l-1) · (n + 0.5) / (fps · d)
- Offset prediction: o = MLP1(fn)
- Deformable points: p = Repeat(pn) + o
- Output feature: fh = MLP2 (Φ (p, {F^l_j}^L_{j=1}))
Loss/Objective:
- Overall training loss: L = Lmatch + Lcls + λ1 · Lenh + λ2 · Lcoop
- Task-aware loss: Lmatch (A^M, A) = Σ^Nf_{i=1} Σ_{u∈{ce,reg}} L_u (A^M_{σ(i)}, A_i) (Hungarian Matching loss, including classification loss Lce and regression loss Lreg).
- Video-level cross-entropy loss: Lcls.
- Enhance loss: Lenh = 1/Nr Σ^Nr_{k=1} cos (rk, f^avg_k)
- Cooperation loss: Lcoop = min_{i<j} (ri; rj)² ≈ ||G – I||² = Σ_{i≠j} G²_{ij} + Σ^Nr_{k=1} (G_kk - γ)² (approximates minimizing mutual information between relay tokens).
Optimization:
- Optimizer: AdamW.
- Learning rate: 2e-4.
- Scheduler: Cosine scheduler with 5-epoch warm-up.
- Epochs: 30 for AV-Deepfake1M, 100 for LAV-DF.
- Batch size: 32.
- Hyperparameters: λ1 = 0.5, λ2 = 0.2.
Data Handling:
- Video sampling: 25 fps.
- Audio sampling: 16 kHz.
- Visual frames: Resized to 224 x 224 pixels.
- Audio: Converted into 80-bin mel-spectrograms using a 40 ms Hamming window with a 40 ms hop size.
- Input span: 8 seconds.
Implementation Details:
- Hardware: Eight NVIDIA RTX 3090 GPUs.

Datasets:
- LAV-DF (Cai et al. 2023): 78K train, 31K val, 26K test samples.
- AV-Deepfake1M (Cai et al. 2024): 746K train, 57K val, 343K test samples.
Metrics:
- Temporal Forgery Localization: mean Average Precision (mAP) at IoU thresholds {0.5, 0.75, 0.9, 0.95}, mean Average Recall (mAR) at different numbers of proposals {5, 10, 20, 30, 50, 100}.
- Video-level detection: Area Under the Curve (AUC).
- Efficiency: Total parameters, trainable parameters, FLOPs, inference time.
Baselines:
- LAV-DF: BA-TFD, BA-TFD+, UMMAFormer, DiMoDif, FullFormer (Ours, Transformer baseline).
- AV-Deepfake1M: BA-TFD, BA-TFD+, UMMAFormer, DiMoDif, FullFormer (Ours, Transformer baseline).
Numeric Results:
- LAV-DF (Table 1):
  - DeformTrace (Ours): mAP@0.5: 97.1, mAP@0.75: 90.7, mAP@0.95: 38.1, mAP Avg.: 75.3. mAR@100: 93.3, mAR@50: 93.1, mAR@20: 92.8, mAR@10: 92.3, mAR Avg.: 92.9.
  - Best baseline (UMMAFormer): mAP@0.5: 98.8, mAP@0.75: 95.5, mAP@0.95: 37.6, mAP Avg.: 77.3. mAR@100: 92.4, mAR@50: 92.5, mAR@20: 92.5, mAR@10: 92.1, mAR Avg.: 92.3.
- AV-Deepfake1M (Table 2):
  - DeformTrace (Ours): mAP@0.5: 92.0, mAP@0.75: 78.7, mAP@0.95: 31.2, mAP Avg.: 52.9. mAR@50: 86.2, mAR@30: 84.2, mAR@20: 81.4, mAR@10: 79.0, mAR@5: 78.0, mAR Avg.: 81.8.
  - Best baseline (DiMoDif): mAP@0.5: 86.9, mAP@0.75: 76.0, mAP@0.95: 28.7, mAP Avg.: 49.3. mAR@50: 81.6, mAR@30: 80.9, mAR@20: 80.3, mAR@10: 78.8, mAR@5: 76.6, mAR Avg.: 79.6.
Runtime/Training Cost (Table 3, on single RTX 3090 GPU):
- DeformTrace (Ours): Total Parameters: 118.3M, Trainable Parameters: 20.8M, FLOPs: 212.4G, Inference Time: 104ms.
- UMMAFormer: Total Parameters: 165.9M, Trainable Parameters: 49.72M, FLOPs: 1563.9G, Inference Time: 857ms.
- BA-TFD+: Total Parameters: 152.9M, Trainable Parameters: 152.9M, FLOPs: 218.2G, Inference Time: 681ms.

No explicit paper-reported limitations were found.

Method evidence: "We propose DeformTrace, which enhances SSMs with deformable dynamics and relay mechanisms to address these challenges. Specifically, Deformable Self-SSM (DS-SSM) introduces dynamic receptive fields into SSMs for precise temporal localization. To further enhance its capacity for temporal reasoning and mitigate long-range decay, a Relay Token Mechanism is integrated into DS-SSM. Besides, Deformable Cross-SSM (DC-SSM) partitions the global state space into query-specific subspaces, reducing non-forgery information accumulation and boosting sensitivity to sparse forgeries." (section: Abstract)
Result evidence: "DeformTrace (Ours) 118.3 20.8 212.4 104" (section: Table 3)
Limitation evidence: No explicit paper-reported limitations were found.

DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures