AI Research Portal

This paper introduces CoT-VLNBench, the first large-scale benchmark and dataset for visual chain-of-thought (CoT) reasoning in quadruped robot navigation. The benchmark features diverse indoor and outdoor scenes, multi-step navigation trajectories, and natural language instructions with fine-grained CoT reasoning traces. Alongside the benchmark, the authors propose CoT-VLN, a novel 7B Vision-Language-Navigation (VLN) model that integrates visual, linguistic, and reasoning modules. Empirical results demonstrate that CoT-VLN significantly outperforms existing non-VLM baselines and non-reasoning approaches on the new benchmark, highlighting the importance of CoT reasoning for interpretable and effective embodied navigation.

The paper identifies several bottlenecks in existing robot-centric datasets and benchmarks:

Lack of support for vision-language navigation (VLN) and advanced reasoning: Most current robot-centric datasets, such as CODa, SIT, and JRDB, primarily focus on traditional 3D tasks like perception and prediction, lacking the necessary annotations and task structures for VLN.

CoT-VLNBench: The first large-scale dataset and benchmark for visual chain-of-thought reasoning in indoor and outdoor quadruped robot navigation.
Fine-grained CoT annotations: Provides annotations that align visual observations, language instructions, and intermediate reasoning steps, enabling interpretable multi-step navigation.
Novel CoT-VLN model: Introduces a model that integrates visual, language, and reasoning modules, demonstrating its effectiveness on the new benchmark.

The CoT-VLNBench dataset and the CoT-VLN model are designed to facilitate visual chain-of-thought reasoning for quadruped robot navigation.

Data Collection and Annotation Pipeline: Data is collected using a Unitree Go2 robot in diverse indoor and outdoor environments. The annotation process for CoT reasoning traces is systematically decomposed into four hierarchical steps, powered by GPT-4.1 and refined through rule-based and manual annotation:

Step 1: Environmental Perception: Focuses on understanding and describing the overall scene context, including weather, time, scene type, and general orientation of each view relative to the ego robot.
Step 2: Key Agents Analysis: Provides detailed analysis of static road features and movable objects, explicitly introducing depth information for comprehensive 3D spatial understanding.
Step 3: Moving Decision Reasoning: Innovatively introduces a multiple-choice format to abstract future trajectories and actions into a finite set of options (Servo status and Path status) to enhance accuracy and consistency.
Step 4: Trajectory Prediction Output: Outputs the final structured action trajectory prediction results, specifically waypoints for the next 3 seconds.

CoT-VLN Framework: The CoT-VLN model is based on a state-of-the-art VLN model, Qwen2.5-VL 7B.

Architecture: The model freezes the parameters of the image encoder and trains only the projection layer and the language model.
Training: The CoT-VLN model is optimized using supervised fine-tuning (SFT).
Inference: During inference, the model adopts a step-by-step CoT process, triggered by the keyword "think," to perform complex reasoning and produce human-like answers.

Robot Setup: A manually modified Unitree Go2 robot was used for data collection.
- Lidar: Hesai XT32 rotating 3D Lidar, operating at 10 Hz.
- Cameras: Five SG2-AR0231C-0202-GMSL-H150 cameras, capturing 1920 × 1080 pixels with 150° field of view, covering 360° view at 10 Hz.
- IMU/GNSS: One IMU and GNSS module, providing fused pose data at 500 Hz.
- Software: Deployed using a Docker image based on Ubuntu 20.04 and ROS2 Humble.
- Calibration: Precise intrinsic and extrinsic calibration performed for all sensors.
Data Collection:
- Scenes: 60 scenes, including densely populated areas (intersections, sidewalks, subway entrances).
- Sequences: Each sequence contains 300 seconds of continuous data sampled at 10 Hz.
- Total Data: 875K images, 175K Lidar frames, approximately 5.25M 3D bounding boxes with object ID annotations, and 8.75 million pairs of fine-grained language annotations.
- Duration: Over 5 hours.
Annotation Labels:
- 3D Bounding Box Labels: Cuboids in Lidar coordinates, derived from 360-degree multi-view cameras and Lidar, with object IDs. Perception annotation range up to 30 meters.
- Categories: Car, Bus, Bicycle, Pedestrian, Half-body, and Cone.
- Meta States: Servo status (Ss) and Path status (Ps) are defined for future trajectory points over the next 3 seconds.
  - Ss = [Keep, Accelerate, Decelerate, Stop]
  - Ps = [Right Turn, Left Turn, Straight, IDLE]
Benchmark Construction:
- Training Instances: 140K (80%)
- Validation Instances: 35K (20%)
- Navigation Task: Predict waypoints for the next 3s (6 sets of trajectory points at 0.5s intervals).
Evaluation Metrics:
- L2 Error: Measured in meters between predicted (pt) and ground-truth (pt) trajectories at time t.
  - L2t = ||pt - pt ||2
  - L2avg = (1/T) Σt=1 to T ||pt - pt||2
- Subjective Evaluation: GPT-4.1 used as an automatic judge to score each step on a scale of 1 to 100 based on factual accuracy, logical consistency, and semantic completeness.
Training Details: All models trained on 8 NVIDIA L40 GPUs.

Datasets:
- CoT-VLNBench (Ours): 175K/5hr 3D Ann frames, 5.25M 3D Bbox, 875K VQA Pairs, ✓ CoT.
- SIT (Bae et al. 2023): 12K/0.33hr 3D Ann frames, 320K 3D Bbox, X VQA Pairs, X CoT.
- CODa (Zhang et al. 2024): 32K/1hr 3D Ann frames, 1.1M 3D Bbox, X VQA Pairs, X CoT.
- OpenLORIS (Shi et al. 2020): X 3D Ann frames, X 3D Bbox, X VQA Pairs, X CoT.
- Rellis-3D (Jiang et al. 2021): 13K/0.3hr 3D Ann frames, X 3D Bbox, X VQA Pairs, X CoT.
- JRDB (Martin-Martin et al. 2021): 60K/2hr 3D Ann frames, 1.8M 3D Bbox, X VQA Pairs, X CoT.
Metrics: L2 Error (m) ↓, Scores (%) ↑.
Baselines (VLM-based): InternVL3-8B, Qwen2.5-VL, Janus-Pro, LLaVA-1.6.
Baselines (Non-VLM): SparseDrive, DiffusionPlanner.
Numeric Results (VLM-based, L2 Error (m)):
- Qwen2.5-VL (Bai et al. 2025) with Reasoning: 0.14 (1s), 0.28 (2s), 0.47 (3s), 0.29 (Avg.).
- Qwen2.5-VL (Bai et al. 2025) Non-Reasoning: 0.16 (1s), 0.30 (2s), 0.52 (3s), 0.33 (Avg.).
- Qwen2.5-VL (Bai et al. 2025) Lang. Navi. Non-Reasoning: 0.20 (1s), 0.34 (2s), 0.62 (3s), 0.39 (Avg.).
- Average L2 Error (m) for all models:
  - Reasoning (Vision Lang. Navi.): 0.34
  - Non-Reasoning (Vision Lang. Navi.): 0.37
  - Non-Reasoning (Lang. Navi.): 0.42
Numeric Results (VLM-based, Scores (%) by GPT-4.1):
- Qwen2.5-VL (Bai et al. 2025): 75.67.
- InternVL3-8B (Zhu et al. 2025): 74.33.
- Janus-Pro (Chen et al. 2025): 69.56.
- LLaVA-1.6 (Liu et al. 2024b): 66.48.
Numeric Results (Non-VLM, L2 Error (m)):
- CoT-VLN(Qwen2.5-VL): 0.14 (1s), 0.28 (2s), 0.47 (3s), 0.29 (Avg.).
- SparseDrive (Sun et al. 2024): 0.20 (1s), 0.41 (2s), 0.65 (3s), 0.42 (Avg.).
- DiffusionPlanner (Zheng et al. 2025): 0.17 (1s), 0.32 (2s), 0.61 (3s), 0.37 (Avg.).
Key Findings: All models achieved highest accuracy under CoT reasoning. Qwen2.5-VL attained the best performance with an average error of 0.29m. Direct inference without CoT reasoning led to a 13.7% (0.04m) performance drop for Qwen2.5-VL. Language-only models without CoT reasoning showed the lowest accuracy (0.39m). Models without reasoning capabilities exhibited an 8.8% increase in average L2 error, and absence of visual input led to a 13.5% decrease in accuracy.

No explicit paper-reported limitations were found.

Method evidence: "Specifically, as illustrated in Figure 4, we systematically decompose the reasoning chain into four hierarchical steps: Step 1: Environmental Perception. Focuses on understanding and describing the overall scene context. Step 2: Key Agents Analysis. Provides detailed analysis of static road features and movable objects in the scene, explicitly introducing depth information to enable comprehensive 3D spatial understanding. Step 3: Walking Decision Reasoning. In this critical step, we innovatively introduce a multiple-choice format, abstracting future trajectories and action decisions into a finite set of options to enhance the accuracy and consistency of the reasoning process. Step 4: Trajectory Prediction Output. Outputs the final structured action trajectory prediction results." (section: VQA Pair Generation)
Result evidence: "Overall, all models achieved their highest accuracy under the CoT reasoning setting, with Qwen2.5-VL (Bai et al. 2025) attaining the best performance, reaching an average error of 0.29m." (section: Main Results)
Limitation evidence: Not stated in the paper.

CoT-VLNBench: A Benchmark for Visual Chain-of-Thought Reasoning in Vision-Language-Navigation Robots

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures