AI Research Portal

VIPACT (Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use) is an agent framework designed to enhance Vision-Language Models (VLMs) for fine-grained visual perception tasks. It achieves this by integrating multi-agent collaboration and specialized vision expert models, enabling more precise visual understanding and comprehensive System-2 reasoning. The core idea is to use an orchestrator agent to manage task analysis, planning, and coordination, specialized agents for detailed visual analysis, and vision expert models for high-precision perceptual information. VIPACT consistently outperforms state-of-the-art baselines across diverse visual perception benchmarks, demonstrating significant performance improvements.

The paper addresses several bottlenecks in current VLM capabilities for visual perception:

Struggle with fine-grained visual perception: State-of-the-art VLMs struggle with tasks requiring detailed pixel-level analysis, such as detecting line intersections or object boundaries, which are trivial for humans.

The paper claims the following key contributions:

VIPACT, a multi-modal agent framework that synergizes multi-agent collaboration with vision expert models to enhance fine-grained visual perception. It is an autonomous system capable of handling diverse visual perception tasks using a single prompt template, leveraging a VLM for task analysis, planning, and invoking multi-agent collaboration, with flexible plug-and-play modular components.
Experiments conducted across diverse visual perception benchmarks, demonstrating VIPACT's advantages over state-of-the-art baselines.
A systematic analysis of previous methods that proved effective in improving the general capabilities of foundation models for fine-grained visual perception, revealing their inconsistent effectiveness.
Comprehensive ablation studies to assess the impact of multi-agent collaboration, visual input for planning, and each component of VIPACT, along with a detailed error analysis identifying the limitations of current VLMs.

VIPACT is a VLM-based framework that integrates multi-agent collaboration and vision expert models for fine-grained visual perception tasks. It consists of three core components:

Orchestrator Agent: This agent manages the entire workflow. It performs Task Requirement Analysis and Planning by analyzing task requirements from images and queries, identifying key elements and critical visual features, and generating a detailed plan. It then handles Tool Selection and Incorporation of Specialized Agents, initiating collaboration with specialized agents or external vision expert models. After tasks are performed, it conducts Evidence Summarization by compiling and summarizing collected evidence, resolving conflicts, and double-checking factuality. Finally, it performs Final Answer Deduction by applying reasoning to the summarized evidence, potentially generating and executing Python code to derive the answer.
Specialized Agents: These agents provide task-specific, detailed visual information to the orchestrator agent through function calling. VIPACT incorporates three specialized agents:
- Focused Image Captioning: Generates detailed image descriptions, optionally emphasizing specific elements relevant to the task via a focus argument.
- Visual Prompt Description: Analyzes visual prompts within images (e.g., colored circles, bounding boxes, arrows, textual labels), generating detailed descriptions of their locations, characteristics, and targets.
- Focused Image Comparison: Analyzes multiple images, identifying similarities and differences with an optional focus on specific elements.
Vision Expert Models: These models provide specialized, task-specific, fine-grained perceptual information, addressing limitations of current VLMs. They return both textual data and processed images. The models used include:
- Visual Prompt Detector: Identifies and localizes annotated elements (circles, bounding boxes, highlighted regions) and returns their coordinates.
- Depth Estimator: Analyzes spatial relationships and relative distances of objects, returning a grey-scale depth image.
- Object Detection: Identifies and localizes objects, providing bounding boxes and textual information.
- Image Segmentation: Offers precise delineation of image regions, returning segmentation masks and textual information.
- Embedding-based Similarity Computation: Quantifies visual similarities across images by generating compact representations and similarity scores (e.g., cosine similarity).

The overall workflow, as described in Algorithm 1, involves initializing the orchestrator agent with a VLM and tools. It formats an initial prompt, then iteratively checks if any tool is required. If so, it selects the most useful tool, executes it, and adds new visual data if generated. Otherwise, it generates VLM output, interprets it, updates the prompt and state, and increments the iteration counter. Finally, it extracts and returns the answer.

Architecture: VIPACT consists of an orchestrator agent, specialized agents (focused image captioning, visual prompt description, focused image comparison), and vision expert models (visual prompt detector, depth estimator, object detection, image segmentation, embedding-based similarity computation).
Core VLM: The primary VLM used in the main paper is GPT-4o (OpenAI 2024). Other VLMs used in experiments include Gemini-1.5-Pro (Team et al. 2024), Claude-3.5-Sonnet (Anthropic 2024), LLaVA-OneVision-7B (Li et al. 2024a), InternVL-2-Pro (Chen et al. 2023d, 2024a), and Llama-3.2-90b-Vision (Dubey et al. 2024).
Key Equations: Not stated in the paper.
Loss/Objective: Not stated in the paper.
Optimization: Not stated in the paper.
Augmentation: Not stated in the paper.
Data Handling: Not stated in the paper.
Implementation Details: The orchestrator agent may generate Python code, which is then executed by an external Python interpreter. The prompts for specialized agents are in Appendix J. The complete function heads, including inputs, outputs, and descriptions for vision expert models, are provided in the initial prompt for the orchestrator agents in Appendix J.

Datasets:
- Blink (Fu et al. 2024): Features diverse visual tasks with visual prompts (bounding boxes, interleaved image-text formats) and multiple images in a single query.
- MMVP (Tong et al. 2024): A benchmark for evaluating visual grounding in VLMs, using image pairs from "CLIP-blind pairs" focusing on nine basic visual patterns.
Metrics:
- Accuracy (%)
Baselines:
- Text-based Prompting: Zero-shot, Chain-of-Thought (CoT), Least-to-Most (LtM), Tree-of-Thought (ToT).
- Few-shot in-context learning.
- Visual Prompting: Set-of-Mark (SoM).
- Vision language agentic frameworks: MM-ReAct, ViperGPT, VisProg.
Numeric Results (using GPT-4o):
- Blink Dataset (Average Accuracy %):
  - VIPACT (Ours): 73.79
  - Zero-shot: 63.74
  - COT: 65.85
  - LtM: 63.51
  - TOT: 64.94
  - SOM: 60.13
  - MM-ReAcT: Not stated in the paper (only partial results reported, e.g., 30.00 for Count, 0.81 for Depth)
  - ViperGPT: Not stated in the paper (only partial results reported, e.g., 29.17 for Count, 0.00 for Depth)
  - VisProg: Not stated in the paper (only partial results reported, e.g., 3.33 for Count, 0.00 for Depth)
- MMVP Dataset (Accuracy %):
  - VIPACT (Ours): 70.7
  - Zero-shot: 68.0
  - COT: 61.0
  - LtM: 66.0
  - TOT: 66.0
  - SOM: 62.0
  - MM-ReAct: 6.67
  - ViperGPT: 53.0
  - VisPro: 39.0
Runtime/Training Cost: Not stated in the paper.

The paper explicitly reports the following limitations based on error analysis of GPT-4o's visual perception capabilities:

Failure to perceive small object parts (17%): The model often overlooks small, semantically important components crucial for precise visual understanding.
Difficulty distinguishing closely positioned visual prompts (15%): The model struggles to differentiate spatially proximate visual prompts.
Challenges in fine-grained spatial reasoning (24%): Tasks requiring high spatial resolution highlight the model's bias towards foreground objects over backgrounds; e.g., misinterpreting a highlight meant for the sky near a car as associated with the car.
Misinterpretation of relative object positions (14%): Errors arise when object arrangements differ from real-world expectations, as the model often lacks the ability to infer spatial relations from objects' perspectives, focusing on camera viewpoint.
Failure to recognize object orientation (13%): Difficulty discerning object orientation leads to errors in recognizing object parts, such as distinguishing left/right bicycle pedals.
Other errors (17%): This includes other issues like failure to detect subtle color differences, inaccuracies in multi-image fine-grained structure correspondence, and instances of refusal or instruction misinterpretation.
Lack of spatial intelligence or imaginative abilities: The model lacks the spatial intelligence or imaginative abilities necessary to infer relative object positions beyond their pixel positions (from the camera's perspective), especially in real-life scenes. These limitations hinder the model's ability to accurately interpret visual prompts and process tasks involving multiple image inputs.

Method evidence: "VIPACT consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks such as image captioning and vision expert models that provide high-precision perceptual information." (section: Abstract)
Result evidence: "Our VIPACT consistently outperforms baselines on all tasks." (section: Table 2: Results for visual reasoning tasks in Blink using GPT-40.)
Limitation evidence: "Failure to perceive small object parts (17%): The model often overlooks small, semantically important components crucial for precise visual understanding." (section: 6 Error Analysis)

VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures