AI Research Portal

Method

AutoDW is a novel execution framework that orchestrates operations for complex document workflow automation through three core modules: stepwise planning, API execution & state tracking, and adaptive rollback.

Method Flow:

Stepwise Planning: Given an initial document state (S₀) and a session of natural language instructions (I = {I₁, I₂, ...}), AutoDW incrementally plans API calls (P = [A₁, A₂, ...]) one at a time. This involves a two-stage strategy:
- Sub-instruction generation: An LLM generates a sub-instruction, corresponding to a single atomic document operation, conditioned on the task objective, original user instruction, previously completed API calls, current document state, detailed guidelines, and JSON output format. This bridges the semantic gap and narrows the API search space.
- API generation: Once a sub-instruction is generated, AutoDW classifies its underlying intent using a fine-tuned BERT model for 8-way intent classification. The top-3 predicted intents are retained. APIs associated with these intents are retrieved and, along with the sub-instruction and other prompt engineering elements, fed into an LLM to generate the final API action (API and its full arguments).
API Execution and State Tracking: The selected API is executed within a Python runtime environment. The system tracks the updated document state, St, which is represented as a tuple of dict objects describing document elements: St = (Ddoc, Depar, Dtable, Dimage, Dpage, Dint, Dstyle) where each D is a list of dict objects for a specific type of component. State parsing failures during execution are treated as invalid outcomes, triggering an API-level rollback.
Adaptive Rollback: This module handles fault recovery.
- Change analysis: An analyze_change function detects changes between document states (St-1, St) after an API call. It outputs a tuple representing various changes: Δ = (As, AC, ∆F, ∆St, ΔΤ, ΔΗ) These represent structural, content, format, style, table, and hyperlink changes, respectively. The content analyzer uses Python's SequenceMatcher, and table analysis uses cell signature comparison: sigi,j = hash(content||structure||position||merge)
- Alignment validation: The analyzed state change (Δ), sub-instruction, current document state, and previously executed APIs are passed to an LLM-based validator. The validator LLM returns a binary decision (pass/fail), a confidence score ([0, 1]), and a textual explanation. An action is accepted if the LLM decision is 'pass' with high confidence (empirically tuned threshold of 0.6) or if the confidence score is low (action accepted by default). Otherwise, rollback is initiated.
- Rollback strategies: AutoDW supports two types of rollback:
  - Argument-level rollback: Regenerates the action using the same API but with updated arguments, considering the validator's explanation.
  - API-level rollback: If argument-level rollback fails validation, a new API is selected altogether. AutoDW defaults to a single round of dual-level rollback.

Architecture:
- Intent Classifier: A fine-tuned 178M BERT model (Devlin et al. 2019) for 8-way intent classification (content creation, content modification, table operation, image operation, chart operations, format/style editing, document structure update, and document lifecycle update). It is fine-tuned on 3,315 instruction-intent pairs and achieves a test accuracy of 98%.
- LLMs: DeepSeek-v3 (default), Qwen-Plus, Gemini 2.5 Pro, and GPT-4.1 are used for evaluation.
- Embedding Model: text-embedding-v4 is used for embedding-based retrieval.
Optimization: Not stated in the paper.
Data Handling:
- DWBench: A benchmark consisting of 250 multi-turn sessions and 1,708 human-annotated instructions. Each instruction is associated with a turn ID, user instruction, initial document state file, human-annotated feasible API calls, and corresponding expected document state.
- APIs: 74 APIs are implemented to support document workflows. Each session averages 34.8 APIs (min 15, max 75). Most instructions require multiple API calls, with a mean of 5.1 API calls per instruction.
Implementation Details:
- Temperature: Fixed at 0.1 for all LLM generation experiments.
- Rollback Threshold: An empirically tuned confidence score threshold of 0.6 is used for alignment validation.
- Rollback Rounds: AutoDW defaults to a single round of dual-level rollback (argument-level then API-level).

Results

Datasets:
- DWBench: 250 sessions, 1,708 human-annotated instructions.
Metrics:
- Instruction-level accuracy (iACC)
- Session-level accuracy (sACC)
- Average number of API calls
- Average token usage
Baselines:
- Retrieval-only
- Reasoning-only
- Hybrid (multi-stage pipeline from PPTC (Guo et al. 2024))
Numeric Results:
- Overall Performance Comparison:
  - Retrieval-only: 13.84% iACC, 4.40% sACC, 4.82 #APIs, 29.6k #Tokens (instruction-level); 15.50 #APIs, 98.7k #Tokens (session-level).
  - Reasoning-only: 39.93% iACC, 25.20% sACC, 5.12 #APIs, 31.6k #Tokens (instruction-level); 34.98 #APIs, 175.4k #Tokens (session-level).
  - Hybrid: 64.46% iACC, 35.20% sACC, 5.30 #APIs, 36.5k #Tokens (instruction-level); 36.21 #APIs, 225.2k #Tokens (session-level).
  - AutoDW (ours): 90.33% iACC, 62.00% sACC, 5.21 #APIs, 42.8k #Tokens (instruction-level); 34.45 #APIs, 284.9k #Tokens (session-level).
  - AutoDW vs. Hybrid: +40.1% iACC, +76.1% sACC, -1.7% #APIs (instruction-level), +17.4% #Tokens (instruction-level); -4.9% #APIs (session-level), +26.5% #Tokens (session-level).
- Robustness Study (by LLM):
  - Qwen-Plus: 82.82% iACC, 53.60% sACC, 6.15 #APIs, 40.1k #Tokens (instruction-level); 38.60 #APIs, 265.4k #Tokens (session-level).
  - DeepSeek-v3: 90.33% iACC, 62.00% sACC, 5.21 #APIs, 42.8k #Tokens (instruction-level); 34.45 #APIs, 284.9k #Tokens (session-level).
  - Gemini-2.5-Pro: 91.79% iACC, 67.20% sACC, 4.92 #APIs, 42.7k #Tokens (instruction-level); 32.88 #APIs, 286.0k #Tokens (session-level).
  - GPT-4.1: 92.03% iACC, 65.20% sACC, 5.14 #APIs, 43.5k #Tokens (instruction-level); 34.82 #APIs, 294.7k #Tokens (session-level).
- Robustness Study (by Difficulty):
  - DeepSeek-v3: 94.54% iACC (Simple), 90.02% iACC (Mediate), 86.33% iACC (Hard).
  - Gemini-2.5-Pro: 95.08% iACC (Simple), 93.46% iACC (Mediate), 86.84% iACC (Hard).
  - GPT-4.1: 95.77% iACC (Simple), 92.48% iACC (Mediate), 87.85% iACC (Hard).
  - Hard instructions (≥ 6 APIs) performance is only 4.4% lower than overall instruction-level accuracy for the three stronger LLMs.
- Ablation Study on Rollback:
  - wo. rollback: 71.8% iACC, 35.6% sACC, 31.5k #Tokens (instruction-level); 183.1k #Tokens (session-level).
  - arg. rollback: 76.8% iACC, 43.6% sACC, 44k #Tokens (instruction-level); 215.2k #Tokens (session-level).
  - AutoDW (single-round dual-level): 90.3% iACC, 62.0% sACC, 42.8k #Tokens (instruction-level); 284.9k #Tokens (session-level).
  - Two-round rollback: 91.4% iACC, 64.8% sACC, 48.6k #Tokens (instruction-level); 337k #Tokens (session-level).
  - Single-round dual-level rollback provides most of the accuracy benefit (+74% relative improvement in session-level success from 35.6% to 62%) while keeping token usage affordable (+56%).
Runtime/Training Cost:
- AutoDW achieves its performance with an extra of 25.6% tokens compared to the Hybrid baseline.
- The single-round dual-level rollback strategy is chosen as the most cost-effective solution, as extending to two-round yields only marginal improvements (1.1% iACC and 2.8% sACC) for a significant increase in token usage.

Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures