AI Research Portal

This paper introduces MTRL-CG, a multi-task reinforcement learning (RL) method that addresses the negative interference issue in multi-task RL by grouping related tasks and separating conflicting ones. The core idea is to quantify inter-task affinity using gradient updates within a shared model, construct an affinity matrix, and then apply spectral clustering to partition tasks into groups. Each task group is subsequently trained with a dedicated policy network, built upon the Soft Actor-Critic (SAC) algorithm, to enable focused learning and mitigate interference. Extensive experiments on the Meta-World benchmark demonstrate that MTRL-CG consistently improves the overall performance of various state-of-the-art multi-task RL methods.

Existing multi-task RL methods face several bottlenecks:

Negative Interference: They often adopt a fully shared policy across all tasks without explicitly distinguishing between related and conflicting ones. This leads to updates beneficial to one task adversely affecting others.

First work to introduce task grouping in multi-task RL, aiming to cluster related tasks and separate conflicting ones to mitigate negative interference.
Proposal of the MTRL-CG method, which leverages spectral clustering to group tasks and performs group-wise training by optimizing separate policy networks for each group, thereby enhancing and reducing negative interference.
Extensive experiments on the Meta-World benchmark, demonstrating superior overall performance compared to state-of-the-art multi-task RL methods.

The MTRL-CG framework consists of two main components: task grouping and group-wise multi-task reinforcement learning.

Task Grouping Stage:

Inter-Task Affinity Computation: The method quantifies the mutual influence among tasks by computing an affinity score for each task pair. This is done by measuring the relative change in the Q-value of task Tj induced by applying a gradient update from task Ti to a shared Q-function. These affinities are then assembled into an inter-task affinity matrix W. A symmetric affinity matrix W is constructed by averaging W with its transpose WT.
- Key Equation (Affinity Score): Wi,j = |Qθτ; (s, a) – Qθ(s, a)| / |Qθ(s,a)|, (s, a) ~ Tj
- Key Equation (Symmetric Affinity Matrix): W = (W + WT) / 2
Spectral Embedding: Given the symmetric affinity matrix W, a signed spectral embedding method is used. This involves:
- Constructing signed affinity matrices (W+ and W-) by separating positive and negative components of W.
- Constructing corresponding signed Laplacians (L+ and L-).
- Solving a generalized eigenvalue problem to compute the spectral embedding matrix V, where each row represents a task in the spectral space.
k-means Clustering: The rows of the spectral embedding matrix V serve as low-dimensional spectral representations for each task. These representations are then clustered using the k-means algorithm to partition the N tasks into K groups.

Group-wise Training Stage:

Each task group Gk is assigned a dedicated Soft Actor-Critic (SAC) learner, comprising a policy function πφk and a Q-function Qθk.
The policy for each group is optimized by maximizing the expected task-specific objective over tasks within that group.
The Q-function for each group is optimized by minimizing the expected Q-loss over tasks within that group.
This approach facilitates intra-group knowledge sharing while mitigating inter-group interference.

Base Algorithm: Soft Actor-Critic (SAC) is employed for policy and value function optimization.
Policy Objective (SAC): Jπ() = Est~~D [Eat~~πφ [alog πφ(at|St) – Qo(t, at)]], where α is an entropy temperature coefficient.
Q-function Objective (SAC): JQ(θ) = E(st,at)~~D[(Qθ(St, at) - (rt + Est+1~~P[Vō(St+1)]))²], where γ is the discount rate.
Multi-Task Objectives: SAC objectives are extended to multi-task settings by maximizing the average expected return across tasks sampled from a distribution p(T) for the policy, and minimizing the average Q loss over tasks for the Q-function.
Affinity Calculation: Inter-task affinity is quantified by measuring the relative change in the Q-value of task Tj induced by applying a gradient update from task Ti to the shared parameters. The update uses a learning rate η.
Spectral Clustering Hyperparameters: The generalized eigenvalue problem for spectral embedding involves hyperparameters τ+ and τ-.
Clustering Algorithm: k-means++ algorithm is used for initializing K cluster centers.
Implementation: The proposed MTRL-CG framework is built upon the SAC algorithm and can be seamlessly incorporated into existing SAC-based multi-task RL methods.
Code Availability: Our code is available at: https://github.com/zhangt603/MTRL-CG.git.

Environments: Meta-World, an open-source simulated environment for multi-task RL, featuring manipulation tasks with the Sawyer robotic arm. Experiments conducted on:
- MT10 (10 distinct manipulation tasks)
- MT50 (50 distinct manipulation tasks)
- Both MT10 and MT50 include "fixed" (fixed initial and goal positions) and "mixed" (randomly sampled initial and goal positions) settings.
Metrics: Well-defined success rate in Meta-World. Results are averaged across 3 random runs.
Baselines: MT-SAC, CARE, CMTA, and PaCo.
Numeric Results (Comparison with State-of-the-art Methods):
- MT10-Fixed:
  - MT-SAC+MTRL-CG: Max success rate 0.87 ± 0.01, Final success rate 0.77 ± 0.04. (Original MT-SAC: Max 0.63 ± 0.02, Final 0.50 ± 0.11)
  - CARE+MTRL-CG: Max success rate 0.97 ± 0.02, Final success rate 0.87 ± 0.02. (Original CARE: Max 0.77 ± 0.02, Final 0.77 ± 0.02)
  - CMTA+MTRL-CG: Max success rate 0.93 ± 0.02, Final success rate 0.83 ± 0.09. (Original CMTA: Max 0.77 ± 0.02, Final 0.73 ± 0.02)
  - PaCo+MTRL-CG: Max success rate 0.73 ± 0.06, Final success rate 0.67 ± 0.05. (Original PaCo: Max 0.73 ± 0.05, Final 0.57 ± 0.02)
- MT10-Mixed:
  - MT-SAC+MTRL-CG: Max success rate 0.77 ± 0.04, Final success rate 0.64 ± 0.02. (Original MT-SAC: Max 0.70 ± 0.04, Final 0.68 ± 0.03)
  - CARE+MTRL-CG: Max success rate 0.87 ± 0.02, Final success rate 0.73 ± 0.02. (Original CARE: Max 0.74 ± 0.03, Final 0.69 ± 0.06)
  - CMTA+MTRL-CG: Max success rate 0.83 ± 0.01, Final success rate 0.53 ± 0.01. (Original CMTA: Max 0.88 ± 0.02, Final 0.58 ± 0.12)
  - PaCo+MTRL-CG: Max success rate 0.89 ± 0.01, Final success rate 0.84 ± 0.01. (Original PaCo: Max 0.85 ± 0.02, Final 0.81 ± 0.01)
Runtime/Training Cost:
- Under MT10-Fixed, MT-SAC converges at approximately 0.30 × 10^7 timesteps, while MT-SAC with MTRL-CG converges at around 0.13 × 10^7 timesteps.
Ablation Study (Grouping Strategy): MTRL-CG achieves the highest max and final success rates on MT10-Fixed and MT10-Mixed compared to "w/o Grouping", "HOA", and "Random" strategies.
- CARE+MTRL-CG (3 groups) achieves Max 0.97 ± 0.02, Final 0.87 ± 0.02 on MT10-Fixed.
- CARE+MTRL-CG (3 groups) achieves Max 0.87 ± 0.02, Final 0.73 ± 0.02 on MT10-Mixed.
Ablation Study (Group Number): MTRL-CG with 3 groups achieves the best overall performance, requiring fewer timesteps to reach higher success rates.

The relatively poor performance of CMTA with MTRL-CG in MT10-Mixed may be due to its design, which amplifies Q value sensitivity to randomness and results in unstable task grouping.
The lower success rate of PaCo with MTRL-CG on MT50-Fixed may stem from its sensitivity to hyperparameter settings.

Method evidence: "In this paper, we propose a Multi-Task Reinforcement Learning method with spectral Clustering-based task Grouping (MTRL-CG), which utilizes spectral clustering to organize related tasks into groups and separate conflicting ones, enabling group-wise policy learning that mitigates negative interference." (section: Abstract)
Result evidence: "For example, under the MT10-Fixed setting, MT-SAC converges at approximately 0.30×10^7 timesteps, while MT-SAC with MTRL-CG converges at around 0.13 × 10^7 timesteps." (section: Experimental Results)
Limitation evidence: "The relatively poor performance of CMTA with MTRL-CG in MT10-Mixed may be due to its design, which amplifies Q value sensitivity to randomness and results in unstable task grouping. Similarly, the lower success rate of PaCo with MTRL-CG on MT50-Fixed may stem from its sensitivity to hyperparameter settings." (section: Experimental Results)

MTRL-CG: Multi-Task Reinforcement Learning Method with Spectral Clustering-Based Task Grouping

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures