AI Research Portal

This paper introduces Offline Multi-Objective Bandits (OffMOB), a comprehensive framework for learning Pareto-optimal policies from static, logged contextual bandit data. The core idea is to integrate the pessimism principle with multi-objective optimization, using neural networks to model vector-valued reward functions and directly approximate the entire Pareto front. OffMOB learns a single, flexible policy model capable of generating optimal actions for any desired trade-off. Empirically, OffMOB significantly outperforms existing methods on complex benchmarks, identifying the complete set of optimal trade-offs where naive extensions and single-objective methods fail.

The paper addresses several bottlenecks in offline policy learning:

Single-objective focus: Existing offline policy learning has been largely confined to single-objective problems, despite real-world scenarios inherently involving multiple, often conflicting, objectives. Optimizing for a simple scalar reward in such scenarios can lead to unintended and detrimental outcomes.

A rigorous formulation for the offline multi-objective contextual bandit problem, defining the concept of a preference-aware policy that can represent the entire Pareto front and introducing the expected Tchebycheff (TCH) sub-optimality metric.
OffMOB, a novel algorithm designed specifically for this setting, which leverages neural networks to model vector-valued reward functions and integrates the pessimism principle into the scalarization framework.
The establishment of the first finite-sample generalization bounds for OffMOB, showing that the algorithm's sub-optimality is bounded under a practical multi-objective data coverage assumption.
Extensive experiments on synthetic data demonstrating that OffMOB significantly outperforms existing methods, identifying the complete set of optimal trade-offs where naive extensions and single-objective methods fail.

OffMOB is designed to learn a preference-aware policy from an offline dataset by combining neural network-based reward estimation with a pessimistic approach to policy evaluation. The algorithm operates in two primary stages:

Neural Reward Estimation The unknown reward functions hi are modeled using fully connected neural networks (NNs) within the Neural Tangent Kernel (NTK) framework. The learning process for the network weights is performed via an online gradient descent update on a ridge regression objective. Loss: For each objective i ∈ [m], the loss function L(W) is minimized: L(W) = (1/2) (fw(xt,at) - rt,i)² + (λNN/2) ||W - W(0)||²

Pessimistic Policy Construction To mitigate distributional shift and unreliable value estimates, OffMOB adopts the principle of pessimism. For a given preference vector λ, the policy πt is constructed by selecting the action that minimizes a pessimistic estimate of the Tchebycheff scalarization. Tchebycheff Scalarization: The Tchebycheff scalarization of a policy π given context x is defined as: lπ(x|λ) = max {λi (zi - vπ,i(x, λ))} i∈[m] where z = supπ vπ,i(x, λ) is the ideal point for objective i given context x. Pessimistic Objective: The objective for action selection is: Lt(u, λ) = max {λi (zi - fi(u))} i∈[m] where fi(u) is the lower confidence bound for the reward of objective i, defined as: fi(u) = fw(t-1)(u) - βt-1||∇fw(t-1)(u) MNN(Λt-1)-1||

Policy Output: At deployment, a decision-maker can provide any preference vector λ, and the policy π(·|x, λ) can instantly compute the corresponding optimal action by solving the pessimistic Tchebycheff minimization problem.

Architecture: Each reward function hi is modeled by a dedicated fully connected neural network with depth L > 2 and width MNN per hidden layer. The ReLU activation function σ(z) = max{z,0} is used. All neural network-based methods use a 2-layer MLP with hidden layer width of MNN.
Key Equations:
- Reward generation: rt,i = hi(xt,at) + εt,i (1)
- Expected vector-valued reward: vπ(x, λ) = E_a~π(·|x,λ)[h(xa)] ∈ R^m (2)
- Tchebycheff scalarization: lπ(x|λ) = max_{i∈[m]} {λi (zi - vπ,i(x, λ))} (3)
- Sub-optimality metric: SubOpt(π) = E_{x~~ρ,λ~~U(Δ^m-1)} [l*(x|λ) - l^π*(x|λ)] (4)
- NN architecture: hi(x; θi) = W_L · σ(W_{L-1} · σ(…σ(W_1x))) (5)
- Linearized reward function: hi(x) = (g(x; θ0), θi - θ0) (6)
- Reward estimation loss: L(W) = (1/2) (fw(xt,at) - rt,i)² + (λNN/2) ||W - W(0)||² (7)
- Pessimistic objective: Lt(u, λ) = max_{i∈[m]} {λi (zi - fi(u))} (8)
- Lower confidence bound: fi(u) = fw(t-1)(u) - βt-1||∇fw(t-1)(u) MNN(Λt-1)-1|| (9)
- Effective dimension: d = (log det (I + H/λNN)) / (log(1 + nK/λNN)) (10)
Optimization: Adam optimizer (Kingma and Ba 2014) is used for trainable neural networks.
Hyperparameters:
- Regularization parameter λNN = 0.1 for all algorithms.
- Uncertainty parameter β: grid search over {0.01, 0.05, 0.1, 1, 5, 10} for LCB-based methods.
- Learning rate η: searched over {0.0001, 0.001} for trainable neural networks.
- l2-regularization parameter: 0.0001 for trainable neural networks.
- RBF kernel bandwidth parameter σ: searched in {0.1, 1, 10} for MOKernLCB.
Data Handling: Offline dataset Dn is generated by a fixed ε-greedy behavior policy with ε = 0.1, which randomly selects actions from the Pareto optima.
Implementation Details: The covariance matrix Λt is approximated by its diagonal. For MOKernLCB, the kernel model is fit on the first 1,000 samples if dataset size exceeds this limit.
Problem Parameters: Context dimension d = 20, number of actions K = 30, neural network width mNN = 50, number of objectives m = 3.

Datasets: Synthetic non-linear reward functions with m = 3 objectives. Three types of reward functions were used: hi(u) = 10(u^T ai)², hi(u) = u^T Ai u, and hi(u) = cos(3u^T ci). Noise scales σ were evaluated at 0.01, 0.1, and 0.5.
Metrics: Expected Tchebycheff sub-optimality (SubOpt(π)).
Baselines: MOLinLCB, MOKernLCB, MONeuralLinLCB, MONeuralLinGreedy, MONeuralGreedy.
Numeric Results: Not stated in the paper.
Qualitative Results:
- OffMOB consistently outperforms all baselines across the three distinct non-linear datasets.
- The significant performance gap between OffMOB and its greedy counterpart, MONeuralGreedy, highlights the critical importance of incorporating pessimism.
- MONeuralLinLCB and MONeuralLinGreedy show nearly identical performance, indicating pessimism is ineffective when the underlying model is misspecified.
- OffMOB consistently maintains its superior performance over all baselines across different noise conditions (σ = 0.01, 0.1, 0.5).
- OffMOB's performance degrades gracefully as noise level increases, consistently retaining a significant advantage.
Runtime/Training Cost: Not stated in the paper.

The current framework is limited to the offline multi-objective contextual bandit setting and would require significant extension to generalize to the full offline multi-objective reinforcement learning setting.
The approach does not explicitly incorporate partial or uncertain user preferences, which could enhance practical utility.

Method evidence: "Our core contribution is a novel algorithm that uniquely integrates the pessimism principle with multi-objective optimization to safely learn from off-policy data." (section: Abstract)
Result evidence: "Figure 1 shows that OffMOB consistently outperforms all baselines across three distinct non-linear datasets." (section: 5.3 Results)
Limitation evidence: "A significant extension would be to generalize our framework to the full offline multi-objective reinforcement learning setting. Finally, exploring methods to incorporate partial or uncertain user preferences could further enhance the practical utility of our approach." (section: 7 Conclusion and Discussion)

Offline Multi-Objective Bandits: From Logged Data to Pareto-Optimal Policies

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence

Additional Figures