AI Research Portal

This paper introduces two new counterfactual decision-making metrics, Probability of Potential Outcome Ranking (PoR) and Probability of achieving the Best potential outcome (PoB), to address limitations of traditional Expected Potential Outcome (RoE) ranking. PoR identifies the most probable ranking of potential outcomes for an individual, while PoB indicates the action most likely to yield the top-ranked outcome. The paper establishes identification theorems and derives bounds for these metrics, presents estimation methods, and empirically demonstrates their finite-sample properties and application to a real-world dataset, showing they can lead to different optimal action choices compared to RoE.

Traditional counterfactual decision-making, often based on the rank of expected potential outcomes (RoE), faces several bottlenecks:

Inadequate for individual preferences: The action that maximizes the expected potential outcome (PO) does not always maximize the PO for each individual, nor does the action with the lowest expected PO necessarily represent the least favorable choice for an individual.

Establishment of identification theorems for PoR and PoB under the rank invariance assumption.
Derivation of bounds for PoR and PoB in the absence of the rank invariance assumption.
Presentation of corresponding estimation methods for PoR and PoB.
Numerical experiments illustrating the finite-sample properties of the estimators.
Demonstration of the application of PoR and PoB to a real-world dataset.

The paper proposes two new metrics for counterfactual decision-making: PoR and PoB, inspired by Discrete Choice Models (DCMs).

Probability of Potential Outcome Ranking (PoR)

Definition: PoR quantifies the probability of a specific ranking of potential outcomes (POs) for an individual. For a rank r = (r₁, ..., rK), PoR is defined as: SPOR(r) = P (Yr₁ > Yr₂ > … > Yrk).
Optimization: Decision-makers can use PoR to find the most probable ranking: rPoR arg maxr∈R SPoR(r).
Purpose: PoR answers the question, "What is the most probable rank of POs for an individual?" It allows for more nuanced decisions by assessing the likelihood of each ranking pattern.

Probability of achieving the Best potential outcome (PoB)

Definition: PoB quantifies the probability that a specific action achieves the best potential outcome for an individual. For an action r₁ ∈ {1, ..., K}, PoB is defined as: SPOB(r1) = P (arg maxk=1,...,K {Yk} = r1).
Optimization: Decision-makers can use PoB to find the action most likely to yield the best outcome: rPoB arg maxr1=1,2,...,K SPoB(r1).
Purpose: PoB answers the question, "Which action is most likely to yield the best outcome for an individual?" It focuses on the proportion of individuals for whom an action yields the best PO.

Identification and Estimation

Assumptions: The identification of PoR and PoB relies on Assumptions 1 (Continuity and Differentiability of CDF of Potential Outcome), 2 (Exogeneity), and 4 (Rank invariance assumption for multiple actions).
Identification Theorems: Theorem 1 identifies PoR (σ(r)) and Theorem 2 identifies PoB (η(r₁)) under these assumptions, expressing them in terms of conditional probabilities of potential outcomes.
Estimation: The paper provides estimators ô(r) for PoR and ἢ(r₁) for PoB, which are consistent estimators.
Bounds: For cases where the rank invariance assumption (Assumption 4) is considered too restrictive, the paper derives bounds for PoR (σL(r) ≤ SPoR(r) ≤ σU(r)) and PoB (ηL(r₁) ≤ SPoB(r₁) ≤ ηU(r₁)) using Fréchet inequalities. These bounds can also be estimated.

Structural Causal Models (SCM): The framework uses SCMs M = < U,V,F,PU > to define potential outcomes Yx(u).
Assumptions:
- Assumption 1 (Continuity and Differentiability of CDF of Potential Outcome): Fy(y) is continuous and differentiable for any x ∈ Ωx and y ∈ ΩY.
- Assumption 2 (Exogeneity): Yx ⊥ X for any x ∈ Ωx. This implies P(Yx) = P(Y|x).
- Assumption 3 (Rank invariance assumption): For binary treatment x = {1,2}, FY₁(Y1) = FY₂(Y2) holds for almost every subject whose POs are (Y1, Y2) = (Y1, Y2).
- Assumption 4 (Rank invariance assumption for multiple actions): FY₁(Y1) = FY₂(Y2) = … = FYK(YK) holds for almost every subject whose POs are (Y1, Y2,...,YK) = (y1, y2,..., yK).
Key Equations (Identification):
- PoR Identification (Theorem 1): σ(r) = P(Y > FYX=r2(FY|X=r1 (Y)) > … > FYX=rK(FY|X=r1 (Y))|X = r1).
- PoB Identification (Theorem 2): η(r1) = P(Y > FYX=r2(FY|X=r1 (Y)), … , Y > FYX=rK(FY|X=r1 (Y))|X = r1).
Estimation:
- Conditional CDF Estimation: FY|X=rk (y) = (1 / N(rk)) Σj=1 to N(rk) I(Y(k)j ≤ y).
- PoR Estimator (ô(r)): ô(r) = (1 / N(r1)) Σi=1 to N(r1) I(Y(1)i > Y(2)([N(r2) FY|X=r2 (Y(1)i)]+1) > … > Y(K)([N(rK) FY|X=rK (Y(1)i)]+1)).
- PoB Estimator (ἢ(r₁)): ἢ(r1) = (1 / N(r1)) Σi=1 to N(r1) I(Y(1)i > Y(2)([N(r2) FY|X=r2 (Y(1)i)]+1), … , Y(1)i > Y(K)([N(rK) FY|X=rK (Y(1)i)]+1)).
Computational Complexity: The computational complexities for ô(r) and ἢ(r₁) are O(Σk=1 to K N(k) log N(k)). When N(k) = N/K, the rate becomes Op(K^(3/2)N^(-1/2)), which worsens as K increases.
Bounds Estimation: The bounds for PoR and PoB are estimated by plugging the conditional CDF estimates into the bound equations (Eqs. 34, 35, 36, 37).

Datasets:
- Synthetic SCMs:
  - SCM (A): Y := −(X − 4)UY, UY ~ Unif(−0.5,1). Satisfies rank invariance.
  - SCM (B): Y := −XUX +U'Y, UY ~ N(−1,1), U'Y ~ Unif(−1, 1). Violates rank invariance.
  - SCM (C): Y := −XUY, UY ~ Unif(−1,1). Satisfies rank invariance.
- Real-world Dataset: "coagulation" (Kropf et al. 2000) from https://cran.r-project.org/web/packages/SimComp/SimComp.pdf. Evaluates three extracorporeal circulation systems (S, H, B) on "Thromb.count" endpoint.
Metrics: PoR, PoB, RoE (E[Yk]), absolute errors, 95% Confidence Intervals (CI).
Baselines: Not stated in the paper, as there are no existing baselines for estimating PoR or PoB when K ≥ 3.
Numerical Results (Synthetic Data):
- Three Candidate Actions (K=3):
  - SCM (A) (Rank Invariance):
    - SPOR(1, 2, 3) estimate: 0.635 [0.560,0.638] (True Value: 0.666) for N'=3000.
    - SPOB(1) estimate: 0.650 [0.580,0.702] (True Value: 0.666) for N'=3000.
    - Estimates become more reliable with increasing sample size (N').
  - SCM (B) (Violates Rank Invariance):
    - UB of SPOR(1,2,3) estimate: 0.873 [0.855,0.887] (True Value: 0.500) for N'=3000.
    - LB of SPOR(1,2,3) estimate: 0.000 [0.000,0.000] (True Value: 0.500) for N'=3000.
    - Computed lower bounds are non-informative for this model.
- Many Candidate Actions (K increases):
  - SCM (C) (Rank Invariance):
    - Absolute errors of estimated PPOR(1, 2, ..., K) and SPOB(1) from ground truth increase as K increases (K=3, 5, 10, 20).
    - For K=20, SPOR(1, 2, ..., K) absolute error: 0.256 [0.147,0.391].
    - For K=20, SPOB(1) absolute error: 0.023 [0.012,0.048].
Numerical Results (Real-World Data - "coagulation"):
- Mean Estimates with 95% CIs:
  - E[YB]: 0.994 [0.859,1.128] (highest RoE).
  - SPOR(B, S, H): 0.277 [0.083,0.583] (most probable ranking).
  - SPOB(B): 0.490 [0.182,0.818] (highest PoB).
- Decision Implications: All three metrics (RoE, PoR, PoB) conclude treatment B is the most effective. However, PoR suggests (B, S, H) as the most probable ranking, preferring S over H, while RoE and PoB prefer H over S. PoB reveals B is optimal for only 49% of patients.
Runtime/Training Cost: Not stated in the paper.

Restrictive Rank Invariance Assumption: The identification results for PoR and PoB rely on the rank invariance assumption (Assumption 4), which is considered restrictive and may limit their practical applications.
Non-sharp Bounds: The derived bounds for PoR and PoB, which do not rely on the rank invariance assumption, are not sharp in general, except for the case of K = 2.
Computational Complexity with Increasing K: The computational complexity of the estimators worsens as the number of candidate actions (K) increases, with a rate of Op(K^(3/2)N^(-1/2)).
Wide Confidence Intervals for Real-World Data: Due to the small sample size in the real-world dataset, the CIs of the estimates for different treatments have considerable overlap, making it difficult to draw reliable conclusions about the effectiveness ranking.

Method evidence: "In this paper, we study new counterfactual decision-making rules by introducing two new metrics: the probabilities of potential outcome ranking (PoR) and the probability of achieving the best potential outcome (PoB)." (section: Abstract)
Result evidence: "Table 5: Mean estimates with 95% CIs of the RoE, POR, and PoB metrics. We highlight the highest values." (section: Results)
Limitation evidence: "Our identification results rely on the restrictive rank invariance assumption, which may limit their practical applications. We provide bounds that do not rely on this assumption, but they are not sharp in general. Deriving tight bounds for PoR and PoB remains an important direction for future research." (section: Conclusion)

Potential Outcome Rankings for Counterfactual Decision Making

Summary

Problem

Key Contributions

Method

Technical Details

Results

Weaknesses / Limitations

Evidence