econml.score.RScorer

class econml.score.RScorer(*, model_y, model_t, discrete_treatment=False, categories='auto', cv=2, mc_iters=None, mc_agg='mean', random_state=None)[source]

Bases: object

Scorer based on the RLearner loss. Fits residual models at fit time and calculates residuals of the evaluation data in a cross-fitting manner:

Yres = Y - E[Y|X, W]
Tres = T - E[T|X, W]

Then for any given cate model calculates the loss:

loss(cate) = E_n[(Yres - <cate(X), Tres>)^2]

Also calculates a baseline loss based on a constant treatment effect model, i.e.:

base_loss = min_{theta} E_n[(Yres - <theta, Tres>)^2]

Returns an analogue of the R-square score for regression:

score = 1 - loss(cate) / base_loss

This corresponds to the extra variance of the outcome explained by introducing heterogeneity in the effect as captured by the cate model, as opposed to always predicting a constant effect. A negative score, means that the cate model performs even worse than a constant effect model and hints at overfitting during training of the cate model.

This method was also advocated in recent work of [Schuleretal2018] when compared among several alternatives for causal model selection and introduced in the work of [NieWager2017].

Parameters
  • model_y (estimator) – The estimator for fitting the response to the features. Must implement fit and predict methods.

  • model_t (estimator) – The estimator for fitting the treatment to the features. Must implement fit and predict methods.

  • discrete_treatment (bool, default False) – Whether the treatment values should be treated as categorical, rather than continuous, quantities

  • categories (‘auto’ or list, default ‘auto’) – The categories to use when encoding discrete treatments (or ‘auto’ to use the unique sorted values). The first category will be treated as the control treatment.

  • cv (int, cross-validation generator or an iterable, default 2) – Determines the cross-validation splitting strategy. Possible inputs for cv are:

    • None, to use the default 3-fold cross-validation,

    • integer, to specify the number of folds.

    • CV splitter

    • An iterable yielding (train, test) splits as arrays of indices.

    For integer/None inputs, if the treatment is discrete StratifiedKFold is used, else, KFold is used (with a random shuffle in either case).

    Unless an iterable is used, we call split(concat[W, X], T) to generate the splits. If all W, X are None, then we call split(ones((T.shape[0], 1)), T).

  • mc_iters (int, optional) – The number of times to rerun the first stage models to reduce the variance of the nuisances.

  • mc_agg ({‘mean’, ‘median’}, default ‘mean’) – How to aggregate the nuisance value for each sample across the mc_iters monte carlo iterations of cross-fitting.

  • random_state (int, RandomState instance, or None, default None) – If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.

References

NieWager2017

X. Nie and S. Wager. Quasi-Oracle Estimation of Heterogeneous Treatment Effects. arXiv preprint arXiv:1712.04912, 2017. https://arxiv.org/pdf/1712.04912.pdf

Schuleretal2018

Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, Nigam Shah. “A comparison of methods for model selection when estimating individual treatment effects.” Arxiv, 2018 https://arxiv.org/pdf/1804.05146.pdf

__init__(*, model_y, model_t, discrete_treatment=False, categories='auto', cv=2, mc_iters=None, mc_agg='mean', random_state=None)[source]

Methods

__init__(*, model_y, model_t[, ...])

best_model(cate_models[, return_scores])

Chooses the best among a list of models

ensemble(cate_models[, eta, return_scores])

Ensembles a list of models based on their performance

fit(y, T[, X, W, sample_weight, groups])

Parameters
  • Y ((n × d_y) matrix or vector of length n) -- Outcomes for each sample

score(cate_model)

Parameters

cate_model (instance of fitted BaseCateEstimator)

best_model(cate_models, return_scores=False)[source]

Chooses the best among a list of models

Parameters
  • cate_models (list of instance of fitted BaseCateEstimator)

  • return_scores (bool, default False) – Whether to return the list scores of each model

Returns

  • best_model (instance of fitted BaseCateEstimator) – The model that achieves the best score

  • best_score (double) – The score of the best model

  • scores (list of double) – The list of scores for each of the input models. Returned only if return_scores=True.

ensemble(cate_models, eta=1000.0, return_scores=False)[source]

Ensembles a list of models based on their performance

Parameters
  • cate_models (list of instance of fitted BaseCateEstimator)

  • eta (double, default 1000) – The soft-max parameter for the ensemble

  • return_scores (bool, default False) – Whether to return the list scores of each model

Returns

  • ensemble_model (instance of fitted EnsembleCateEstimator) – A fitted ensemble cate model that calculates effects based on a weighted version of the input cate models, weighted by a softmax of their score performance

  • ensemble_score (double) – The score of the ensemble model

  • scores (list of double) – The list of scores for each of the input models. Returned only if return_scores=True.

fit(y, T, X=None, W=None, sample_weight=None, groups=None)[source]
Parameters
  • Y ((n × d_y) matrix or vector of length n) – Outcomes for each sample

  • T ((n × dₜ) matrix or vector of length n) – Treatments for each sample

  • X ((n × dₓ) matrix, optional) – Features for each sample

  • W ((n × d_w) matrix, optional) – Controls for each sample

  • sample_weight ((n,) vector, optional) – Weights for each row

  • groups ((n,) vector, optional) – All rows corresponding to the same group will be kept together during splitting. If groups is not None, the cv argument passed to this class’s initializer must support a ‘groups’ argument to its split method.

Return type

self

score(cate_model)[source]
Parameters

cate_model (instance of fitted BaseCateEstimator)

Returns

score – An analogue of the R-square loss for the causal setting.

Return type

double