Decision-centric active learning of binary-outcome models.

It can be expensive to acquire the data required for businesses to employ data-driven predictive modeling--for example, to model consumer preferences to optimize targeting. Prior research has introduced "active-learning" policies for identifying data that are particularly useful for model induction, with the goal of decreasing the statistical error for a given acquisition cost (error-centric approaches). However, predictive models are used as part of a decision-making process, and costly improvements in model accuracy do not always result in better decisions. This paper introduces a new approach for active data acquisition that specifically targets decision making. The new decision-centric approach departs from traditional active learning by placing emphasis on acquisitions that are more likely to affect decision making. We describe two different types of decision-centric techniques. Next, using direct-marketing data, we compare various data-acquisition techniques. We demonstrate that strategies for reducing statistical error can be wasteful in a decision-making context, and show that one decision-centric technique in particular can improve targeting decisions significantly. We also show that this method is robust in the face of decreasing quality of utility estimations, eventually converging to uniform random sampling, and that it can be extended to situations where different data acquisitions have different costs. The results suggest that businesses should consider modifying their strategies for acquiring information through normal business transactions. For example, a firm such as Amazon.com that models consumer preferences for customized marketing may accelerate learning by proactively offering recommendations--not merely to induce immediate sales, but for improving recommendations in the future.

Key words: decision-support systems; active learning; classifier induction; decision making; predictive modeling

1. Introduction

With advances in computing power, network reach, availability of data, and the maturity of induction algorithms, businesses are taking advantage of automated predictive modeling to influence repetitive decisions, often as tools for extracting customer, competitor, and market intelligence (Berry and Linoff 2004). Consider an example: telecommunications companies face severe customer retention problems as customers switch back and forth between carriers (the problem of churn). For each customer, at each point in time, the company faces a decision between doing nothing and intervening in an attempt to retain the customer. This paper focuses on modeling uncertain outcomes for a certain type of decision problem (1): a firm or individual must repeatedly decide either to take a specific action or not to take the action. If the action is taken, it results in one of two uncertain but well-defined outcomes. We assume that the utilities associated with each outcome and with not taking the action can be estimated or are known.

Such decisions can be based on predictive models built from data on known outcomes; however, acquiring such data can be costly. Consider our customer-retention example. (2) Outcomes can be modeled based on data about customers and their responses to the firm's actions, and firms collect such data in various ways. They undertake direct solicitations--for example, via experimental special offers, via customer surveys, and via interactions such as Amazon's online acquisition of product ratings. Firms also acquire data directly from third parties. For example, Acxiom (3) sells detailed consumer demographic and lifestyle data to many firms in support of their marketing efforts; other firms, such as Abacus, (4) maintain and sell specialized consumer purchase information. Firms also collect information indirectly in the course of normal business interactions--for example, by observing responses to offers or the results of everyday merchandizing decisions. All these acquisitions involve costs to the firms.

For this paper, we consider the acquisition of a particular kind of data. Following the terminology used by Hastie et al. (2001), we refer to the data used to induce the predictive model as the training data. Each training data point is a historical example of the phenomenon being modeled (e.g., the targeting of a particular customer), described by various variables including a target variable (e.g., indicating whether or not the customer responded to the offer). Of particular importance to this paper, in training data each historical example's label, the target variable's value, is known. We focus on the acquisition of these labels, which can be costly. For example, obtaining response data for individual consumers involves solicitation costs, incentives required for revealing preferences, negative reactions to solicitations, etc. Firms also incur opportunity costs when labels are acquired over time through normal business interactions. For example, making a particular offer to a sample of website visitors for the purpose of acquiring training labels may preclude making another offer already known to be profitable.

To reduce the cost of label acquisition, researchers have studied the selective acquisition of labels (see [section]2.2), reasoning that focused selection of cases for label acquisition should yield more accurate models for a given acquisition budget, as compared to the standard approach of acquiring labels for cases sampled uniformly at random. There are various label-acquisition strategies for reducing statistical error (error-centric approaches). However, business applications employ predictive models to help make particular business decisions. Of course, a more accurate model may lead to better decisions, but concentrating on the particular decisions themselves (a decision-centric approach) has the potential to produce a more economical allocation of the acquisition budget. Prior work has not addressed label acquisition to facilitate decision making directly.