Title:Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality

Abstract:Matching for causal inference is a well-studied problem, but standard methods fail when the units to match are text documents: the high dimensional and rich nature of the data renders exact or coarsened exact matching infeasible, causes propensity scores to produce incomparable matches, and makes assessing match quality nearly impossible. In this paper, we develop a framework for characterizing text matching methods based on (1) a text representation, and (2) a distance metric. We investigate what types of text matching within this framework perform best through a systematic multifactor evaluation experiment using human subjects. Altogether we evaluate 84 unique text matching methods from our framework along with 5 standards taken from the literature. Our experimental results identify matching procedures that produce matches with higher subjective match quality than current state-of-the-art techniques. We enhance the precision of this approach by developing a predictive model to predict the match quality of pairs of text documents as a function of our 84 distance scores. This model, which we find successfully mimics human judgment, also allows for approximate evaluation of new procedures. Finally, we apply our framework to a substantive debate in the study of media bias using a data set of front-page news articles from thirteen news sources. In our case study we find that both components contribute significantly to media bias, though some news sources have more than one component than the other.