Bayesian Variable Selection for Linear Models Using I-Priors

Abstract

The Bayesian approach to modelling differs from the frequentist approach primarily in the supplementation of additional information about the parameters to the data. If we specify a “good” prior, in the sense that the prior nudges the likelihood in the right direction, then the estimates will also be good. This is what we aim to do in the case of variable selection problems, whereby the Bayesian method reduces the selection problem to one of estimation from a true search of the variable space for the model which optimises a certain criterion. We contribute to the vastly available literature of variable selection methods by using I-priors [5]—a class of Gaussian distributions which has the distinguishing property of having covariance proportional to the Fisher information (of the model parameters). The original motivation behind the I-prior methodology was to develop a novel unifying approach to various regression models. In this work, we detail the I-prior model used, and showcase some simulation results and several real-world applications in which the I-prior performs favourably compared to other prior distributions and/or variable selection techniques in terms of model size, predictive ability, and so on.