Estimating Latent Processes on a Network From Indirect Measurements
- 15 March 2013
- journal article
- research article
- Published by Taylor & Francis in Journal of the American Statistical Association
- Vol. 108 (501) , 149-164
Abstract
In a communication network, point-to-point traffic volumes over time are critical for designing protocols that route information efficiently and for maintaining security, whether at the scale of an Internet service provider or within a corporation. While technically feasible, the direct measurement of point-to-point traffic imposes a heavy burden on network performance and is typically not implemented. Instead, indirect aggregate traffic volumes are routinely collected. We consider the problem of estimating point-to-point traffic volumes, , from aggregate traffic volumes, , given information about the network routing protocol encoded in a matrix A. This estimation task can be reformulated as finding the solutions to a sequence of ill-posed linear inverse problems, , since the number of origin-destination routes of interest is higher than the number of aggregate measurements available. Here, we introduce a novel multilevel state-space model (SSM) of aggregate traffic volumes with realistic features. We implement a naïve strategy for estimating unobserved point-to-point traffic volumes from indirect measurements of aggregate traffic, based on particle filtering. We then develop a more efficient two-stage inference strategy that relies on model-based regularization: a simple model is used to calibrate regularization parameters that lead to efficient/scalable inference in the multilevel SSM. We apply our methods to corporate and academic networks, where we show that the proposed inference strategy outperforms existing approaches and scales to larger networks. We also design a simulation study to explore the factors that influence the performance. Our results suggest that model-based regularization may be an efficient strategy for inference in other complex multilevel models. Supplementary materials for this article are available online.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Data augmentation in multi-way contingency tables with fixed marginal totalsJournal of Statistical Planning and Inference, 2006
- Traffic matrix estimationACM SIGCOMM Computer Communication Review, 2002
- Following a Moving Target—Monte Carlo Inference for Dynamic Bayesian ModelsJournal of the Royal Statistical Society Series B: Statistical Methodology, 2001
- Time-Varying Network Tomography: Router Link DataJournal of the American Statistical Association, 2000
- Bayesian Inference on Network Traffic Using Link Count DataJournal of the American Statistical Association, 1998
- Network Tomography: Estimating Source-Destination Traffic Intensities from Link DataJournal of the American Statistical Association, 1996
- Blind Deconvolution via Sequential ImputationsJournal of the American Statistical Association, 1995
- Multiple Imputation of Industry and Occupation Codes in Census Public-use Samples Using Bayesian Logistic RegressionJournal of the American Statistical Association, 1991
- A Statistical Model for Positron Emission TomographyJournal of the American Statistical Association, 1985
- On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are KnownThe Annals of Mathematical Statistics, 1940