“School of Mathematics”
Back to Papers HomeBack to Papers of School of Mathematics
| Paper IPM / M / 16311 |
|
| Abstract: | |
|
Big data enables reliable estimation of continuous probability density, cumulative distribution, survival, hazard rate, and mean residual functions (MRFs). We illustrate that plot of the MRF provides the best resolution for distinguishing between distributions. At each point, the MRF gives the mean excess of the
data beyond the threshold. Graph of the empirical MRF, called here the MRplot, provides an effective visualization tool. A variety of theoretical and data driven examples illustrate that MR plots of big data preserve the shape of theMRF and complex models require bigger data. The MRF is an optimal predictor of the excess of the random variable.With a suitable prior, the expected MRF gives the
Bayes risk in the form of the entropy functional of the survival function, called here the survival entropy.We show that the survival entropy is dominated by the standard deviation (SD) and the equality between the two measures characterizes the exponential distribution. The empirical survival entropy provides a data concentration statistic which is strongly consistent, easy to compute, and less sensitive than the SD to heavy tailed data. An application uses theNewYork City Taxi database with millions of trip times to illustrate the MR plot as a powerful tool for distinguishing distributions.
Download TeX format |
|
| back to top | |
