Online 1. Largescale inference with block structure [electronic resource] [2017]
 Kou, Jiyao.
 2017.
 Description
 Book — 1 online resource.
 Summary

The detection of weak and rare effects in large amounts of data arises in a number of modern data analysis problems. Known results show that in this situation the potential of statistical inference is severely limited by the largescale multiple testing that is inherent in these problems. Here we show that fundamentally more powerful statistical inference is possible when there is some structure in the signal that can be exploited, e.g. if the signal is clustered in many small blocks, as is the case in relevant applications. We derive the detection boundary in such a situation where we allow both the number of blocks and the block length to grow polynomially with sample size. This result recovers as special cases the heterogeneous mixture detection problem (1) where there is no structure in the signal, as well as scan problem (2) where the signal comprises a single interval. We develop methodology that allows optimal adaptive detection in the general setting, thus exploiting the structure if it is present without incurring a penalty in the case where there is no structure. The advantage of this methodology can be considerable, as in the latter case the means need to increase at the rate of square root of the log of the problem size to ensure detection, while in the former case the means may decrease at a polynomial rate. The identification version of this problem is also considered in this thesis, where the length of the block is allowed to grow polynomially with the sample size while the number of blocks is assumed to grow at most logarithmically with the sample size. This setting greatly generalizes previous results. The multivariate version of this problem is also considered, in which we try to identify the support of the rectangular signal(s) in the hyperrectangle. A lower bound below which the identification is impossible is presented and an asymptotically optimal and computationally efficient procedure is proposed under Gaussian white noise. This signal identification problem is shown to have the same statistical difficulty as the corresponding detection problem, in the sense that whenever we can detect the signal, we can identify the support of the signal. We also discuss about the signal identification problem under the exponential family setting and the robust identification problem where the noise distribution is unspecified.
 Also online at

Special Collections
Special Collections  Status 

University Archives  Request via Aeon (opens in new tab) 
3781 2017 K  Inlibrary use 