![]() ![]() Following a review of relevant previous work, we give a general formulation of the sequential Holm procedure in Section 2. We call this new procedure the sequential Holm procedure because of its relation to Holm’s (1979) seminal fixed-sample “step-down” procedure which controls the FWER. The current paper introduces a procedure to test k hypotheses while simultaneously controlling both the type I and II FWERs (defined precisely below) at prescribed levels in the same general setting: No assumptions are made about the dependence between the different data streams. Their procedure requires only the existence of basic sequential tests for each data stream and makes no assumptions about the dependence between the different data streams in particular, the error control holds when the streams are highly positively correlated, as is often the case in the application areas mentioned above. This scenario described above was addressed by Bartroff and Lai (2010) who gave a procedure that sequentially (or group sequentially) tests k hypotheses while controlling the type I familywise error rate (FWER, see Hochberg and Tamhane, 1987), i.e., the probability of rejecting any true hypotheses, at a prescribed level. We point out that our use of the word “sequential” here and below refers to the manner of sampling (or equivalently, observation) and differs from the way the word is sometimes used in the literature on fixed-sample multiple testing procedures to describe the stepwise analysis of fixed-sample test statistics, e.g., p-values. If we think of each experiment as a hypothesis test about the corresponding data stream, then what is needed is a combination of a multiple hypothesis test and a sequential hypothesis test. The preceding scenario occurs in a number of real applications including multiple endpoint (or multi-arm) clinical trials ( Jennison and Turnbull, 2000, Chapter 15), multi-channel changepoint detection ( Tartakovsky et al., 2003) and its applications to biosurveillance ( Mei, 2010), genetics and genomics ( Dudoit and van der Laan, 2008), acceptance sampling with multiple criteria ( Baillie, 1987), and financial trading strategies ( Romano and Wolf, 2005). The between-stream data may be very dissimilar in distribution and dimension, but at the same time may be highly correlated, or even duplicated exactly in some cases, since they all are related to some phenomenon. The scientist would like to control the overall error rate of the battery of experiments in order to be able to draw statistically-valid conclusions for each experiment once all experimentation has ceased, but also needs to be as efficient as possible with the finite resources available by “dropping” certain experiments (i.e., stopping experimentation) when additional data is no longer needed from that stream to reach a conclusion. The proposed procedure, which we call the sequential Holm procedure because of its inspiration from Holm’s (1979) seminal fixed-sample procedure, shows simultaneous savings in expected sample size and less conservative error control relative to fixed sample, sequential Bonferroni, and other recently proposed sequential procedures in a simulation study. Treating each experiment as a hypothesis test and adopting the familywise error rate (FWER) metric, we give a procedure that sequentially tests each hypothesis while controlling both the type I and II FWERs regardless of the between-stream correlation, and only requires arbitrary sequential test statistics that control the error rates for a given stream in isolation. The between-stream data may differ in distribution and dimension but also may be highly correlated, even duplicated exactly in some cases. The scientist would like to control the overall error rate in order to draw statistically-valid conclusions from each experiment, while being as efficient as possible. This paper addresses the following general scenario: A scientist wishes to perform a battery of experiments, each generating a sequential stream of data, to investigate some phenomenon. ![]()
0 Comments
Leave a Reply. |