A note on tests for relevant differences with extremely large sample sizes.

A well-known problem in classical two-tailed hypothesis testing is that P-values go to zero when the sample size goes to infinity, irrespectively of the effect size. This pitfall can make the testing of data consisting of large sample sizes potentially unreliable. In this note, we propose to test for relevant differences to overcome this issue. We illustrate the proposed test a on real data set of about 40 million privately insured patients.