Title:Comparing basic statistical concepts with diagnostic probabilities based on directly observed proportions to help understand the replication crisis

Abstract:Instead of regarding an observed proportion as a sample from a population with an unknown parameter, diagnosticians intuitively use the observed proportion as a direct estimate of the posterior probability of a diagnosis. Therefore, a diagnostician might also regard a continuous Gaussian probability distribution of an outcome conditional on a study selection criterion to represents posterior probabilities. Fitting a distribution to its mean and standard deviation (SD) can be regarded as pooling data from an infinite number of imaginary or theoretical studies with an identical mean and SD but randomly different numerical values. For a distribution of possible means based on a SEM, the posterior probability Q of any theoretically true mean falling into a specified tail would be equal to the tail area as a proportion of the whole. If the reverse likelihood distribution of possible study means conditional on the same hypothetical tail threshold is assumed to be the same as the posterior probability distribution of means (as is customary) then by Bayes rule the P value equals Q. Replication involves doing two independent studies, thus doubling the variance for the combined posterior probability distribution. Thus, if the original effect size was 1.96, the number of observations was 100, the SEM was 1 and the original P value was 0.025, the theoretical probability of a replicating study getting a P value of up to 0.025 again is only 0.283. By applying the double variance to power calculations, the required number of observations is doubled compared to conventional approaches. If these theoretical probabilities of replication are consistent with empirical replication study results, this might explain the replication crisis and make the concepts of statistics easier to understand by diagnosticians and others.