Do FDR-corrected p values in R retain a significance threshold?

Question

I have a set of uncorrected p-values after running a piece of code, which outputs uncorrected p values. I had to correct them prior to describing my results in a scientific paper.

I read my uncorrected p values as a vector in R and then used R's multiple comparison correction method with FDR as a preferred choice:

p.adjust(unc_pvalues, method="fdr")

After correcting the uncorrected p values in R, I obtain a list of values where about 50 are <0.0001 and then multiple are in the range 0.1-0.99.

I get that these have all been FDR corrected already (at 0.05 as that's the default in R). What is confusing me is where to set the threshold of significance of these already corrected p values though (I hope that makes sense).

I basically have the below:

[48 values that are = 0.0000

and then:

0.0001

0.0005

....

0.6884

0.6967

0.6989

0.7761

0.7761

0.8741

0.8863

0.9305

0.9644

0.9644

0.9884

0.9989]

Which of these values are statistically significant? Is that all of them, even the ones <0.05, given that they have already been FDR-corrected? And if I only include those values that are smaller than either .05 or .001, am I double-correcting for false positives?

Hope that makes sense - many thanks all for your advice in advance!

Answer 1

When correcting for multiplicity you can do this in two ways: how the calculations are usually done is that you adjust your alpha (type I assertion rate) for each individual comparison. Let's use Bonferroni as an example because the calculations are straightforward, though the idea applies to all multiplicity corrections.

In the case of Bonferroni, the alpha for each individual test is the global alpha divided by the number of tests. Let's say you have the P-values 0.12 0.06 0.03 0.01 and want to test at a global alpha of 0.05. You'd compare each of these to 0.05/4 = 0.0125, and only the smallest one would be "significant".

What p.adjust does is instead of changing the local alpha (which may depend on the order and the values of all of your tests for less simple methods), it changes your P-values in a very specific way: for every test, it returns you the global alpha at which that test could be declared significant.

p.adjust(c(.12, .06, .03, .01), method="bonferroni")
#> 0.48 0.24 0.12 0.04

As before the last test is below (global) alpha=0.05, for the next smallest we see that we would have to accept alpha=0.12 (and indeed 0.12/4 = 0.03), and so on. Again, in the Bonferroni case this is all very simple but the same ideas hold for any output of p.adjust: instead of changing the (local) alpha at which you reject a test, you change its P-value so that you can use the original alpha.

Which of your P-values are significant therefore still depends on the alpha you want to use, if it's say 0.05 or 0.10 globally you would compare your adjusted P-values to these thresholds & anything below is significant.