gravitational waves - Meaning of $5.1sigma$ significance with regards to GW150914

I couldn't find any publication by LIGO that explains how we should interpret this value. The closest I have found is the following quote:

This means that a noise event mimicking GW150914 would be exceedingly rare - indeed we expect an event as strong as GW150914 to appear by chance only once in about 200,000 years of such data! This false alarm rate can be translated into a number of "sigma" (denoted by s), which is commonly used in statistical analysis to measure the significance of a detection claim. This search identifies GW150914 as a real event, with a significance of more than 5 sigma.

http://www.ligo.org/science/Publication-GW150914/index.php

From my reading, it appears that $5.1\sigma$ significance refers to:

The probability of observing such a signal given that the model of background noise correctly describes all input to the detectors at the time of the signal.

I would like to verify that the above interpretation is correct and is different from the probability GW150914:

1. arose due to chance


2. was caused by a gravitational wave


3. was caused by a BH-BH merger

I ask because I have seen posts on this site and elsewhere (both news and blogs) that seem to imply differently. I worry I may be misunderstanding some terminology specific to astrophysics.

Also, does anyone know what calculations were used to convert false alarm rate to # of sigmas? This detail seems to have been left out of the papers, so I assume it is something trivial I am missing due to lack of background in this area.

Edit:

Let me clarify (what I have learned is an incorrect) interpretation #1 above. This is Bayes' rule:

$ p(H|O)=\frac{p(H)p(O|H)}{p(O)} \tag{1} $
where,
$$H=\text{Hypothesis (model of background noise describes}\\ \text{all input to the detectors at the time of the signal)}$$ $$O= \text{Observation (the GW150914 signal)}$$

Just to be 100% clear:
$$ p(H|O)=\text{The probability H is true given O has been observed} \\ p(O|H)=\text{The probability of observing O given H is true} \\ p(H)=\text{The probability H is true} \textit{ independent } \text{of observation O} \\ p(O)=\text{The probability of observing O} \textit{ independent } \text{of whether H is true} $$

The last term can be rewritten as: $$ p(O)= p(H)p(O|H)+p(\neg H)p(O|\neg H) \tag{2} $$ where the probability H is false is denoted by
$$p(\neg H)=1-p(H)\tag{3}$$

In the answers, we established the $\sigma$-level is a simple transformation of the p-value, which equals $p(O|H)$. It is clear that $p(H|O)$ must have a different numerical value than the p-value except under some very specific circumstances, i.e. when $p(H)=p(O)$. The p-value is calculated under the assumption that $H$ is true, and from equations 1/2/3 we see that $p(H|O)$ explicitly depends on both $p(H)$ and the probability of observing such a signal if $H$ is false: $p(O|\neg H)$.

If our hypothesis is true, I think we all agree the only way to get a signal like GW150914 is a chance coincidence of noise patterns between the two LIGO detectors. So when writing we often use shorthand such as: $$H=\text{any signal is due to, i.e. caused by, chance coincidence}$$ or $$H=\text{any signal is not real}$$

There are many shorthand ways of saying the same thing that confuses things. The point is that the p-value is not the probability GW150914 was caused by (arose from; is due to) chance (background noise; random coincidence). It is also not the probability GW150914 "isn't real", or "how unlikely" it is that GW150914 is due to chance.

In this case, the p-value is apparently $p(O|H)\approx2\times10^{-7}$. Also, apparently the only other plausible explanation is a BH-BH merger. In an earlier question we estimated the prior probability of this to be $\approx10^{-4} \text{ to } 10^{-1}$. If we suppose that is the only other possible explanation, that must be the probability that H is false independent of observing GW150914: $p(\neg H)$.

First, lets use the lower bound: $p(\neg H)\approx10^{-4}$. From equation 3, then $p(H)\approx0.9999$. Also, GW150914 apparently matched prediction exactly. Therefore, probability of seeing such a signal given that H is false is $p(O|\neg H)\approx1$. Plugging in these values we get:

$$p(H|O)=\frac{0.9999\times2\times10^{-7}}{0.9999\times2\times10^{-7}+10^{-4}\times1}\approx0.002$$

Doing the same for the upper bound I get $p(H|O)\approx 1.8\times10^{-6}$. Now we can say "the probability GW150914 occurred due to chance ranges from $2\times10^{-3} \text{ to } 1.8\times10^{-6}$," which is quite different from the p-value. Any mistakes in this reasoning?

Answer

I see where you are going with your question. Let me feed the flames.

The sigma value that is quoted is equivalent to a false alarm probability. It tells you how unlikely it is for your experiment, given your understanding (theoretically and empirically) of the noise characteristics, to have produced a signal that looked like GWs from a merging BH.

Personally, I prefer the statement in the text you quote. Such an event would have been seen (in both detectors) about once every 200,000 years. Given that the observations were for 16 days, that means an expectation there would be $2.2 \times 10^{-7}$ such events in the data. i.e a one in 4.6 million chance.

The LIGO team have just converted this number into a numbers of sigma significance using an integral under one tail of the normal distribution. Using one of the readily available calculators e.g http://www.danielsoper.com/statcalc3/calc.aspx?id=20 we see that 5.0-5.1$\sigma$ (known as z-scores) corresponds to p-values of $2.7\times 10^{-7}$ to $1.7\times10^{-7}$, bracketing the value found above.

However this is not the confidence level that this is a gravitational wave or a merging black hole. There is always the possibility that some unanticipated source of error could have crept in that mimics a GW signal (but note that it needs to affect both detectors) or that some other astrophysical source could be capable of producing the signal. As far as I am aware, apart from the usual conspiracy theories (yawn), nobody has come up with a plausible alternative to GWs from a merging BH.