The reason I ask is because I’ve been digging deep into Brülosophy’s website, and while some of the time fermentation temps have some difference in taste, it seems like most of the time it honestly doesn’t matter. And when the study shows that they were able to pick the odd beer out, he states it could’ve been by random chance.
Keep in mind, though, that he uses a methodology designed to guard very carefully against incorrectly concluding the variable, like ferm temp, had an effect when in fact the results were due to random chance. Not a criticism at all; he's employing what's used in science.
But the downside of this method is that, by guarding very carefully against making that kind of error, you increase your chance of making the other type: Concluding the results were due to random chance when in fact the variable DID have an effect*. In "real science", that's the price to be paid because the other type of error is much worse to make.
That's in real science, but is it the worse error in homebrewing? Depends on the situation, but I'd say usually not. Sure, if there were a very expensive piece of equipment that is purported to make better beer, then, yes, I'd want to be very confident the results of that triangle test were not just random chance before I went out and bought it. But if it's something like whether dropping in 2 cents worth of gypsum helps bring out hop flavor in a IPA, I'd settle for less confidence.
To stick with that example, let's say brulosophy found that out of the 20 people, 13 were able to distinguish the difference between the beer with the gypsum and the one without, but that only reached a significance level of 8%. The conclusion would then "have to" be, assuming the usual 5% criteron were used, that gypsum doesn't matter. But is that appropriate? What you're kind of saying** is "I'm only about 90% sure gypsum works, so I won't use it." Again, it makes sense in science to say "I'm only about 90% sure there's a real effect here, so I must conclude there isn't." because you REALLY don't want to claim you've found a real effect and be wrong about it. But in homebrewing?
*it's worth noting that the type of statistical tests used for the kind of data collected in brulosophy exbeeriments is especially prone to this kind of error.
**only kind of, stats nerds
...I'd love your thoughts, but know I'm aware of the liberties I'm taking.