On the quality of Brulosophy experiments:
I don't think they're that bad. Really. I'm a scientist and I don't see a lot of places there that might be improved.
They're doing two batches as identically as they can, changing one variable (usually). Experimenters will do things like split a yeast starter, ensuring each gets the same thing. The temperatures of mash, fermentation, and so on are very close, such that no reasonable, experienced brewer would expect, say, a .4 degree difference in mash temp to produce a noticeable final result.
The essence of experimentation is control of extraneous variables. Unless they're lying to us about what they're doing (and I don't think they are), they're doing a pretty nice job controlling.
There's a flaw in understanding research that sometimes people make--that is, the fact that you might not have perfectly controlled everything doesn't mean that what you didn't perfectly control had a negative influence on the results. It might have, might not have. If the variable not perfectly controlled differs significantly between experimental and control batch, it might be an alternative explanation for the results. Might be. This is why we replicate.
****************
My issue w/ the Brulosophy approach is on the back end--deciding if the beers are different or not.
The problem is what different people perceive. My signature says what it says for a reason. I don't care for Belgian beers. There's something about the flavor that turns me off. I can judge them (had one from
@Morrey that was an excellent Belgian--but I'd never order one
). People who like Belgians would, I'm sure, like Morrey's. No off flavors, good mouthfeel, flavor is rich and full. And I don't care for it!
So when we get a panel of tasters in brulosophy experiments, we're seeing if a bunch of people can tell the difference between two beers. We know nothing about what they've been drinking prior to testing, whether they like the style or not, if they're super tasters versus people like me who are not, of what beer drinkers, if any, the panel is representative.
Further, they don't do the triangle tests correctly, at least as food-science people would do. I attended an off-flavor workshop last summer where they had us do a triangle test--and they randomized the presentation of the beers to different people. In other words, AAB, ABA, BAA, BBA, BAB, ABB. There are six orders where someone could taste the beer, and everybody shouldn't be tasting them the same way.
And we don't know what the panel tasters were drinking just prior. I did a triangle test a little more than a year ago and asked tasters to not have any beer prior to tasting. I don't know what brulosophy tasters are doing.
****************
When I teach this I have my students, w/r/t sampling, imagine doing a student survey and asking passersby to take that survey. Would their results be different if they were asking student passersby inside the Engineering building compare to those inside the Art building?
I don't know which building the Exbeeriment sample tasters come from. The IPA building? The Sours building? The BMC building?
****************
On top of that, a typical panel will have more than half the tasters unable to identify the odd-beer-out. Think about that. People are focused on whether the results are significant, which is fine, but there's another very interesting element of the data--most can't tell the beers apart!
****************
Further, and this part gets me, the writeup usually says, if the result is not significant, something like "tasters were unable to reliably distinguish between the two beers."
Um.....no. Reliability is repeatability or consistency of measurement. If each taster were identical, sure, but they're most assuredly not. I'd be more impressed with the results if tasters came back 3 or 4 days in a row and were able to repeat the same conclusion each time. What we don't know is how many "correct" guesses were in fact that--guesses.
What we have is analogous in some ways to a one-shot case study. No way to know whether the results would be the same tomorrow w/ these same tasters. Or a different set of tasters.
****************
I've read a little, and seen a few videos, on how the food industry does taste tests. Much more controlled. Among other things, I'd love to see if tasters could, really, distinguish between these beers on successive days. And before drinking any other beer.
Are tasters in the experiments suffering (if that's the right word) from palate fatigue? If the exbeerimental taste test was their first beer of the day would the results be different than after drinking a couple IPAs? Or a slew of other beers?
These are questions I'd want to have answered before drawing any great and weighty conclusions.
****************
Just to give Marshall a little love, he's trying to do some things. Most of us critical of the methods or the measurements aren't doing a damned thing, including yours truly.
Marshall and his cohorts are "
in the arena."
I have done one of these, comparing two beers (IPAs) using Brewtan-B in one, not in the other. Tasters, under more controlled circumstances, were unable to tell the difference.
Now that I can brew 10-gallon batches, maybe I'll try some more.
