Thank you! Great article. I've made it a candidate for my Top Twelve of 2022.
The original idea of meta-analysis was that by combining many small studies, you'd have a big sample so statistically significant effects would have a chance to show up. Or, to not show up-- but you'd have higher power, more confidence that it wasn't just a small sample size that meant you got a null result.
In this homeopathy meta-study, though, what's happening is that the sixth study, the single-blind one has such a big effect that it's generating the entire result, it looks like to me-- even though it's being DILUTED, not ENHANCED, by the other studies. The purpose of a meta-study like this is just to camouflage the deficiencies of the one study that's generating the general conclusion from combining studies. Plus, you can claim that aggregating 6 studies produced result X, instead of just that one or two studies produced it, and other studies rejecting it. So the meta-analysis is just a rhetorical tool, a tool of obfuscation.
Depressingly, though I hadn't looked at this particular meta-analysis, this matches many experiences I've had, in reviewing and in critiquing published papers. You have to check all or most of the papers that were included, maybe redo some of the calculations with dubious studies left out, and then you're still not sure because it takes too long to redo the literature search. Obviously there are some very good meta-analyses too, but meta-analyses are so much more demanding than other kinds of study to check for quality.
Whilst I don’t doubt that you are right that this meta-analysis is a pile of garbage, the logic of your argument is hilarious. You essentially don’t believe it because you don’t believe it. If you believe the vaccine studies are better done than the work on ivermectin, I don’t know how to help you.
ADHD is poorly defined and prone to wildly subjective analysis and description of symptoms. So while I agree that homeopathy is almost certainly magical thinking at best, ADHD is not a scientifically rigorous diagnostic criteria by which to measure the effectiveness of any treatment. Stimulants ‘work’ behaviourally but aren’t a treatment. So what were the parameters? It’s all cargo cult science anyway.
I like the way you break down the why of why you think the meta-analysis is "fucked", as you put it, but wish you had at some point defined the term "homeopathy". If you did I must have missed it. I also agree that most of what passes as "science" nowadays is plain garbage. It has been corrupted by government money.
So depressing. Since ivermectin I haven't known how to talk about meta-analyses with students. The truth is how I evaluate them is ~90% down to whether I trust the PI.
Lay person here: many years ago I read a compelling article in SciAm arguing we should not be dismissing, but rather harnessing, the strikingly high efficacy of placebo in almost every medical condition, but especially conditions such as depression where “effective” treatments get a pass even where effectiveness threshold is relatively low and where effectiveness of placebo can be quite impressive)
The writer (a clinician) argued: rather than throw away placebo, HARNESS IT. The ethical method he proposed, and uses on his patients, was really simple. Using depression as an example he would make the true statement ;
“I can prescribe you x pill which may have y benefits and x side effects, but I suggest that before we go that route we try St. John’s Wort which has been shownnto be effective for many of my other patients (or) which studies show has been effective in some patients ” He consciously uses his authority as doctor, knowing the white coat is important to the placebo setup.
To my mind this is where homeopathy fits. REAL sugar pills that have a demonstrable effect that is just as strong as placebo,
I had an experience in my 20s of homeopathy being an effective antihistamine for occasional hives. Me? Susceptible to placebo? Surely not! …but nobody thinks their body can be tricked by placebo, but a zillion placebo controlled studies suggest otherwise!
So if homeopathy showed any statistical effect on ADHD why NOT prescribe it before amphetamine? In a cohort that is 5-11 it may be the placebo effect on the parent that is helping the child. Don’t toss that effect in the bin!!
Thank you! Great article. I've made it a candidate for my Top Twelve of 2022.
The original idea of meta-analysis was that by combining many small studies, you'd have a big sample so statistically significant effects would have a chance to show up. Or, to not show up-- but you'd have higher power, more confidence that it wasn't just a small sample size that meant you got a null result.
In this homeopathy meta-study, though, what's happening is that the sixth study, the single-blind one has such a big effect that it's generating the entire result, it looks like to me-- even though it's being DILUTED, not ENHANCED, by the other studies. The purpose of a meta-study like this is just to camouflage the deficiencies of the one study that's generating the general conclusion from combining studies. Plus, you can claim that aggregating 6 studies produced result X, instead of just that one or two studies produced it, and other studies rejecting it. So the meta-analysis is just a rhetorical tool, a tool of obfuscation.
Depressingly, though I hadn't looked at this particular meta-analysis, this matches many experiences I've had, in reviewing and in critiquing published papers. You have to check all or most of the papers that were included, maybe redo some of the calculations with dubious studies left out, and then you're still not sure because it takes too long to redo the literature search. Obviously there are some very good meta-analyses too, but meta-analyses are so much more demanding than other kinds of study to check for quality.
Whilst I don’t doubt that you are right that this meta-analysis is a pile of garbage, the logic of your argument is hilarious. You essentially don’t believe it because you don’t believe it. If you believe the vaccine studies are better done than the work on ivermectin, I don’t know how to help you.
ADHD is poorly defined and prone to wildly subjective analysis and description of symptoms. So while I agree that homeopathy is almost certainly magical thinking at best, ADHD is not a scientifically rigorous diagnostic criteria by which to measure the effectiveness of any treatment. Stimulants ‘work’ behaviourally but aren’t a treatment. So what were the parameters? It’s all cargo cult science anyway.
6 studies with a total sample size of 309.. that barely seems to meet the standards for a single study, let alone a meta-analysis.
I like the way you break down the why of why you think the meta-analysis is "fucked", as you put it, but wish you had at some point defined the term "homeopathy". If you did I must have missed it. I also agree that most of what passes as "science" nowadays is plain garbage. It has been corrupted by government money.
So depressing. Since ivermectin I haven't known how to talk about meta-analyses with students. The truth is how I evaluate them is ~90% down to whether I trust the PI.
Lay person here: many years ago I read a compelling article in SciAm arguing we should not be dismissing, but rather harnessing, the strikingly high efficacy of placebo in almost every medical condition, but especially conditions such as depression where “effective” treatments get a pass even where effectiveness threshold is relatively low and where effectiveness of placebo can be quite impressive)
The writer (a clinician) argued: rather than throw away placebo, HARNESS IT. The ethical method he proposed, and uses on his patients, was really simple. Using depression as an example he would make the true statement ;
“I can prescribe you x pill which may have y benefits and x side effects, but I suggest that before we go that route we try St. John’s Wort which has been shownnto be effective for many of my other patients (or) which studies show has been effective in some patients ” He consciously uses his authority as doctor, knowing the white coat is important to the placebo setup.
To my mind this is where homeopathy fits. REAL sugar pills that have a demonstrable effect that is just as strong as placebo,
I had an experience in my 20s of homeopathy being an effective antihistamine for occasional hives. Me? Susceptible to placebo? Surely not! …but nobody thinks their body can be tricked by placebo, but a zillion placebo controlled studies suggest otherwise!
So if homeopathy showed any statistical effect on ADHD why NOT prescribe it before amphetamine? In a cohort that is 5-11 it may be the placebo effect on the parent that is helping the child. Don’t toss that effect in the bin!!
>It’s time we downgraded meta-analyses in our mental hierarchy of evidence.
Obvious question/challenge - what would you rank above them in that hierarchy?
I didn't realize that homeopathy had a moment in 2010.