A recent meta-analysis looked like good news for the effectiveness of "nudge" theory. Does a new set of rebuttal letters throw the whole idea into doubt?
I'm sorry to bother you with this as I am sure you must be thinking about it, but what do you think of the recent review which claimed that serotonin has no relationship to depression? Would love to read your take on it.
One major assumption here that is fundamentally wrong in the analysis of publication bias, is the requirement that all the reported effect "clouds" must be symmetrical even *throughout the zero effect*. Let me demonstrate with an example. You design a study to see whether say.. injecting bleach can cure Covid-19. You gather 100 people, inject 50 with bleach, 50 with saline. Hmm, weird, you replicate the study. Ten times, hundred times. You can plot funnels, clouds, but even if by some miracle you see a dot on the right, expecting any kind of symmetry is detached from reality.
So, yes, the variability in experimental design plays a role, but that's not all. You need to account for the fact that these experiments have been designed with a specific goal to achieve positive effects. And that may be a faulty mindset in science generally, but it is the fundamental nature of what nudges are about! In some cases, observing a negative effect size is actually absurd or even impossible. If you observe how many people sign up for organ donation form that's innocuously bundled to a driver's license application - would you expect that someone flips the table, storms off, and writes a last will saying: "When I die, dissolve my body in a vat of acid." ?
And even though thorough statistical interpretations are very much needed, sometimes the world is too complex for us to reduce into numbers.
i think a better term than "nudge" would be lying, manipulation, coercion, control etc. "nudge" is a propaganda term used to minimize the reality of how evil and sociopathic "nudging" (manipulating) people really is.
There's something I don't understand about the treatment here. Aren't all nudges apples and oranges, even closely related ones of the same type?
If one study finds with high certainty that a wasp picture in urinals have a positive effect and another study finds with high certainty that a butterfly picture in urinals have a negative effect (of the same size), is the relevant meta-conclusion that urinal-picture-nudges have no effect on average? Wouldn't the relevant conclusion for real-world practical applications be that nudges can be effective, but you have to test the effectiveness of the given nudge because you can't presume to know what it'll be in advance?
How does it make any kind of sense to look at an "average effect" of studies that study different nudges (even of the same type) and conclude that merely because the average is zero or very small - or would be if not for publication bias - this means we have no evidence for effectiveness of nudges in that domain overall? Just because a randomly designed nudge will have little to no effect on average doesn't mean nudges can't be effectively employed with the proper methodology for designing and testing them. Testing just has to be a part of the deployment process itself, rather than something left for academics to do later. or do I misunderstand something here?
The problem with having a nudge department is that, no matter what the situation, they're never going to say 'let's just be honest with the public and lay out the facts before them.'
As a nudger, your job security depends on persuading your bosses that the general public are a bunch of suicidal buffoons. Which can't be good for society.
You are British and the "nudge" guys are American yeah? I saw you say "pension scheme" and I knew you had to be British because in American English "scheme" is incredibly negative in connotation whereas "nudge" is about as neutral as you can get for what it is describing. We'd usually call them "pension plans" rather than "pension schemes". Planners are good and schemers are bad.
Many thanks Stuart for this excellent article. It would make a great addition to the nudge theory articles series I've been publishing at https://realkm.com/nudge-theory/
I wonder if I could obtain your permission for republication? The article would be republished verbatim, and I'm very happy to add backlinks and attribution as required.
This may be overly optimistic, but perhaps we can take encouragement that the open data provided alongside the original meta-analysis must surely be a factor which facilitated the speed of the corrections and commentary?
Nudged off a cliff
I'm sorry to bother you with this as I am sure you must be thinking about it, but what do you think of the recent review which claimed that serotonin has no relationship to depression? Would love to read your take on it.
(I sincerely hope) there’s a special place in hell for government bureaucrats and nudge-ists who think “changing the default” is a benign “nudge”.
Making something opt-out rather than opt-in is not some brilliant science-based choice architecture hack.
It’s simply coercing users by taking away their freedom to choose.
Compare “changing the default” to other judges and it’s clear that this is the most coercive technique (yet).
It is therefore no surprise that this is mentioned as one of the more effective methods.
Sorry for the rant but I had to get this off my chest.
Thank you for this essay/post (sorry, no idea what substack posts should be called), I really enjoy your writing
One major assumption here that is fundamentally wrong in the analysis of publication bias, is the requirement that all the reported effect "clouds" must be symmetrical even *throughout the zero effect*. Let me demonstrate with an example. You design a study to see whether say.. injecting bleach can cure Covid-19. You gather 100 people, inject 50 with bleach, 50 with saline. Hmm, weird, you replicate the study. Ten times, hundred times. You can plot funnels, clouds, but even if by some miracle you see a dot on the right, expecting any kind of symmetry is detached from reality.
So, yes, the variability in experimental design plays a role, but that's not all. You need to account for the fact that these experiments have been designed with a specific goal to achieve positive effects. And that may be a faulty mindset in science generally, but it is the fundamental nature of what nudges are about! In some cases, observing a negative effect size is actually absurd or even impossible. If you observe how many people sign up for organ donation form that's innocuously bundled to a driver's license application - would you expect that someone flips the table, storms off, and writes a last will saying: "When I die, dissolve my body in a vat of acid." ?
And even though thorough statistical interpretations are very much needed, sometimes the world is too complex for us to reduce into numbers.
i think a better term than "nudge" would be lying, manipulation, coercion, control etc. "nudge" is a propaganda term used to minimize the reality of how evil and sociopathic "nudging" (manipulating) people really is.
This does seem to come up often. I wonder if we should train ourselves to think "Meta-analysis - Is it publication bias?"
There's something I don't understand about the treatment here. Aren't all nudges apples and oranges, even closely related ones of the same type?
If one study finds with high certainty that a wasp picture in urinals have a positive effect and another study finds with high certainty that a butterfly picture in urinals have a negative effect (of the same size), is the relevant meta-conclusion that urinal-picture-nudges have no effect on average? Wouldn't the relevant conclusion for real-world practical applications be that nudges can be effective, but you have to test the effectiveness of the given nudge because you can't presume to know what it'll be in advance?
How does it make any kind of sense to look at an "average effect" of studies that study different nudges (even of the same type) and conclude that merely because the average is zero or very small - or would be if not for publication bias - this means we have no evidence for effectiveness of nudges in that domain overall? Just because a randomly designed nudge will have little to no effect on average doesn't mean nudges can't be effectively employed with the proper methodology for designing and testing them. Testing just has to be a part of the deployment process itself, rather than something left for academics to do later. or do I misunderstand something here?
The problem with having a nudge department is that, no matter what the situation, they're never going to say 'let's just be honest with the public and lay out the facts before them.'
As a nudger, your job security depends on persuading your bosses that the general public are a bunch of suicidal buffoons. Which can't be good for society.
You are British and the "nudge" guys are American yeah? I saw you say "pension scheme" and I knew you had to be British because in American English "scheme" is incredibly negative in connotation whereas "nudge" is about as neutral as you can get for what it is describing. We'd usually call them "pension plans" rather than "pension schemes". Planners are good and schemers are bad.
Many thanks Stuart for this excellent article. It would make a great addition to the nudge theory articles series I've been publishing at https://realkm.com/nudge-theory/
I wonder if I could obtain your permission for republication? The article would be republished verbatim, and I'm very happy to add backlinks and attribution as required.
I look forward to hearing from you, and can be contacted through https://realkm.com/contact/
Many thanks,
Bruce Boyes.
This may be overly optimistic, but perhaps we can take encouragement that the open data provided alongside the original meta-analysis must surely be a factor which facilitated the speed of the corrections and commentary?