Author of that irritating and tiresome paper here!
I of course agree that your paper should be accurate. Accuracy means honesty, and honesty is just naturally interesting, and often funny.
You’d think it would be otherwise, but in my other life I teach improv comedy, and this is one of the surprising things you learn. (The foundational text of improv is called Truth in Comedy.) People are boring when they try to be funny; they are way funnier when they just say the things that come to mind. That’s how people make each other laugh all the time in normal conversation.
Often papers are boring for the same reason: they are trying to lie to you. They want to pretend they know what’s going on, or that nothing went wrong, or that their results are extremely important. Many papers are founded on the lie that the authors care deeply about what they’re writing, and they aren’t desperately trying to publish a paper, any paper, so they can get a job. (I have written such a paper myself!) Lying takes up a lot of space, and it is incredibly tedious both to write and to read.
When we wrote in our paper that we forgot why we ran one of the studies, for instance, that was just true. We didn’t plan it that way, and tried to avoid admitting it until we decided to be honest instead. I know that makes me look dumb, but oh well! I’d rather be an honest fool than a venerable liar.
(This is also why we posted all the data, code, materials, preregistrations, etc. if we made mistakes, why hide them? We *want* to know if we made mistakes, so we can fix them!)
Humans are naturally interested in all sorts of niche and complicated scientific questions. I think that’s one of the best things about humans. I don’t believe that some corners of science are simply boring, and that anyone studying them must resign themselves to having a bad time, or that boring yourself and others is somehow a noble and necessary part of science. I think this is an unfortunate result of forcing people to suppress their natural curiosity, whether that happens in school, or whether people do it to themselves when their curiosity gets in the way of their career.
Of course people can try to force their papers to be interesting by following bad advice like the stuff these marine biologists provided. I’ll bet you that the papers that result are far more tedious than they would be if the authors were simply honest.
In short, when people say “tell a good story,” they really mean “lie,” and that’s both obviously wrong and it won’t work anyway. I say: be completely honest, and a good story will result. It just may not be the one you thought you’d tell.
(Sorry you didn’t like the jokes. Can’t please everyone! 🙂)
I write legal briefs for a living. When I read a scientific study that sounds like I wrote it, I immediately dismiss the findings because I know how easy it is to backfill and reach the result you wanted all along.
That there are people out there saying scientists should write more like lawyers is troubling. Many Covid studies already read like legal briefs. We need a lot less of this science-ganda, not more. I would go so far as to say that scientists thinking, speaking, and writing like advocates for some cause or policy, as opposed to neutral arbiters of a messy reality, is one of the gravest threats facing society. Having activists produce our foundational knowledge is like driving while wearing kaleidoscope glasses.
Why would "writing backgrounds" even make for a better story? Something really annoys me about that. Surely a good science "story" plays up the messiness and mystery, inviting the reader to join in the speculation about why things don't add up.
Ditto for conference talks IMO. Every psychology talk should end with a slide on what the presenter thinks still doesn't add up. Or maybe a slide about what would convince them to reject their current theory. The messiness should spark a fun discussion instead of defensiveness.
There was much to be annoyed by in that editorial. You are right parts of it seems to invite harking and selective reporting of results, both quite dangerous to the truth.
But I have to disagree with your premise: Scientific writing and scientific presentations are indeed story telling.... and that is a good thing.
An introduction needs to make a compelling case that your question is important, that your hypotheses are both novel, and theoretically grounded, and that a study such as what you've done is appropriate for answering the question? If it doesn't, why read the methods and results? The methods needs to make the case that you've done the study accurately and with attention to confounds and limits to generalization. If it doesn't, just put down the paper. And your results cannot just be a table with the output of your LMER or whatever fancy R package you are compelled to use this week -- you have to use those outputs to establish the validity of your work, evaluate your hypotheses, and rule out competing interpretations.
What is that, if not story telling?
The data does not speak for itself and it can't, particularly in a field like cognitive or social psychology. You can show me the figure or statistics of a paper even in my own subfield (psycholinguistics) and it will mean nothing to me. The data are meaningless absent a strong derivation chain (the set of assumptions linking a theory to an observable result; c.f., Paul Meehl, 1990, Psychological Reports), and a strong link to how similar measures have been used, interpreted (and challenged) in the past. That is, a 20 msec RT differences might be meangingless to a psychobiologist, but a massive effect in masked priming or an IAT study.
Psychology is not an observational science that just observes and reports -- we test hypotheses that are grounded in a theory. Data are meaningless outside of the collective narrative of the field. This is, of course, the very meaning of a Kuhnian scientific paradigm, and scientists do not operate out side of one.
Science writing must be story telling. Otherwise we will make no progress -- our collective understanding of language, cognition social behavior, or even marine biology cannot just consist of a massive table of "data". We need clear findings that can accumulate to understanding.
But scientific story telling need not be a fairy tale, nor should it be a sales brochure. It must be a faithful and honest story of what the researchers thought and why, and of what they learned from their study and why the believe it. There's no reason a compelling scientific story can't acknowledge inconsistencies (is that like an odd post-modernist short story that doesn't quite wrap it all up?) or even wrong hypotheses (is that the equivalent of a tragedy?) while still fitting into a broader and compelling narrative.
In our zeal for openness, and for all the critical values that have become more important in the paradigm motivated by reproducibility crisis, we should not forget that the goal here is collective understanding, and that requires stories. But story telling does not require nor demand dishonesty.
Thanks for this. I’m a big fan of your writing. And I enjoyed the ‘tiresome’ humorously written study and accompanying critique of peer review. Could you elaborate on the reasons you found it tiresome? I’m surprised at being so at odds with your response, and would be happy to identify and consider my blindspot.
What a terrible idea that "write your paper backwards from the end" is! As you state, it definitely creates opportunities for bad behavior and crappy science. It's unfortunate too, because many scientists DO need to learn how craft a narrative around their research to make it more understandable, but reverse-engineering a TL;DR from what you think is the final take-home point ain't it. Plenty of great science and medical writers like Dr. Atul Gawande and Dr. Siddhartha Murkherjee know how to present research as a narrative without getting it terribly, terribly wrong
Hello, this is off-topic. On Twitter, you retweeted a link to Ellen Pasternak's post, which discusses statistical competence among researchers (https://worksinprogress.substack.com/p/the-stats-gap). Comments are closed on her post, and I'm not on Twitter, so please allow me to share these thoughts here instead.
One of Pasternak's proposals is this:
> What I’d love to see would be departments like my department of biology hiring a handful of dedicated people for statistics support for all the researchers that work there. Just as you can pop in to see the people at IT support for help with your computer, you ought to be able to fire off a quick email to the stats office for advice on the analysis of your data.
My first reaction was, 'That would probably be great.'
Now I'm searching for what would make that first reaction wrong, and I have a few ideas. What do you think?
If a later study were to fail to replicate the initial study's results, would the initial study's statistician share in the blame? If so, that would give the statistician a strong incentive to spend hours or days on each little request from the researcher - as if the statistician were a real co-author, not help staff - and to keep records of everything on their own computer. Contrast that with speedy IT support staff, whose responses to requests are usually along the lines of 'Try X, and come back if it doesn't work.' (My assumption is that IT staff are so efficient and casual because they have little/no incentive to get everything right on the first try.)
How would the department react were the statistician, bored/jealous from years of playing a supporting role, to request to take the lead on their own study?
What if the statistician's data analysis were to produce results more 'harmful' than the researcher expected? Facing personal and professional incentives, the researcher might choose not to submit the study for publication. In that case, would the statistician be permitted to complain about academic misconduct?
Zooming out: Researchers gain statistical skills in the course of pursuing e.g. their passion for wildlife (Pasternak's example). Later, some of them can't find a position in their field, so they find unrelated jobs. These unrelated jobs sometimes involve the ex-researchers' skills with data. Were Pasternak's proposal adopted, these ex-researchers would have fewer skills with data. What happens to the average salary of these ex-researchers? What happens to employers' ability to find workers who are fairly competent with data?
I'm a little bit undecided on whether the idea of "telling a story" can or should be thrown overboard, because even a very complete and "dry" study report will have to make some decisions with regard to what's relevant and what not. I think thats a party of every communication to an audience. And as someone else already commented, data in psychology does not speak for itself and you will always have to make a case for even doing a study. You say there might be a middle ground here, but i'm not really wiser on how to decide where it is?
Something else that got me thinking is that I (as an ECR) have now read quite a bit about story telling and how to do it (or not) but so little about arguments. Maybe I'm missing something but I haven't read quite as much discussion on how to build an argument for something, what are good arguments, what are bad arguments, how to criticize arguments and their presuppositions and so on. I find this focus on story telling instead of on sound arguments when it comes to writing a bit strange.
I'm no scientist, so take my input with the requisite pinch of sodium chloride, but this whole Covid travesty had me brushing down my brain and doing my utmost to parse through scientific papers on natural/acquired/innate immunity, gene therapies vs vaccines etc. to try and figure out what was going on. And not because I aspire to be a scientist, or have any delusions about my abilities - but because these very scientist fvckers (excuse my french, but that's my only word for them) were trying to get under my very skin. And that is both literal and metaphorical. So it became personal, and as far as I was concerned, they lost all claim to keeping the layman out of their territory. So read them, I did. And no, I was not able to decipher the bulk of the statistical/data hieroglyphics or bespoke jargon - although some of it filtered through - but what I can do, is read for meaning. And when it came to abstracts and conclusions (the "storyline" or "synopsis" of scientific papers, if you will), then all I can say it doesn't take a scientist, or even a half midwit, to see that there is something very, very wrong with the scientific system. The number of papers I read, where clearly the scientists are wriggling, in great discomfort, on their ethically compromised and grant-laden hooks, is just so patently obvious, it became laughable.
In essence, you were seeing things like:
"The vaccine has a 95.372% efficacy rate and kills only 1.356% people, with a risk reduction of 98%(editor note: relative risk reduction in a cohort study of 162 very healthy people only, but we're going to try and bury that). Also, even if you recovered from the Rona, take it as well, because we've diddled some stats on that for you as well. So our conclusion is that the vaccine is safe and effective, so we urge you to toodle along for your shot forthwith, and so I can load my next grant tranche from my patron saint and benefactor, Bill."
Ok, so I took a bit of poetic licence, but all I did was write the quiet bits and add a dollop of snark. I will defy anyone to prove me wrong (actually, don't - I don't care what you prove now).
THAT is the now indelible impression that I have of the Scientific Process. And it's broken beyond repair. Or, at least, it will take many decades to sift through the excrement that has been produced in the last three years to put it to rights. If anyone bothers to. The damage is done and I, for one, would love to see a Samson bring the roof down crashing on all of their calculating, corrupt and ethically-compromised heads. Even if I have to try and live out the rest of my life without the fruits of their benighted labours.
Hi Stuart. Love your writing, loved your book and am always pretty much 100% on board with what you have to say. I'm an open science advocate myself, so I appreciate your argument here. However, I am also coaching people in and outside of academia to craft better presentations and dataViz to get their points across and my advice in these cases is dangerously close to the editorial, you are picking apart here: Engage the audience with clear communication, wrapped up in a narrative, supported by visual aids. Obviously a scientific paper and a scientific presentation are two completely different modalities and communication contexts, but I was wondering about your stand on these matters. IMO if the data & materials are open, if the paper is sound (ideally preregistered to avoid post-hoc story telling), I think certain communication channels should make use of story telling techniques to make your audience care and pay attention, to make them understand and to make them remember your possibly important research. Curious to hear your thoughts. Thanks again for all you have done as a science communicator. We appreciate it.
Author of that irritating and tiresome paper here!
I of course agree that your paper should be accurate. Accuracy means honesty, and honesty is just naturally interesting, and often funny.
You’d think it would be otherwise, but in my other life I teach improv comedy, and this is one of the surprising things you learn. (The foundational text of improv is called Truth in Comedy.) People are boring when they try to be funny; they are way funnier when they just say the things that come to mind. That’s how people make each other laugh all the time in normal conversation.
Often papers are boring for the same reason: they are trying to lie to you. They want to pretend they know what’s going on, or that nothing went wrong, or that their results are extremely important. Many papers are founded on the lie that the authors care deeply about what they’re writing, and they aren’t desperately trying to publish a paper, any paper, so they can get a job. (I have written such a paper myself!) Lying takes up a lot of space, and it is incredibly tedious both to write and to read.
When we wrote in our paper that we forgot why we ran one of the studies, for instance, that was just true. We didn’t plan it that way, and tried to avoid admitting it until we decided to be honest instead. I know that makes me look dumb, but oh well! I’d rather be an honest fool than a venerable liar.
(This is also why we posted all the data, code, materials, preregistrations, etc. if we made mistakes, why hide them? We *want* to know if we made mistakes, so we can fix them!)
Humans are naturally interested in all sorts of niche and complicated scientific questions. I think that’s one of the best things about humans. I don’t believe that some corners of science are simply boring, and that anyone studying them must resign themselves to having a bad time, or that boring yourself and others is somehow a noble and necessary part of science. I think this is an unfortunate result of forcing people to suppress their natural curiosity, whether that happens in school, or whether people do it to themselves when their curiosity gets in the way of their career.
Of course people can try to force their papers to be interesting by following bad advice like the stuff these marine biologists provided. I’ll bet you that the papers that result are far more tedious than they would be if the authors were simply honest.
In short, when people say “tell a good story,” they really mean “lie,” and that’s both obviously wrong and it won’t work anyway. I say: be completely honest, and a good story will result. It just may not be the one you thought you’d tell.
(Sorry you didn’t like the jokes. Can’t please everyone! 🙂)
I write legal briefs for a living. When I read a scientific study that sounds like I wrote it, I immediately dismiss the findings because I know how easy it is to backfill and reach the result you wanted all along.
That there are people out there saying scientists should write more like lawyers is troubling. Many Covid studies already read like legal briefs. We need a lot less of this science-ganda, not more. I would go so far as to say that scientists thinking, speaking, and writing like advocates for some cause or policy, as opposed to neutral arbiters of a messy reality, is one of the gravest threats facing society. Having activists produce our foundational knowledge is like driving while wearing kaleidoscope glasses.
Why would "writing backgrounds" even make for a better story? Something really annoys me about that. Surely a good science "story" plays up the messiness and mystery, inviting the reader to join in the speculation about why things don't add up.
Ditto for conference talks IMO. Every psychology talk should end with a slide on what the presenter thinks still doesn't add up. Or maybe a slide about what would convince them to reject their current theory. The messiness should spark a fun discussion instead of defensiveness.
Stuart-
There was much to be annoyed by in that editorial. You are right parts of it seems to invite harking and selective reporting of results, both quite dangerous to the truth.
But I have to disagree with your premise: Scientific writing and scientific presentations are indeed story telling.... and that is a good thing.
An introduction needs to make a compelling case that your question is important, that your hypotheses are both novel, and theoretically grounded, and that a study such as what you've done is appropriate for answering the question? If it doesn't, why read the methods and results? The methods needs to make the case that you've done the study accurately and with attention to confounds and limits to generalization. If it doesn't, just put down the paper. And your results cannot just be a table with the output of your LMER or whatever fancy R package you are compelled to use this week -- you have to use those outputs to establish the validity of your work, evaluate your hypotheses, and rule out competing interpretations.
What is that, if not story telling?
The data does not speak for itself and it can't, particularly in a field like cognitive or social psychology. You can show me the figure or statistics of a paper even in my own subfield (psycholinguistics) and it will mean nothing to me. The data are meaningless absent a strong derivation chain (the set of assumptions linking a theory to an observable result; c.f., Paul Meehl, 1990, Psychological Reports), and a strong link to how similar measures have been used, interpreted (and challenged) in the past. That is, a 20 msec RT differences might be meangingless to a psychobiologist, but a massive effect in masked priming or an IAT study.
Psychology is not an observational science that just observes and reports -- we test hypotheses that are grounded in a theory. Data are meaningless outside of the collective narrative of the field. This is, of course, the very meaning of a Kuhnian scientific paradigm, and scientists do not operate out side of one.
Science writing must be story telling. Otherwise we will make no progress -- our collective understanding of language, cognition social behavior, or even marine biology cannot just consist of a massive table of "data". We need clear findings that can accumulate to understanding.
But scientific story telling need not be a fairy tale, nor should it be a sales brochure. It must be a faithful and honest story of what the researchers thought and why, and of what they learned from their study and why the believe it. There's no reason a compelling scientific story can't acknowledge inconsistencies (is that like an odd post-modernist short story that doesn't quite wrap it all up?) or even wrong hypotheses (is that the equivalent of a tragedy?) while still fitting into a broader and compelling narrative.
In our zeal for openness, and for all the critical values that have become more important in the paradigm motivated by reproducibility crisis, we should not forget that the goal here is collective understanding, and that requires stories. But story telling does not require nor demand dishonesty.
It's good advice how to prepare a TED talk.
It also may be good advice for teaching, especially at introductory or one-shot audiences. One does, ultimately, want to get the point across.
But I agree that this can too easily slide into thinking and researching in reverse.
Hi Stuart,
Thanks for this. I’m a big fan of your writing. And I enjoyed the ‘tiresome’ humorously written study and accompanying critique of peer review. Could you elaborate on the reasons you found it tiresome? I’m surprised at being so at odds with your response, and would be happy to identify and consider my blindspot.
Thank you.
What a terrible idea that "write your paper backwards from the end" is! As you state, it definitely creates opportunities for bad behavior and crappy science. It's unfortunate too, because many scientists DO need to learn how craft a narrative around their research to make it more understandable, but reverse-engineering a TL;DR from what you think is the final take-home point ain't it. Plenty of great science and medical writers like Dr. Atul Gawande and Dr. Siddhartha Murkherjee know how to present research as a narrative without getting it terribly, terribly wrong
Hello, this is off-topic. On Twitter, you retweeted a link to Ellen Pasternak's post, which discusses statistical competence among researchers (https://worksinprogress.substack.com/p/the-stats-gap). Comments are closed on her post, and I'm not on Twitter, so please allow me to share these thoughts here instead.
One of Pasternak's proposals is this:
> What I’d love to see would be departments like my department of biology hiring a handful of dedicated people for statistics support for all the researchers that work there. Just as you can pop in to see the people at IT support for help with your computer, you ought to be able to fire off a quick email to the stats office for advice on the analysis of your data.
My first reaction was, 'That would probably be great.'
Now I'm searching for what would make that first reaction wrong, and I have a few ideas. What do you think?
If a later study were to fail to replicate the initial study's results, would the initial study's statistician share in the blame? If so, that would give the statistician a strong incentive to spend hours or days on each little request from the researcher - as if the statistician were a real co-author, not help staff - and to keep records of everything on their own computer. Contrast that with speedy IT support staff, whose responses to requests are usually along the lines of 'Try X, and come back if it doesn't work.' (My assumption is that IT staff are so efficient and casual because they have little/no incentive to get everything right on the first try.)
How would the department react were the statistician, bored/jealous from years of playing a supporting role, to request to take the lead on their own study?
What if the statistician's data analysis were to produce results more 'harmful' than the researcher expected? Facing personal and professional incentives, the researcher might choose not to submit the study for publication. In that case, would the statistician be permitted to complain about academic misconduct?
Zooming out: Researchers gain statistical skills in the course of pursuing e.g. their passion for wildlife (Pasternak's example). Later, some of them can't find a position in their field, so they find unrelated jobs. These unrelated jobs sometimes involve the ex-researchers' skills with data. Were Pasternak's proposal adopted, these ex-researchers would have fewer skills with data. What happens to the average salary of these ex-researchers? What happens to employers' ability to find workers who are fairly competent with data?
P.S. I wish you luck in your new role at iNews!
The suggestions (e.g., bullet-point main findings up front) aren't about how to *do the science." They are explicitly about *how to write the parent*.
I'm a little bit undecided on whether the idea of "telling a story" can or should be thrown overboard, because even a very complete and "dry" study report will have to make some decisions with regard to what's relevant and what not. I think thats a party of every communication to an audience. And as someone else already commented, data in psychology does not speak for itself and you will always have to make a case for even doing a study. You say there might be a middle ground here, but i'm not really wiser on how to decide where it is?
Something else that got me thinking is that I (as an ECR) have now read quite a bit about story telling and how to do it (or not) but so little about arguments. Maybe I'm missing something but I haven't read quite as much discussion on how to build an argument for something, what are good arguments, what are bad arguments, how to criticize arguments and their presuppositions and so on. I find this focus on story telling instead of on sound arguments when it comes to writing a bit strange.
I'm no scientist, so take my input with the requisite pinch of sodium chloride, but this whole Covid travesty had me brushing down my brain and doing my utmost to parse through scientific papers on natural/acquired/innate immunity, gene therapies vs vaccines etc. to try and figure out what was going on. And not because I aspire to be a scientist, or have any delusions about my abilities - but because these very scientist fvckers (excuse my french, but that's my only word for them) were trying to get under my very skin. And that is both literal and metaphorical. So it became personal, and as far as I was concerned, they lost all claim to keeping the layman out of their territory. So read them, I did. And no, I was not able to decipher the bulk of the statistical/data hieroglyphics or bespoke jargon - although some of it filtered through - but what I can do, is read for meaning. And when it came to abstracts and conclusions (the "storyline" or "synopsis" of scientific papers, if you will), then all I can say it doesn't take a scientist, or even a half midwit, to see that there is something very, very wrong with the scientific system. The number of papers I read, where clearly the scientists are wriggling, in great discomfort, on their ethically compromised and grant-laden hooks, is just so patently obvious, it became laughable.
In essence, you were seeing things like:
"The vaccine has a 95.372% efficacy rate and kills only 1.356% people, with a risk reduction of 98%(editor note: relative risk reduction in a cohort study of 162 very healthy people only, but we're going to try and bury that). Also, even if you recovered from the Rona, take it as well, because we've diddled some stats on that for you as well. So our conclusion is that the vaccine is safe and effective, so we urge you to toodle along for your shot forthwith, and so I can load my next grant tranche from my patron saint and benefactor, Bill."
Ok, so I took a bit of poetic licence, but all I did was write the quiet bits and add a dollop of snark. I will defy anyone to prove me wrong (actually, don't - I don't care what you prove now).
THAT is the now indelible impression that I have of the Scientific Process. And it's broken beyond repair. Or, at least, it will take many decades to sift through the excrement that has been produced in the last three years to put it to rights. If anyone bothers to. The damage is done and I, for one, would love to see a Samson bring the roof down crashing on all of their calculating, corrupt and ethically-compromised heads. Even if I have to try and live out the rest of my life without the fruits of their benighted labours.
Hi Stuart. Love your writing, loved your book and am always pretty much 100% on board with what you have to say. I'm an open science advocate myself, so I appreciate your argument here. However, I am also coaching people in and outside of academia to craft better presentations and dataViz to get their points across and my advice in these cases is dangerously close to the editorial, you are picking apart here: Engage the audience with clear communication, wrapped up in a narrative, supported by visual aids. Obviously a scientific paper and a scientific presentation are two completely different modalities and communication contexts, but I was wondering about your stand on these matters. IMO if the data & materials are open, if the paper is sound (ideally preregistered to avoid post-hoc story telling), I think certain communication channels should make use of story telling techniques to make your audience care and pay attention, to make them understand and to make them remember your possibly important research. Curious to hear your thoughts. Thanks again for all you have done as a science communicator. We appreciate it.