Things You Learn from Skimming 1350 Academic Journal Articles
Thoughts on the usefulness of wastefulness
I’ve been working on a meta-analysis for the past several months. It’s been a doozy. It’s taken up an enormous amount of my time. And it will probably take me another year to complete, at least.
A meta-analysis is the workhorse of the psychological sciences — and really, just about any science, now that you have hundreds of thousands of researchers all doing their own thing. A meta-analysis is like a review, except it’s packed full of nutritious numbers to the point where it is ready to burst. The goal is to comb through other people’s research, record their numbers, and then compare and combine numbers across studies to see how they all fit into the big picture.
So, with that being said, I’m not going to throw numbers at you. I know. I know. Please breathe — you’re safe here. We all have deep trauma associated with math. I won’t trigger yours today. I’m not going to talk statistics; instead, I just want to talk about the experience of going through so many papers.
There aren’t many places where you get to see such a broad crush of humanity on display. Airports. Amusement parks. Crowded streets in Tokyo. Good parks, if you’re the kind of person who can sit patiently on a bench and observe others without looking creepy.
Meta-analyses are one of those places where you get a chance to see humanity on display. They’re like a private viewing available only to scholars. When you conduct a meta-analysis you look through dozens — sometimes hundreds, sometimes thousands — of papers on a topic. To give you an idea, I have a list of about 1350 that I have to go through.
There is no better time than a meta-analysis to get a sense of the broad range of scientists working on a topic, or what they’ve done, or what they hope to do, or the stories they tell themselves about the work they’re doing.
Plus, the quirks; when you read enough papers you stumble across gems like an entire line of papers on marriage and children by two authors with the last names Musick and Bumpass. Or the man with the most scholarly name I have ever seen in print — the inimitable U.R. Dumdum.
The problem is that, small joys aside, you can only make it a short distance into a meta-analysis before you start to feel that most of what science has done is a pile of hot garbage. There is a reason that most papers in the psychological sciences are hardly ever cited by other scientists: most of them add nothing, or at least nothing noteworthy.
There are reasons for this sense of wastefulness. One reason is redundancy: Scientists unwittingly trace the same contours endlessly, repeating themselves with small variations on the same theme. In the psychological sciences there aren’t many researchers who are daring enough to do something new — not when their tenure depends on producing a steady stream of acceptable papers.
Another is the smallness of any given study. Say that a researcher does want to do something new. Often that “new” thing takes the form of a program; the individual studies don’t add much except that they advance the argument the researcher is trying to make, one lemma at a time. It’s only upon synthesis that they produce a useful meta-level insight that might add something of value to science.
Another reason is bad practices. It’s well known in the social sciences that you can take any two measures of negative concepts — say, anxiety and procrastination (which happens to be the topic of my meta-analysis) — and they will be related to each other by default. Negativity is transferrable; a person who checks a bubble on a survey saying that they believe they’re a hot pile of garbage is unlikely to have good things to say about politics and they are unlikely to be optimistic about the future.
So when you start a meta-analysis on the relationship between two negative variables? It’s going to get redundant quickly.
With all the waste you might think that the social sciences would be better served if we were to just get rid of all those junk papers and be efficient. There are problems with this type of thinking, though. Three, really, which are all inter-related, and which we could spend a long time unpacking, but I will try to state them succinctly here.
I. What is waste anyway?
Who decides what is waste and what is not? And what criteria do you use? You might suggest that waste is obvious and easy to spot, and all you need to do is put a judicious regulator in charge, but that implies that a regulator is capable not only of judging the present worth but also the future worth of a line of inquiry.
My favorite example of this is from the biological sciences: The polymerase chain reaction is a necessary part of genetic research; it is used to rapidly create millions to billions of copies of a segment of DNA. The technology itself is dependent on thermostable polymerases which are derived from work on extremophiles — organisms which can survive in extreme conditions that most life would not.
But that research began with one fellow who grew particularly interested in a green sludge that he spotted in a thermal vent in Yellowstone natural park. Nobody cared much about it, then — it wasn’t until decades later that it blossomed into something that had undeniable application.
II. Who is the research for?
Every research paper comes with a sense of who-ness and if you’re not attuned to it you could completely miss why it was done.
Researchers who evaluate scientific literature after it has been published each have their own purpose for it in mind, and that purpose shapes their judgments of its worth.
Say, for example, that your goal is to provide broad insight on the relationship between anxiety and procrastination. It doesn’t take more than a few papers before everything else on the topic becomes redundant to you. After the tenth paper showing that anxiety correlates positively with procrastination you can almost certainly guess the eleventh accurately, and the fiftieth. Really, the reason you add the remaining four hundred papers to your database is to be thorough, not because you lack an answer.
But the people who wrote the papers had other aims in mind. Say that one researcher wrote a paper on the relationship between anxiety and procrastination in schizophrenic patients, and another researcher wrote one on the relationship between anxiety and bedtime procrastination. To an outside reviewer these might seem redundant (and, having reviewed several of each, I can say that the results are near identical). But to the ones who wrote them? They may have important meaning.
That paper on anxiety and procrastination in schizophrenics may have been written by a researcher who works with schizophrenics and wanted to address several patterns of self-sabotaging delay that their clients exhibited. In the absence of a meaningful paper dealing with the schizophrenic population, the researcher can only draw on broad principles to make their argument. With a published paper under their belt they can say “no, we have evidence that this is a pervasive problem affecting people exactly like the ones we work with every day.”
Ditto with the researcher who decided to write a paper on bedtime procrastination. I am certain that it matters to many people receiving treatment for sleep disorders — because I have a sleep disorder as well, and it seems largely driven by a tendency to put off sleep in favor of work. Was there even a language for this until somebody decided to call it “bedtime procrastination” and conduct research into it? No matter how derivative it is of the original literature, it still has value in sparking a conversation among the professionals who work with it and the people who do it.
This is one of the overlooked functions of research papers: often they allow a person to open up a dialogue on a new topic, not with the world, but with other professionals in their immediate vicinity, and to discuss it with some authority. Is it right, then, to judge the utility of a paper solely on whether it advances some broad theoretical agenda?
III. Is waste actually wasteful?
A third consideration, who says waste is wasteful? Back in the realm of DNA, researchers have known for many decades that the majority of DNA is non-productive, in the sense that it does not produce proteins that are used to construct life. But that does not mean it is useless. Researchers have identified many potential (and actual) uses of non-productive DNA over the decades, but my favorite comes from a study conducted in 2019.
The authors of the study found that the non-productive DNA was a site for the spontaneous assembly of new proteins that were not currently in circulation. Or (and I may be taking liberties here), it appeared to be one way in which evolution occurs; the random matching and mismatching of all that non-productive DNA spontaneously produces “options” that may become fodder for natural selection.
Okay. So, how does this apply to you?
So far this has been a purely abstract set of considerations, grounded in the type of science talk that a lot of people may find incurably boring. But I’m going to take a slightly strange direction with it. I think that we can take the principles above and apply them to life, and specifically your own life. There’s a good lesson in here about reserving judgment.
To do that we just take the principles above about judging the wastefulness (or productivity) of an academic journal article and apply it more broadly to personal projects and life goals. So, here goes:
What appears wasteful to you in the present may be of great value to you in the future if it is given the time to develop to its logical end. Don’t judge the usefulness of your actions too quickly.
What appears wasteful to you may be of immense value to somebody else — don’t judge the usefulness of other people’s actions based only on how they affect you, and don’t judge the usefulness of your own actions based only on how they affect you.
What appears wasteful to you may be the raw stuff that new ideas and new directions are born from. Don’t judge the usefulness of your actions until you have given them enough time to surprise you.
All of that is to say, don’t be too hard on yourself. Productivity-heads are prone to believing that life must be optimized and that waste must be removed in favor of laser-sharp focus on a few essential things (chosen ahead of time) that truly matter.
I am not sure if that is an appropriate or healthy attitude to take towards life. It seems to me like it filters out the serendipity.
Close friends are produced from thousands of interpersonal connections, most of which never bear fruit. Think of your best friend — and now imagine that if you hadn’t bumped into a thousand random people during your childhood, most of whom didn’t matter, you may never have met them. Were the other thousand encounters really junk? Or were they all worth it for that one golden connection?
Another way to think of this is that you can’t optimize until the mess of life has given you something to work with. You can’t produce something optimal by cutting out mess — you can only focus on something optimal that has already been produced. Those are different things. The mess itself, with all its serendipity and false paths and boring and exciting and distracting moments — that’s the thing that dredges up the gold for you. So don’t shy away from it too quickly.
J
I'm going to use this as an opening reading for new grad students 😁. Should make for interesting discussion! Also, I teach writing which means I teach reading which means I teach strategies for dealing with reading and writing, one of which is forbearance (hmm, is that the right word?), a type of mental stoicism that acknowledges slogging through in hope toward the eventual insight.