Learning to do a meta-analysis

My last post about what a meta-analysis is, was partially because I decided to learn how to do a meta-analysis. I decided that while I was walking home last Friday and I realized I could have more than one blog. I quickly came up with a fantastic name for a new blog that had something to do with meta-analyses and then promptly forgot about it. I don't know if I'll start a whole new blog to discuss my process of learning about and/or carrying out a meta-analysis but I figured I would start blogging here.

I have a tendency to want to learn about new statistical techniques without then using them to do anything. I think I have to agree that actually using (or thinking about using) a new technique is much more useful in the end. Kind of like you may not think that doing example problems will help you understand a concept but (at least when going through some of the meta-analysis material I've been looking at) it can be really helpful. I have decided that the question I am going to plan / actually pursue is transactive memory's role on performance and turnover's role as a moderator. Not only is this an area I am interested in so I have a lot of understanding already, but there are not any meta-analyses I know of looking at this topic. DeChurch and Mesmer-Magnus did a meta-analysis in 2010 about team cognition which encompassed TMS but I think that a more narrow approach may be enlightening.

I am basing my current exploration on of the article I mentioned in my last post "How to do a meta-analysis". The accompanying website for the article is not super easy to find but is here: resource page. The first author's website, Discovering Statistics (aka Statistics Hell) seems to have a lot of good resources as well. The researchers who wrote the article, also wrote several scripts that can be used in SPSS and R (two statistical packages, the second is free). The webpage doesn't describe the process of preparing the data (you'll want to read the paper or this short article for that) but it does provide some example data for you already. The authors claim this data is (or is based on) published articles, so I'm guessing that I should be able to replicate the work those researchers did.

If I continue this exploration further, I'll keep you all in the loop.

What is a meta-analysis?

Many of you may have heard the term meta-analysis either in this blog or other places on the web. Because of the amount of data and the power of our analysis software, these kinds of analyses are done fairly frequently in a lot of topics. So what is a meta-analysis? Essentially, it is a statistical way of adding a bunch of different studies together that are looking at the same thing to determine what the real effects are. Let me use an example.

If you are looking at 2 studies (or 2 articles reporting on studies) that are looking at the same question but come to different results, what can you use to determine which is the most valid? There are a few general rules of thumb. If one of them comes from a more notable research institution, it may be better because these schools typically have stricter institutional controls. Another important factor is sample size and population. If the sample is entirely college students, there are reasons why you might not trust that finding as much as if the sample was more diverse. Also, if one study had 50 people and the other had 500, then you might trust the larger one more.

It may seem obvious, but why actually do we trust the more diverse or the larger studies? Studies where they find effects even with diverse samples suggests that the effect is likely to be more prevalent. Diversity always adds some amount of variation to human subjects research. In a study I am running, we are using computers and we found that it was a good idea to limit the age of participants because some participants had much more trouble since they were not as familiar with computers as the younger participants. So, reducing the diversity of the sample can let researchers narrow in on results they are interested in. Sample size effects the likelihood of finding an effect to begin with. As the sample size increases, a number called the standard error decreases in the analyses. This means that the analyses can become more confident of the effects each variable has.

What a meta-analysis is, is a tool that lets researchers combine multiple studies together. Through that process, the sample size gets bigger which allows us to be more confident and, due to the aggregation process, the sample also becomes more diverse because studies will have used different kinds of people and possibly different methods in carrying out the experiment. Meta-analyses can be done incorrectly and can be misleading, but a good rule of thumb would be to trust a meta-analysis about a topic more than any single study.

Extra Fun Facts: File-Drawer Effect

The process of doing a meta-analysis of course adds some difficulties for the researcher in trying to 'wash out' the potential added noise (a term for unintended variance) from the analysis. There are many possible problems such as the 'file-drawer effect'. It is well-known that a lot of the work that scientists do never gets published. A big factor of this is non-significant effects. If you run an experiment, for example, and do not find what you are looking for, you may assume that you did something wrong. One professor I had mentioned that he ran one study over 3 times, never quite finding the effects he was interested in. Because of this, he never published any of the studies. [Later on he did a small meta-analysis of just these experiments and found that there was a small effect that he was only able to see when adding all of the data he had collected together.]

There are two main reasons for the file-drawer effect. Researchers may be embarrassed or not see value in proclaiming to the world that they found nothing (significant effects are 8 times more likely to be submitted), and academic journals are hesitant to publish articles without significant effects for the primary variables of interest (7 times less likely to be published). There are some legitimate reasons for this hesitancy. A study can fail to find effects for a lot of different reasons (actually no effect, poor design, too small sample size, inappropriate analysis, etc.), but there are fewer conditions under which a study will find effects when there are none. Therefore, if you did a meta-analysis only using the data that were published, there may be an over-representation of the actual effect than in reality. If you are trying to determine the average grade for the class but only included students that made above a certain grade or attended every class session, you will get an average that is likely to be different from the actual average. There are various and sometimes complex ways that researchers try to deal with these problems but it is always a concern.

* I used Field & Gillett (2010) "How to do a Meta-Analysis" significantly in this post.

The presentation of science and scientists in movies and television

I have been watching the new version of the Cosmos television program, and it started making me think about how science has been presented in general. In Cosmos and other educational media, science is typically presented in a positive instructive light, but other presentations of science and scientists vary in the way they construct the image of the scientist. The mad scientist is a common cultural construction that has been prevalent since at least Frankenstein created his monster. This and many other slightly less mad scientists have seemed to often present scenarios that the scientist had never considered and show how a massive problem could stem from it. Frankenstein never thought that his monster would be violent for example and never took precautions.

Then there are the scientists that are immune to human danger when something worthy of research becomes apparent. Walter in the television show Fringe acted in such a way when he experimented on children, and many scientists in media have exclaimed the opportunities for research as a grand explosion/tear in space time/contact with aliens or other such calamities begin their desolation. Scientists in these situations are painted as naive, narrow-minded, or lacking in common sense. When I was in middle school I was also told that I lacked common sense. It has sometimes made me wonder if this societal endowment of unintentional malice on the scientist is a way of aggrandizing the every-man to a higher level than the thinker because he has common sense to know that the alien is probably going to kill everyone. Militarists often get this presentation, such as in Aliens [most recently to me in Final Fantasy the Spirits Within (which notably has a rather positive presentation of scientists though they happen to practice something more akin to magic)].

Social scientists typically are presented in the media only to the extent that psychotherapists and the occasional academic appears on television. I know little about psychotherapy but their presentation in the media I consume often highlights the Freudian influenced therapies of the past due to their notable eccentricities as opposed to more reasonable modern therapies. Group psychology as it is brought up in procedural crime shows such as Criminal Minds typically centers on cases like Kitty Genovese's. She was molested and murdered while in a public area without intervention from bystanders. I have heard characters spout of other social psychy sorts of things but often guffaw at the ridiculousness of the claim. Granted, the writers of the show are most likely to have experienced a Psych 101 type course where these more shocking cases are typically presented. The hard scientists seem to sometime have an easier time in their presentation, I think often due to the physical nature of their work.

I just finished watching a short film where apparently the designers of a robot somehow allowed it to be possible for him to be abused into murdering a family. This type of plot device seems to assume ignorance on the part of the designers. A logical extension could be, we should hold back such and such work because, as we saw in such and such a film, there can be unintended consequences. We shouldn't research AI because robots might kill us. This is the type of concept that can only really slow down the work that is being done. I often wish I could talk to the creator of the art and ask "Why do you as an ignorant observer feel like you know more about a scientist's work than they do."

A recent example of this occuring in the real world would be when the Food babe exclaimed that a chemical used in yoga mats is also used in Subway's bread. Her ignorance of science was held up by many in the media and the public as a liberating force in the war against non-natural food. Subway was forced by public opinion to promise removal of this chemical though it has not been found to be dangerous. Her subsequent judgement on cookies given out freely by Doubletree appears to be based on a misreading of the chemical list. A possibly honest, but illuminating, mistake that highlights the authors lack of journalistic fact checking.


One of the more interesting presentations of a research scientist I have seen recently was actually from the 1954 film Godzilla. This was the original Japanese version though I have also seen the American version (which makes some notable changes to the character of the scientist that I will address) several years ago. In this film, the main scientist has made a decision long ago that his research should not be made public until he has found a way to use it in a non-violent way. The product of his research is the "oxygen destroyer" which will remove all the oxygen from a body of water, effectively killing any living thing in that water. The scientist, Dr. Serizawa, makes a very ethical stance in this case, if the work is revealed at this stage, it will only be seen as a force for destruction. A German colleague outs the relevance of his work to a reporter in the wake of a desolating attack of Godzilla on Tokyo. After some convincing, the scientist agrees to use the oxygen destroyer on Godzilla after he has destroyed all of his research. After he knows Godzilla has been destroyed, he ultimately commits suicide to seal the secret of the oxygen destroyer away. In the version of the movie for the US, an American reported called Dr. Serizawa to help convince him to use the "oxygen destroyer" on Godzilla. I do not recall specifics but I think the reporter, played by Raymond Burr, thinks the scientists concerns are not very credible and dismisses his hesitancy. Ultimately, Serizawa takes the only option he thinks is logical, the use of his weapon to save Japan, but the prevention of his weapon being used to destroy the world. Serizawa was always looking out for the long-term good of human life as opposed to the short-term needs of the people around him. Though this may seem as dissmissive of the current struggles, I think that this representation shows the forethought and consideration Serizawa put into his decision.

This type of representation of a scientist is very different from films like Day of the Dead where the scientist is seen as an insane and strange pseudo-villain whose work is nonsensical and dangerous with no regard for those around him. The scientist has long abandoned his work to cure the zombies but instead has sought to teach them. He is then murdered, to comical effect, for his belief that zombies could be civilized. The assumption the director is implying is that there is no solution to the problem, there is no hope except isolation. Research is pointless and can only lead to danger and death.

There is also the long line of fiction where the scientist builds some great machine that spectacularly fails, destroying the scientist and everything around him/her. In many of these works, the simple solution is the correct one. If the powers that be had only listened to the hero (who often has low status) then bad things could have been avoided (there are many examples of this in Japanese cinema, Final Fantasy the Spirits Within an example).