Comments on Data Analysis and Statistics

Analyzing data is quite an odd experience in the research world. In statistics classes, you learn about a lot of complicated models, tests, and assumptions. But, my experience analyzing data from experiments is that much of what is learned in the classroom is ignored. That isn't to say that I willingly defy the instruction I received in my classes. Instead it is that the things I learned in the majority of my stats classes are not that important for studying experiments.

Why is there this disconnect? First, most experiments are technically immune from a lot of the potential problems that stats classes teach you about. If there is random assignment to condition then individual differences shouldn't matter. Manipulations and some dependent measures can be thought of as perfect measures because they are the thing itself. I don't have a huge amount of experience in this area but the data I have typically gotten from experiments does not typically violate assumptions or is unable to violate the assumptions on ANOVA or regression such as independence. A social psychologist once implied to me that, in our field, if you use fancy statistical techniques or describe all of the tests for assumptions that you ran on the data that it can make any effects less believable. The researcher proposed this because most of the effects we investigate are measurable by ANOVA or linear regression. The researcher may be using fancy statistics because that is the only occasion when the effect exists.

This is an interesting situation because, if this is really the case, it suggests that at least part of social psychology is unwilling to accept advances in statistical procedures or statistical rigor because they don't want to be seen as hiding behind the math. If a paper doesn't use one of a small handful of methods then they are open to criticism for their statistical methods. If they use simple analyses, however, they are less open to complaints about their statistics. This may not be the true case or the case in the majority of social psychology, but I have reason to believe that it exists. It is also certainly true that a sign of experimenter 'p-hacking' is the use of convoluted analyses that may not be entirely appropriate, leading to spurious effects.

I don't mean to suggest that new innovations never make it into social psychological research. Preacher and Hayes have made a huge splash in the psychological community by introducing a way to more accurately gauge the existence and the effect size for many kinds of statistical mediations. I think that a partial reason for this acceptance, however, was their demonstration that the traditional ways of testing for mediations were more likely than their method to say that there is no mediation when a mediation does actually exist. This fact made the acceptance of a new method more appealing to the community at large, partially because it is more accurate and especially because it is more accurate in such a direction that mediations that were previously not supported may now be supported.

It is an interesting world that I honestly do not know much about. As far as journal publications go, if the editor and reviewers (normally no more than 5 people in total) think the stats you use are okay then your work can be published. If the stats are easier to understand, then your work is more likely to be published. Also, if your stats are very hard to understand because the methods are very obscure or new can also lead to your work being published. Though there were multiple issues with Daryl Bem's 2011 paper in JPSP (a very prestigious journal), but one criticism was that the stats he used were too complex and picked up on subtle, random differences. I think that the analytical world I live in is very interesting, but I just don't understand it sometimes.

Turnover and enactment of change

In much of the literature about turnover, it is unclear how newcomers are influencing the outcomes for the groups that they join. More recent literature has attempted to categorize the ways that newcomers adapt to the groups they join and how groups adapt to the newcomers. In the study I describe today, the researchers were curious what factors influence whether a newcomer is able or willing to share their ideas with the rest of the group. There is an assumption in much of the management literature that newcomer's primary value is in the new ideas they bring to the group. But, under what conditions that occurs is less clear. It certainly is not as often as possible or there would be much more value in the world.

The study I want to describe is Kane, Argote, & Levine (2005). These researchers decided to use an experiment to investigate some of their ideas using the frame of social identification. The researchers proposed that if group member shared a common social identity with a newcomer that they would be more willing to accept new ideas into the group. Unfortunately, this study was not able to determine directionality (whether the effect is newcomers' willingness to give ideas or oldtimers' willingness to accept them) but this step was a great step forward in this research.

In this task, the participants made paper boats in assembly lines. The researchers demonstrated how paper boats could be made but were clear that their requirement was to make as many paper boats that fit the requirements for the task and not just this specific boat. Some groups learned a method of making paper boats that required 7 folds while other groups learned a method that took 12 folds. Though the group with the smaller number of folds had one fold that was somewhat complex, they were much more efficient in general than groups with 12 folds (based on pretesting). The groups were told to use an assembly line to construct the boats and the more difficult fold is done by the middle member.

The other manipulation in the study was whether the groups shared a sense of collective social identity with each other. In each experimental session, 2 groups were brought into the lab at the same time. They both also participated in a training period in the same room. However, in the high social identity condition, the groups were given the same names, seated in an integrated fashion differently, and given a reward scheme where the performance of both groups would lead to better outcomes for all the individuals. In the low social identity condition, these three factors were changed so the groups seemed less similar to one another and their reward was not interdependent with the other team.

The other action in the study that the experimenters made was very clever. The middle members for each group switched from participating with one group to participating in another group. Therefore, for some groups the new member had the same experience as the group they are entering (experience with low or high efficiency folding techniques) whereas for other groups there was a difference (the new member has the low or high efficiency folding technique but the group they join has the opposite). The new member is therefore in a position where they need to learn the technique the group is using, or the member needs to try and get the group to accept the way of doing the task that they are most used to.

Skipping ahead to the results, almost no groups accepted the newcomers folding strategy if the strategy was worse than the one that they already have. There was also a main effect of the identity. Shen the groups had a shared identity, they were much more likely to accept the new member's strategy into their group. If the groups shared an identity (from having the same group name and having a shared reward structure, then they accepted the new member's better way of constructing the boat about 70% of the time. If they didn't share that same identity, however, then the group didn't accept the new member's superior way of making the boat that often (only 25% of the time).

The results for performance were a bit harder to interpret. All groups performed better over time, generating more boats in the last trial than in the first one. But there wasn't a strong direct relationship of the new member having a superior routine and performance. The researchers found that when the new member introduced a better routine to the group, that they experienced a larger increase in their performance than when the new member has a worse routine. These differences, however, are only for groups that share an identity with the new member. If the group didn't share an identity with the new member, then it didn't matter whether the new member had a better or worse routine, partially because these groups accepted that routine infrequently.

In context, this study was very significant for a few reasons. First, Argote had done significant work on learning within groups and organizations. This, however, was one of the first study that demonstrates both how learning can occur within a group and that there are certain variables that influence whether a group can learn from a new member. The variable of interet here was social identity but other work has looked at a lot of other factors (see Rink et al., 2013 or a review). Second, this study demonstrated that groups have the ability to recognize advantageous strategies and use them. This was demonstrated in some earlier work by McGrath, but Kane's study is a very clean experimental setting. Lastly, the results suggest that learning new strategies can be costly to a group, hence the small differences in performance for groups where the new member had a better routine compared to groups that received a new member with a less efficient routine.

The research process and study design

I was talking to another PhD student the other day that was presenting a schedule of the work that she was planning to do over the next few months. One project that she is deeply involved in at the moment in is analyzing data that has been collected from a set of real organizations. However, she also wants to then test these findings in the lab. I thought this might be a good opportunity to talk briefly about study design and what different kinds of research exist within social science.

Ideally, the research process goes in an order vaguely like this: A researcher comes up with an idea about how the world works, the relationship between some set of variables, etc. In most branches of social science, the researcher then creates a set of predictions about how different variables will be related to one another. This is less necessary in some fields such as (non-behavioral) economics. The next thing the researcher decides is what is the best way to determine if this relationship exists. Sometimes the question itself will inform what data should be used to test for an effect. If the question is, for example, about the relationship between stock price and employee stealing, then looking at a real organization may be ideal. Once a data source is identified, the researcher collects the data and does analysis on these data. After the researcher has interpreted the results, the work will go onto the publication process, either into a journal article, book, book chapter, or conference presentation.

I work in a very small world where I have used experiments in all of my work. My experiments, though not identical, have certain elements of design that I consistently use which adds familiarity to the design process for my studies. I know the manipulations and the kinds of acceptable tasks very well. Though the specifics have taken some time in the past to work out, I don't think it took me more than a few days to design each of the studies that I have used. The longest time has always been determining the task to use. The difficulty with tasks sometimes is the balance between creating a new, novel task that the participants won't be familiar with and choosing a task that has been tried and tested by you or your colleagues.

When I looked at the other student's schedule, I was genuinely surprised that she had 3 weeks scheduled for study design. When I talked to another student, she thought that 3 weeks was just about enough time. This interaction got me thinking why I was so surprised that the student chose such a long period of time to dedicate to study design. I don't think I am overly skilled at study design, but I could be using a different definition of study design than they were.

When a lab study is designed, the major decisions that have to be made are the task, the manipulations, and the measures. My manipulations have always been rather blunt and heavily tested: employee turnover or restricting communication. The manipulation of more delicate factors, such as feelings of group belonging, of fear, or feelings surrounding the exchange of favors, are, I imagine, much more difficult and may have smaller impacts on people. There are huge literatures investigating these factors, which may actually instead the time it takes to choose a manipulation because the researcher may feel like they need to be familiar with most of the prior work. I don't mean to come off as dismissive of other work, but if you spend all of your time reading all the published literature in your area, you'll never add to that literature yourself. It is a dangerous game of academia, unless you work along narrow specialties (which has been my strategy).

Once the core vision of a study has been determined and the three decisions mentioned earlier (task, manipulation, and measure) have been chosen, the materials have to be put together. I don't typically think of this as design, but it is a necessary part of the research process. This is the phase where study materials are drafted, the specifics of the task are decided, materials are purchased, and advertisement materials are readied. Another unsung, but important aspect of this process is the writing of a script. I was fortunate to have a reader on a student project strongly suggest I write one for my first solo project and graciously provided me an example. The script lists all the actions the experimenter does to prepare for the study, all the things the experimenter says, when things occur in relationship to one another, and the timeline of the study. Writing the script always has a way of highlighting to me glaring issues with the design of the study in both a shallow sense (operalization) and a deeper (theoretical) sense.

I hope this post provides you with some insight into the nuts and bolts of the social science research processes and may provide some tips to other scientists.