jumping to conclusions

Filed under: by: Aaron Price on Friday, August 19, 2011 @ 11:34 PM

There has been a lot of news in the last couple of weeks about a research study that found 3D movies do not enhance enjoyment, cause stress-related physical strain and do not help memory. But let's take a closer look before jumping to the conclusion that 3D in movies sucks.

First, the study was presented at an industry conference and has not been published in a peer-reviewed journal. That doesn't mean it is poor, just that it has not been checked out and, more importantly, it means we don't have the details of the study. All have is what was reported to the media at the conference. I'll quote this article:

Carrier and colleagues had 400 students watch one of three movies in either 2-D or 3-D: "Alice in Wonderland," "Clash of the Titans" or "How to Train Your Dragon."

After viewing the film, participants went home and completed an online survey. They were asked to rate how realistic the movie was to them and to report emotions and sensations they experienced, which they selected from a list of 60 words. The words ranged from mild emotions, such as "enjoyment," to more intense ones, including "anger" and "rage." Participants were also quizzed on their knowledge of the film.

The survey showed that neither group remembered the movie better than the other. It also showed that 3-D movie-watchers did not experience a greater sense of immersion in the movie's world, nor did they pay more attention to the film or report experiencing more intense emotions.

I have a number of beefs. First of all, self reported studies always should be taken with a grain of salt. Sometimes in psychology that is all you have because studying the human brain (emotions, etc.) is quite difficult. But such research design is not very sensitive, mainly because there is no zero point. That is the subjects are not on the same scale. For example, what "anger" means to one person is different than it means to someone else. Thus, the difference between "anger" and "rage" can be different between people. Secondly, the sample sizes are small. You have 400 people divided into two groups, with each group divided into 3, thus you have about 65-70 people per film per condition. Unless the results were very strong, I doubt it would be statistically significant. Again, we'll be able to check for all this if the paper is published (a high percentage of conference presentations never make it to publication - I'd put it at around 50%). Third, how can one judge whether a cartoon (such as How to Train Your Dragon) is more realistic in 2D or 3D?

I could go on and on but will stop because it is somewhat unfair to criticize research at a presentation I did not attend. But it was publicly released and the authors have made free with the media, so it is fair to ask questions.

The authors say: ""It didn’t seem to enhance your memory at all," Carrier said. "That’s an unfortunate implication."

Again, how did they determine that? Without an experimental study I just don't see how they can say that based on the study design described in the press reports. In science the importance of a claim needs to be correlated with the strength of your evidence (as Carl Sagan said: "extraordinary claims require extraordinary evidence").

I'm not a 3D champion. My first published study on this subject actually agrees with their core result! In my experience, 3D increased cognitive load and did not have any effect on accuracy of spatial cognition tasks I gave the subjects. And I buy the argument that 3D may not increase enjoyment of a film. Anecdotally I think audiences are starting to show that. And I think this study adds a new data point to the scholarly pool. But I think it's a minor data point and it does not mean what the press says it means. The evidence is not enough to say that 3D increases discomfort and does not affect memory. Not yet.