Scientists Avoid Bias When They Know They’re Being Tested

The test for this was rather complicated. In the first year of the experiment, all the committee members were made aware the experiment was taking place. As such, everyone would be expected to be thinking about the possibility of bias and be able to overcome their implicit bias in favor of males. The year following, however, it was assumed that this knowledge had worn off and that defaulting to implicit bias would be common. The exception would be those groups of reviewers whose default state is to be aware of bias issues.

To get a measure of this last possibility, in the first year, everyone was asked to complete a survey in which they were asked about the gender disparities in science and were given the choice of possible reasons, ranging from things like the challenges of balancing work and family, personal choices, or a lack of ability. This was converted to a score that represented an awareness of some of the hurdles women face in the sciences.

So, their hypothesis was that, in the first year, everyone would be in a position to overcome their innate biases. In the second, only the groups that had a default awareness of the hurdles women face would be in a position to do so.

Analysis and limitations

The researchers figured out the gender mix in each committee’s pool of candidates, then determined whether the gender mix of the successful candidates was consistent with the starting percentage. Overall, there was no evidence of significant gender differences in either year of the testing, suggesting that, as a whole, the committees did a good job of overcoming their biases.

Still, there were differences among the committees. Nearly half of them had a majority of members who felt that women’s lack of progress in the sciences was influenced by gender discrimination. And, if that belief was combined with a high level of implicit bias, then these committees had the largest decline in selection of women in the second year of testing. In other words, if a high implicit bias is combined with a low sense that women face barriers to advancement, then the committee selected relatively fewer women in the second year of testing.

This supports the researchers’ hypothesis that an awareness of the issues that women face in the sciences provides a degree of protection against the implicit biases that many of us have internalized. Fortunately, that’s now true for a majority of the committees in France, which seems to be enough to prevent an overall bias from creeping into this selection process.

Or at least creeping in at a statistically significant level. As the researchers acknowledge, their study is very small, as it was forced to do evaluations at the committee level, rather than analyzing the individual choices of the 400-plus committee members. There are a lot of tests that could potentially provide us with more information, but the data was too limited to show a significant effect. Another issue they point out is correlation; while the data they see is consistent with their hypothesis, it doesn’t demonstrate the causal link among these factors.

To an extent, the strongest thing the researchers demonstrate is how hard it is to get good data on this issue. If studies are done using surveys, then scientists will likely end up aware that they’re part of a test and may be more conscious of their implicit biases. If it’s done in a real-world context, as in this study, then the need to keep everything confidential can limit the data available.