Research in physics, biology or other natural sciences uncovers regularities that help us to better understand the world around us. But what about the social sciences, which deal with individual or collective human behavior? Can cross-contextual regularities be identified here as well? Particularly in psychology, economics, management, and related fields, this remains controversial.
An international team of researchers, including Anna Dreber who works at the Department of Economics at the University of Innsbruck and at the Stockholm School of Economics, has now been able to show in a recently in PNAS published study that generalizability is possible – and predictable – in social science research as well. The most important indicator of whether a finding is generally true across contexts is whether a study can be reproduced in the same context.
Archival data examined for the first time
While research is already being conducted on the generalizability of social science laboratory experiments, this project, led by Andrew Delios (National University of Singapore), Elena Clemente (Stockholm School of Economics), Tao Wu (The Chinese University of Hong Kong), and Eric Uhlmann (INSEAD), has now investigated for the first time whether it is also possible to generalize study results whose statistical analyses are based on complex data sets. "We already know from experimental studies that laboratory effects either generally apply to many samples, regardless of different geographic regions or time periods, or they cannot be replicated across the board," says Anna Dreber. However, it was previously unknown whether this is also true for the complex archival datasets with many variables and observations which are widely used across scientific fields.
In the current study, the scientists systematically investigated whether findings from the field of strategic management also occur when other temporal or geographic sections of the data sets used are analyzed applying the same methodological approach. "We were able to show that original findings, that were statistically reliable in the first place, were generally found again in new analyses. This indicates surprisingly little context dependence. Findings that relate to a specific period of time or a specific region can thus also provide a meaningful indication of general observations in the social sciences," explains Andrew Delois. This is an important insight not only for numerous scientific fields, but also for practitioners who rely on such analysis results when making important strategic decisions.
Reproducibility as an indicator
The researchers repeated statistical analyses from a total of 29 studies and applied them to 52 new time periods or regions. All 29 studies were based on the same dataset on foreign direct investment by Japanese companies, which was compiled by a single author from various sources and then used by numerous researchers for different studies. Their results show that 45 percent of the analyses conducted by the researchers using the same methodology as in the original study also provided the same result. When data of new temporal or regional contexts were analyzed, the results also corresponded to the original study 55 percent of the time. However, the most important indicator of the generalizability of study results is seen in whether they can be directly reproduced. "If repeating the analysis of a study showed the same results as the original study using the same observations, then its findings were transferable to other time periods and regions, 84 percent and 57 percent of the time, respectively," Anna Dreber said.
The researchers also included a forecasting survey, where they asked independent scientists to anticipate which effects would find support in tests in new samples. These independent scientists were largely able to predict which result would generalize.
Further research needed
The researchers see their initiative as just the beginning of many more metascientific investigations to further advance understanding of the generalizability of social science research findings. "There is a lot more work to be done on this – looking at which archival analyses across fields are replicated in other contexts such as time periods, populations, and geographic areas, and whether scientists can predict these results. In recent years, the focus of replication studies has been primarily on behavioral laboratory experiments. However, studies of research with observational and archival data in management, sociology, economics, and other fields are also needed," Dreber emphasizes.
Good examples include the Systematizing Confidence in Open Research and Evidence project and the newly formed Institute for Replication, which focuses on economics and political science.
Examining the generalizability of research findings from archival data: Delios, A., Clemente, E., Wu, T. et al. Examining the generalizability of research findings from archival data. PNAS 119 (30) e2120377119 (2022). https://doi.org/10.1073/pnas.2120377119