Crowdsourcing Program Tests Combined Opinions of Scientists and Technology Enthusiasts to Predict Scientific and Technological Events


Researchers are looking at crowdsourcing to gain a better view into future significant scientific and technological developments, particularly in areas such as information systems and energy where corporations may want to focus development efforts and investments. Crowdsourcing often provides more accurate forecasts by aggregating the collective opinions of an informed, diverse group rather than relying on a single expert.

But just how reliable and accurate are crowdsourced forecasts? The Forecasting Science and Technology (ForeST) program of the U.S. Intelligence Agency Research Projects Activity (IARPA) within the Office of the Director of National Intelligence (ODNI) aims to test the reliability and effectiveness of crowdsourcing techniques to predict advances in science and technology (S&T).

ForeST is a joint extension of two other IARPA programs, Aggregative Contingent Estimation (ACE) and Foresight and Understanding from Scientific Exposition (FUSE). ACE focuses on advanced techniques that elicit, weigh and combine the judgments of many people making forecasts about geo-political events. FUSE is aimed at recognizing very early stages of scientific or technical emergence through data-driven artificial intelligence techniques that analyze information in millions of scientific, technical and patent documents. The FUSE technologies can then automatically identify and recognize areas that are experiencing rapid growth or gaining attention from the scientific community.

SRI researchers are helping to apply FUSE technologies to the ForeST program to generate questions that will lead to forecasts of real-world events. The project is run through SciCast, a George Mason University website where members of the scientific community and technology enthusiasts can use their knowledge to answer questions in a number of areas, such as biology, medicine, energy and information systems.

SciCast is a combinatorial prediction market in which participants use a point system to place “bets” on answers to forecast questions. The more correct forecasts they make, the more influence they will have in other forecast questions. Participants can track their predictions on a scoreboard and see how they compare to their peers in accurately predicting the future of S&T. At the same time, researchers involved in the ForeST program are able to test the extent to which crowdsourcing methods are effective in generating accurate forecasts.

In addition, participants have the opportunity to pose their own questions and see how the crowd responds. Along with writing questions, participants must submit relevant background and supporting material for the crowd to refer to when providing their answers. And for people who are interested in contributing questions, there is an opportunity to work with the SRI research team on the question-generation process by suggesting specific sources, journals and terminology.

The ForeST program is an experiment that in principle represents scientific progress in our understanding of people’s interaction and prediction making. We are looking to members of the scientific community to participate, which will also increase participants’ own scientific knowledge.

Interested scientists and technology enthusiasts can register to become SciCast forecasters. It’s an exciting opportunity to gain a view into the future of S&T, and we’re looking forward to working with the community to combine their collective information and knowledge in a reliable way to more accurately predict advances.

This research was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior National Business Center (DoI/NBC). The views and conclusions expressed herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. government.

Read more from SRI