Relative Risks of State-Led Mass Killing Onset in 2016: Results from a Comparison Survey

The Early Warning Project ran a wiki survey in December to get another take on countries' relative risks of state-led mass killing onsets.

Smaller countries are hard to see on that map, and it’s difficult to compare colors across countries that aren't adjacent. To show the results for all countries covered, the dot plot below lists all countries in rank order and shows their survey scores. (NOTE: The scores plotted here are not estimated probabilities of mass-killing onset. Instead, they are assessments of relative risk. In other words, these scores tell us which countries our crowd thinks we should worry about more, not how likely our crowd thinks an onset of state-led mass killing is.)

Countries with red dots are ones that probably had ongoing episodes of state-led mass killing at the end of 2015: Democratic Republic of the Congo, Egypt, Myanmar, Nigeria, North Korea, South Sudan, Sudan, and Syria. It is possible that these countries will experience additional onsets in 2016, but we wonder if some of our respondents didn’t confuse the risk of a new episode—in which the state targets a different group than the one(s) targeted in the episode(s) already underway—with the presence or intensity of an ongoing one. And, of course, there are at least a few ongoing episodes of mass killing being perpetrated by non-state actors (e.g., Boko Haram in Nigeria and the Islamic State in Iraq), but that’s not what we asked our crowd to consider in this survey.

We’re also interested in seeing which countries moved sharply up or down from year to year. The plot below compares the results of our wiki survey for 2015 with the 2016 scores. Red arrows identify countries that climbed at least 10 spots in the rankings; blue arrows do the same for countries that fell at least 10 spots. Most of the darkly colored arrows point to countries that score lower in the overall rankings, but there are some notable exceptions. Among the biggest gainers in risk, according to our wiki surveys, were Burundi (32nd to 4th), Ukraine (68th to 18th), Rwanda (47th to 27th), and Turkey (63rd to 28th). Among the biggest decliners were Guinea (13th to 36th) and Uzbekistan (29th to 58th).

These survey scores don't contribute directly to our statistical risk assessments, but they do inform the selection of cases we track in real time with our Expert Opinion Pool. On January 1, 2016, we launched opinion-pool questions about risks of new episodes of mass killing in roughly 20 countries worldwide. That set includes the top 15 from last year’s statistical risk assessments; any countries in the top 15 on the wiki survey that do not appear in the statistical top 15 (e.g., Burundi, Libya, and Ukraine); and a few other countries that are not in the top 15 on either list but are salient to some of our core audiences.

As with all of our forecasts, we plan to track and report on the accuracy of the assessments this wiki survey produces. Because the survey results are relative rankings, we can't score them the way we would score probabilities, but we can see if any countries that suffered onsets of state-led mass killing had ranked near the top of this list. At the moment, our researchers are still reviewing information from 2015 to determine where episodes of mass killing began, continued, or ended. After that review has been completed and we have posted about it, we will also take a preliminary look at the accuracy of our various risk-assessment methods—statistical modeling, wiki survey, and opinion pool—over the past year.

Methodology

We used All Our Ideas to conduct this pairwise wiki survey. For more on how a pairwise wiki survey works, see this working paper by the instrument’s creators. For details on what we mean by “state-led mass killing” and “onset,” see the blog post that led respondents to the survey.

We launched this survey with that blog post on December 1, 2015 and closed it on the morning of January 1, 2016. We publicized the survey through a few channels, including emails from the Center for the Prevention of Genocide and Ben Valentino to partners and colleagues who work on this topic; a posting to a Conflict Research group on Facebook; and occasional tweets throughout the month.

As for who are respondents were, we don’t and can’t know. Participation in All Our Ideas surveys is anonymous, and our blog post was shared publicly. From the survey's Results page, though, we know that the participants cast 6,952 votes—more than twice as many as last year’s 3,050—in 160 unique user sessions. Some individuals voted in more than one session, so the count of respondents was some unknown number smaller than 160. Several people affiliated with the Early Warning Project participated, but we actively discouraged them from casting a large number of votes so they wouldn’t have undue influence on the results.

As the chart below shows, voting got off to a slow start this time, but it picked up markedly after we emailed partners and colleagues in mid-December to encourage them to participate. Later peaks followed tweets reminding the Early Warning Project’s Twitter followers about the survey.

As the map below shows, and as in previous years, most of the votes in our wiki survey came from the United States and Europe. We were heartened to see some votes from Africa, the Middle East, and the South Pacific as well, but we hope to attract more participation from those regions and from Latin America and Asia in future surveys.