Call for participation and interest

Social Simulation has a variety of perfectly valid purposes other than prediction (see Epstein 2008; Edmonds 2017). Further, as an area that recognizes the inherent complexity of social systems, researchers in our community are inevitably and understandably cautious about the confidence with which they make any predictions. Prediction is nevertheless important in establishing agent-based modelling approaches as critical to reasoning about social systems, and creating expectations that future, credible social scientists will be acquainted with those approaches. And with agent-based modelling being increasingly used in policy-making and participatory contexts, we should encourage activities that gauge the limitations and requirements for making predictions using social simulations in complex social systems. If we are asking people in the real world to use evidence provided by our models when making decisions, decisions that may cost them money, we ourselves should be willing to ‘put our money* where our mouth is’ with our models.

*Tokens…

We propose to run a prediction workshop at the Social Simulation Conference in Stockholm in August 2018. This will take the form of a light-hearted competition in which some participants will present models that make predictions on a chosen topic, and others will use tokens to express their confidence in those models’ predictions. The rules will be explained below.

The chosen topic for 2018 is the Swedish General Election, which is to take place shortly after the conference on 9 September 2018. Whether justifiably or not (https://www.theguardian.com/politics/2018/mar/12/polls-as-accurate-as-they-have-ever-been-study-says, https://www.nature.com/articles/s41562-018-0315-6), opinion polls have had some significant blows to their reputation over the past few years, particularly in the UK and the USA (Loughran 2016). Opinion dynamics is also a well-established area of work in social simulation. Surely we, as agent-based modellers, with our more sophisticated explicit representations of societies, can do better at predicting election outcomes? If nothing else, the exercise serves as a useful opportunity for those of us coming to Sweden for the Social Simulation Conference to learn a bit more about the country we will be visiting in August.

The following resources might be useful to you:

• https://en.wikipedia.org/wiki/Swedish_general_election,_2018
• https://www.gdeltproject.org/ and https://blog.gdeltproject.org/gdelt-translingual-translating-the-planet/
• http://www.scb.se/en/ (Statistics Sweden)
• https://github.com/MansMeg/SwedishPolls

Timetable:

27 April 2018: Deadline for expression of intent to prepare a model. Please send an email to Gary Polhill with the subject “Prediction competition declaration of intent“.

4 May 2018: Confirmation that the competition will proceed. (At least two declarations of intent received.)

31 July 2018: Deadline for submission of predictions (see rules below)

20 August 2018: Date of workshop at SSC 2018 and announcement of category A winner(s).

9 September 2018: Swedish general election

ESSA Newsletter after the Speaker of the Riksdag announces the Prime Minister to the Monarch (or another general election is called): Announcement of category B winner(s).

Participation:

Participants in the workshop have their choice of one or more of the following roles:

• Modeller. As a modeller, you will present your model of the Swedish general election, and use it to state what you think the (likely) outcome(s) will be. You should express your prediction as one or more outcomes each with a probability in units of 1%. You will also provide a short paper describing your model. After the election result is announced, you will be asked to provide a paragraph explaining what you have learned about building and using your model for prediction, what additional information or data you would like to have had (if any), and what you would do differently.
• Punter. As a punter, you will be expected to listen to the modellers’ descriptions of models and associated predictions, and assign tokens to zero or more models in accordance with your confidence in those models’ predictions.
  • All punters will have 20 nominal tokens that they can assign to one or more models in accordance with their confidence in those models’ predictions.
  • Punters will be expected to make brief notes, to be shared with each modeller on why they have chosen to assign them the number of tokens they have – including those modellers whom they have assigned zero tokens.

The rules:

1. All models must be agent-based models.
2. All data used by modellers must be publicly available.
3. Modellers must be willing to release the source code of their models using an open source licence (GNU General Public Licence or similar).
4. All modellers must describe the following in their short paper submission to the workshop:

• In what way their model is agent-based
• The data they have used (if any)
• The structure of the model (Entities, State Variables and Scales in ODD, or UML class diagram)
• The schedule of the model (Process Overview and Scheduling in ODD, or UML activity diagram)
• Any theoretical principles underlying the construction of the model (Basic Principles Design Concept in ODD)
• Any steps taken to study the sensitivity of the model to its parameters, and results thereof
• Any steps taken to validate the model, and results thereof
• Any references to previously developed models for election prediction or related phenomena
• The prediction of the model, as a table with one or more data rows as shown below, where Outcome is defined using BNF notation (https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form):

   <outcome> ::= <government> <basis> | "other"
   <government> ::= <party> | <coalition>
   <coalition> ::= <party> "&" <coalition>
   <coalition> ::= <party>
   <party> ::= "Socialdemokraterna" | "Moderaterna" | "Sverigedemokraterna" | "Miljöpartiet" | "Centerpartiet" | "Vänsterpartiet" | "Liberalerna" | "Kristdemokraterna" | "Feministiskt Initiativ" | "Piratpartiet" | "Other Party"
   <basis> ::= "majority" | "minority"

  and Probability % is an integer in the range [1, 100]. The sum of the Probability % entries in each row must be <= 100.

• A summary statement stating why they (the modellers) think punters should have confidence in the recommendations

5. Modellers must predict the outcome of the general election as held on 9 September 2018. If no government is formed, and a subsequent general election is held, for example, the outcome of the latter is not the subject of the prediction made by the modellers. The “other” outcome option is provided for modellers to predict such an outcome from the 9 September 2018 election.
6. Punters will be given 20 tokens each, to assign to zero or more models in accordance with their confidence in those models.
7. Punters need not assign all 20 of their tokens.
8. Punters who are also modellers may not assign tokens to their own model.
9. Punters who do not use all of their tokens may not give their tokens to other punters.
10. The competition has the following categories of winning model:
    1. The model receiving the most tokens in the workshop. This represents the model that was most convincing to the participants. Where two or more models receive an equal highest number of tokens, they shall be declared joint winners in this category. This category of winner will be announced at the workshop.
    2. The model that made the best prediction. This is the model with the actual outcome from the election listed as one of its outcomes in its outcome table with the highest probability %. Where two or more models have the same probability % for the outcome that actually happened, they shall be declared joint winners in this category. This category of winner will be announced in the next newsletter of ESSA.
11. There is no prize (other than kudos) for winning in any category.

References

Edmonds, B. (2017) Different modelling purposes. In Simulating Social Complexity: A Handbook (2^nd Edition). Springer https://doi.org/10.1007/978-3-319-66948-9_4

Epstein, J. (2008) Why model? JASSS 11 (4), 2 http://jasss.soc.surrey.ac.uk/11/4/2.html

Loughran, T. (2016) Failed political predictions and the future of opinion polls. Manchester Policy Blogs. http://blog.policy.manchester.ac.uk/posts/2016/12/failed-political-predictions-and-the-future-of-opinion-polls/