Question writing and submission guidelines
Welcome to our question writing guide! Here, you’ll learn about our best practices for writing and submitting questions, as well as our content rules and guidelines.
We greatly value the contributions of our diverse community of forecasters, question authors, and forum participants, and we hope that these guidelines will promote, enhance, and safeguard a vibrant community forecasting space for many years to come.
- What types of questions are suitable for Metaculus?
- Our guidelines
- Guidelines for resolution conditions
- Best practices for question submissions
- Setting closing and resolution dates
- Moderation rules
- About the Community Moderators
Submitted questions are reviewed by a group of volunteer Community Moderators. The Community Moderators team tries its best to approve all the questions that conform to our guidelines and best-practices swiftly, typically within a week or two. Questions that are not immediately ready for publication are provided feedback by Community Moderators.
What types of questions are suitable for Metaculus?
Metaculus hosts questions on almost any topic—science, technology, politics, business, law, economics—you name it! That said, we have a thematic focus on scientific and technological advances, and an inclination towards issues of global importance (such as climate change, COVID-19, Artificial Intelligence, and so on.)
Currently, we will consider any question that satisfies our Moderation rules for publication on the platform.
Writing incisive questions for forecasting can be challenging, and requires a keen eye for detail, careful precision, creativity and imagination, and a host of other skills. Fortunately, many of these skills can be cultivated with a bit of practice. Here are some guidelines for writing good questions.
Guidelines for resolution conditions
Resolution conditions are the backbone of any forecasting question—they spell out how and when a question will resolve. It's therefore key to spell these out clearly.
- Aim for tight resolution conditions. The resolution conditions should leave little room for discretion in deciding the resolution. As best you can, try to limit the scope for ex-post quarrels about what really happened, and who was right.
- Define your terms. Questions of the sort “will X occur?” often hinge on how X is defined. It is therefore important to spell out your definitions with extra care. Don’t worry about being a little pedantic here!
- Be concrete. Try to specify precisely and in detail which steps should or shouldn’t be followed when resolving the question. Examples are helpful for making these instructions concrete.
- Use authoritative sources, when possible. Good options are numerical data regularly published by a reliable publicly available source. Note that you should be sure that the sources will be available at the time of resolution, or otherwise you might want to specify alternative sources of information.
- Consider and account for edge-cases. Try to imagine scenarios for which the resolution conditions fail to cleanly apply, or cases that are just on the edge of counting towards resolution. If such scenarios or edge-cases are plausible, you should clarify how the question should resolve when such events rear their head.
- Consider fall-back criteria. When you have a resolution that should be easy to check assuming all goes well, try to handle also the case where all doesn't go well. What if the data source you specified stops being published? Is there anything else odd that might happen to make the outcome unclear?
- Try to account for unknown unknowns. Think about how the resolution criteria behave when something you don’t expect happens anyway.
Best practices for question submissions
Metaculus has developed its own style and norms for writing and submitting questions. Heeding these best-practices will be appreciated by Metaculus forecasters and Community Moderators alike.
- Write in a neutral and objective tone. Questions in a neutral point of view represent topics fairly, proportionately, and, as far as possible, without editorial bias (think Wikipedia-style).
- Put effort into your question submissions. Don’t expect moderators to develop your question from scratch. If you have early ideas for questions but have yet to work these out, you can add a comment in the question-suggestions discussion post.
- Keep an eye out for feedback on your questions. After submitting your question, keep up with the feedback provided by moderators, including requests for revisions.
- Explain why your question is interesting or important. The significance or context of a question is not always common knowledge, so it is sometimes important to explain it. Quoting predictions from public figures or institutions is a great way to show the significance of a question.
- The statement of the question should match the resolution conditions. Make sure that the question statement does not give a mistaken impression of when and how the question might resolve.
- Delegate the boring stuff to the fine-print. In some cases, the resolution criteria might become quite detailed and involved. To keep questions streamlined, it can be convenient to relegate some of the gory details to the fine-print.
- Select ranges that (likely) contain the resolution value. For range questions, make sure the bounds cover most possible outcomes, not just all outcomes that appear most likely. Select open bounds if there is even a small chance that values could fall outside the range.
- Avoid excessively wide ranges. Select a range wide enough that the true value is very unlikely to fall outside, but no wider.
- Double check. Proof-read your submission, double-check your resolution conditions and ranges, and make sure the formulation of your question is consistent throughout.
- Acknowledge the contributions of others. When copying text from an existing question, acknowledge and reference the previous question.
- Use appropriate markdown formatting. When posting URLs, remember to embed these. When writing equations and formulas, try using MathJax.
- Select appropriate category tags. When submitting a question, make sure that you categorize your question appropriately by selecting relevant tags.
- Specify units and times. When the question asks about amounts, be sure to specify the units precisely (e.g. thousands of kilograms, nominal US dollars, temperature in degrees celsius). When specifying dates and times, remember to indicate the timezone or a UTC offset, and preferably write dates in the year-month-day format.
Setting closing and resolution dates
When submitted a question, you are asked to specify the closing date (when the question is no longer available for predicting) and resolution date (when the resolution is expected to occur).
Resolution dates should be the date when you expect there will be enough information for the question to resolve. The resolution date lets Metaculus administrators know when the question might be ready for resolution. If the date comes and not enough information is available yet, resolution can simply be postponed. As such, the resolution date is often just a suggestion.
In some cases, questions must resolve at the resolution date according to the best available information. In such cases, it becomes important to choose the resolution date carefully, and it must be explicit in the question text. Try to set it to dates that make for interesting and insightful questions!
For closing dates, here are some rules of thumb:
- If the outcome of the question will very likely or assuredly be determined at a fixed known time, then the closing time should be immediately before this time (example: a scheduled contest between known competitors)
- If the outcome of a question will be determined by some process that will occur at an unknown time, but the outcome is likely to be independent of this time, then it should be specified that the question retroactively closes some appropriate time before the process begins. (Example: success of a rocket
- launch occurring at an unknown time.)
- If the outcome of a question depends on a discrete event that may or may not happen, the close time should be specified of order 2/3 to 3/4 of the way between the open and resolution time, and if the event occurs before the close time, the question should be resolved just prior to the event with no retroactive closure. (Example: successful accomplishment of some technical challenge or publication of some type of paper.)
- If the outcome of a question depends upon a steady stream of events so that the outcome becomes steadily more clear with time, the question's close time may be chosen relatively freely; given the present scoring system a time of order 3/4 or more of the time between opening and resolution time is generally appropriate. (Example: cases of an COVID-19 confirmed by a given date.)
- If it is valuable to have updated predictions on a question up until the question resolves, it may be appropriate to push the close-date closer to the resolution date (say, somewhere between 80% to 90% of the way between the open and resolution time).
- Questions should not contain inappropriate or offensive material. Please refrain from posting content that is predictably disruptive.
- Questions should be on topics that are notable. Questions should reasonably be expected to be of interest to at least some other users. If your question idea is only interesting to you or a small group of friends, you can make a private question, and share it with them.
- Public questions should not concern the personal lives of non-public figures. Questions about non-public figures are generally only appropriate as private questions.
- Questions should not concern the mortality of individuals. Questions should never aim to predict mortality of individual people or even small groups. In cases of public interest (such as court appointees and political figures), the question should be phrased in other more directly relevant terms such as "when will X no longer serve on the court". When the topic is death (or longevity) itself questions should treat people in aggregate or hypothetically.
- Questions should typically not concern harmful acts. Questions should avoid being written in a way that incentivizes harmful or illicit acts—that is, if one were to imagine that the stakes of getting a question correct were high enough to motivate someone to interfere in real-world events to change a question's resolution, those actions should not be by their nature illegal or harmful. Exceptions may be made in cases where such predictions are clearly of wide public interest, which is to be decided by Community Moderators.
Moderation Rules (with the exception of the notability requirement, and the no non-public figures rule) apply equally to private questions as they do to public ones.
About the Community Moderators
Metaculus Community Moderators are committed members of the community who volunteer to help the platform run smoothly. We greatly appreciate their time and talents, and we hope you’ll do the same!
The main responsibilities of Community Moderators include:
- Providing constructive feedback on user-submitted questions, including asking for revisions when necessary.
- Accepting community-suggested questions that are ready to go live for forecasting.
- Mediating discussions about question resolutions.
- Answering users’ questions about how the platform works, when they arise.
Additionally, Community Moderators can help draw extra attention to bugs, feature requests, or other issues that affect the platform. Community Moderators are appointed democratically, by way of election. If you’d like to register interest in becoming a Community Moderator, please do so by commenting to this effect in the comment section of this discussion thread, and we’ll reach out to you before the next election takes place.