In this survey, you will rate hypotheses on their quality. Your participation will help us understand how people can generate high quality hypotheses.
We present you hypotheses generated by the crowd on a certain data science problem. We ask you to rate hypotheses on their quality, and the quality is defined based on certain criteria. We have a total of 164 hypotheses that need rating, and each hypothesis has the same rating criteria and questions.
To rate one hypothesis, it takes less than 1 minute. You get faster, the more you rate.
Progress:
We currently have 570 out of 656 required ratings.
At the current submission rate, each person would need to rate at least 25-35 minutes. Please rate as many as you
can!
You can always return to this survey if you have time and rate more.
We are highly dependent on your input and want to thank you in advance for your much appreciated work.
Table of Contents
Introduction
Quality Rating Criterias
Examples for Criterias
START RATING! Button
Leaderboard
Quality Rating Criterias
- Does this hypothesis make sense? (yes/no)
- Does this hypothesis include empirical observations? (yes/no)
- Does the hypothesis include at least 2 variables? (yes/no)
- Does it make use of features related to the problem scenario? (yes/no)
- How precisely is this hypothesis written? (rating from 1 to 5)
- How easy is it to interpret? (rating from 1 to 5)
- How likely is it to be true? (rating from 1 to 5)
Examples
(Volcanic activity include volcanos and their erruptions, and everything around its consequences)
Sense
Question: Does this Hypothesis make sense?
| Example Hypothesis | Explanation |
| Hypothesis: "T-Shirts are more likely to speak when held under water" | Does this Hypothesis make sense? NO! Because T-Shirts cannot speak. This hypothesis does not make sense. |
Empirical
Question: Does this Hypothesis include empirical observations?
| Example Hypothesis | Explanation |
| Hypothesis: "Magic did cause the volcano to errupt." | Does this Hypothesis include empirical observations? NO! An empirical observation would include why the volcano errupted, and not just magic. |
Question: Does the hypothesis include at least 2 variables?
| Example Hypothesis | Explanation |
| Hypothesis: "Volcano number 45 has the hottest water" | Does the hypothesis include at least 2 variables? NO! There are not 2 variables that are in a relationship to each other in this hypothesis |
Question: Does it make use of features related to the problem scenario?
(Imagine a dataset about volcanic activity on Hawaii)
| Example Hypothesis | Explanation |
Hypothesis: "Paddleboats on the lake of Zurich are faster because of the electric motor" |
Does it make use of features related to the problem scenario? NO! The problem scenario is about volcanic activity! |
Question: How precisely is this hypothesis written?
| Example Hypothesis | Explanation |
| Hypothesis A : "The earth got denser because of an earthquake caused by a volcano" Hypothesis B : "The earth got 3 kilogram per cubic meter denser because of earthquake of magnitude 3, which was caused by the erruption of a volcano" |
How precisely is this hypothesis written? Hypothesis A would score lower than Hypothesis B on a scale from 1 to 5 |
Question: How easy is it to interpret?
| Example Hypothesis | Explanation |
| Hypothesis A : "The thing from the top left turned into ten units of something because of the other
thing from down there" Hypothesis B : "Volcano number 45 turned into ten small island because of the erruption from 1923" |
How easy is it to interpret? Hypothesis A would score lower than Hypothesis B on a scale from 1 to 5 |
Question: How likely is it to be true?
| Example Hypothesis | Explanation |
| Hypothesis A : "Volcanic erruption causes the earth to fly into the sun" Hypothesis B : "Volcanic erruption causes plane flights to be cancelled" |
How likely is it to be true? Hypothesis A would score lower than Hypothesis B on a scale from 1 to 5 |
If you have any question or need more information, feel free to contact us via chat, in person or via email: hypotheses_experiments@ifi.uzh.ch
Cookie Consent: A cookie will be stored in your browser in order to not show you any hypotheses twice and to save your gamification progress. If you continue, you agree with this.