Guidelines
(Click to expand)Overview
Welcome to our task on hypotheses ranking. In this task, we will give you a set of predefined hypotheses for a data set, and we will ask you to rank them based on how interesting you think they are.
As stated by De Jong et al., a hypothesis is "a statement about the relations between variables in a theoretical model" (De Jong, 1998).
For example, when exploring a data set in order to identify relevant features for predicting credit risk, it could be useful to test hypotheses such as the following:
- Customers with poor credit history in the past are more likely to default in future.
- Customers with high (loan_value / income) are likely to default more than those with a low ratio.
- Customers doing impulsive shopping are more likely to be at a higher credit risk.
If needed, you can access the basic definiton and examples for hypothesis, variables and relation between variables by clicking the 'Instructions' tab at the top left corner of this page.
The answers collected in these HITS will be used for research purposes, within the CrowdAlytics project, as a means to evaluate a hypotheses ranking approach. Thank you very much for taking the time to participate in our task.
Kindly, researchers of the University of Zurich.
If you have any questions or concerns about the task, please do not hesitate to contact us via hypotheses_experiments@ifi.uzh.ch.
INSTRUCTIONS
We have a large set of hypotheses that one can test on a data set. However, we would like to know if these hypotheses are equally interesting or not. Which ones would you like to be tested first by a data scientist? Which ones are the most interesting ones for you?
Steps:
- Read the given analysis goal carefully.
- You will see two hypotheses (left and right), select the one you find as most interesting.
- Evaluate a few statements based on your selection.
- If asked, please elaborate in a few sentences the reason(s) of your selection.
- We ask you to rank a set of five pairs of hypotheses. To assess the next pair, please click on the "Next" button below the third question.
- Click 'submit' only after finishing the entire task.
Task: Given the data science problem stated below, your task is to choose the hypothesis you would like to be tested by a data scientist. Use the 'Next' button to assess the next set of hypotheses.
{{ Problem_Statement }}
Hypothesis 1
{{ B1_hypothesis1 }}
Hypothesis 2
{{ B1_hypothesis2 }}
1. Which hypothesis is most interesting to you?
2. Why do you find that hypothesis more interesting than the other?
2.1 Evaluate the following statements on a 5-point scale:
| Statement | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
|---|---|---|---|---|---|
| 1. The selected hypothesis has a better-defined relation(s) between one or more variables compared to the other hypotheses/hypotheses. | |||||
| 2. The selected hypothesis is more precise/conceptually clear than the other hypothesis/hypotheses. | |||||
| 3. The selected hypothesis is more aligned with the given data science problem than the other hypothesis/hypotheses. | |||||
| 4. The variables in the selected hypothesis are more interesting than that of other hypotheses. | |||||
| 5. The stated relation between the variables in the selected hypotheses is more interesting than the ones in the other hypotheses. | |||||
| 6. The selected hypothesis is expected to be FALSE. | |||||
| 7. The selected hypothesis is expected to be TRUE. | |||||
| 8. I do have more prior knowledge about the selected hypothesis compared to the other hypothesis/hypotheses. | |||||
| 9. The selected hypothesis relates better to my personal experience compared to the other hypothesis/hypotheses. |
3. How confident are you about your selection?
Hypothesis 1
{{ B2_hypothesis1 }}
Hypothesis 2
{{ B2_hypothesis2 }}
1. Which hypothesis is most interesting to you?
2. Why do you find that hypothesis more interesting than the other?
2.1 Evaluate the following statements on a 5-point scale:
| Statement | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
|---|---|---|---|---|---|
| 1. The selected hypothesis has a better better-defined relation(s) between one or more variables compared to the other hypotheses/hypotheses. | |||||
| 2. The selected hypothesis is more precise/conceptually clear than the other hypothesis/hypotheses. | |||||
| 3. The selected hypothesis is more aligned with the given data science problem than the other hypothesis/hypotheses. | |||||
| 4. The variables in the selected hypothesis are more interesting than that of other hypotheses. | |||||
| 5. The stated relation between the variables in the selected hypotheses is more interesting than the ones in the other hypotheses. | |||||
| 6. The selected hypothesis is expected to be FALSE. | |||||
| 7. The selected hypothesis is expected to be TRUE. | |||||
| 8. I do have more prior knowledge about the selected hypothesis compared to the other hypothesis/hypotheses. | |||||
| 9. The selected hypothesis relates better to my personal experience compared to the other hypothesis/hypotheses. |
3. How confident are you about your selection?
Hypothesis 1
{{ B3_hypothesis1 }}
Hypothesis 2
{{ B3_hypothesis2 }}
1. Which hypothesis is most interesting to you?
2. Why do you find that hypothesis more interesting than the other?
2.1 Evaluate the following statements on a 5-point scale:
| Statement | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
|---|---|---|---|---|---|
| 1. The selected hypothesis has a better-defined relation(s) between one or more variables compared to the other hypotheses/hypotheses. | |||||
| 2. The selected hypothesis is more precise/conceptually clear than the other hypothesis/hypotheses. | |||||
| 3. The selected hypothesis is more aligned with the given data science problem than the other hypothesis/hypotheses. | |||||
| 4. The variables in the selected hypothesis are more interesting than that of other hypotheses. | |||||
| 5. The stated relation between the variables in the selected hypotheses is more interesting than the ones in the other hypotheses. | |||||
| 6. The selected hypothesis is expected to be FALSE. | |||||
| 7. The selected hypothesis is expected to be TRUE. | |||||
| 8. I do have more prior knowledge about the selected hypothesis compared to the other hypothesis/hypotheses. | |||||
| 9. The selected hypothesis relates better to my personal experience compared to the other hypothesis/hypotheses. |
3. How confident are you about your selection?
Hypothesis 1
{{ B4_hypothesis1 }}
Hypothesis 2
{{ B4_hypothesis2 }}
1. Which hypothesis is most interesting to you?
2. Why do you find that hypothesis more interesting than the other?
2.1 Evaluate the following statements on a 5-point scale:
| Statement | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
|---|---|---|---|---|---|
| 1. The selected hypothesis has a better-defined relation(s) between one or more variables compared to the other hypotheses/hypotheses. | |||||
| 2. The selected hypothesis is more precise/conceptually clear than the other hypothesis/hypotheses. | |||||
| 3. The selected hypothesis is more aligned with the given data science problem than the other hypothesis/hypotheses. | |||||
| 4. The variables in the selected hypothesis are more interesting than that of other hypotheses. | |||||
| 5. The stated relation between the variables in the selected hypotheses is more interesting than the ones in the other hypotheses. | |||||
| 6. The selected hypothesis is expected to be FALSE. | |||||
| 7. The selected hypothesis is expected to be TRUE. | |||||
| 8. I do have more prior knowledge about the selected hypothesis compared to the other hypothesis/hypotheses. | |||||
| 9. The selected hypothesis relates better to my personal experience compared to the other hypothesis/hypotheses. |
3. How confident are you about your selection?
Hypothesis 1
{{ B5_hypothesis1 }}
Hypothesis 2
{{ B5_hypothesis2 }}
1. Which hypothesis is most interesting to you?
2. Why do you find that hypothesis more interesting than the other?
2.1 Evaluate the following statements on a 5-point scale:
| Statement | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
|---|---|---|---|---|---|
| 1. The selected hypothesis has a better-defined relation(s) between one or more variables compared to the other hypotheses/hypotheses. | |||||
| 2. The selected hypothesis is more precise/conceptually clear than the other hypothesis/hypotheses. | |||||
| 3. The selected hypothesis is more aligned with the given data science problem than the other hypothesis/hypotheses. | |||||
| 4. The variables in the selected hypothesis are more interesting than that of other hypotheses. | |||||
| 5. The stated relation between the variables in the selected hypotheses is more interesting than the ones in the other hypotheses. | |||||
| 6. The selected hypothesis is expected to be FALSE. | |||||
| 7. The selected hypothesis is expected to be TRUE. | |||||
| 8. I do have more prior knowledge about the selected hypothesis compared to the other hypothesis/hypotheses. | |||||
| 9. The selected hypothesis relates better to my personal experience compared to the other hypothesis/hypotheses. |
2.2 Please explain the reasons for your selection (in a sentence or two).
3. How confident are you about your selection?
4. If you have any feedback for this task, please let us know.
If you want to look into definition and examples of hypothesis, variables and relations; click on the below link 'More Instructions'.
