# Questions Questions are saved in files in the [YAML](http://www.yaml.org/start.html) format. Each file contains a list of questions like ```yaml - type: radio ref: myquestion1 ... - type: checkbox ref: myquestion2 ... ``` where each question is specified in a dictionary. The `type` key is mandatory and specifies the type of question (multiple choice, text, etc). The other keys available will depend on the type of question. The field `ref` is not strictly required but still recommended, if not defined it will default to a string with the filename and the question index, e.g., `questions.yaml:12`. The following types of questions are supported: type | kind of answer -----|----------------- [radio](#radio) | Choose exactly one option from a list of options. [checkbox](#checkbox) | Choose zero, one or more options. [text](#text) | Line of text which is compared against a list of acceptable answers. [text-regex](#text-regex) | Similar to text, but the answer is validated by a regular expression. [numeric-interval](#numeric-interval) | Answer is interpreted as a floating point value (e.g. `1.2e-3`), and is checked against a closed interval. [textarea](#textarea) | The answer is a multiline block of text that is sent to an external program for assessment. The printed output of the external program is parsed (yaml) to obtain the result. [information, warning, alert, success](#information-warning-alert-and-success) | These are not really questions, just information panels intended to be used to convey information. There is no space for an answer and the correction returns always correct. [generator](#generator) | This is not a real question type. It means that this question will be generated by an external program, and the actual type is defined there. ## Question types ### radio Only one option can be selected as the answer. If no option is selected, the question is considered unanswered. The general format is ```yaml - type: radio ref: question_reference title: My first question text: | Please select one option. options: - this one is the correct one - wrong - not correct but also not completely wrong correct: [1, 0, 0.1] # default: first option shuffle: yes # default: yes choose: 2 # default: all options discount: yes # default: yes ``` All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true`. The `correct` field can be used in multiple ways and in combination with `shuffle`, `discount` and `choose` fields: - if not present, the first option is considered correct (options are shuffled by default when presented to the student). - it can be the index (0-based) of the correct option, e.g., `correct: 0` for the first option. - it can be a list of numbers between 0 and 1, e.g., `correct: [1, 0, 0]`. In this case, the first option is 100% correct while the others are 0%. If `discount: true` (the default), then the wrong ones will be penalized by $-1/(n-1)=-\tfrac{1}{2}$, where $n$ is the number of options. - there can be more than one correct option in the list, which is then marked in the correct field, e.g. `correct: [1, 1, 0]`. In this case, one of the correct options will be randomly selected, and the remaining wrong ones appended. - there can also be a long list of right and wrong options from which to build the question options. E.g. if `correct: [1,1,1,0,0,0,0]` and `choose: 3` is defined, then 1 correct option and 2 wrong ones are randomly selected from the list. - finally it's also possible to have a question that is *"not-completely-right"* or *"not-completely-wrong"*. This can be done using numbers between 0 and 1, e.g., `correct: [1, 0.3, 0]`. This practice is discouraged. In some situations one may not want the options to be shuffled. In that case use `shuffle: false`. ### checkbox Zero, one or multiple options can be selected. The question is always considered as answered, even if no options are selected. The simplest format is ```yaml - type: checkbox ref: question_reference title: My second question text: | Please mark the correct options. options: - this one is correct - wrong - this one is also correct correct: [1, -1, 1] # default: [1, 1, 1] shuffle: yes # default: yes choose: 2 # default: choose all options discount: yes # default: yes ``` All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true` and `choose` to the total number of options. When correcting an answer, each correctly marked/unmarked option gets the corresponding value from the list `correct: [1, -1, 1]` and each wrong gets its symmetrical. So in the previous example, to have a completely right answer the checboxes should be: marked, unmarked, marked. If `discount: no` then wrong options are given a value of 0. Options are shuffled by default. A smaller number of options may be randomly selected by setting the option `choose`. A more advanced format is to have two versions for each option, one right and one wrong. One of the versions is randomly selected when the question is generated. For example, ```yaml options: - ['this one is correct', 'this is wrong'] - 'wrong' - ['wrong again', 'this one is also correct'] correct: [1, -1, -1] ``` If the first version is selected then the corresponding `correct` value is used, otherwise if the second version is selected, then the symmetrical value is used instead. This format is useful to write questions that are presented in different ways to different students. It also minimizes solution memorization. Example: ```yaml options: - ['$\pi$ is a real number', '$\pi$ is an integer'] - ['there are more reals than integers', 'there are more integers than reals'] ``` ### text The answer is a single line of text. Just compare the answered text with the strings provided in a list of answers considered to be right. ```yaml - type: text ref: question-reference-3 title: My third question text: Seven days are called a... correct: ['week', 'Week'] # default: [] always wrong ``` ### text-regex Similar to text, but answers are validated by a regular expression. ```yaml - type: text-regex ref: question-reference-4 title: My fourth question text: Seven days are called a... correct: !regex '[wW]eek' # default: '$.^' always wrong ``` The regular expression is in a string and must be prefixed by the keyword `!regex`. ### numeric-interval Similar to text, but expects an integer or floating point number. The answer is converted to a float and is considered correct if the number is in the given closed interval. ```yaml - type: numeric-interval ref: question-reference-5 title: My fifth question text: What are the first 3 fractional digits of $\pi$? correct: [3.141, 3.142] # default: [1.0, -1.0] always wrong ``` ### textarea Provides a multiline textarea for the answer. The answered text is sent to the standard input of an external program for grading. The printed output to standard output of the program is parsed as YAML to get the grade and optional comments. ```yaml - type: textarea ref: question-reference-6 title: My sixth question text: Write a program in C that computes whatever. lines: 20 # default: 8 correct: path/to/program # default: '' always wrong timeout: 15 # default: 5 ``` The external program prints a dictionary with the grade and optional comments. For example, ```yaml grade: 0.8 comments: Almost there ``` It can also just print the grade as a single number. ### information, warning, alert and success These are not really questions, but just provides information for the student. ```yaml - type: information ref: question-reference-7 title: Calculator text: You can use your calculator. ``` Grading these type of "questions" yields always correct. ### generator This allows great flexibility, and allows each instance of a question to be always different. ```yaml type: generator ref: some-reference script: executable_program arg: 10,20 ``` A generator is an external program that generates a question dynamically. In the example above, the program to be run is `executable_program`. The `arg` is sent to the standard input of the `executable_program`. Questions should be printed to the stdout in YAML format, similarly to how they are defined above (but without the list dash). The printed question is then parsed to a dictionary which is then used to update the question. The `type` is redefined from generator to something else and the other fields are also updated. A generator can be any executable program (written in any language) that prints to the standard output. Example of a generator written in python: ```python #!/usr/bin/env python3 from random import randint import sys arg = sys.stdin.read() # read arguments from stdin as a string a,b = (int(n) for n in arg.split(',')) q = fr''' type: checkbox text: | Indique quais das seguintes adições resultam em overflow quando se considera a adição de números com sinal (complemento para 2) em registos de 8 bits. Os números foram gerados aleatoriamente no intervalo de {a} a {b}. options: ''' correct = [] for i in range(5): x = randint(a, b) y = randint(a, b) q += f'- "`{x} + {y}`"\n' correct.append(1 if x + y > 127 else -1) q += 'correct: ' + str(correct) print(q) ``` A generator cannot generate another generator, only real questions are acceptable. ## Writing text The text in the questions is interpreted as markdown with support for LaTeX formulas. The best way to write text is to use indentation like this: ```yaml text: | Yes. this is ok: If not indented, "Yes" would be a boolean and colon would be interpreted as a dictionary key. Images placed in the `public` subdirectory are accessible by ![image](image.png "sunset") LaTeX inline $\pi$ is ok, and $$ \frac{\sqrt{2\pi\sigma^2}}{2} $$ is also ok. Tables are simple: header1 | header2 --------|--------- value1 | value2 ``` Options of radio and checkbox questions are also interpreted as markdown.