Questions

Questions are saved in files in the YAML format. Each file contains a list of questions like

- type: radio
  ref: myquestion1
  ...

- type: checkbox
  ref: myquestion2
  ...

where each question is specified in a dictionary. The type key is mandatory and specifies the type of question (multiple choice, text, etc). The other keys available will depend on the type of question. The field ref is not strictly required but still recommended, if not defined it will default to a string with the filename and the question index, e.g., questions.yaml:12.

The following types of questions are supported:

Question types

radio

Only one option can be selected as the answer. If no option is selected, the question is considered unanswered.

The general format is

- type: radio
  ref: question_reference 
  title: My first question
  text: |                 
    Please select one option.
  options:
    - this one is the correct one
    - wrong
    - not correct but also not completely wrong
  correct: [1, 0, 0.1]              # default: first option
  shuffle: yes                      # default: yes
  choose: 2                         # default: all options
  discount: yes                     # default: yes

All fields are optional except type and options. title and text default to empty strings, shuffle and discount to true.

The correct field can be used in multiple ways and in combination with shuffle, discount and choose fields:

• if not present, the first option is considered correct (options are shuffled by default when presented to the student).
• it can be the index (0-based) of the correct option, e.g., correct: 0 for the first option.
• it can be a list of numbers between 0 and 1, e.g., correct: [1, 0, 0]. In this case, the first option is 100% correct while the others are 0%. If discount: true (the default), then the wrong ones will be penalized by $-1/(n-1)=-\tfrac{1}{2}$, where $n$ is the number of options.
• there can be more than one correct option in the list, which is then marked in the correct field, e.g. correct: [1, 1, 0]. In this case, one of the correct options will be randomly selected, and the remaining wrong ones appended.
• there can also be a long list of right and wrong options from which to build the question options. E.g. if correct: [1,1,1,0,0,0,0] and choose: 3 is defined, then 1 correct option and 2 wrong ones are randomly selected from the list.
• finally it's also possible to have a question that is "not-completely-right" or "not-completely-wrong". This can be done using numbers between 0 and 1, e.g., correct: [1, 0.3, 0]. This practice is discouraged.

In some situations one may not want the options to be shuffled. In that case use shuffle: false.

checkbox

Zero, one or multiple options can be selected. The question is always considered as answered, even if no options are selected.

The simplest format is

- type: checkbox
  ref: question_reference 
  title: My second question
  text: |                 
    Please mark the correct options.
  options:
    - this one is correct
    - wrong
    - this one is also correct
  correct: [1, -1, 1]            # default: [0, 0, 0]
  shuffle: yes                   # default: yes
  choose: 2                      # default: choose all options
  discount: yes                  # default: yes

All fields are optional except type and options. title and text default to empty strings, shuffle and discount to true and choose to the total number of options.

When correcting an answer, each correctly marked/unmarked option gets the corresponding value from the list correct: [1, -1, 1] and each wrong gets its symmetrical. So in the previous example, to have a completely right answer the checboxes should be: marked, unmarked, marked.

If discount: no then wrong options are given a value of 0. Options are shuffled by default. A smaller number of options may be randomly selected by setting the option choose.

A more advanced format is to have

  options:
    - ['this one is correct', 'this is wrong']
    - 'wrong'
    - ['wrong again', 'this one is also correct']
  correct: [1, -1, -1]

In this case, there are options that contain a list of 2 suboptions. When the question is generated, one of the suboptions is randomly selected. If it's the first then the corresponding correct value is used, if it's the second then it's symmetrical is used instead.

This format is useful to write options with 2 different versions to avoid cheating. Example:

  options:
    - ['$\pi$ is a real number', '$\pi$ is an integer']
    - ['there are more reals than integers', 'there are more integers than reals']

text

Answer is a line of text. Just compare the answered text with string provided in a list of acceptable answers.

- type: text
  ref: question-reference-3
  title: My third question
  text: Seven days are called a...
  correct: ['week', 'Week']      # default: [] always wrong

text-regex

Similar to text, but validade using a regular expression.

- type: text-regex
  ref: question-reference-4
  title: My fourth question
  text: Seven days are called a...
  correct: !regex '[wW]eek'      # default: '$.^' always wrong

numeric-interval

Similar to text, but expects an integer or floating point number. The answer is correct if the number is in the given interval.

- type: numeric-interval
  ref: question-reference-5
  title: My fifth question
  text: What are the first 3 fractional digits of $\pi$?
  correct: [3.141, 3.142]      # default: [1.0, -1.0] always wrong

textarea

Provides a multiline textarea for the answer. The answered text is sent to the stdin of an external program for assessment. The printed stdout output of the program is parsed as YAML to get the grade and optional comments.

- type: textarea
  ref: question-reference-6
  title: My sixth question
  text: Write a program in C that computes whatever.
  lines: 20                     # default: 8
  correct: path/to/program      # default: '' always wrong
  timeout: 15                   # default: 5

Example program output:

grade: 0.8
comments: Almost there

information, warning, alert and success

- type: information
  ref: question-reference-7
  title: Calculator
  text: You can use your calculator.

generator

Generators are external programs that generate a question dynamically. Questions should be printed to the stdout in YAML format (without the list dash). The parsed output is used to update the question dict, redefining type and other fields.

Example of a generator written in python (any language can do):

#!/usr/bin/env python3

from random import randint
import sys

arg = sys.stdin.read()  # read arguments
a,b = (int(n) for n in arg.split(','))

q = fr'''
type: checkbox
text: |
  Indique quais das seguintes adições resultam em overflow quando se considera a adição de números com sinal (complemento para 2) em registos de 8 bits.

  Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
options:
'''

correct = []
for i in range(5):
    x = randint(a, b)
    y = randint(a, b)
    q += f'- "`{x} + {y}`"\n'
    correct.append(1 if x + y > 127 else -1)

q += 'correct: ' + str(correct)

print(q)

Writing text

The text in the questions is interpreted as markdown with LaTeX formulas. The best way to avoid gotchas is to indent text like this:

  text: |
    Yes. this is ok: If not indented, "Yes" would be a boolean 
    and colon would be interpreted as a dictionary key.

    Images placed in the `public` subdirectory are accessible by
    ![image](image.png "sunset")

    LaTeX inline $\pi$ is ok, and
    $$
    \frac{\sqrt{2\pi\sigma^2}}{2}
    $$
    is also ok.

    Tables are simple:

    header1 | header2
    --------|---------
    value1  | value2