Questions

Questions are saved in files in the YAML format. Each file contains a list of questions like

- type: radio
  ref: myquestion1
  ...

- type: checkbox
  ref: myquestion2
  ...

where each question is specified in a dictionary. The type key is mandatory and specifies the type of question (multiple choice, text, etc). The other keys available will depend on the type of question. The field ref is not strictly required but still recommended, if not defined it will default to a string with the filename and the question index, e.g., questions.yaml:12.

The following types of questions are supported:

Question types

radio

Only one option can be selected as the answer. If no option is selected, the question is considered unanswered.

The general format is

- type: radio
  ref: question_reference
  title: My first question
  text: |
    Please select one option.
  options:
    - this one is the correct one
    - wrong
    - not correct but also not completely wrong
  correct: [1, 0, 0.1]              # default: first option
  shuffle: yes                      # default: yes
  choose: 2                         # default: all options
  discount: yes                     # default: yes

All fields are optional except type and options. title and text default to empty strings, shuffle and discount to true.

The correct field can be used in multiple ways and in combination with shuffle, discount and choose fields:

• if not present, the first option is considered correct (options are shuffled by default when presented to the student).
• it can be the index (0-based) of the correct option, e.g., correct: 0 for the first option.
• it can be a list of numbers between 0 and 1, e.g., correct: [1, 0, 0]. In this case, the first option is 100% correct while the others are 0%. If discount: true (the default), then the wrong ones will be penalized by $-1/(n-1)=-\tfrac{1}{2}$, where $n$ is the number of options.
• there can be more than one correct option in the list, which is then marked in the correct field, e.g. correct: [1, 1, 0]. In this case, one of the correct options will be randomly selected, and the remaining wrong ones appended.
• there can also be a long list of right and wrong options from which to build the question options. E.g. if correct: [1,1,1,0,0,0,0] and choose: 3 is defined, then 1 correct option and 2 wrong ones are randomly selected from the list.
• finally it's also possible to have a question that is "not-completely-right" or "not-completely-wrong". This can be done using numbers between 0 and 1, e.g., correct: [1, 0.3, 0]. This practice is discouraged.

In some situations one may not want the options to be shuffled. In that case use shuffle: false.

checkbox

Zero, one or multiple options can be selected. The question is always considered as answered, even if no options are selected.

The simplest format is

- type: checkbox
  ref: question_reference
  title: My second question
  text: |
    Please mark the correct options.
  options:
    - this one is correct
    - wrong
    - this one is also correct
  correct: [1, -1, 1]            # default: [1, 1, 1]
  shuffle: yes                   # default: yes
  choose: 2                      # default: choose all options
  discount: yes                  # default: yes

All fields are optional except type and options. title and text default to empty strings, shuffle and discount to true and choose to the total number of options.

When correcting an answer, each correctly marked/unmarked option gets the corresponding value from the list correct: [1, -1, 1] and each wrong gets its symmetrical. So in the previous example, to have a completely right answer the checboxes should be: marked, unmarked, marked.

If discount: no then wrong options are given a value of 0. Options are shuffled by default. A smaller number of options may be randomly selected by setting the option choose.

A more advanced format is to have two versions for each option, one right and one wrong. One of the versions is randomly selected when the question is generated. For example,

  options:
    - ['this one is correct', 'this is wrong']
    - 'wrong'
    - ['wrong again', 'this one is also correct']
  correct: [1, -1, -1]

If the first version is selected then the corresponding correct value is used, otherwise if the second version is selected, then the symmetrical value is used instead.

This format is useful to write questions that are presented in different ways to different students. It also minimizes solution memorization. Example:

  options:
    - ['$\pi$ is a real number', '$\pi$ is an integer']
    - ['there are more reals than integers', 'there are more integers than reals']

text

The answer is a single line of text. Just compare the answered text with the strings provided in a list of answers considered to be right.

- type: text
  ref: question-reference-3
  title: My third question
  text: Seven days are called a...
  correct: ['week', 'Week']      # default: [] always wrong

text-regex

Similar to text, but answers are validated by a regular expression.

- type: text-regex
  ref: question-reference-4
  title: My fourth question
  text: Seven days are called a...
  correct: !regex '[wW]eek'      # default: '$.^' always wrong

The regular expression is in a string and must be prefixed by the keyword !regex.

numeric-interval

Similar to text, but expects an integer or floating point number. The answer is converted to a float and is considered correct if the number is in the given closed interval.

- type: numeric-interval
  ref: question-reference-5
  title: My fifth question
  text: What are the first 3 fractional digits of $\pi$?
  correct: [3.141, 3.142]      # default: [1.0, -1.0] always wrong

textarea

Provides a multiline textarea for the answer. The answered text is sent to the standard input of an external program for grading. The printed output to standard output of the program is parsed as YAML to get the grade and optional comments.

- type: textarea
  ref: question-reference-6
  title: My sixth question
  text: Write a program in C that computes whatever.
  lines: 20                     # default: 8
  correct: path/to/program      # default: '' always wrong
  timeout: 15                   # default: 5

The external program prints a dictionary with the grade and optional comments. For example,

grade: 0.8
comments: Almost there

It can also just print the grade as a single number.

information, warning, alert and success

These are not really questions, but just provides information for the student.

- type: information
  ref: question-reference-7
  title: Calculator
  text: You can use your calculator.

Grading these type of "questions" yields always correct.

generator

This allows great flexibility, and allows each instance of a question to be always different.

type: generator
ref: some-reference
script: executable_program
arg: 10,20

A generator is an external program that generates a question dynamically. In the example above, the program to be run is executable_program. The arg is sent to the standard input of the executable_program.

Questions should be printed to the stdout in YAML format, similarly to how they are defined above (but without the list dash). The printed question is then parsed to a dictionary which is then used to update the question. The type is redefined from generator to something else and the other fields are also updated.

A generator can be any executable program (written in any language) that prints to the standard output. Example of a generator written in python:

#!/usr/bin/env python3

from random import randint
import sys

arg = sys.stdin.read()  # read arguments from stdin as a string
a,b = (int(n) for n in arg.split(','))

q = fr'''
type: checkbox
text: |
  Indique quais das seguintes adições resultam em overflow quando se considera
  a adição de números com sinal (complemento para 2) em registos de 8 bits.

  Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
options:
'''

correct = []
for i in range(5):
    x = randint(a, b)
    y = randint(a, b)
    q += f'- "`{x} + {y}`"\n'
    correct.append(1 if x + y > 127 else -1)

q += 'correct: ' + str(correct)

print(q)

A generator cannot generate another generator, only real questions are acceptable.

Writing text

The text in the questions is interpreted as markdown with support for LaTeX formulas. The best way to write text is to use indentation like this:

text: |
    Yes. this is ok: If not indented, "Yes" would be a boolean
    and colon would be interpreted as a dictionary key.

    Images placed in the `public` subdirectory are accessible by
    ![image](image.png "sunset")

    LaTeX inline $\pi$ is ok, and
    $$
    \frac{\sqrt{2\pi\sigma^2}}{2}
    $$
    is also ok.

    Tables are simple:

    header1 | header2
    --------|---------
    value1  | value2

Options of radio and checkbox questions are also interpreted as markdown.