- added documentation in QUESTIONS.md and fixed in README.md.

- changed the certificate names and permissions.

- added documentation in QUESTIONS.md and fixed in README.md.
- changed the certificate names and permissions.
Miguel Barão
1 parent c55940af
Showing 4 changed files with 373 additions and 85 deletions Show diff stats
QUESTIONS.md
README.md
questions.py
serve.py
@@ -0,0 +1,244 @@
+# Questions
+
+Questions are saved in files in the [YAML](http://www.yaml.org/start.html) format. Each file has a list of questions with the following structure:
+
+```yaml
+- type: radio
+  ref: question1
+  ...
+
+- type: checkbox
+  ref: question2
+  ...
+```
+
+The following kinds kinds of questions are supported:
+
+type | kind of answer
+-----|-----------------
+[radio](#radio) | Choose exactly one option from list of options.
+[checkbox](#checkbox) | Choose zero, one or more options.
+[text](#text) | Line of text which is matched against a list of acceptable answers.
+[text-regex](#text-regex) | Similar to text, but the answer is validated by a regular expression.
+[numeric-interval](#numeric-interval) | Answer is interpreted as a floating point value (e.g. `1.2e-3`), which is checked against a closed interval.
+[textarea](#textarea) | The answer is a multiline block of text that is sent to an external program for assessment. The printed output of the external program is parsed to obtain the result.
+[information, warning, alert and success](#information-warning-alert-and-success) | These are not really questions, just information panels intended to be used in tests to convey information. There is no answer and it's always correct.
+[generator](#generator) | This is not a really a question type. It means that this question will be generated by an external program, and the actual type is defined there.
+
+In all questions, the field `type` is required. The field `ref` is not strictly required but still recommended, if not defined will default to FIXME.
+
+## radio
+
+Only one option can be selected as the answer. If no option is selected, the question is considered unanswered.
+
+The general format is
+
+```yaml
+- type: radio
+  ref: question_reference 
+  title: My first question
+  text: |                 
+    Please select one option.
+  options:
+    - this one is the correct one
+    - wrong
+    - not correct but also not completely wrong
+  correct: [1, 0, 0.1]              # default: first option
+  shuffle: yes                      # default: yes
+  choose: 2                         # default: all options
+  discount: yes                     # default: yes
+```
+
+All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true`.
+
+The `correct` field can be used in multiple ways and in combination with `shuffle`, `discount` and `choose` fields:
+
+- if not present, the first option is considered correct (options are usually shuffled...).
+- it can be the index (0-based) of the correct option, e.g., `correct: 0`.
+- it can be a list of numbers between 0 and 1, e.g., `correct: [1, 0, 0]`. In this case, the first option is 100% correct while the others are 0%. If `discount: true` (the default), then the wrong ones will be penalized by $-1/(n-1)=-\tfrac{1}{2}$, where $n$ is the number of options.
+- there can be more than one correct option in the list, which is then marked in the correct field, e.g. `correct: [1, 1, 0]`. In this case, one of the correct options will be randomly selected, and the remaining wrong ones appended.
+- there can also be a long list of right and wrong options from which to build the question options. E.g. if `correct: [1,1,1,0,0,0,0]` and `choose: 3` is defined, then 1 correct option and 2 wrong ones are randomly selected from the list.
+- finally it's also possible to have a question that is *"not-completely-right"* or *"not-completely-wrong"*. This can be done using numbers between 0 and 1, e.g., `correct: [1, 0.3, 0]`. This practice is discouraged.
+
+In some situations one may not want the options to be shuffled, e.g., if they show several steps of a proof and the student should mark the wrong step. In that case use `shuffle: false`.
+
+## checkbox
+
+Zero, one or multiple options can be selected. The question is always considered as answered, even if no options are selected, which is also a valid answed.
+
+The simplest format is 
+
+```yaml
+- type: checkbox
+  ref: question_reference 
+  title: My second question
+  text: |                 
+    Please mark the correct options.
+  options:
+    - this one is correct
+    - wrong
+    - this one is also correct
+  correct: [1, -1, 1]            # default: [0, 0, 0]
+  shuffle: yes                   # default: yes
+  choose: 2                      # default: choose all options
+  discount: yes                  # default: yes
+```
+
+All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true` and `choose` to the total number of options.
+
+When correcting an answer, each correctly marked/unmarked option gets the corresponding value from the list `correct: [1, -1, 1]` and each wrong gets its symmetrical. So in the previous example, to have a completely right answer the checboxes should be: marked, unmarked, marked.
+
+If `discount: no` then wrong options are given a value of 0.
+Options are shuffled by default. A smaller number of options may be randomly selected by setting the option `choose`.
+
+A more advanced format is to have
+
+```yaml
+  options:
+    - ['this one is correct', 'this is wrong']
+    - 'wrong'
+    - ['wrong again', 'this one is also correct']
+  correct: [1, -1, -1]
+```
+
+In this case, there are options that contain a list of 2 suboptions.
+When the question is generated, one of the suboptions is randomly  selected. If it's the first then the corresponding `correct` value is used, if it's the second then it's symmetrical is used instead.
+
+This format is useful to write options with 2 different versions to avoid cheating. Example: 
+
+```yaml
+  options:
+    - ['$\pi$ is a real number', '$\pi$ is an integer']
+    - ['there are more reals than integers', 'there are more integers than reals']
+
+```
+
+## text
+
+Answer is a line of text. Just compare the answered text with string provided in a list of acceptable answers.
+
+```yaml
+- type: text
+  ref: question-reference-3
+  title: My third question
+  text: Seven days are called a...
+  correct: ['week', 'Week']      # default: [] always wrong
+```
+
+## text-regex
+
+Similar to text, but validade using a regular expression.
+
+```yaml
+- type: text-regex
+  ref: question-reference-4
+  title: My fourth question
+  text: Seven days are called a...
+  correct: !regex '[wW]eek'      # default: '$.^' always wrong
+```
+
+
+## numeric-interval
+
+Similar to text, but expects an integer or floating point number. The answer is correct if the number is in the given interval.
+
+```yaml
+- type: numeric-interval
+  ref: question-reference-5
+  title: My fifth question
+  text: What are the first 3 fractional digits of $\pi$?
+  correct: [3.141, 3.142]      # default: [1.0, -1.0] always wrong
+```
+
+## textarea
+
+Provides a multiline textarea for the answer. The answered text is sent to the stdin of an external program for assessment. The printed stdout output of the program is parsed as YAML to get the grade and optional comments.
+
+```yaml
+- type: textarea
+  ref: question-reference-6
+  title: My sixth question
+  text: Write a program in C that computes whatever.
+  lines: 20                     # default: 8
+  correct: path/to/program      # default: '' always wrong
+  timeout: 15                   # default: 5
+```
+
+Example program output:
+
+```yaml
+grade: 0.8
+comments: Almost there
+```
+
+## information, warning, alert and success
+
+```yaml
+- type: information
+  ref: question-reference-7
+  title: Calculator
+  text: You can use your calculator.
+```
+
+## generator
+
+Generators are external programs that generate a question dynamically.
+Questions should be printed to the stdout in YAML format (without the list dash). The parsed output is used to update the question dict, redefining `type` and other fields.
+
+Example of a generator written in python (any language can do):
+
+```python
+#!/usr/bin/env python3
+
+from random import randint
+import sys
+
+arg = sys.stdin.read()  # read arguments
+a,b = (int(n) for n in arg.split(','))
+
+q = fr'''
+type: checkbox
+text: |
+  Indique quais das seguintes adições resultam em overflow quando se considera a adição de números com sinal (complemento para 2) em registos de 8 bits.
+
+  Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
+options:
+'''
+
+correct = []
+for i in range(5):
+    x = randint(a, b)
+    y = randint(a, b)
+    q += f'- "`{x} + {y}`"\n'
+    correct.append(1 if x + y > 127 else -1)
+
+q += 'correct: ' + str(correct)
+
+print(q)
+```
+
+# Writing text
+
+The text in the questions is interpreted as markdown with LaTeX formulas.
+The best way to avoid gotchas is to indent text like this:
+
+```yaml
+  text: |
+    Yes. this is ok: If not indented, "Yes" would be a boolean 
+    and colon would be interpreted as a dictionary key.
+
+    Images placed in the `public` subdirectory are accessible by
+    ![image](image.png "sunset")
+
+    LaTeX inline $\pi$ is ok, and
+    $$
+    \frac{\sqrt{2\pi\sigma^2}}{2}
+    $$
+    is also ok.
+
+    Tables are simple:
+
+    header1 | header2
+    --------|---------
+    value1  | value2
+```
 \ No newline at end of file
-# Get Started
+# Getting Started
  
  
 ## Requirements
  
-We will need to install python3.6 with sqlite3 support.
+Before installing the server, we will need to install python with some additional packages.
+
+### Install python3.6 with sqlite3 support
+
 This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.
  
 - Installing from the system package manager:
-    - OSX: `port install python36`
-    - FreeBSD: `pkg install python36 py36-sqlite3`
-    - Linux: `apt-get install ???` (not available yet?)
+    ```sh
+    sudo port install python36                # MacOS
+    sudo pkg install python36 py36-sqlite3    # FreeBSD
+    sudo apt install ?not available yet?      # Linux, install from source
+    ```
 - Installing from source:
-    - Download from [http://www.python.org]()
-    - `unxz Python-3.6.tar.xz`
-    - `tar xvf Python-3.6.tar`
-    - `cd Python-3.6`
-    - `./configure --prefix=$HOME/.local/bin`
-    - `make && make install`
+     Download from [http://www.python.org]() and
+
+    ```sh
+    unxz Python-3.6.tar.xz
+    tar xvf Python-3.6.tar
+    cd Python-3.6
+    ./configure --prefix=$HOME/.local/bin
+    make && make install
+    ```
+
+    This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).
  
-This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in OSX or FreeBSD).
+### Install pip
  
-Next install pip (if not yet installed):
+If the `pip` command is not yet installed, run
  
-    python3.6 -m ensurepip --user
+```sh
+python3.6 -m ensurepip --user
+```
  
-This will install pip in your account under `~/.local/bin`.
+This will install pip in your user account under `~/.local/bin`.
 In the end you should be able to run `pip3 --version` and `python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`, `pip3.6` or `pip-3.6`).
  
-Install additional python packages locally on the user area:
+If you want to always install python modules on the user account (recommended), edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or `Library/Application Support/pip/pip.conf` (MacOS) and add the lines
  
-    pip3 install --user \
-        tornado \
-        sqlalchemy \
-        pyyaml \
-        pygments \
-        markdown \
-        bcrypt \
-        networkx
+```ini
+[global]
+user = yes
+```
  
-These are usually installed under
+### Install additional python packages:
  
-- OSX: `~/Library/python/3.6/lib/python/site-packages/`
-- Linux/FreeBSD: `~/.local/lib/python3.6/site-packages/`
+```sh
+pip3 install --user \
+    tornado \
+    sqlalchemy \
+    pyyaml \
+    pygments \
+    markdown \
+    bcrypt \
+    networkx
+```
  
-Note: If you want to always install python modules on the user account, edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or `Library/Application Support/pip/pip.conf` (OSX) and add the lines
+These are usually installed under
  
-    [global]
-    user = yes
+- `~/.local/lib/python3.6/site-packages/` in Linux/FreeBSD.
+- `~/Library/python/3.6/lib/python/site-packages/` in MacOS.
  
 ## Installation
  
 Replace USER by your bitbucket username:
  
-    cd path/to/some/directory
-    git clone https://USER@bitbucket.org/mjsb/aprendizations.git
-
-A directory `aprendizations` will be created with the software:
-
-    cd aprendizations
+```sh
+cd path/to/some/directory
+git clone https://USER@bitbucket.org/mjsb/aprendizations.git
+cd aprendizations
+```
  
 ## Configuration
  
 ### Database
  
-First we need to create a database:
+The user data is maintained in a sqlite3 database file. We first need to create the database:
  
-    ./initdb.py                 # initialize user `0` and empty password
-    ./initdb.py --pw alibaba    # initialize user `0` and given password
-    ./initdb.py --help          # for the available options
+```sh
+cd aprendizations
+./initdb.py --help          # for the available options
+./initdb.py                 # show current database or initialize empty if nonexisting
+./initdb.py inscricoes.csv  # add students from CSV, passwords are the numbers
+./initdb.py --add 1184 "Ana Bola"           # add new user (password=1184)
+./initdb.py --update 1184 --pw alibaba      # update password
+```
  
 ### SSL Certificates
  
-We need certificates for https. Certificates can be self-signed or certificates validated by a trusted authority.
-Self-signed can be used for development, but browsers will complain.
-LetsEncrypt issues trusted and free certificates, but the served must have a fixed IP and a domain name (not dynamic).
+We need certificates for https. Certificates can be self-signed or validated by a trusted authority.
  
-#### Selfsigned
+Self-signed can be used for testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a fixed IP and a registered domain name in the DNS (dynamic DNS does not work).
  
-Generate selfsigned certificates using
+#### Selfsigned
  
-    openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
+Generate a selfsigned certificate and place it in `aprendizations/certs`.
  
-and place them in `aprendizations/certs`.
+```sh
+openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
+```
  
 #### LetsEncrypt
  
-    sudo pkg install py27-certbot       # FreeBSD
+```sh
+sudo pkg install py27-certbot       # FreeBSD
+```
  
-Shutdown any server running and the firewall, and then run the script to generate the certificate:
+Shutdown the firewall and any server running. Then run the script to generate the certificate:
  
-    sudo pfctl -d                                   # disable pf firewall
-    sudo certbot certonly --standalone -d bit.xdi.uevora.pt
-    sudo pfctl -e; sudo pfctl -f /etc/pf.conf       # enable pf firewall
+```sh
+sudo pfctl -d                                   # disable pf firewall (FreeBSD)
+sudo certbot certonly --standalone -d www.example.com
+sudo pfctl -e; sudo pfctl -f /etc/pf.conf       # enable pf firewall
+```
  
-Certificates are saved under `/usr/local/etc/letsencrypt/live/bit.xdi.uevora.pt/` which is readable only by root.
+Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `aprendizations/certs` and change permissions to be readable:
  
-Copy them to `aprendizations/certs` with names `cert.pem` and `key.pem`. And change permissions to be readble (FIXME how to do it securily?)
+```sh
+sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
+sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
+chmod 400 cert.pem privkey.pem
+```
  
 Renews can be done running `certbot renew` (untested!).
  
  
 ### Testing
  
-Run a demonstration:
+The application includes a small example in `demo/demo.yaml`. Run it with
  
-    ./serve.py demo/demo.yaml
+```sh
+./serve.py demo/demo.yaml
+```
  
-and open a browser at `https://127.0.0.1:8443`.
+and open a browser at [https://127.0.0.1:8443](). If it everything looks good, check at the correct address `https://www.example.com` (requires port forward in the firewall).
  
-Logging level can be adjusted in `config/logger.yaml`.
  
 ### Firewall configuration
  
-Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as a regular user on higher ports like 8080 for http and 8443 for https. In this case, we should configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
+Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as an unprivileged user on higher ports like 8080 for http and 8443 for https. For this, we can configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
  
 #### FreeBSD and pf
  
 Edit `/etc/pf.conf`:
  
-    ext_if="em0"
-    rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
-    rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
+```sh
+ext_if="em0"                # this should be the correct network interface
+rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
+rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
+```
  
-or `ext_if="vtnet0"` for guest additions under virtual box.
+Under virtualbox with guest additions use `ext_if="vtnet0"`.
  
-Edit `rc.conf`
+Edit `/etc/rc.conf`
  
-    pf_enable="YES"
-    pf_flags=""
-    pf_rules="/etc/pf.conf"
+```sh
+pf_enable="YES"
+pf_flags=""
+pf_rules="/etc/pf.conf"
  
-    # optional logging:
-    pflog_enable="YES"
-    pflog_flags=""
-    pflog_logfile="/var/log/pflog"
+# optional logging:
+pflog_enable="YES"
+pflog_flags=""
+pflog_logfile="/var/log/pflog"
+```
  
 Reboot or `sudo service pf start`.
  
 ## Troubleshooting
  
+To help with troubleshooting, use the option `--debug` when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.
+
+Logging levels can be adjusted in `config/logger.yaml` and `config/logger-debug.yaml`.
+
+
 #### UnicodeEncodeError
  
 The server should not generate this error, but when using external scripts to generate questions or to correct, these scripts can print unicode strings to stdout. If the terminal does not support unicode, python will generate this exception.
  
 - FreeBSD fix: edit `~/.login_conf` to use UTF-8, for example:
  
-    me:\
-            :charset=UTF-8:\
-            :lang=en_US.UTF-8:
+```sh
+me:\
+    :charset=UTF-8:\
+    :lang=en_US.UTF-8:
+```
  
 - Debian fix: check `locale`...
  
-## Useful sqlite3 queries
-
-Which students did at least one topic?
-
-    sqlite3 students.db "select distinct student_id from studenttopic"
+## FAQ
  
+- Which students did at least one topic?
  
-How many topics had each student done?
+```sh
+sqlite3 students.db "select distinct student_id from studenttopic"
+```
  
-    sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
+- How many topics had each student done?
  
+```sh
+sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
+```
  
-Which questions have more wrong answers?
+- Which questions have more wrong answers?
  
-    sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
+```sh
+sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
+```
 \ No newline at end of file
@@ -157,7 +157,7 @@ class QuestionCheckbox(Question):
         if len(self['correct']) != n:
             logger.error(f'Options and correct size mismatch in "{self["ref"]}", file "{self["filename"]}".')
  
-        # if a option is a list of (right, wrong), pick one
+        # if an option is a list of (right, wrong), pick one
         # FIXME it's possible that all options are chosen wrong
         options = []
         correct = []
@@ -323,7 +323,7 @@ def main():
     http_server = tornado.httpserver.HTTPServer(webapp,
         ssl_options={
             "certfile": "certs/cert.pem",
-            "keyfile": "certs/key.pem"
+            "keyfile": "certs/privkey.pem"
         })
     http_server.listen(8443)
...	...	@@ -0,0 +1,244 @@
	1	+# Questions
	2	+
	3	+Questions are saved in files in the [YAML](http://www.yaml.org/start.html) format. Each file has a list of questions with the following structure:
	4	+
	5	+```yaml
	6	+- type: radio
	7	+ ref: question1
	8	+ ...
	9	+
	10	+- type: checkbox
	11	+ ref: question2
	12	+ ...
	13	+```
	14	+
	15	+The following kinds kinds of questions are supported:
	16	+
	17	+type \| kind of answer
	18	+-----\|-----------------
	19	+[radio](#radio) \| Choose exactly one option from list of options.
	20	+[checkbox](#checkbox) \| Choose zero, one or more options.
	21	+[text](#text) \| Line of text which is matched against a list of acceptable answers.
	22	+[text-regex](#text-regex) \| Similar to text, but the answer is validated by a regular expression.
	23	+[numeric-interval](#numeric-interval) \| Answer is interpreted as a floating point value (e.g. `1.2e-3`), which is checked against a closed interval.
	24	+[textarea](#textarea) \| The answer is a multiline block of text that is sent to an external program for assessment. The printed output of the external program is parsed to obtain the result.
	25	+[information, warning, alert and success](#information-warning-alert-and-success) \| These are not really questions, just information panels intended to be used in tests to convey information. There is no answer and it's always correct.
	26	+[generator](#generator) \| This is not a really a question type. It means that this question will be generated by an external program, and the actual type is defined there.
	27	+
	28	+In all questions, the field `type` is required. The field `ref` is not strictly required but still recommended, if not defined will default to FIXME.
	29	+
	30	+## radio
	31	+
	32	+Only one option can be selected as the answer. If no option is selected, the question is considered unanswered.
	33	+
	34	+The general format is
	35	+
	36	+```yaml
	37	+- type: radio
	38	+ ref: question_reference
	39	+ title: My first question
	40	+ text: \|
	41	+ Please select one option.
	42	+ options:
	43	+ - this one is the correct one
	44	+ - wrong
	45	+ - not correct but also not completely wrong
	46	+ correct: [1, 0, 0.1] # default: first option
	47	+ shuffle: yes # default: yes
	48	+ choose: 2 # default: all options
	49	+ discount: yes # default: yes
	50	+```
	51	+
	52	+All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true`.
	53	+
	54	+The `correct` field can be used in multiple ways and in combination with `shuffle`, `discount` and `choose` fields:
	55	+
	56	+- if not present, the first option is considered correct (options are usually shuffled...).
	57	+- it can be the index (0-based) of the correct option, e.g., `correct: 0`.
	58	+- it can be a list of numbers between 0 and 1, e.g., `correct: [1, 0, 0]`. In this case, the first option is 100% correct while the others are 0%. If `discount: true` (the default), then the wrong ones will be penalized by $-1/(n-1)=-\tfrac{1}{2}$, where $n$ is the number of options.
	59	+- there can be more than one correct option in the list, which is then marked in the correct field, e.g. `correct: [1, 1, 0]`. In this case, one of the correct options will be randomly selected, and the remaining wrong ones appended.
	60	+- there can also be a long list of right and wrong options from which to build the question options. E.g. if `correct: [1,1,1,0,0,0,0]` and `choose: 3` is defined, then 1 correct option and 2 wrong ones are randomly selected from the list.
	61	+- finally it's also possible to have a question that is "not-completely-right" or "not-completely-wrong". This can be done using numbers between 0 and 1, e.g., `correct: [1, 0.3, 0]`. This practice is discouraged.
	62	+
	63	+In some situations one may not want the options to be shuffled, e.g., if they show several steps of a proof and the student should mark the wrong step. In that case use `shuffle: false`.
	64	+
	65	+## checkbox
	66	+
	67	+Zero, one or multiple options can be selected. The question is always considered as answered, even if no options are selected, which is also a valid answed.
	68	+
	69	+The simplest format is
	70	+
	71	+```yaml
	72	+- type: checkbox
	73	+ ref: question_reference
	74	+ title: My second question
	75	+ text: \|
	76	+ Please mark the correct options.
	77	+ options:
	78	+ - this one is correct
	79	+ - wrong
	80	+ - this one is also correct
	81	+ correct: [1, -1, 1] # default: [0, 0, 0]
	82	+ shuffle: yes # default: yes
	83	+ choose: 2 # default: choose all options
	84	+ discount: yes # default: yes
	85	+```
	86	+
	87	+All fields are optional except `type` and `options`. `title` and `text` default to empty strings, `shuffle` and `discount` to `true` and `choose` to the total number of options.
	88	+
	89	+When correcting an answer, each correctly marked/unmarked option gets the corresponding value from the list `correct: [1, -1, 1]` and each wrong gets its symmetrical. So in the previous example, to have a completely right answer the checboxes should be: marked, unmarked, marked.
	90	+
	91	+If `discount: no` then wrong options are given a value of 0.
	92	+Options are shuffled by default. A smaller number of options may be randomly selected by setting the option `choose`.
	93	+
	94	+A more advanced format is to have
	95	+
	96	+```yaml
	97	+ options:
	98	+ - ['this one is correct', 'this is wrong']
	99	+ - 'wrong'
	100	+ - ['wrong again', 'this one is also correct']
	101	+ correct: [1, -1, -1]
	102	+```
	103	+
	104	+In this case, there are options that contain a list of 2 suboptions.
	105	+When the question is generated, one of the suboptions is randomly selected. If it's the first then the corresponding `correct` value is used, if it's the second then it's symmetrical is used instead.
	106	+
	107	+This format is useful to write options with 2 different versions to avoid cheating. Example:
	108	+
	109	+```yaml
	110	+ options:
	111	+ - ['$\pi$ is a real number', '$\pi$ is an integer']
	112	+ - ['there are more reals than integers', 'there are more integers than reals']
	113	+
	114	+```
	115	+
	116	+## text
	117	+
	118	+Answer is a line of text. Just compare the answered text with string provided in a list of acceptable answers.
	119	+
	120	+```yaml
	121	+- type: text
	122	+ ref: question-reference-3
	123	+ title: My third question
	124	+ text: Seven days are called a...
	125	+ correct: ['week', 'Week'] # default: [] always wrong
	126	+```
	127	+
	128	+## text-regex
	129	+
	130	+Similar to text, but validade using a regular expression.
	131	+
	132	+```yaml
	133	+- type: text-regex
	134	+ ref: question-reference-4
	135	+ title: My fourth question
	136	+ text: Seven days are called a...
	137	+ correct: !regex '[wW]eek' # default: '$.^' always wrong
	138	+```
	139	+
	140	+
	141	+## numeric-interval
	142	+
	143	+Similar to text, but expects an integer or floating point number. The answer is correct if the number is in the given interval.
	144	+
	145	+```yaml
	146	+- type: numeric-interval
	147	+ ref: question-reference-5
	148	+ title: My fifth question
	149	+ text: What are the first 3 fractional digits of $\pi$?
	150	+ correct: [3.141, 3.142] # default: [1.0, -1.0] always wrong
	151	+```
	152	+
	153	+## textarea
	154	+
	155	+Provides a multiline textarea for the answer. The answered text is sent to the stdin of an external program for assessment. The printed stdout output of the program is parsed as YAML to get the grade and optional comments.
	156	+
	157	+```yaml
	158	+- type: textarea
	159	+ ref: question-reference-6
	160	+ title: My sixth question
	161	+ text: Write a program in C that computes whatever.
	162	+ lines: 20 # default: 8
	163	+ correct: path/to/program # default: '' always wrong
	164	+ timeout: 15 # default: 5
	165	+```
	166	+
	167	+Example program output:
	168	+
	169	+```yaml
	170	+grade: 0.8
	171	+comments: Almost there
	172	+```
	173	+
	174	+## information, warning, alert and success
	175	+
	176	+```yaml
	177	+- type: information
	178	+ ref: question-reference-7
	179	+ title: Calculator
	180	+ text: You can use your calculator.
	181	+```
	182	+
	183	+## generator
	184	+
	185	+Generators are external programs that generate a question dynamically.
	186	+Questions should be printed to the stdout in YAML format (without the list dash). The parsed output is used to update the question dict, redefining `type` and other fields.
	187	+
	188	+Example of a generator written in python (any language can do):
	189	+
	190	+```python
	191	+#!/usr/bin/env python3
	192	+
	193	+from random import randint
	194	+import sys
	195	+
	196	+arg = sys.stdin.read() # read arguments
	197	+a,b = (int(n) for n in arg.split(','))
	198	+
	199	+q = fr'''
	200	+type: checkbox
	201	+text: \|
	202	+ Indique quais das seguintes adições resultam em overflow quando se considera a adição de números com sinal (complemento para 2) em registos de 8 bits.
	203	+
	204	+ Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
	205	+options:
	206	+'''
	207	+
	208	+correct = []
	209	+for i in range(5):
	210	+ x = randint(a, b)
	211	+ y = randint(a, b)
	212	+ q += f'- "`{x} + {y}`"\n'
	213	+ correct.append(1 if x + y > 127 else -1)
	214	+
	215	+q += 'correct: ' + str(correct)
	216	+
	217	+print(q)
	218	+```
	219	+
	220	+# Writing text
	221	+
	222	+The text in the questions is interpreted as markdown with LaTeX formulas.
	223	+The best way to avoid gotchas is to indent text like this:
	224	+
	225	+```yaml
	226	+ text: \|
	227	+ Yes. this is ok: If not indented, "Yes" would be a boolean
	228	+ and colon would be interpreted as a dictionary key.
	229	+
	230	+ Images placed in the `public` subdirectory are accessible by
	231	+ ![image](image.png "sunset")
	232	+
	233	+ LaTeX inline $\pi$ is ok, and
	234	+ $$
	235	+ \frac{\sqrt{2\pi\sigma^2}}{2}
	236	+ $$
	237	+ is also ok.
	238	+
	239	+ Tables are simple:
	240	+
	241	+ header1 \| header2
	242	+ --------\|---------
	243	+ value1 \| value2
	244	+```
0	245	\ No newline at end of file
...	...
1		-# Get Started
	1	+# Getting Started
2	2
3	3
4	4	## Requirements
5	5
6		-We will need to install python3.6 with sqlite3 support.
	6	+Before installing the server, we will need to install python with some additional packages.
	7	+
	8	+### Install python3.6 with sqlite3 support
	9	+
7	10	This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.
8	11
9	12	- Installing from the system package manager:
10		- - OSX: `port install python36`
11		- - FreeBSD: `pkg install python36 py36-sqlite3`
12		- - Linux: `apt-get install ???` (not available yet?)
	13	+ ```sh
	14	+ sudo port install python36 # MacOS
	15	+ sudo pkg install python36 py36-sqlite3 # FreeBSD
	16	+ sudo apt install ?not available yet? # Linux, install from source
	17	+ ```
13	18	- Installing from source:
14		- - Download from [http://www.python.org]()
15		- - `unxz Python-3.6.tar.xz`
16		- - `tar xvf Python-3.6.tar`
17		- - `cd Python-3.6`
18		- - `./configure --prefix=$HOME/.local/bin`
19		- - `make && make install`
	19	+ Download from [http://www.python.org]() and
	20	+
	21	+ ```sh
	22	+ unxz Python-3.6.tar.xz
	23	+ tar xvf Python-3.6.tar
	24	+ cd Python-3.6
	25	+ ./configure --prefix=$HOME/.local/bin
	26	+ make && make install
	27	+ ```
	28	+
	29	+ This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).
20	30
21		-This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in OSX or FreeBSD).
	31	+### Install pip
22	32
23		-Next install pip (if not yet installed):
	33	+If the `pip` command is not yet installed, run
24	34
25		- python3.6 -m ensurepip --user
	35	+```sh
	36	+python3.6 -m ensurepip --user
	37	+```
26	38
27		-This will install pip in your account under `~/.local/bin`.
	39	+This will install pip in your user account under `~/.local/bin`.
28	40	In the end you should be able to run `pip3 --version` and `python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`, `pip3.6` or `pip-3.6`).
29	41
30		-Install additional python packages locally on the user area:
	42	+If you want to always install python modules on the user account (recommended), edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or `Library/Application Support/pip/pip.conf` (MacOS) and add the lines
31	43
32		- pip3 install --user \
33		- tornado \
34		- sqlalchemy \
35		- pyyaml \
36		- pygments \
37		- markdown \
38		- bcrypt \
39		- networkx
	44	+```ini
	45	+[global]
	46	+user = yes
	47	+```
40	48
41		-These are usually installed under
	49	+### Install additional python packages:
42	50
43		-- OSX: `~/Library/python/3.6/lib/python/site-packages/`
44		-- Linux/FreeBSD: `~/.local/lib/python3.6/site-packages/`
	51	+```sh
	52	+pip3 install --user \
	53	+ tornado \
	54	+ sqlalchemy \
	55	+ pyyaml \
	56	+ pygments \
	57	+ markdown \
	58	+ bcrypt \
	59	+ networkx
	60	+```
45	61
46		-Note: If you want to always install python modules on the user account, edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or `Library/Application Support/pip/pip.conf` (OSX) and add the lines
	62	+These are usually installed under
47	63
48		- [global]
49		- user = yes
	64	+- `~/.local/lib/python3.6/site-packages/` in Linux/FreeBSD.
	65	+- `~/Library/python/3.6/lib/python/site-packages/` in MacOS.
50	66
51	67	## Installation
52	68
53	69	Replace USER by your bitbucket username:
54	70
55		- cd path/to/some/directory
56		- git clone https://USER@bitbucket.org/mjsb/aprendizations.git
57		-
58		-A directory `aprendizations` will be created with the software:
59		-
60		- cd aprendizations
	71	+```sh
	72	+cd path/to/some/directory
	73	+git clone https://USER@bitbucket.org/mjsb/aprendizations.git
	74	+cd aprendizations
	75	+```
61	76
62	77	## Configuration
63	78
64	79	### Database
65	80
66		-First we need to create a database:
	81	+The user data is maintained in a sqlite3 database file. We first need to create the database:
67	82
68		- ./initdb.py # initialize user `0` and empty password
69		- ./initdb.py --pw alibaba # initialize user `0` and given password
70		- ./initdb.py --help # for the available options
	83	+```sh
	84	+cd aprendizations
	85	+./initdb.py --help # for the available options
	86	+./initdb.py # show current database or initialize empty if nonexisting
	87	+./initdb.py inscricoes.csv # add students from CSV, passwords are the numbers
	88	+./initdb.py --add 1184 "Ana Bola" # add new user (password=1184)
	89	+./initdb.py --update 1184 --pw alibaba # update password
	90	+```
71	91
72	92	### SSL Certificates
73	93
74		-We need certificates for https. Certificates can be self-signed or certificates validated by a trusted authority.
75		-Self-signed can be used for development, but browsers will complain.
76		-LetsEncrypt issues trusted and free certificates, but the served must have a fixed IP and a domain name (not dynamic).
	94	+We need certificates for https. Certificates can be self-signed or validated by a trusted authority.
77	95
78		-#### Selfsigned
	96	+Self-signed can be used for testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a fixed IP and a registered domain name in the DNS (dynamic DNS does not work).
79	97
80		-Generate selfsigned certificates using
	98	+#### Selfsigned
81	99
82		- openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
	100	+Generate a selfsigned certificate and place it in `aprendizations/certs`.
83	101
84		-and place them in `aprendizations/certs`.
	102	+```sh
	103	+openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
	104	+```
85	105
86	106	#### LetsEncrypt
87	107
88		- sudo pkg install py27-certbot # FreeBSD
	108	+```sh
	109	+sudo pkg install py27-certbot # FreeBSD
	110	+```
89	111
90		-Shutdown any server running and the firewall, and then run the script to generate the certificate:
	112	+Shutdown the firewall and any server running. Then run the script to generate the certificate:
91	113
92		- sudo pfctl -d # disable pf firewall
93		- sudo certbot certonly --standalone -d bit.xdi.uevora.pt
94		- sudo pfctl -e; sudo pfctl -f /etc/pf.conf # enable pf firewall
	114	+```sh
	115	+sudo pfctl -d # disable pf firewall (FreeBSD)
	116	+sudo certbot certonly --standalone -d www.example.com
	117	+sudo pfctl -e; sudo pfctl -f /etc/pf.conf # enable pf firewall
	118	+```
95	119
96		-Certificates are saved under `/usr/local/etc/letsencrypt/live/bit.xdi.uevora.pt/` which is readable only by root.
	120	+Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `aprendizations/certs` and change permissions to be readable:
97	121
98		-Copy them to `aprendizations/certs` with names `cert.pem` and `key.pem`. And change permissions to be readble (FIXME how to do it securily?)
	122	+```sh
	123	+sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
	124	+sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
	125	+chmod 400 cert.pem privkey.pem
	126	+```
99	127
100	128	Renews can be done running `certbot renew` (untested!).
101	129
102	130
103	131	### Testing
104	132
105		-Run a demonstration:
	133	+The application includes a small example in `demo/demo.yaml`. Run it with
106	134
107		- ./serve.py demo/demo.yaml
	135	+```sh
	136	+./serve.py demo/demo.yaml
	137	+```
108	138
109		-and open a browser at `https://127.0.0.1:8443`.
	139	+and open a browser at [https://127.0.0.1:8443](). If it everything looks good, check at the correct address `https://www.example.com` (requires port forward in the firewall).
110	140
111		-Logging level can be adjusted in `config/logger.yaml`.
112	141
113	142	### Firewall configuration
114	143
115		-Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as a regular user on higher ports like 8080 for http and 8443 for https. In this case, we should configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
	144	+Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as an unprivileged user on higher ports like 8080 for http and 8443 for https. For this, we can configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
116	145
117	146	#### FreeBSD and pf
118	147
119	148	Edit `/etc/pf.conf`:
120	149
121		- ext_if="em0"
122		- rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
123		- rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
	150	+```sh
	151	+ext_if="em0" # this should be the correct network interface
	152	+rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
	153	+rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
	154	+```
124	155
125		-or `ext_if="vtnet0"` for guest additions under virtual box.
	156	+Under virtualbox with guest additions use `ext_if="vtnet0"`.
126	157
127		-Edit `rc.conf`
	158	+Edit `/etc/rc.conf`
128	159
129		- pf_enable="YES"
130		- pf_flags=""
131		- pf_rules="/etc/pf.conf"
	160	+```sh
	161	+pf_enable="YES"
	162	+pf_flags=""
	163	+pf_rules="/etc/pf.conf"
132	164
133		- # optional logging:
134		- pflog_enable="YES"
135		- pflog_flags=""
136		- pflog_logfile="/var/log/pflog"
	165	+# optional logging:
	166	+pflog_enable="YES"
	167	+pflog_flags=""
	168	+pflog_logfile="/var/log/pflog"
	169	+```
137	170
138	171	Reboot or `sudo service pf start`.
139	172
140	173	## Troubleshooting
141	174
	175	+To help with troubleshooting, use the option `--debug` when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.
	176	+
	177	+Logging levels can be adjusted in `config/logger.yaml` and `config/logger-debug.yaml`.
	178	+
	179	+
142	180	#### UnicodeEncodeError
143	181
144	182	The server should not generate this error, but when using external scripts to generate questions or to correct, these scripts can print unicode strings to stdout. If the terminal does not support unicode, python will generate this exception.
145	183
146	184	- FreeBSD fix: edit `~/.login_conf` to use UTF-8, for example:
147	185
148		- me:\
149		- :charset=UTF-8:\
150		- :lang=en_US.UTF-8:
	186	+```sh
	187	+me:\
	188	+ :charset=UTF-8:\
	189	+ :lang=en_US.UTF-8:
	190	+```
151	191
152	192	- Debian fix: check `locale`...
153	193
154		-## Useful sqlite3 queries
155		-
156		-Which students did at least one topic?
157		-
158		- sqlite3 students.db "select distinct student_id from studenttopic"
	194	+## FAQ
159	195
	196	+- Which students did at least one topic?
160	197
161		-How many topics had each student done?
	198	+```sh
	199	+sqlite3 students.db "select distinct student_id from studenttopic"
	200	+```
162	201
163		- sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
	202	+- How many topics had each student done?
164	203
	204	+```sh
	205	+sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
	206	+```
165	207
166		-Which questions have more wrong answers?
	208	+- Which questions have more wrong answers?
167	209
168		- sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
	210	+```sh
	211	+sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
	212	+```
169	213	\ No newline at end of file
...	...
...	...	@@ -157,7 +157,7 @@ class QuestionCheckbox(Question):
157	157	if len(self['correct']) != n:
158	158	logger.error(f'Options and correct size mismatch in "{self["ref"]}", file "{self["filename"]}".')
159	159
160		- # if a option is a list of (right, wrong), pick one
	160	+ # if an option is a list of (right, wrong), pick one
161	161	# FIXME it's possible that all options are chosen wrong
162	162	options = []
163	163	correct = []
...	...
...	...	@@ -323,7 +323,7 @@ def main():
323	323	http_server = tornado.httpserver.HTTPServer(webapp,
324	324	ssl_options={
325	325	"certfile": "certs/cert.pem",
326		- "keyfile": "certs/key.pem"
	326	+ "keyfile": "certs/privkey.pem"
327	327	})
328	328	http_server.listen(8443)
329	329
...	...