Getting Started

Requirements

Before installing the server, we will need to install python with some additional packages.

Install python3.7 with sqlite3 support

This can be done using the system package management, downloaded from http://www.python.org, or compiled from sources.

• Installing from the system package manager: sh sudo port install python37 # MacOS sudo pkg install python37 py37-sqlite3 # FreeBSD sudo apt install ?not available yet? # Linux, install from source

• Installing from source: Download from http://www.python.org and

  unxz Python-3.7.tar.xz
  tar xvf Python-3.7.tar
  cd Python-3.7
  ./configure --prefix=$HOME/.local/bin
  make && make install
  

  This will install python locally under ~/.local/bin. Make sure to add it to your PATH (edit ~/.profile in MacOS or FreeBSD).

Install pip

If the pip command is not yet installed, run

python3.7 -m ensurepip --user

This will install pip in your user account under ~/.local/bin. In the end you should be able to run pip3 --version and python3 -c "import sqlite3" without errors (sometimes pip3 is pip, pip3.7 or pip-3.7).

If you want to always install python modules on the user account (recommended), edit the pip configuration file ~/.config/pip/pip.conf (FreeBSD, Linux) or Library/Application Support/pip/pip.conf (MacOS) and add the lines

[global]
user = yes

Install additional python packages:

pip3 install --user \
    tornado \
    sqlalchemy \
    pyyaml \
    pygments \
    mistune \
    bcrypt \
    networkx

These are usually installed under

• ~/.local/lib/python3.7/site-packages/ in Linux/FreeBSD.
• ~/Library/python/3.7/lib/python/site-packages/ in MacOS.

Installation

Replace USER by your bitbucket username:

cd path/to/some/directory
git clone https://USER@bitbucket.org/mjsb/aprendizations.git
cd aprendizations

Configuration

Database

The user data is maintained in a sqlite3 database file. We first need to create the database:

cd aprendizations
./initdb.py                 # show current database or initialize empty if nonexisting
./initdb.py inscricoes.csv  # add students from CSV, passwords are the numbers
./initdb.py --add 1184 "Aladino da Silva"   # add new user (default password=1184)
./initdb.py --update 1184 --pw alibaba      # update password of given student
./initdb.py --help          # for the available options

SSL Certificates

We need certificates for https. Certificates can be self-signed or validated by a trusted authority.

Self-signed can be used locally for development and testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.

Selfsigned

Generate a selfsigned certificate and place it in aprendizations/certs.

openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes

LetsEncrypt

sudo pkg install py27-certbot       # FreeBSD

Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:

sudo service pf stop                            # disable pf firewall (FreeBSD)
sudo certbot certonly --standalone -d www.example.com
sudo service pf start                           # enable pf firewall

Certificates are saved under /usr/local/etc/letsencrypt/live/www.example.com/. Copy them to aprendizations/certs and change permissions to be readable:

sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
chmod 400 cert.pem privkey.pem

Renews can be done as follows:

sudo service pf stop   # shutdown firewall
sudo certbot renew
sudo service pf start  # start firewall

and then copy the cert.pem and privkey.pem files to aprendizations/certs directory. Change permissions and ownership as appropriate.

Testing

The application includes a small example in demo/demo.yaml. Run it with

./serve.py demo/demo.yaml

and open a browser at https://127.0.0.1:8443. If it everything looks good, check at the correct address https://www.example.com (requires port forward in the firewall). The option --debug provides more verbose logging and might be useful during testing.

Firewall configuration

Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as an unprivileged user on higher ports like 8080 for http and 8443 for https. For this, we can configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.

FreeBSD and pf

Edit /etc/pf.conf:

ext_if="em0"                # this should be the correct network interface
rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443

Under virtualbox with guest additions use ext_if="vtnet0".

Edit /etc/rc.conf

pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"

# optional logging:
pflog_enable="YES"
pflog_flags=""
pflog_logfile="/var/log/pflog"

Reboot or sudo service pf start.

Troubleshooting

To help with troubleshooting, use the option --debug when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.

Logging levels can be adjusted in config/logger.yaml and config/logger-debug.yaml.

UnicodeEncodeError

The server should not generate this error, but when using external scripts to generate questions or to correct, these scripts can print unicode strings to stdout. If the terminal does not support unicode, python will generate this exception.

• FreeBSD fix: edit ~/.login_conf to use UTF-8, for example:

me:\
    :charset=UTF-8:\
    :lang=en_US.UTF-8:

• Debian fix: check locale...

FAQ

• Which students did at least one topic?

sqlite3 students.db "select distinct student_id from studenttopic"

• How many topics had each student done?

sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"

• Which questions have more wrong answers?

sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"