Getting Started

Requirements

This application requires python3.7+ and a few additional python packages. It also uses npm (Node package management) to install javascript libraries.

Install python3.7 with sqlite3 support and npm

This can be done using the system package management, downloaded from http://www.python.org, or compiled from sources.

• Installing from the system package manager:

sudo port install python37 npm5              # MacOS
sudo pkg install python37 py37-sqlite3 npm   # FreeBSD
sudo apt install python3.7 npm               # Linux

• Installing from source:

Download from http://www.python.org and

unxz Python-3.7.tar.xz
tar xvf Python-3.7.tar
cd Python-3.7
./configure --prefix=$HOME/.local/bin
make && make install

This will install python locally under ~/.local/bin. Make sure to add it to your PATH (edit ~/.profile in MacOS or FreeBSD).

Install pip

If the pip command is not yet installed, run one of these:

sudo apt install python3.7-pip     # Ubuntu
sudo pkg py37-pip                  # FreeBSD
sudo port install py37-pip         # MacOS
python3.7 -m ensurepip --user      # otherwise

The latter will install pip in your user account under ~/.local/bin. In the end you should be able to run pip3 --version and python3 -c "import sqlite3" without errors (sometimes pip3 is pip, pip3.7 or pip-3.7).

If you want to always install python modules on the user account (recommended), edit the pip configuration file ~/.config/pip/pip.conf (FreeBSD, Linux) or Library/Application Support/pip/pip.conf (MacOS) and add the lines

[global]
user = yes

Install python packages and javascript libraries:

Replace USER by your bitbucket username:

cd somewhere
git clone https://git.xdi.uevora.pt/mjsb/aprendizations.git
cd aprendizations
pip install .           # install aprendizations and dependencies
npm install             # install javascript libraries

Python packages are usually installed in:

• ~/.local/lib/python3.7/site-packages/ in Linux/FreeBSD.
• ~/Library/python/3.7/lib/python/site-packages/ in MacOS.

Javascript libraries are installed in aprendizations/node_modules directory. This libraries have symbolic links from aprendizations/aprendizations/static.

At this point aprendizations is installed in

~/Library/Python/3.7/bin              # MacOS
~/.local/bin                          # FreeBSD/Linux

Make sure this directory is in your $PATH.

The server can be run with the command aprendizations from the terminal.

Configuration

Database

The user data is maintained in a sqlite3 database file. We first need to create the database. At the moment, the database should be located in the same directory as the main configuration file (see below). As an example, do

cd demo                               # contains a small example
initdb-aprendizations                 # show or initialize database
initdb-aprendizations --admin         # add admin user
initdb-aprendizations inscricoes.csv  # add students from CSV
initdb-aprendizations --add 1184 "Aladino da Silva"   # add user
initdb-aprendizations --update 1184 --pw alibaba      # update password
initdb-aprendizations --help          # for the available options

The default password is equal to the user name used to login.

SSL Certificates

We need certificates for https. Certificates can be self-signed or validated by a trusted authority.

Self-signed can be used locally for development and testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.

Selfsigned

Generate a selfsigned certificate and place it in ~/.local/share/certs.

openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes

LetsEncrypt

sudo pkg install py27-certbot       # FreeBSD

Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:

sudo service pf stop                            # disable pf firewall (FreeBSD)
sudo certbot certonly --standalone -d www.example.com
sudo service pf start                           # enable pf firewall

Certificates are saved under /usr/local/etc/letsencrypt/live/www.example.com/. Copy them to ~/.local/share/certs and change permissions to be readable:

sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
chmod 400 cert.pem privkey.pem

Renews can be done as follows:

sudo service pf stop   # shutdown firewall
sudo certbot renew
sudo service pf start  # start firewall

and then copy the cert.pem and privkey.pem files to ~/.local/share/certs directory. Change permissions and ownership as appropriate.

Testing

The application includes a small example in demo/demo.yaml. Run it with

cd demo
aprendizations demo.yaml

and open a browser at https://127.0.0.1:8443. If it everything looks good, check at the correct address https://www.example.com (requires port forward in the firewall). The option --debug provides more verbose logging and might be useful during testing. The option --check generates all the questions once before running the server to check for any obvious syntax error.

Firewall configuration

Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as an unprivileged user on higher ports like 8080 for http and 8443 for https. For this, we can configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.

FreeBSD and pf

Edit /etc/pf.conf:

ext_if="em0"                # this should be the correct network interface
rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443

Under virtualbox with guest additions use ext_if="vtnet0".

Edit /etc/rc.conf

pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"

# optional logging:
pflog_enable="YES"
pflog_flags=""
pflog_logfile="/var/log/pflog"

Reboot or sudo service pf start.

Troubleshooting

To help with troubleshooting, use the option --debug when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.

Logging levels can be adjusted in ~/.config/aprendizations/logger.yaml and ~/.config/aprendizations/logger-debug.yaml.

If these files do not yet exist, there are examples in aprendizations/config that can be copied to ~/.config/aprendizations.

UnicodeEncodeError

The server should not generate this error, but when using external scripts to generate questions or to correct, these scripts can print unicode strings to stdout. If the terminal does not support unicode, python will generate this exception.

• FreeBSD fix: edit ~/.login_conf to use UTF-8, for example:

me:\
    :charset=UTF-8:\
    :lang=en_US.UTF-8:

• Debian fix: check locale...

FAQ

• Which students did at least one topic?

sqlite3 students.db "select distinct student_id from studenttopic"

• How many topics has each student done?

sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"

• Which questions have more wrong answers?

sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"