# Getting Started

## Requirements

This application requires python3.7+ and a few additional python packages.

It also uses npm (Node package management) to install javascript libraries.

### Install python3.7 with sqlite3 support and npm

This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.

- Installing from the system package manager:

```sh

sudo port install python37 npm5              # MacOS

sudo pkg install python37 py37-sqlite3 npm   # FreeBSD

sudo apt install python3.7 npm               # Linux

```

- Installing from source:

Download from [http://www.python.org]() and

```sh

unxz Python-3.7.tar.xz
tar xvf Python-3.7.tar

cd Python-3.7
./configure --prefix=$HOME/.local/bin

make && make install

```

This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).

### Install pip

If the `pip` command is not yet installed, run one of these:

```sh

sudo apt install python3.7-pip     # Ubuntu

sudo pkg py37-pip                  # FreeBSD

sudo port install py37-pip         # MacOS
python3.7 -m ensurepip --user      # otherwise

```

The latter will install `pip` in your user account under `~/.local/bin`.
In the end you should be able to run `pip3 --version` and 
`python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`, 
`pip3.7` or `pip-3.7`).

If you want to always install python modules on the user account (recommended), 

edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or 

`Library/Application Support/pip/pip.conf` (MacOS) and add the lines

```ini
[global]

user = yes

```

### Install python packages and javascript libraries:

Replace USER by your bitbucket username:

```sh

cd somewhere

git clone https://git.xdi.uevora.pt/mjsb/aprendizations.git

cd aprendizations

npm install             # install javascript libraries

pip install .           # install aprendizations and dependencies

```

Python packages are usually installed in:

- `~/.local/lib/python3.7/site-packages/` in Linux/FreeBSD.
- `~/Library/python/3.7/lib/python/site-packages/` in MacOS.

Javascript libraries are installed in `aprendizations/node_modules` directory.

This libraries have symbolic links from `aprendizations/aprendizations/static`.

At this point aprendizations is installed in

```sh
~/Library/Python/3.7/bin              # MacOS
~/.local/bin                          # FreeBSD/Linux

```

Make sure this directory is in your `$PATH`.

The server can be run with the command `aprendizations` from the terminal.

## Configuration

### Database

The user data is maintained in a sqlite3 database file. We first need to create 

the database.

At the moment, the database should be located in the same directory as the main 

configuration file (see below). As an example, do

```sh

cd demo                               # contains a small example
initdb-aprendizations                 # show or initialize database
initdb-aprendizations --admin         # add admin user

initdb-aprendizations inscricoes.csv  # add students from CSV

initdb-aprendizations --add 1184 "Aladino da Silva"   # add user
initdb-aprendizations --update 1184 --pw alibaba      # update password
initdb-aprendizations --help          # for the available options

```

The default password is equal to the user name used to login.

### SSL Certificates

We need certificates for https. Certificates can be self-signed or validated by 

a trusted authority.

Self-signed can be used locally for development and testing, but browsers will 
complain. LetsEncrypt issues trusted and free certificates, but the server must 

have a registered publicly accessible domain name.

#### Selfsigned

Generate a selfsigned certificate and place it in `~/.local/share/certs`.

```sh

openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes

```

#### LetsEncrypt

```sh

sudo pkg install py27-certbot       # FreeBSD
```

Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:

```sh

sudo service pf stop                            # disable pf firewall (FreeBSD)

sudo certbot certonly --standalone -d www.example.com
sudo service pf start                           # enable pf firewall
```

Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `~/.local/share/certs` and change permissions to be readable:

```sh
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
chmod 400 cert.pem privkey.pem
```

Renews can be done as follows:

```sh

sudo service pf stop   # shutdown firewall

sudo certbot renew

sudo service pf start  # start firewall

```

and then copy the `cert.pem` and `privkey.pem` files to `~/.local/share/certs` directory. Change permissions and ownership as appropriate.

### Testing

The application includes a small example in `demo/demo.yaml`. Run it with

```sh

cd demo
aprendizations demo.yaml

```

and open a browser at [https://127.0.0.1:8443](https://127.0.0.1:8443). 

If it everything looks good, 
check at the correct address `https://www.example.com` (requires port forward 
in the firewall). The option `--debug` provides more verbose logging and might 

be useful during testing. The option `--check` generates all the questions once
before running the server to check for any obvious syntax error.


### Firewall configuration

Ports 80 and 443 are only usable by root. For security reasons it is better to 

run the server as an unprivileged user on higher ports like 8080 for http and 
8443 for https. For this, we can configure port forwarding in the firewall to 
redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.

#### FreeBSD and pf

Edit `/etc/pf.conf`:

```sh
ext_if="em0"                # this should be the correct network interface
rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443

```

Under virtualbox with guest additions use `ext_if="vtnet0"`.

Edit `/etc/rc.conf`

```sh

pf_enable="YES"

pf_flags=""

pf_rules="/etc/pf.conf"

# optional logging:
pflog_enable="YES"
pflog_flags=""
pflog_logfile="/var/log/pflog"

```

Reboot or `sudo service pf start`.

## Troubleshooting

To help with troubleshooting, use the option `--debug` when running the server. 
This will increase logs in the terminal and will present the python exception 
errors in the browser.

Logging levels can be adjusted in `~/.config/aprendizations/logger.yaml` and 

`~/.config/aprendizations/logger-debug.yaml`.

If these files do not yet exist, there are examples in `aprendizations/config` that can be copied to `~/.config/aprendizations`.

#### UnicodeEncodeError

The server should not generate this error, but when using external scripts to 

generate questions or to correct, these scripts can print unicode strings to 

stdout. If the terminal does not support unicode, python will generate this 

exception.

- FreeBSD fix: edit `~/.login_conf` to use UTF-8, for example:

```sh

me:\

    :charset=UTF-8:\

    :lang=en_US.UTF-8:

```

- Debian fix: check `locale`...

## FAQ

- Which students did at least one topic?

```sh
sqlite3 students.db "select distinct student_id from studenttopic"
```

- How many topics has each student done?

```sh
sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
```

- Which questions have more wrong answers?

```sh
sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
```