# Getting Started


## Requirements

Before installing the server, we will need to install python with some additional packages.

### Install python3.6 with sqlite3 support

This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.

- Installing from the system package manager:
    ```sh
    sudo port install python36                # MacOS
    sudo pkg install python36 py36-sqlite3    # FreeBSD
    sudo apt install ?not available yet?      # Linux, install from source
    ```
- Installing from source:
     Download from [http://www.python.org]() and

    ```sh
    unxz Python-3.6.tar.xz
    tar xvf Python-3.6.tar
    cd Python-3.6
    ./configure --prefix=$HOME/.local/bin
    make && make install
    ```

    This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).

### Install pip

If the `pip` command is not yet installed, run

```sh
python3.6 -m ensurepip --user
```

This will install pip in your user account under `~/.local/bin`.
In the end you should be able to run `pip3 --version` and `python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`, `pip3.6` or `pip-3.6`).

If you want to always install python modules on the user account (recommended), edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or `Library/Application Support/pip/pip.conf` (MacOS) and add the lines

```ini
[global]
user = yes
```

### Install additional python packages:

```sh
pip3 install --user \
    tornado \
    sqlalchemy \
    pyyaml \
    pygments \
    markdown \
    bcrypt \
    networkx
```

These are usually installed under

- `~/.local/lib/python3.6/site-packages/` in Linux/FreeBSD.
- `~/Library/python/3.6/lib/python/site-packages/` in MacOS.

## Installation

Replace USER by your bitbucket username:

```sh
cd path/to/some/directory
git clone https://USER@bitbucket.org/mjsb/aprendizations.git
cd aprendizations
```

## Configuration

### Database

The user data is maintained in a sqlite3 database file. We first need to create the database:

```sh
cd aprendizations
./initdb.py --help          # for the available options
./initdb.py                 # show current database or initialize empty if nonexisting
./initdb.py inscricoes.csv  # add students from CSV, passwords are the numbers
./initdb.py --add 1184 "Ana Bola"           # add new user (password=1184)
./initdb.py --update 1184 --pw alibaba      # update password
```

### SSL Certificates

We need certificates for https. Certificates can be self-signed or validated by a trusted authority.

Self-signed can be used for testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a fixed IP and a registered domain name in the DNS (dynamic DNS does not work).

#### Selfsigned

Generate a selfsigned certificate and place it in `aprendizations/certs`.

```sh
openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
```

#### LetsEncrypt

```sh
sudo pkg install py27-certbot       # FreeBSD
```

Shutdown the firewall and any server running. Then run the script to generate the certificate:

```sh
sudo service pf stop                            # disable pf firewall (FreeBSD)
sudo certbot certonly --standalone -d www.example.com
sudo service pf start                           # enable pf firewall
```

Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `aprendizations/certs` and change permissions to be readable:

```sh
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
chmod 400 cert.pem privkey.pem
```

Renews can be done as follows:

```sh
sudo service pf stop   # shutdown firewall
sudo certbot renew
sudo service pf start  # start firewall
```

and then copy the `cert.pem` and `privkey.pem` files to `aprendizations/certs` directory. Change permissions and ownership as appropriate.


### Testing

The application includes a small example in `demo/demo.yaml`. Run it with

```sh
./serve.py demo/demo.yaml
```

and open a browser at [https://127.0.0.1:8443](). If it everything looks good, check at the correct address `https://www.example.com` (requires port forward in the firewall).


### Firewall configuration

Ports 80 and 443 are only usable by root. For security reasons it is better to run the server as an unprivileged user on higher ports like 8080 for http and 8443 for https. For this, we can configure port forwarding in the firewall to redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.

#### FreeBSD and pf

Edit `/etc/pf.conf`:

```sh
ext_if="em0"                # this should be the correct network interface
rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
```

Under virtualbox with guest additions use `ext_if="vtnet0"`.

Edit `/etc/rc.conf`

```sh
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"

# optional logging:
pflog_enable="YES"
pflog_flags=""
pflog_logfile="/var/log/pflog"
```

Reboot or `sudo service pf start`.

## Troubleshooting

To help with troubleshooting, use the option `--debug` when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.

Logging levels can be adjusted in `config/logger.yaml` and `config/logger-debug.yaml`.


#### UnicodeEncodeError

The server should not generate this error, but when using external scripts to generate questions or to correct, these scripts can print unicode strings to stdout. If the terminal does not support unicode, python will generate this exception.

- FreeBSD fix: edit `~/.login_conf` to use UTF-8, for example:

```sh
me:\
    :charset=UTF-8:\
    :lang=en_US.UTF-8:
```

- Debian fix: check `locale`...

## FAQ

- Which students did at least one topic?

```sh
sqlite3 students.db "select distinct student_id from studenttopic"
```

- How many topics had each student done?

```sh
sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
```

- Which questions have more wrong answers?

```sh
sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
```