# Getting Started

To complete the installation we will need to perform the following steps:

1. install python3.7, pip and npm
1. download aprendizations from the repository

1. install javascript libraries (with npm)

1. install aprendizations (with pip)

1. initialize database
1. generate SSL certificates
1. configure the firewall (optional)
1. try running `aprendizations demo.yaml`

These steps are explained next in detail.

### Install python3.7 with sqlite3 support and npm

Python can be installed either from the system package management or compiled from sources.

#### Installing from the system package manager

```sh

sudo apt install python3.7 npm               # Linux (Ubuntu)

sudo pkg install python37 py37-sqlite3 npm   # FreeBSD
sudo port install python37 npm6              # MacOS

```

#### Installing from source
     

Make sure that the build tools and libraries are installed:

```sh

# Ubuntu:

sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev

```

Download python from [http://www.python.org]() and

```sh

tar xvfJ Python-3.7.tar.xz

cd Python-3.7

./configure --prefix=$HOME/.local --enable-optimizations

make && make install
```

This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` in `~/.profile`. If `~/bin` is already in the path, you may just make a symbolic link `ln -s ~/.local/bin ~/bin`.

### Install pip

Python usually includes pip which is accessible through `python -m pip install something`, but it's also convenient to have the `pip` command installed. 
If the `pip` command is not yet installed, run one of these:

```sh

sudo apt install python3.7-pip     # Ubuntu 19.04+
python3.7 -m pip install pip       # Ubuntu 18.04
sudo pkg py37-pip                  # FreeBSD
sudo port install py37-pip         # MacOS

```

The latter will install `pip` in your user account under `~/.local/bin`.

In the end you should be able to run `pip --version` and 

`python3 -c "import sqlite3"` without errors.

Sometimes the `pip` command is named `pip3`, 

`pip3.7` or `pip-3.7`.

Edit the configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or 

`Library/Application Support/pip/pip.conf` (MacOS) and add the lines

```ini

[global]

user = yes

```

This will set pip to install modules in the user area (recommended).

### Install aprendizations and dependencies:

```sh

git clone https://git.xdi.uevora.pt/mjsb/aprendizations.git
cd aprendizations

npm install             # install javascript libraries

pip install .           # install aprendizations and dependencies

```

Javascript libraries are initially installed in `aprendizations/node_modules` directory.

These libraries already have symbolic links from `aprendizations/aprendizations/static`.

Python packages are usually installed in:

- `~/.local/lib/python3.7/site-packages/` in Linux/FreeBSD.

- `~/Library/python/3.7/lib/python/site-packages/` in MacOS.

When aprendizations is installed with pip, all the dependencies are also installed. The javascript libraries previously installed with npm are copied to the above directory and the cloned repository is no longer needed.

At this point, aprendizations is installed in

```sh

~/.local/bin                          # Linux/FreeBSD

~/Library/Python/3.7/bin              # MacOS

```

and can be run from the terminal:

```sh

aprendizations --version

aprendizations --help
```

## Configuration

### Database

User data is maintained in a sqlite3 database which has to be created manually using the command `initdb-aprendizations`.

The database file should be located in the same directory as the main 

YAML configuration file.

For example, to run the included demo do:

```sh
cd demo                               # contains a small example

initdb-aprendizations                 # show or initialize database

initdb-aprendizations --admin         # add admin user

initdb-aprendizations inscricoes.csv  # add students from CSV
initdb-aprendizations --add 1184 "Aladino da Silva"   # add new user
initdb-aprendizations --update 1184 --pw alibaba      # update password

initdb-aprendizations --help          # for available options

```

The default password is equal to the user name, if left undefined.


### SSL Certificates

We need certificates for https. Certificates can be self-signed or validated by a trusted authority.

Self-signed can be used locally for development and testing, but browsers will 
complain. LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.

#### Generating selfsigned certificates

Generate a selfsigned certificate and place it in `~/.local/share/certs`.

```sh
openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
```

#### LetsEncrypt certificates

Install the certbot from LetsEncrypt:

```sh
sudo pkg install py36-certbot       # FreeBSD
sudo apt install certbot            # Linux Ubuntu
```

To generate or renew the certificates, ports 80 and 443 have to be accessible. The firewall and webserver have to be stopped. 

```sh

sudo certbot certonly --standalone -d www.example.com   # first time

sudo certbot renew                                      # renew

```

Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `~/.local/share/certs` and change permissions to be readable:

```sh

sudo cp /usr/local/etc/letsencrypt/live/www.example.com/cert.pem .
sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .

chmod 400 cert.pem privkey.pem

```

### Running the demo

The application includes a small example in `demo/demo.yaml` that can be used for initial testing. Run it with

```sh
cd demo

aprendizations demo.yaml
```

Open the browser at [https://127.0.0.1:8443](https://127.0.0.1:8443). 
If it everything looks good, check at the correct address 

`https://www.example.com:8443`.

The option `--debug` provides more verbose logging and might 

be useful during testing.

### Firewall configuration

Ports 80 and 443 are only usable by root. For security reasons the server runs as an unprivileged user on port 8443 for https. 

To access the server in the default https port (443), port forwarding can be configured in the firewall. 

#### FreeBSD and pf

Edit `/etc/pf.conf`:

```sh

ext_if="em0"                # change em0 to the correct network interface

rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
```

Under virtualbox with guest additions use `ext_if="vtnet0"`.

Edit `/etc/rc.conf`

```sh
pf_enable="YES"

pf_flags=""
pf_rules="/etc/pf.conf"

# optional logging:

pflog_enable="YES"
pflog_flags=""

pflog_logfile="/var/log/pflog"
```

Reboot or `sudo service pf start`.

### Testing the system

Make sure the following steps have been done:

- installed python3.7, pip and npm

- git-cloned the aprendizations from the main repository

- installed javascript libraries with npm
- installed aprendizations with pip

- initialized database with at least 1 user
- generate and copy certificates to the appropriate place

- (optional) configure the firewall to do port forwarding
- run `aprendizations demo.yaml --check`

## Troubleshooting

To help with troubleshooting, use the option `--debug` when running the server. 

This will increase logs in the terminal and will present the python exception 

errors in the browser.

Logging levels can be adjusted in `~/.config/aprendizations/logger.yaml` and 

`~/.config/aprendizations/logger-debug.yaml`.

If these files do not yet exist, there are examples in `aprendizations/config` that can be copied to `~/.config/aprendizations`.

#### UnicodeEncodeError

The server should not generate this error, but when using external scripts to 

generate questions or to correct, these scripts can print unicode strings to 

stdout. If the terminal does not support unicode, python will generate this 

exception.

- FreeBSD fix: edit `~/.login_conf` to use UTF-8, for example:

```sh

me:\

    :charset=UTF-8:\
    :lang=en_US.UTF-8:
```

- Debian fix: check `locale`...

## FAQ

Common database manipulations:

```sh
initdb-aprendizations -u 12345 --pw alibaba     # reset student password
initdb-aprendizations -a 12345 --pw alibaba     # add new student
```

Common database queries:

```sh

# Which students did at least one topic?

sqlite3 students.db "select distinct student_id from studenttopic"

# How many topics has each student done?
sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"

# Which questions have more wrong answers?

sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"

```