updates README.md.

adds the --version option. shows instructions in the terminal for some failures in configuration. fix error with exception not detected when database not usable.

updates README.md.
adds the --version option. shows instructions in the terminal for some failures in configuration. fix error with exception not detected when database not usable.
Miguel Barão
1 parent aea7e9d8
Showing 4 changed files with 154 additions and 97 deletions Show diff stats
BUGS.md
README.md
aprendizations/learnapp.py
aprendizations/serve.py
@@ -3,7 +3,7 @@
  
 - quando termina topico devia apagar as perguntas todas (se falhar a gerar novo topico, aparecem perguntas do antigo)
 - apos clicar no botao responder, inactivar o input (importante quando o tempo de correcção é grande)
-- devia mostrar timout para o aluno saber a razao.
+- devia mostrar timeout para o aluno saber a razao.
 - permitir configuracao para escolher entre static files locais ou remotos
 - sqlalchemy.pool.impl.NullPool: Exception during reset or similar
 sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread.
@@ -40,6 +40,7 @@ sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in
  
 # FIXED
  
+- add aprendizatons --version
 - se aluno abre dois tabs no browser, conseque navegar em simultaneo para perguntas diferentes. quando submete uma delas dá asneira. Tem de haver um campo hidden que tenha um céodigo único que indique qual a pergunta. do lado do servidor apnas há o codigo da pergunta corrente, se forem diferentes faz redirect para /.
 - nos topicos learn.yaml, qd falha acrescenta no fim. nao faz sentido.
 - não esta a fazer render correcto de tabelas nas opcoes checkbox. e.g. information-theory/source-coding-theory/block-codes
 # Getting Started
  
+To complete the installation we will need to perform the following steps:
  
-## Requirements
+1. install python3.7, pip and npm
+1. download aprendizations from the repository
+1. install javascript libraries (with npm)
+1. install aprendizations (with pip)
+1. initialize database
+1. generate SSL certificates
+1. configure the firewall (optional)
+1. try running `aprendizations demo.yaml`
  
-This application requires python3.7+ and a few additional python packages.
-It also uses npm (Node package management) to install javascript libraries.
+These steps are explained next in detail.
  
 ### Install python3.7 with sqlite3 support and npm
  
-This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.
+Python can be installed either from the system package management or compiled from sources.
  
-- Installing from the system package manager:
+#### Installing from the system package manager
  
 ```sh
-sudo port install python37 npm5              # MacOS
+sudo apt install python3.7 npm               # Linux (Ubuntu)
 sudo pkg install python37 py37-sqlite3 npm   # FreeBSD
-sudo apt install python3.7 npm               # Linux
+sudo port install python37 npm6              # MacOS
 ```
  
-- Installing from source:
+#### Installing from source
  
-Download from [http://www.python.org]() and
+Make sure that the build tools and libraries are installed:
  
 ```sh
-unxz Python-3.7.tar.xz
-tar xvf Python-3.7.tar
+# Ubuntu:
+sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev
+```
+
+Download python from [http://www.python.org]() and
+
+```sh
+tar xvfJ Python-3.7.tar.xz
 cd Python-3.7
-./configure --prefix=$HOME/.local/bin
+./configure --prefix=$HOME/.local --enable-optimizations
 make && make install
 ```
  
-This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).
-
+This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` in `~/.profile`. If `~/bin` is already in the path, you may just make a symbolic link `ln -s ~/.local/bin ~/bin`.
  
 ### Install pip
  
+Python usually includes pip which is accessible through `python -m pip install something`, but it's also convenient to have the `pip` command installed. 
 If the `pip` command is not yet installed, run one of these:
  
 ```sh
-sudo apt install python3.7-pip     # Ubuntu
+sudo apt install python3.7-pip     # Ubuntu 19.04+
+python3.7 -m pip install pip       # Ubuntu 18.04
 sudo pkg py37-pip                  # FreeBSD
 sudo port install py37-pip         # MacOS
-python3.7 -m ensurepip --user      # otherwise
 ```
  
 The latter will install `pip` in your user account under `~/.local/bin`.
-In the end you should be able to run `pip3 --version` and 
-`python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`, 
-`pip3.7` or `pip-3.7`).
+In the end you should be able to run `pip --version` and 
+`python3 -c "import sqlite3"` without errors.
+Sometimes the `pip` command is named `pip3`, 
+`pip3.7` or `pip-3.7`.
  
-If you want to always install python modules on the user account (recommended), 
-edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or 
+Edit the configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or 
 `Library/Application Support/pip/pip.conf` (MacOS) and add the lines
  
 ```ini
@@ -58,69 +71,72 @@ edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or
 user = yes
 ```
  
-### Install python packages and javascript libraries:
+This will set pip to install modules in the user area (recommended).
  
-Replace USER by your bitbucket username:
+### Install aprendizations and dependencies:
  
 ```sh
-cd somewhere
 git clone https://git.xdi.uevora.pt/mjsb/aprendizations.git
 cd aprendizations
 npm install             # install javascript libraries
 pip install .           # install aprendizations and dependencies
 ```
  
+Javascript libraries are initially installed in `aprendizations/node_modules` directory.
+These libraries already have symbolic links from `aprendizations/aprendizations/static`.
+
 Python packages are usually installed in:
  
 - `~/.local/lib/python3.7/site-packages/` in Linux/FreeBSD.
 - `~/Library/python/3.7/lib/python/site-packages/` in MacOS.
  
-Javascript libraries are installed in `aprendizations/node_modules` directory.
-This libraries have symbolic links from `aprendizations/aprendizations/static`.
+When aprendizations is installed with pip, all the dependencies are also installed. The javascript libraries previously installed with npm are copied to the above directory and the cloned repository is no longer needed.
  
-At this point aprendizations is installed in
+At this point, aprendizations is installed in
  
 ```sh
+~/.local/bin                          # Linux/FreeBSD
 ~/Library/Python/3.7/bin              # MacOS
-~/.local/bin                          # FreeBSD/Linux
 ```
  
-Make sure this directory is in your `$PATH`.
+and can be run from the terminal:
  
-The server can be run with the command `aprendizations` from the terminal.
+```sh
+aprendizations --version
+aprendizations --help
+```
  
 ## Configuration
  
 ### Database
  
-The user data is maintained in a sqlite3 database file. We first need to create 
-the database.
-At the moment, the database should be located in the same directory as the main 
-configuration file (see below). As an example, do
+User data is maintained in a sqlite3 database which has to be created manually using the command `initdb-aprendizations`.
+The database file should be located in the same directory as the main 
+YAML configuration file.
+
+For example, to run the included demo do:
  
 ```sh
 cd demo                               # contains a small example
 initdb-aprendizations                 # show or initialize database
 initdb-aprendizations --admin         # add admin user
 initdb-aprendizations inscricoes.csv  # add students from CSV
-initdb-aprendizations --add 1184 "Aladino da Silva"   # add user
+initdb-aprendizations --add 1184 "Aladino da Silva"   # add new user
 initdb-aprendizations --update 1184 --pw alibaba      # update password
-initdb-aprendizations --help          # for the available options
+initdb-aprendizations --help          # for available options
 ```
  
-The default password is equal to the user name used to login.
+The default password is equal to the user name, if left undefined.
  
  
 ### SSL Certificates
  
-We need certificates for https. Certificates can be self-signed or validated by 
-a trusted authority.
+We need certificates for https. Certificates can be self-signed or validated by a trusted authority.
  
 Self-signed can be used locally for development and testing, but browsers will 
-complain. LetsEncrypt issues trusted and free certificates, but the server must 
-have a registered publicly accessible domain name.
+complain. LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.
  
-#### Selfsigned
+#### Generating selfsigned certificates
  
 Generate a selfsigned certificate and place it in `~/.local/share/certs`.
  
@@ -128,18 +144,20 @@ Generate a selfsigned certificate and place it in `~/.local/share/certs`.
 openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
 ```
  
-#### LetsEncrypt
+#### LetsEncrypt certificates
+
+Install the certbot from LetsEncrypt:
  
 ```sh
-sudo pkg install py27-certbot       # FreeBSD
+sudo pkg install py36-certbot       # FreeBSD
+sudo apt install certbot            # Linux Ubuntu
 ```
  
-Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:
+To generate or renew the certificates, ports 80 and 443 have to be accessible. The firewall and webserver have to be stopped. 
  
 ```sh
-sudo service pf stop                            # disable pf firewall (FreeBSD)
-sudo certbot certonly --standalone -d www.example.com
-sudo service pf start                           # enable pf firewall
+sudo certbot certonly --standalone -d www.example.com   # first time
+sudo certbot renew                                      # renew
 ```
  
 Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `~/.local/share/certs` and change permissions to be readable:
@@ -150,47 +168,32 @@ sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
 chmod 400 cert.pem privkey.pem
 ```
  
-Renews can be done as follows:
-
-```sh
-sudo service pf stop   # shutdown firewall
-sudo certbot renew
-sudo service pf start  # start firewall
-```
-
-and then copy the `cert.pem` and `privkey.pem` files to `~/.local/share/certs` directory. Change permissions and ownership as appropriate.
+### Running the demo
  
-
-### Testing
-
-The application includes a small example in `demo/demo.yaml`. Run it with
+The application includes a small example in `demo/demo.yaml` that can be used for initial testing. Run it with
  
 ```sh
 cd demo
 aprendizations demo.yaml
 ```
  
-and open a browser at [https://127.0.0.1:8443](https://127.0.0.1:8443). 
-If it everything looks good, 
-check at the correct address `https://www.example.com` (requires port forward 
-in the firewall). The option `--debug` provides more verbose logging and might 
-be useful during testing. The option `--check` generates all the questions once
-before running the server to check for any obvious syntax error.
-
+Open the browser at [https://127.0.0.1:8443](https://127.0.0.1:8443). 
+If it everything looks good, check at the correct address 
+`https://www.example.com:8443`.
+The option `--debug` provides more verbose logging and might 
+be useful during testing.
  
 ### Firewall configuration
  
-Ports 80 and 443 are only usable by root. For security reasons it is better to 
-run the server as an unprivileged user on higher ports like 8080 for http and 
-8443 for https. For this, we can configure port forwarding in the firewall to 
-redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
+Ports 80 and 443 are only usable by root. For security reasons the server runs as an unprivileged user on port 8443 for https. 
+To access the server in the default https port (443), port forwarding can be configured in the firewall. 
  
 #### FreeBSD and pf
  
 Edit `/etc/pf.conf`:
  
 ```sh
-ext_if="em0"                # this should be the correct network interface
+ext_if="em0"                # change em0 to the correct network interface
 rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
 rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
 ```
@@ -212,6 +215,18 @@ pflog_logfile=&quot;/var/log/pflog&quot;
  
 Reboot or `sudo service pf start`.
  
+### Testing the system
+
+Make sure the following steps have been done:
+
+- installed python3.7, pip and npm
+- git-cloned the aprendizations from the main repository
+- installed javascript libraries with npm
+- installed aprendizations with pip
+- initialized database with at least 1 user
+- generate and copy certificates to the appropriate place
+- (optional) configure the firewall to do port forwarding
+- run `aprendizations demo.yaml --check`
  
 ## Troubleshooting
  
@@ -243,20 +258,23 @@ me:\
  
 ## FAQ
  
-- Which students did at least one topic?
+Common database manipulations:
  
 ```sh
-sqlite3 students.db "select distinct student_id from studenttopic"
+initdb-aprendizations -u 12345 --pw alibaba     # reset student password
+initdb-aprendizations -a 12345 --pw alibaba     # add new student
 ```
  
-- How many topics has each student done?
+Common database queries:
  
 ```sh
-sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
-```
+# Which students did at least one topic?
+sqlite3 students.db "select distinct student_id from studenttopic"
  
-- Which questions have more wrong answers?
+# How many topics has each student done?
+sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
  
-```sh
+# Which questions have more wrong answers?
 sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
-```
 \ No newline at end of file
+```
+
@@ -29,6 +29,10 @@ class LearnException(Exception):
     pass
  
  
+class DatabaseUnusableException(LearnException):
+    pass
+
+
 # ============================================================================
 # LearnApp - application logic
 # ============================================================================
@@ -46,6 +50,7 @@ class LearnApp(object):
         except Exception:
             logger.error('DB rollback!!!')
             session.rollback()
+            raise
         finally:
             session.close()
  
@@ -105,7 +110,7 @@ class LearnApp(object):
             logger.error(f'{errors:>6} errors found.')
             raise LearnException('Sanity checks')
         else:
-            logger.info('No errors found.')
+            logger.info('     0 errors found.')
  
     # ------------------------------------------------------------------------
     # login
@@ -274,9 +279,9 @@ class LearnApp(object):
                 n = s.query(Student).count()
                 m = s.query(Topic).count()
                 q = s.query(Answer).count()
-        except Exception as e:
-            logger.critical(f'Database "{db}" not usable!')
-            raise e
+        except Exception:
+            logger.error(f'Database "{db}" not usable!')
+            raise DatabaseUnusableException()
         else:
             logger.info(f'{n:6} students')
             logger.info(f'{m:6} topics')
@@ -21,9 +21,9 @@ import tornado.web
 from tornado.escape import to_unicode
  
 # this project
-from .learnapp import LearnApp
+from .learnapp import LearnApp, DatabaseUnusableException
 from .tools import load_yaml, md_to_html
-from . import APP_NAME
+from . import APP_NAME, APP_VERSION
  
  
 # ----------------------------------------------------------------------------
@@ -380,39 +380,44 @@ def signal_handler(signal, frame):
 # ----------------------------------------------------------------------------
 def parse_cmdline_arguments():
     argparser = argparse.ArgumentParser(
-        description='Server for online learning. Enrolled students and topics '
+        description='Server for online learning. Students and topics '
         'have to be previously configured. Please read the documentation '
         'included with this software before running the server.'
         )
  
     argparser.add_argument(
-        'conffile', type=str, nargs='+',
+        'conffile', type=str, nargs='*',
         help='Topics configuration file in YAML format.'
         )
  
     argparser.add_argument(
         '--prefix', type=str, default='.',
-        help='Path where the topic directories can be found, e.g. ~/topics'
+        help='Path where the topic directories can be found (default: .)'
         )
  
     argparser.add_argument(
         '--port', type=int, default=8443,
-        help='Port to be used by the HTTPS server, e.g. 8443'
+        help='Port to be used by the HTTPS server (default: 8443)'
         )
  
     argparser.add_argument(
         '--db', type=str, default='students.db',
-        help='SQLite3 database file, e.g. students.db'
+        help='SQLite3 database file (default: students.db)'
         )
  
     argparser.add_argument(
         '--check', action='store_true',
-        help='Sanity check all questions'
+        help='Sanity check questions (can take awhile)'
         )
  
     argparser.add_argument(
         '--debug', action='store_true',
-        help='Enable debug messages'
+        help='Enable debug mode'
+        )
+
+    argparser.add_argument(
+        '--version', action='store_true',
+        help='Print version information'
         )
  
     return argparser.parse_args()
@@ -434,8 +439,7 @@ def get_logger_config(debug=False):
         'version': 1,
         'formatters': {
             'standard': {
-                'format': '%(asctime)s %(name)-24s %(levelname)-10s - '
-                          '%(message)s',
+                'format': '%(asctime)s | %(levelname)-10s | %(message)s',
                 'datefmt': '%Y-%m-%d %H:%M:%S',
                 },
             },
@@ -472,6 +476,11 @@ def main():
     # --- Commandline argument parsing
     arg = parse_cmdline_arguments()
  
+    if arg.version:
+        print(APP_NAME + ' ' + APP_VERSION)
+        print('Python ' + sys.version)
+        sys.exit(0)
+
     # --- Setup logging
     logger_config = get_logger_config(arg.debug)
     logging.config.dictConfig(logger_config)
@@ -489,6 +498,17 @@ def main():
     try:
         learnapp = LearnApp(arg.conffile, prefix=arg.prefix, db=arg.db,
                             check=arg.check)
+    except DatabaseUnusableException:
+        logging.critical('Failed to start application.')
+        print('--------------------------------------------------------------')
+        print('Could not find a usable database. Use one of the follwing     ')
+        print('commands to initialize:                                       ')
+        print('                                                              ')
+        print('  initdb-aprendizations --admin            # add admin        ')
+        print('  initdb-aprendizations -a 86 "Max Smart"  # add student      ')
+        print('  initdb-aprendizations students.csv       # add many students')
+        print('--------------------------------------------------------------')
+        sys.exit(1)
     except Exception:
         logging.critical('Failed to start application.')
         sys.exit(1)
@@ -513,6 +533,19 @@ def main():
                                 path.join(certs_dir, 'privkey.pem'))
     except FileNotFoundError:
         logging.critical(f'SSL certificates missing in {certs_dir}')
+        print('--------------------------------------------------------------')
+        print('Certificates should be issued by a certificate authority (CA),')
+        print('such as https://letsencrypt.org, and then copied to:          ')
+        print('                                                              ')
+        print(f'  {certs_dir:<62}')
+        print('                                                              ')
+        print('For testing purposes a selfsigned certificate can be generated')
+        print('locally by running:                                           ')
+        print('                                                              ')
+        print('  openssl req -x509 -newkey rsa:4096 -keyout privkey.pem \\   ')
+        print('          -out cert.pem -days 365 -nodes                      ')
+        print('                                                              ')
+        print('--------------------------------------------------------------')
         sys.exit(1)
  
     # --- create webserver
...	...	@@ -3,7 +3,7 @@
3	3
4	4	- quando termina topico devia apagar as perguntas todas (se falhar a gerar novo topico, aparecem perguntas do antigo)
5	5	- apos clicar no botao responder, inactivar o input (importante quando o tempo de correcção é grande)
6		-- devia mostrar timout para o aluno saber a razao.
	6	+- devia mostrar timeout para o aluno saber a razao.
7	7	- permitir configuracao para escolher entre static files locais ou remotos
8	8	- sqlalchemy.pool.impl.NullPool: Exception during reset or similar
9	9	sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread.
...	...	@@ -40,6 +40,7 @@ sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in
40	40
41	41	# FIXED
42	42
	43	+- add aprendizatons --version
43	44	- se aluno abre dois tabs no browser, conseque navegar em simultaneo para perguntas diferentes. quando submete uma delas dá asneira. Tem de haver um campo hidden que tenha um céodigo único que indique qual a pergunta. do lado do servidor apnas há o codigo da pergunta corrente, se forem diferentes faz redirect para /.
44	45	- nos topicos learn.yaml, qd falha acrescenta no fim. nao faz sentido.
45	46	- não esta a fazer render correcto de tabelas nas opcoes checkbox. e.g. information-theory/source-coding-theory/block-codes
...	...
1	1	# Getting Started
2	2
	3	+To complete the installation we will need to perform the following steps:
3	4
4		-## Requirements
	5	+1. install python3.7, pip and npm
	6	+1. download aprendizations from the repository
	7	+1. install javascript libraries (with npm)
	8	+1. install aprendizations (with pip)
	9	+1. initialize database
	10	+1. generate SSL certificates
	11	+1. configure the firewall (optional)
	12	+1. try running `aprendizations demo.yaml`
5	13
6		-This application requires python3.7+ and a few additional python packages.
7		-It also uses npm (Node package management) to install javascript libraries.
	14	+These steps are explained next in detail.
8	15
9	16	### Install python3.7 with sqlite3 support and npm
10	17
11		-This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.
	18	+Python can be installed either from the system package management or compiled from sources.
12	19
13		-- Installing from the system package manager:
	20	+#### Installing from the system package manager
14	21
15	22	```sh
16		-sudo port install python37 npm5 # MacOS
	23	+sudo apt install python3.7 npm # Linux (Ubuntu)
17	24	sudo pkg install python37 py37-sqlite3 npm # FreeBSD
18		-sudo apt install python3.7 npm # Linux
	25	+sudo port install python37 npm6 # MacOS
19	26	```
20	27
21		-- Installing from source:
	28	+#### Installing from source
22	29
23		-Download from [http://www.python.org]() and
	30	+Make sure that the build tools and libraries are installed:
24	31
25	32	```sh
26		-unxz Python-3.7.tar.xz
27		-tar xvf Python-3.7.tar
	33	+# Ubuntu:
	34	+sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev
	35	+```
	36	+
	37	+Download python from [http://www.python.org]() and
	38	+
	39	+```sh
	40	+tar xvfJ Python-3.7.tar.xz
28	41	cd Python-3.7
29		-./configure --prefix=$HOME/.local/bin
	42	+./configure --prefix=$HOME/.local --enable-optimizations
30	43	make && make install
31	44	```
32	45
33		-This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` (edit `~/.profile` in MacOS or FreeBSD).
34		-
	46	+This will install python locally under `~/.local/bin`. Make sure to add it to your `PATH` in `~/.profile`. If `~/bin` is already in the path, you may just make a symbolic link `ln -s ~/.local/bin ~/bin`.
35	47
36	48	### Install pip
37	49
	50	+Python usually includes pip which is accessible through `python -m pip install something`, but it's also convenient to have the `pip` command installed.
38	51	If the `pip` command is not yet installed, run one of these:
39	52
40	53	```sh
41		-sudo apt install python3.7-pip # Ubuntu
	54	+sudo apt install python3.7-pip # Ubuntu 19.04+
	55	+python3.7 -m pip install pip # Ubuntu 18.04
42	56	sudo pkg py37-pip # FreeBSD
43	57	sudo port install py37-pip # MacOS
44		-python3.7 -m ensurepip --user # otherwise
45	58	```
46	59
47	60	The latter will install `pip` in your user account under `~/.local/bin`.
48		-In the end you should be able to run `pip3 --version` and
49		-`python3 -c "import sqlite3"` without errors (sometimes `pip3` is `pip`,
50		-`pip3.7` or `pip-3.7`).
	61	+In the end you should be able to run `pip --version` and
	62	+`python3 -c "import sqlite3"` without errors.
	63	+Sometimes the `pip` command is named `pip3`,
	64	+`pip3.7` or `pip-3.7`.
51	65
52		-If you want to always install python modules on the user account (recommended),
53		-edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or
	66	+Edit the configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or
54	67	`Library/Application Support/pip/pip.conf` (MacOS) and add the lines
55	68
56	69	```ini
...	...	@@ -58,69 +71,72 @@ edit the pip configuration file `~/.config/pip/pip.conf` (FreeBSD, Linux) or
58	71	user = yes
59	72	```
60	73
61		-### Install python packages and javascript libraries:
	74	+This will set pip to install modules in the user area (recommended).
62	75
63		-Replace USER by your bitbucket username:
	76	+### Install aprendizations and dependencies:
64	77
65	78	```sh
66		-cd somewhere
67	79	git clone https://git.xdi.uevora.pt/mjsb/aprendizations.git
68	80	cd aprendizations
69	81	npm install # install javascript libraries
70	82	pip install . # install aprendizations and dependencies
71	83	```
72	84
	85	+Javascript libraries are initially installed in `aprendizations/node_modules` directory.
	86	+These libraries already have symbolic links from `aprendizations/aprendizations/static`.
	87	+
73	88	Python packages are usually installed in:
74	89
75	90	- `~/.local/lib/python3.7/site-packages/` in Linux/FreeBSD.
76	91	- `~/Library/python/3.7/lib/python/site-packages/` in MacOS.
77	92
78		-Javascript libraries are installed in `aprendizations/node_modules` directory.
79		-This libraries have symbolic links from `aprendizations/aprendizations/static`.
	93	+When aprendizations is installed with pip, all the dependencies are also installed. The javascript libraries previously installed with npm are copied to the above directory and the cloned repository is no longer needed.
80	94
81		-At this point aprendizations is installed in
	95	+At this point, aprendizations is installed in
82	96
83	97	```sh
	98	+~/.local/bin # Linux/FreeBSD
84	99	~/Library/Python/3.7/bin # MacOS
85		-~/.local/bin # FreeBSD/Linux
86	100	```
87	101
88		-Make sure this directory is in your `$PATH`.
	102	+and can be run from the terminal:
89	103
90		-The server can be run with the command `aprendizations` from the terminal.
	104	+```sh
	105	+aprendizations --version
	106	+aprendizations --help
	107	+```
91	108
92	109	## Configuration
93	110
94	111	### Database
95	112
96		-The user data is maintained in a sqlite3 database file. We first need to create
97		-the database.
98		-At the moment, the database should be located in the same directory as the main
99		-configuration file (see below). As an example, do
	113	+User data is maintained in a sqlite3 database which has to be created manually using the command `initdb-aprendizations`.
	114	+The database file should be located in the same directory as the main
	115	+YAML configuration file.
	116	+
	117	+For example, to run the included demo do:
100	118
101	119	```sh
102	120	cd demo # contains a small example
103	121	initdb-aprendizations # show or initialize database
104	122	initdb-aprendizations --admin # add admin user
105	123	initdb-aprendizations inscricoes.csv # add students from CSV
106		-initdb-aprendizations --add 1184 "Aladino da Silva" # add user
	124	+initdb-aprendizations --add 1184 "Aladino da Silva" # add new user
107	125	initdb-aprendizations --update 1184 --pw alibaba # update password
108		-initdb-aprendizations --help # for the available options
	126	+initdb-aprendizations --help # for available options
109	127	```
110	128
111		-The default password is equal to the user name used to login.
	129	+The default password is equal to the user name, if left undefined.
112	130
113	131
114	132	### SSL Certificates
115	133
116		-We need certificates for https. Certificates can be self-signed or validated by
117		-a trusted authority.
	134	+We need certificates for https. Certificates can be self-signed or validated by a trusted authority.
118	135
119	136	Self-signed can be used locally for development and testing, but browsers will
120		-complain. LetsEncrypt issues trusted and free certificates, but the server must
121		-have a registered publicly accessible domain name.
	137	+complain. LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.
122	138
123		-#### Selfsigned
	139	+#### Generating selfsigned certificates
124	140
125	141	Generate a selfsigned certificate and place it in `~/.local/share/certs`.
126	142
...	...	@@ -128,18 +144,20 @@ Generate a selfsigned certificate and place it in `~/.local/share/certs`.
128	144	openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
129	145	```
130	146
131		-#### LetsEncrypt
	147	+#### LetsEncrypt certificates
	148	+
	149	+Install the certbot from LetsEncrypt:
132	150
133	151	```sh
134		-sudo pkg install py27-certbot # FreeBSD
	152	+sudo pkg install py36-certbot # FreeBSD
	153	+sudo apt install certbot # Linux Ubuntu
135	154	```
136	155
137		-Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:
	156	+To generate or renew the certificates, ports 80 and 443 have to be accessible. The firewall and webserver have to be stopped.
138	157
139	158	```sh
140		-sudo service pf stop # disable pf firewall (FreeBSD)
141		-sudo certbot certonly --standalone -d www.example.com
142		-sudo service pf start # enable pf firewall
	159	+sudo certbot certonly --standalone -d www.example.com # first time
	160	+sudo certbot renew # renew
143	161	```
144	162
145	163	Certificates are saved under `/usr/local/etc/letsencrypt/live/www.example.com/`. Copy them to `~/.local/share/certs` and change permissions to be readable:
...	...	@@ -150,47 +168,32 @@ sudo cp /usr/local/etc/letsencrypt/live/www.example.com/privkey.pem .
150	168	chmod 400 cert.pem privkey.pem
151	169	```
152	170
153		-Renews can be done as follows:
154		-
155		-```sh
156		-sudo service pf stop # shutdown firewall
157		-sudo certbot renew
158		-sudo service pf start # start firewall
159		-```
160		-
161		-and then copy the `cert.pem` and `privkey.pem` files to `~/.local/share/certs` directory. Change permissions and ownership as appropriate.
	171	+### Running the demo
162	172
163		-
164		-### Testing
165		-
166		-The application includes a small example in `demo/demo.yaml`. Run it with
	173	+The application includes a small example in `demo/demo.yaml` that can be used for initial testing. Run it with
167	174
168	175	```sh
169	176	cd demo
170	177	aprendizations demo.yaml
171	178	```
172	179
173		-and open a browser at [https://127.0.0.1:8443](https://127.0.0.1:8443).
174		-If it everything looks good,
175		-check at the correct address `https://www.example.com` (requires port forward
176		-in the firewall). The option `--debug` provides more verbose logging and might
177		-be useful during testing. The option `--check` generates all the questions once
178		-before running the server to check for any obvious syntax error.
179		-
	180	+Open the browser at [https://127.0.0.1:8443](https://127.0.0.1:8443).
	181	+If it everything looks good, check at the correct address
	182	+`https://www.example.com:8443`.
	183	+The option `--debug` provides more verbose logging and might
	184	+be useful during testing.
180	185
181	186	### Firewall configuration
182	187
183		-Ports 80 and 443 are only usable by root. For security reasons it is better to
184		-run the server as an unprivileged user on higher ports like 8080 for http and
185		-8443 for https. For this, we can configure port forwarding in the firewall to
186		-redirect incoming tcp traffic from 80 to 8080 and 443 to 8443.
	188	+Ports 80 and 443 are only usable by root. For security reasons the server runs as an unprivileged user on port 8443 for https.
	189	+To access the server in the default https port (443), port forwarding can be configured in the firewall.
187	190
188	191	#### FreeBSD and pf
189	192
190	193	Edit `/etc/pf.conf`:
191	194
192	195	```sh
193		-ext_if="em0" # this should be the correct network interface
	196	+ext_if="em0" # change em0 to the correct network interface
194	197	rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
195	198	rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
196	199	```
...	...	@@ -212,6 +215,18 @@ pflog_logfile="/var/log/pflog"
212	215
213	216	Reboot or `sudo service pf start`.
214	217
	218	+### Testing the system
	219	+
	220	+Make sure the following steps have been done:
	221	+
	222	+- installed python3.7, pip and npm
	223	+- git-cloned the aprendizations from the main repository
	224	+- installed javascript libraries with npm
	225	+- installed aprendizations with pip
	226	+- initialized database with at least 1 user
	227	+- generate and copy certificates to the appropriate place
	228	+- (optional) configure the firewall to do port forwarding
	229	+- run `aprendizations demo.yaml --check`
215	230
216	231	## Troubleshooting
217	232
...	...	@@ -243,20 +258,23 @@ me:\
243	258
244	259	## FAQ
245	260
246		-- Which students did at least one topic?
	261	+Common database manipulations:
247	262
248	263	```sh
249		-sqlite3 students.db "select distinct student_id from studenttopic"
	264	+initdb-aprendizations -u 12345 --pw alibaba # reset student password
	265	+initdb-aprendizations -a 12345 --pw alibaba # add new student
250	266	```
251	267
252		-- How many topics has each student done?
	268	+Common database queries:
253	269
254	270	```sh
255		-sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
256		-```
	271	+# Which students did at least one topic?
	272	+sqlite3 students.db "select distinct student_id from studenttopic"
257	273
258		-- Which questions have more wrong answers?
	274	+# How many topics has each student done?
	275	+sqlite3 students.db "select student_id, count(topic_id) from studenttopic group by student_id order by count(topic_id) desc"
259	276
260		-```sh
	277	+# Which questions have more wrong answers?
261	278	sqlite3 students.db "select count(ref), ref from answers where grade<1.0 group by ref order by count(ref) desc"
262		-```
263	279	\ No newline at end of file
	280	+```
	281	+
...	...
...	...	@@ -29,6 +29,10 @@ class LearnException(Exception):
29	29	pass
30	30
31	31
	32	+class DatabaseUnusableException(LearnException):
	33	+ pass
	34	+
	35	+
32	36	# ============================================================================
33	37	# LearnApp - application logic
34	38	# ============================================================================
...	...	@@ -46,6 +50,7 @@ class LearnApp(object):
46	50	except Exception:
47	51	logger.error('DB rollback!!!')
48	52	session.rollback()
	53	+ raise
49	54	finally:
50	55	session.close()
51	56
...	...	@@ -105,7 +110,7 @@ class LearnApp(object):
105	110	logger.error(f'{errors:>6} errors found.')
106	111	raise LearnException('Sanity checks')
107	112	else:
108		- logger.info('No errors found.')
	113	+ logger.info(' 0 errors found.')
109	114
110	115	# ------------------------------------------------------------------------
111	116	# login
...	...	@@ -274,9 +279,9 @@ class LearnApp(object):
274	279	n = s.query(Student).count()
275	280	m = s.query(Topic).count()
276	281	q = s.query(Answer).count()
277		- except Exception as e:
278		- logger.critical(f'Database "{db}" not usable!')
279		- raise e
	282	+ except Exception:
	283	+ logger.error(f'Database "{db}" not usable!')
	284	+ raise DatabaseUnusableException()
280	285	else:
281	286	logger.info(f'{n:6} students')
282	287	logger.info(f'{m:6} topics')
...	...
...	...	@@ -21,9 +21,9 @@ import tornado.web
21	21	from tornado.escape import to_unicode
22	22
23	23	# this project
24		-from .learnapp import LearnApp
	24	+from .learnapp import LearnApp, DatabaseUnusableException
25	25	from .tools import load_yaml, md_to_html
26		-from . import APP_NAME
	26	+from . import APP_NAME, APP_VERSION
27	27
28	28
29	29	# ----------------------------------------------------------------------------
...	...	@@ -380,39 +380,44 @@ def signal_handler(signal, frame):
380	380	# ----------------------------------------------------------------------------
381	381	def parse_cmdline_arguments():
382	382	argparser = argparse.ArgumentParser(
383		- description='Server for online learning. Enrolled students and topics '
	383	+ description='Server for online learning. Students and topics '
384	384	'have to be previously configured. Please read the documentation '
385	385	'included with this software before running the server.'
386	386	)
387	387
388	388	argparser.add_argument(
389		- 'conffile', type=str, nargs='+',
	389	+ 'conffile', type=str, nargs='*',
390	390	help='Topics configuration file in YAML format.'
391	391	)
392	392
393	393	argparser.add_argument(
394	394	'--prefix', type=str, default='.',
395		- help='Path where the topic directories can be found, e.g. ~/topics'
	395	+ help='Path where the topic directories can be found (default: .)'
396	396	)
397	397
398	398	argparser.add_argument(
399	399	'--port', type=int, default=8443,
400		- help='Port to be used by the HTTPS server, e.g. 8443'
	400	+ help='Port to be used by the HTTPS server (default: 8443)'
401	401	)
402	402
403	403	argparser.add_argument(
404	404	'--db', type=str, default='students.db',
405		- help='SQLite3 database file, e.g. students.db'
	405	+ help='SQLite3 database file (default: students.db)'
406	406	)
407	407
408	408	argparser.add_argument(
409	409	'--check', action='store_true',
410		- help='Sanity check all questions'
	410	+ help='Sanity check questions (can take awhile)'
411	411	)
412	412
413	413	argparser.add_argument(
414	414	'--debug', action='store_true',
415		- help='Enable debug messages'
	415	+ help='Enable debug mode'
	416	+ )
	417	+
	418	+ argparser.add_argument(
	419	+ '--version', action='store_true',
	420	+ help='Print version information'
416	421	)
417	422
418	423	return argparser.parse_args()
...	...	@@ -434,8 +439,7 @@ def get_logger_config(debug=False):
434	439	'version': 1,
435	440	'formatters': {
436	441	'standard': {
437		- 'format': '%(asctime)s %(name)-24s %(levelname)-10s - '
438		- '%(message)s',
	442	+ 'format': '%(asctime)s \| %(levelname)-10s \| %(message)s',
439	443	'datefmt': '%Y-%m-%d %H:%M:%S',
440	444	},
441	445	},
...	...	@@ -472,6 +476,11 @@ def main():
472	476	# --- Commandline argument parsing
473	477	arg = parse_cmdline_arguments()
474	478
	479	+ if arg.version:
	480	+ print(APP_NAME + ' ' + APP_VERSION)
	481	+ print('Python ' + sys.version)
	482	+ sys.exit(0)
	483	+
475	484	# --- Setup logging
476	485	logger_config = get_logger_config(arg.debug)
477	486	logging.config.dictConfig(logger_config)
...	...	@@ -489,6 +498,17 @@ def main():
489	498	try:
490	499	learnapp = LearnApp(arg.conffile, prefix=arg.prefix, db=arg.db,
491	500	check=arg.check)
	501	+ except DatabaseUnusableException:
	502	+ logging.critical('Failed to start application.')
	503	+ print('--------------------------------------------------------------')
	504	+ print('Could not find a usable database. Use one of the follwing ')
	505	+ print('commands to initialize: ')
	506	+ print(' ')
	507	+ print(' initdb-aprendizations --admin # add admin ')
	508	+ print(' initdb-aprendizations -a 86 "Max Smart" # add student ')
	509	+ print(' initdb-aprendizations students.csv # add many students')
	510	+ print('--------------------------------------------------------------')
	511	+ sys.exit(1)
492	512	except Exception:
493	513	logging.critical('Failed to start application.')
494	514	sys.exit(1)
...	...	@@ -513,6 +533,19 @@ def main():
513	533	path.join(certs_dir, 'privkey.pem'))
514	534	except FileNotFoundError:
515	535	logging.critical(f'SSL certificates missing in {certs_dir}')
	536	+ print('--------------------------------------------------------------')
	537	+ print('Certificates should be issued by a certificate authority (CA),')
	538	+ print('such as https://letsencrypt.org, and then copied to: ')
	539	+ print(' ')
	540	+ print(f' {certs_dir:<62}')
	541	+ print(' ')
	542	+ print('For testing purposes a selfsigned certificate can be generated')
	543	+ print('locally by running: ')
	544	+ print(' ')
	545	+ print(' openssl req -x509 -newkey rsa:4096 -keyout privkey.pem \\ ')
	546	+ print(' -out cert.pem -days 365 -nodes ')
	547	+ print(' ')
	548	+ print('--------------------------------------------------------------')
516	549	sys.exit(1)
517	550
518	551	# --- create webserver
...	...