Merge branch 'dev'

Miguel Barão
2 parents f6fdbdbd a0ab1672
Showing 30 changed files with 1174 additions and 1520 deletions Show diff stats
BUGS.md
README.md
demo/demo.yaml
demo/questions/generate-question.py
demo/questions/generators/generate-question.py
demo/questions/questions-tutorial.yaml
mypy.ini
package.json
perguntations/__init__.py
perguntations/app.py
perguntations/initdb.py
perguntations/main.py
perguntations/models.py
perguntations/parser_markdown.py
perguntations/questions.py
perguntations/serve.py
perguntations/static/js/admin.js
perguntations/templates/admin.html
perguntations/templates/grade.html
perguntations/templates/question-information.html
 # BUGS
-- show_points nao esta a funcionar (mostra sempre)
-- esta a corrigir código JOBE mesmo que nao tenha respondido???
-- correct devia poder ser corrido mais que uma vez (por exemplo para alterar
-  cotacoes, corrigir perguntas)
-- nao esta a mostrar imagens?? internal server error?
-- guardar testes em JSON assim que sao atribuidos aos alunos (ou guardados
-  inicialmente com um certo nome, e atribuidos posteriormente ao aluno).
-- cookies existe um `perguntations_user` e um user. De onde vem o user?
-- QuestionCode falta reportar nos comments os vários erros que podem ocorrer
-  (timeout, etc)
-- algumas vezes a base de dados guarda o mesmo teste em duplicado. ver se dois
-  submits dao origem a duas correcções.  talvez a base de dados devesse ter
-  como chave do teste um id que fosse único desse teste particular (não um auto
-  counter, nem ref do teste)
-- em caso de timeout na submissão (e.g. JOBE ou script nao responde) a
-  correcção não termina e o teste não é guardado.
+- correct devia poder ser corrido mais que uma vez (por exemplo para alterar cotacoes, corrigir perguntas)
+- guardar testes em JSON assim que sao atribuidos aos alunos (ou guardados inicialmente com um certo nome, e atribuidos posteriormente ao aluno).
+- cookies existe um perguntations_user e um user. De onde vem o user?
+- QuestionCode falta reportar nos comments os vários erros que podem ocorrer (timeout, etc)
+- algumas vezes a base de dados guarda o mesmo teste em duplicado. ver se dois submits dao origem a duas correcções.
+talvez a base de dados devesse ter como chave do teste um id que fosse único desse teste particular (não um auto counter, nem ref do teste)
+- em caso de timeout na submissão (e.g. JOBE ou script nao responde) a correcção não termina e o teste não é guardado.
 - grade gives internal server error??
 - reload do teste recomeça a contagem no inicio do tempo.
-- em admin, quando `scale_max` não é 20, as cores das barras continuam a
-  reflectir a escala 0,20. a tabela teste na DB não tem a escala desse teste.
-- em grade.html as barras estao normalizadas para os limites `scale_min` e max
-  do teste actual e nao dos testes realizados no passado (tabela test devia
-      guardar a escala).
-- codigo `hello   world` nao esta a preservar o whitespace. O renderer de
-  markdown gera a tag `<code>` que não preserva whitespace. Necessario
-  adicionar `<pre>.`
-- mensagems de erro do assembler aparecem na mesma linha na correcao e nao
-  fazerm rendering do `$t`, ver se servidor faz parse do markdown dessas
-  mensagens.
+- em admin, quando scale_max não é 20, as cores das barras continuam a reflectir a escala 0,20. a tabela teste na DB não tem a escala desse teste.
+- em grade.html as barras estao normalizadas para os limites scale_min e max do teste actual e nao dos testes realizados no passado (tabela test devia guardar a escala).
+- codigo `hello   world` nao esta a preservar o whitespace. O renderer de markdown gera a tag <code> que não preserva whitespace. Necessario adicionar <pre>.
+- mensagems de erro do assembler aparecem na mesma linha na correcao e nao fazerm rendering do `$t`, ver se servidor faz parse do markdown dessas mensagens.
 - a revisao do teste não mostra as imagens.
-- `Test.reset_answers()` unused.
+- Test.reset_answers() unused.
 - teste nao esta a mostrar imagens de vez em quando.???
 - show-ref nao esta a funcionar na correccao (pelo menos)
 # TODO
-- permitir que textarea nao tenha correct definido, da sempre 0 e é para
-  corrigir manualmente.
-- permitir corrigir mais que uma vez, actualizando a base de dados.
-- yaml schemas para validar os ficheiros yaml (ver cerberus e jsonschema)
+- JOBE correct async
+- esta a corrigir código JOBE mesmo que nao tenha respondido???
 - permitir remover alunos que estão online para poderem comecar de novo.
-- guardar nota final grade truncado em zero e sem ser truncado (quando é
-  necessário fazer correcções à mão às perguntas, é necessário o valor não
-  truncado)
-- stress tests. use [locust](https://locust.io)
+- guardar nota final grade truncado em zero e sem ser truncado (quando é necessário fazer correcções à mão às perguntas, é necessário o valor não truncado)
+- stress tests. use https://locust.io
 - wait for admin to start test. (students can be allowed earlier)
-- impedir os eventos copy/paste. alunos usam isso para trazer codigo ja feito
-  nos computadores. Obrigar a fazer reset? fazer um copy automaticamente?
-- na pagina admin, mostrar com cor vermelha as horas de entrada dos alunos que
-  ja tenham excedido o tempo
-- retornar None quando nao ha alteracoes relativamente à última vez.  ou usar
-  push (websockets?)
-- mudar ref do test para `test_id` (ref já é usado nas perguntas)
+- impedir os eventos copy/paste. alunos usam isso para trazer codigo ja feito nos computadores. Obrigar a fazer reset? fazer um copy automaticamente?
+- na pagina admin, mostrar com cor vermelha as horas de entrada dos alunos que ja tenham excedido o tempo
+- retornar None quando nao ha alteracoes relativamente à última vez.
+ou usar push (websockets?)
+- mudar ref do test para test_id (ref já é usado nas perguntas)
 - servidor ntpd no x220 para configurar a data/hora dos portateis dell
-- sala de espera: autorização dada, mas teste não disponível até que seja dada
-  ordem para começar.
-- alunos com necessidades especiais nao podem ter autosubmit. ter um
-  `autosubmit_exceptions`: ['123', '456']
+- sala de espera: autorização dada, mas teste não disponível até que seja dada ordem para começar.
+- alunos com necessidades especiais nao podem ter autosubmit. ter um autosubmit_exceptions: ['123', '456']
 - submissao fazer um post ajax?
 - adicionar opcao para eliminar um teste em curso.
 - enviar resposta de cada pergunta individualmente.
-- experimentar gerador de svg que inclua no markdown da pergunta e ver se
-  funciona.
-- quando ha varias perguntas para escolher, escolher sucessivamente em vez de
-  aleatoriamente.
+- experimentar gerador de svg que inclua no markdown da pergunta e ver se funciona.
+- quando ha varias perguntas para escolher, escolher sucessivamente em vez de aleatoriamente.
 - como refrescar a tabela de admin sem fazer reload da pagina?
-- botao "testar resposta" que valida codigo relativamente a syntax, mas nao
-  classifica. perguntas devem ter opcao validate: script.py. Aluno pressiona
-  botao e codigo é enviado para servidor para validação, feedback é mostrado na
-  pagina de teste.
-- test: botao submeter valida se esta online com um post `willing_to_submit`,
-  se estiver online, mostra mensagem de confirmacao, caso contrario avisa que
-  nao esta online.
+- botao "testar resposta" que valida codigo relativamente a syntax, mas nao classifica. perguntas devem ter opcao validate: script.py. Aluno pressiona botao e codigo é enviado para servidor para validação, feedback é mostrado na pagina de teste.
+- test: botao submeter valida se esta online com um post willing_to_submit, se estiver online, mostra mensagem de confirmacao, caso contrario avisa que nao esta online.
 - test: Cada pergunta respondida é logo submetida.
 - test: calculadora javascript.
 - admin: histograma das notas.
 - admin: mostrar as horas a que o teste terminou para os testes terminados.
 - admin: histograma das notas.
 - admin: mostrar teste gerado para aluno (tipo review).
-- fazer renderer para formulas com mathjax serverside (mathjax-node) ou usar
-  katex.
+- fazer renderer para formulas com mathjax serverside (mathjax-node) ou usar katex.
 - fazer renderer para imagens, com links /file?ref=xpto;name=zzz.jpg
 - fazer renderer para linguagem assembly mips?
-- cancelar teste no menu admin. Dado o numero de aluno remove teste e faz
-  logout do aluno.
+- cancelar teste no menu admin. Dado o numero de aluno remove teste e faz logout do aluno.
+- mathjax-node:
+    sudo pkg install node npm
+    npm install mathjax-node mathjax-node-cli    # pacotes em ~/node_modules
+    node_modules/mathjax-node-cli/bin/tex2svg '\sqrt{x}'
+  usar isto para gerar svg que passa a fazer parte do texto da pergunta (markdown suporta tags svg?)
+  fazer funçao tex() que recebe formula e converte para svg. exemplo:
+  fr'''A formula é {tex("\sqrt{x]}")}'''
 - Gerar pdf's com todos os testes no final (pdfkit).
-- manter registo dos unfocus durante o teste e de qual a pergunta visivel nesse
-  momento
+- manter registo dos unfocus durante o teste e de qual a pergunta visivel nesse momento
 - permitir varios testes, aluno escolhe qual o teste que quer fazer.
 - se ocorrer um erro na correcçao avisar aluno para contactar o professor.
-- abrir o teste numa janela maximizada e que nao permite que o aluno a
-  redimensione/mova?
-- detectar scroll e enviar posição para servidor (analise de scroll para
-  detectar copianço? ou simplesmente para analisar como os alunos percorrem o
-  teste)
-- aviso na pagina principal para quem usa browser incompativel ou settings
-  esquisitos... Apos login pode ser enviado e submetido um exemplo de teste
-  para verificar se browser consegue submeter? ha alunos com javascript
-  bloqueado?
+- abrir o teste numa janela maximizada e que nao permite que o aluno a redimensione/mova?
+- detectar scroll e enviar posição para servidor (analise de scroll para detectar copianço? ou simplesmente para analisar como os alunos percorrem o teste)
+- aviso na pagina principal para quem usa browser incompativel ou settings esquisitos... Apos login pode ser enviado e submetido um exemplo de teste para verificar se browser consegue submeter? ha alunos com javascript bloqueado?
 - criar perguntas de outros tipos, e.g. associação, ordenação, varios textinput
-- perguntas para professor corrigir mais tarde. permitir que review possa
-  alterar as notas
+- perguntas para professor corrigir mais tarde. permitir que review possa alterar as notas
 - fazer uma calculadora javascript e por no menu. surge como modal
 # FIXED
-- JOBE correct async
 - testar as perguntas todas no início do teste como o aprendizations.
-- adicionar identificacao do aluno no jumbotron inicial do teste, para que ao
-  imprimir para pdf a identificacao do aluno venha escrita no documento.
+- adicionar identificacao do aluno no jumbotron inicial do teste, para que ao imprimir para pdf a identificacao do aluno venha escrita no documento.
 - internal server error quando em --review, download csv detalhado.
-- perguntas repetidas (mesma ref) dao asneira, porque a referencia é usada como
-  chave em varios sitios e as chaves nao podem ser dupplicadas.  da asneira
-  pelo menos na funcao `get_questions_csv`. na base de dados tem de estar
-  registado tb o numero da pergunta, caso contrario é impossível saber a qual
-  corresponde.
+- perguntas repetidas (mesma ref) dao asneira, porque a referencia é usada como chave em varios sitios e as chaves nao podem ser dupplicadas.
+  da asneira pelo menos na funcao get_questions_csv. na base de dados tem de estar registado tb o numero da pergunta, caso contrario é impossível saber a qual corresponde.
 - mostrar unfocus e window area em /admin
-- CRITICAL se answer for `i<n` a revisão de provas mostra apenas i (interpreta
-  `<` como tag?)
+- CRITICAL se answer for `i<n` a revisão de provas mostra apenas i (interpreta `<` como tag?)
 - botao de autorizar desliga-se, fazer debounce.
 - link na pagina com a nota para voltar ao principio.
 - default logger config mostrar horas com segundos
@@ -128,45 +88,33 @@
 - servidor nao esta a lidar com eventos resize.
 - sock.bind(sockaddr)  OSError: [Errno 48] Address already in use
 - dizer quanto desconta em cada pergunta de escolha multipla
-- se houver erros a abrir ficheiros .yaml de perguntas, depois dos testes diz
-  "No errors found".
-- se faltarem files na especificação do teste, o check não detecta e factory
-  não gera para essas perguntas.
+- se houver erros a abrir ficheiros .yaml de perguntas, depois dos testes diz "No errors found".
+- se faltarem files na especificação do teste, o check não detecta e factory não gera para essas perguntas.
 - nao esta a usar points das perguntas
 - quando se clica no texto de uma opcao, salta para outro lado na pagina.
-- suportar cotacao to teste diferente de 20 (e.g. para juntar perguntas em
-  papel). opcao "points: 18" que normaliza total para 18 em vez de 20.
+- suportar cotacao to teste diferente de 20 (e.g. para juntar perguntas em papel). opcao "points: 18" que normaliza total para 18 em vez de 20.
 - fazer package para instalar perguntations com pip.
 - pymips: nao pode executar syscalls do spim.
 - exception sqlalchemy relacionada com threads.
 - acrescentar logger.conf que sirva de base.
-- questions.py textarea has a abspath which does not make sense! why is it
-  there? not working for perguntations, but seems to work for aprendizations
-- textarea foi modificado em aprendizations para receber cmd line args.
-  corrigir aqui tb.
+- questions.py textarea has a abspath which does not make sense! why is it there? not working for perguntations, but seems to work for aprendizations
+- textarea foi modificado em aprendizations para receber cmd line args. corrigir aqui tb.
 - usar npm para instalar javascript
 - se aluno entrar com l12345 rebenta. numa funcao get_... ver imagem no ipad
-- no test3 está contar 1.0 valores numa pergunta do tipo info? acontece para
-  type: info, e não para type: information
-- default correct in checkbox must be 1.0, so that the pairs (right,wrong)
-  still work with `correct` left undefined.
+- no test3 está contar 1.0 valores numa pergunta do tipo info? acontece para type: info, e não para type: information
+- default correct in checkbox must be 1.0, so that the pairs (right,wrong) still work with `correct` left undefined.
 - textarea com codemirror
 - decorador para user 0, evita o "if uid==0" em muitas funcoes.
-- numeric interval deve converter respostas que usam virgulas para pontos
-  decimais
-- `self.testapp.get_json_filename_of_test(test_id)` retorna None quando test_id
-  nao existe.
-- o eventloop está a bloquear. correção do teste é blocking. usar
-  threadpoolexecutor?
-- substituir get_event_loop por get_runnint_loop
+- numeric interval deve converter respostas que usam virgulas para pontos decimais
+- self.testapp.get_json_filename_of_test(test_id) retorna None quando test_id nao existe.
+- o eventloop está a bloquear. correção do teste é blocking. usar threadpoolexecutor?
+- substituir get_event_loop por get_runnint_loop (ver https://docs.python.org/3/library/asyncio-eventloop.html)
 - review nao esta a funcionar
-- servir imagens das perguntas não funciona. Necessario passar a ref da
-  pergunta no link para poder ajustar o path no FileHandler.
+- servir imagens das perguntas não funciona. Necessario passar a ref da pergunta no link para poder ajustar o path no FileHandler.
 - a primeira coluna da tabela admin deveria estar sempre ordenada.
 - abortar depois de testar todas as perguntas, caso haja algum erro.
 - imagens jpg/png nas perguntas.
-- initdb está a inicializar com passwords iguais aos numeros. deveria ser vazio
-  para alunos definirem.
+- initdb está a inicializar com passwords iguais aos numeros. deveria ser vazio para alunos definirem.
 - upgrade popper e fazer link.
 - mover scripts js para head, com defer. ver todos os templates.
 - update fontawesome to 5.
@@ -176,8 +124,7 @@
 - md_to_html() nao usa o segundo argumento q. pode retirar-se dos templates?
 - config/logger.yaml ainda é do cherrypy...
 - uniformizar question.py com a de aprendizations...
-- qual a diferenca entre md_to_html e md_to_html_review, parece desnecessario
-  haver dois.
+- qual a diferenca entre md_to_html e md_to_html_review, parece desnecessario haver dois.
 - converter markdown para mistune.
 - como alterar configuracao para mostrar logs de debug?
 - espaco no final das tabelas.
@@ -193,43 +140,31 @@
 - text-numeric não está a gerar a pergunta. faltam templates?
 - testar perguntas warning/warn
 - qd user 0 faz logout rebenta.
-- Quando grava JSON do teste deve usar 'path' tal como definido na configuração
-  e não expandido. Isto porque em OSX /home é /Users e quando se muda de um
-  sistema para outro não encontra os testes. Assim, usando ~ na configuração
-  deveria funcionar sempre.
-- configuração do teste não joga bem com o do aprendizations. Em particular os
-  scripts não ficam com o mesmo path!!!
+- Quando grava JSON do teste deve usar 'path' tal como definido na configuração e não expandido. Isto porque em OSX /home é /Users e quando se muda de um sistema para outro não encontra os testes. Assim, usando ~ na configuração deveria funcionar sempre.
+- configuração do teste não joga bem com o do aprendizations. Em particular os scripts não ficam com o mesmo path!!!
 - configurar pf em freebsd, port forward 80 -> 8080. documentacao
 - barras com notas em grade estão desalinhadas.
 - erros nos generators devem ser ERROR e não WARNING.
-- se directorio "logs" não existir no directorio actual aborta com mensagem de
-  erro.
-- se um teste tiver a mesma pergunta repetida (ref igual), rebenta na
-  correcçao. As respostas são agregadas numa lista para cada ref. Ex: {'ref1':
-  'resposta1', 'ref2': ['resposta2a', 'resposta2b']}
-- usar [](http://fontawesome.io/examples/) em vez dos do bootstrap3
+- se directorio "logs" não existir no directorio actual aborta com mensagem de erro.
+- se um teste tiver a mesma pergunta repetida (ref igual), rebenta na correcçao. As respostas são agregadas numa lista para cada ref. Ex: {'ref1': 'resposta1', 'ref2': ['resposta2a', 'resposta2b']}
+- usar http://fontawesome.io/examples/ em vez dos do bootstrap3
 - se pergunta tiver 'type:' errado, rebenta.
 - se submeter um teste so com information, da divisao por zero.
-- se save_answers nao existir, da warning que nao serao gravados, mas sao
-  sempre gravados! pagina de administracao diz --not being saved--
+- se save_answers nao existir, da warning que nao serao gravados, mas sao sempre gravados! pagina de administracao diz --not being saved--
 - first login é INFO e não WARNING
 - /review não mostra imagens porque precisa que teste esteja a decorrer...
 - visualizar um teste ja realizado na página de administração
-- Depois da correcção, mostra testes realizados que não foram realizados pelo
-  próprio
-- detectar se janela perde focus e alertar o prof
-- server nao esta a receber eventos focus/blur dos utilizadores diferentes de
-  '0', estranho...
+- Depois da correcção, mostra testes realizados que não foram realizados pelo próprio
+- detectar se janela perde focus e alertar o prof (http://stackoverflow.com/questions/1060008/is-there-a-way-to-detect-if-a-browser-window-is-not-currently-active)
+- server nao esta a receber eventos focus/blur dos utilizadores diferentes de '0', estranho...
 - permitir adicionar imagens nas perguntas.
 - detect_unfocus.js so funciona se estiver inline no html. porquê???
 - inserir novo aluno /admin não fecha.
 - se aluno desistir, ainda fica marcado como online
 - give dá None em vez de 0.0
 - debug mode: log levels not working
-- Se aluno fizer logout, o teste não é gravado e ficamos sem registo do teste
-  que o aluno viu.
-- criar sqlalchemy sessions dentro de app de modo a estarem associadas a
-  requests. ver se é facil usar with db:(...) para criar e fechar sessão.
+- Se aluno fizer logout, o teste não é gravado e ficamos sem registo do teste que o aluno viu.
+- criar sqlalchemy sessions dentro de app de modo a estarem associadas a requests. ver se é facil usar with db:(...) para criar e fechar sessão.
 - sqlalchemy queixa-se de threads.
 - SQLAlchemy em vez da classe database.
 - replace sys.exit calls
@@ -238,21 +173,20 @@
 - configuracao dos logs cherrypy para se darem bem com os outros
 - browser e ip usados gravado no test.
 - botões allow all/deny all.
-- mostrar botão de reset apenas no final da pagina, com edit para escrever o
-  número.
-- aluno faz login, mas fecha browser, ficando no estado (online,deny). Ao
-  tentar login com outro browser está deny e o prof não consegue pô-lo em allow
-  pois já não está na lista. => solucao é manter todos os alunos numa tabela.
-- pagina de login nao esta a apresentar bem. parece que precisa de autorizacao
-  para aceder a /static...
+- mostrar botão de reset apenas no final da pagina, com edit para escrever o número.
+- aluno faz login, mas fecha browser, ficando no estado (online,deny). Ao tentar login com outro browser está deny e o prof não consegue pô-lo em allow pois já não está na lista. => solucao é manter todos os alunos numa tabela.
+- pagina de login nao esta a apresentar bem. parece que precisa de autorizacao para aceder a /static...
 - Não mostrar Professor nos activos em /admin
 - /admin mostrar actualizações automaticamente?
 - se no teste uma das "ref" nao existir nos ficheiros de perguntas, rebenta.
-- alunos podem estar online, mas browser perder sessao => nao conseguem mais
-  entrar porque a App pensa que estão online. Permitir login e dar o mesmo
-  teste.
-- pagina de management dos alunos.  mostrar online ordenados por hora de login,
-  offline por número.  permitir reset da pw e allow/disallow
+- alunos podem estar online, mas browser perder sessao => nao conseguem mais entrar porque a App pensa que estão online. Permitir login e dar o mesmo teste.
+- pagina de management dos alunos.
+    mostrar online ordenados por hora de login, offline por número.
+    permitir reset da pw e allow/disallow
 - script de correcção pode enviar dicionario yaml com grade e comentarios. ex:
-  grade: 0.5 comments: Falhou na função xpto.  os comentários são guardados no
-  teste (ficheiro) ou enviados para o browser no modo practice.
+    grade: 0.5
+    comments: Falhou na função xpto.
+    os comentários são guardados no teste (ficheiro) ou enviados para o browser no modo practice.
+- testar regex na definicao das perguntas. como se faz rawstring em yaml?
+    singlequote? problemas de backslash??? sim... necessário fazer \\ em varios casos, mas não é claro! e.g. \n é convertido em espaço mas \w é convertido em \\ e w. Solução (http://stackoverflow.com/questions/10771163/python-interpreting-a-regex-from-a-yaml-config-file) é fazer
+            correct: !regex '^(yes|no)'
@@ -11,26 +11,27 @@
 ## Requirements
-The webserver is a python application that requires `>=python3.7` and the package installer for python `pip`. The node package management `npm` is also necessary in order to install the javascript libraries.
+The webserver is a python application that requires `>=python3.8` and the
+package installer for python `pip`. The node package management `npm` is also
+necessary in order to install the javascript libraries.
 ```sh
 sudo apt install python3 python3-pip npm                     # Ubuntu
-sudo pkg install python37 py37-sqlite3 py37-pip npm          # FreeBSD
-sudo port install python37 py37-pip py37-setuptools npm6     # MacOS (macports)
+sudo pkg install python38 py38-sqlite3 py38-pip npm          # FreeBSD
+sudo port install python38 py38-pip py38-setuptools npm6     # MacOS (macports)
 ```
-To make the `pip` install packages to a local directory, the file `pip.conf` should be configured as follows:
+To make the `pip` install packages to a local directory, the file `pip.conf`
+should be configured as follows:
 ```ini
 [global]
 user=yes
-
-[list]
-format=columns
 ```
 This file is usually in `~/.config/pip/` in Linux and FreeBSD. In MacOS it's in
-`~/Library/Application Support/pip/`. You may have to create it, if it doesn't exist yet.
+`~/Library/Application Support/pip/`. You may have to create it, if it doesn't
+exist yet.
 ---
@@ -42,19 +43,21 @@ Download and install:
 git clone https://git.xdi.uevora.pt/mjsb/perguntations.git
 cd perguntations
 npm install
-pip3 install .
+pip install .
 ```
-The command `npm` installs the javascript libraries and then `pip3` installs the python webserver. This will also install any required dependencies.
+The command `npm` installs the javascript libraries and then `pip` installs the
+python webserver. This will also install any required dependencies.
-**Atention!** `pip3 install .` must run **after** `npm install`, otherwise the javascript libraries will not be found during the install.
+**Atention!** `pip install .` must be run **after** `npm install`, otherwise
+the javascript libraries will not be found during the install.
 ---
 ## Setup
-The server will run a `https` server and requires valid certificates.
-There are two possibilities to generate the certificates:
+The server will run a `https` server and requires valid certificates.  There
+are two possibilities to generate the certificates:
 - public server with static IP address and registered domain name;
 - private server on a local network isolated from the internet.
@@ -74,38 +77,35 @@ cd ~/.local/share/certs
 openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -nodes
 ```
-To have a true certificate, like the ones issued by LetsEncrypt, the host
-must have a registered domain name.
-See instructions for FreeBSD at the end of this page.
+To have a true certificate, like the ones issued by LetsEncrypt, the host must
+have a registered domain name.  See instructions for FreeBSD at the end of this
+page.
 ---
 ## Running a demo
-The directory `demo` in the repository includes a demo test that can be used
-as a template for your own tests and questions.
-
-To run the demonstration test you need to initialize the database using one of
-the following methods:
+The directory `demo` in the repository includes a demo test that can be used as
+a template for your own tests and questions.  To run the demonstration test you
+need to initialize the database using one of the following methods:
 ```sh
 cd perguntations/demo
-
 initdb students.csv                 # initialize from a CSV file
 initdb --admin                      # only adds the administrator account
 initdb --add 123 "Asterix Gaules"   # add one student
 ```
 This will create or update a `students.db` file that contains a sqlite3
-database.
-The database stores user passwords and grades, but not the actual tests.
+database.  The database stores user passwords and grades, but not the actual
+tests.
-A test is specified in a single `yaml` file.
-The demo already includes the `demo.yaml` that you can play with.
+A test is specified in a single `yaml` file.  The file `demo.yaml` is a test
+specification that you can play with.
-The complete tests submitted by the students are stored in JSON files in the
-directory defined in `demo.yaml` under the option `answers_dir: ans`.
-We also have to create this directory manually:
+The answered tests submitted by the students are stored in JSON files in a
+directory defined in `demo.yaml` under the option `answers_dir: ans`.  We also
+have to create this directory manually:
 ```sh
 mkdir ans                           # directory where the tests will be saved
@@ -114,38 +114,36 @@ mkdir ans                           # directory where the tests will be saved
 Start the server and run the `demo.yaml` test:
 ```sh
-perguntations demo.yaml         # run demo test
+perguntations demo.yaml             # run demo test
 ```
 Several options are available, run `perguntations --help` for a list.
-The server listens on port 8443 of all IPs of all network interfaces.
-Open the browser at `http://127.0.0.1:8443/` and login as user number `0`
+The server listens on port 8443 of all IPs of all network interfaces.  Open the
+browser at `http://127.0.0.1:8443/` and login as user number `0`
 (administrator) and choose any password you want. The password is defined on
 first login.
+
 After logging in, you will be redirected to the administration page that shows
 all the students and their current state.
 1. Authorize students by clicking the checkboxes.
 2. Open a different browser (or exit administrator) at `http://127.0.0.1:8443/`
-and login as one of the authorized students.
-Answer the questions and submit. You should get a grade at the end.
+   and login as one of the authorized students.  Answer the questions and
+   submit. You should get a grade at the end.
-The server can be stoped from the terminal with `^C`.
+The server can be stoped from the terminal with `^C` (Control+C).
 ---
 ## Running on lower ports
 Ports 80 and 443 are reserved for the root user and and this software **should
-NOT be run as root**.
-Instead, tcp traffic can be forwarded from port 443 to 8443 where the server is
-listening.
-The details depend on the operating system/firewall.
-
-### debian
+NOT be run as root**.  Instead, tcp traffic can be forwarded from port 443 to
+8443 where the server is listening.  The details depend on the operating
+system/firewall.
-FIXME: Untested
+### debian (FIXME: Untested)
 ```.sh
 iptables -t nat -I PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-ports 8443
@@ -166,25 +164,29 @@ Explanation:
 Edit `/etc/pf.conf`:
-    ext_if="wlan1"
-    rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
-    rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
+```conf
+ext_if="wlan1"
+rdr on $ext_if proto tcp from any to any port 80 -> 127.0.0.1 port 8080
+rdr on $ext_if proto tcp from any to any port 443 -> 127.0.0.1 port 8443
+```
-The `wlan1` should be the name of the network interface.
-Use `ext_if="vtnet0"` for guest additions under virtual box.
+The `wlan1` should be the name of the network interface.  Use `ext_if="vtnet0"`
+for guest additions under virtual box.
 Start firewall with `sudo service pf start`.
 Optionally, to activate pf on boot, edit `rc.conf`:
-    pf_enable="YES"
-    pf_flags=""
-    pf_rules="/etc/pf.conf"
+```conf
+pf_enable="YES"
+pf_flags=""
+pf_rules="/etc/pf.conf"
-    # optional logging:
-    pflog_enable="YES"
-    pflog_flags=""
-    pflog_logfile="/var/log/pflog"
+# optional logging:
+pflog_enable="YES"
+pflog_flags=""
+pflog_logfile="/var/log/pflog"
+```
 ## Generating certificates with LetsEncript (FreeBSD)
@@ -192,7 +194,7 @@ Generating certificates for a public server (FreeBSD) requires a registered
 domain with fixed IP.
 ```sh
-sudo pkg install py27-certbot                   # FreeBSD
+sudo pkg install py38-certbot                   # FreeBSD 13
 sudo service pf stop                            # disable pf firewall (FreeBSD)
 sudo certbot certonly --standalone -d www.example.com
 sudo service pf start                           # enable pf firewall
@@ -208,47 +210,52 @@ chmod 400 cert.pem privkey.pem
 Certificate renewals can be done as follows:
 ```sh
-sudo service pf stop   # shutdown firewall
+sudo service pf stop        # shutdown firewall
 sudo certbot renew
-sudo service pf start  # start firewall
+sudo service pf start       # start firewall
 ```
-Again, copy certificate files `privkey.pem` and `cert.pem` to `~/.local/share/certs`.
+Again, copy certificate files `privkey.pem` and `cert.pem` to
+`~/.local/share/certs`.
 ---
 ## Upgrading
-From time to time there can be updates to perguntations, python packages and javascript libraries.
-
-Python packages can be upgraded independently of the rest using pip:
+From time to time there can be updates to perguntations, python packages and
+javascript libraries.  Python packages can be upgraded independently of the
+rest using pip:
 ```sh
-pip list --outdated         # lists upgradable packages
+pip list --outdated --user  # lists upgradable packages
 pip install -U something    # upgrade something
 ```
+Attention: don't upgrade blindly. Some upgrades can be incompatible with
+perguntations and break things! All upgrades require testing of all the
+funcionalities.
+
 To upgrade perguntations and javascript libraries do:
 ```sh
 cd perguntations
 git pull                    # get latest version of perguntations
 npm update                  # get latest versions of javascript libraries
-pip3 install -U .           # upgrade perguntations
+pip install -U .            # upgrade perguntations
 ```
 ## Troubleshooting
 - The server tries to run `python3` so this command must be accessible from
-user accounts. Currently, the minimum supported python version is 3.7.
-
+  user accounts. Currently, the minimum supported python version is 3.8.
 - If you are getting any `UnicodeEncodeError` type of errors that's because the
-terminal is not supporting UTF-8.
-This error may occur when a unicode character is printed to the screen by the
-server or, when running question generator or correction scripts, a message is
-piped between the server and the scripts that includes unicode characters.
-Try running `locale` on the terminal and see if there are any error messages.
-Solutions:
+  terminal is not supporting UTF-8.  This error may occur when a unicode
+  character is printed to the screen by the server or, when running question
+  generator or correction scripts, a message is piped between the server and
+  the scripts that includes unicode characters.  Try running `locale` on the
+  terminal and see if there are any error messages.
+  Solutions:
+
   - debian: `sudo dpkg-reconfigure locales` and select your UTF-8 locales.
   - FreeBSD: edit `~/.login_conf` to use UTF-8, for example:
@@ -14,9 +14,6 @@ database: students.db
 # Directory where the submitted and corrected test are stored for later review.
 answers_dir: ans
-# Server used to compile & execute code
-jobe_server: 192.168.1.85
-
 # --- optional settings: -----------------------------------------------------
 # Title of this test, e.g. course name, year or test number
@@ -41,43 +38,46 @@ autocorrect: true
 # (default: true)
 show_points: true
-# scale final grade to an interval, e.g. [0, 20], keeping the relative weight
-# of the points declared in the questions below.
+# Scale the points of the questions so that the final grade is in the given
+# interval.
 # (default: no scaling, just use question points)
-scale: [0, 5]
+scale: [0, 20]
 # ----------------------------------------------------------------------------
-# Base path applied to the questions files and all the scripts
-# including question generators and correctors.
-# Either absolute path or relative to current directory can be used.
-questions_dir: .
-
-# (optional) List of files containing questions in yaml format.
-# Selected questions will be obtained from these files.
-# If undefined, all yaml files in questions_dir are loaded (not recommended).
+# Files to import. Each file contains a list of questions in yaml format.
 files:
   - questions/questions-tutorial.yaml
 # This is the list of questions that will make up the test.
 # The order is preserved.
-# There are several ways to define each question (explained below).
+# Each question is a dictionary with a question `ref` or a list of `ref`.
+# If a list is given, one question will be choosen randomly to each student.
+# The `points` for each question is optional and is 1.0 by default for normal
+# questions. Informative type of "questions" will have 0.0 points.
+# Points are automatically scaled if `scale` key is defined.
 questions:
   - ref: tut-test
-  - tut-questions
+  - ref: tut-questions
+
+  # these will have 1.0 points
+  - ref: tut-radio
+  - ref: tut-checkbox
+  - ref: tut-text
+  - ref: tut-text-regex
+  - ref: tut-numeric-interval
-  - tut-radio
-  - tut-checkbox
-  - tut-text
-  - tut-text-regex
-  - tut-numeric-interval
+  # this question will have 2.0 points
   - ref: tut-textarea
     points: 2.0
-  - tut-information
-  - tut-success
-  - tut-warning
-  - [tut-alert1, tut-alert2]
-  - tut-generator
-  - tut-yamllint
-  # - tut-code
+  # these will have 0.0 points:
+  - ref: tut-information
+  - ref: tut-success
+  - ref: tut-warning
+
+  # choose one from the list:
+  - ref: [tut-alert1, tut-alert2]
+
+  - ref: tut-generator
+  - ref: tut-yamllint
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+
+'''
+Example of a question generator.
+Arguments are read from stdin.
+'''
+
+from random import randint
+import sys
+
+# read two arguments from the field `args` specified in the question yaml file
+a, b = (int(n) for n in sys.argv[1:])
+
+x = randint(a, b)
+y = randint(a, b)
+r = x + y
+
+print(f"""---
+type: text
+title: Geradores de perguntas
+text: |
+
+  As perguntas podem ser estáticas (como as que vimos até aqui), ou serem
+  geradas dinâmicamente por um programa externo. Para gerar uma pergunta, o
+  programa deve escrever texto no `stdout` em formato `yaml` tal como os
+  exemplos das perguntas estáticas dos tipos apresentados anteriormente. Pode
+  também receber argumentos de linha de comando para parametrizar a pergunta.
+  Aqui está um exemplo de uma pergunta gerada por um script python:
+
+  ```python
+      #!/usr/bin/env python3
+
+      from random import randint
+      import sys
+
+      a, b = (int(n) for n in sys.argv[1:])   # argumentos da linha de comando
+
+      x = randint(a, b)   # número inteiro no intervalo a..b
+      y = randint(a, b)   # número inteiro no intervalo a..b
+      r = x + y           # calcula resultado correcto
+
+      print(f'''---
+      type: text
+      title: Contas de somar
+      text: |
+        Calcule o resultado de ${{x}} + {{y}}$.
+      correct: '{{r}}'
+      solution: |
+        A solução é {{r}}.''')
+  ```
+
+  Este script deve ter permissões para poder ser executado no terminal.
+  Podemos testar o programa no terminal `./gen-somar.py 1 100` e verificar que
+  o output é uma pergunta válida em formato `yaml`. Agora é necessário indicar
+  que este script deve ser usado para gerar uma pergunta.
+
+  Uma pergunta gerada por um programa externo é declarada com
+
+  ```yaml
+      - type: generator
+        ref: gen-somar
+        script: gen-somar.py
+        # argumentos opcionais
+        args: [1, 100]
+  ```
+
+  O programa pode receber uma lista de argumentos de linha de comando
+  declarados em `args`.
+
+  ---
+
+  Calcule o resultado de ${x} + {y}$.
+
+  Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
+correct: '{r}'
+solution: |
+  A solução é {r}.""")
@@ -1,77 +0,0 @@
-#!/usr/bin/env python3
-
-'''
-Example of a question generator.
-Arguments are read from stdin.
-'''
-
-from random import randint
-import sys
-
-# read two arguments from the field `args` specified in the question yaml file
-a, b = (int(n) for n in sys.argv[1:])
-
-x = randint(a, b)
-y = randint(a, b)
-r = x + y
-
-print(f"""---
-type: text
-title: Geradores de perguntas
-text: |
-
-  As perguntas podem ser estáticas (como as que vimos até aqui), ou serem
-  geradas dinâmicamente por um programa externo. Para gerar uma pergunta, o
-  programa deve escrever texto no `stdout` em formato `yaml` tal como os
-  exemplos das perguntas estáticas dos tipos apresentados anteriormente. Pode
-  também receber argumentos de linha de comando para parametrizar a pergunta.
-  Aqui está um exemplo de uma pergunta gerada por um script python:
-
-  ```python
-      #!/usr/bin/env python3
-
-      from random import randint
-      import sys
-
-      a, b = (int(n) for n in sys.argv[1:])   # argumentos da linha de comando
-
-      x = randint(a, b)   # número inteiro no intervalo a..b
-      y = randint(a, b)   # número inteiro no intervalo a..b
-      r = x + y           # calcula resultado correcto
-
-      print(f'''---
-      type: text
-      title: Contas de somar
-      text: |
-        Calcule o resultado de ${{x}} + {{y}}$.
-      correct: '{{r}}'
-      solution: |
-        A solução é {{r}}.''')
-  ```
-
-  Este script deve ter permissões para poder ser executado no terminal.
-  Podemos testar o programa no terminal `./gen-somar.py 1 100` e verificar que
-  o output é uma pergunta válida em formato `yaml`. Agora é necessário indicar
-  que este script deve ser usado para gerar uma pergunta.
-
-  Uma pergunta gerada por um programa externo é declarada com
-
-  ```yaml
-      - type: generator
-        ref: gen-somar
-        script: gen-somar.py
-        # argumentos opcionais
-        args: [1, 100]
-  ```
-
-  O programa pode receber uma lista de argumentos de linha de comando
-  declarados em `args`.
-
-  ---
-
-  Calcule o resultado de ${x} + {y}$.
-
-  Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
-correct: '{r}'
-solution: |
-  A solução é {r}.""")
@@ -20,24 +20,20 @@
     database: students.db      # base de dados previamente criada com initdb
     answers_dir: ans           # directório onde ficam os testes dos alunos
-    # opcional
+    # opcionais
     duration: 60               # duração da prova em minutos (default: inf)
     autosubmit: true           # submissão automática (default: false)
     show_points: true          # mostra cotação das perguntas (default: true)
-    scale: [0, 20]             # limites inferior e superior da escala (default: [0,20])
-    scale_points: true         # normaliza cotações para a escala definida
-    jobe_server: moodle-jobe.uevora.pt  # server used to compile & execute code
-    debug: false               # mostra informação de debug no browser
+    scale: [0, 20]             # normaliza cotações para o intervalo indicado.
+                               # não normaliza por defeito (default: None)
     # --------------------------------------------------------------------------
-    questions_dir: ~/topics    # raíz da árvore de directórios das perguntas
-
     # Ficheiros de perguntas a importar (relativamente a `questions_dir`)
     files:
       - tabelas.yaml
-      - topic_A/questions.yaml
-      - topic_B/part_1/questions.yaml
-      - topic_B/part_2/questions.yaml
+      - topic1/questions.yaml
+      - topic2/part1/questions.yaml
+      - topic2/part2/questions.yaml
     # --------------------------------------------------------------------------
     # Especificação das perguntas do teste e respectivas cotações.
@@ -50,13 +46,10 @@
       - ref: pergunta2
         points: 2.0
-      # a cotação é 1.0 por defeito
+      # por defeinto, a cotação da pergunta é 1.0 valor
       - ref: pergunta3
-      # uma string (não dict), é interpretada como referência
-      - tabela-auxiliar
-
-      # escolhe aleatoriamente uma das variantes
+      # escolhe aleatoriamente uma das variantes da pergunta
       - ref: [pergunta3a, pergunta3b]
         points: 0.5
@@ -96,8 +89,7 @@
       text: |
         Quando o texto da pergunta tem várias linhas, dá jeito usar o símbolo
         `|` de pipe, para indicar que tudo o que estiver indentado faz parte do
-        texto.
-        É o caso desta pergunta.
+        texto. É o caso desta pergunta.
         O texto das perguntas é escrito em `markdown` e suporta fórmulas em
         LaTeX.
@@ -105,8 +97,8 @@
     #---------------------------------------------------------------------------
     ```
-    As chaves são usadas para construir o teste e não se podem repetir, mesmo em
-    ficheiros diferentes.
+    As chaves são usadas para construir o teste e não se podem repetir, mesmo
+    em ficheiros diferentes.
     De seguida mostram-se exemplos dos vários tipos de perguntas.
 # ----------------------------------------------------------------------------
@@ -431,6 +423,10 @@
     pode estar previamente preenchida como neste caso (use `answer: texto`).
   correct: correct/correct-question.py
   timeout: 5
+  tests_right:
+    - 'red green blue'
+  # tests_wrong:
+  #   - 'blue gray yellow'
 # ---------------------------------------------------------------------------
 - type: information
@@ -504,6 +500,7 @@
             return 0;   // comentario
         }
         ```
+
     ```
 # ---------------------------------------------------------------------------
@@ -586,7 +583,7 @@
 # ----------------------------------------------------------------------------
 - type: generator
   ref: tut-generator
-  script: generators/generate-question.py
+  script: generate-question.py
   args: [1, 100]
 # ----------------------------------------------------------------------------
 [mypy]
-python_version = 3.7
-# warn_return_any = True
-# warn_unused_configs = True
-
-[mypy-sqlalchemy.*]
+python_version = 3.9
 ignore_missing_imports = True
-[mypy-pygments.*]
-ignore_missing_imports = True
-[mypy-bcrypt.*]
-ignore_missing_imports = True
+; [mypy-setuptools.*]
+; ignore_missing_imports = True
-[mypy-mistune.*]
-ignore_missing_imports = True
+; [mypy-sqlalchemy.*]
+; ignore_missing_imports = True
+
+; [mypy-pygments.*]
+; ignore_missing_imports = True
+
+; [mypy-mistune.*]
+; ignore_missing_imports = True
@@ -2,12 +2,12 @@
     "description": "Javascript libraries required to run the server",
     "email": "mjsb@uevora.pt",
     "dependencies": {
-        "@fortawesome/fontawesome-free": "^5.15.1",
-        "bootstrap": "^4.5",
-        "codemirror": "^5.58.1",
+        "@fortawesome/fontawesome-free": "^5.15.3",
+        "bootstrap": "^4.6.0",
+        "codemirror": "^5.61.1",
         "datatables": "^1.10",
-        "jquery": "^3.5.1",
-        "mathjax": "^3.1.2",
-        "underscore": "^1.11.0"
+        "jquery": "^3.6.0",
+        "mathjax": "^3.1.4",
+        "underscore": "^1.13.1"
     }
 }
-# Copyright (C) 2020 Miguel Barão
+# Copyright (C) 2022 Miguel Barão
 #
 # THE MIT License
 #
@@ -32,10 +32,10 @@ proof of submission and for review.
 '''
 APP_NAME = 'perguntations'
-APP_VERSION = '2021.02.dev1'
-APP_DESCRIPTION = __doc__
+APP_VERSION = '2022.01.dev1'
+APP_DESCRIPTION = str(__doc__)
 __author__ = 'Miguel Barão'
-__copyright__ = 'Copyright 2020, Miguel Barão'
+__copyright__ = 'Copyright 2022, Miguel Barão'
 __license__ = 'MIT license'
 __version__ = APP_VERSION
 '''
-Main application module
+File:           perguntations/app.py
+Description:    Main application logic.
 '''
 # python standard libraries
 import asyncio
-from contextlib import contextmanager  # `with` statement in db sessions
 import csv
 import io
 import json
 import logging
-from os import path
+import os
+from typing import Optional
 # installed packages
 import bcrypt
-from sqlalchemy import create_engine, exc
-from sqlalchemy.orm import sessionmaker
+from sqlalchemy import create_engine, select
+from sqlalchemy.exc import OperationalError, NoResultFound, IntegrityError
+from sqlalchemy.orm import Session
+import yaml
 # this project
-from perguntations.models import Student, Test, Question
-from perguntations.tools import load_yaml
-from perguntations.testfactory import TestFactory, TestFactoryException
-import perguntations.test
-from perguntations.questions import question_from
+from .models import Student, Test, Question
+from .tools import load_yaml
+from .testfactory import TestFactory, TestFactoryException
+from .test import Test as TestInstance
+from .questions import question_from
+# setup logger for this module
 logger = logging.getLogger(__name__)
-
-# ============================================================================
-class AppException(Exception):
-    '''Exception raised in this module'''
-
-
-# ============================================================================
-# helper functions
-# ============================================================================
-async def check_password(try_pw, hashed_pw):
+async def check_password(password: str, hashed: bytes) -> bool:
     '''check password in executor'''
-    try_pw = try_pw.encode('utf-8')
     loop = asyncio.get_running_loop()
-    hashed = await loop.run_in_executor(None, bcrypt.hashpw, try_pw, hashed_pw)
-    return hashed_pw == hashed
-
+    return await loop.run_in_executor(None, bcrypt.checkpw,
+            password.encode('utf-8'), hashed)
-async def hash_password(password):
-    '''hash password in executor'''
+async def hash_password(password: str) -> bytes:
+    '''get hash for password'''
     loop = asyncio.get_running_loop()
     return await loop.run_in_executor(None, bcrypt.hashpw,
-                                      password.encode('utf-8'),
-                                      bcrypt.gensalt())
+            password.encode('utf-8'), bcrypt.gensalt())
+# ============================================================================
+class AppException(Exception):
+    '''Exception raised in this module'''
 # ============================================================================
 # main application
 # ============================================================================
 class App():
     '''
-    This is the main application
-    state:
-      self.Session
-      self.online - {uid:
-                         {'student':{...}, 'test': {...}},
-                          ...
-                     }
-      self.allowd - {'123', '124', ...}
-      self.testfactory - TestFactory
+    Main application
     '''
     # ------------------------------------------------------------------------
-    @contextmanager
-    def _db_session(self):
-        '''
-        helper to manage db sessions using the `with` statement, for example:
-        with self._db_session() as s:  s.query(...)
-        '''
-        session = self.Session()
-        try:
-            yield session
-            session.commit()
-        except exc.SQLAlchemyError:
-            logger.error('DB rollback!!!')
-            session.rollback()
-            raise
-        finally:
-            session.close()
+    def __init__(self, config):
+        self.debug = config['debug']
+        self._make_test_factory(config['testfile'])
+        self._db_setup()  # setup engine and load all students
-    # ------------------------------------------------------------------------
-    def __init__(self, conf) -> None:
-        self.online = dict()    # {uid: {'student':{...}, 'test': {...}}, ...}
-        self.allowed = set()    # '0' is hardcoded to allowed elsewhere
-        self.unfocus = set()    # set of students that have no browser focus
-        self.area = dict()      # {uid: percent_area}
-        self.pregenerated_tests = []  # list of tests to give to students
-
-        self._make_test_factory(conf)
-
-        # connect to database and check registered students
-        dbfile = self.testfactory['database']
-        database = f'sqlite:///{path.expanduser(dbfile)}'
-        engine = create_engine(database, echo=False)
-        self.Session = sessionmaker(bind=engine)
-        try:
-            with self._db_session() as sess:
-                num = sess.query(Student).filter(Student.id != '0').count()
-        except Exception as exc:
-            raise AppException(f'Database unusable {dbfile}.') from exc
-        logger.info('Database "%s" has %s students.', dbfile, num)
-
-        # command line option --allow-all
-        if conf['allow_all']:
-            self.allow_all_students()
-        elif conf['allow_list'] is not None:
-            self.allow_list(conf['allow_list'])
-        else:
-            logger.info('Students not yet allowed to login.')
+        # FIXME get_event_loop will be deprecated in python3.10
+        asyncio.get_event_loop().run_until_complete(self._assign_tests())
-        # pre-generate tests for allowed students
-        if self.allowed:
-            logger.info('Generating %d tests. May take awhile...',
-                        len(self.allowed))
-            self._pregenerate_tests(len(self.allowed))
+        # command line options:  --allow-all, --allow-list filename
+        if config['allow_all']:
+            self.allow_all_students()
+        elif config['allow_list'] is not None:
+            self.allow_from_list(config['allow_list'])
         else:
-            logger.info('No tests were generated.')
+            logger.info('Students not allowed to login')
-        if conf['correct']:
+        if config['correct']:
             self._correct_tests()
     # ------------------------------------------------------------------------
-    def _correct_tests(self) -> None:
-        with self._db_session() as sess:
-            # Find which tests have to be corrected.
-            # already corrected tests are not included.
-            dbtests = sess.query(Test)\
-                    .filter(Test.ref == self.testfactory['ref'])\
-                    .filter(Test.state == "SUBMITTED")\
-                    .all()
-
-            logger.info('Correcting %d tests...', len(dbtests))
-            for dbtest in dbtests:
-                try:
-                    with open(dbtest.filename) as file:
-                        testdict = json.load(file)
-                except FileNotFoundError:
-                    logger.error('File not found: %s', dbtest.filename)
-                    continue
+    def _db_setup(self) -> None:
+        '''
+        Create database engine and checks for admin and students
+        '''
+        dbfile = os.path.expanduser(self._testfactory['database'])
+        logger.debug('Checking database "%s"...', dbfile)
+        if not os.path.exists(dbfile):
+            raise AppException('No database, use "initdb" to create')
-                # creates a class Test with the methods to correct it
-                # the questions are still dictionaries, so we have to call
-                # question_from() to produce Question() instances that can be
-                # corrected. Finally the test can be corrected.
-                test = perguntations.test.Test(testdict)
-                test['questions'] = [question_from(q) for q in test['questions']]
-                test.correct()
-                logger.info('Student %s:  grade = %f', test['student']['number'], test['grade'])
+        # connect to database and check for admin & registered students
+        self._engine = create_engine(f'sqlite:///{dbfile}', future=True)
+        try:
+            with Session(self._engine, future=True) as session:
+                query = select(Student.id, Student.name)\
+                        .where(Student.id != '0')
+                dbstudents = session.execute(query).all()
+                session.execute(select(Student).where(Student.id == '0')).one()
+        except NoResultFound:
+            msg = 'Database has no administrator (user "0")'
+            logger.error(msg)
+            raise AppException(msg) from None
+        except OperationalError:
+            msg = f'Database "{dbfile}" unusable'
+            logger.error(msg)
+            raise AppException(msg) from None
+        logger.info('Database has %d students', len(dbstudents))
-                # save JSON file (overwriting the old one)
-                uid = test['student']['number']
-                ref = test['ref']
-                finish_time = test['finish_time']
-                answers_dir = test['answers_dir']
-                fname = f'{uid}--{ref}--{finish_time}.json'
-                fpath = path.join(answers_dir, fname)
-                test.save_json(fpath)
-                logger.info('%s saved JSON file.', uid)
+        self._students = {uid: {
+                'name': name,
+                'state': 'offline',
+                'test': None,
+            } for uid, name in dbstudents}
-                # update database
-                dbtest.grade = test['grade']
-                dbtest.state = test['state']
-                dbtest.questions = [
-                    Question(
-                        number=n,
-                        ref=q['ref'],
-                        grade=q['grade'],
-                        comment=q.get('comment', ''),
-                        starttime=str(test['start_time']),
-                        finishtime=str(test['finish_time']),
-                        test_id=test['ref']
-                        )
-                    for n, q in enumerate(test['questions'])
-                    ]
-                logger.info('%s database updated.', uid)
+    # ------------------------------------------------------------------------
+    async def _assign_tests(self) -> None:
+        '''Generate tests for all students that don't yet have a test'''
+        logger.info('Generating tests...')
+        for student in self._students.values():
+            if student.get('test', None) is None:
+                student['test'] = await self._testfactory.generate()
+        logger.info('Tests assigned to all students')
     # ------------------------------------------------------------------------
-    async def login(self, uid, try_pw, headers=None):
-        '''login authentication'''
-        if uid not in self.allowed and uid != '0':      # not allowed
-            logger.warning('"%s" unauthorized.', uid)
-            return 'unauthorized'
-
-        with self._db_session() as sess:
-            name, hashed_pw = sess.query(Student.name, Student.password)\
-                                 .filter_by(id=uid)\
-                                 .one()
-
-        if hashed_pw == '':              # update password on first login
-            await self.update_student_password(uid, try_pw)
-            pw_ok = True
-        else:                           # check password
-            pw_ok = await check_password(try_pw, hashed_pw)  # async bcrypt
-
-        if not pw_ok:        # wrong password
-            logger.info('"%s" wrong password.', uid)
+    async def login(self, uid: str, password: str, headers: dict) -> Optional[str]:
+        '''
+        Login authentication
+        If successful returns None, else returns an error message
+        '''
+        try:
+            with Session(self._engine, future=True) as session:
+                query = select(Student.password).where(Student.id == uid)
+                hashed = session.execute(query).scalar_one()
+        except NoResultFound:
+            logger.warning('"%s" does not exist', uid)
+            return 'nonexistent'
+
+        if uid != '0' and self._students[uid]['state'] != 'allowed':
+            logger.warning('"%s" login not allowed', uid)
+            return 'not_allowed'
+
+        if hashed == '':  # set password on first login
+            await self.set_password(uid, password)
+        elif not await check_password(password, hashed):
+            logger.info('"%s" wrong password', uid)
             return 'wrong_password'
         # success
-        self.allowed.discard(uid)  # remove from set of allowed students
-
-        if uid in self.online:
-            logger.warning('"%s" login again from %s (reusing state).',
-                           uid, headers['remote_ip'])
-            # FIXME invalidate previous login
+        if uid == '0':
+            logger.info('Admin login from %s', headers['remote_ip'])
         else:
-            self.online[uid] = {'student': {
-                                    'name': name,
-                                    'number': uid,
-                                    'headers': headers}}
-            logger.info('"%s" login from %s.', uid, headers['remote_ip'])
+            student = self._students[uid]
+            student['test'].start(uid)
+            student['state'] = 'online'
+            student['headers'] = headers
+            student['unfocus'] = False
+            student['area'] = 0.0
+            logger.info('"%s" login from %s', uid, headers['remote_ip'])
+        return None
     # ------------------------------------------------------------------------
-    def logout(self, uid):
+    async def set_password(self, uid: str, password: str) -> None:
+        '''change password on the database'''
+        with Session(self._engine, future=True) as session:
+            query = select(Student).where(Student.id == uid)
+            student = session.execute(query).scalar_one()
+            student.password = await hash_password(password) if password else ''
+            session.commit()
+        logger.info('"%s" password updated', uid)
+
+    # ------------------------------------------------------------------------
+    def logout(self, uid: str) -> None:
         '''student logout'''
-        self.online.pop(uid, None)  # remove from dict if exists
-        logger.info('"%s" logged out.', uid)
+        student = self._students.get(uid, None)
+        if student is not None:
+            # student['test'] = None
+            student['state'] = 'offline'
+            student.pop('headers', None)
+            student.pop('unfocus', None)
+            student.pop('area', None)
+            logger.info('"%s" logged out', uid)
     # ------------------------------------------------------------------------
-    def _make_test_factory(self, conf):
+    def _make_test_factory(self, filename: str) -> None:
         '''
         Setup a factory for the test
         '''
         # load configuration from yaml file
-        logger.info('Loading test configuration "%s".', conf["testfile"])
         try:
-            testconf = load_yaml(conf['testfile'])
-        except Exception as exc:
-            msg = 'Error loading test configuration YAML.'
-            logger.critical(msg)
+            testconf = load_yaml(filename)
+            testconf['testfile'] = filename
+        except (OSError, yaml.YAMLError) as exc:
+            msg = f'Cannot read test configuration "{filename}"'
+            logger.error(msg)
             raise AppException(msg) from exc
-        # command line options override configuration
-        testconf.update(conf)
-
-        # start test factory
+        # make test factory
         logger.info('Running test factory...')
         try:
-            self.testfactory = TestFactory(testconf)
+            self._testfactory = TestFactory(testconf)
         except TestFactoryException as exc:
-            logger.critical(exc)
+            logger.error(exc)
             raise AppException('Failed to create test factory!') from exc
     # ------------------------------------------------------------------------
-    def _pregenerate_tests(self, num):  # TODO needs improvement
-        event_loop = asyncio.get_event_loop()
-        self.pregenerated_tests += [
-                event_loop.run_until_complete(self.testfactory.generate())
-                for _ in range(num)]
-
-    # ------------------------------------------------------------------------
-    async def get_test_or_generate(self, uid):
-        '''get current test or generate a new one'''
-        try:
-            student = self.online[uid]
-        except KeyError as exc:
-            msg = f'"{uid}" is not online. get_test_or_generate() FAILED'
-            logger.error(msg)
-            raise AppException(msg) from exc
-
-        # get current test. if test does not exist then generate a new one
-        if not 'test' in student:
-            await self._new_test(uid)
-
-        return student['test']
-
-    def get_test(self, uid):
-        '''get test from online student or raise exception'''
-        return self.online[uid]['test']
-
-    # ------------------------------------------------------------------------
-    async def _new_test(self, uid):
-        '''
-        assign a test to a given student. if there are pregenerated tests then
-        use one of them, otherwise generate one.
-        the student must be online
-        '''
-        student = self.online[uid]['student']  # {'name': ?, 'number': ?}
-
-        try:
-            test = self.pregenerated_tests.pop()
-        except IndexError:
-            logger.info('"%s" generating new test...', uid)
-            test = await self.testfactory.generate()
-            logger.info('"%s" test is ready.', uid)
-        else:
-            logger.info('"%s" using a pregenerated test.', uid)
-
-        test.start(student)           # student signs the test
-        self.online[uid]['test'] = test
-
-    # ------------------------------------------------------------------------
-    async def submit_test(self, uid, ans):
+    async def submit_test(self, uid, ans) -> None:
         '''
         Handles test submission and correction.
         ans is a dictionary {question_index: answer, ...} with the answers for
         the complete test. For example:  {0:'hello', 1:[1,2]}
         '''
-        test = self.online[uid]['test']
+        if self._students[uid]['state'] != 'online':
+            logger.warning('"%s" INVALID SUBMISSION! STUDENT NOT ONLINE', uid)
+            return
         # --- submit answers and correct test
+        logger.info('"%s" submitted %d answers', uid, len(ans))
+        test = self._students[uid]['test']
         test.submit(ans)
-        logger.info('"%s" submitted %d answers.', uid, len(ans))
         if test['autocorrect']:
             await test.correct_async()
-            logger.info('"%s" grade = %g points.', uid, test['grade'])
+            logger.info('"%s" grade = %g points', uid, test['grade'])
         # --- save test in JSON format
         fname = f'{uid}--{test["ref"]}--{test["finish_time"]}.json'
-        fpath = path.join(test['answers_dir'], fname)
+        fpath = os.path.join(test['answers_dir'], fname)
         test.save_json(fpath)
-        logger.info('"%s" saved JSON.', uid)
+        logger.info('"%s" saved JSON', uid)
         # --- insert test and questions into the database
         #     only corrected questions are added
@@ -348,13 +253,62 @@ class App():
                 for n, q in enumerate(test['questions'])
                 ]
-        with self._db_session() as sess:
-            sess.add(test_row)
-        logger.info('"%s" database updated.', uid)
+        with Session(self._engine, future=True) as session:
+            session.add(test_row)
+            session.commit()
+        logger.info('"%s" database updated', uid)
     # ------------------------------------------------------------------------
-    def get_student_grade(self, uid):
-        return self.online[uid]['test'].get('grade', None)
+    def _correct_tests(self) -> None:
+        with Session(self._engine, future=True) as session:
+            # Find which tests have to be corrected
+            query = select(Test) \
+                    .where(Test.ref == self._testfactory['ref']) \
+                    .where(Test.state == "SUBMITTED")
+            dbtests = session.execute(query).scalars().all()
+            if not dbtests:
+                logger.info('No tests to correct')
+                return
+
+            logger.info('Correcting %d tests...', len(dbtests))
+            for dbtest in dbtests:
+                try:
+                    with open(dbtest.filename) as file:
+                        testdict = json.load(file)
+                except OSError:
+                    logger.error('Failed: %s', dbtest.filename)
+                    continue
+
+                # creates a class Test with the methods to correct it
+                # the questions are still dictionaries, so we have to call
+                # question_from() to produce Question() instances that can be
+                # corrected. Finally the test can be corrected.
+                test = TestInstance(testdict)
+                test['questions'] = [question_from(q) for q in test['questions']]
+                test.correct()
+                logger.info('  %s:  %f', test['student'], test['grade'])
+
+                # save JSON file (overwriting the old one)
+                uid = test['student']
+                test.save_json(dbtest.filename)
+                logger.debug('%s saved JSON file', uid)
+
+                # update database
+                dbtest.grade = test['grade']
+                dbtest.state = test['state']
+                dbtest.questions = [
+                    Question(
+                        number=n,
+                        ref=q['ref'],
+                        grade=q['grade'],
+                        comment=q.get('comment', ''),
+                        starttime=str(test['start_time']),
+                        finishtime=str(test['finish_time']),
+                        test_id=test['ref']
+                        ) for n, q in enumerate(test['questions'])
+                    ]
+            session.commit()
+            logger.info('Database updated')
     # ------------------------------------------------------------------------
     # def giveup_test(self, uid):
@@ -366,7 +320,7 @@ class App():
     #     fields = (test['student']['number'], test['ref'],
     #               str(test['finish_time']))
     #     fname = '--'.join(fields) + '.json'
-    #     fpath = path.join(test['answers_dir'], fname)
+    #     fpath = os.path.join(test['answers_dir'], fname)
     #     test.save_json(fpath)
     #     # insert test into database
@@ -381,11 +335,11 @@ class App():
     #                       state=test['state'],
     #                       comment=''))
-    #     logger.info('"%s" gave up.', uid)
+    #     logger.info('"%s" gave up', uid)
     #     return test
     # ------------------------------------------------------------------------
-    def event_test(self, uid, cmd, value):
+    def register_event(self, uid, cmd, value):
         '''handles browser events the occur during the test'''
         if cmd == 'focus':
             if value:
@@ -395,202 +349,200 @@ class App():
         elif cmd == 'size':
             self._set_screen_area(uid, value)
-    # ------------------------------------------------------------------------
-    # --- GETTERS
-    # ------------------------------------------------------------------------
+    # ========================================================================
+    #   GETTERS
+    # ========================================================================
+    def get_test(self, uid: str) -> Optional[dict]:
+        '''return student test'''
+        return self._students[uid]['test']
-    # def get_student_name(self, uid):
-    #     return self.online[uid]['student']['name']
+    # ------------------------------------------------------------------------
+    def get_name(self, uid: str) -> str:
+        '''return name of student'''
+        return self._students[uid]['name']
-    def get_questions_csv(self):
-        '''generates a CSV with the grades of the test'''
-        test_ref = self.testfactory['ref']
-        with self._db_session() as sess:
-            questions = sess.query(Test.id, Test.student_id, Test.starttime,
-                                   Question.number, Question.grade)\
-                            .join(Question)\
-                            .filter(Test.ref == test_ref)\
-                            .all()
-
-            qnums = set()  # keeps track of all the questions in the test
-            tests = {}      # {test_id: {student_id, starttime, 0: grade, ...}}
-            for question in questions:
-                test_id, student_id, starttime, num, grade = question
-                default_test_id = {'Aluno': student_id, 'Início': starttime}
-                tests.setdefault(test_id, default_test_id)[num] = grade
-                qnums.add(num)
+    # ------------------------------------------------------------------------
+    def get_test_config(self) -> dict:
+        '''return brief test configuration to use as header in /admin'''
+        return {'title': self._testfactory['title'],
+                'ref': self._testfactory['ref'],
+                'filename': self._testfactory['testfile'],
+                'database': self._testfactory['database'],
+                'answers_dir': self._testfactory['answers_dir']
+                }
+    # ------------------------------------------------------------------------
+    def get_test_csv(self):
+        '''generates a CSV with the grades of the test currently running'''
+        test_ref = self._testfactory['ref']
+        with Session(self._engine, future=True) as session:
+            query = select(Test.student_id, Test.grade,
+                           Test.starttime, Test.finishtime)\
+                    .where(Test.ref == test_ref)\
+                    .order_by(Test.student_id)
+            tests = session.execute(query).all()
         if not tests:
             logger.warning('Empty CSV: there are no tests!')
             return test_ref, ''
-        cols = ['Aluno', 'Início'] + list(qnums)
-
         csvstr = io.StringIO()
-        writer = csv.DictWriter(csvstr, fieldnames=cols, restval=None,
-                                delimiter=';', quoting=csv.QUOTE_ALL)
-        writer.writeheader()
-        writer.writerows(tests.values())
+        writer = csv.writer(csvstr, delimiter=';', quoting=csv.QUOTE_ALL)
+        writer.writerow(('Aluno', 'Nota', 'Início', 'Fim'))
+        writer.writerows(tests)
         return test_ref, csvstr.getvalue()
-    def get_test_csv(self):
-        '''generates a CSV with the grades of the test currently running'''
-        test_ref = self.testfactory['ref']
-        with self._db_session() as sess:
-            tests = sess.query(Test.student_id,
-                                Test.grade,
-                                Test.starttime, Test.finishtime)\
-                         .filter(Test.ref == test_ref)\
-                         .order_by(Test.student_id)\
-                         .all()
+    # ------------------------------------------------------------------------
+    def get_detailed_grades_csv(self):
+        '''generates a CSV with the grades of the test'''
+        test_ref = self._testfactory['ref']
+        with Session(self._engine, future=True) as session:
+            query = select(Test.id, Test.student_id, Test.starttime,
+                           Question.number, Question.grade)\
+                    .join(Question)\
+                    .where(Test.ref == test_ref)
+            questions = session.execute(query).all()
+
+        cols = ['Aluno', 'Início']
+        tests = {}      # {test_id: {student_id, starttime, 0: grade, ...}}
+        for test_id, student_id, starttime, num, grade in questions:
+            default_test_id = {'Aluno': student_id, 'Início': starttime}
+            tests.setdefault(test_id, default_test_id)[num] = grade
+            if num not in cols:
+                cols.append(num)
         if not tests:
             logger.warning('Empty CSV: there are no tests!')
             return test_ref, ''
         csvstr = io.StringIO()
-        writer = csv.writer(csvstr, delimiter=';', quoting=csv.QUOTE_ALL)
-        writer.writerow(('Aluno', 'Nota', 'Início', 'Fim'))
-        writer.writerows(tests)
-
+        writer = csv.DictWriter(csvstr, fieldnames=cols, restval=None,
+                                delimiter=';', quoting=csv.QUOTE_ALL)
+        writer.writeheader()
+        writer.writerows(tests.values())
         return test_ref, csvstr.getvalue()
     # ------------------------------------------------------------------------
-    def get_student_grades_from_all_tests(self, uid):
-        '''get grades of student from all tests'''
-        with self._db_session() as sess:
-            return sess.query(Test.title, Test.grade, Test.finishtime)\
-                       .filter_by(student_id=uid)\
-                       .order_by(Test.finishtime)
-
     def get_json_filename_of_test(self, test_id):
         '''get JSON filename from database given the test_id'''
-        with self._db_session() as sess:
-            return sess.query(Test.filename)\
-                       .filter_by(id=test_id)\
-                       .scalar()
+        with Session(self._engine, future=True) as session:
+            query = select(Test.filename).where(Test.id == test_id)
+            return session.execute(query).scalar()
-    def get_student_grades_from_test(self, uid, testid):
+    # ------------------------------------------------------------------------
+    def get_grades(self, uid, ref):
         '''get grades of student for a given testid'''
-        with self._db_session() as sess:
-            return sess.query(Test.grade, Test.finishtime, Test.id)\
-                       .filter_by(student_id=uid)\
-                       .filter_by(ref=testid)\
-                       .all()
-
-    def get_students_state(self):
-        '''get list of states of every student'''
-        return [{
-            'uid': uid,
-            'name': name,
-            'allowed': uid in self.allowed,
-            'online': uid in self.online,
-            'start_time': self.online.get(uid, {}).get('test', {})
-                          .get('start_time', ''),
-            'password_defined': pw != '',
-            'unfocus': uid in self.unfocus,
-            'area': self.area.get(uid, None),
-            'grades': self.get_student_grades_from_test(
-                uid, self.testfactory['ref'])
-            } for uid, name, pw in self._get_all_students()]
-
-    # --- private methods ----------------------------------------------------
-    def _get_all_students(self):
-        '''get all students from database'''
-        with self._db_session() as sess:
-            return sess.query(Student.id, Student.name, Student.password)\
-                       .filter(Student.id != '0')\
-                       .order_by(Student.id)
-
-    # def get_allowed_students(self):
-    #     # set of 'uid' allowed to login
-    #     return self.allowed
-
-    # def get_file(self, uid, ref, key):
-    #     # get filename of (uid, ref, name) if declared in the question
-    #     t = self.get_student_test(uid)
-    #     for q in t['questions']:
-    #         if q['ref'] == ref and key in q['files']:
-    #             return path.abspath(path.join(q['path'], q['files'][key]))
+        with Session(self._engine, future=True) as session:
+            query = select(Test.grade, Test.finishtime, Test.id)\
+                    .where(Test.student_id == uid)\
+                    .where(Test.ref == ref)
+            grades = session.execute(query).all()
+        return [tuple(grade) for grade in grades]
     # ------------------------------------------------------------------------
-    # --- SETTERS
-    # ------------------------------------------------------------------------
-
-    def allow_student(self, uid):
+    def get_students_state(self) -> list:
+        '''get list of states of every student to show in /admin page'''
+        return [{   'uid': uid,
+                    'name': student['name'],
+                    'allowed': student['state'] == 'allowed',
+                    'online': student['state'] == 'online',
+                    'start_time': student.get('test', {}).get('start_time', ''),
+                    'unfocus': student.get('unfocus', False),
+                    'area': student.get('area', 1.0),
+                    'grades': self.get_grades(uid, self._testfactory['ref']) }
+                for uid, student in self._students.items()]
+
+    # ========================================================================
+    #   SETTERS
+    # ========================================================================
+    def allow_student(self, uid: str) -> None:
         '''allow a single student to login'''
-        self.allowed.add(uid)
-        logger.info('"%s" allowed to login.', uid)
+        self._students[uid]['state'] = 'allowed'
+        logger.info('"%s" allowed to login', uid)
-    def deny_student(self, uid):
+    # ------------------------------------------------------------------------
+    def deny_student(self, uid: str) -> None:
         '''deny a single student to login'''
-        self.allowed.discard(uid)
+        student = self._students[uid]
+        if student['state'] == 'allowed':
+            student['state'] = 'offline'
         logger.info('"%s" denied to login', uid)
-    def allow_all_students(self):
+    # ------------------------------------------------------------------------
+    def allow_all_students(self) -> None:
         '''allow all students to login'''
-        all_students = self._get_all_students()
-        self.allowed.update(s[0] for s in all_students)
-        logger.info('Allowed all %d students.', len(self.allowed))
+        for student in self._students.values():
+            student['state'] = 'allowed'
+        logger.info('Allowed %d students', len(self._students))
-    def deny_all_students(self):
+    # ------------------------------------------------------------------------
+    def deny_all_students(self) -> None:
         '''deny all students to login'''
         logger.info('Denying all students...')
-        self.allowed.clear()
+        for student in self._students.values():
+            if student['state'] == 'allowed':
+                student['state'] = 'offline'
+
+    # ------------------------------------------------------------------------
+    async def insert_new_student(self, uid: str, name: str) -> None:
+        '''insert new student into the database'''
+        with Session(self._engine, future=True) as session:
+            try:
+                session.add(Student(id=uid, name=name, password=''))
+                session.commit()
+            except IntegrityError:
+                logger.warning('"%s" already exists!', uid)
+                session.rollback()
+                return
+        logger.info('New student added: %s %s', uid, name)
+        self._students[uid] = {
+                'name': name,
+                'state': 'offline',
+                'test': await self._testfactory.generate(),
+                }
-    def allow_list(self, filename):
-        '''allow students listed in file (one number per line)'''
+    # ------------------------------------------------------------------------
+    def allow_from_list(self, filename: str) -> None:
+        '''allow students listed in text file (one number per line)'''
+        # parse list of students to allow (one number per line)
         try:
-            with open(filename, 'r') as file:
-                allowed_in_file = {s.strip() for s in file} - {''}
-        except Exception as exc:
+            with open(filename, 'r', encoding='utf-8') as file:
+                allowed = {line.strip() for line in file}
+                allowed.discard('')
+        except OSError as exc:
             error_msg = f'Cannot read file {filename}'
             logger.critical(error_msg)
             raise AppException(error_msg) from exc
-        enrolled = set(s[0] for s in self._get_all_students())  # in database
-        self.allowed.update(allowed_in_file & enrolled)
-        logger.info('Allowed %d students provided in "%s"', len(self.allowed),
-                    filename)
+        # update allowed state (missing are students allowed that don't exist)
+        missing = 0
+        for uid in allowed:
+            try:
+                self.allow_student(uid)
+            except KeyError:
+                logger.warning('Allowed student "%s" does not exist!', uid)
+                missing += 1
-        not_enrolled = allowed_in_file - enrolled
-        if not_enrolled:
-            logger.warning('  but found students not in the database:  %s',
-                           ', '.join(not_enrolled))
+        logger.info('Allowed %d students', len(allowed)-missing)
+        if missing:
+            logger.warning('  %d missing!', missing)
+    # ------------------------------------------------------------------------
     def _focus_student(self, uid):
         '''set student in focus state'''
-        self.unfocus.discard(uid)
+        self._students[uid]['unfocus'] = False
         logger.info('"%s" focus', uid)
+    # ------------------------------------------------------------------------
     def _unfocus_student(self, uid):
         '''set student in unfocus state'''
-        self.unfocus.add(uid)
+        self._students[uid]['unfocus'] = True
         logger.info('"%s" unfocus', uid)
+    # ------------------------------------------------------------------------
     def _set_screen_area(self, uid, sizes):
         '''set current browser area as detected in resize event'''
         scr_y, scr_x, win_y, win_x = sizes
         area = win_x * win_y / (scr_x * scr_y) * 100
-        self.area[uid] = area
+        self._students[uid]['area'] = area
         logger.info('"%s" area=%g%%, window=%dx%d, screen=%dx%d',
                     uid, area, win_x, win_y, scr_x, scr_y)
-
-    async def update_student_password(self, uid, password=''):
-        '''change password on the database'''
-        if password:
-            password = await hash_password(password)
-        with self._db_session() as sess:
-            student = sess.query(Student).filter_by(id=uid).one()
-            student.password = password
-        logger.info('"%s" password updated.', uid)
-
-    def insert_new_student(self, uid, name):
-        '''insert new student into the database'''
-        try:
-            with self._db_session() as sess:
-                sess.add(Student(id=uid, name=name, password=''))
-        except exc.SQLAlchemyError:
-            logger.error('Insert failed: student %s already exists?', uid)
-        else:
-            logger.info('New student: "%s", "%s"', uid, name)
@@ -13,17 +13,18 @@ from concurrent.futures import ThreadPoolExecutor
 # installed packages
 import bcrypt
-import sqlalchemy as sa
-import sqlalchemy.orm
+from sqlalchemy import create_engine, select
+from sqlalchemy.orm import Session
 from sqlalchemy.exc import IntegrityError
 # this project
-from perguntations.models import Base, Student
+from .models import Base, Student
 # ============================================================================
 def parse_commandline_arguments():
     '''Parse command line options'''
+
     parser = argparse.ArgumentParser(
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
         description='Insert new users into a database. Users can be imported '
@@ -104,7 +105,7 @@ def get_students_from_csv(filename):
 # ============================================================================
-def hashpw(student, password=None):
+def hashpw(student, password=None) -> None:
     '''replace password by hash for a single student'''
     print('.', end='', flush=True)
     if password is None:
@@ -115,12 +116,13 @@ def hashpw(student, password=None):
 # ============================================================================
-def insert_students_into_db(session, students):
+def insert_students_into_db(session, students) -> None:
     '''insert list of students into the database'''
     try:
         session.add_all([Student(id=s['uid'], name=s['name'], password=s['pw'])
                          for s in students])
         session.commit()
+
     except IntegrityError:
         print('!!! Integrity error. Users already in database. Aborted !!!\n')
         session.rollback()
@@ -129,11 +131,12 @@ def insert_students_into_db(session, students):
 # ============================================================================
 def show_students_in_database(session, verbose=False):
     '''get students from database'''
-    users = session.query(Student).all()
+    users = session.execute(select(Student)).scalars().all()
+    # users = session.query(Student).all()
+    total = len(users)
-    total_users = len(users)
     print('Registered users:')
-    if total_users == 0:
+    if total == 0:
         print('  -- none --')
     else:
         users.sort(key=lambda u: f'{u.id:>12}')  # sort by number
@@ -142,13 +145,13 @@ def show_students_in_database(session, verbose=False):
                 print(f'{user.id:>12}   {user.name}')
         else:
             print(f'{users[0].id:>12}   {users[0].name}')
-            if total_users > 1:
+            if total > 1:
                 print(f'{users[1].id:>12}   {users[1].name}')
-            if total_users > 3:
+            if total > 3:
                 print('           |   |')
-            if total_users > 2:
+            if total > 2:
                 print(f'{users[-1].id:>12}   {users[-1].name}')
-    print(f'Total: {total_users}.')
+    print(f'Total: {total}.')
 # ============================================================================
@@ -158,39 +161,45 @@ def main():
     args = parse_commandline_arguments()
     # --- database
-    print(f'Using database: {args.db}')
-    engine = sa.create_engine(f'sqlite:///{args.db}', echo=False)
+    print(f'Database: {args.db}')
+    engine = create_engine(f'sqlite:///{args.db}', echo=False, future=True)
     Base.metadata.create_all(engine)  # Criates schema if needed
-    SessionMaker = sqlalchemy.orm.sessionmaker(bind=engine)
-    session = SessionMaker()
+    session = Session(engine, future=True)
     # --- make list of students to insert
-    new_students = []
+    students = []
     if args.admin:
         print('Adding user: 0, Admin.')
-        new_students.append({'uid': '0', 'name': 'Admin'})
+        students.append({'uid': '0', 'name': 'Admin'})
     for csvfile in args.csvfile:
         print('Adding users from:', csvfile)
-        new_students.extend(get_students_from_csv(csvfile))
+        students.extend(get_students_from_csv(csvfile))
+
+    for csvfile in args.csvfile:
+        print('Adding users from:', csvfile)
+        students.extend(get_students_from_csv(csvfile))
     if args.add:
         for uid, name in args.add:
             print(f'Adding user: {uid}, {name}.')
-            new_students.append({'uid': uid, 'name': name})
+            students.append({'uid': uid, 'name': name})
     # --- insert new students
-    if new_students:
+    if students:
         print('Generating password hashes', end='')
-        with ThreadPoolExecutor() as executor:  # hashing in parallel
-            executor.map(lambda s: hashpw(s, args.pw), new_students)
-        print(f'\nInserting {len(new_students)}')
-        insert_students_into_db(session, new_students)
+        with ThreadPoolExecutor() as executor:  # hashing
+            executor.map(lambda s: hashpw(s, args.pw), students)
+        print(f'\nInserting {len(students)}')
+        insert_students_into_db(session, students)
     # --- update all students
     if args.update_all:
-        all_students = session.query(Student).filter(Student.id != '0').all()
+        all_students = session.execute(
+            select(Student).where(Student.id != '0')
+        ).scalars().all()
+
         print(f'Updating password of {len(all_students)} users', end='')
         for student in all_students:
             password = (args.pw or student.id).encode('utf-8')
@@ -203,9 +212,12 @@ def main():
     else:
         for student_id in args.update:
             print(f'Updating password of {student_id}')
-            student = session.query(Student).get(student_id)
-            password = (args.pw or student_id).encode('utf-8')
-            student.password = bcrypt.hashpw(password, bcrypt.gensalt())
+            student = session.execute(
+                select(Student).
+                where(Student.id == student_id)
+            ).scalar_one()
+            new_password = (args.pw or student_id).encode('utf-8')
+            student.password = bcrypt.hashpw(new_password, bcrypt.gensalt())
         session.commit()
     show_students_in_database(session, args.verbose)
 #!/usr/bin/env python3
 '''
-Main file that starts the application and the web server
+Start application and web server
 '''
@@ -10,20 +10,17 @@ import argparse
 import logging
 import logging.config
 import os
-from os import environ, path
 import ssl
 import sys
-# from typing import Any, Dict
 # this project
-from perguntations.app import App, AppException
-from perguntations.serve import run_webserver
-from perguntations.tools import load_yaml
-from perguntations import APP_NAME, APP_VERSION
-
+from .app import App, AppException
+from .serve import run_webserver
+from .tools import load_yaml
+from . import APP_NAME, APP_VERSION
 # ----------------------------------------------------------------------------
-def parse_cmdline_arguments():
+def parse_cmdline_arguments() -> argparse.Namespace:
     '''
     Get command line arguments
     '''
@@ -33,7 +30,6 @@ def parse_cmdline_arguments():
         'included with this software before running the server.')
     parser.add_argument('testfile',
                         type=str,
-                        # nargs='+',  # TODO
                         help='tests in YAML format')
     parser.add_argument('--allow-all',
                         action='store_true',
@@ -44,9 +40,6 @@ def parse_cmdline_arguments():
     parser.add_argument('--debug',
                         action='store_true',
                         help='Enable debug messages')
-    parser.add_argument('--show-ref',
-                        action='store_true',
-                        help='Show question references')
     parser.add_argument('--review',
                         action='store_true',
                         help='Review mode: doesn\'t generate test')
@@ -63,56 +56,50 @@ def parse_cmdline_arguments():
                         help='Show version information and exit')
     return parser.parse_args()
-
 # ----------------------------------------------------------------------------
 def get_logger_config(debug=False) -> dict:
     '''
     Load logger configuration from ~/.config directory if exists,
     otherwise set default paramenters.
     '''
+
+    file = 'logger-debug.yaml' if debug else 'logger.yaml'
+    path = os.path.expanduser(os.environ.get('XDG_CONFIG_HOME', '~/.config/'))
+    try:
+        return load_yaml(os.path.join(path, APP_NAME, file))
+    except OSError:
+        print('Using default logger configuration...')
+
     if debug:
-        filename = 'logger-debug.yaml'
         level = 'DEBUG'
+        fmt = '%(asctime)s %(levelname)-8s %(module)-12s%(lineno)4d| %(message)s'
+        dateformat = ''
     else:
-        filename = 'logger.yaml'
         level = 'INFO'
-
-    config_dir = environ.get('XDG_CONFIG_HOME', '~/.config/')
-    config_file = path.join(path.expanduser(config_dir), APP_NAME, filename)
-
-    default_config = {
-        'version': 1,
-        'formatters': {
-            'standard': {
-                'format': '%(asctime)s %(levelname)-8s %(message)s',
-                'datefmt': '%H:%M:%S',
-                },
-            },
-        'handlers': {
-            'default': {
-                'level': level,
-                'class': 'logging.StreamHandler',
-                'formatter': 'standard',
-                'stream': 'ext://sys.stdout',
+        fmt = '%(asctime)s| %(levelname)-8s| %(message)s'
+        dateformat = '%Y-%m-%d %H:%M:%S'
+    modules = ['main', 'serve', 'app', 'models', 'questions', 'test',
+               'testfactory', 'tools']
+    logger = {'handlers': ['default'], 'level': level, 'propagate': False}
+    return {
+            'version': 1,
+            'formatters': {
+                'standard': {
+                    'format': fmt,
+                    'datefmt': dateformat,
+                    },
                 },
-            },
-        'loggers': {
-            '': {  # configuration for serve.py
-                'handlers': ['default'],
-                'level': level,
+            'handlers': {
+                'default': {
+                    'level': level,
+                    'class': 'logging.StreamHandler',
+                    'formatter': 'standard',
+                    'stream': 'ext://sys.stdout',
+                    },
                 },
-            },
+            'loggers': {f'{APP_NAME}.{module}': logger for module in modules}
         }
-    modules = ['app', 'models', 'questions', 'test', 'testfactory', 'tools']
-    logger = {'handlers': ['default'], 'level': level, 'propagate': False}
-
-    default_config['loggers'].update({f'{APP_NAME}.{module}': logger
-                                      for module in modules})
-
-    return load_yaml(config_file, default=default_config)
-
-
 # ----------------------------------------------------------------------------
 def main() -> None:
     '''
@@ -122,15 +109,16 @@ def main() -&gt; None:
     # --- Setup logging ------------------------------------------------------
     logging.config.dictConfig(get_logger_config(args.debug))
-    logging.info('====================== Start Logging ======================')
+    logger = logging.getLogger(__name__)
+
+    logger.info('================== Start Logging ==================')
     # --- start application --------------------------------------------------
     config = {
         'testfile': args.testfile,
-        'debug': args.debug,
         'allow_all': args.allow_all,
         'allow_list': args.allow_list,
-        'show_ref': args.show_ref,
+        'debug': args.debug,
         'review': args.review,
         'correct': args.correct,
         }
@@ -138,24 +126,24 @@ def main() -&gt; None:
     try:
         app = App(config)
     except AppException:
-        logging.critical('Failed to start application.')
-        sys.exit(-1)
+        logger.critical('Failed to start application!')
+        sys.exit(1)
     # --- get SSL certificates -----------------------------------------------
     if 'XDG_DATA_HOME' in os.environ:
-        certs_dir = path.join(os.environ['XDG_DATA_HOME'], 'certs')
+        certs_dir = os.path.join(os.environ['XDG_DATA_HOME'], 'certs')
     else:
-        certs_dir = path.expanduser('~/.local/share/certs')
+        certs_dir = os.path.expanduser('~/.local/share/certs')
     ssl_opt = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
     try:
-        ssl_opt.load_cert_chain(path.join(certs_dir, 'cert.pem'),
-                                path.join(certs_dir, 'privkey.pem'))
+        ssl_opt.load_cert_chain(os.path.join(certs_dir, 'cert.pem'),
+                                os.path.join(certs_dir, 'privkey.pem'))
     except FileNotFoundError:
-        logging.critical('SSL certificates missing in %s', certs_dir)
-        sys.exit(-1)
+        logger.critical('SSL certificates missing in %s', certs_dir)
+        sys.exit(1)
-    # --- run webserver ----------------------------------------------------
+    # --- run webserver ------------------------------------------------------
     run_webserver(app=app, ssl_opt=ssl_opt, port=args.port, debug=args.debug)
 '''
+perguntations/models.py
 SQLAlchemy ORM
-
-The classes below correspond to database tables
 '''
+from typing import Any
 from sqlalchemy import Column, ForeignKey, Integer, Float, String
-from sqlalchemy.ext.declarative import declarative_base
-from sqlalchemy.orm import relationship
+from sqlalchemy.orm import declarative_base, relationship
-# ============================================================================
-#   Declare ORM
-Base = declarative_base()
+# FIXME Any is a workaround for static type checking
+# (https://github.com/python/mypy/issues/6372)
+Base: Any = declarative_base()
 # ----------------------------------------------------------------------------
@@ -26,11 +25,11 @@ class Student(Base):
     # ---
     tests = relationship('Test', back_populates='student')
-    def __str__(self):
-        return (f'Student:\n'
-                f'  id:         "{self.id}"\n'
-                f'  name:       "{self.name}"\n'
-                f'  password:   "{self.password}"\n')
+    def __repr__(self):
+        return (f'Student('
+            f'id={self.id!r}, '
+            f'name={self.name!r}, '
+            f'password={self.password!r})')
 # ----------------------------------------------------------------------------
@@ -52,18 +51,18 @@ class Test(Base):
     student = relationship('Student', back_populates='tests')
     questions = relationship('Question', back_populates='test')
-    def __str__(self):
-        return (f'Test:\n'
-                f'  id:         {self.id}\n'
-                f'  ref:        "{self.ref}"\n'
-                f'  title:      "{self.title}"\n'
-                f'  grade:      {self.grade}\n'
-                f'  state:      "{self.state}"\n'
-                f'  comment:    "{self.comment}"\n'
-                f'  starttime:  "{self.starttime}"\n'
-                f'  finishtime: "{self.finishtime}"\n'
-                f'  filename:   "{self.filename}"\n'
-                f'  student_id: "{self.student_id}"\n')
+    def __repr__(self):
+        return (f'Test('
+                f'id={self.id!r}, '
+                f'ref={self.ref!r}, '
+                f'title={self.title!r}, '
+                f'grade={self.grade!r}, '
+                f'state={self.state!r}, '
+                f'comment={self.comment!r}, '
+                f'starttime={self.starttime!r}, '
+                f'finishtime={self.finishtime!r}, '
+                f'filename={self.filename!r}, '
+                f'student_id={self.student_id!r})')
 # ---------------------------------------------------------------------------
@@ -82,13 +81,13 @@ class Question(Base):
     # ---
     test = relationship('Test', back_populates='questions')
-    def __str__(self):
-        return (f'Question:\n'
-                f'  id:         {self.id}\n'
-                f'  number:     {self.number}\n'
-                f'  ref:        "{self.ref}"\n'
-                f'  grade:      {self.grade}\n'
-                f'  comment:    "{self.comment}"\n'
-                f'  starttime:  "{self.starttime}"\n'
-                f'  finishtime: "{self.finishtime}"\n'
-                f'  test_id:    "{self.test_id}"\n')
+    def __repr__(self):
+        return (f'Question('
+                f'id={self.id!r}, '
+                f'number={self.number!r}, '
+                f'ref={self.ref!r}, '
+                f'grade={self.grade!r}, '
+                f'comment={self.comment!r}, '
+                f'starttime={self.starttime!r}, '
+                f'finishtime={self.finishtime!r}, '
+                f'test_id={self.test_id!r})')
@@ -137,9 +137,9 @@ class HighlightRenderer(mistune.Renderer):
         return '<table class="table table-sm"><thead class="thead-light">' \
                + header + '</thead><tbody>' + body + '</tbody></table>'
-    def image(self, src, title, alt):
+    def image(self, src, title, text):
         '''render image'''
-        alt = mistune.escape(alt, quote=True)
+        alt = mistune.escape(text, quote=True)
         if title is not None:
             if title:  # not empty string, show as caption
                 title = mistune.escape(title, quote=True)
 '''
-Classes the implement several types of questions.
+File:           perguntations/questions.py
+Description:    Classes the implement several types of questions.
 '''
 # python standard library
-import asyncio
 from datetime import datetime
 import logging
-from os import path
+import os
 import random
 import re
 from typing import Any, Dict, NewType
 import uuid
-
-# from urllib.error import HTTPError
-# import json
-# import http.client
-
-
 # this project
-from perguntations.tools import run_script, run_script_async
+from .tools import run_script, run_script_async
 # setup logger for this module
 logger = logging.getLogger(__name__)
-
 QDict = NewType('QDict', Dict[str, Any])
-
-
 class QuestionException(Exception):
     '''Exceptions raised in this module'''
@@ -45,8 +36,6 @@ class Question(dict):
     for each student.
     Instances can shuffle options or automatically generate questions.
     '''
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
     def gen(self) -> None:
         '''
@@ -96,9 +85,6 @@ class QuestionRadio(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         '''
         Sets defaults, performs checks and generates the actual question
@@ -128,8 +114,7 @@ class QuestionRadio(Question):
         # e.g.  correct: 2 --> correct: [0,0,1,0,0]
         if isinstance(self['correct'], int):
             if not 0 <= self['correct'] < nopts:
-                msg = (f'`correct` out of range 0..{nopts-1}. '
-                       f'In question "{self["ref"]}"')
+                msg = f'"{self["ref"]}": correct out of range 0..{nopts-1}'
                 logger.error(msg)
                 raise QuestionException(msg)
@@ -139,8 +124,7 @@ class QuestionRadio(Question):
         elif isinstance(self['correct'], list):
             # must match number of options
             if len(self['correct']) != nopts:
-                msg = (f'{nopts} options vs {len(self["correct"])} correct. '
-                       f'In question "{self["ref"]}"')
+                msg = f'"{self["ref"]}": number of options/correct mismatch'
                 logger.error(msg)
                 raise QuestionException(msg)
@@ -148,23 +132,20 @@ class QuestionRadio(Question):
             try:
                 self['correct'] = [float(x) for x in self['correct']]
             except (ValueError, TypeError) as exc:
-                msg = ('`correct` must be list of numbers or booleans.'
-                       f'In "{self["ref"]}"')
+                msg = f'"{self["ref"]}": correct must contain floats or bools'
                 logger.error(msg)
                 raise QuestionException(msg) from exc
             # check grade boundaries
             if self['discount'] and not all(0.0 <= x <= 1.0
                                             for x in self['correct']):
-                msg = ('`correct` values must be in the interval [0.0, 1.0]. '
-                       f'In "{self["ref"]}"')
+                msg = f'"{self["ref"]}": correct must be in [0.0, 1.0]'
                 logger.error(msg)
                 raise QuestionException(msg)
             # at least one correct option
             if all(x < 1.0 for x in self['correct']):
-                msg = ('At least one correct option is required. '
-                       f'In "{self["ref"]}"')
+                msg = f'"{self["ref"]}": has no correct options'
                 logger.error(msg)
                 raise QuestionException(msg)
@@ -231,9 +212,6 @@ class QuestionCheckbox(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
@@ -288,19 +266,6 @@ class QuestionCheckbox(Question):
                    f'Please fix "{self["ref"]}" in "{self["path"]}"')
             logger.error(msg)
             raise QuestionException(msg)
-            # msg0 = ('+--------------- BEHAVIOR CHANGE NOTICE ---------------+')
-            # msg1 = ('| Correct values in checkbox questions must be in the  |')
-            # msg2 = ('| interval [0.0, 1.0]. I will convert to the new       |')
-            # msg3 = ('| behavior, for now, but you should fix it.            |')
-            # msg4 = ('+------------------------------------------------------+')
-            # logger.warning(msg0)
-            # logger.warning(msg1)
-            # logger.warning(msg2)
-            # logger.warning(msg3)
-            # logger.warning(msg4)
-            # logger.warning('please fix "%s"', self["ref"])
-            # # normalize to [0,1]
-            # self['correct'] = [(x+1)/2 for x in self['correct']]
         # if an option is a list of (right, wrong), pick one
         options = []
@@ -356,9 +321,6 @@ class QuestionText(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
         self.set_defaults(QDict({
@@ -389,12 +351,13 @@ class QuestionText(Question):
     def transform(self, ans):
         '''apply optional filters to the answer'''
+        # apply transformations in sequence
         for transform in self['transform']:
             if transform == 'remove_space':  # removes all spaces
                 ans = ans.replace(' ', '')
             elif transform == 'trim':        # removes spaces around
                 ans = ans.strip()
-            elif transform == 'normalize_space':  # replaces multiple spaces by one
+            elif transform == 'normalize_space':  # replaces many spaces by one
                 ans = re.sub(r'\s+', ' ', ans.strip())
             elif transform == 'lower':       # convert to lowercase
                 ans = ans.lower()
@@ -410,7 +373,7 @@ class QuestionText(Question):
         super().correct()
         if self['answer'] is not None:
-            answer = self.transform(self['answer'])  # apply transformations
+            answer = self.transform(self['answer'])
             self['grade'] = 1.0 if answer in self['correct'] else 0.0
@@ -427,9 +390,6 @@ class QuestionTextRegex(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
@@ -442,14 +402,6 @@ class QuestionTextRegex(Question):
         if not isinstance(self['correct'], list):
             self['correct'] = [self['correct']]
-        # converts patterns to compiled versions
-        # try:
-        #     self['correct'] = [re.compile(a) for a in self['correct']]
-        # except Exception as exc:
-        #     msg = f'Failed to compile regex in "{self["ref"]}"'
-        #     logger.error(msg)
-        #     raise QuestionException(msg) from exc
-
     # ------------------------------------------------------------------------
     def correct(self) -> None:
         super().correct()
@@ -464,15 +416,6 @@ class QuestionTextRegex(Question):
                                  regex, self['answer'])
             self['grade'] = 0.0
-                # try:
-                #     if regex.match(self['answer']):
-                #         self['grade'] = 1.0
-                #         return
-                # except TypeError:
-                #     logger.error('While matching regex %s with answer "%s".',
-                #                  regex.pattern, self["answer"])
-
-
 # ============================================================================
 class QuestionNumericInterval(Question):
     '''An instance of QuestionTextNumeric will always have the keys:
@@ -484,9 +427,6 @@ class QuestionNumericInterval(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
@@ -548,9 +488,6 @@ class QuestionTextArea(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
@@ -561,7 +498,7 @@ class QuestionTextArea(Question):
             'args': []
             }))
-        self['correct'] = path.join(self['path'], self['correct'])
+        self['correct'] = os.path.join(self['path'], self['correct'])
     # ------------------------------------------------------------------------
     def correct(self) -> None:
@@ -625,129 +562,6 @@ class QuestionTextArea(Question):
 # ============================================================================
-# class QuestionCode(Question):
-#     '''
-#     Submits answer to a JOBE server to compile and run against the test cases.
-#     '''
-
-#     _outcomes = {
-#         0:  'JOBE outcome: Successful run',
-#         11: 'JOBE outcome: Compile error',
-#         12: 'JOBE outcome: Runtime error',
-#         13: 'JOBE outcome: Time limit exceeded',
-#         15: 'JOBE outcome: Successful run',
-#         17: 'JOBE outcome: Memory limit exceeded',
-#         19: 'JOBE outcome: Illegal system call',
-#         20: 'JOBE outcome: Internal error, please report',
-#         21: 'JOBE outcome: Server overload',
-#         }
-
-#     # ------------------------------------------------------------------------
-#     def __init__(self, q: QDict) -> None:
-#         super().__init__(q)
-
-#         self.set_defaults(QDict({
-#             'text': '',
-#             'timeout': 5,   # seconds
-#             'server': '127.0.0.1',  # JOBE server
-#             'language': 'c',
-#             'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
-#             }))
-
-    # ------------------------------------------------------------------------
-    # def correct(self) -> None:
-    #     super().correct()
-
-    #     if self['answer'] is None:
-    #         return
-
-    #     # submit answer to JOBE server
-    #     resource = '/jobe/index.php/restapi/runs/'
-    #     headers = {"Content-type": "application/json; charset=utf-8",
-    #                "Accept": "application/json"}
-
-    #     for expected in self['correct']:
-    #         data_json = json.dumps({
-    #             'run_spec' : {
-    #                 'language_id': self['language'],
-    #                 'sourcecode': self['answer'],
-    #                 'input': expected.get('stdin', ''),
-    #                  },
-    #             })
-
-    #         try:
-    #             connect = http.client.HTTPConnection(self['server'])
-    #             connect.request(
-    #                 method='POST',
-    #                 url=resource,
-    #                 body=data_json,
-    #                 headers=headers
-    #                 )
-    #             response = connect.getresponse()
-    #             logger.debug('JOBE response status %d', response.status)
-    #             if response.status != 204:
-    #                 content = response.read().decode('utf8')
-    #                 if content:
-    #                     result = json.loads(content)
-    #             connect.close()
-
-    #         except (HTTPError, ValueError):
-    #             logger.error('HTTPError while connecting to JOBE server')
-
-    #         try:
-    #             outcome = result['outcome']
-    #         except (NameError, TypeError, KeyError):
-    #             logger.error('Bad result returned from JOBE server: %s', result)
-    #             return
-    #         logger.debug(self._outcomes[outcome])
-
-
-
-    #         if result['cmpinfo']:  # compiler errors and warnings
-    #             self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
-    #             self['grade'] = 0.0
-    #             return
-
-    #         if result['stdout'] != expected.get('stdout', ''):
-    #             self['comments'] = 'O output gerado é diferente do esperado.'  # FIXME mostrar porque?
-    #             self['grade'] = 0.0
-    #             return
-
-    #     self['comments'] = 'Ok!'
-    #     self['grade'] = 1.0
-
-
-    # # ------------------------------------------------------------------------
-    # async def correct_async(self) -> None:
-    #     self.correct()  # FIXME there is no async correction!!!
-
-
-            # out = run_script(
-            #     script=self['correct'],
-            #     args=self['args'],
-            #     stdin=self['answer'],
-            #     timeout=self['timeout']
-            #     )
-
-            # if out is None:
-            #     logger.warning('No grade after running "%s".', self["correct"])
-            #     self['comments'] = 'O programa de correcção abortou...'
-            #     self['grade'] = 0.0
-            # elif isinstance(out, dict):
-            #     self['comments'] = out.get('comments', '')
-            #     try:
-            #         self['grade'] = float(out['grade'])
-            #     except ValueError:
-            #         logger.error('Output error in "%s".', self["correct"])
-            #     except KeyError:
-            #         logger.error('No grade in "%s".', self["correct"])
-            # else:
-            #     try:
-            #         self['grade'] = float(out)
-            #     except (TypeError, ValueError):
-            #         logger.error('Invalid grade in "%s".', self["correct"])
-
-# ============================================================================
 class QuestionInformation(Question):
     '''
     Not really a question, just an information panel.
@@ -755,9 +569,6 @@ class QuestionInformation(Question):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, q: QDict) -> None:
-    #     super().__init__(q)
-
     def gen(self) -> None:
         super().gen()
         self.set_defaults(QDict({
@@ -770,7 +581,6 @@ class QuestionInformation(Question):
         self['grade'] = 1.0  # always "correct" but points should be zero!
-
 # ============================================================================
 def question_from(qdict: QDict) -> Question:
     '''
@@ -783,7 +593,6 @@ def question_from(qdict: QDict) -&gt; Question:
         'text-regex':       QuestionTextRegex,
         'numeric-interval': QuestionNumericInterval,
         'textarea':         QuestionTextArea,
-        # 'code':             QuestionCode,
         # -- informative panels --
         'information':      QuestionInformation,
         'success':          QuestionInformation,
@@ -856,17 +665,17 @@ class QFactory():
         logger.debug('generating %s...', self.qdict["ref"])
         # Shallow copy so that script generated questions will not replace
         # the original generators
-        qdict = self.qdict.copy()
+        qdict = QDict(self.qdict.copy())
         qdict['qid'] = str(uuid.uuid4())  # unique for each question
         # If question is of generator type, an external program will be run
         # which will print a valid question in yaml format to stdout. This
         # output is then yaml parsed into a dictionary `q`.
         if qdict['type'] == 'generator':
-            logger.debug(' \\_ Running "%s".', qdict['script'])
+            logger.debug(' \\_ Running "%s"', qdict['script'])
             qdict.setdefault('args', [])
             qdict.setdefault('stdin', '')
-            script = path.join(qdict['path'], qdict['script'])
+            script = os.path.join(qdict['path'], qdict['script'])
             out = await run_script_async(script=script,
                                          args=qdict['args'],
                                          stdin=qdict['stdin'])
@@ -875,8 +684,3 @@ class QFactory():
         question = question_from(qdict)  # returns a Question instance
         question.gen()
         return question
-
-    # ------------------------------------------------------------------------
-    def generate(self) -> Question:
-        '''generate question (synchronous version)'''
-        return asyncio.get_event_loop().run_until_complete(self.gen_async())
@@ -10,23 +10,27 @@ import asyncio
 import base64
 import functools
 import json
-import logging.config
+import logging
 import mimetypes
 from os import path
 import re
 import signal
 import sys
 from timeit import default_timer as timer
+from typing import Dict, Tuple
 import uuid
 # user installed libraries
 import tornado.ioloop
 import tornado.web
-# import tornado.websocket
 import tornado.httpserver
 # this project
-from perguntations.parser_markdown import md_to_html
+from .parser_markdown import md_to_html
+
+
+# setup logger for this module
+logger = logging.getLogger(__name__)
 # ----------------------------------------------------------------------------
@@ -41,8 +45,6 @@ class WebApplication(tornado.web.Application):
             (r'/review', ReviewHandler),
             (r'/admin', AdminHandler),
             (r'/file', FileHandler),
-            # (r'/root', MainHandler),
-            # (r'/ws', AdminSocketHandler),
             (r'/adminwebservice', AdminWebservice),
             (r'/studentwebservice', StudentWebservice),
             (r'/', RootHandler),
@@ -64,11 +66,10 @@ class WebApplication(tornado.web.Application):
 # ----------------------------------------------------------------------------
 def admin_only(func):
     '''
-    Decorator used to restrict access to the administrator.
-    Example:
+    Decorator to restrict access to the administrator:
-        @admin_only
-        def get(self): ...
+    @admin_only
+    def get(self):
     '''
     @functools.wraps(func)
     async def wrapper(self, *args, **kwargs):
@@ -92,6 +93,11 @@ class BaseHandler(tornado.web.RequestHandler):
         '''simplifies access to the application a little bit'''
         return self.application.testapp
+    # @property
+    # def debug(self) -> bool:
+    #     '''check if is running in debug mode'''
+    #     return self.application.testapp.debug
+
     def get_current_user(self):
         '''
         Since HTTP is stateless, a cookie is used to identify the user.
@@ -104,72 +110,15 @@ class BaseHandler(tornado.web.RequestHandler):
 # ----------------------------------------------------------------------------
-# class MainHandler(BaseHandler):
-
-#     @tornado.web.authenticated
-#     @admin_only
-#     def get(self):
-#         self.render("admin-ws.html", students=self.testapp.get_students_state())
-
-
-# # ----------------------------------------------------------------------------
-# class AdminSocketHandler(tornado.websocket.WebSocketHandler):
-#     waiters = set()
-#     # cache = []
-
-#     # def get_compression_options(self):
-#     #     return {}   # Non-None enables compression with default options.
-
-#     # called when opening connection
-#     def open(self):
-#         logging.debug('[AdminSocketHandler.open]')
-#         AdminSocketHandler.waiters.add(self)
-
-#     # called when closing connection
-#     def on_close(self):
-#         logging.debug('[AdminSocketHandler.on_close]')
-#         AdminSocketHandler.waiters.remove(self)
-
-#     # @classmethod
-#     # def update_cache(cls, chat):
-#     #     logging.debug(f'[AdminSocketHandler.update_cache] "{chat}"')
-#     #     cls.cache.append(chat)
-
-#     # @classmethod
-#     # def send_updates(cls, chat):
-#     #     logging.info("sending message to %d waiters", len(cls.waiters))
-#     #     for waiter in cls.waiters:
-#     #         try:
-#     #             waiter.write_message(chat)
-#     #         except Exception:
-#     #             logging.error("Error sending message", exc_info=True)
-
-#     # handle incomming messages
-#     def on_message(self, message):
-#         logging.info(f"[AdminSocketHandler.onmessage] got message {message}")
-#         parsed = tornado.escape.json_decode(message)
-#         print(parsed)
-#         chat = {"id": str(uuid.uuid4()), "body": parsed["body"]}
-#         print(chat)
-#         chat["html"] = tornado.escape.to_basestring(
-#             '<div>' + chat['body'] + '</div>'
-#             # self.render_string("message.html", message=chat)
-#         )
-#         print(chat)
-
-#         AdminSocketHandler.update_cache(chat)  # store msgs
-#         AdminSocketHandler.send_updates(chat)  # send to clients
-
-# ----------------------------------------------------------------------------
 # pylint: disable=abstract-method
 class LoginHandler(BaseHandler):
     '''Handles /login'''
     _prefix = re.compile(r'[a-z]')
     _error_msg = {
-        'wrong_password': 'Password errada',
-        'already_online': 'Já está online, não pode entrar duas vezes',
-        'unauthorized': 'Não está autorizado a fazer o teste'
+        'wrong_password': 'Senha errada',
+        'not_allowed': 'Não está autorizado a fazer o teste',
+        'nonexistent': 'Número de aluno inválido'
     }
     def get(self):
@@ -178,7 +127,7 @@ class LoginHandler(BaseHandler):
     async def post(self):
         '''Authenticates student and login.'''
-        uid = self._prefix.sub('', self.get_body_argument('uid'))
+        uid = self.get_body_argument('uid')
         password = self.get_body_argument('pw')
         headers = {
             'remote_ip': self.request.remote_ip,
@@ -187,7 +136,7 @@ class LoginHandler(BaseHandler):
         error = await self.testapp.login(uid, password, headers)
-        if error:
+        if error is not None:
             await asyncio.sleep(3)  # delay to avoid spamming the server...
             self.render('login.html', error=self._error_msg[error])
         else:
@@ -203,8 +152,8 @@ class LogoutHandler(BaseHandler):
     @tornado.web.authenticated
     def get(self):
         '''Logs out a user.'''
-        self.clear_cookie('perguntations_user')
         self.testapp.logout(self.current_user)
+        self.clear_cookie('perguntations_user')
         self.render('login.html', error='')
@@ -214,7 +163,7 @@ class LogoutHandler(BaseHandler):
 # pylint: disable=abstract-method
 class RootHandler(BaseHandler):
     '''
-    Generates test to student.
+    Presents test to student.
     Receives answers, corrects the test and sends back the grade.
     Redirects user 0 to /admin.
     '''
@@ -227,7 +176,6 @@ class RootHandler(BaseHandler):
         'text-regex':   'question-text.html',
         'numeric-interval': 'question-text.html',
         'textarea':     'question-textarea.html',
-        'code':         'question-textarea.html',
         # -- information panels --
         'information':  'question-information.html',
         'success':      'question-information.html',
@@ -243,77 +191,67 @@ class RootHandler(BaseHandler):
         Sends test to student or redirects 0 to admin page.
         Multiple calls to this function will return the same test.
         '''
-
         uid = self.current_user
-        logging.debug('"%s" GET /', uid)
+        logger.debug('"%s" GET /', uid)
         if uid == '0':
             self.redirect('/admin')
-            return
-
-        test = await self.testapp.get_test_or_generate(uid)
-        self.render('test.html', t=test, md=md_to_html, templ=self._templates)
-
+        else:
+            test = self.testapp.get_test(uid)
+            name = self.testapp.get_name(uid)
+            self.render('test.html', t=test, uid=uid, name=name, md=md_to_html,
+                        templ=self._templates, debug=self.testapp.debug)
     # --- POST
     @tornado.web.authenticated
     async def post(self):
         '''
         Receives answers, fixes some html weirdness, corrects test and
-        sends back the grade.
+        renders the grade.
         self.request.arguments = {'answered-0': [b'on'], '0': [b'13.45']}
-        builds dictionary ans={0: 'answer0', 1:, 'answer1', ...}
-        unanswered questions not included.
+        builds dictionary ans = {0: 'answer0', 1:, 'answer1', ...}
+        unanswered questions are not included.
         '''
-        timeit_start = timer()  # performance timer
+        starttime = timer()  # performance timer
         uid = self.current_user
-        logging.debug('"%s" POST /', uid)
+        logger.debug('"%s" POST /', uid)
-        try:
-            test = self.testapp.get_test(uid)
-        except KeyError as exc:
-            logging.warning('"%s" POST / raised 403 Forbidden', uid)
-            raise tornado.web.HTTPError(403) from exc # Forbidden
+        test = self.testapp.get_test(uid)
+        if test is None:
+            logger.warning('"%s" submitted but no test running - Err 403', uid)
+            raise tornado.web.HTTPError(403)  # Forbidden
         ans = {}
         for i, question in enumerate(test['questions']):
             qid = str(i)
-            if 'answered-' + qid in self.request.arguments:
+            if f'answered-{qid}' in self.request.arguments:
                 ans[i] = self.get_body_arguments(qid)
                 # remove enclosing list in some question types
                 if question['type'] == 'radio':
-                    if not ans[i]:
-                        ans[i] = None
-                    else:
-                        ans[i] = ans[i][0]
+                    ans[i] = ans[i][0] if ans[i] else None
                 elif question['type'] in ('text', 'text-regex', 'textarea',
-                                          'numeric-interval', 'code'):
+                                          'numeric-interval'):
                     ans[i] = ans[i][0]
         # submit answered questions, correct
         await self.testapp.submit_test(uid, ans)
-        # show final grade and grades of other tests in the database
-        # allgrades = self.testapp.get_student_grades_from_all_tests(uid)
-        # grade = self.testapp.get_student_grade(uid)
-
-        self.render('grade.html', t=test)
+        name = self.testapp.get_name(uid)
+        self.render('grade.html', t=test, uid=uid, name=name)
         self.clear_cookie('perguntations_user')
         self.testapp.logout(uid)
-
-        timeit_finish = timer()
-        logging.info('  elapsed time: %fs', timeit_finish-timeit_start)
+        logger.info('  elapsed time: %fs', timer() - starttime)
 # ----------------------------------------------------------------------------
 # pylint: disable=abstract-method
 class StudentWebservice(BaseHandler):
     '''
-    Receive ajax from students in the test in response from focus, unfocus and
-    resize events.
+    Receive ajax from students during the test in response to the events
+    focus, unfocus and resize.
     '''
     @tornado.web.authenticated
@@ -321,8 +259,9 @@ class StudentWebservice(BaseHandler):
         '''handle ajax post'''
         uid = self.current_user
         cmd = self.get_body_argument('cmd', None)
-        value = json.loads(self.get_body_argument('value', None))
-        self.testapp.event_test(uid, cmd, value)
+        value = self.get_body_argument('value', None)
+        if cmd is not None and value is not None:
+            self.testapp.register_event(uid, cmd, json.loads(value))
 # ----------------------------------------------------------------------------
@@ -337,16 +276,17 @@ class AdminWebservice(BaseHandler):
     async def get(self):
         '''admin webservices that do not change state'''
         cmd = self.get_query_argument('cmd')
+        logger.debug('GET /adminwebservice %s', cmd)
+
         if cmd == 'testcsv':
-            test_ref, data = self.testapp.get_test_csv()
+            test_ref, data = self.testapp.get_grades_csv()
             self.set_header('Content-Type', 'text/csv')
             self.set_header('content-Disposition',
                             f'attachment; filename={test_ref}.csv')
             self.write(data)
             await self.flush()
-
-        if cmd == 'questionscsv':
-            test_ref, data = self.testapp.get_questions_csv()
+        elif cmd == 'questionscsv':
+            test_ref, data = self.testapp.get_detailed_grades_csv()
             self.set_header('Content-Type', 'text/csv')
             self.set_header('content-Disposition',
                             f'attachment; filename={test_ref}-detailed.csv')
@@ -359,6 +299,7 @@ class AdminWebservice(BaseHandler):
 class AdminHandler(BaseHandler):
     '''Handle /admin'''
+    # --- GET
     @tornado.web.authenticated
     @admin_only
     async def get(self):
@@ -366,24 +307,18 @@ class AdminHandler(BaseHandler):
         Admin page.
         '''
         cmd = self.get_query_argument('cmd', default=None)
+        logger.debug('GET /admin (cmd=%s)', cmd)
-        if cmd == 'students_table':
-            data = {'data': self.testapp.get_students_state()}
-            self.write(json.dumps(data, default=str))
+        if cmd is None:
+            self.render('admin.html')
         elif cmd == 'test':
-            data = {
-                'data': {
-                    'title': self.testapp.testfactory['title'],
-                    'ref': self.testapp.testfactory['ref'],
-                    'filename': self.testapp.testfactory['testfile'],
-                    'database': self.testapp.testfactory['database'],
-                    'answers_dir': self.testapp.testfactory['answers_dir'],
-                    }
-                }
+            data = { 'data': self.testapp.get_test_config() }
+            self.write(json.dumps(data, default=str))
+        elif cmd == 'students_table':
+            data = {'data': self.testapp.get_students_state()}
             self.write(json.dumps(data, default=str))
-        else:
-            self.render('admin.html')
+    # --- POST
     @tornado.web.authenticated
     @admin_only
     async def post(self):
@@ -392,6 +327,7 @@ class AdminHandler(BaseHandler):
         '''
         cmd = self.get_body_argument('cmd', None)
         value = self.get_body_argument('value', None)
+        logger.debug('POST /admin (cmd=%s, value=%s)')
         if cmd == 'allow':
             self.testapp.allow_student(value)
@@ -402,15 +338,11 @@ class AdminHandler(BaseHandler):
         elif cmd == 'deny_all':
             self.testapp.deny_all_students()
         elif cmd == 'reset_password':
-            await self.testapp.update_student_password(uid=value, password='')
-
-        elif cmd == 'insert_student':
+            await self.testapp.set_password(uid=value, pw='')
+        elif cmd == 'insert_student' and value is not None:
             student = json.loads(value)
-            self.testapp.insert_new_student(uid=student['number'],
-                                            name=student['name'])
-
-        else:
-            logging.error('Unknown command: "%s"', cmd)
+            await self.testapp.insert_new_student(uid=student['number'],
+                                                  name=student['name'])
 # ----------------------------------------------------------------------------
@@ -422,6 +354,7 @@ class FileHandler(BaseHandler):
     Handles static files from questions like images, etc.
     '''
+    _filecache: Dict[Tuple[str, str], bytes] = {}
     @tornado.web.authenticated
     async def get(self):
@@ -429,40 +362,47 @@ class FileHandler(BaseHandler):
         Returns requested file. Files are obtained from the 'public' directory
         of each question.
         '''
-
         uid = self.current_user
         ref = self.get_query_argument('ref', None)
         image = self.get_query_argument('image', None)
+        logger.debug('GET /file (ref=%s, image=%s)', ref, image)
+
+        if ref is None or image is None:
+            return
+
         content_type = mimetypes.guess_type(image)[0]
-        if uid != '0':
-            test = self.testapp.get_student_test(uid)
-        else:
-            logging.error('FIXME Cannot serve images for review.')
-            raise tornado.web.HTTPError(404)  # FIXME admin
+        if (ref, image) in self._filecache:
+            logger.debug('using cached file')
+            self.write(self._filecache[(ref, image)])
+            if content_type is not None:
+                self.set_header("Content-Type", content_type)
+            await self.flush()
+            return
-        if test is None:
-            raise tornado.web.HTTPError(404)  # Not Found
+        try:
+            test = self.testapp.get_test(uid)
+        except KeyError:
+            logger.warning('Could not get test to serve image file')
+            raise tornado.web.HTTPError(404) from None  # Not Found
+        # search for the question that contains the image
         for question in test['questions']:
-            # search for the question that contains the image
             if question['ref'] == ref:
                 filepath = path.join(question['path'], 'public', image)
+
                 try:
-                    file = open(filepath, 'rb')
-                except FileNotFoundError:
-                    logging.error('File not found: %s', filepath)
-                except PermissionError:
-                    logging.error('No permission: %s', filepath)
+                    with open(filepath, 'rb') as file:
+                        data = file.read()
                 except OSError:
-                    logging.error('Error opening file: %s', filepath)
-                else:
-                    data = file.read()
-                    file.close()
+                    logger.error('Error reading file "%s"', filepath)
+                    return
+                self._filecache[(ref, image)] = data
+                self.write(data)
+                if content_type is not None:
                     self.set_header("Content-Type", content_type)
-                    self.write(data)
-                    await self.flush()
-                break
+                await self.flush()
+                return
 # --- REVIEW -----------------------------------------------------------------
@@ -479,7 +419,6 @@ class ReviewHandler(BaseHandler):
         'text-regex':   'review-question-text.html',
         'numeric-interval': 'review-question-text.html',
         'textarea':     'review-question-text.html',
-        'code':         'review-question-text.html',
         # -- information panels --
         'information':  'review-question-information.html',
         'success':      'review-question-information.html',
@@ -494,26 +433,28 @@ class ReviewHandler(BaseHandler):
         Opens JSON file with a given corrected test and renders it
         '''
         test_id = self.get_query_argument('test_id', None)
-        logging.info('Review test %s.', test_id)
+        logger.info('Review test %s.', test_id)
         fname = self.testapp.get_json_filename_of_test(test_id)
         if fname is None:
             raise tornado.web.HTTPError(404)  # Not Found
         try:
-            with open(path.expanduser(fname)) as jsonfile:
+            with open(path.expanduser(fname), encoding='utf-8') as jsonfile:
                 test = json.load(jsonfile)
         except OSError:
             msg = f'Cannot open "{fname}" for review.'
-            logging.error(msg)
+            logger.error(msg)
             raise tornado.web.HTTPError(status_code=404, reason=msg) from None
         except json.JSONDecodeError as exc:
             msg = f'JSON error in "{fname}": {exc}'
-            logging.error(msg)
+            logger.error(msg)
             raise tornado.web.HTTPError(status_code=404, reason=msg)
-        self.render('review.html', t=test, md=md_to_html,
-                    templ=self._templates)
+        uid = test['student']
+        name = self.testapp.get_name(uid)
+        self.render('review.html', t=test, uid=uid, name=name, md=md_to_html,
+                    templ=self._templates, debug=self.testapp.debug)
 # ----------------------------------------------------------------------------
@@ -524,7 +465,7 @@ def signal_handler(*_):
     reply = input(' --> Stop webserver? (yes/no) ')
     if reply.lower() == 'yes':
         tornado.ioloop.IOLoop.current().stop()
-        logging.critical('Webserver stopped.')
+        logger.critical('Webserver stopped.')
         sys.exit(0)
 # ----------------------------------------------------------------------------
@@ -534,33 +475,33 @@ def run_webserver(app, ssl_opt, port, debug):
     '''
     # --- create web application
-    logging.info('-----------------------------------------------------------')
-    logging.info('Starting WebApplication (tornado)')
+    logger.info('-------- Starting WebApplication (tornado) --------')
     try:
         webapp = WebApplication(app, debug=debug)
     except Exception:
-        logging.critical('Failed to start web application.')
+        logger.critical('Failed to start web application.')
         raise
+    # --- create httpserver
     try:
         httpserver = tornado.httpserver.HTTPServer(webapp, ssl_options=ssl_opt)
     except ValueError:
-        logging.critical('Certificates cert.pem, privkey.pem not found')
+        logger.critical('Certificates cert.pem, privkey.pem not found')
         sys.exit(1)
     try:
         httpserver.listen(port)
     except OSError:
-        logging.critical('Cannot bind port %d. Already in use?', port)
+        logger.critical('Cannot bind port %d. Already in use?', port)
         sys.exit(1)
-    logging.info('Webserver listening on %d...  (Ctrl-C to stop)', port)
+    logger.info('Listening on port %d...  (Ctrl-C to stop)', port)
     signal.signal(signal.SIGINT, signal_handler)
     # --- run webserver
     try:
         tornado.ioloop.IOLoop.current().start()  # running...
     except Exception:
-        logging.critical('Webserver stopped!')
+        logger.critical('Webserver stopped!')
         tornado.ioloop.IOLoop.current().stop()
         raise
@@ -117,16 +117,13 @@ $(document).ready(function() {
                         d = json.data[i];
                         var uid = d['uid'];
                         var checked = d['allowed'] ? 'checked' : '';
-                        var password_defined = d['password_defined'] ? ' <span class="badge badge-secondary"><i class="fa fa-key" aria-hidden="true"></i></span>' : '';
+                        // var password_defined = d['password_defined'] ? ' <span class="badge badge-secondary"><i class="fa fa-key" aria-hidden="true"></i></span>' : '';
                         var hora_inicio = d['start_time'] ? ' <span class="badge badge-success"><i class="fas fa-hourglass-start"></i> ' + d['start_time'].slice(11,16) + '</span>': '';
                         var unfocus = d['unfocus'] ? ' <span class="badge badge-danger">unfocus</span>' : '';
-                        var area = '';
-                        if (d['start_time'] ) {
-                            if (d['area'] > 75)
-                                area = ' <span class="badge badge-success"><i class="fas fa-desktop"></i> ' + Math.round(d['area']) + '%</span>';
-                            else
-                                area = ' <span class="badge badge-danger"><i class="fas fa-desktop"></i> ' + Math.round(d['area']) + '%</span>';
-                        };
+                        if (d['area'] > 75)
+                            area = ' <span class="badge badge-success"><i class="fas fa-desktop"></i>' + Math.round(d['area']) + '%</span>';
+                        else
+                            area = ' <span class="badge badge-danger"><i class="fas fa-desktop"></i> ' + Math.round(d['area']) + '%</span>';
                         var g = d['grades'];
                         t[i] = [];
@@ -134,7 +131,7 @@ $(document).ready(function() {
                         t[i][1] = '<input type="checkbox" name="' + uid + '" value="true"' + checked + '> ';
                         t[i][2] = uid;
                         t[i][3] = d['name'];
-                        t[i][4] = password_defined + hora_inicio + area + unfocus;
+                        t[i][4] = d['online'] ? hora_inicio + area + unfocus : '';
                         var gbar = '';
                         for (var j=0; j < g.length; j++)
@@ -53,8 +53,8 @@
           Acções
         </a>
         <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownAluno">
-          <a class="dropdown-item" href="#" id="novo_aluno" data-toggle="modal" data-target="#novo_aluno_modal">Inserir novo aluno...</a>
-          <a class="dropdown-item" href="#" id="reset_password_menu" data-toggle="modal" data-target="#reset_password_modal">Reset password do aluno...</a>
+          <a class="dropdown-item" href="#" id="novo_aluno" data-toggle="modal" data-target="#novo_aluno_modal">Novo aluno...</a>
+          <a class="dropdown-item" href="#" id="reset_password_menu" data-toggle="modal" data-target="#reset_password_modal">Limpar password...</a>
           <a class="dropdown-item" href="#" id="allow_all">Autorizar todos</a>
           <a class="dropdown-item" href="#" id="deny_all">Desautorizar todos</a>
           <div class="dropdown-divider"></div>
@@ -72,7 +72,7 @@
     <p>
       Referência: <code id="ref">--</code><br>
       Ficheiro de configuração do teste: <code id="filename">--</code><br>
-      Testes em formato JSON no directório: <code id="answers_dir">--</code><br>
+      Directório com os testes entregues: <code id="answers_dir">--</code><br>
       Base de dados: <code id="database">--</code><br>
     </p>
     <p>
@@ -31,8 +31,8 @@
     </ul>
     <span class="navbar-text">
       <i class="fas fa-user" aria-hidden="true"></i>
-      <span id="name">{{ escape(t['student']['name']) }}</span>
-      (<span id="number">{{ escape(t['student']['number']) }}</span>)
+      <span id="name">{{ escape(name) }}</span>
+      (<span id="number">{{ escape(uid) }}</span>)
       <span class="caret"></span>
     </span>
   </div>
@@ -17,9 +17,9 @@
     {{ md(q['text']) }}
   </div>
-  {% if show_ref %}
+  {% if debug %}
     <hr>
     file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
     ref:  <code>{{ q['ref'] }}</code>
   {% end %}
-</div>
 \ No newline at end of file
+</div>
@@ -29,11 +29,11 @@
     </p>
   </div>
-  {% if show_ref %}
+  {% if debug %}
     <div class="card-footer">
       file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
       ref:  <code>{{ q['ref'] }}</code>
     </div>
   {% end %}
 </div>
-{% end %}
 \ No newline at end of file
+{% end %}
@@ -16,9 +16,9 @@
 	<div id="text">
 		{{ md(q['text']) }}
 	</div>
-    {% if t['show_ref'] %}
+    {% if debug %}
         <hr>
         file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
         ref:  <code>{{ q['ref'] }}</code>
     {% end %}
-</div>
 \ No newline at end of file
+</div>
@@ -65,7 +65,7 @@
             {% end %}
           {% end %}
-          {% if t['show_ref'] %}
+          {% if debug %}
             <hr>
             file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
             ref:  <code>{{ q['ref'] }}</code>
@@ -109,7 +109,7 @@
           {% end %}
         </p>
-        {% if t['show_ref'] %}
+        {% if debug %}
           <hr>
           file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
           ref:  <code>{{ q['ref'] }}</code>
@@ -118,4 +118,4 @@
       </div> <!-- card-footer -->
     </div> <!-- card -->
   {% end %} <!-- if answer not None -->
-{% end %} <!-- block -->
 \ No newline at end of file
+{% end %} <!-- block -->
@@ -59,8 +59,8 @@
         <li class="nav-item">
           <span class="navbar-text">
             <i class="fas fa-user" aria-hidden="true"></i>
-            <span id="name">{{ escape(t['student']['name']) }}</span>
-            (<span id="number">{{ escape(t['student']['number']) }}</span>)
+            <span id="name">{{ escape(name) }}</span>
+            (<span id="number">{{ escape(uid) }}</span>)
             <span class="caret"></span>
           </span>
         </li>
@@ -97,9 +97,12 @@
       <div class="row">
         <label for="nota" class="col-sm-2">Nota:</label>
         <div class="col-sm-10" id="nota">
-          <span class="badge badge-primary">{{ round(t['grade'], 2) }}</span> valores
-          {% if t['state'] == 'QUIT' %}
-              (DESISTÊNCIA)
+          {% if t['state'] == 'CORRECTED' %}
+            <span class="badge badge-primary">{{ round(t['grade'], 2) }}</span> valores
+          {% elif t['state'] == 'SUBMITTED' %}
+            (não corrigido)
+          {% elif t['state'] == 'QUIT' %}
+            (DESISTIU)
           {% end %}
         </div>
       </div>
@@ -113,7 +116,7 @@
   </div> <!-- jumbotron -->
   {% for i, q in enumerate(t['questions']) %}
-    {% module Template(templ[q['type']], i=i, q=q, md=md(q['ref']), t=t) %}
+    {% module Template(templ[q['type']], i=i, q=q, md=md(q['ref']), t=t, debug=debug) %}
   {% end %}
 </div> <!-- container -->
@@ -74,8 +74,8 @@
         <li class="nav-item">
           <span class="navbar-text">
             <i class="fas fa-user" aria-hidden="true"></i>
-            <span id="name">{{ escape(t['student']['name']) }}</span>
-            (<span id="number">{{ escape(t['student']['number']) }}</span>)
+            <span id="name">{{ escape(name) }}</span>
+            (<span id="number">{{ escape(uid) }}</span>)
             <span class="caret"></span>
           </span>
         </li>
@@ -93,11 +93,11 @@
       <div class="row">
         <label for="nome" class="col-sm-3">Nome:</label>
-        <div class="col-sm-9" id="nome">{{ escape(t['student']['name']) }}</div>
+        <div class="col-sm-9" id="nome">{{ escape(name) }}</div>
       </div>
       <div class="row">
         <label for="numero" class="col-sm-3">Número:</label>
-        <div class="col-sm-9" id="numero">{{ escape(t['student']['number']) }}</div>
+        <div class="col-sm-9" id="numero">{{ escape(uid) }}</div>
       </div>
       <div class="row">
@@ -114,12 +114,14 @@
     {% module xsrf_form_html() %}
     {% for i, q in enumerate(t['questions']) %}
-      {% module Template(templ[q['type']], i=i, q=q, md=md(q['ref']), show_ref=t['show_ref']) %}
+      {% module Template(templ[q['type']], i=i, q=q, md=md(q['ref']), debug=debug) %}
     {% end %}
     <div class="form-row">
       <div class="col-12">
-        <button type="button" class="btn btn-success btn-lg btn-block" data-toggle="modal" data-target="#confirmar" id="form-button-submit">Submeter teste</button>
+        <button type="button" class="btn btn-success btn-lg btn-block" data-toggle="modal" data-target="#confirmar" id="form-button-submit">
+          Submeter teste
+        </button>
       </div>
     </div>
   </form>
@@ -138,11 +140,16 @@
       <div class="modal-body">
         O teste será enviado para classificação e já não poderá voltar atrás.
         Antes de submeter, verifique se respondeu a todas as questões.
-        Desactive as perguntas que não pretende classificar para evitar eventuais penalizações.
+        Desactive as perguntas que não pretende classificar para evitar
+        eventuais penalizações.
       </div>
       <div class="modal-footer">
-        <button type="button" class="btn btn-danger btn-lg" data-dismiss="modal">Oops, NÃO!!!</button>
-        <button form="test" type="submit" class="btn btn-success btn-lg">Sim, submeter...</button>
+        <button type="button" class="btn btn-danger btn-lg" data-dismiss="modal">
+          Oops, NÃO!!!
+        </button>
+        <button form="test" type="submit" class="btn btn-success btn-lg">
+          Sim, submeter...
+        </button>
       </div>
     </div>
   </div>
@@ -7,9 +7,8 @@ from datetime import datetime
 import json
 import logging
 from math import nan
-from os import path
-# Logger configuration
+
 logger = logging.getLogger(__name__)
@@ -30,17 +29,17 @@ class Test(dict):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, d):
+    def __init__(self, d: dict):
         super().__init__(d)
         self['grade'] = nan
         self['comment'] = ''
     # ------------------------------------------------------------------------
-    def start(self, student: dict) -> None:
+    def start(self, uid: str) -> None:
         '''
-        Write student id in the test and register start time
+        Register student id and start time in the test
         '''
-        self['student'] = student
+        self['student'] = uid
         self['start_time'] = datetime.now()
         self['finish_time'] = None
         self['state'] = 'ACTIVE'
@@ -57,14 +56,14 @@ class Test(dict):
         self['questions'][ref].set_answer(ans)
     # ------------------------------------------------------------------------
-    def submit(self, answers_dict) -> None:
+    def submit(self, answers: dict) -> None:
         '''
         Given a dictionary ans={'ref': 'some answer'} updates the answers of
         multiple questions in the test.
         Only affects the questions referred in the dictionary.
         '''
         self['finish_time'] = datetime.now()
-        for ref, ans in answers_dict.items():
+        for ref, ans in answers.items():
             self['questions'][ref].set_answer(ans)
         self['state'] = 'SUBMITTED'
@@ -104,15 +103,11 @@ class Test(dict):
         self['grade'] = 0.0
     # ------------------------------------------------------------------------
-    def save_json(self, pathfile) -> None:
+    def save_json(self, filename: str) -> None:
         '''save test in JSON format'''
-        with open(pathfile, 'w') as file:
+        with open(filename, 'w', encoding='utf-8') as file:
             json.dump(self, file, indent=2, default=str)  # str for datetime
     # ------------------------------------------------------------------------
     def __str__(self) -> str:
         return '\n'.join([f'{k}: {v}' for k,v in self.items()])
-        # return ('Test:\n'
-        #         f'  student:  {self.get("student", "--")}\n'
-        #         f'  start_time:  {self.get("start_time", "--")}\n'
-        #         f'  questions:  {", ".join(q["ref"] for q in self["questions"])}\n')
@@ -3,20 +3,65 @@ TestFactory - generates tests for students
 '''
 # python standard library
+import asyncio
 from os import path
 import random
 import logging
-import re
-from typing import Any, Dict
+
+# other libraries
+import schema
 # this project
-from perguntations.questions import QFactory, QuestionException
-from perguntations.test import Test
-from perguntations.tools import load_yaml
+from .questions import QFactory, QuestionException, QDict
+from .test import Test
+from .tools import load_yaml
 # Logger configuration
 logger = logging.getLogger(__name__)
+# --- test validation --------------------------------------------------------
+def check_answers_directory(ans: str) -> bool:
+    '''Checks is answers_dir exists and is writable'''
+    testfile = path.join(path.expanduser(ans), 'REMOVE-ME')
+    try:
+        with open(testfile, 'w', encoding='utf-8') as file:
+            file.write('You can safely remove this file.')
+    except OSError:
+        return False
+    return True
+
+def check_import_files(files: list) -> bool:
+    '''Checks if the question files exist'''
+    if not files:
+        return False
+    for file in files:
+        if not path.isfile(file):
+            return False
+    return True
+
+def normalize_question_list(questions: list) -> None:
+    '''convert question ref from string to list of string'''
+    for question in questions:
+        if isinstance(question['ref'], str):
+            question['ref'] = [question['ref']]
+
+test_schema = schema.Schema({
+    'ref': schema.Regex('^[a-zA-Z0-9_-]+$'),
+    'database': schema.And(str, path.isfile),
+    'answers_dir': schema.And(str, check_answers_directory),
+    'title': str,
+    schema.Optional('duration'): int,
+    schema.Optional('autosubmit'): bool,
+    schema.Optional('autocorrect'): bool,
+    schema.Optional('show_points'): bool,
+    schema.Optional('scale'): schema.And([schema.Use(float)],
+                                         lambda s: len(s) == 2),
+    'files': schema.And([str], check_import_files),
+    'questions': [{
+        'ref': schema.Or(str, [str]),
+        schema.Optional('points'): float
+        }]
+    }, ignore_extra_keys=True)
 # ============================================================================
 class TestFactoryException(Exception):
@@ -32,34 +77,30 @@ class TestFactory(dict):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, conf: Dict[str, Any]) -> None:
+    def __init__(self, conf) -> None:
         '''
         Loads configuration from yaml file, then overrides some configurations
         using the conf argument.
         Base questions are added to a pool of questions factories.
         '''
+        test_schema.validate(conf)
+
         # --- set test defaults and then use given configuration
         super().__init__({  # defaults
-            'title': '',
             'show_points': True,
             'scale': None,
             'duration': 0,  # 0=infinite
             'autosubmit': False,
             'autocorrect': True,
-            'debug': False,
-            'show_ref': False,
             })
         self.update(conf)
+        normalize_question_list(self['questions'])
         # --- for review, we are done. no factories needed
-        if self['review']:
-            logger.info('Review mode. No questions loaded. No factories.')
-            return
-
-        # --- perform sanity checks and normalize the test questions
-        self.sanity_checks()
-        logger.info('Sanity checks PASSED.')
+        # if self['review']:  FIXME
+        #     logger.info('Review mode. No questions loaded. No factories.')
+        #     return
         # --- find refs of all questions used in the test
         qrefs = {r for qq in self['questions'] for r in qq['ref']}
@@ -70,7 +111,7 @@ class TestFactory(dict):
         self['question_factory'] = {}
         for file in self["files"]:
-            fullpath = path.normpath(path.join(self["questions_dir"], file))
+            fullpath = path.normpath(file)
             logger.info('Loading "%s"...', fullpath)
             questions = load_yaml(fullpath)   # , default=[])
@@ -81,32 +122,25 @@ class TestFactory(dict):
                     msg = f'Question {i} in {file} is not a dictionary'
                     raise TestFactoryException(msg)
-                # check if ref is missing, then set to '/path/file.yaml:3'
+                # check if ref is missing, then set to '//file.yaml:3'
                 if 'ref' not in question:
                     question['ref'] = f'{file}:{i:04}'
                     logger.warning('Missing ref set to "%s"', question["ref"])
                 # check for duplicate refs
-                if question['ref'] in self['question_factory']:
-                    other = self['question_factory'][question['ref']]
+                qref = question['ref']
+                if qref in self['question_factory']:
+                    other = self['question_factory'][qref]
                     otherfile = path.join(other.question['path'],
                                           other.question['filename'])
-                    msg = (f'Duplicate reference "{question["ref"]}" in files '
-                           f'"{otherfile}" and "{fullpath}".')
+                    msg = f'Duplicate "{qref}" in {otherfile} and {fullpath}'
                     raise TestFactoryException(msg)
                 # make factory only for the questions used in the test
-                if question['ref'] in qrefs:
+                if qref in qrefs:
                     question.update(zip(('path', 'filename', 'index'),
                                         path.split(fullpath) + (i,)))
-                    if question['type'] == 'code' and 'server' not in question:
-                        try:
-                            question['server'] = self['jobe_server']
-                        except KeyError as exc:
-                            msg = f'Missing JOBE server in "{question["ref"]}"'
-                            raise TestFactoryException(msg) from exc
-
-                    self['question_factory'][question['ref']] = QFactory(question)
+                    self['question_factory'][qref] = QFactory(QDict(question))
         qmissing = qrefs.difference(set(self['question_factory'].keys()))
         if qmissing:
@@ -118,157 +152,124 @@ class TestFactory(dict):
     # ------------------------------------------------------------------------
-    def check_test_ref(self) -> None:
-        '''Test must have a `ref`'''
-        if 'ref' not in self:
-            raise TestFactoryException('Missing "ref" in configuration!')
-        if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
-            raise TestFactoryException('Test "ref" can only contain the '
-                                       'characters a-zA-Z0-9_-')
-
-    def check_missing_database(self) -> None:
-        '''Test must have a database'''
-        if 'database' not in self:
-            raise TestFactoryException('Missing "database" in configuration')
-        if not path.isfile(path.expanduser(self['database'])):
-            msg = f'Database "{self["database"]}" not found!'
-            raise TestFactoryException(msg)
-
-    def check_missing_answers_directory(self) -> None:
-        '''Test must have a answers directory'''
-        if 'answers_dir' not in self:
-            msg = 'Missing "answers_dir" in configuration'
-            raise TestFactoryException(msg)
-
-    def check_answers_directory_writable(self) -> None:
-        '''Answers directory must be writable'''
-        testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
-        try:
-            with open(testfile, 'w') as file:
-                file.write('You can safely remove this file.')
-        except OSError as exc:
-            msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
-            raise TestFactoryException(msg) from exc
-
-    def check_questions_directory(self) -> None:
-        '''Check if questions directory is missing or not accessible.'''
-        if 'questions_dir' not in self:
-            logger.warning('Missing "questions_dir". Using "%s"',
-                           path.abspath(path.curdir))
-            self['questions_dir'] = path.curdir
-        elif not path.isdir(path.expanduser(self['questions_dir'])):
-            raise TestFactoryException(f'Can\'t find questions directory '
-                                       f'"{self["questions_dir"]}"')
-
-    def check_import_files(self) -> None:
-        '''Check if there are files to import (with questions)'''
-        if 'files' not in self:
-            msg = ('Missing "files" in configuration with the list of '
-                   'question files to import!')
-            raise TestFactoryException(msg)
-
-        if isinstance(self['files'], str):
-            self['files'] = [self['files']]
-
-    def check_question_list(self) -> None:
-        '''normalize question list'''
-        if 'questions' not in self:
-            raise TestFactoryException('Missing "questions" in configuration')
-
-        for i, question in enumerate(self['questions']):
-            # normalize question to a dict and ref to a list of references
-            if isinstance(question, str):  # e.g.,    - some_ref
-                question = {'ref': [question]}    # becomes  - ref: [some_ref]
-            elif isinstance(question, dict) and isinstance(question['ref'], str):
-                question['ref'] = [question['ref']]
-            elif isinstance(question, list):
-                question = {'ref': [str(a) for a in question]}
-
-            self['questions'][i] = question
-
-    def check_missing_title(self) -> None:
-        '''Warns if title is missing'''
-        if not self['title']:
-            logger.warning('Title is undefined!')
-
-    def check_grade_scaling(self) -> None:
-        '''Just informs the scale limits'''
-        if 'scale_points' in self:
-            msg = ('*** DEPRECATION WARNING: *** scale_points, scale_min, '
-                   'scale_max were replaced by "scale: [min, max]".')
-            logger.warning(msg)
-            self['scale'] = [self['scale_min'], self['scale_max']]
+    # def check_test_ref(self) -> None:
+    #     '''Test must have a `ref`'''
+    #     if 'ref' not in self:
+    #         raise TestFactoryException('Missing "ref" in configuration!')
+    #     if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
+    #         raise TestFactoryException('Test "ref" can only contain the '
+    #                                    'characters a-zA-Z0-9_-')
+
+    # def check_missing_database(self) -> None:
+    #     '''Test must have a database'''
+    #     if 'database' not in self:
+    #         raise TestFactoryException('Missing "database" in configuration')
+    #     if not path.isfile(path.expanduser(self['database'])):
+    #         msg = f'Database "{self["database"]}" not found!'
+    #         raise TestFactoryException(msg)
+
+    # def check_missing_answers_directory(self) -> None:
+    #     '''Test must have a answers directory'''
+    #     if 'answers_dir' not in self:
+    #         msg = 'Missing "answers_dir" in configuration'
+    #         raise TestFactoryException(msg)
+
+    # def check_answers_directory_writable(self) -> None:
+    #     '''Answers directory must be writable'''
+    #     testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
+    #     try:
+    #         with open(testfile, 'w', encoding='utf-8') as file:
+    #             file.write('You can safely remove this file.')
+    #     except OSError as exc:
+    #         msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
+    #         raise TestFactoryException(msg) from exc
+
+    # def check_questions_directory(self) -> None:
+    #     '''Check if questions directory is missing or not accessible.'''
+    #     if 'questions_dir' not in self:
+    #         logger.warning('Missing "questions_dir". Using "%s"',
+    #                        path.abspath(path.curdir))
+    #         self['questions_dir'] = path.curdir
+    #     elif not path.isdir(path.expanduser(self['questions_dir'])):
+    #         raise TestFactoryException(f'Can\'t find questions directory '
+    #                                    f'"{self["questions_dir"]}"')
+
+    # def check_import_files(self) -> None:
+    #     '''Check if there are files to import (with questions)'''
+    #     if 'files' not in self:
+    #         msg = ('Missing "files" in configuration with the list of '
+    #                'question files to import!')
+    #         raise TestFactoryException(msg)
+
+        # if isinstance(self['files'], str):
+        #     self['files'] = [self['files']]
+
+    # def check_question_list(self) -> None:
+    #     '''normalize question list'''
+    #     if 'questions' not in self:
+    #         raise TestFactoryException('Missing "questions" in configuration')
+
+    #     for i, question in enumerate(self['questions']):
+    #         # normalize question to a dict and ref to a list of references
+    #         if isinstance(question, str):  # e.g.,    - some_ref
+    #             logger.warning(f'Question "{question}" should be a dictionary')
+    #             question = {'ref': [question]}    # becomes  - ref: [some_ref]
+    #         elif isinstance(question, dict) and isinstance(question['ref'], str):
+    #             question['ref'] = [question['ref']]
+    #         elif isinstance(question, list):
+    #             question = {'ref': [str(a) for a in question]}
+
+    #         self['questions'][i] = question
+
+    # def check_missing_title(self) -> None:
+    #     '''Warns if title is missing'''
+    #     if not self['title']:
+    #         logger.warning('Title is undefined!')
+
+    # def check_grade_scaling(self) -> None:
+    #     '''Just informs the scale limits'''
+    #     if 'scale_points' in self:
+    #         msg = ('*** DEPRECATION WARNING: *** scale_points, scale_min, '
+    #                'scale_max were replaced by "scale: [min, max]".')
+    #         logger.warning(msg)
+    #         self['scale'] = [self['scale_min'], self['scale_max']]
     # ------------------------------------------------------------------------
-    def sanity_checks(self) -> None:
-        '''
-        Checks for valid keys and sets default values.
-        Also checks if some files and directories exist
-        '''
-        self.check_test_ref()
-        self.check_missing_database()
-        self.check_missing_answers_directory()
-        self.check_answers_directory_writable()
-        self.check_questions_directory()
-        self.check_import_files()
-        self.check_question_list()
-        self.check_missing_title()
-        self.check_grade_scaling()
+    # def sanity_checks(self) -> None:
+    #     '''
+    #     Checks for valid keys and sets default values.
+    #     Also checks if some files and directories exist
+    #     '''
+        # self.check_test_ref()
+        # self.check_missing_database()
+        # self.check_missing_answers_directory()
+        # self.check_answers_directory_writable()
+        # self.check_questions_directory()
+        # self.check_import_files()
+        # self.check_question_list()
+        # self.check_missing_title()
+        # self.check_grade_scaling()
     # ------------------------------------------------------------------------
     def check_questions(self) -> None:
         '''
         checks if questions can be correctly generated and corrected
         '''
-        logger.info('Checking if questions can be generated and corrected...')
+        logger.info('Checking questions...')
+        # FIXME get_event_loop will be deprecated in python3.10
+        loop = asyncio.get_event_loop()
         for i, (qref, qfact) in enumerate(self['question_factory'].items()):
             try:
-                question = qfact.generate()
+                question = loop.run_until_complete(qfact.gen_async())
             except Exception as exc:
                 msg = f'Failed to generate "{qref}"'
                 raise TestFactoryException(msg) from exc
             else:
                 logger.info('%4d.  %s:  Ok', i, qref)
-                # logger.info('         generate Ok')
-
-            if question['type'] in ('code', 'textarea'):
-                if 'tests_right' in question:
-                    for tnum, right_answer in enumerate(question['tests_right']):
-                        try:
-                            question.set_answer(right_answer)
-                            question.correct()
-                        except Exception as exc:
-                            msg = f'Failed to correct "{qref}"'
-                            raise TestFactoryException(msg) from exc
-
-                        if question['grade'] == 1.0:
-                            logger.info('         test %i Ok', tnum)
-                        else:
-                            logger.error('         TEST %i IS WRONG!!!', tnum)
-                elif 'tests_wrong' in question:
-                    for tnum, wrong_answer in enumerate(question['tests_wrong']):
-                        try:
-                            question.set_answer(wrong_answer)
-                            question.correct()
-                        except Exception as exc:
-                            msg = f'Failed to correct "{qref}"'
-                            raise TestFactoryException(msg) from exc
-
-                        if question['grade'] < 1.0:
-                            logger.info('         test %i Ok', tnum)
-                        else:
-                            logger.error('         TEST %i IS WRONG!!!', tnum)
-                else:
-                    try:
-                        question.set_answer('')
-                        question.correct()
-                    except Exception as exc:
-                        msg = f'Failed to correct "{qref}"'
-                        raise TestFactoryException(msg) from exc
-                    else:
-                        logger.info('         correct Ok but no tests to run')
+            if question['type'] == 'textarea':
+                _runtests_textarea(qref, question)
     # ------------------------------------------------------------------------
     async def generate(self):
         '''
@@ -326,16 +327,51 @@ class TestFactory(dict):
             logger.error('%s errors found!', nerr)
         # copy these from the test configuratoin to each test instance
-        inherit = {'ref', 'title', 'database', 'answers_dir',
-                   'questions_dir', 'files',
-                   'duration', 'autosubmit', 'autocorrect',
-                   'scale', 'show_points',
-                   'show_ref', 'debug', }
-        # NOT INCLUDED:  testfile, allow_all, review
+        inherit = ['ref', 'title', 'database', 'answers_dir', 'files', 'scale',
+                   'duration', 'autosubmit', 'autocorrect', 'show_points']
         return Test({'questions': questions, **{k:self[k] for k in inherit}})
     # ------------------------------------------------------------------------
     def __repr__(self):
         testsettings = '\n'.join(f'  {k:14s}: {v}' for k, v in self.items())
-        return '{\n' + testsettings + '\n}'
+        return 'TestFactory({\n' + testsettings + '\n})'
+
+# ============================================================================
+def _runtests_textarea(qref, question):
+    '''
+    Checks if correction script works and runs tests if available
+    '''
+    try:
+        question.set_answer('')
+        question.correct()
+    except Exception as exc:
+        msg = f'Failed to correct "{qref}"'
+        raise TestFactoryException(msg) from exc
+    logger.info('         correction works')
+
+    for tnum, right_answer in enumerate(question.get('tests_right', {})):
+        try:
+            question.set_answer(right_answer)
+            question.correct()
+        except Exception as exc:
+            msg = f'Failed to correct "{qref}"'
+            raise TestFactoryException(msg) from exc
+
+        if question['grade'] == 1.0:
+            logger.info('         tests_right[%i] Ok', tnum)
+        else:
+            logger.error('         tests_right[%i] FAILED!!!', tnum)
+
+    for tnum, wrong_answer in enumerate(question.get('tests_wrong', {})):
+        try:
+            question.set_answer(wrong_answer)
+            question.correct()
+        except Exception as exc:
+            msg = f'Failed to correct "{qref}"'
+            raise TestFactoryException(msg) from exc
+
+        if question['grade'] < 1.0:
+            logger.info('         tests_wrong[%i] Ok', tnum)
+        else:
+            logger.error('         tests_wrong[%i] FAILED!!!', tnum)
 '''
-This module contains helper functions to:
-- load yaml files and report errors
-- run external programs (sync and async)
+File:           perguntations/tools.py
+Description:    Helper functions to load yaml files and run external programs.
 '''
@@ -20,28 +19,11 @@ import yaml
 logger = logging.getLogger(__name__)
-# ---------------------------------------------------------------------------
-def load_yaml(filename: str, default: Any = None) -> Any:
-    '''load data from yaml file'''
-
-    filename = path.expanduser(filename)
-    try:
-        file = open(filename, 'r', encoding='utf-8')
-    except Exception as exc:
-        logger.error(exc)
-        if default is not None:
-            return default
-        raise
-
-    with file:
-        try:
-            return yaml.safe_load(file)
-        except yaml.YAMLError as exc:
-            logger.error(str(exc).replace('\n', ' '))
-            if default is not None:
-                return default
-            raise
-
+# ----------------------------------------------------------------------------
+def load_yaml(filename: str) -> Any:
+    '''load yaml file or raise exception on error'''
+    with open(path.expanduser(filename), 'r', encoding='utf-8') as file:
+        return yaml.safe_load(file)
 # ---------------------------------------------------------------------------
 def run_script(script: str,
@@ -53,7 +35,7 @@ def run_script(script: str,
     The script is run in another process but this function blocks waiting
     for its termination.
     '''
-    logger.info('run_script "%s"', script)
+    logger.debug('run_script "%s"', script)
     output = None
     script = path.expanduser(script)
@@ -22,14 +22,15 @@ setup(
     url="https://git.xdi.uevora.pt/mjsb/perguntations.git",
     packages=find_packages(),
     include_package_data=True,   # install files from MANIFEST.in
-    python_requires='>=3.7.*',
+    python_requires='>=3.8.*',
     install_requires=[
-        'tornado>=6.0',
-        'mistune',
+        'bcrypt>=3.1',
+        'mistune<2.0',
         'pyyaml>=5.1',
         'pygments',
-        'sqlalchemy',
-        'bcrypt>=3.1'
+        'schema>=0.7.5',
+        'sqlalchemy>=1.4',
+        'tornado>=6.1',
         ],
     entry_points={
         'console_scripts': [
@@ -3,4 +3,3 @@
 git pull
 npm update
 pip install -U .
-
@@ -0,0 +1,77 @@		@@ -0,0 +1,77 @@
	1	+#!/usr/bin/env python3
	2	+
	3	+'''
	4	+Example of a question generator.
	5	+Arguments are read from stdin.
	6	+'''
	7	+
	8	+from random import randint
	9	+import sys
	10	+
	11	+# read two arguments from the field `args` specified in the question yaml file
	12	+a, b = (int(n) for n in sys.argv[1:])
	13	+
	14	+x = randint(a, b)
	15	+y = randint(a, b)
	16	+r = x + y
	17	+
	18	+print(f"""---
	19	+type: text
	20	+title: Geradores de perguntas
	21	+text: \|
	22	+
	23	+ As perguntas podem ser estáticas (como as que vimos até aqui), ou serem
	24	+ geradas dinâmicamente por um programa externo. Para gerar uma pergunta, o
	25	+ programa deve escrever texto no `stdout` em formato `yaml` tal como os
	26	+ exemplos das perguntas estáticas dos tipos apresentados anteriormente. Pode
	27	+ também receber argumentos de linha de comando para parametrizar a pergunta.
	28	+ Aqui está um exemplo de uma pergunta gerada por um script python:
	29	+
	30	+ ```python
	31	+ #!/usr/bin/env python3
	32	+
	33	+ from random import randint
	34	+ import sys
	35	+
	36	+ a, b = (int(n) for n in sys.argv[1:]) # argumentos da linha de comando
	37	+
	38	+ x = randint(a, b) # número inteiro no intervalo a..b
	39	+ y = randint(a, b) # número inteiro no intervalo a..b
	40	+ r = x + y # calcula resultado correcto
	41	+
	42	+ print(f'''---
	43	+ type: text
	44	+ title: Contas de somar
	45	+ text: \|
	46	+ Calcule o resultado de ${{x}} + {{y}}$.
	47	+ correct: '{{r}}'
	48	+ solution: \|
	49	+ A solução é {{r}}.''')
	50	+ ```
	51	+
	52	+ Este script deve ter permissões para poder ser executado no terminal.
	53	+ Podemos testar o programa no terminal `./gen-somar.py 1 100` e verificar que
	54	+ o output é uma pergunta válida em formato `yaml`. Agora é necessário indicar
	55	+ que este script deve ser usado para gerar uma pergunta.
	56	+
	57	+ Uma pergunta gerada por um programa externo é declarada com
	58	+
	59	+ ```yaml
	60	+ - type: generator
	61	+ ref: gen-somar
	62	+ script: gen-somar.py
	63	+ # argumentos opcionais
	64	+ args: [1, 100]
	65	+ ```
	66	+
	67	+ O programa pode receber uma lista de argumentos de linha de comando
	68	+ declarados em `args`.
	69	+
	70	+ ---
	71	+
	72	+ Calcule o resultado de ${x} + {y}$.
	73	+
	74	+ Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
	75	+correct: '{r}'
	76	+solution: \|
	77	+ A solução é {r}.""")
	@@ -1,77 +0,0 @@	@@ -1,77 +0,0 @@
1	-#!/usr/bin/env python3
2	-
3	-'''
4	-Example of a question generator.
5	-Arguments are read from stdin.
6	-'''
7	-
8	-from random import randint
9	-import sys
10	-
11	-# read two arguments from the field `args` specified in the question yaml file
12	-a, b = (int(n) for n in sys.argv[1:])
13	-
14	-x = randint(a, b)
15	-y = randint(a, b)
16	-r = x + y
17	-
18	-print(f"""---
19	-type: text
20	-title: Geradores de perguntas
21	-text: \|
22	-
23	- As perguntas podem ser estáticas (como as que vimos até aqui), ou serem
24	- geradas dinâmicamente por um programa externo. Para gerar uma pergunta, o
25	- programa deve escrever texto no `stdout` em formato `yaml` tal como os
26	- exemplos das perguntas estáticas dos tipos apresentados anteriormente. Pode
27	- também receber argumentos de linha de comando para parametrizar a pergunta.
28	- Aqui está um exemplo de uma pergunta gerada por um script python:
29	-
30	- ```python
31	- #!/usr/bin/env python3
32	-
33	- from random import randint
34	- import sys
35	-
36	- a, b = (int(n) for n in sys.argv[1:]) # argumentos da linha de comando
37	-
38	- x = randint(a, b) # número inteiro no intervalo a..b
39	- y = randint(a, b) # número inteiro no intervalo a..b
40	- r = x + y # calcula resultado correcto
41	-
42	- print(f'''---
43	- type: text
44	- title: Contas de somar
45	- text: \|
46	- Calcule o resultado de ${{x}} + {{y}}$.
47	- correct: '{{r}}'
48	- solution: \|
49	- A solução é {{r}}.''')
50	- ```
51	-
52	- Este script deve ter permissões para poder ser executado no terminal.
53	- Podemos testar o programa no terminal `./gen-somar.py 1 100` e verificar que
54	- o output é uma pergunta válida em formato `yaml`. Agora é necessário indicar
55	- que este script deve ser usado para gerar uma pergunta.
56	-
57	- Uma pergunta gerada por um programa externo é declarada com
58	-
59	- ```yaml
60	- - type: generator
61	- ref: gen-somar
62	- script: gen-somar.py
63	- # argumentos opcionais
64	- args: [1, 100]
65	- ```
66	-
67	- O programa pode receber uma lista de argumentos de linha de comando
68	- declarados em `args`.
69	-
70	- ---
71	-
72	- Calcule o resultado de ${x} + {y}$.
73	-
74	- Os números foram gerados aleatoriamente no intervalo de {a} a {b}.
75	-correct: '{r}'
76	-solution: \|
77	- A solução é {r}.""")
	@@ -137,9 +137,9 @@ class HighlightRenderer(mistune.Renderer):		@@ -137,9 +137,9 @@ class HighlightRenderer(mistune.Renderer):
137	return '<table class="table table-sm"><thead class="thead-light">' \	137	return '<table class="table table-sm"><thead class="thead-light">' \
138	+ header + '</thead><tbody>' + body + '</tbody></table>'	138	+ header + '</thead><tbody>' + body + '</tbody></table>'
139		139
140	- def image(self, src, title, alt):	140	+ def image(self, src, title, text):
141	'''render image'''	141	'''render image'''
142	- alt = mistune.escape(alt, quote=True)	142	+ alt = mistune.escape(text, quote=True)
143	if title is not None:	143	if title is not None:
144	if title: # not empty string, show as caption	144	if title: # not empty string, show as caption
145	title = mistune.escape(title, quote=True)	145	title = mistune.escape(title, quote=True)
1	'''	1	'''
2	-Classes the implement several types of questions.	2	+File: perguntations/questions.py
		3	+Description: Classes the implement several types of questions.
3	'''	4	'''
4		5
5		6
6	# python standard library	7	# python standard library
7	-import asyncio
8	from datetime import datetime	8	from datetime import datetime
9	import logging	9	import logging
10	-from os import path	10	+import os
11	import random	11	import random
12	import re	12	import re
13	from typing import Any, Dict, NewType	13	from typing import Any, Dict, NewType
14	import uuid	14	import uuid
15		15
16	-
17	-# from urllib.error import HTTPError
18	-# import json
19	-# import http.client
20	-
21	-
22	# this project	16	# this project
23	-from perguntations.tools import run_script, run_script_async	17	+from .tools import run_script, run_script_async
24		18
25	# setup logger for this module	19	# setup logger for this module
26	logger = logging.getLogger(__name__)	20	logger = logging.getLogger(__name__)
27		21
28	-
29	QDict = NewType('QDict', Dict[str, Any])	22	QDict = NewType('QDict', Dict[str, Any])
30		23
31		24
32	-
33	-
34	class QuestionException(Exception):	25	class QuestionException(Exception):
35	'''Exceptions raised in this module'''	26	'''Exceptions raised in this module'''
36		27
	@@ -45,8 +36,6 @@ class Question(dict):		@@ -45,8 +36,6 @@ class Question(dict):
45	for each student.	36	for each student.
46	Instances can shuffle options or automatically generate questions.	37	Instances can shuffle options or automatically generate questions.
47	'''	38	'''
48	- # def __init__(self, q: QDict) -> None:
49	- # super().__init__(q)
50		39
51	def gen(self) -> None:	40	def gen(self) -> None:
52	'''	41	'''
	@@ -96,9 +85,6 @@ class QuestionRadio(Question):		@@ -96,9 +85,6 @@ class QuestionRadio(Question):
96	'''	85	'''
97		86
98	# ------------------------------------------------------------------------	87	# ------------------------------------------------------------------------
99	- # def __init__(self, q: QDict) -> None:
100	- # super().__init__(q)
101	-
102	def gen(self) -> None:	88	def gen(self) -> None:
103	'''	89	'''
104	Sets defaults, performs checks and generates the actual question	90	Sets defaults, performs checks and generates the actual question
	@@ -128,8 +114,7 @@ class QuestionRadio(Question):		@@ -128,8 +114,7 @@ class QuestionRadio(Question):
128	# e.g. correct: 2 --> correct: [0,0,1,0,0]	114	# e.g. correct: 2 --> correct: [0,0,1,0,0]
129	if isinstance(self['correct'], int):	115	if isinstance(self['correct'], int):
130	if not 0 <= self['correct'] < nopts:	116	if not 0 <= self['correct'] < nopts:
131	- msg = (f'`correct` out of range 0..{nopts-1}. '
132	- f'In question "{self["ref"]}"')	117	+ msg = f'"{self["ref"]}": correct out of range 0..{nopts-1}'
133	logger.error(msg)	118	logger.error(msg)
134	raise QuestionException(msg)	119	raise QuestionException(msg)
135		120
	@@ -139,8 +124,7 @@ class QuestionRadio(Question):		@@ -139,8 +124,7 @@ class QuestionRadio(Question):
139	elif isinstance(self['correct'], list):	124	elif isinstance(self['correct'], list):
140	# must match number of options	125	# must match number of options
141	if len(self['correct']) != nopts:	126	if len(self['correct']) != nopts:
142	- msg = (f'{nopts} options vs {len(self["correct"])} correct. '
143	- f'In question "{self["ref"]}"')	127	+ msg = f'"{self["ref"]}": number of options/correct mismatch'
144	logger.error(msg)	128	logger.error(msg)
145	raise QuestionException(msg)	129	raise QuestionException(msg)
146		130
	@@ -148,23 +132,20 @@ class QuestionRadio(Question):		@@ -148,23 +132,20 @@ class QuestionRadio(Question):
148	try:	132	try:
149	self['correct'] = [float(x) for x in self['correct']]	133	self['correct'] = [float(x) for x in self['correct']]
150	except (ValueError, TypeError) as exc:	134	except (ValueError, TypeError) as exc:
151	- msg = ('`correct` must be list of numbers or booleans.'
152	- f'In "{self["ref"]}"')	135	+ msg = f'"{self["ref"]}": correct must contain floats or bools'
153	logger.error(msg)	136	logger.error(msg)
154	raise QuestionException(msg) from exc	137	raise QuestionException(msg) from exc
155		138
156	# check grade boundaries	139	# check grade boundaries
157	if self['discount'] and not all(0.0 <= x <= 1.0	140	if self['discount'] and not all(0.0 <= x <= 1.0
158	for x in self['correct']):	141	for x in self['correct']):
159	- msg = ('`correct` values must be in the interval [0.0, 1.0]. '
160	- f'In "{self["ref"]}"')	142	+ msg = f'"{self["ref"]}": correct must be in [0.0, 1.0]'
161	logger.error(msg)	143	logger.error(msg)
162	raise QuestionException(msg)	144	raise QuestionException(msg)
163		145
164	# at least one correct option	146	# at least one correct option
165	if all(x < 1.0 for x in self['correct']):	147	if all(x < 1.0 for x in self['correct']):
166	- msg = ('At least one correct option is required. '
167	- f'In "{self["ref"]}"')	148	+ msg = f'"{self["ref"]}": has no correct options'
168	logger.error(msg)	149	logger.error(msg)
169	raise QuestionException(msg)	150	raise QuestionException(msg)
170		151
	@@ -231,9 +212,6 @@ class QuestionCheckbox(Question):		@@ -231,9 +212,6 @@ class QuestionCheckbox(Question):
231	'''	212	'''
232		213
233	# ------------------------------------------------------------------------	214	# ------------------------------------------------------------------------
234	- # def __init__(self, q: QDict) -> None:
235	- # super().__init__(q)
236	-
237	def gen(self) -> None:	215	def gen(self) -> None:
238	super().gen()	216	super().gen()
239		217
	@@ -288,19 +266,6 @@ class QuestionCheckbox(Question):		@@ -288,19 +266,6 @@ class QuestionCheckbox(Question):
288	f'Please fix "{self["ref"]}" in "{self["path"]}"')	266	f'Please fix "{self["ref"]}" in "{self["path"]}"')
289	logger.error(msg)	267	logger.error(msg)
290	raise QuestionException(msg)	268	raise QuestionException(msg)
291	- # msg0 = ('+--------------- BEHAVIOR CHANGE NOTICE ---------------+')
292	- # msg1 = ('\| Correct values in checkbox questions must be in the \|')
293	- # msg2 = ('\| interval [0.0, 1.0]. I will convert to the new \|')
294	- # msg3 = ('\| behavior, for now, but you should fix it. \|')
295	- # msg4 = ('+------------------------------------------------------+')
296	- # logger.warning(msg0)
297	- # logger.warning(msg1)
298	- # logger.warning(msg2)
299	- # logger.warning(msg3)
300	- # logger.warning(msg4)
301	- # logger.warning('please fix "%s"', self["ref"])
302	- # # normalize to [0,1]
303	- # self['correct'] = [(x+1)/2 for x in self['correct']]
304		269
305	# if an option is a list of (right, wrong), pick one	270	# if an option is a list of (right, wrong), pick one
306	options = []	271	options = []
	@@ -356,9 +321,6 @@ class QuestionText(Question):		@@ -356,9 +321,6 @@ class QuestionText(Question):
356	'''	321	'''
357		322
358	# ------------------------------------------------------------------------	323	# ------------------------------------------------------------------------
359	- # def __init__(self, q: QDict) -> None:
360	- # super().__init__(q)
361	-
362	def gen(self) -> None:	324	def gen(self) -> None:
363	super().gen()	325	super().gen()
364	self.set_defaults(QDict({	326	self.set_defaults(QDict({
	@@ -389,12 +351,13 @@ class QuestionText(Question):		@@ -389,12 +351,13 @@ class QuestionText(Question):
389	def transform(self, ans):	351	def transform(self, ans):
390	'''apply optional filters to the answer'''	352	'''apply optional filters to the answer'''
391		353
		354	+ # apply transformations in sequence
392	for transform in self['transform']:	355	for transform in self['transform']:
393	if transform == 'remove_space': # removes all spaces	356	if transform == 'remove_space': # removes all spaces
394	ans = ans.replace(' ', '')	357	ans = ans.replace(' ', '')
395	elif transform == 'trim': # removes spaces around	358	elif transform == 'trim': # removes spaces around
396	ans = ans.strip()	359	ans = ans.strip()
397	- elif transform == 'normalize_space': # replaces multiple spaces by one	360	+ elif transform == 'normalize_space': # replaces many spaces by one
398	ans = re.sub(r'\s+', ' ', ans.strip())	361	ans = re.sub(r'\s+', ' ', ans.strip())
399	elif transform == 'lower': # convert to lowercase	362	elif transform == 'lower': # convert to lowercase
400	ans = ans.lower()	363	ans = ans.lower()
	@@ -410,7 +373,7 @@ class QuestionText(Question):		@@ -410,7 +373,7 @@ class QuestionText(Question):
410	super().correct()	373	super().correct()
411		374
412	if self['answer'] is not None:	375	if self['answer'] is not None:
413	- answer = self.transform(self['answer']) # apply transformations	376	+ answer = self.transform(self['answer'])
414	self['grade'] = 1.0 if answer in self['correct'] else 0.0	377	self['grade'] = 1.0 if answer in self['correct'] else 0.0
415		378
416		379
	@@ -427,9 +390,6 @@ class QuestionTextRegex(Question):		@@ -427,9 +390,6 @@ class QuestionTextRegex(Question):
427	'''	390	'''
428		391
429	# ------------------------------------------------------------------------	392	# ------------------------------------------------------------------------
430	- # def __init__(self, q: QDict) -> None:
431	- # super().__init__(q)
432	-
433	def gen(self) -> None:	393	def gen(self) -> None:
434	super().gen()	394	super().gen()
435		395
	@@ -442,14 +402,6 @@ class QuestionTextRegex(Question):		@@ -442,14 +402,6 @@ class QuestionTextRegex(Question):
442	if not isinstance(self['correct'], list):	402	if not isinstance(self['correct'], list):
443	self['correct'] = [self['correct']]	403	self['correct'] = [self['correct']]
444		404
445	- # converts patterns to compiled versions
446	- # try:
447	- # self['correct'] = [re.compile(a) for a in self['correct']]
448	- # except Exception as exc:
449	- # msg = f'Failed to compile regex in "{self["ref"]}"'
450	- # logger.error(msg)
451	- # raise QuestionException(msg) from exc
452	-
453	# ------------------------------------------------------------------------	405	# ------------------------------------------------------------------------
454	def correct(self) -> None:	406	def correct(self) -> None:
455	super().correct()	407	super().correct()
	@@ -464,15 +416,6 @@ class QuestionTextRegex(Question):		@@ -464,15 +416,6 @@ class QuestionTextRegex(Question):
464	regex, self['answer'])	416	regex, self['answer'])
465	self['grade'] = 0.0	417	self['grade'] = 0.0
466		418
467	- # try:
468	- # if regex.match(self['answer']):
469	- # self['grade'] = 1.0
470	- # return
471	- # except TypeError:
472	- # logger.error('While matching regex %s with answer "%s".',
473	- # regex.pattern, self["answer"])
474	-
475	-
476	# ============================================================================	419	# ============================================================================
477	class QuestionNumericInterval(Question):	420	class QuestionNumericInterval(Question):
478	'''An instance of QuestionTextNumeric will always have the keys:	421	'''An instance of QuestionTextNumeric will always have the keys:
	@@ -484,9 +427,6 @@ class QuestionNumericInterval(Question):		@@ -484,9 +427,6 @@ class QuestionNumericInterval(Question):
484	'''	427	'''
485		428
486	# ------------------------------------------------------------------------	429	# ------------------------------------------------------------------------
487	- # def __init__(self, q: QDict) -> None:
488	- # super().__init__(q)
489	-
490	def gen(self) -> None:	430	def gen(self) -> None:
491	super().gen()	431	super().gen()
492		432
	@@ -548,9 +488,6 @@ class QuestionTextArea(Question):		@@ -548,9 +488,6 @@ class QuestionTextArea(Question):
548	'''	488	'''
549		489
550	# ------------------------------------------------------------------------	490	# ------------------------------------------------------------------------
551	- # def __init__(self, q: QDict) -> None:
552	- # super().__init__(q)
553	-
554	def gen(self) -> None:	491	def gen(self) -> None:
555	super().gen()	492	super().gen()
556		493
	@@ -561,7 +498,7 @@ class QuestionTextArea(Question):		@@ -561,7 +498,7 @@ class QuestionTextArea(Question):
561	'args': []	498	'args': []
562	}))	499	}))
563		500
564	- self['correct'] = path.join(self['path'], self['correct'])	501	+ self['correct'] = os.path.join(self['path'], self['correct'])
565		502
566	# ------------------------------------------------------------------------	503	# ------------------------------------------------------------------------
567	def correct(self) -> None:	504	def correct(self) -> None:
	@@ -625,129 +562,6 @@ class QuestionTextArea(Question):		@@ -625,129 +562,6 @@ class QuestionTextArea(Question):
625		562
626		563
627	# ============================================================================	564	# ============================================================================
628	-# class QuestionCode(Question):
629	-# '''
630	-# Submits answer to a JOBE server to compile and run against the test cases.
631	-# '''
632	-
633	-# _outcomes = {
634	-# 0: 'JOBE outcome: Successful run',
635	-# 11: 'JOBE outcome: Compile error',
636	-# 12: 'JOBE outcome: Runtime error',
637	-# 13: 'JOBE outcome: Time limit exceeded',
638	-# 15: 'JOBE outcome: Successful run',
639	-# 17: 'JOBE outcome: Memory limit exceeded',
640	-# 19: 'JOBE outcome: Illegal system call',
641	-# 20: 'JOBE outcome: Internal error, please report',
642	-# 21: 'JOBE outcome: Server overload',
643	-# }
644	-
645	-# # ------------------------------------------------------------------------
646	-# def __init__(self, q: QDict) -> None:
647	-# super().__init__(q)
648	-
649	-# self.set_defaults(QDict({
650	-# 'text': '',
651	-# 'timeout': 5, # seconds
652	-# 'server': '127.0.0.1', # JOBE server
653	-# 'language': 'c',
654	-# 'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
655	-# }))
656	-
657	- # ------------------------------------------------------------------------
658	- # def correct(self) -> None:
659	- # super().correct()
660	-
661	- # if self['answer'] is None:
662	- # return
663	-
664	- # # submit answer to JOBE server
665	- # resource = '/jobe/index.php/restapi/runs/'
666	- # headers = {"Content-type": "application/json; charset=utf-8",
667	- # "Accept": "application/json"}
668	-
669	- # for expected in self['correct']:
670	- # data_json = json.dumps({
671	- # 'run_spec' : {
672	- # 'language_id': self['language'],
673	- # 'sourcecode': self['answer'],
674	- # 'input': expected.get('stdin', ''),
675	- # },
676	- # })
677	-
678	- # try:
679	- # connect = http.client.HTTPConnection(self['server'])
680	- # connect.request(
681	- # method='POST',
682	- # url=resource,
683	- # body=data_json,
684	- # headers=headers
685	- # )
686	- # response = connect.getresponse()
687	- # logger.debug('JOBE response status %d', response.status)
688	- # if response.status != 204:
689	- # content = response.read().decode('utf8')
690	- # if content:
691	- # result = json.loads(content)
692	- # connect.close()
693	-
694	- # except (HTTPError, ValueError):
695	- # logger.error('HTTPError while connecting to JOBE server')
696	-
697	- # try:
698	- # outcome = result['outcome']
699	- # except (NameError, TypeError, KeyError):
700	- # logger.error('Bad result returned from JOBE server: %s', result)
701	- # return
702	- # logger.debug(self._outcomes[outcome])
703	-
704	-
705	-
706	- # if result['cmpinfo']: # compiler errors and warnings
707	- # self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
708	- # self['grade'] = 0.0
709	- # return
710	-
711	- # if result['stdout'] != expected.get('stdout', ''):
712	- # self['comments'] = 'O output gerado é diferente do esperado.' # FIXME mostrar porque?
713	- # self['grade'] = 0.0
714	- # return
715	-
716	- # self['comments'] = 'Ok!'
717	- # self['grade'] = 1.0
718	-
719	-
720	- # # ------------------------------------------------------------------------
721	- # async def correct_async(self) -> None:
722	- # self.correct() # FIXME there is no async correction!!!
723	-
724	-
725	- # out = run_script(
726	- # script=self['correct'],
727	- # args=self['args'],
728	- # stdin=self['answer'],
729	- # timeout=self['timeout']
730	- # )
731	-
732	- # if out is None:
733	- # logger.warning('No grade after running "%s".', self["correct"])
734	- # self['comments'] = 'O programa de correcção abortou...'
735	- # self['grade'] = 0.0
736	- # elif isinstance(out, dict):
737	- # self['comments'] = out.get('comments', '')
738	- # try:
739	- # self['grade'] = float(out['grade'])
740	- # except ValueError:
741	- # logger.error('Output error in "%s".', self["correct"])
742	- # except KeyError:
743	- # logger.error('No grade in "%s".', self["correct"])
744	- # else:
745	- # try:
746	- # self['grade'] = float(out)
747	- # except (TypeError, ValueError):
748	- # logger.error('Invalid grade in "%s".', self["correct"])
749	-
750	-# ============================================================================
751	class QuestionInformation(Question):	565	class QuestionInformation(Question):
752	'''	566	'''
753	Not really a question, just an information panel.	567	Not really a question, just an information panel.
	@@ -755,9 +569,6 @@ class QuestionInformation(Question):		@@ -755,9 +569,6 @@ class QuestionInformation(Question):
755	'''	569	'''
756		570
757	# ------------------------------------------------------------------------	571	# ------------------------------------------------------------------------
758	- # def __init__(self, q: QDict) -> None:
759	- # super().__init__(q)
760	-
761	def gen(self) -> None:	572	def gen(self) -> None:
762	super().gen()	573	super().gen()
763	self.set_defaults(QDict({	574	self.set_defaults(QDict({
	@@ -770,7 +581,6 @@ class QuestionInformation(Question):		@@ -770,7 +581,6 @@ class QuestionInformation(Question):
770	self['grade'] = 1.0 # always "correct" but points should be zero!	581	self['grade'] = 1.0 # always "correct" but points should be zero!
771		582
772		583
773	-
774	# ============================================================================	584	# ============================================================================
775	def question_from(qdict: QDict) -> Question:	585	def question_from(qdict: QDict) -> Question:
776	'''	586	'''
	@@ -783,7 +593,6 @@ def question_from(qdict: QDict) -> Question:		@@ -783,7 +593,6 @@ def question_from(qdict: QDict) -> Question:
783	'text-regex': QuestionTextRegex,	593	'text-regex': QuestionTextRegex,
784	'numeric-interval': QuestionNumericInterval,	594	'numeric-interval': QuestionNumericInterval,
785	'textarea': QuestionTextArea,	595	'textarea': QuestionTextArea,
786	- # 'code': QuestionCode,
787	# -- informative panels --	596	# -- informative panels --
788	'information': QuestionInformation,	597	'information': QuestionInformation,
789	'success': QuestionInformation,	598	'success': QuestionInformation,
	@@ -856,17 +665,17 @@ class QFactory():		@@ -856,17 +665,17 @@ class QFactory():
856	logger.debug('generating %s...', self.qdict["ref"])	665	logger.debug('generating %s...', self.qdict["ref"])
857	# Shallow copy so that script generated questions will not replace	666	# Shallow copy so that script generated questions will not replace
858	# the original generators	667	# the original generators
859	- qdict = self.qdict.copy()	668	+ qdict = QDict(self.qdict.copy())
860	qdict['qid'] = str(uuid.uuid4()) # unique for each question	669	qdict['qid'] = str(uuid.uuid4()) # unique for each question
861		670
862	# If question is of generator type, an external program will be run	671	# If question is of generator type, an external program will be run
863	# which will print a valid question in yaml format to stdout. This	672	# which will print a valid question in yaml format to stdout. This
864	# output is then yaml parsed into a dictionary `q`.	673	# output is then yaml parsed into a dictionary `q`.
865	if qdict['type'] == 'generator':	674	if qdict['type'] == 'generator':
866	- logger.debug(' \\_ Running "%s".', qdict['script'])	675	+ logger.debug(' \\_ Running "%s"', qdict['script'])
867	qdict.setdefault('args', [])	676	qdict.setdefault('args', [])
868	qdict.setdefault('stdin', '')	677	qdict.setdefault('stdin', '')
869	- script = path.join(qdict['path'], qdict['script'])	678	+ script = os.path.join(qdict['path'], qdict['script'])
870	out = await run_script_async(script=script,	679	out = await run_script_async(script=script,
871	args=qdict['args'],	680	args=qdict['args'],
872	stdin=qdict['stdin'])	681	stdin=qdict['stdin'])
	@@ -875,8 +684,3 @@ class QFactory():		@@ -875,8 +684,3 @@ class QFactory():
875	question = question_from(qdict) # returns a Question instance	684	question = question_from(qdict) # returns a Question instance
876	question.gen()	685	question.gen()
877	return question	686	return question
878	-
879	- # ------------------------------------------------------------------------
880	- def generate(self) -> Question:
881	- '''generate question (synchronous version)'''
882	- return asyncio.get_event_loop().run_until_complete(self.gen_async())
1	'''	1	'''
2	-This module contains helper functions to:
3	-- load yaml files and report errors
4	-- run external programs (sync and async)	2	+File: perguntations/tools.py
		3	+Description: Helper functions to load yaml files and run external programs.
5	'''	4	'''
6		5
7		6
	@@ -20,28 +19,11 @@ import yaml		@@ -20,28 +19,11 @@ import yaml
20	logger = logging.getLogger(__name__)	19	logger = logging.getLogger(__name__)
21		20
22		21
23	-# ---------------------------------------------------------------------------
24	-def load_yaml(filename: str, default: Any = None) -> Any:
25	- '''load data from yaml file'''
26	-
27	- filename = path.expanduser(filename)
28	- try:
29	- file = open(filename, 'r', encoding='utf-8')
30	- except Exception as exc:
31	- logger.error(exc)
32	- if default is not None:
33	- return default
34	- raise
35	-
36	- with file:
37	- try:
38	- return yaml.safe_load(file)
39	- except yaml.YAMLError as exc:
40	- logger.error(str(exc).replace('\n', ' '))
41	- if default is not None:
42	- return default
43	- raise
44	-	22	+# ----------------------------------------------------------------------------
		23	+def load_yaml(filename: str) -> Any:
		24	+ '''load yaml file or raise exception on error'''
		25	+ with open(path.expanduser(filename), 'r', encoding='utf-8') as file:
		26	+ return yaml.safe_load(file)
45		27
46	# ---------------------------------------------------------------------------	28	# ---------------------------------------------------------------------------
47	def run_script(script: str,	29	def run_script(script: str,
	@@ -53,7 +35,7 @@ def run_script(script: str,		@@ -53,7 +35,7 @@ def run_script(script: str,
53	The script is run in another process but this function blocks waiting	35	The script is run in another process but this function blocks waiting
54	for its termination.	36	for its termination.
55	'''	37	'''
56	- logger.info('run_script "%s"', script)	38	+ logger.debug('run_script "%s"', script)
57		39
58	output = None	40	output = None
59	script = path.expanduser(script)	41	script = path.expanduser(script)
	@@ -3,4 +3,3 @@		@@ -3,4 +3,3 @@
3	git pull	3	git pull
4	npm update	4	npm update
5	pip install -U .	5	pip install -U .
6	-