large refactoring

allow offline correction of tests BREAKING CHANGES: Incompatible database tables from previous versions! Must start with empty database.

large refactoring
allow offline correction of tests BREAKING CHANGES: Incompatible database tables from previous versions! Must start with empty database.
Miguel Barão
1 parent 053cf0e0
Showing 18 changed files with 1080 additions and 713 deletions Show diff stats
BUGS.md
demo/demo.yaml
demo/questions/questions-tutorial.yaml
perguntations/__init__.py
perguntations/app.py
perguntations/initdb.py
perguntations/main.py
perguntations/models.py
perguntations/questions.py
perguntations/serve.py
perguntations/templates/grade.html
perguntations/templates/login.html
perguntations/templates/review-question.html
perguntations/templates/review.html
perguntations/templates/test.html
perguntations/test.py
perguntations/testfactory.py
setup.py
 # BUGS
-- grade gives internal server error
+- cookies existe um perguntations_user e um user. De onde vem o user?
+- nao esta a mostrar imagens?? internal server error?
+- JOBE correct async
+- esta a corrigir código JOBE mesmo que nao tenha respondido???
+- QuestionCode falta reportar nos comments os vários erros que podem ocorrer (timeout, etc)
+
+- algumas vezes a base de dados guarda o mesmo teste em duplicado. ver se dois submits dao origem a duas correcções.
+talvez a base de dados devesse ter como chave do teste um id que fosse único desse teste particular (não um auto counter, nem ref do teste)
+- em caso de timeout na submissão (e.g. JOBE ou script nao responde) a correcção não termina e o teste não é guardado.
+- grade gives internal server error??
 - reload do teste recomeça a contagem no inicio do tempo.
 - em admin, quando scale_max não é 20, as cores das barras continuam a reflectir a escala 0,20. a tabela teste na DB não tem a escala desse teste.
 - em grade.html as barras estao normalizadas para os limites scale_min e max do teste actual e nao dos testes realizados no passado (tabela test devia guardar a escala).
@@ -11,9 +20,12 @@
 - Test.reset_answers() unused.
 - teste nao esta a mostrar imagens de vez em quando.???
 - testar as perguntas todas no início do teste como o aprendizations.
+- show-ref nao esta a funcionar na correccao (pelo menos)
 # TODO
+- permitir remover alunos que estão online para poderem comecar de novo.
+- guardar nota final grade truncado em zero e sem ser truncado (quando é necessário fazer correcções à mão às perguntas, é necessário o valor não truncado)
 - stress tests. use https://locust.io
 - wait for admin to start test. (students can be allowed earlier)
 - impedir os eventos copy/paste. alunos usam isso para trazer codigo ja feito nos computadores. Obrigar a fazer reset? fazer um copy automaticamente?
@@ -62,6 +74,9 @@ ou usar push (websockets?)
 # FIXED
+- internal server error quando em --review, download csv detalhado.
+- perguntas repetidas (mesma ref) dao asneira, porque a referencia é usada como chave em varios sitios e as chaves nao podem ser dupplicadas.
+  da asneira pelo menos na funcao get_questions_csv. na base de dados tem de estar registado tb o numero da pergunta, caso contrario é impossível saber a qual corresponde.
 - mostrar unfocus e window area em /admin
 - CRITICAL se answer for `i<n` a revisão de provas mostra apenas i (interpreta `<` como tag?)
 - botao de autorizar desliga-se, fazer debounce.
@@ -14,6 +14,9 @@ database: students.db
 # Directory where the submitted and corrected test are stored for later review.
 answers_dir: ans
+# Server used to compile & execute code
+jobe_server: 192.168.1.85
+
 # --- optional settings: -----------------------------------------------------
 # Title of this test, e.g. course name, year or test number
@@ -26,7 +29,13 @@ duration: 20
 # Automatic test submission after the given 'duration' timeout
 # (default: false)
-autosubmit: true
+autosubmit: false
+
+# If true, the test will be corrected on submission, the grade calculated and
+# shown to the student. If false, the test is saved but not corrected.
+# No grade is shown to the student.
+# (default: true)
+autocorrect: true
 # Show points for each question (min and max).
 # (default: true)
@@ -37,6 +46,7 @@ show_points: true
 # (default: no scaling, just use question points)
 scale: [0, 5]
+
 # ----------------------------------------------------------------------------
 # Base path applied to the questions files and all the scripts
 # including question generators and correctors.
@@ -70,19 +80,4 @@ questions:
   - [tut-alert1, tut-alert2]
   - tut-generator
   - tut-yamllint
-
-# test:
-#   - ref1
-#   - block: a
-#   - block: [b, c]
-#   - ref2
-
-# blocks:
-#   a:
-#     - ref1
-#     - ref2
-#     - ref3
-#   b:
-#     - rr4
-#     - rr5
-#     - rr6
+  # - tut-code
@@ -26,6 +26,7 @@
     show_points: true          # mostra cotação das perguntas (default: true)
     scale: [0, 20]             # limites inferior e superior da escala (default: [0,20])
     scale_points: true         # normaliza cotações para a escala definida
+    jobe_server: moodle-jobe.uevora.pt  # server used to compile & execute code
     debug: false               # mostra informação de debug no browser
     # --------------------------------------------------------------------------
@@ -575,7 +576,7 @@
 # ----------------------------------------------------------------------------
 - type: information
   text: |
-    This question is not included in the test and will not shown up.
+    This question is not included in the test and will not show up.
     It also lacks a "ref" and is automatically named
     `questions/questions-tutorial.yaml:0013`.
     A warning is shown on the console about this.
@@ -609,3 +610,52 @@
         generate-question | yamllint -
         correct-answer | yamllint -
     ```
+
+# ----------------------------------------------------------------------------
+# - type: code
+#   ref: tut-code
+#   title: Submissão de código (JOBE)
+#   text: |
+#     É possível enviar código para ser compilado e executado por um servidor
+#     JOBE instalado separadamente, ver [JOBE](https://github.com/trampgeek/jobe).
+
+#     ```yaml
+#     - type: code
+#       ref: tut-code
+#       title: Submissão de código (JOBE)
+#       text: |
+#         Escreva um programa em C que recebe uma string no standard input e
+#         mostra a mensagem `hello ` seguida da string.
+#         Por exemplo, se o input for `Maria`, o output deverá ser `hello Maria`.
+#       language: c
+#       correct:
+#         - stdin: 'Maria'
+#           stdout: 'hello Maria'
+#         - stdin: 'xyz'
+#           stdout: 'hello xyz'
+#     ```
+
+#     Existem várias linguagens suportadas pelo servidor JOBE (C, C++, Java,
+#     Python2, Python3, Octave, Pascal, PHP).
+#     O campo `correct` deverá ser uma lista de casos a testar.
+#     Se um caso incluir `stdin`, este será enviado para o programa e o `stdout`
+#     obtido será comparado com o declarado. A pergunta é considerada correcta se
+#     todos os outputs coincidirem.
+
+#     Por defeito é o usado o servidor JOBE declarado no teste. Para usar outro
+#     diferente nesta pergunta usa-se a opção `server: 127.0.0.1` com o endereço
+#     apropriado.
+#   answer: |
+#     #include <stdio.h>
+#     int main() {
+#       char name[20];
+#       scanf("%s", name);
+#       printf("hello %s", name);
+#     }
+#   # server: 192.168.1.85
+#   language: c
+#   correct:
+#     - stdin: 'Maria'
+#       stdout: 'hello Maria'
+#     - stdin: 'xyz'
+#       stdout: 'hello xyz'
@@ -32,7 +32,7 @@ proof of submission and for review.
 '''
 APP_NAME = 'perguntations'
-APP_VERSION = '2020.11.dev2'
+APP_VERSION = '2020.12.dev1'
 APP_DESCRIPTION = __doc__
 __author__ = 'Miguel Barão'
@@ -20,7 +20,9 @@ from sqlalchemy.orm import sessionmaker
 # this project
 from perguntations.models import Student, Test, Question
 from perguntations.tools import load_yaml
-from perguntations.test import TestFactory, TestFactoryException
+from perguntations.testfactory import TestFactory, TestFactoryException
+import perguntations.test
+from perguntations.questions import question_from
 logger = logging.getLogger(__name__)
@@ -33,12 +35,12 @@ class AppException(Exception):
 # ============================================================================
 # helper functions
 # ============================================================================
-async def check_password(try_pw, password):
+async def check_password(try_pw, hashed_pw):
     '''check password in executor'''
     try_pw = try_pw.encode('utf-8')
     loop = asyncio.get_running_loop()
-    hashed = await loop.run_in_executor(None, bcrypt.hashpw, try_pw, password)
-    return password == hashed
+    hashed = await loop.run_in_executor(None, bcrypt.hashpw, try_pw, hashed_pw)
+    return hashed_pw == hashed
 async def hash_password(password):
@@ -113,8 +115,7 @@ class App():
         else:
             logger.info('Students not yet allowed to login.')
-        # pre-generate tests
-
+        # pre-generate tests for allowed students
         if self.allowed:
             logger.info('Generating %d tests. May take awhile...',
                         len(self.allowed))
@@ -122,37 +123,98 @@ class App():
         else:
             logger.info('No tests were generated.')
+        if conf['correct']:
+            self._correct_tests()
+
+    # ------------------------------------------------------------------------
+    def _correct_tests(self):
+        with self._db_session() as sess:
+            # Find which tests have to be corrected
+            dbtests = sess.query(Test)\
+                    .filter(Test.ref == self.testfactory['ref'])\
+                    .filter(Test.state == "SUBMITTED")\
+                    .all()
+
+            logger.info('Correcting %d tests...', len(dbtests))
+            for dbtest in dbtests:
+                try:
+                    with open(dbtest.filename) as file:
+                        testdict = json.load(file)
+                except FileNotFoundError:
+                    logger.error('File not found: %s', dbtest.filename)
+                    continue
+
+                # creates a class Test with the methods to correct it
+                # the questions are still dictionaries, so we have to call
+                # question_from() to produce Question() instances that can be
+                # corrected. Finally the test can be corrected.
+                test = perguntations.test.Test(testdict)
+                test['questions'] = [question_from(q) for q in test['questions']]
+                test.correct()
+                logger.info('Student %s:  grade = %f', test['student']['number'], test['grade'])
+
+                # save JSON file (overwriting the old one)
+                uid = test['student']['number']
+                ref = test['ref']
+                finish_time = test['finish_time']
+                answers_dir = test['answers_dir']
+                fname = f'{uid}--{ref}--{finish_time}.json'
+                fpath = path.join(answers_dir, fname)
+                test.save_json(fpath)
+                logger.info('%s saved JSON file.', uid)
+
+                # update database
+                dbtest.grade = test['grade']
+                dbtest.state = test['state']
+                dbtest.questions = [
+                    Question(
+                        number=n,
+                        ref=q['ref'],
+                        grade=q['grade'],
+                        comment=q.get('comment', ''),
+                        starttime=str(test['start_time']),
+                        finishtime=str(test['finish_time']),
+                        test_id=test['ref']
+                        )
+                    for n, q in enumerate(test['questions'])
+                    ]
+                logger.info('%s database updated.', uid)
+
     # ------------------------------------------------------------------------
-    async def login(self, uid, try_pw):
+    async def login(self, uid, try_pw, headers=None):
         '''login authentication'''
         if uid not in self.allowed and uid != '0':      # not allowed
-            logger.warning('"%s" not allowed to login.', uid)
-            return False
+            logger.warning('"%s" unauthorized.', uid)
+            return 'unauthorized'
-        # get name+password from db
         with self._db_session() as sess:
-            name, password = sess.query(Student.name, Student.password)\
+            name, hashed_pw = sess.query(Student.name, Student.password)\
                                  .filter_by(id=uid)\
                                  .one()
-        # first login updates the password
-        if password == '':              # update password on first login
+        if hashed_pw == '':              # update password on first login
             await self.update_student_password(uid, try_pw)
             pw_ok = True
         else:                           # check password
-            pw_ok = await check_password(try_pw, password)  # async bcrypt
-
-        if pw_ok:      # success
-            self.allowed.discard(uid)  # remove from set of allowed students
-            if uid in self.online:
-                logger.warning('"%s" already logged in.', uid)
-            else:                      # make student online
-                self.online[uid] = {'student': {'name': name, 'number': uid}}
-                logger.info('"%s" logged in.', uid)
-            return True
-                        # wrong password
-        logger.info('"%s" wrong password.', uid)
-        return False
+            pw_ok = await check_password(try_pw, hashed_pw)  # async bcrypt
+
+        if not pw_ok:        # wrong password
+            logger.info('"%s" wrong password.', uid)
+            return 'wrong_password'
+
+        # success
+        self.allowed.discard(uid)  # remove from set of allowed students
+
+        if uid in self.online:
+            logger.warning('"%s" login again from %s (reusing state).',
+                           uid, headers['remote_ip'])
+            # FIXME invalidate previous login
+        else:
+            self.online[uid] = {'student': {
+                                    'name': name,
+                                    'number': uid,
+                                    'headers': headers}}
+            logger.info('"%s" login from %s.', uid, headers['remote_ip'])
     # ------------------------------------------------------------------------
     def logout(self, uid):
@@ -179,31 +241,48 @@ class App():
         testconf.update(conf)
         # start test factory
-        logger.info('Making test factory...')
+        logger.info('Running test factory...')
         try:
             self.testfactory = TestFactory(testconf)
         except TestFactoryException as exc:
             logger.critical(exc)
-            raise AppException('Failed to create test factory!') from  exc
-
-        logger.info('Test factory ready. No errors found.')
+            raise AppException('Failed to create test factory!') from exc
     # ------------------------------------------------------------------------
-    def _pregenerate_tests(self, num):
+    def _pregenerate_tests(self, num):  # TODO needs improvement
         event_loop = asyncio.get_event_loop()
-        # for _ in range(num):
-        #     test = event_loop.run_until_complete(self.testfactory.generate())
-        #     self.pregenerated_tests.append(test)
-
         self.pregenerated_tests += [
                 event_loop.run_until_complete(self.testfactory.generate())
                 for _ in range(num)]
     # ------------------------------------------------------------------------
-    async def generate_test(self, uid):
-        '''generate a test for a given student. the student must be online'''
+    async def get_test_or_generate(self, uid):
+        '''get current test or generate a new one'''
+        try:
+            student = self.online[uid]
+        except KeyError as exc:
+            msg = f'"{uid}" is not online. get_test_or_generate() FAILED'
+            logger.error(msg)
+            raise AppException(msg) from exc
+
+        # get current test. if test does not exist then generate a new one
+        if not 'test' in student:
+            await self._new_test(uid)
+
+        return student['test']
+
+    def get_test(self, uid):
+        '''get test from online student or raise exception'''
+        return self.online[uid]['test']
-        student_id = self.online[uid]['student']  # {'name': ?, 'number': ?}
+    # ------------------------------------------------------------------------
+    async def _new_test(self, uid):
+        '''
+        assign a test to a given student. if there are pregenerated tests then
+        use one of them, otherwise generate one.
+        the student must be online
+        '''
+        student = self.online[uid]['student']  # {'name': ?, 'number': ?}
         try:
             test = self.pregenerated_tests.pop()
@@ -214,15 +293,13 @@ class App():
         else:
             logger.info('"%s" using a pregenerated test.', uid)
-        test.start(student_id)           # student signs the test
-        self.online[uid]['test'] = test  # register test for this student
-
-        return self.online[uid]['test']
+        test.start(student)           # student signs the test
+        self.online[uid]['test'] = test
     # ------------------------------------------------------------------------
-    async def correct_test(self, uid, ans):
+    async def submit_test(self, uid, ans):
         '''
-        Corrects test
+        Handles test submission and correction.
         ans is a dictionary {question_index: answer, ...} with the answers for
         the complete test. For example:  {0:'hello', 1:[1,2]}
@@ -230,72 +307,81 @@ class App():
         test = self.online[uid]['test']
         # --- submit answers and correct test
-        test.update_answers(ans)
+        test.submit(ans)
         logger.info('"%s" submitted %d answers.', uid, len(ans))
-        grade = await test.correct()
-        logger.info('"%s" grade = %g points.', uid, grade)
+        if test['autocorrect']:
+            await test.correct_async()
+            logger.info('"%s" grade = %g points.', uid, test['grade'])
         # --- save test in JSON format
-        fields = (uid, test['ref'], str(test['finish_time']))
-        fname = '--'.join(fields) + '.json'
+        fname = f'{uid}--{test["ref"]}--{test["finish_time"]}.json'
         fpath = path.join(test['answers_dir'], fname)
-        with open(path.expanduser(fpath), 'w') as file:
-            # default=str required for datetime objects
-            json.dump(test, file, indent=2, default=str)
+        test.save_json(fpath)
         logger.info('"%s" saved JSON.', uid)
-        # --- insert test and questions into database
-        with self._db_session() as sess:
-            sess.add(Test(
-                ref=test['ref'],
-                title=test['title'],
-                grade=test['grade'],
-                starttime=str(test['start_time']),
-                finishtime=str(test['finish_time']),
-                filename=fpath,
-                student_id=uid,
-                state=test['state'],
-                comment=''))
-            sess.add_all([Question(
-                ref=q['ref'],
-                grade=q['grade'],
-                starttime=str(test['start_time']),
-                finishtime=str(test['finish_time']),
-                student_id=uid,
-                test_id=test['ref'])
-                          for q in test['questions'] if 'grade' in q])
+        # --- insert test and questions into the database
+        #     only corrected questions are added
+        test_row = Test(
+            ref=test['ref'],
+            title=test['title'],
+            grade=test['grade'],
+            state=test['state'],
+            comment=test['comment'],
+            starttime=str(test['start_time']),
+            finishtime=str(test['finish_time']),
+            filename=fpath,
+            student_id=uid)
+
+        if test['state'] == 'CORRECTED':
+            test_row.questions = [
+                Question(
+                    number=n,
+                    ref=q['ref'],
+                    grade=q['grade'],
+                    comment=q.get('comment', ''),
+                    starttime=str(test['start_time']),
+                    finishtime=str(test['finish_time']),
+                    test_id=test['ref']
+                    )
+                for n, q in enumerate(test['questions'])
+                ]
+        with self._db_session() as sess:
+            sess.add(test_row)
         logger.info('"%s" database updated.', uid)
-        return grade
     # ------------------------------------------------------------------------
-    def giveup_test(self, uid):
-        '''giveup test - not used??'''
-        test = self.online[uid]['test']
-        test.giveup()
+    def get_student_grade(self, uid):
+        return self.online[uid]['test'].get('grade', None)
-        # save JSON with the test
-        fields = (test['student']['number'], test['ref'],
-                  str(test['finish_time']))
-        fname = '--'.join(fields) + '.json'
-        fpath = path.join(test['answers_dir'], fname)
-        test.save_json(fpath)
-
-        # insert test into database
-        with self._db_session() as sess:
-            sess.add(Test(ref=test['ref'],
-                          title=test['title'],
-                          grade=test['grade'],
-                          starttime=str(test['start_time']),
-                          finishtime=str(test['finish_time']),
-                          filename=fpath,
-                          student_id=test['student']['number'],
-                          state=test['state'],
-                          comment=''))
-
-        logger.info('"%s" gave up.', uid)
-        return test
+    # ------------------------------------------------------------------------
+    # def giveup_test(self, uid):
+    #     '''giveup test - not used??'''
+    #     test = self.online[uid]['test']
+    #     test.giveup()
+
+    #     # save JSON with the test
+    #     fields = (test['student']['number'], test['ref'],
+    #               str(test['finish_time']))
+    #     fname = '--'.join(fields) + '.json'
+    #     fpath = path.join(test['answers_dir'], fname)
+    #     test.save_json(fpath)
+
+    #     # insert test into database
+    #     with self._db_session() as sess:
+    #         sess.add(Test(ref=test['ref'],
+    #                       title=test['title'],
+    #                       grade=test['grade'],
+    #                       starttime=str(test['start_time']),
+    #                       finishtime=str(test['finish_time']),
+    #                       filename=fpath,
+    #                       # student_id=test['student']['number'],
+    #                       state=test['state'],
+    #                       comment=''))
+
+    #     logger.info('"%s" gave up.', uid)
+    #     return test
     # ------------------------------------------------------------------------
     def event_test(self, uid, cmd, value):
@@ -317,57 +403,58 @@ class App():
     def get_questions_csv(self):
         '''generates a CSV with the grades of the test'''
-        test_id = self.testfactory['ref']
-
+        test_ref = self.testfactory['ref']
         with self._db_session() as sess:
-            grades = sess.query(Question.student_id, Question.starttime,
-                                Question.ref, Question.grade)\
-                         .filter(Question.test_id == test_id)\
-                         .order_by(Question.student_id)\
-                         .all()
-
-        cols = ['Aluno', 'Início'] + \
-               [r for question in self.testfactory['questions']
-                for r in question['ref']]
-
-        tests = {}
-        for question in grades:
-            student, qref, qgrade = question[:2], *question[2:]
-            tests.setdefault(student, {})[qref] = qgrade
-
-        rows = [{'Aluno': test[0], 'Início': test[1], **q}
-                for test, q in tests.items()]
+            questions = sess.query(Test.id, Test.student_id, Test.starttime,
+                                   Question.number, Question.grade)\
+                            .join(Question)\
+                            .filter(Test.ref == test_ref)\
+                            .all()
+
+            qnums = set()  # keeps track of all the questions in the test
+            tests = {}      # {test_id: {student_id, starttime, 0: grade, ...}}
+            for question in questions:
+                test_id, student_id, starttime, num, grade = question
+                default_test_id = {'Aluno': student_id, 'Início': starttime}
+                tests.setdefault(test_id, default_test_id)[num] = grade
+                qnums.add(num)
+
+        if not tests:
+            logger.warning('Empty CSV: there are no tests!')
+            return test_ref, ''
+
+        cols = ['Aluno', 'Início'] + list(qnums)
         csvstr = io.StringIO()
         writer = csv.DictWriter(csvstr, fieldnames=cols, restval=None,
                                 delimiter=';', quoting=csv.QUOTE_ALL)
         writer.writeheader()
-        writer.writerows(rows)
-        return test_id, csvstr.getvalue()
-
+        writer.writerows(tests.values())
+        return test_ref, csvstr.getvalue()
     def get_test_csv(self):
-        '''generates a CSV with the grades of the test'''
+        '''generates a CSV with the grades of the test currently running'''
+        test_ref = self.testfactory['ref']
         with self._db_session() as sess:
-            grades = sess.query(Test.student_id, Test.grade,
+            tests = sess.query(Test.student_id,
+                                Test.grade,
                                 Test.starttime, Test.finishtime)\
-                         .filter(Test.ref == self.testfactory['ref'])\
+                         .filter(Test.ref == test_ref)\
                          .order_by(Test.student_id)\
                          .all()
+        if not tests:
+            logger.warning('Empty CSV: there are no tests!')
+            return test_ref, ''
+
         csvstr = io.StringIO()
         writer = csv.writer(csvstr, delimiter=';', quoting=csv.QUOTE_ALL)
         writer.writerow(('Aluno', 'Nota', 'Início', 'Fim'))
-        writer.writerows(grades)
-        return self.testfactory['ref'], csvstr.getvalue()
+        writer.writerows(tests)
-    def get_student_test(self, uid):
-        '''get test from online student or None if no test was generated yet'''
-        return self.online[uid].get('test', None)
-
-    # def get_questions_dir(self):
-    #     return self.testfactory['questions_dir']
+        return test_ref, csvstr.getvalue()
+    # ------------------------------------------------------------------------
     def get_student_grades_from_all_tests(self, uid):
         '''get grades of student from all tests'''
         with self._db_session() as sess:
@@ -19,7 +19,7 @@ import sqlalchemy as sa
 from perguntations.models import Base, Student
-# ===========================================================================
+# ============================================================================
 def parse_commandline_arguments():
     '''Parse command line options'''
     parser = argparse.ArgumentParser(
@@ -68,7 +68,7 @@ def parse_commandline_arguments():
     return parser.parse_args()
-# ===========================================================================
+# ============================================================================
 def get_students_from_csv(filename):
     '''
     SIIUE names have alien strings like "(TE)" and are sometimes capitalized
@@ -97,7 +97,7 @@ def get_students_from_csv(filename):
     return students
-# ===========================================================================
+# ============================================================================
 def hashpw(student, password=None):
     '''replace password by hash for a single student'''
     print('.', end='', flush=True)
@@ -108,7 +108,7 @@ def hashpw(student, password=None):
                                       bcrypt.gensalt())
-# ===========================================================================
+# ============================================================================
 def insert_students_into_db(session, students):
     '''insert list of students into the database'''
     try:
@@ -49,13 +49,16 @@ def parse_cmdline_arguments():
     parser.add_argument('--review',
                         action='store_true',
                         help='Review mode: doesn\'t generate test')
+    parser.add_argument('--correct',
+                        action='store_true',
+                        help='Correct test and update JSON files and database')
     parser.add_argument('--port',
                         type=int,
                         default=8443,
                         help='port for the HTTPS server (default: 8443)')
     parser.add_argument('--version',
                         action='version',
-                        version=APP_VERSION,
+                        version=f'{APP_VERSION} - python {sys.version}',
                         help='Show version information and exit')
     return parser.parse_args()
@@ -99,13 +102,12 @@ def get_logger_config(debug=False):
                 },
             },
         }
-    default_config['loggers'].update({
-        APP_NAME+'.'+module: {
-            'handlers': ['default'],
-            'level': level,
-            'propagate': False,
-            } for module in ['app', 'models', 'factory', 'questions',
-                             'test', 'tools']})
+
+    modules = ['app', 'models', 'questions', 'test', 'testfactory', 'tools']
+    logger = {'handlers': ['default'], 'level': level, 'propagate': False}
+
+    default_config['loggers'].update({f'{APP_NAME}.{module}': logger
+                                      for module in modules})
     return load_yaml(config_file, default=default_config)
@@ -124,11 +126,12 @@ def main():
     # --- start application --------------------------------------------------
     config = {
         'testfile': args.testfile,
-        'debug':    args.debug,
+        'debug': args.debug,
         'allow_all': args.allow_all,
         'allow_list': args.allow_list,
         'show_ref': args.show_ref,
-        'review':   args.review,
+        'review': args.review,
+        'correct': args.correct,
         }
     try:
@@ -25,9 +25,8 @@ class Student(Base):
     # ---
     tests = relationship('Test', back_populates='student')
-    questions = relationship('Question', back_populates='student')
-    def __repr__(self):
+    def __str__(self):
         return (f'Student:\n'
                 f'  id:         "{self.id}"\n'
                 f'  name:       "{self.name}"\n'
@@ -42,7 +41,7 @@ class Test(Base):
     ref = Column(String)
     title = Column(String)
     grade = Column(Float)
-    state = Column(String)  # ACTIVE, FINISHED, QUIT, NULL
+    state = Column(String)  # ACTIVE, SUBMITTED, CORRECTED, QUIT, NULL
     comment = Column(String)
     starttime = Column(String)
     finishtime = Column(String)
@@ -53,12 +52,12 @@ class Test(Base):
     student = relationship('Student', back_populates='tests')
     questions = relationship('Question', back_populates='test')
-    def __repr__(self):
+    def __str__(self):
         return (f'Test:\n'
-                f'  id:         "{self.id}"\n'
+                f'  id:         {self.id}\n'
                 f'  ref:        "{self.ref}"\n'
                 f'  title:      "{self.title}"\n'
-                f'  grade:      "{self.grade}"\n'
+                f'  grade:      {self.grade}\n'
                 f'  state:      "{self.state}"\n'
                 f'  comment:    "{self.comment}"\n'
                 f'  starttime:  "{self.starttime}"\n'
@@ -72,23 +71,24 @@ class Question(Base):
     '''Question table'''
     __tablename__ = 'questions'
     id = Column(Integer, primary_key=True)  # auto_increment
+    number = Column(Integer)    # question number (ref may be not be unique)
     ref = Column(String)
     grade = Column(Float)
+    comment = Column(String)
     starttime = Column(String)
     finishtime = Column(String)
-    student_id = Column(String, ForeignKey('students.id'))
     test_id = Column(String, ForeignKey('tests.id'))
     # ---
-    student = relationship('Student', back_populates='questions')
     test = relationship('Test', back_populates='questions')
-    def __repr__(self):
+    def __str__(self):
         return (f'Question:\n'
-                f'  id:         "{self.id}"\n'
+                f'  id:         {self.id}\n'
+                f'  number:     {self.number}\n'
                 f'  ref:        "{self.ref}"\n'
-                f'  grade:      "{self.grade}"\n'
+                f'  grade:      {self.grade}\n'
+                f'  comment:    "{self.comment}"\n'
                 f'  starttime:  "{self.starttime}"\n'
                 f'  finishtime: "{self.finishtime}"\n'
-                f'  student_id: "{self.student_id}"\n'  # FIXME normal form
                 f'  test_id:    "{self.test_id}"\n')
@@ -13,6 +13,12 @@ import re
 from typing import Any, Dict, NewType
 import uuid
+
+# from urllib.error import HTTPError
+# import json
+# import http.client
+
+
 # this project
 from perguntations.tools import run_script, run_script_async
@@ -23,6 +29,8 @@ logger = logging.getLogger(__name__)
 QDict = NewType('QDict', Dict[str, Any])
+
+
 class QuestionException(Exception):
     '''Exceptions raised in this module'''
@@ -37,8 +45,13 @@ class Question(dict):
     for each student.
     Instances can shuffle options or automatically generate questions.
     '''
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        '''
+        Sets defaults that are valid for any question type
+        '''
         # add required keys if missing
         self.set_defaults(QDict({
@@ -83,9 +96,15 @@ class QuestionRadio(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+    def gen(self) -> None:
+        '''
+        Sets defaults, performs checks and generates the actual question
+        by modifying the options and correct values
+        '''
+        super().gen()
         try:
             nopts = len(self['options'])
         except KeyError as exc:
@@ -212,8 +231,11 @@ class QuestionCheckbox(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        super().gen()
         try:
             nopts = len(self['options'])
@@ -334,9 +356,11 @@ class QuestionText(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+    def gen(self) -> None:
+        super().gen()
         self.set_defaults(QDict({
             'text': '',
             'correct': [],  # no correct answers, always wrong
@@ -403,8 +427,11 @@ class QuestionTextRegex(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        super().gen()
         self.set_defaults(QDict({
             'text': '',
@@ -416,26 +443,34 @@ class QuestionTextRegex(Question):
             self['correct'] = [self['correct']]
         # converts patterns to compiled versions
-        try:
-            self['correct'] = [re.compile(a) for a in self['correct']]
-        except Exception as exc:
-            msg = f'Failed to compile regex in "{self["ref"]}"'
-            logger.error(msg)
-            raise QuestionException(msg) from exc
+        # try:
+        #     self['correct'] = [re.compile(a) for a in self['correct']]
+        # except Exception as exc:
+        #     msg = f'Failed to compile regex in "{self["ref"]}"'
+        #     logger.error(msg)
+        #     raise QuestionException(msg) from exc
     # ------------------------------------------------------------------------
     def correct(self) -> None:
         super().correct()
         if self['answer'] is not None:
-            self['grade'] = 0.0
             for regex in self['correct']:
                 try:
-                    if regex.match(self['answer']):
+                    if re.fullmatch(regex, self['answer']):
                         self['grade'] = 1.0
                         return
                 except TypeError:
-                    logger.error('While matching regex %s with answer "%s".',
-                                 regex.pattern, self["answer"])
+                    logger.error('While matching regex "%s" with answer "%s".',
+                                 regex, self['answer'])
+            self['grade'] = 0.0
+
+                # try:
+                #     if regex.match(self['answer']):
+                #         self['grade'] = 1.0
+                #         return
+                # except TypeError:
+                #     logger.error('While matching regex %s with answer "%s".',
+                #                  regex.pattern, self["answer"])
 # ============================================================================
@@ -449,8 +484,11 @@ class QuestionNumericInterval(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        super().gen()
         self.set_defaults(QDict({
             'text': '',
@@ -510,8 +548,11 @@ class QuestionTextArea(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        super().gen()
         self.set_defaults(QDict({
             'text': '',
@@ -584,6 +625,129 @@ class QuestionTextArea(Question):
 # ============================================================================
+# class QuestionCode(Question):
+#     '''
+#     Submits answer to a JOBE server to compile and run against the test cases.
+#     '''
+
+#     _outcomes = {
+#         0:  'JOBE outcome: Successful run',
+#         11: 'JOBE outcome: Compile error',
+#         12: 'JOBE outcome: Runtime error',
+#         13: 'JOBE outcome: Time limit exceeded',
+#         15: 'JOBE outcome: Successful run',
+#         17: 'JOBE outcome: Memory limit exceeded',
+#         19: 'JOBE outcome: Illegal system call',
+#         20: 'JOBE outcome: Internal error, please report',
+#         21: 'JOBE outcome: Server overload',
+#         }
+
+#     # ------------------------------------------------------------------------
+#     def __init__(self, q: QDict) -> None:
+#         super().__init__(q)
+
+#         self.set_defaults(QDict({
+#             'text': '',
+#             'timeout': 5,   # seconds
+#             'server': '127.0.0.1',  # JOBE server
+#             'language': 'c',
+#             'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
+#             }))
+
+    # ------------------------------------------------------------------------
+    # def correct(self) -> None:
+    #     super().correct()
+
+    #     if self['answer'] is None:
+    #         return
+
+    #     # submit answer to JOBE server
+    #     resource = '/jobe/index.php/restapi/runs/'
+    #     headers = {"Content-type": "application/json; charset=utf-8",
+    #                "Accept": "application/json"}
+
+    #     for expected in self['correct']:
+    #         data_json = json.dumps({
+    #             'run_spec' : {
+    #                 'language_id': self['language'],
+    #                 'sourcecode': self['answer'],
+    #                 'input': expected.get('stdin', ''),
+    #                  },
+    #             })
+
+    #         try:
+    #             connect = http.client.HTTPConnection(self['server'])
+    #             connect.request(
+    #                 method='POST',
+    #                 url=resource,
+    #                 body=data_json,
+    #                 headers=headers
+    #                 )
+    #             response = connect.getresponse()
+    #             logger.debug('JOBE response status %d', response.status)
+    #             if response.status != 204:
+    #                 content = response.read().decode('utf8')
+    #                 if content:
+    #                     result = json.loads(content)
+    #             connect.close()
+
+    #         except (HTTPError, ValueError):
+    #             logger.error('HTTPError while connecting to JOBE server')
+
+    #         try:
+    #             outcome = result['outcome']
+    #         except (NameError, TypeError, KeyError):
+    #             logger.error('Bad result returned from JOBE server: %s', result)
+    #             return
+    #         logger.debug(self._outcomes[outcome])
+
+
+
+    #         if result['cmpinfo']:  # compiler errors and warnings
+    #             self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
+    #             self['grade'] = 0.0
+    #             return
+
+    #         if result['stdout'] != expected.get('stdout', ''):
+    #             self['comments'] = 'O output gerado é diferente do esperado.'  # FIXME mostrar porque?
+    #             self['grade'] = 0.0
+    #             return
+
+    #     self['comments'] = 'Ok!'
+    #     self['grade'] = 1.0
+
+
+    # # ------------------------------------------------------------------------
+    # async def correct_async(self) -> None:
+    #     self.correct()  # FIXME there is no async correction!!!
+
+
+            # out = run_script(
+            #     script=self['correct'],
+            #     args=self['args'],
+            #     stdin=self['answer'],
+            #     timeout=self['timeout']
+            #     )
+
+            # if out is None:
+            #     logger.warning('No grade after running "%s".', self["correct"])
+            #     self['comments'] = 'O programa de correcção abortou...'
+            #     self['grade'] = 0.0
+            # elif isinstance(out, dict):
+            #     self['comments'] = out.get('comments', '')
+            #     try:
+            #         self['grade'] = float(out['grade'])
+            #     except ValueError:
+            #         logger.error('Output error in "%s".', self["correct"])
+            #     except KeyError:
+            #         logger.error('No grade in "%s".', self["correct"])
+            # else:
+            #     try:
+            #         self['grade'] = float(out)
+            #     except (TypeError, ValueError):
+            #         logger.error('Invalid grade in "%s".', self["correct"])
+
+# ============================================================================
 class QuestionInformation(Question):
     '''
     Not really a question, just an information panel.
@@ -591,8 +755,11 @@ class QuestionInformation(Question):
     '''
     # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
+    # def __init__(self, q: QDict) -> None:
+    #     super().__init__(q)
+
+    def gen(self) -> None:
+        super().gen()
         self.set_defaults(QDict({
             'text': '',
             }))
@@ -603,6 +770,46 @@ class QuestionInformation(Question):
         self['grade'] = 1.0  # always "correct" but points should be zero!
+
+# ============================================================================
+def question_from(qdict: QDict) -> Question:
+    '''
+    Converts a question specified in a dict into an instance of Question()
+    '''
+    types = {
+        'radio':            QuestionRadio,
+        'checkbox':         QuestionCheckbox,
+        'text':             QuestionText,
+        'text-regex':       QuestionTextRegex,
+        'numeric-interval': QuestionNumericInterval,
+        'textarea':         QuestionTextArea,
+        # 'code':             QuestionCode,
+        # -- informative panels --
+        'information':      QuestionInformation,
+        'success':          QuestionInformation,
+        'warning':          QuestionInformation,
+        'alert':            QuestionInformation,
+        }
+
+    # Get class for this question type
+    try:
+        qclass = types[qdict['type']]
+    except KeyError:
+        logger.error('Invalid type "%s" in "%s"',
+                     qdict['type'], qdict['ref'])
+        raise
+
+    # Create an instance of Question() of appropriate type
+    try:
+        qinstance = qclass(QDict(qdict))
+    except QuestionException:
+        logger.error('Error generating "%s" in %s/%s',
+                     qdict['ref'], qdict['path'], qdict['filename'])
+        raise
+
+    return qinstance
+
+
 # ============================================================================
 class QFactory():
     '''
@@ -636,24 +843,8 @@ class QFactory():
       grade = question['grade']        # get grade
     '''
-    # Depending on the type of question, a different question class will be
-    # instantiated. All these classes derive from the base class `Question`.
-    _types = {
-        'radio':            QuestionRadio,
-        'checkbox':         QuestionCheckbox,
-        'text':             QuestionText,
-        'text-regex':       QuestionTextRegex,
-        'numeric-interval': QuestionNumericInterval,
-        'textarea':         QuestionTextArea,
-        # -- informative panels --
-        'information':      QuestionInformation,
-        'success':          QuestionInformation,
-        'warning':          QuestionInformation,
-        'alert':            QuestionInformation,
-        }
-
     def __init__(self, qdict: QDict = QDict({})) -> None:
-        self.question = qdict
+        self.qdict = qdict
     # ------------------------------------------------------------------------
     async def gen_async(self) -> Question:
@@ -662,44 +853,28 @@ class QFactory():
         which is a descendent of base class Question.
         '''
-        logger.debug('generating %s...', self.question["ref"])
+        logger.debug('generating %s...', self.qdict["ref"])
         # Shallow copy so that script generated questions will not replace
         # the original generators
-        question = self.question.copy()
-        question['qid'] = str(uuid.uuid4())  # unique for each question
+        qdict = self.qdict.copy()
+        qdict['qid'] = str(uuid.uuid4())  # unique for each question
         # If question is of generator type, an external program will be run
         # which will print a valid question in yaml format to stdout. This
         # output is then yaml parsed into a dictionary `q`.
-        if question['type'] == 'generator':
-            logger.debug(' \\_ Running "%s".', question['script'])
-            question.setdefault('args', [])
-            question.setdefault('stdin', '')
-            script = path.join(question['path'], question['script'])
+        if qdict['type'] == 'generator':
+            logger.debug(' \\_ Running "%s".', qdict['script'])
+            qdict.setdefault('args', [])
+            qdict.setdefault('stdin', '')
+            script = path.join(qdict['path'], qdict['script'])
             out = await run_script_async(script=script,
-                                         args=question['args'],
-                                         stdin=question['stdin'])
-            question.update(out)
+                                         args=qdict['args'],
+                                         stdin=qdict['stdin'])
+            qdict.update(out)
-        # Get class for this question type
-        try:
-            qclass = self._types[question['type']]
-        except KeyError:
-            logger.error('Invalid type "%s" in "%s"',
-                         question['type'], question['ref'])
-            raise
-
-        # Finally create an instance of Question()
-        try:
-            qinstance = qclass(QDict(question))
-        except QuestionException:
-            logger.error('Error generating question "%s". See "%s/%s"',
-                         question['ref'],
-                         question['path'],
-                         question['filename'])
-            raise
-
-        return qinstance
+        question = question_from(qdict)  # returns a Question instance
+        question.gen()
+        return question
     # ------------------------------------------------------------------------
     def generate(self) -> Question:
@@ -5,8 +5,8 @@ Handles the web, http &amp; html part of the application interface.
 Uses the tornadoweb framework.
 '''
-
 # python standard library
+import asyncio
 import base64
 import functools
 import json
@@ -161,6 +161,54 @@ class BaseHandler(tornado.web.RequestHandler):
 #         AdminSocketHandler.send_updates(chat)  # send to clients
 # ----------------------------------------------------------------------------
+# pylint: disable=abstract-method
+class LoginHandler(BaseHandler):
+    '''Handles /login'''
+
+    _prefix = re.compile(r'[a-z]')
+    _error_msg = {
+        'wrong_password': 'Password errada',
+        'already_online': 'Já está online, não pode entrar duas vezes',
+        'unauthorized': 'Não está autorizado a fazer o teste'
+    }
+
+    def get(self):
+        '''Render login page.'''
+        self.render('login.html', error='')
+
+    async def post(self):
+        '''Authenticates student and login.'''
+        uid = self._prefix.sub('', self.get_body_argument('uid'))
+        password = self.get_body_argument('pw')
+        headers = {
+            'remote_ip': self.request.remote_ip,
+            'user_agent': self.request.headers.get('User-Agent')
+        }
+
+        error = await self.testapp.login(uid, password, headers)
+
+        if error:
+            await asyncio.sleep(3)  # delay to avoid spamming the server...
+            self.render('login.html', error=self._error_msg[error])
+        else:
+            self.set_secure_cookie('perguntations_user', str(uid))
+            self.redirect('/')
+
+
+# ----------------------------------------------------------------------------
+# pylint: disable=abstract-method
+class LogoutHandler(BaseHandler):
+    '''Handle /logout'''
+
+    @tornado.web.authenticated
+    def get(self):
+        '''Logs out a user.'''
+        self.clear_cookie('perguntations_user')
+        self.testapp.logout(self.current_user)
+        self.render('login.html', error='')
+
+
+# ----------------------------------------------------------------------------
 # Test shown to students
 # ----------------------------------------------------------------------------
 # pylint: disable=abstract-method
@@ -179,6 +227,7 @@ class RootHandler(BaseHandler):
         'text-regex':   'question-text.html',
         'numeric-interval': 'question-text.html',
         'textarea':     'question-textarea.html',
+        'code':         'question-textarea.html',
         # -- information panels --
         'information':  'question-information.html',
         'success':      'question-information.html',
@@ -190,18 +239,19 @@ class RootHandler(BaseHandler):
     @tornado.web.authenticated
     async def get(self):
         '''
-        Sends test to student or redirects 0 to admin page
+        Handles GET /
+        Sends test to student or redirects 0 to admin page.
+        Multiple calls to this function will return the same test.
         '''
         uid = self.current_user
-        logging.info('"%s" GET /', uid)
+        logging.debug('"%s" GET /', uid)
+
         if uid == '0':
             self.redirect('/admin')
+            return
-        test = self.testapp.get_student_test(uid)  # reloading returns same test
-        if test is None:
-            test = await self.testapp.generate_test(uid)
-
+        test = await self.testapp.get_test_or_generate(uid)
         self.render('test.html', t=test, md=md_to_html, templ=self._templates)
@@ -222,7 +272,7 @@ class RootHandler(BaseHandler):
         logging.debug('"%s" POST /', uid)
         try:
-            test = self.testapp.get_student_test(uid)
+            test = self.testapp.get_test(uid)
         except KeyError as exc:
             logging.warning('"%s" POST / raised 403 Forbidden', uid)
             raise tornado.web.HTTPError(403) from exc # Forbidden
@@ -232,7 +282,6 @@ class RootHandler(BaseHandler):
             qid = str(i)
             if 'answered-' + qid in self.request.arguments:
                 ans[i] = self.get_body_arguments(qid)
-                # print(i, ans[i])
                 # remove enclosing list in some question types
                 if question['type'] == 'radio':
@@ -241,62 +290,22 @@ class RootHandler(BaseHandler):
                     else:
                         ans[i] = ans[i][0]
                 elif question['type'] in ('text', 'text-regex', 'textarea',
-                                          'numeric-interval'):
+                                          'numeric-interval', 'code'):
                     ans[i] = ans[i][0]
-        # correct answered questions and logout
-        await self.testapp.correct_test(uid, ans)
+        # submit answered questions, correct
+        await self.testapp.submit_test(uid, ans)
         # show final grade and grades of other tests in the database
-        allgrades = self.testapp.get_student_grades_from_all_tests(uid)
+        # allgrades = self.testapp.get_student_grades_from_all_tests(uid)
+        grade = self.testapp.get_student_grade(uid)
+        self.render('grade.html', t=test)
         self.clear_cookie('perguntations_user')
-        self.render('grade.html', t=test, allgrades=allgrades)
         self.testapp.logout(uid)
         timeit_finish = timer()
-        logging.info('  correction took %fs', timeit_finish-timeit_start)
-
-# ----------------------------------------------------------------------------
-# pylint: disable=abstract-method
-class LoginHandler(BaseHandler):
-    '''Handles /login'''
-
-    _prefix = re.compile(r'[a-z]')
-
-    def get(self):
-        '''Render login page.'''
-        self.render('login.html', error='')
-
-    async def post(self):
-        '''Authenticates student and login.'''
-        uid = self._prefix.sub('', self.get_body_argument('uid'))
-        password = self.get_body_argument('pw')
-        login_ok = await self.testapp.login(uid, password)
-
-        if login_ok:
-            self.set_secure_cookie('perguntations_user', str(uid), expires_days=1)
-            self.redirect('/')
-        else:
-            self.render('login.html', error='Não autorizado ou senha inválida')
-
-
-# ----------------------------------------------------------------------------
-# pylint: disable=abstract-method
-class LogoutHandler(BaseHandler):
-    '''Handle /logout'''
-
-    @tornado.web.authenticated
-    def get(self):
-        '''Logs out a user.'''
-        self.clear_cookie('perguntations_user')
-        self.testapp.logout(self.current_user)
-        self.redirect('/')
-
-    def on_finish(self):
-        self.testapp.logout(self.current_user)
-
-
+        logging.info('  elapsed time: %fs', timeit_finish-timeit_start)
 # ----------------------------------------------------------------------------
@@ -456,7 +465,6 @@ class FileHandler(BaseHandler):
                 break
-
 # --- REVIEW -----------------------------------------------------------------
 # pylint: disable=abstract-method
 class ReviewHandler(BaseHandler):
@@ -471,6 +479,7 @@ class ReviewHandler(BaseHandler):
         'text-regex':   'review-question-text.html',
         'numeric-interval': 'review-question-text.html',
         'textarea':     'review-question-text.html',
+        'code':         'review-question-text.html',
         # -- information panels --
         'information':  'review-question-information.html',
         'success':      'review-question-information.html',
@@ -40,68 +40,22 @@
 <!-- ================================================================== -->
 <div class="container">
   <div class="jumbotron">
-    {% if t['state'] == 'FINISHED' %}
-      <h1>Resultado:
-        <strong>{{ f'{round(t["grade"], 3)}' }}</strong>
-        valores na escala de {{t['scale'][0]}} a {{t['scale'][1]}}.
-      </h1>
-      <p>O seu teste foi correctamente entregue e a nota registada.</p>
-      <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para sair do teste</a></p>
+    {% if t['state'] == 'CORRECTED' %}
       {% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}
         <i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>
       {% end %}
+      <h3>Resultado:
+        <strong>{{ f'{round(t["grade"], 3)}' }}</strong>
+        valores na escala [{{t['scale'][0]}},{{t['scale'][1]}}].
+      </h3>
+    {% elif t['state'] == 'SUBMITTED' %}
+      <h3>A prova foi submetida com sucesso. Vai ser corrigida mais tarde.</h3>
     {% elif t['state'] == 'QUIT' %}
-      <p>Foi registada a sua desistência da prova.</p>
+      <h3>Foi registada a sua desistência da prova.</h3>
     {% end %}
+    <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para terminar</a></p>
   </div> <!-- jumbotron -->
-
-  <div class="card">
-    <h5 class="card-header">
-        Histórico de resultados
-    </h5>
-    <table class="table table-condensed noleftmargin">
-      <thead>
-          <tr>
-              <th>Prova</th>
-              <th>Data</th>
-              <th>Hora</th>
-              <th>Nota</th>
-          </tr>
-      </thead>
-      <tbody>
-        {% for g in allgrades %}
-          <tr>
-            <td>{{g[0]}}</td>  <!-- teste -->
-            <td>{{g[2][:10]}}</td>  <!-- data -->
-            <td>{{g[2][11:19]}}</td>  <!-- hora -->
-            <td> <!-- progress column -->
-              <div class="progress" style="height: 20px;">
-                <div class="progress-bar
-                  {% if g[1] - t['scale'][0] < 0.5*(t['scale'][1] - t['scale'][0]) %}
-                      bg-danger
-                  {% elif g[1]  - t['scale'][0] < 0.75*(t['scale'][1] - t['scale'][0]) %}
-                      bg-warning
-                  {% else %}
-                      bg-success
-                  {% end %}
-                  "
-                  role="progressbar"
-                  aria-valuenow="{{ 100*(g[1] - t['scale'][0])/(t['scale'][1] - t['scale'][0]) }}"
-                  aria-valuemin="0"
-                  aria-valuemax="100"
-                  style="min-width: 2em; width: {{ 100*(g[1]-t['scale'][0])/(t['scale'][1]-t['scale'][0]) }}%;">
-
-                    {{ str(round(g[1], 1)) }}
-
-                </div> <!-- progress-bar -->
-              </div> <!-- progress -->
-            </td> <!-- progress column -->
-          </tr>
-        {% end %}
-      </tbody>
-    </table>
-  </div> <!-- panel -->
 </div> <!-- container -->
 </body>
 </html>
@@ -12,7 +12,6 @@
   <!-- Scripts -->
   <script src="/static/jquery/jquery.min.js"></script>
-  <!-- <script defer src="/static/popper.js/popper.min.js"></script> -->
   <script defer src="/static/fontawesome-free/js/all.min.js"></script>
   <script defer src="/static/bootstrap/js/bootstrap.bundle.min.js"></script>
@@ -32,45 +32,46 @@
         </p>
       </div> <!-- card-body -->
-      <div class="card-footer">
-        {% if q['grade'] > 0.99 %}
-          <p class="text-success">
-            <i class="far fa-thumbs-up fa-3x" aria-hidden="true"></i>
-            {{ round(q['grade'] * q['points'], 2) }}
-            pontos
-          </p>
-          <p class="text-success">{{ q['comments'] }}</p>
-        {% elif q['grade'] > 0.49 %}
-          <p class="text-warning">
-            <i class="fas fa-exclamation-triangle fa-3x" aria-hidden="true"></i>
-            {{ round(q['grade'] * q['points'], 2) }}
-            pontos
-          </p>
-          <p class="text-warning">{{ q['comments'] }}</p>
-          {% if q.get('solution', '') %}
-            <hr>
-            {{ md('**Solução:** \n\n' + q['solution']) }}
+      {% if 'grade' in q %}
+        <div class="card-footer">
+          {% if q['grade'] > 0.999 %}
+            <p class="text-success">
+              <i class="far fa-thumbs-up fa-3x" aria-hidden="true"></i>
+              {{ round(q['grade'] * q['points'], 2) }}
+              pontos
+            </p>
+            <p class="text-success">{{ md(q['comments']) }}</p>
+          {% elif q['grade'] >= 0.5 %}
+            <p class="text-warning">
+              <i class="fas fa-exclamation-triangle fa-3x" aria-hidden="true"></i>
+              {{ round(q['grade'] * q['points'], 2) }}
+              pontos
+            </p>
+            <p class="text-warning">{{ md(q['comments']) }}</p>
+            {% if q['solution'] %}
+              <hr>
+              {{ md('**Solução:** \n\n' + q['solution']) }}
+            {% end %}
+          {% else %}
+            <p class="text-danger">
+              <i class="far fa-thumbs-down fa-3x" aria-hidden="true"></i>
+              {{ round(q['grade'] * q['points'], 2) }}
+              pontos
+            </p>
+            <p class="text-danger">{{ md(q['comments']) }}</p>
+            {% if q['solution'] %}
+              <hr>
+              {{ md('**Solução:** \n\n' + q['solution']) }}
+            {% end %}
           {% end %}
-        {% else %}
-          <p class="text-danger">
-            <i class="far fa-thumbs-down fa-3x" aria-hidden="true"></i>
-            {{ round(q['grade'] * q['points'], 2) }}
-            pontos
-          </p>
-          <p class="text-danger">{{ q['comments'] }}</p>
-          {% if q.get('solution', '') %}
+
+          {% if t['show_ref'] %}
             <hr>
-            {{ md('**Solução:** \n\n' + q['solution']) }}
+            file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
+            ref:  <code>{{ q['ref'] }}</code>
           {% end %}
-        {% end %}
-
-        {% if t['show_ref'] %}
-          <hr>
-          file: <code>{{ q['path'] }}/{{ q['filename'] }}</code><br>
-          ref:  <code>{{ q['ref'] }}</code>
-        {% end %}
-
-      </div> <!-- card-footer -->
+        </div> <!-- card-footer -->
+      {% end %}
     </div> <!-- card -->
   {% else %}
@@ -97,12 +98,12 @@
           </small>
         </p>
       </div> <!-- card-body -->
+
       <div class="card-footer">
         <p class="text-secondary">
           <i class="fas fa-ban fa-3x" aria-hidden="true"></i>
-          {{ round(q['grade'] * q['points'], 2) }} pontos<br>
-          {{ q['comments'] }}
-          {% if q.get('solution', '') %}
+          {{ md(q['comments']) }}
+          {% if q['solution'] %}
             <hr>
             {{ md('**Solução:** \n\n' + q['solution']) }}
           {% end %}
@@ -97,7 +97,7 @@
       <div class="row">
         <label for="nota" class="col-sm-2">Nota:</label>
         <div class="col-sm-10" id="nota">
-          <span class="badge badge-primary">{{ round(t['grade'], 1) }}</span> valores
+          <span class="badge badge-primary">{{ round(t['grade'], 2) }}</span> valores
           {% if t['state'] == 'QUIT' %}
               (DESISTÊNCIA)
           {% end %}
@@ -44,7 +44,7 @@
 <!-- ===================================================================== -->
 <body>
 <!-- ===================================================================== -->
-<div class="progress fixed-top" style="height: 61px; border-radius: 0px;">
+<div class="progress fixed-top" style="height: 62px; border-radius: 0px;">
   <div class="progress-bar bg-secondary" role="progressbar" style="width: 100%" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"></div>
 </div>
@@ -94,10 +94,6 @@
     <h5>
       <div class="row">
-        <label for="inicio" class="col-sm-3">Início:</label>
-        <div class="col-sm-9" id="inicio">{{ str(t['start_time'].time())[:8]}}</div>
-      </div>
-      <div class="row">
         <label for="duracao" class="col-sm-3">Duração:</label>
         <div class="col-sm-9" id="duracao">{{ str(t['duration'])+' minutos' if t['duration'] > 0 else 'sem limite de tempo' }}</div>
       </div>
 '''
-TestFactory - generates tests for students
 Test - instances of this class are individual tests
 '''
-
 # python standard library
-from os import path
-import random
 from datetime import datetime
+import json
 import logging
-import re
-
-# this project
-from perguntations.questions import QFactory, QuestionException
-from perguntations.tools import load_yaml
+from math import nan
+from os import path
 # Logger configuration
 logger = logging.getLogger(__name__)
 # ============================================================================
-class TestFactoryException(Exception):
-    '''exception raised in this module'''
-
-
-# ============================================================================
-class TestFactory(dict):
-    '''
-    Each instance of TestFactory() is a test generator.
-    For example, if we want to serve two different tests, then we need two
-    instances of TestFactory(), one for each test.
-    '''
-
-    # ------------------------------------------------------------------------
-    def __init__(self, conf):
-        '''
-        Loads configuration from yaml file, then overrides some configurations
-        using the conf argument.
-        Base questions are added to a pool of questions factories.
-        '''
-
-        # --- set test defaults and then use given configuration
-        super().__init__({  # defaults
-            'title': '',
-            'show_points': True,
-            'scale': None,  # or [0, 20]
-            'duration': 0,  # 0=infinite
-            'autosubmit': False,
-            'debug': False,
-            'show_ref': False,
-            })
-        self.update(conf)
-
-        # --- perform sanity checks and normalize the test questions
-        self.sanity_checks()
-        logger.info('Sanity checks PASSED.')
-
-        # --- find refs of all questions used in the test
-        qrefs = {r for qq in self['questions'] for r in qq['ref']}
-        logger.info('Declared %d questions (each test uses %d).',
-                    len(qrefs), len(self["questions"]))
-
-        # --- for review, we are done. no factories needed
-        if self['review']:
-            logger.info('Review mode. No questions loaded. No factories.')
-            return
-
-        # --- load and build question factories
-        self.question_factory = {}
-
-        counter = 1
-        for file in self["files"]:
-            fullpath = path.normpath(path.join(self["questions_dir"], file))
-            (dirname, filename) = path.split(fullpath)
-
-            logger.info('Loading "%s"...', fullpath)
-            questions = load_yaml(fullpath)   # , default=[])
-
-            for i, question in enumerate(questions):
-                # make sure every question in the file is a dictionary
-                if not isinstance(question, dict):
-                    msg = f'Question {i} in {file} is not a dictionary'
-                    raise TestFactoryException(msg)
-
-                # check if ref is missing, then set to '/path/file.yaml:3'
-                if 'ref' not in question:
-                    question['ref'] = f'{file}:{i:04}'
-                    logger.warning('Missing ref set to "%s"', question["ref"])
-
-                # check for duplicate refs
-                if question['ref'] in self.question_factory:
-                    other = self.question_factory[question['ref']]
-                    otherfile = path.join(other.question['path'],
-                                          other.question['filename'])
-                    msg = (f'Duplicate reference "{question["ref"]}" in files '
-                           f'"{otherfile}" and "{fullpath}".')
-                    raise TestFactoryException(msg)
-
-                # make factory only for the questions used in the test
-                if question['ref'] in qrefs:
-                    question.setdefault('type', 'information')
-                    question.update({
-                        'filename': filename,
-                        'path': dirname,
-                        'index': i            # position in the file, 0 based
-                        })
-
-                    self.question_factory[question['ref']] = QFactory(question)
-
-                    # check if all the questions can be correctly generated
-                    try:
-                        self.question_factory[question['ref']].generate()
-                    except Exception as exc:
-                        msg = f'Failed to generate "{question["ref"]}"'
-                        raise TestFactoryException(msg) from exc
-                    else:
-                        logger.info('%4d.  "%s" Ok.', counter, question["ref"])
-                    counter += 1
-
-        qmissing = qrefs.difference(set(self.question_factory.keys()))
-        if qmissing:
-            raise TestFactoryException(f'Could not find questions {qmissing}.')
-
-    # ------------------------------------------------------------------------
-    def check_test_ref(self):
-        '''Test must have a `ref`'''
-        if 'ref' not in self:
-            raise TestFactoryException('Missing "ref" in configuration!')
-        if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
-            raise TestFactoryException('Test "ref" can only contain the '
-                                       'characters a-zA-Z0-9_-')
-
-    def check_missing_database(self):
-        '''Test must have a database'''
-        if 'database' not in self:
-            raise TestFactoryException('Missing "database" in configuration')
-        if not path.isfile(path.expanduser(self['database'])):
-            msg = f'Database "{self["database"]}" not found!'
-            raise TestFactoryException(msg)
-
-    def check_missing_answers_directory(self):
-        '''Test must have a answers directory'''
-        if 'answers_dir' not in self:
-            msg = 'Missing "answers_dir" in configuration'
-            raise TestFactoryException(msg)
-
-    def check_answers_directory_writable(self):
-        '''Answers directory must be writable'''
-        testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
-        try:
-            with open(testfile, 'w') as file:
-                file.write('You can safely remove this file.')
-        except OSError as exc:
-            msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
-            raise TestFactoryException(msg) from exc
-
-    def check_questions_directory(self):
-        '''Check if questions directory is missing or not accessible.'''
-        if 'questions_dir' not in self:
-            logger.warning('Missing "questions_dir". Using "%s"',
-                           path.abspath(path.curdir))
-            self['questions_dir'] = path.curdir
-        elif not path.isdir(path.expanduser(self['questions_dir'])):
-            raise TestFactoryException(f'Can\'t find questions directory '
-                                       f'"{self["questions_dir"]}"')
-
-    def check_import_files(self):
-        '''Check if there are files to import (with questions)'''
-        if 'files' not in self:
-            msg = ('Missing "files" in configuration with the list of '
-                   'question files to import!')
-            raise TestFactoryException(msg)
-
-        if isinstance(self['files'], str):
-            self['files'] = [self['files']]
-
-    def check_question_list(self):
-        '''normalize question list'''
-        if 'questions' not in self:
-            raise TestFactoryException('Missing "questions" in configuration')
-
-        for i, question in enumerate(self['questions']):
-            # normalize question to a dict and ref to a list of references
-            if isinstance(question, str):  # e.g.,    - some_ref
-                question = {'ref': [question]}    # becomes  - ref: [some_ref]
-            elif isinstance(question, dict) and isinstance(question['ref'], str):
-                question['ref'] = [question['ref']]
-            elif isinstance(question, list):
-                question = {'ref': [str(a) for a in question]}
-
-            self['questions'][i] = question
-
-    def check_missing_title(self):
-        '''Warns if title is missing'''
-        if not self['title']:
-            logger.warning('Title is undefined!')
-
-    def check_grade_scaling(self):
-        '''Just informs the scale limits'''
-        if 'scale_points' in self:
-            msg = ('*** DEPRECATION WARNING: *** scale_points, scale_min, '
-                   'scale_max were replaced by "scale: [min, max]".')
-            logger.warning(msg)
-            self['scale'] = [self['scale_min'], self['scale_max']]
-
-
-    # ------------------------------------------------------------------------
-    def sanity_checks(self):
-        '''
-        Checks for valid keys and sets default values.
-        Also checks if some files and directories exist
-        '''
-        self.check_test_ref()
-        self.check_missing_database()
-        self.check_missing_answers_directory()
-        self.check_answers_directory_writable()
-        self.check_questions_directory()
-        self.check_import_files()
-        self.check_question_list()
-        self.check_missing_title()
-        self.check_grade_scaling()
-
-    # ------------------------------------------------------------------------
-    async def generate(self):
-        '''
-        Given a dictionary with a student dict {'name':'john', 'number': 123}
-        returns instance of Test() for that particular student
-        '''
-
-        # make list of questions
-        questions = []
-        qnum = 1  # track question number
-        nerr = 0  # count errors during questions generation
-
-        for qlist in self['questions']:
-            # choose list of question variants
-            choose = qlist.get('choose', 1)
-            qrefs = random.sample(qlist['ref'], k=choose)
-
-            for qref in qrefs:
-                # generate instance of question
-                try:
-                    question = await self.question_factory[qref].gen_async()
-                except QuestionException:
-                    logger.error('Can\'t generate question "%s". Skipping.', qref)
-                    nerr += 1
-                    continue
-
-                # some defaults
-                if question['type'] in ('information', 'success', 'warning',
-                                        'alert'):
-                    question['points'] = qlist.get('points', 0.0)
-                else:
-                    question['points'] = qlist.get('points', 1.0)
-                    question['number'] = qnum  # counter for non informative panels
-                    qnum += 1
-
-                questions.append(question)
-
-        # setup scale
-        total_points = sum(q['points'] for q in questions)
-
-        if total_points > 0:
-            # normalize question points to scale
-            if self['scale'] is not None:
-                scale_min, scale_max = self['scale']
-                for question in questions:
-                    question['points'] *= (scale_max - scale_min) / total_points
-            else:
-                self['scale'] = [0, total_points]
-        else:
-            logger.warning('Total points is **ZERO**.')
-            if self['scale'] is None:
-                self['scale'] = [0, 20]  # default
-
-        if nerr > 0:
-            logger.error('%s errors found!', nerr)
-
-        # copy these from the test configuratoin to each test instance
-        inherit = {'ref', 'title', 'database', 'answers_dir',
-                   'questions_dir', 'files',
-                   'duration', 'autosubmit',
-                   'scale', 'show_points',
-                   'show_ref', 'debug', }
-        # NOT INCLUDED:  testfile, allow_all, review
-
-        return Test({'questions': questions, **{k:self[k] for k in inherit}})
-
-    # ------------------------------------------------------------------------
-    def __repr__(self):
-        testsettings = '\n'.join(f'  {k:14s}: {v}' for k, v in self.items())
-        return '{\n' + testsettings + '\n}'
-
-
-# ============================================================================
 class Test(dict):
     '''
     Each instance Test() is a concrete test of a single student.
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, d):
-    #     super().__init__(d)
+    def __init__(self, d):
+        super().__init__(d)
+        self['grade'] = nan
+        self['comment'] = ''
     # ------------------------------------------------------------------------
-    def start(self, student):
+    def start(self, student: dict) -> None:
         '''
         Write student id in the test and register start time
         '''
@@ -312,59 +34,75 @@ class Test(dict):
         self['start_time'] = datetime.now()
         self['finish_time'] = None
         self['state'] = 'ACTIVE'
-        self['comment'] = ''
     # ------------------------------------------------------------------------
-    def reset_answers(self):
+    def reset_answers(self) -> None:
         '''Removes all answers from the test (clean)'''
         for question in self['questions']:
             question['answer'] = None
     # ------------------------------------------------------------------------
-    def update_answer(self, ref, ans):
+    def update_answer(self, ref: str, ans) -> None:
         '''updates one answer in the test'''
         self['questions'][ref].set_answer(ans)
     # ------------------------------------------------------------------------
-    def update_answers(self, answers_dict):
+    def submit(self, answers_dict) -> None:
         '''
         Given a dictionary ans={'ref': 'some answer'} updates the answers of
         multiple questions in the test.
         Only affects the questions referred in the dictionary.
         '''
+        self['finish_time'] = datetime.now()
         for ref, ans in answers_dict.items():
             self['questions'][ref].set_answer(ans)
-            # self['questions'][ref]['answer'] = ans
+        self['state'] = 'SUBMITTED'
     # ------------------------------------------------------------------------
-    async def correct(self):
+    async def correct_async(self) -> None:
         '''Corrects all the answers of the test and computes the final grade'''
-        self['finish_time'] = datetime.now()
-        self['state'] = 'FINISHED'
-
         grade = 0.0
         for question in self['questions']:
             await question.correct_async()
             grade += question['grade'] * question['points']
             logger.debug('Correcting %30s: %3g%%',
-                         question["ref"], question["grade"]*100)
+                         question['ref'], question['grade']*100)
+
+        # truncate to avoid negative final grade and adjust scale
+        self['grade'] = max(0.0, grade) + self['scale'][0]
+        self['state'] = 'CORRECTED'
+
+    # ------------------------------------------------------------------------
+    def correct(self) -> None:
+        '''Corrects all the answers of the test and computes the final grade'''
+        grade = 0.0
+        for question in self['questions']:
+            question.correct()
+            grade += question['grade'] * question['points']
+            logger.debug('Correcting %30s: %3g%%',
+                         question['ref'], question['grade']*100)
         # truncate to avoid negative final grade and adjust scale
         self['grade'] = max(0.0, grade) + self['scale'][0]
-        return self['grade']
+        self['state'] = 'CORRECTED'
     # ------------------------------------------------------------------------
-    def giveup(self):
+    def giveup(self) -> None:
         '''Test is marqued as QUIT and is not corrected'''
         self['finish_time'] = datetime.now()
         self['state'] = 'QUIT'
         self['grade'] = 0.0
-        logger.info('Student %s:  gave up.', self["student"]["number"])
-        return self['grade']
     # ------------------------------------------------------------------------
-    def __str__(self):
-        return ('Test:\n'
-                f'  student:  {self.get("student", "--")}\n'
-                f'  start_time:  {self.get("start_time", "--")}\n'
-                f'  questions:  {", ".join(q["ref"] for q in self["questions"])}\n')
+    def save_json(self, pathfile) -> None:
+        '''save test in JSON format'''
+        with open(pathfile, 'w') as file:
+            json.dump(self, file, indent=2, default=str)  # str for datetime
+
+    # ------------------------------------------------------------------------
+    def __str__(self) -> str:
+        return '\n'.join([f'{k}: {v}' for k,v in self.items()])
+        # return ('Test:\n'
+        #         f'  student:  {self.get("student", "--")}\n'
+        #         f'  start_time:  {self.get("start_time", "--")}\n'
+        #         f'  questions:  {", ".join(q["ref"] for q in self["questions"])}\n')
@@ -0,0 +1,341 @@
+'''
+TestFactory - generates tests for students
+'''
+
+# python standard library
+from os import path
+import random
+import logging
+import re
+from typing import Any, Dict
+
+# this project
+from perguntations.questions import QFactory, QuestionException
+from perguntations.test import Test
+from perguntations.tools import load_yaml
+
+# Logger configuration
+logger = logging.getLogger(__name__)
+
+
+# ============================================================================
+class TestFactoryException(Exception):
+    '''exception raised in this module'''
+
+
+# ============================================================================
+class TestFactory(dict):
+    '''
+    Each instance of TestFactory() is a test generator.
+    For example, if we want to serve two different tests, then we need two
+    instances of TestFactory(), one for each test.
+    '''
+
+    # ------------------------------------------------------------------------
+    def __init__(self, conf: Dict[str, Any]) -> None:
+        '''
+        Loads configuration from yaml file, then overrides some configurations
+        using the conf argument.
+        Base questions are added to a pool of questions factories.
+        '''
+
+        # --- set test defaults and then use given configuration
+        super().__init__({  # defaults
+            'title': '',
+            'show_points': True,
+            'scale': None,
+            'duration': 0,  # 0=infinite
+            'autosubmit': False,
+            'autocorrect': True,
+            'debug': False,
+            'show_ref': False,
+            })
+        self.update(conf)
+
+        # --- for review, we are done. no factories needed
+        if self['review']:
+            logger.info('Review mode. No questions loaded. No factories.')
+            return
+
+        # --- perform sanity checks and normalize the test questions
+        self.sanity_checks()
+        logger.info('Sanity checks PASSED.')
+
+        # --- find refs of all questions used in the test
+        qrefs = {r for qq in self['questions'] for r in qq['ref']}
+        logger.info('Declared %d questions (each test uses %d).',
+                    len(qrefs), len(self["questions"]))
+
+        # --- load and build question factories
+        self['question_factory'] = {}
+
+        for file in self["files"]:
+            fullpath = path.normpath(path.join(self["questions_dir"], file))
+
+            logger.info('Loading "%s"...', fullpath)
+            questions = load_yaml(fullpath)   # , default=[])
+
+            for i, question in enumerate(questions):
+                # make sure every question in the file is a dictionary
+                if not isinstance(question, dict):
+                    msg = f'Question {i} in {file} is not a dictionary'
+                    raise TestFactoryException(msg)
+
+                # check if ref is missing, then set to '/path/file.yaml:3'
+                if 'ref' not in question:
+                    question['ref'] = f'{file}:{i:04}'
+                    logger.warning('Missing ref set to "%s"', question["ref"])
+
+                # check for duplicate refs
+                if question['ref'] in self['question_factory']:
+                    other = self['question_factory'][question['ref']]
+                    otherfile = path.join(other.question['path'],
+                                          other.question['filename'])
+                    msg = (f'Duplicate reference "{question["ref"]}" in files '
+                           f'"{otherfile}" and "{fullpath}".')
+                    raise TestFactoryException(msg)
+
+                # make factory only for the questions used in the test
+                if question['ref'] in qrefs:
+                    question.update(zip(('path', 'filename', 'index'),
+                                        path.split(fullpath) + (i,)))
+                    if question['type'] == 'code' and 'server' not in question:
+                        try:
+                            question['server'] = self['jobe_server']
+                        except KeyError as exc:
+                            msg = f'Missing JOBE server in "{question["ref"]}"'
+                            raise TestFactoryException(msg) from exc
+
+                    self['question_factory'][question['ref']] = QFactory(question)
+
+        qmissing = qrefs.difference(set(self['question_factory'].keys()))
+        if qmissing:
+            raise TestFactoryException(f'Could not find questions {qmissing}.')
+
+        self.check_questions()
+
+        logger.info('Test factory ready. No errors found.')
+
+
+    # ------------------------------------------------------------------------
+    def check_test_ref(self) -> None:
+        '''Test must have a `ref`'''
+        if 'ref' not in self:
+            raise TestFactoryException('Missing "ref" in configuration!')
+        if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
+            raise TestFactoryException('Test "ref" can only contain the '
+                                       'characters a-zA-Z0-9_-')
+
+    def check_missing_database(self) -> None:
+        '''Test must have a database'''
+        if 'database' not in self:
+            raise TestFactoryException('Missing "database" in configuration')
+        if not path.isfile(path.expanduser(self['database'])):
+            msg = f'Database "{self["database"]}" not found!'
+            raise TestFactoryException(msg)
+
+    def check_missing_answers_directory(self) -> None:
+        '''Test must have a answers directory'''
+        if 'answers_dir' not in self:
+            msg = 'Missing "answers_dir" in configuration'
+            raise TestFactoryException(msg)
+
+    def check_answers_directory_writable(self) -> None:
+        '''Answers directory must be writable'''
+        testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
+        try:
+            with open(testfile, 'w') as file:
+                file.write('You can safely remove this file.')
+        except OSError as exc:
+            msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
+            raise TestFactoryException(msg) from exc
+
+    def check_questions_directory(self) -> None:
+        '''Check if questions directory is missing or not accessible.'''
+        if 'questions_dir' not in self:
+            logger.warning('Missing "questions_dir". Using "%s"',
+                           path.abspath(path.curdir))
+            self['questions_dir'] = path.curdir
+        elif not path.isdir(path.expanduser(self['questions_dir'])):
+            raise TestFactoryException(f'Can\'t find questions directory '
+                                       f'"{self["questions_dir"]}"')
+
+    def check_import_files(self) -> None:
+        '''Check if there are files to import (with questions)'''
+        if 'files' not in self:
+            msg = ('Missing "files" in configuration with the list of '
+                   'question files to import!')
+            raise TestFactoryException(msg)
+
+        if isinstance(self['files'], str):
+            self['files'] = [self['files']]
+
+    def check_question_list(self) -> None:
+        '''normalize question list'''
+        if 'questions' not in self:
+            raise TestFactoryException('Missing "questions" in configuration')
+
+        for i, question in enumerate(self['questions']):
+            # normalize question to a dict and ref to a list of references
+            if isinstance(question, str):  # e.g.,    - some_ref
+                question = {'ref': [question]}    # becomes  - ref: [some_ref]
+            elif isinstance(question, dict) and isinstance(question['ref'], str):
+                question['ref'] = [question['ref']]
+            elif isinstance(question, list):
+                question = {'ref': [str(a) for a in question]}
+
+            self['questions'][i] = question
+
+    def check_missing_title(self) -> None:
+        '''Warns if title is missing'''
+        if not self['title']:
+            logger.warning('Title is undefined!')
+
+    def check_grade_scaling(self) -> None:
+        '''Just informs the scale limits'''
+        if 'scale_points' in self:
+            msg = ('*** DEPRECATION WARNING: *** scale_points, scale_min, '
+                   'scale_max were replaced by "scale: [min, max]".')
+            logger.warning(msg)
+            self['scale'] = [self['scale_min'], self['scale_max']]
+
+
+    # ------------------------------------------------------------------------
+    def sanity_checks(self) -> None:
+        '''
+        Checks for valid keys and sets default values.
+        Also checks if some files and directories exist
+        '''
+        self.check_test_ref()
+        self.check_missing_database()
+        self.check_missing_answers_directory()
+        self.check_answers_directory_writable()
+        self.check_questions_directory()
+        self.check_import_files()
+        self.check_question_list()
+        self.check_missing_title()
+        self.check_grade_scaling()
+
+    # ------------------------------------------------------------------------
+    def check_questions(self) -> None:
+        '''
+        checks if questions can be correctly generated and corrected
+        '''
+        logger.info('Checking if questions can be generated and corrected...')
+        for i, (qref, qfact) in enumerate(self['question_factory'].items()):
+            try:
+                question = qfact.generate()
+            except Exception as exc:
+                msg = f'Failed to generate "{qref}"'
+                raise TestFactoryException(msg) from exc
+            else:
+                logger.info('%4d.  %s:  Ok', i, qref)
+                # logger.info('         generate Ok')
+
+            if question['type'] in ('code', 'textarea'):
+                if 'tests_right' in question:
+                    for i, right_answer in enumerate(question['tests_right']):
+                        try:
+                            question.set_answer(right_answer)
+                            question.correct()
+                        except Exception as exc:
+                            msg = f'Failed to correct "{qref}"'
+                            raise TestFactoryException(msg) from exc
+
+                        if question['grade'] == 1.0:
+                            logger.info('         test %i Ok', i)
+                        else:
+                            logger.error('         TEST %i IS WRONG!!!', i)
+                elif 'tests_wrong' in question:
+                    for i, wrong_answer in enumerate(question['tests_wrong']):
+                        try:
+                            question.set_answer(wrong_answer)
+                            question.correct()
+                        except Exception as exc:
+                            msg = f'Failed to correct "{qref}"'
+                            raise TestFactoryException(msg) from exc
+
+                        if question['grade'] < 1.0:
+                            logger.info('         test %i Ok', i)
+                        else:
+                            logger.error('         TEST %i IS WRONG!!!', i)
+                else:
+                    try:
+                        question.set_answer('')
+                        question.correct()
+                    except Exception as exc:
+                        msg = f'Failed to correct "{qref}"'
+                        raise TestFactoryException(msg) from exc
+                    else:
+                        logger.info('         correct Ok but no tests to run')
+
+    # ------------------------------------------------------------------------
+    async def generate(self):
+        '''
+        Given a dictionary with a student dict {'name':'john', 'number': 123}
+        returns instance of Test() for that particular student
+        '''
+
+        # make list of questions
+        questions = []
+        qnum = 1  # track question number
+        nerr = 0  # count errors during questions generation
+
+        for qlist in self['questions']:
+            # choose list of question variants
+            choose = qlist.get('choose', 1)
+            qrefs = random.sample(qlist['ref'], k=choose)
+
+            for qref in qrefs:
+                # generate instance of question
+                try:
+                    question = await self['question_factory'][qref].gen_async()
+                except QuestionException:
+                    logger.error('Can\'t generate question "%s". Skipping.', qref)
+                    nerr += 1
+                    continue
+
+                # some defaults
+                if question['type'] in ('information', 'success', 'warning',
+                                        'alert'):
+                    question['points'] = qlist.get('points', 0.0)
+                else:
+                    question['points'] = qlist.get('points', 1.0)
+                    question['number'] = qnum  # counter for non informative panels
+                    qnum += 1
+
+                questions.append(question)
+
+        # setup scale
+        total_points = sum(q['points'] for q in questions)
+
+        if total_points > 0:
+            # normalize question points to scale
+            if self['scale'] is not None:
+                scale_min, scale_max = self['scale']
+                for question in questions:
+                    question['points'] *= (scale_max - scale_min) / total_points
+            else:
+                self['scale'] = [0, total_points]
+        else:
+            logger.warning('Total points is **ZERO**.')
+            if self['scale'] is None:
+                self['scale'] = [0, 20]  # default
+
+        if nerr > 0:
+            logger.error('%s errors found!', nerr)
+
+        # copy these from the test configuratoin to each test instance
+        inherit = {'ref', 'title', 'database', 'answers_dir',
+                   'questions_dir', 'files',
+                   'duration', 'autosubmit', 'autocorrect',
+                   'scale', 'show_points',
+                   'show_ref', 'debug', }
+        # NOT INCLUDED:  testfile, allow_all, review
+
+        return Test({'questions': questions, **{k:self[k] for k in inherit}})
+
+    # ------------------------------------------------------------------------
+    def __repr__(self):
+        testsettings = '\n'.join(f'  {k:14s}: {v}' for k, v in self.items())
+        return '{\n' + testsettings + '\n}'
+'''
+Perguntations setup
+'''
+
 from setuptools import setup, find_packages
 from perguntations import (__author__, __license__,
	@@ -26,6 +26,7 @@		@@ -26,6 +26,7 @@
26	show_points: true # mostra cotação das perguntas (default: true)	26	show_points: true # mostra cotação das perguntas (default: true)
27	scale: [0, 20] # limites inferior e superior da escala (default: [0,20])	27	scale: [0, 20] # limites inferior e superior da escala (default: [0,20])
28	scale_points: true # normaliza cotações para a escala definida	28	scale_points: true # normaliza cotações para a escala definida
		29	+ jobe_server: moodle-jobe.uevora.pt # server used to compile & execute code
29	debug: false # mostra informação de debug no browser	30	debug: false # mostra informação de debug no browser
30		31
31	# --------------------------------------------------------------------------	32	# --------------------------------------------------------------------------
	@@ -575,7 +576,7 @@		@@ -575,7 +576,7 @@
575	# ----------------------------------------------------------------------------	576	# ----------------------------------------------------------------------------
576	- type: information	577	- type: information
577	text: \|	578	text: \|
578	- This question is not included in the test and will not shown up.	579	+ This question is not included in the test and will not show up.
579	It also lacks a "ref" and is automatically named	580	It also lacks a "ref" and is automatically named
580	`questions/questions-tutorial.yaml:0013`.	581	`questions/questions-tutorial.yaml:0013`.
581	A warning is shown on the console about this.	582	A warning is shown on the console about this.
	@@ -609,3 +610,52 @@		@@ -609,3 +610,52 @@
609	generate-question \| yamllint -	610	generate-question \| yamllint -
610	correct-answer \| yamllint -	611	correct-answer \| yamllint -
611	```	612	```
		613	+
		614	+# ----------------------------------------------------------------------------
		615	+# - type: code
		616	+# ref: tut-code
		617	+# title: Submissão de código (JOBE)
		618	+# text: \|
		619	+# É possível enviar código para ser compilado e executado por um servidor
		620	+# JOBE instalado separadamente, ver [JOBE](https://github.com/trampgeek/jobe).
		621	+
		622	+# ```yaml
		623	+# - type: code
		624	+# ref: tut-code
		625	+# title: Submissão de código (JOBE)
		626	+# text: \|
		627	+# Escreva um programa em C que recebe uma string no standard input e
		628	+# mostra a mensagem `hello ` seguida da string.
		629	+# Por exemplo, se o input for `Maria`, o output deverá ser `hello Maria`.
		630	+# language: c
		631	+# correct:
		632	+# - stdin: 'Maria'
		633	+# stdout: 'hello Maria'
		634	+# - stdin: 'xyz'
		635	+# stdout: 'hello xyz'
		636	+# ```
		637	+
		638	+# Existem várias linguagens suportadas pelo servidor JOBE (C, C++, Java,
		639	+# Python2, Python3, Octave, Pascal, PHP).
		640	+# O campo `correct` deverá ser uma lista de casos a testar.
		641	+# Se um caso incluir `stdin`, este será enviado para o programa e o `stdout`
		642	+# obtido será comparado com o declarado. A pergunta é considerada correcta se
		643	+# todos os outputs coincidirem.
		644	+
		645	+# Por defeito é o usado o servidor JOBE declarado no teste. Para usar outro
		646	+# diferente nesta pergunta usa-se a opção `server: 127.0.0.1` com o endereço
		647	+# apropriado.
		648	+# answer: \|
		649	+# #include <stdio.h>
		650	+# int main() {
		651	+# char name[20];
		652	+# scanf("%s", name);
		653	+# printf("hello %s", name);
		654	+# }
		655	+# # server: 192.168.1.85
		656	+# language: c
		657	+# correct:
		658	+# - stdin: 'Maria'
		659	+# stdout: 'hello Maria'
		660	+# - stdin: 'xyz'
		661	+# stdout: 'hello xyz'
	@@ -32,7 +32,7 @@ proof of submission and for review.		@@ -32,7 +32,7 @@ proof of submission and for review.
32	'''	32	'''
33		33
34	APP_NAME = 'perguntations'	34	APP_NAME = 'perguntations'
35	-APP_VERSION = '2020.11.dev2'	35	+APP_VERSION = '2020.12.dev1'
36	APP_DESCRIPTION = __doc__	36	APP_DESCRIPTION = __doc__
37		37
38	__author__ = 'Miguel Barão'	38	__author__ = 'Miguel Barão'
	@@ -19,7 +19,7 @@ import sqlalchemy as sa		@@ -19,7 +19,7 @@ import sqlalchemy as sa
19	from perguntations.models import Base, Student	19	from perguntations.models import Base, Student
20		20
21		21
22	-# ===========================================================================	22	+# ============================================================================
23	def parse_commandline_arguments():	23	def parse_commandline_arguments():
24	'''Parse command line options'''	24	'''Parse command line options'''
25	parser = argparse.ArgumentParser(	25	parser = argparse.ArgumentParser(
	@@ -68,7 +68,7 @@ def parse_commandline_arguments():		@@ -68,7 +68,7 @@ def parse_commandline_arguments():
68	return parser.parse_args()	68	return parser.parse_args()
69		69
70		70
71	-# ===========================================================================	71	+# ============================================================================
72	def get_students_from_csv(filename):	72	def get_students_from_csv(filename):
73	'''	73	'''
74	SIIUE names have alien strings like "(TE)" and are sometimes capitalized	74	SIIUE names have alien strings like "(TE)" and are sometimes capitalized
	@@ -97,7 +97,7 @@ def get_students_from_csv(filename):		@@ -97,7 +97,7 @@ def get_students_from_csv(filename):
97	return students	97	return students
98		98
99		99
100	-# ===========================================================================	100	+# ============================================================================
101	def hashpw(student, password=None):	101	def hashpw(student, password=None):
102	'''replace password by hash for a single student'''	102	'''replace password by hash for a single student'''
103	print('.', end='', flush=True)	103	print('.', end='', flush=True)
	@@ -108,7 +108,7 @@ def hashpw(student, password=None):		@@ -108,7 +108,7 @@ def hashpw(student, password=None):
108	bcrypt.gensalt())	108	bcrypt.gensalt())
109		109
110		110
111	-# ===========================================================================	111	+# ============================================================================
112	def insert_students_into_db(session, students):	112	def insert_students_into_db(session, students):
113	'''insert list of students into the database'''	113	'''insert list of students into the database'''
114	try:	114	try:
	@@ -13,6 +13,12 @@ import re		@@ -13,6 +13,12 @@ import re
13	from typing import Any, Dict, NewType	13	from typing import Any, Dict, NewType
14	import uuid	14	import uuid
15		15
		16	+
		17	+# from urllib.error import HTTPError
		18	+# import json
		19	+# import http.client
		20	+
		21	+
16	# this project	22	# this project
17	from perguntations.tools import run_script, run_script_async	23	from perguntations.tools import run_script, run_script_async
18		24
	@@ -23,6 +29,8 @@ logger = logging.getLogger(__name__)		@@ -23,6 +29,8 @@ logger = logging.getLogger(__name__)
23	QDict = NewType('QDict', Dict[str, Any])	29	QDict = NewType('QDict', Dict[str, Any])
24		30
25		31
		32	+
		33	+
26	class QuestionException(Exception):	34	class QuestionException(Exception):
27	'''Exceptions raised in this module'''	35	'''Exceptions raised in this module'''
28		36
	@@ -37,8 +45,13 @@ class Question(dict):		@@ -37,8 +45,13 @@ class Question(dict):
37	for each student.	45	for each student.
38	Instances can shuffle options or automatically generate questions.	46	Instances can shuffle options or automatically generate questions.
39	'''	47	'''
40	- def __init__(self, q: QDict) -> None:
41	- super().__init__(q)	48	+ # def __init__(self, q: QDict) -> None:
		49	+ # super().__init__(q)
		50	+
		51	+ def gen(self) -> None:
		52	+ '''
		53	+ Sets defaults that are valid for any question type
		54	+ '''
42		55
43	# add required keys if missing	56	# add required keys if missing
44	self.set_defaults(QDict({	57	self.set_defaults(QDict({
	@@ -83,9 +96,15 @@ class QuestionRadio(Question):		@@ -83,9 +96,15 @@ class QuestionRadio(Question):
83	'''	96	'''
84		97
85	# ------------------------------------------------------------------------	98	# ------------------------------------------------------------------------
86	- def __init__(self, q: QDict) -> None:
87	- super().__init__(q)	99	+ # def __init__(self, q: QDict) -> None:
		100	+ # super().__init__(q)
88		101
		102	+ def gen(self) -> None:
		103	+ '''
		104	+ Sets defaults, performs checks and generates the actual question
		105	+ by modifying the options and correct values
		106	+ '''
		107	+ super().gen()
89	try:	108	try:
90	nopts = len(self['options'])	109	nopts = len(self['options'])
91	except KeyError as exc:	110	except KeyError as exc:
	@@ -212,8 +231,11 @@ class QuestionCheckbox(Question):		@@ -212,8 +231,11 @@ class QuestionCheckbox(Question):
212	'''	231	'''
213		232
214	# ------------------------------------------------------------------------	233	# ------------------------------------------------------------------------
215	- def __init__(self, q: QDict) -> None:
216	- super().__init__(q)	234	+ # def __init__(self, q: QDict) -> None:
		235	+ # super().__init__(q)
		236	+
		237	+ def gen(self) -> None:
		238	+ super().gen()
217		239
218	try:	240	try:
219	nopts = len(self['options'])	241	nopts = len(self['options'])
	@@ -334,9 +356,11 @@ class QuestionText(Question):		@@ -334,9 +356,11 @@ class QuestionText(Question):
334	'''	356	'''
335		357
336	# ------------------------------------------------------------------------	358	# ------------------------------------------------------------------------
337	- def __init__(self, q: QDict) -> None:
338	- super().__init__(q)	359	+ # def __init__(self, q: QDict) -> None:
		360	+ # super().__init__(q)
339		361
		362	+ def gen(self) -> None:
		363	+ super().gen()
340	self.set_defaults(QDict({	364	self.set_defaults(QDict({
341	'text': '',	365	'text': '',
342	'correct': [], # no correct answers, always wrong	366	'correct': [], # no correct answers, always wrong
	@@ -403,8 +427,11 @@ class QuestionTextRegex(Question):		@@ -403,8 +427,11 @@ class QuestionTextRegex(Question):
403	'''	427	'''
404		428
405	# ------------------------------------------------------------------------	429	# ------------------------------------------------------------------------
406	- def __init__(self, q: QDict) -> None:
407	- super().__init__(q)	430	+ # def __init__(self, q: QDict) -> None:
		431	+ # super().__init__(q)
		432	+
		433	+ def gen(self) -> None:
		434	+ super().gen()
408		435
409	self.set_defaults(QDict({	436	self.set_defaults(QDict({
410	'text': '',	437	'text': '',
	@@ -416,26 +443,34 @@ class QuestionTextRegex(Question):		@@ -416,26 +443,34 @@ class QuestionTextRegex(Question):
416	self['correct'] = [self['correct']]	443	self['correct'] = [self['correct']]
417		444
418	# converts patterns to compiled versions	445	# converts patterns to compiled versions
419	- try:
420	- self['correct'] = [re.compile(a) for a in self['correct']]
421	- except Exception as exc:
422	- msg = f'Failed to compile regex in "{self["ref"]}"'
423	- logger.error(msg)
424	- raise QuestionException(msg) from exc	446	+ # try:
		447	+ # self['correct'] = [re.compile(a) for a in self['correct']]
		448	+ # except Exception as exc:
		449	+ # msg = f'Failed to compile regex in "{self["ref"]}"'
		450	+ # logger.error(msg)
		451	+ # raise QuestionException(msg) from exc
425		452
426	# ------------------------------------------------------------------------	453	# ------------------------------------------------------------------------
427	def correct(self) -> None:	454	def correct(self) -> None:
428	super().correct()	455	super().correct()
429	if self['answer'] is not None:	456	if self['answer'] is not None:
430	- self['grade'] = 0.0
431	for regex in self['correct']:	457	for regex in self['correct']:
432	try:	458	try:
433	- if regex.match(self['answer']):	459	+ if re.fullmatch(regex, self['answer']):
434	self['grade'] = 1.0	460	self['grade'] = 1.0
435	return	461	return
436	except TypeError:	462	except TypeError:
437	- logger.error('While matching regex %s with answer "%s".',
438	- regex.pattern, self["answer"])	463	+ logger.error('While matching regex "%s" with answer "%s".',
		464	+ regex, self['answer'])
		465	+ self['grade'] = 0.0
		466	+
		467	+ # try:
		468	+ # if regex.match(self['answer']):
		469	+ # self['grade'] = 1.0
		470	+ # return
		471	+ # except TypeError:
		472	+ # logger.error('While matching regex %s with answer "%s".',
		473	+ # regex.pattern, self["answer"])
439		474
440		475
441	# ============================================================================	476	# ============================================================================
	@@ -449,8 +484,11 @@ class QuestionNumericInterval(Question):		@@ -449,8 +484,11 @@ class QuestionNumericInterval(Question):
449	'''	484	'''
450		485
451	# ------------------------------------------------------------------------	486	# ------------------------------------------------------------------------
452	- def __init__(self, q: QDict) -> None:
453	- super().__init__(q)	487	+ # def __init__(self, q: QDict) -> None:
		488	+ # super().__init__(q)
		489	+
		490	+ def gen(self) -> None:
		491	+ super().gen()
454		492
455	self.set_defaults(QDict({	493	self.set_defaults(QDict({
456	'text': '',	494	'text': '',
	@@ -510,8 +548,11 @@ class QuestionTextArea(Question):		@@ -510,8 +548,11 @@ class QuestionTextArea(Question):
510	'''	548	'''
511		549
512	# ------------------------------------------------------------------------	550	# ------------------------------------------------------------------------
513	- def __init__(self, q: QDict) -> None:
514	- super().__init__(q)	551	+ # def __init__(self, q: QDict) -> None:
		552	+ # super().__init__(q)
		553	+
		554	+ def gen(self) -> None:
		555	+ super().gen()
515		556
516	self.set_defaults(QDict({	557	self.set_defaults(QDict({
517	'text': '',	558	'text': '',
	@@ -584,6 +625,129 @@ class QuestionTextArea(Question):		@@ -584,6 +625,129 @@ class QuestionTextArea(Question):
584		625
585		626
586	# ============================================================================	627	# ============================================================================
		628	+# class QuestionCode(Question):
		629	+# '''
		630	+# Submits answer to a JOBE server to compile and run against the test cases.
		631	+# '''
		632	+
		633	+# _outcomes = {
		634	+# 0: 'JOBE outcome: Successful run',
		635	+# 11: 'JOBE outcome: Compile error',
		636	+# 12: 'JOBE outcome: Runtime error',
		637	+# 13: 'JOBE outcome: Time limit exceeded',
		638	+# 15: 'JOBE outcome: Successful run',
		639	+# 17: 'JOBE outcome: Memory limit exceeded',
		640	+# 19: 'JOBE outcome: Illegal system call',
		641	+# 20: 'JOBE outcome: Internal error, please report',
		642	+# 21: 'JOBE outcome: Server overload',
		643	+# }
		644	+
		645	+# # ------------------------------------------------------------------------
		646	+# def __init__(self, q: QDict) -> None:
		647	+# super().__init__(q)
		648	+
		649	+# self.set_defaults(QDict({
		650	+# 'text': '',
		651	+# 'timeout': 5, # seconds
		652	+# 'server': '127.0.0.1', # JOBE server
		653	+# 'language': 'c',
		654	+# 'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
		655	+# }))
		656	+
		657	+ # ------------------------------------------------------------------------
		658	+ # def correct(self) -> None:
		659	+ # super().correct()
		660	+
		661	+ # if self['answer'] is None:
		662	+ # return
		663	+
		664	+ # # submit answer to JOBE server
		665	+ # resource = '/jobe/index.php/restapi/runs/'
		666	+ # headers = {"Content-type": "application/json; charset=utf-8",
		667	+ # "Accept": "application/json"}
		668	+
		669	+ # for expected in self['correct']:
		670	+ # data_json = json.dumps({
		671	+ # 'run_spec' : {
		672	+ # 'language_id': self['language'],
		673	+ # 'sourcecode': self['answer'],
		674	+ # 'input': expected.get('stdin', ''),
		675	+ # },
		676	+ # })
		677	+
		678	+ # try:
		679	+ # connect = http.client.HTTPConnection(self['server'])
		680	+ # connect.request(
		681	+ # method='POST',
		682	+ # url=resource,
		683	+ # body=data_json,
		684	+ # headers=headers
		685	+ # )
		686	+ # response = connect.getresponse()
		687	+ # logger.debug('JOBE response status %d', response.status)
		688	+ # if response.status != 204:
		689	+ # content = response.read().decode('utf8')
		690	+ # if content:
		691	+ # result = json.loads(content)
		692	+ # connect.close()
		693	+
		694	+ # except (HTTPError, ValueError):
		695	+ # logger.error('HTTPError while connecting to JOBE server')
		696	+
		697	+ # try:
		698	+ # outcome = result['outcome']
		699	+ # except (NameError, TypeError, KeyError):
		700	+ # logger.error('Bad result returned from JOBE server: %s', result)
		701	+ # return
		702	+ # logger.debug(self._outcomes[outcome])
		703	+
		704	+
		705	+
		706	+ # if result['cmpinfo']: # compiler errors and warnings
		707	+ # self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
		708	+ # self['grade'] = 0.0
		709	+ # return
		710	+
		711	+ # if result['stdout'] != expected.get('stdout', ''):
		712	+ # self['comments'] = 'O output gerado é diferente do esperado.' # FIXME mostrar porque?
		713	+ # self['grade'] = 0.0
		714	+ # return
		715	+
		716	+ # self['comments'] = 'Ok!'
		717	+ # self['grade'] = 1.0
		718	+
		719	+
		720	+ # # ------------------------------------------------------------------------
		721	+ # async def correct_async(self) -> None:
		722	+ # self.correct() # FIXME there is no async correction!!!
		723	+
		724	+
		725	+ # out = run_script(
		726	+ # script=self['correct'],
		727	+ # args=self['args'],
		728	+ # stdin=self['answer'],
		729	+ # timeout=self['timeout']
		730	+ # )
		731	+
		732	+ # if out is None:
		733	+ # logger.warning('No grade after running "%s".', self["correct"])
		734	+ # self['comments'] = 'O programa de correcção abortou...'
		735	+ # self['grade'] = 0.0
		736	+ # elif isinstance(out, dict):
		737	+ # self['comments'] = out.get('comments', '')
		738	+ # try:
		739	+ # self['grade'] = float(out['grade'])
		740	+ # except ValueError:
		741	+ # logger.error('Output error in "%s".', self["correct"])
		742	+ # except KeyError:
		743	+ # logger.error('No grade in "%s".', self["correct"])
		744	+ # else:
		745	+ # try:
		746	+ # self['grade'] = float(out)
		747	+ # except (TypeError, ValueError):
		748	+ # logger.error('Invalid grade in "%s".', self["correct"])
		749	+
		750	+# ============================================================================
587	class QuestionInformation(Question):	751	class QuestionInformation(Question):
588	'''	752	'''
589	Not really a question, just an information panel.	753	Not really a question, just an information panel.
	@@ -591,8 +755,11 @@ class QuestionInformation(Question):		@@ -591,8 +755,11 @@ class QuestionInformation(Question):
591	'''	755	'''
592		756
593	# ------------------------------------------------------------------------	757	# ------------------------------------------------------------------------
594	- def __init__(self, q: QDict) -> None:
595	- super().__init__(q)	758	+ # def __init__(self, q: QDict) -> None:
		759	+ # super().__init__(q)
		760	+
		761	+ def gen(self) -> None:
		762	+ super().gen()
596	self.set_defaults(QDict({	763	self.set_defaults(QDict({
597	'text': '',	764	'text': '',
598	}))	765	}))
	@@ -603,6 +770,46 @@ class QuestionInformation(Question):		@@ -603,6 +770,46 @@ class QuestionInformation(Question):
603	self['grade'] = 1.0 # always "correct" but points should be zero!	770	self['grade'] = 1.0 # always "correct" but points should be zero!
604		771
605		772
		773	+
		774	+# ============================================================================
		775	+def question_from(qdict: QDict) -> Question:
		776	+ '''
		777	+ Converts a question specified in a dict into an instance of Question()
		778	+ '''
		779	+ types = {
		780	+ 'radio': QuestionRadio,
		781	+ 'checkbox': QuestionCheckbox,
		782	+ 'text': QuestionText,
		783	+ 'text-regex': QuestionTextRegex,
		784	+ 'numeric-interval': QuestionNumericInterval,
		785	+ 'textarea': QuestionTextArea,
		786	+ # 'code': QuestionCode,
		787	+ # -- informative panels --
		788	+ 'information': QuestionInformation,
		789	+ 'success': QuestionInformation,
		790	+ 'warning': QuestionInformation,
		791	+ 'alert': QuestionInformation,
		792	+ }
		793	+
		794	+ # Get class for this question type
		795	+ try:
		796	+ qclass = types[qdict['type']]
		797	+ except KeyError:
		798	+ logger.error('Invalid type "%s" in "%s"',
		799	+ qdict['type'], qdict['ref'])
		800	+ raise
		801	+
		802	+ # Create an instance of Question() of appropriate type
		803	+ try:
		804	+ qinstance = qclass(QDict(qdict))
		805	+ except QuestionException:
		806	+ logger.error('Error generating "%s" in %s/%s',
		807	+ qdict['ref'], qdict['path'], qdict['filename'])
		808	+ raise
		809	+
		810	+ return qinstance
		811	+
		812	+
606	# ============================================================================	813	# ============================================================================
607	class QFactory():	814	class QFactory():
608	'''	815	'''
	@@ -636,24 +843,8 @@ class QFactory():		@@ -636,24 +843,8 @@ class QFactory():
636	grade = question['grade'] # get grade	843	grade = question['grade'] # get grade
637	'''	844	'''
638		845
639	- # Depending on the type of question, a different question class will be
640	- # instantiated. All these classes derive from the base class `Question`.
641	- _types = {
642	- 'radio': QuestionRadio,
643	- 'checkbox': QuestionCheckbox,
644	- 'text': QuestionText,
645	- 'text-regex': QuestionTextRegex,
646	- 'numeric-interval': QuestionNumericInterval,
647	- 'textarea': QuestionTextArea,
648	- # -- informative panels --
649	- 'information': QuestionInformation,
650	- 'success': QuestionInformation,
651	- 'warning': QuestionInformation,
652	- 'alert': QuestionInformation,
653	- }
654	-
655	def __init__(self, qdict: QDict = QDict({})) -> None:	846	def __init__(self, qdict: QDict = QDict({})) -> None:
656	- self.question = qdict	847	+ self.qdict = qdict
657		848
658	# ------------------------------------------------------------------------	849	# ------------------------------------------------------------------------
659	async def gen_async(self) -> Question:	850	async def gen_async(self) -> Question:
	@@ -662,44 +853,28 @@ class QFactory():		@@ -662,44 +853,28 @@ class QFactory():
662	which is a descendent of base class Question.	853	which is a descendent of base class Question.
663	'''	854	'''
664		855
665	- logger.debug('generating %s...', self.question["ref"])	856	+ logger.debug('generating %s...', self.qdict["ref"])
666	# Shallow copy so that script generated questions will not replace	857	# Shallow copy so that script generated questions will not replace
667	# the original generators	858	# the original generators
668	- question = self.question.copy()
669	- question['qid'] = str(uuid.uuid4()) # unique for each question	859	+ qdict = self.qdict.copy()
		860	+ qdict['qid'] = str(uuid.uuid4()) # unique for each question
670		861
671	# If question is of generator type, an external program will be run	862	# If question is of generator type, an external program will be run
672	# which will print a valid question in yaml format to stdout. This	863	# which will print a valid question in yaml format to stdout. This
673	# output is then yaml parsed into a dictionary `q`.	864	# output is then yaml parsed into a dictionary `q`.
674	- if question['type'] == 'generator':
675	- logger.debug(' \\_ Running "%s".', question['script'])
676	- question.setdefault('args', [])
677	- question.setdefault('stdin', '')
678	- script = path.join(question['path'], question['script'])	865	+ if qdict['type'] == 'generator':
		866	+ logger.debug(' \\_ Running "%s".', qdict['script'])
		867	+ qdict.setdefault('args', [])
		868	+ qdict.setdefault('stdin', '')
		869	+ script = path.join(qdict['path'], qdict['script'])
679	out = await run_script_async(script=script,	870	out = await run_script_async(script=script,
680	- args=question['args'],
681	- stdin=question['stdin'])
682	- question.update(out)	871	+ args=qdict['args'],
		872	+ stdin=qdict['stdin'])
		873	+ qdict.update(out)
683		874
684	- # Get class for this question type
685	- try:
686	- qclass = self._types[question['type']]
687	- except KeyError:
688	- logger.error('Invalid type "%s" in "%s"',
689	- question['type'], question['ref'])
690	- raise
691	-
692	- # Finally create an instance of Question()
693	- try:
694	- qinstance = qclass(QDict(question))
695	- except QuestionException:
696	- logger.error('Error generating question "%s". See "%s/%s"',
697	- question['ref'],
698	- question['path'],
699	- question['filename'])
700	- raise
701	-
702	- return qinstance	875	+ question = question_from(qdict) # returns a Question instance
		876	+ question.gen()
		877	+ return question
703		878
704	# ------------------------------------------------------------------------	879	# ------------------------------------------------------------------------
705	def generate(self) -> Question:	880	def generate(self) -> Question:
	@@ -40,68 +40,22 @@		@@ -40,68 +40,22 @@
40	<!-- ================================================================== -->	40	<!-- ================================================================== -->
41	<div class="container">	41	<div class="container">
42	<div class="jumbotron">	42	<div class="jumbotron">
43	- {% if t['state'] == 'FINISHED' %}
44	- <h1>Resultado:
45	- <strong>{{ f'{round(t["grade"], 3)}' }}</strong>
46	- valores na escala de {{t['scale'][0]}} a {{t['scale'][1]}}.
47	- </h1>
48	- <p>O seu teste foi correctamente entregue e a nota registada.</p>
49	- <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para sair do teste</a></p>	43	+ {% if t['state'] == 'CORRECTED' %}
50	{% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}	44	{% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}
51	<i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>	45	<i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>
52	{% end %}	46	{% end %}
		47	+ <h3>Resultado:
		48	+ <strong>{{ f'{round(t["grade"], 3)}' }}</strong>
		49	+ valores na escala [{{t['scale'][0]}},{{t['scale'][1]}}].
		50	+ </h3>
		51	+ {% elif t['state'] == 'SUBMITTED' %}
		52	+ <h3>A prova foi submetida com sucesso. Vai ser corrigida mais tarde.</h3>
53	{% elif t['state'] == 'QUIT' %}	53	{% elif t['state'] == 'QUIT' %}
54	- <p>Foi registada a sua desistência da prova.</p>	54	+ <h3>Foi registada a sua desistência da prova.</h3>
55	{% end %}	55	{% end %}
56		56
		57	+ <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para terminar</a></p>
57	</div> <!-- jumbotron -->	58	</div> <!-- jumbotron -->
58	-
59	- <div class="card">
60	- <h5 class="card-header">
61	- Histórico de resultados
62	- </h5>
63	- <table class="table table-condensed noleftmargin">
64	- <thead>
65	- <tr>
66	- <th>Prova</th>
67	- <th>Data</th>
68	- <th>Hora</th>
69	- <th>Nota</th>
70	- </tr>
71	- </thead>
72	- <tbody>
73	- {% for g in allgrades %}
74	- <tr>
75	- <td>{{g[0]}}</td> <!-- teste -->
76	- <td>{{g[2][:10]}}</td> <!-- data -->
77	- <td>{{g[2][11:19]}}</td> <!-- hora -->
78	- <td> <!-- progress column -->
79	- <div class="progress" style="height: 20px;">
80	- <div class="progress-bar
81	- {% if g[1] - t['scale'][0] < 0.5*(t['scale'][1] - t['scale'][0]) %}
82	- bg-danger
83	- {% elif g[1] - t['scale'][0] < 0.75*(t['scale'][1] - t['scale'][0]) %}
84	- bg-warning
85	- {% else %}
86	- bg-success
87	- {% end %}
88	- "
89	- role="progressbar"
90	- aria-valuenow="{{ 100*(g[1] - t['scale'][0])/(t['scale'][1] - t['scale'][0]) }}"
91	- aria-valuemin="0"
92	- aria-valuemax="100"
93	- style="min-width: 2em; width: {{ 100*(g[1]-t['scale'][0])/(t['scale'][1]-t['scale'][0]) }}%;">
94	-
95	- {{ str(round(g[1], 1)) }}
96	-
97	- </div> <!-- progress-bar -->
98	- </div> <!-- progress -->
99	- </td> <!-- progress column -->
100	- </tr>
101	- {% end %}
102	- </tbody>
103	- </table>
104	- </div> <!-- panel -->
105	</div> <!-- container -->	59	</div> <!-- container -->
106	</body>	60	</body>
107	</html>	61	</html>
	@@ -12,7 +12,6 @@		@@ -12,7 +12,6 @@
12		12
13	<!-- Scripts -->	13	<!-- Scripts -->
14	<script src="/static/jquery/jquery.min.js"></script>	14	<script src="/static/jquery/jquery.min.js"></script>
15	- <!-- <script defer src="/static/popper.js/popper.min.js"></script> -->
16	<script defer src="/static/fontawesome-free/js/all.min.js"></script>	15	<script defer src="/static/fontawesome-free/js/all.min.js"></script>
17	<script defer src="/static/bootstrap/js/bootstrap.bundle.min.js"></script>	16	<script defer src="/static/bootstrap/js/bootstrap.bundle.min.js"></script>
18		17
	@@ -97,7 +97,7 @@		@@ -97,7 +97,7 @@
97	<div class="row">	97	<div class="row">
98	<label for="nota" class="col-sm-2">Nota:</label>	98	<label for="nota" class="col-sm-2">Nota:</label>
99	<div class="col-sm-10" id="nota">	99	<div class="col-sm-10" id="nota">
100	- <span class="badge badge-primary">{{ round(t['grade'], 1) }}</span> valores	100	+ <span class="badge badge-primary">{{ round(t['grade'], 2) }}</span> valores
101	{% if t['state'] == 'QUIT' %}	101	{% if t['state'] == 'QUIT' %}
102	(DESISTÊNCIA)	102	(DESISTÊNCIA)
103	{% end %}	103	{% end %}
1	'''	1	'''
2	-TestFactory - generates tests for students
3	Test - instances of this class are individual tests	2	Test - instances of this class are individual tests
4	'''	3	'''
5		4
6	-
7	# python standard library	5	# python standard library
8	-from os import path
9	-import random
10	from datetime import datetime	6	from datetime import datetime
		7	+import json
11	import logging	8	import logging
12	-import re
13	-
14	-# this project
15	-from perguntations.questions import QFactory, QuestionException
16	-from perguntations.tools import load_yaml	9	+from math import nan
		10	+from os import path
17		11
18	# Logger configuration	12	# Logger configuration
19	logger = logging.getLogger(__name__)	13	logger = logging.getLogger(__name__)
20		14
21		15
22	# ============================================================================	16	# ============================================================================
23	-class TestFactoryException(Exception):
24	- '''exception raised in this module'''
25	-
26	-
27	-# ============================================================================
28	-class TestFactory(dict):
29	- '''
30	- Each instance of TestFactory() is a test generator.
31	- For example, if we want to serve two different tests, then we need two
32	- instances of TestFactory(), one for each test.
33	- '''
34	-
35	- # ------------------------------------------------------------------------
36	- def __init__(self, conf):
37	- '''
38	- Loads configuration from yaml file, then overrides some configurations
39	- using the conf argument.
40	- Base questions are added to a pool of questions factories.
41	- '''
42	-
43	- # --- set test defaults and then use given configuration
44	- super().__init__({ # defaults
45	- 'title': '',
46	- 'show_points': True,
47	- 'scale': None, # or [0, 20]
48	- 'duration': 0, # 0=infinite
49	- 'autosubmit': False,
50	- 'debug': False,
51	- 'show_ref': False,
52	- })
53	- self.update(conf)
54	-
55	- # --- perform sanity checks and normalize the test questions
56	- self.sanity_checks()
57	- logger.info('Sanity checks PASSED.')
58	-
59	- # --- find refs of all questions used in the test
60	- qrefs = {r for qq in self['questions'] for r in qq['ref']}
61	- logger.info('Declared %d questions (each test uses %d).',
62	- len(qrefs), len(self["questions"]))
63	-
64	- # --- for review, we are done. no factories needed
65	- if self['review']:
66	- logger.info('Review mode. No questions loaded. No factories.')
67	- return
68	-
69	- # --- load and build question factories
70	- self.question_factory = {}
71	-
72	- counter = 1
73	- for file in self["files"]:
74	- fullpath = path.normpath(path.join(self["questions_dir"], file))
75	- (dirname, filename) = path.split(fullpath)
76	-
77	- logger.info('Loading "%s"...', fullpath)
78	- questions = load_yaml(fullpath) # , default=[])
79	-
80	- for i, question in enumerate(questions):
81	- # make sure every question in the file is a dictionary
82	- if not isinstance(question, dict):
83	- msg = f'Question {i} in {file} is not a dictionary'
84	- raise TestFactoryException(msg)
85	-
86	- # check if ref is missing, then set to '/path/file.yaml:3'
87	- if 'ref' not in question:
88	- question['ref'] = f'{file}:{i:04}'
89	- logger.warning('Missing ref set to "%s"', question["ref"])
90	-
91	- # check for duplicate refs
92	- if question['ref'] in self.question_factory:
93	- other = self.question_factory[question['ref']]
94	- otherfile = path.join(other.question['path'],
95	- other.question['filename'])
96	- msg = (f'Duplicate reference "{question["ref"]}" in files '
97	- f'"{otherfile}" and "{fullpath}".')
98	- raise TestFactoryException(msg)
99	-
100	- # make factory only for the questions used in the test
101	- if question['ref'] in qrefs:
102	- question.setdefault('type', 'information')
103	- question.update({
104	- 'filename': filename,
105	- 'path': dirname,
106	- 'index': i # position in the file, 0 based
107	- })
108	-
109	- self.question_factory[question['ref']] = QFactory(question)
110	-
111	- # check if all the questions can be correctly generated
112	- try:
113	- self.question_factory[question['ref']].generate()
114	- except Exception as exc:
115	- msg = f'Failed to generate "{question["ref"]}"'
116	- raise TestFactoryException(msg) from exc
117	- else:
118	- logger.info('%4d. "%s" Ok.', counter, question["ref"])
119	- counter += 1
120	-
121	- qmissing = qrefs.difference(set(self.question_factory.keys()))
122	- if qmissing:
123	- raise TestFactoryException(f'Could not find questions {qmissing}.')
124	-
125	- # ------------------------------------------------------------------------
126	- def check_test_ref(self):
127	- '''Test must have a `ref`'''
128	- if 'ref' not in self:
129	- raise TestFactoryException('Missing "ref" in configuration!')
130	- if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
131	- raise TestFactoryException('Test "ref" can only contain the '
132	- 'characters a-zA-Z0-9_-')
133	-
134	- def check_missing_database(self):
135	- '''Test must have a database'''
136	- if 'database' not in self:
137	- raise TestFactoryException('Missing "database" in configuration')
138	- if not path.isfile(path.expanduser(self['database'])):
139	- msg = f'Database "{self["database"]}" not found!'
140	- raise TestFactoryException(msg)
141	-
142	- def check_missing_answers_directory(self):
143	- '''Test must have a answers directory'''
144	- if 'answers_dir' not in self:
145	- msg = 'Missing "answers_dir" in configuration'
146	- raise TestFactoryException(msg)
147	-
148	- def check_answers_directory_writable(self):
149	- '''Answers directory must be writable'''
150	- testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
151	- try:
152	- with open(testfile, 'w') as file:
153	- file.write('You can safely remove this file.')
154	- except OSError as exc:
155	- msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
156	- raise TestFactoryException(msg) from exc
157	-
158	- def check_questions_directory(self):
159	- '''Check if questions directory is missing or not accessible.'''
160	- if 'questions_dir' not in self:
161	- logger.warning('Missing "questions_dir". Using "%s"',
162	- path.abspath(path.curdir))
163	- self['questions_dir'] = path.curdir
164	- elif not path.isdir(path.expanduser(self['questions_dir'])):
165	- raise TestFactoryException(f'Can\'t find questions directory '
166	- f'"{self["questions_dir"]}"')
167	-
168	- def check_import_files(self):
169	- '''Check if there are files to import (with questions)'''
170	- if 'files' not in self:
171	- msg = ('Missing "files" in configuration with the list of '
172	- 'question files to import!')
173	- raise TestFactoryException(msg)
174	-
175	- if isinstance(self['files'], str):
176	- self['files'] = [self['files']]
177	-
178	- def check_question_list(self):
179	- '''normalize question list'''
180	- if 'questions' not in self:
181	- raise TestFactoryException('Missing "questions" in configuration')
182	-
183	- for i, question in enumerate(self['questions']):
184	- # normalize question to a dict and ref to a list of references
185	- if isinstance(question, str): # e.g., - some_ref
186	- question = {'ref': [question]} # becomes - ref: [some_ref]
187	- elif isinstance(question, dict) and isinstance(question['ref'], str):
188	- question['ref'] = [question['ref']]
189	- elif isinstance(question, list):
190	- question = {'ref': [str(a) for a in question]}
191	-
192	- self['questions'][i] = question
193	-
194	- def check_missing_title(self):
195	- '''Warns if title is missing'''
196	- if not self['title']:
197	- logger.warning('Title is undefined!')
198	-
199	- def check_grade_scaling(self):
200	- '''Just informs the scale limits'''
201	- if 'scale_points' in self:
202	- msg = ('* DEPRECATION WARNING: * scale_points, scale_min, '
203	- 'scale_max were replaced by "scale: [min, max]".')
204	- logger.warning(msg)
205	- self['scale'] = [self['scale_min'], self['scale_max']]
206	-
207	-
208	- # ------------------------------------------------------------------------
209	- def sanity_checks(self):
210	- '''
211	- Checks for valid keys and sets default values.
212	- Also checks if some files and directories exist
213	- '''
214	- self.check_test_ref()
215	- self.check_missing_database()
216	- self.check_missing_answers_directory()
217	- self.check_answers_directory_writable()
218	- self.check_questions_directory()
219	- self.check_import_files()
220	- self.check_question_list()
221	- self.check_missing_title()
222	- self.check_grade_scaling()
223	-
224	- # ------------------------------------------------------------------------
225	- async def generate(self):
226	- '''
227	- Given a dictionary with a student dict {'name':'john', 'number': 123}
228	- returns instance of Test() for that particular student
229	- '''
230	-
231	- # make list of questions
232	- questions = []
233	- qnum = 1 # track question number
234	- nerr = 0 # count errors during questions generation
235	-
236	- for qlist in self['questions']:
237	- # choose list of question variants
238	- choose = qlist.get('choose', 1)
239	- qrefs = random.sample(qlist['ref'], k=choose)
240	-
241	- for qref in qrefs:
242	- # generate instance of question
243	- try:
244	- question = await self.question_factory[qref].gen_async()
245	- except QuestionException:
246	- logger.error('Can\'t generate question "%s". Skipping.', qref)
247	- nerr += 1
248	- continue
249	-
250	- # some defaults
251	- if question['type'] in ('information', 'success', 'warning',
252	- 'alert'):
253	- question['points'] = qlist.get('points', 0.0)
254	- else:
255	- question['points'] = qlist.get('points', 1.0)
256	- question['number'] = qnum # counter for non informative panels
257	- qnum += 1
258	-
259	- questions.append(question)
260	-
261	- # setup scale
262	- total_points = sum(q['points'] for q in questions)
263	-
264	- if total_points > 0:
265	- # normalize question points to scale
266	- if self['scale'] is not None:
267	- scale_min, scale_max = self['scale']
268	- for question in questions:
269	- question['points'] *= (scale_max - scale_min) / total_points
270	- else:
271	- self['scale'] = [0, total_points]
272	- else:
273	- logger.warning('Total points is ZERO.')
274	- if self['scale'] is None:
275	- self['scale'] = [0, 20] # default
276	-
277	- if nerr > 0:
278	- logger.error('%s errors found!', nerr)
279	-
280	- # copy these from the test configuratoin to each test instance
281	- inherit = {'ref', 'title', 'database', 'answers_dir',
282	- 'questions_dir', 'files',
283	- 'duration', 'autosubmit',
284	- 'scale', 'show_points',
285	- 'show_ref', 'debug', }
286	- # NOT INCLUDED: testfile, allow_all, review
287	-
288	- return Test({'questions': questions, **{k:self[k] for k in inherit}})
289	-
290	- # ------------------------------------------------------------------------
291	- def __repr__(self):
292	- testsettings = '\n'.join(f' {k:14s}: {v}' for k, v in self.items())
293	- return '{\n' + testsettings + '\n}'
294	-
295	-
296	-# ============================================================================
297	class Test(dict):	17	class Test(dict):
298	'''	18	'''
299	Each instance Test() is a concrete test of a single student.	19	Each instance Test() is a concrete test of a single student.
300	'''	20	'''
301		21
302	# ------------------------------------------------------------------------	22	# ------------------------------------------------------------------------
303	- # def __init__(self, d):
304	- # super().__init__(d)	23	+ def __init__(self, d):
		24	+ super().__init__(d)
		25	+ self['grade'] = nan
		26	+ self['comment'] = ''
305		27
306	# ------------------------------------------------------------------------	28	# ------------------------------------------------------------------------
307	- def start(self, student):	29	+ def start(self, student: dict) -> None:
308	'''	30	'''
309	Write student id in the test and register start time	31	Write student id in the test and register start time
310	'''	32	'''
	@@ -312,59 +34,75 @@ class Test(dict):		@@ -312,59 +34,75 @@ class Test(dict):
312	self['start_time'] = datetime.now()	34	self['start_time'] = datetime.now()
313	self['finish_time'] = None	35	self['finish_time'] = None
314	self['state'] = 'ACTIVE'	36	self['state'] = 'ACTIVE'
315	- self['comment'] = ''
316		37
317	# ------------------------------------------------------------------------	38	# ------------------------------------------------------------------------
318	- def reset_answers(self):	39	+ def reset_answers(self) -> None:
319	'''Removes all answers from the test (clean)'''	40	'''Removes all answers from the test (clean)'''
320	for question in self['questions']:	41	for question in self['questions']:
321	question['answer'] = None	42	question['answer'] = None
322		43
323	# ------------------------------------------------------------------------	44	# ------------------------------------------------------------------------
324	- def update_answer(self, ref, ans):	45	+ def update_answer(self, ref: str, ans) -> None:
325	'''updates one answer in the test'''	46	'''updates one answer in the test'''
326	self['questions'][ref].set_answer(ans)	47	self['questions'][ref].set_answer(ans)
327		48
328	# ------------------------------------------------------------------------	49	# ------------------------------------------------------------------------
329	- def update_answers(self, answers_dict):	50	+ def submit(self, answers_dict) -> None:
330	'''	51	'''
331	Given a dictionary ans={'ref': 'some answer'} updates the answers of	52	Given a dictionary ans={'ref': 'some answer'} updates the answers of
332	multiple questions in the test.	53	multiple questions in the test.
333	Only affects the questions referred in the dictionary.	54	Only affects the questions referred in the dictionary.
334	'''	55	'''
		56	+ self['finish_time'] = datetime.now()
335	for ref, ans in answers_dict.items():	57	for ref, ans in answers_dict.items():
336	self['questions'][ref].set_answer(ans)	58	self['questions'][ref].set_answer(ans)
337	- # self['questions'][ref]['answer'] = ans	59	+ self['state'] = 'SUBMITTED'
338		60
339	# ------------------------------------------------------------------------	61	# ------------------------------------------------------------------------
340	- async def correct(self):	62	+ async def correct_async(self) -> None:
341	'''Corrects all the answers of the test and computes the final grade'''	63	'''Corrects all the answers of the test and computes the final grade'''
342	- self['finish_time'] = datetime.now()
343	- self['state'] = 'FINISHED'
344	-
345	grade = 0.0	64	grade = 0.0
346	for question in self['questions']:	65	for question in self['questions']:
347	await question.correct_async()	66	await question.correct_async()
348	grade += question['grade'] * question['points']	67	grade += question['grade'] * question['points']
349	logger.debug('Correcting %30s: %3g%%',	68	logger.debug('Correcting %30s: %3g%%',
350	- question["ref"], question["grade"]*100)	69	+ question['ref'], question['grade']*100)
		70	+
		71	+ # truncate to avoid negative final grade and adjust scale
		72	+ self['grade'] = max(0.0, grade) + self['scale'][0]
		73	+ self['state'] = 'CORRECTED'
		74	+
		75	+ # ------------------------------------------------------------------------
		76	+ def correct(self) -> None:
		77	+ '''Corrects all the answers of the test and computes the final grade'''
		78	+ grade = 0.0
		79	+ for question in self['questions']:
		80	+ question.correct()
		81	+ grade += question['grade'] * question['points']
		82	+ logger.debug('Correcting %30s: %3g%%',
		83	+ question['ref'], question['grade']*100)
351		84
352	# truncate to avoid negative final grade and adjust scale	85	# truncate to avoid negative final grade and adjust scale
353	self['grade'] = max(0.0, grade) + self['scale'][0]	86	self['grade'] = max(0.0, grade) + self['scale'][0]
354	- return self['grade']	87	+ self['state'] = 'CORRECTED'
355		88
356	# ------------------------------------------------------------------------	89	# ------------------------------------------------------------------------
357	- def giveup(self):	90	+ def giveup(self) -> None:
358	'''Test is marqued as QUIT and is not corrected'''	91	'''Test is marqued as QUIT and is not corrected'''
359	self['finish_time'] = datetime.now()	92	self['finish_time'] = datetime.now()
360	self['state'] = 'QUIT'	93	self['state'] = 'QUIT'
361	self['grade'] = 0.0	94	self['grade'] = 0.0
362	- logger.info('Student %s: gave up.', self["student"]["number"])
363	- return self['grade']
364		95
365	# ------------------------------------------------------------------------	96	# ------------------------------------------------------------------------
366	- def __str__(self):
367	- return ('Test:\n'
368	- f' student: {self.get("student", "--")}\n'
369	- f' start_time: {self.get("start_time", "--")}\n'
370	- f' questions: {", ".join(q["ref"] for q in self["questions"])}\n')	97	+ def save_json(self, pathfile) -> None:
		98	+ '''save test in JSON format'''
		99	+ with open(pathfile, 'w') as file:
		100	+ json.dump(self, file, indent=2, default=str) # str for datetime
		101	+
		102	+ # ------------------------------------------------------------------------
		103	+ def __str__(self) -> str:
		104	+ return '\n'.join([f'{k}: {v}' for k,v in self.items()])
		105	+ # return ('Test:\n'
		106	+ # f' student: {self.get("student", "--")}\n'
		107	+ # f' start_time: {self.get("start_time", "--")}\n'
		108	+ # f' questions: {", ".join(q["ref"] for q in self["questions"])}\n')
@@ -0,0 +1,341 @@		@@ -0,0 +1,341 @@
	1	+'''
	2	+TestFactory - generates tests for students
	3	+'''
	4	+
	5	+# python standard library
	6	+from os import path
	7	+import random
	8	+import logging
	9	+import re
	10	+from typing import Any, Dict
	11	+
	12	+# this project
	13	+from perguntations.questions import QFactory, QuestionException
	14	+from perguntations.test import Test
	15	+from perguntations.tools import load_yaml
	16	+
	17	+# Logger configuration
	18	+logger = logging.getLogger(__name__)
	19	+
	20	+
	21	+# ============================================================================
	22	+class TestFactoryException(Exception):
	23	+ '''exception raised in this module'''
	24	+
	25	+
	26	+# ============================================================================
	27	+class TestFactory(dict):
	28	+ '''
	29	+ Each instance of TestFactory() is a test generator.
	30	+ For example, if we want to serve two different tests, then we need two
	31	+ instances of TestFactory(), one for each test.
	32	+ '''
	33	+
	34	+ # ------------------------------------------------------------------------
	35	+ def __init__(self, conf: Dict[str, Any]) -> None:
	36	+ '''
	37	+ Loads configuration from yaml file, then overrides some configurations
	38	+ using the conf argument.
	39	+ Base questions are added to a pool of questions factories.
	40	+ '''
	41	+
	42	+ # --- set test defaults and then use given configuration
	43	+ super().__init__({ # defaults
	44	+ 'title': '',
	45	+ 'show_points': True,
	46	+ 'scale': None,
	47	+ 'duration': 0, # 0=infinite
	48	+ 'autosubmit': False,
	49	+ 'autocorrect': True,
	50	+ 'debug': False,
	51	+ 'show_ref': False,
	52	+ })
	53	+ self.update(conf)
	54	+
	55	+ # --- for review, we are done. no factories needed
	56	+ if self['review']:
	57	+ logger.info('Review mode. No questions loaded. No factories.')
	58	+ return
	59	+
	60	+ # --- perform sanity checks and normalize the test questions
	61	+ self.sanity_checks()
	62	+ logger.info('Sanity checks PASSED.')
	63	+
	64	+ # --- find refs of all questions used in the test
	65	+ qrefs = {r for qq in self['questions'] for r in qq['ref']}
	66	+ logger.info('Declared %d questions (each test uses %d).',
	67	+ len(qrefs), len(self["questions"]))
	68	+
	69	+ # --- load and build question factories
	70	+ self['question_factory'] = {}
	71	+
	72	+ for file in self["files"]:
	73	+ fullpath = path.normpath(path.join(self["questions_dir"], file))
	74	+
	75	+ logger.info('Loading "%s"...', fullpath)
	76	+ questions = load_yaml(fullpath) # , default=[])
	77	+
	78	+ for i, question in enumerate(questions):
	79	+ # make sure every question in the file is a dictionary
	80	+ if not isinstance(question, dict):
	81	+ msg = f'Question {i} in {file} is not a dictionary'
	82	+ raise TestFactoryException(msg)
	83	+
	84	+ # check if ref is missing, then set to '/path/file.yaml:3'
	85	+ if 'ref' not in question:
	86	+ question['ref'] = f'{file}:{i:04}'
	87	+ logger.warning('Missing ref set to "%s"', question["ref"])
	88	+
	89	+ # check for duplicate refs
	90	+ if question['ref'] in self['question_factory']:
	91	+ other = self['question_factory'][question['ref']]
	92	+ otherfile = path.join(other.question['path'],
	93	+ other.question['filename'])
	94	+ msg = (f'Duplicate reference "{question["ref"]}" in files '
	95	+ f'"{otherfile}" and "{fullpath}".')
	96	+ raise TestFactoryException(msg)
	97	+
	98	+ # make factory only for the questions used in the test
	99	+ if question['ref'] in qrefs:
	100	+ question.update(zip(('path', 'filename', 'index'),
	101	+ path.split(fullpath) + (i,)))
	102	+ if question['type'] == 'code' and 'server' not in question:
	103	+ try:
	104	+ question['server'] = self['jobe_server']
	105	+ except KeyError as exc:
	106	+ msg = f'Missing JOBE server in "{question["ref"]}"'
	107	+ raise TestFactoryException(msg) from exc
	108	+
	109	+ self['question_factory'][question['ref']] = QFactory(question)
	110	+
	111	+ qmissing = qrefs.difference(set(self['question_factory'].keys()))
	112	+ if qmissing:
	113	+ raise TestFactoryException(f'Could not find questions {qmissing}.')
	114	+
	115	+ self.check_questions()
	116	+
	117	+ logger.info('Test factory ready. No errors found.')
	118	+
	119	+
	120	+ # ------------------------------------------------------------------------
	121	+ def check_test_ref(self) -> None:
	122	+ '''Test must have a `ref`'''
	123	+ if 'ref' not in self:
	124	+ raise TestFactoryException('Missing "ref" in configuration!')
	125	+ if not re.match(r'^[a-zA-Z0-9_-]+$', self['ref']):
	126	+ raise TestFactoryException('Test "ref" can only contain the '
	127	+ 'characters a-zA-Z0-9_-')
	128	+
	129	+ def check_missing_database(self) -> None:
	130	+ '''Test must have a database'''
	131	+ if 'database' not in self:
	132	+ raise TestFactoryException('Missing "database" in configuration')
	133	+ if not path.isfile(path.expanduser(self['database'])):
	134	+ msg = f'Database "{self["database"]}" not found!'
	135	+ raise TestFactoryException(msg)
	136	+
	137	+ def check_missing_answers_directory(self) -> None:
	138	+ '''Test must have a answers directory'''
	139	+ if 'answers_dir' not in self:
	140	+ msg = 'Missing "answers_dir" in configuration'
	141	+ raise TestFactoryException(msg)
	142	+
	143	+ def check_answers_directory_writable(self) -> None:
	144	+ '''Answers directory must be writable'''
	145	+ testfile = path.join(path.expanduser(self['answers_dir']), 'REMOVE-ME')
	146	+ try:
	147	+ with open(testfile, 'w') as file:
	148	+ file.write('You can safely remove this file.')
	149	+ except OSError as exc:
	150	+ msg = f'Cannot write answers to directory "{self["answers_dir"]}"'
	151	+ raise TestFactoryException(msg) from exc
	152	+
	153	+ def check_questions_directory(self) -> None:
	154	+ '''Check if questions directory is missing or not accessible.'''
	155	+ if 'questions_dir' not in self:
	156	+ logger.warning('Missing "questions_dir". Using "%s"',
	157	+ path.abspath(path.curdir))
	158	+ self['questions_dir'] = path.curdir
	159	+ elif not path.isdir(path.expanduser(self['questions_dir'])):
	160	+ raise TestFactoryException(f'Can\'t find questions directory '
	161	+ f'"{self["questions_dir"]}"')
	162	+
	163	+ def check_import_files(self) -> None:
	164	+ '''Check if there are files to import (with questions)'''
	165	+ if 'files' not in self:
	166	+ msg = ('Missing "files" in configuration with the list of '
	167	+ 'question files to import!')
	168	+ raise TestFactoryException(msg)
	169	+
	170	+ if isinstance(self['files'], str):
	171	+ self['files'] = [self['files']]
	172	+
	173	+ def check_question_list(self) -> None:
	174	+ '''normalize question list'''
	175	+ if 'questions' not in self:
	176	+ raise TestFactoryException('Missing "questions" in configuration')
	177	+
	178	+ for i, question in enumerate(self['questions']):
	179	+ # normalize question to a dict and ref to a list of references
	180	+ if isinstance(question, str): # e.g., - some_ref
	181	+ question = {'ref': [question]} # becomes - ref: [some_ref]
	182	+ elif isinstance(question, dict) and isinstance(question['ref'], str):
	183	+ question['ref'] = [question['ref']]
	184	+ elif isinstance(question, list):
	185	+ question = {'ref': [str(a) for a in question]}
	186	+
	187	+ self['questions'][i] = question
	188	+
	189	+ def check_missing_title(self) -> None:
	190	+ '''Warns if title is missing'''
	191	+ if not self['title']:
	192	+ logger.warning('Title is undefined!')
	193	+
	194	+ def check_grade_scaling(self) -> None:
	195	+ '''Just informs the scale limits'''
	196	+ if 'scale_points' in self:
	197	+ msg = ('* DEPRECATION WARNING: * scale_points, scale_min, '
	198	+ 'scale_max were replaced by "scale: [min, max]".')
	199	+ logger.warning(msg)
	200	+ self['scale'] = [self['scale_min'], self['scale_max']]
	201	+
	202	+
	203	+ # ------------------------------------------------------------------------
	204	+ def sanity_checks(self) -> None:
	205	+ '''
	206	+ Checks for valid keys and sets default values.
	207	+ Also checks if some files and directories exist
	208	+ '''
	209	+ self.check_test_ref()
	210	+ self.check_missing_database()
	211	+ self.check_missing_answers_directory()
	212	+ self.check_answers_directory_writable()
	213	+ self.check_questions_directory()
	214	+ self.check_import_files()
	215	+ self.check_question_list()
	216	+ self.check_missing_title()
	217	+ self.check_grade_scaling()
	218	+
	219	+ # ------------------------------------------------------------------------
	220	+ def check_questions(self) -> None:
	221	+ '''
	222	+ checks if questions can be correctly generated and corrected
	223	+ '''
	224	+ logger.info('Checking if questions can be generated and corrected...')
	225	+ for i, (qref, qfact) in enumerate(self['question_factory'].items()):
	226	+ try:
	227	+ question = qfact.generate()
	228	+ except Exception as exc:
	229	+ msg = f'Failed to generate "{qref}"'
	230	+ raise TestFactoryException(msg) from exc
	231	+ else:
	232	+ logger.info('%4d. %s: Ok', i, qref)
	233	+ # logger.info(' generate Ok')
	234	+
	235	+ if question['type'] in ('code', 'textarea'):
	236	+ if 'tests_right' in question:
	237	+ for i, right_answer in enumerate(question['tests_right']):
	238	+ try:
	239	+ question.set_answer(right_answer)
	240	+ question.correct()
	241	+ except Exception as exc:
	242	+ msg = f'Failed to correct "{qref}"'
	243	+ raise TestFactoryException(msg) from exc
	244	+
	245	+ if question['grade'] == 1.0:
	246	+ logger.info(' test %i Ok', i)
	247	+ else:
	248	+ logger.error(' TEST %i IS WRONG!!!', i)
	249	+ elif 'tests_wrong' in question:
	250	+ for i, wrong_answer in enumerate(question['tests_wrong']):
	251	+ try:
	252	+ question.set_answer(wrong_answer)
	253	+ question.correct()
	254	+ except Exception as exc:
	255	+ msg = f'Failed to correct "{qref}"'
	256	+ raise TestFactoryException(msg) from exc
	257	+
	258	+ if question['grade'] < 1.0:
	259	+ logger.info(' test %i Ok', i)
	260	+ else:
	261	+ logger.error(' TEST %i IS WRONG!!!', i)
	262	+ else:
	263	+ try:
	264	+ question.set_answer('')
	265	+ question.correct()
	266	+ except Exception as exc:
	267	+ msg = f'Failed to correct "{qref}"'
	268	+ raise TestFactoryException(msg) from exc
	269	+ else:
	270	+ logger.info(' correct Ok but no tests to run')
	271	+
	272	+ # ------------------------------------------------------------------------
	273	+ async def generate(self):
	274	+ '''
	275	+ Given a dictionary with a student dict {'name':'john', 'number': 123}
	276	+ returns instance of Test() for that particular student
	277	+ '''
	278	+
	279	+ # make list of questions
	280	+ questions = []
	281	+ qnum = 1 # track question number
	282	+ nerr = 0 # count errors during questions generation
	283	+
	284	+ for qlist in self['questions']:
	285	+ # choose list of question variants
	286	+ choose = qlist.get('choose', 1)
	287	+ qrefs = random.sample(qlist['ref'], k=choose)
	288	+
	289	+ for qref in qrefs:
	290	+ # generate instance of question
	291	+ try:
	292	+ question = await self['question_factory'][qref].gen_async()
	293	+ except QuestionException:
	294	+ logger.error('Can\'t generate question "%s". Skipping.', qref)
	295	+ nerr += 1
	296	+ continue
	297	+
	298	+ # some defaults
	299	+ if question['type'] in ('information', 'success', 'warning',
	300	+ 'alert'):
	301	+ question['points'] = qlist.get('points', 0.0)
	302	+ else:
	303	+ question['points'] = qlist.get('points', 1.0)
	304	+ question['number'] = qnum # counter for non informative panels
	305	+ qnum += 1
	306	+
	307	+ questions.append(question)
	308	+
	309	+ # setup scale
	310	+ total_points = sum(q['points'] for q in questions)
	311	+
	312	+ if total_points > 0:
	313	+ # normalize question points to scale
	314	+ if self['scale'] is not None:
	315	+ scale_min, scale_max = self['scale']
	316	+ for question in questions:
	317	+ question['points'] *= (scale_max - scale_min) / total_points
	318	+ else:
	319	+ self['scale'] = [0, total_points]
	320	+ else:
	321	+ logger.warning('Total points is ZERO.')
	322	+ if self['scale'] is None:
	323	+ self['scale'] = [0, 20] # default
	324	+
	325	+ if nerr > 0:
	326	+ logger.error('%s errors found!', nerr)
	327	+
	328	+ # copy these from the test configuratoin to each test instance
	329	+ inherit = {'ref', 'title', 'database', 'answers_dir',
	330	+ 'questions_dir', 'files',
	331	+ 'duration', 'autosubmit', 'autocorrect',
	332	+ 'scale', 'show_points',
	333	+ 'show_ref', 'debug', }
	334	+ # NOT INCLUDED: testfile, allow_all, review
	335	+
	336	+ return Test({'questions': questions, **{k:self[k] for k in inherit}})
	337	+
	338	+ # ------------------------------------------------------------------------
	339	+ def __repr__(self):
	340	+ testsettings = '\n'.join(f' {k:14s}: {v}' for k, v in self.items())
	341	+ return '{\n' + testsettings + '\n}'
		1	+'''
		2	+Perguntations setup
		3	+'''
		4	+
1	from setuptools import setup, find_packages	5	from setuptools import setup, find_packages
2		6
3	from perguntations import (__author__, __license__,	7	from perguntations import (__author__, __license__,