- trying to implement a --correct option to corrected previously submitted tests.

(NOT YET FUNCTIONAL) - removed question type code since it can be done in textarea using the jobe_submit module and is more flexible

- trying to implement a --correct option to corrected previously submitted tests.
(NOT YET FUNCTIONAL) - removed question type code since it can be done in textarea using the jobe_submit module and is more flexible
Miguel Barão
1 parent 45e6dc40
Showing 11 changed files with 319 additions and 258 deletions Show diff stats
BUGS.md
demo/demo.yaml
demo/questions/questions-tutorial.yaml
perguntations/app.py
perguntations/main.py
perguntations/models.py
perguntations/questions.py
perguntations/serve.py
perguntations/templates/grade.html
perguntations/test.py
perguntations/testfactory.py
 # BUGS
+- cookies existe um perguntations_user e um user. De onde vem o user?
+- nao esta a mostrar imagens?? internal server error?
 - JOBE correct async
 - esta a corrigir código JOBE mesmo que nao tenha respondido???
 - QuestionCode falta reportar nos comments os vários erros que podem ocorrer (timeout, etc)
+
 - algumas vezes a base de dados guarda o mesmo teste em duplicado. ver se dois submits dao origem a duas correcções.
 talvez a base de dados devesse ter como chave do teste um id que fosse único desse teste particular (não um auto counter, nem ref do teste)
 - em caso de timeout na submissão (e.g. JOBE ou script nao responde) a correcção não termina e o teste não é guardado.
@@ -31,6 +31,12 @@ duration: 20
 # (default: false)
 autosubmit: true
+# If true, the test will be corrected on submission, the grade calculated and
+# shown to the student. If false, the test is saved but not corrected.
+# No grade is shown to the student.
+# (default: true)
+autocorrect: false
+
 # Show points for each question (min and max).
 # (default: true)
 show_points: true
@@ -74,5 +80,4 @@ questions:
   - [tut-alert1, tut-alert2]
   - tut-generator
   - tut-yamllint
-  - tut-code
-
+  # - tut-code
@@ -576,7 +576,7 @@
 # ----------------------------------------------------------------------------
 - type: information
   text: |
-    This question is not included in the test and will not shown up.
+    This question is not included in the test and will not show up.
     It also lacks a "ref" and is automatically named
     `questions/questions-tutorial.yaml:0013`.
     A warning is shown on the console about this.
@@ -612,50 +612,50 @@
     ```
 # ----------------------------------------------------------------------------
-- type: code
-  ref: tut-code
-  title: Submissão de código (JOBE)
-  text: |
-    É possível enviar código para ser compilado e executado por um servidor
-    JOBE instalado separadamente, ver [JOBE](https://github.com/trampgeek/jobe).
-
-    ```yaml
-    - type: code
-      ref: tut-code
-      title: Submissão de código (JOBE)
-      text: |
-        Escreva um programa em C que recebe uma string no standard input e
-        mostra a mensagem `hello ` seguida da string.
-        Por exemplo, se o input for `Maria`, o output deverá ser `hello Maria`.
-      language: c
-      correct:
-        - stdin: 'Maria'
-          stdout: 'hello Maria'
-        - stdin: 'xyz'
-          stdout: 'hello xyz'
-    ```
-
-    Existem várias linguagens suportadas pelo servidor JOBE (C, C++, Java,
-    Python2, Python3, Octave, Pascal, PHP).
-    O campo `correct` deverá ser uma lista de casos a testar.
-    Se um caso incluir `stdin`, este será enviado para o programa e o `stdout`
-    obtido será comparado com o declarado. A pergunta é considerada correcta se
-    todos os outputs coincidirem.
-
-    Por defeito é o usado o servidor JOBE declarado no teste. Para usar outro
-    diferente nesta pergunta usa-se a opção `server: 127.0.0.1` com o endereço
-    apropriado.
-  answer: |
-    #include <stdio.h>
-    int main() {
-      char name[20];
-      scanf("%s", name);
-      printf("hello %s", name);
-    }
-  # server: 192.168.1.85
-  language: c
-  correct:
-    - stdin: 'Maria'
-      stdout: 'hello Maria'
-    - stdin: 'xyz'
-      stdout: 'hello xyz'
+# - type: code
+#   ref: tut-code
+#   title: Submissão de código (JOBE)
+#   text: |
+#     É possível enviar código para ser compilado e executado por um servidor
+#     JOBE instalado separadamente, ver [JOBE](https://github.com/trampgeek/jobe).
+
+#     ```yaml
+#     - type: code
+#       ref: tut-code
+#       title: Submissão de código (JOBE)
+#       text: |
+#         Escreva um programa em C que recebe uma string no standard input e
+#         mostra a mensagem `hello ` seguida da string.
+#         Por exemplo, se o input for `Maria`, o output deverá ser `hello Maria`.
+#       language: c
+#       correct:
+#         - stdin: 'Maria'
+#           stdout: 'hello Maria'
+#         - stdin: 'xyz'
+#           stdout: 'hello xyz'
+#     ```
+
+#     Existem várias linguagens suportadas pelo servidor JOBE (C, C++, Java,
+#     Python2, Python3, Octave, Pascal, PHP).
+#     O campo `correct` deverá ser uma lista de casos a testar.
+#     Se um caso incluir `stdin`, este será enviado para o programa e o `stdout`
+#     obtido será comparado com o declarado. A pergunta é considerada correcta se
+#     todos os outputs coincidirem.
+
+#     Por defeito é o usado o servidor JOBE declarado no teste. Para usar outro
+#     diferente nesta pergunta usa-se a opção `server: 127.0.0.1` com o endereço
+#     apropriado.
+#   answer: |
+#     #include <stdio.h>
+#     int main() {
+#       char name[20];
+#       scanf("%s", name);
+#       printf("hello %s", name);
+#     }
+#   # server: 192.168.1.85
+#   language: c
+#   correct:
+#     - stdin: 'Maria'
+#       stdout: 'hello Maria'
+#     - stdin: 'xyz'
+#       stdout: 'hello xyz'
@@ -21,6 +21,8 @@ from sqlalchemy.orm import sessionmaker
 from perguntations.models import Student, Test, Question
 from perguntations.tools import load_yaml
 from perguntations.testfactory import TestFactory, TestFactoryException
+import perguntations.test
+from perguntations.questions import QuestionFrom
 logger = logging.getLogger(__name__)
@@ -33,12 +35,12 @@ class AppException(Exception):
 # ============================================================================
 # helper functions
 # ============================================================================
-async def check_password(try_pw, password):
+async def check_password(try_pw, hashed_pw):
     '''check password in executor'''
     try_pw = try_pw.encode('utf-8')
     loop = asyncio.get_running_loop()
-    hashed = await loop.run_in_executor(None, bcrypt.hashpw, try_pw, password)
-    return password == hashed
+    hashed = await loop.run_in_executor(None, bcrypt.hashpw, try_pw, hashed_pw)
+    return hashed_pw == hashed
 async def hash_password(password):
@@ -121,6 +123,39 @@ class App():
         else:
             logger.info('No tests were generated.')
+        if conf['correct']:
+            self._correct_tests()
+
+    # ------------------------------------------------------------------------
+    def _correct_tests(self):
+        with self._db_session() as sess:
+            filenames = sess.query(Test.filename)\
+                    .filter(Test.ref == self.testfactory['ref'])\
+                    .filter(Test.state == "SUBMITTED")\
+                    .all()
+            # print([(x.filename, x.state, x.grade) for x in a])
+        logger.info('Correcting %d tests...', len(filenames))
+
+        for filename, in filenames:
+            try:
+                with open(filename) as file:
+                    testdict = json.load(file)
+            except FileNotFoundError:
+                logger.error('File not found: %s', filename)
+                continue
+
+            test = perguntations.test.Test(testdict)
+            print(test['questions'][7]['correct'])
+            test['questions'] = [QuestionFrom(q) for q in test['questions']]
+
+            print(test['questions'][7]['correct'])
+            test.correct()
+            logger.info('Student %s:  grade = %f', test['student']['number'], test['grade'])
+
+
+        # FIXME update JSON and database
+
+
     # ------------------------------------------------------------------------
     async def login(self, uid, try_pw, headers=None):
         '''login authentication'''
@@ -129,15 +164,15 @@ class App():
             return 'unauthorized'
         with self._db_session() as sess:
-            name, password = sess.query(Student.name, Student.password)\
+            name, hashed_pw = sess.query(Student.name, Student.password)\
                                  .filter_by(id=uid)\
                                  .one()
-        if password == '':              # update password on first login
+        if hashed_pw == '':              # update password on first login
             await self.update_student_password(uid, try_pw)
             pw_ok = True
         else:                           # check password
-            pw_ok = await check_password(try_pw, password)  # async bcrypt
+            pw_ok = await check_password(try_pw, hashed_pw)  # async bcrypt
         if not pw_ok:        # wrong password
             logger.info('"%s" wrong password.', uid)
@@ -216,6 +251,7 @@ class App():
         '''get test from online student or raise exception'''
         return self.online[uid]['test']
+    # ------------------------------------------------------------------------
     async def _new_test(self, uid):
         '''
         assign a test to a given student. if there are pregenerated tests then
@@ -233,13 +269,13 @@ class App():
         else:
             logger.info('"%s" using a pregenerated test.', uid)
-        test.register(student)           # student signs the test
+        test.start(student)           # student signs the test
         self.online[uid]['test'] = test
     # ------------------------------------------------------------------------
-    async def correct_test(self, uid, ans):
+    async def submit_test(self, uid, ans):
         '''
-        Corrects test
+        Handles test submission and correction.
         ans is a dictionary {question_index: answer, ...} with the answers for
         the complete test. For example:  {0:'hello', 1:[1,2]}
@@ -247,49 +283,55 @@ class App():
         test = self.online[uid]['test']
         # --- submit answers and correct test
-        test.update_answers(ans)
+        test.submit(ans)
         logger.info('"%s" submitted %d answers.', uid, len(ans))
-        grade = await test.correct()
-        logger.info('"%s" grade = %g points.', uid, grade)
+        if test['autocorrect']:
+            await test.correct_async()
+            logger.info('"%s" grade = %g points.', uid, test['grade'])
         # --- save test in JSON format
         fields = (uid, test['ref'], str(test['finish_time']))
         fname = '--'.join(fields) + '.json'
         fpath = path.join(test['answers_dir'], fname)
         with open(path.expanduser(fpath), 'w') as file:
-            json.dump(test, file, indent=2, default=str)
-            # option default=str is required for datetime objects
-
+            json.dump(test, file, indent=2, default=str)  # str for datetime
         logger.info('"%s" saved JSON.', uid)
-        # --- insert test and questions into database
+        # --- insert test and questions into the database
+        #     only corrected questions are added
         test_row = Test(
             ref=test['ref'],
             title=test['title'],
             grade=test['grade'],
             state=test['state'],
-            comment='',
+            comment=test['comment'],
             starttime=str(test['start_time']),
             finishtime=str(test['finish_time']),
             filename=fpath,
             student_id=uid)
-        test_row.questions = [Question(
-            number=n,
-            ref=q['ref'],
-            grade=q['grade'],
-            comment=q.get('comment', ''),
-            starttime=str(test['start_time']),
-            finishtime=str(test['finish_time']),
-            test_id=test['ref'])
-                for n, q in enumerate(test['questions'])
-                if 'grade' in q
+
+        test_row.questions = [
+            Question(
+                number=n,
+                ref=q['ref'],
+                grade=q['grade'],
+                comment=q.get('comment', ''),
+                starttime=str(test['start_time']),
+                finishtime=str(test['finish_time']),
+                test_id=test['ref']
+                )
+            for n, q in enumerate(test['questions'])
+            if 'grade' in q
             ]
+
         with self._db_session() as sess:
             sess.add(test_row)
-
         logger.info('"%s" database updated.', uid)
-        return grade
+
+    # ------------------------------------------------------------------------
+    def get_student_grade(self, uid):
+        return self.online[uid]['test'].get('grade', None)
     # ------------------------------------------------------------------------
     # def giveup_test(self, uid):
@@ -49,6 +49,9 @@ def parse_cmdline_arguments():
     parser.add_argument('--review',
                         action='store_true',
                         help='Review mode: doesn\'t generate test')
+    parser.add_argument('--correct',
+                        action='store_true',
+                        help='Correct test and update JSON files and database')
     parser.add_argument('--port',
                         type=int,
                         default=8443,
@@ -123,11 +126,12 @@ def main():
     # --- start application --------------------------------------------------
     config = {
         'testfile': args.testfile,
-        'debug':    args.debug,
+        'debug': args.debug,
         'allow_all': args.allow_all,
         'allow_list': args.allow_list,
         'show_ref': args.show_ref,
-        'review':   args.review,
+        'review': args.review,
+        'correct': args.correct,
         }
     try:
@@ -41,7 +41,7 @@ class Test(Base):
     ref = Column(String)
     title = Column(String)
     grade = Column(Float)
-    state = Column(String)  # ACTIVE, FINISHED, QUIT, NULL
+    state = Column(String)  # ACTIVE, SUBMITTED, CORRECTED, QUIT, NULL
     comment = Column(String)
     starttime = Column(String)
     finishtime = Column(String)
@@ -32,6 +32,44 @@ QDict = NewType(&#39;QDict&#39;, Dict[str, Any])
 class QuestionException(Exception):
     '''Exceptions raised in this module'''
+# FIXME if this works, use it below
+def QuestionFrom(question: dict):
+    types = {
+        'radio':            QuestionRadio,
+        'checkbox':         QuestionCheckbox,
+        'text':             QuestionText,
+        'text-regex':       QuestionTextRegex,
+        'numeric-interval': QuestionNumericInterval,
+        'textarea':         QuestionTextArea,
+        # 'code':             QuestionCode,
+        # -- informative panels --
+        'information':      QuestionInformation,
+        'success':          QuestionInformation,
+        'warning':          QuestionInformation,
+        'alert':            QuestionInformation,
+        }
+
+    # Get class for this question type
+    try:
+        qclass = types[question['type']]
+    except KeyError:
+        logger.error('Invalid type "%s" in "%s"',
+                     question['type'], question['ref'])
+        raise
+
+    # Finally create an instance of Question()
+    try:
+        qinstance = qclass(QDict(question))
+    except QuestionException:
+        logger.error('Error generating question "%s". See "%s/%s"',
+                     question['ref'],
+                     question['path'],
+                     question['filename'])
+        raise
+
+    return qinstance
+
+
 # ============================================================================
 # Questions derived from Question are already instantiated and ready to be
@@ -590,101 +628,101 @@ class QuestionTextArea(Question):
 # ============================================================================
-class QuestionCode(Question):
-    '''
-    Submits answer to a JOBE server to compile and run against the test cases.
-    '''
-
-    _outcomes = {
-        0:  'JOBE outcome: Successful run',
-        11: 'JOBE outcome: Compile error',
-        12: 'JOBE outcome: Runtime error',
-        13: 'JOBE outcome: Time limit exceeded',
-        15: 'JOBE outcome: Successful run',
-        17: 'JOBE outcome: Memory limit exceeded',
-        19: 'JOBE outcome: Illegal system call',
-        20: 'JOBE outcome: Internal error, please report',
-        21: 'JOBE outcome: Server overload',
-        }
-
-    # ------------------------------------------------------------------------
-    def __init__(self, q: QDict) -> None:
-        super().__init__(q)
-
-        self.set_defaults(QDict({
-            'text': '',
-            'timeout': 5,   # seconds
-            'server': '127.0.0.1',  # JOBE server
-            'language': 'c',
-            'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
-            }))
-
-    # ------------------------------------------------------------------------
-    def correct(self) -> None:
-        super().correct()
-
-        if self['answer'] is None:
-            return
-
-        # submit answer to JOBE server
-        resource = '/jobe/index.php/restapi/runs/'
-        headers = {"Content-type": "application/json; charset=utf-8",
-                   "Accept": "application/json"}
-
-        for expected in self['correct']:
-            data_json = json.dumps({
-                'run_spec' : {
-                    'language_id': self['language'],
-                    'sourcecode': self['answer'],
-                    'input': expected.get('stdin', ''),
-                     },
-                })
-
-            try:
-                connect = http.client.HTTPConnection(self['server'])
-                connect.request(
-                    method='POST',
-                    url=resource,
-                    body=data_json,
-                    headers=headers
-                    )
-                response = connect.getresponse()
-                logger.debug('JOBE response status %d', response.status)
-                if response.status != 204:
-                    content = response.read().decode('utf8')
-                    if content:
-                        result = json.loads(content)
-                connect.close()
-
-            except (HTTPError, ValueError):
-                logger.error('HTTPError while connecting to JOBE server')
-
-            try:
-                outcome = result['outcome']
-            except (NameError, TypeError, KeyError):
-                logger.error('Bad result returned from JOBE server: %s', result)
-                return
-            logger.debug(self._outcomes[outcome])
-
-
-
-            if result['cmpinfo']:  # compiler errors and warnings
-                self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
-                self['grade'] = 0.0
-                return
-
-            if result['stdout'] != expected.get('stdout', ''):
-                self['comments'] = 'O output gerado é diferente do esperado.'  # FIXME mostrar porque?
-                self['grade'] = 0.0
-                return
-
-        self['comments'] = 'Ok!'
-        self['grade'] = 1.0
-
+# class QuestionCode(Question):
+#     '''
+#     Submits answer to a JOBE server to compile and run against the test cases.
+#     '''
+
+#     _outcomes = {
+#         0:  'JOBE outcome: Successful run',
+#         11: 'JOBE outcome: Compile error',
+#         12: 'JOBE outcome: Runtime error',
+#         13: 'JOBE outcome: Time limit exceeded',
+#         15: 'JOBE outcome: Successful run',
+#         17: 'JOBE outcome: Memory limit exceeded',
+#         19: 'JOBE outcome: Illegal system call',
+#         20: 'JOBE outcome: Internal error, please report',
+#         21: 'JOBE outcome: Server overload',
+#         }
+
+#     # ------------------------------------------------------------------------
+#     def __init__(self, q: QDict) -> None:
+#         super().__init__(q)
+
+#         self.set_defaults(QDict({
+#             'text': '',
+#             'timeout': 5,   # seconds
+#             'server': '127.0.0.1',  # JOBE server
+#             'language': 'c',
+#             'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
+#             }))
     # ------------------------------------------------------------------------
-    async def correct_async(self) -> None:
-        self.correct()
+    # def correct(self) -> None:
+    #     super().correct()
+
+    #     if self['answer'] is None:
+    #         return
+
+    #     # submit answer to JOBE server
+    #     resource = '/jobe/index.php/restapi/runs/'
+    #     headers = {"Content-type": "application/json; charset=utf-8",
+    #                "Accept": "application/json"}
+
+    #     for expected in self['correct']:
+    #         data_json = json.dumps({
+    #             'run_spec' : {
+    #                 'language_id': self['language'],
+    #                 'sourcecode': self['answer'],
+    #                 'input': expected.get('stdin', ''),
+    #                  },
+    #             })
+
+    #         try:
+    #             connect = http.client.HTTPConnection(self['server'])
+    #             connect.request(
+    #                 method='POST',
+    #                 url=resource,
+    #                 body=data_json,
+    #                 headers=headers
+    #                 )
+    #             response = connect.getresponse()
+    #             logger.debug('JOBE response status %d', response.status)
+    #             if response.status != 204:
+    #                 content = response.read().decode('utf8')
+    #                 if content:
+    #                     result = json.loads(content)
+    #             connect.close()
+
+    #         except (HTTPError, ValueError):
+    #             logger.error('HTTPError while connecting to JOBE server')
+
+    #         try:
+    #             outcome = result['outcome']
+    #         except (NameError, TypeError, KeyError):
+    #             logger.error('Bad result returned from JOBE server: %s', result)
+    #             return
+    #         logger.debug(self._outcomes[outcome])
+
+
+
+    #         if result['cmpinfo']:  # compiler errors and warnings
+    #             self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
+    #             self['grade'] = 0.0
+    #             return
+
+    #         if result['stdout'] != expected.get('stdout', ''):
+    #             self['comments'] = 'O output gerado é diferente do esperado.'  # FIXME mostrar porque?
+    #             self['grade'] = 0.0
+    #             return
+
+    #     self['comments'] = 'Ok!'
+    #     self['grade'] = 1.0
+
+
+    # # ------------------------------------------------------------------------
+    # async def correct_async(self) -> None:
+    #     self.correct()  # FIXME there is no async correction!!!
             # out = run_script(
@@ -731,7 +769,6 @@ class QuestionInformation(Question):
         super().correct()
         self['grade'] = 1.0  # always "correct" but points should be zero!
-
 # ============================================================================
 class QFactory():
     '''
@@ -774,7 +811,7 @@ class QFactory():
         'text-regex':       QuestionTextRegex,
         'numeric-interval': QuestionNumericInterval,
         'textarea':         QuestionTextArea,
-        'code':             QuestionCode,
+        # 'code':             QuestionCode,
         # -- informative panels --
         'information':      QuestionInformation,
         'success':          QuestionInformation,
@@ -187,12 +187,12 @@ class LoginHandler(BaseHandler):
         error = await self.testapp.login(uid, password, headers)
-        if error is None:
-            self.set_secure_cookie('perguntations_user', str(uid))
-            self.redirect('/')
-        else:
+        if error:
             await asyncio.sleep(3)  # delay to avoid spamming the server...
             self.render('login.html', error=self._error_msg[error])
+        else:
+            self.set_secure_cookie('perguntations_user', str(uid))
+            self.redirect('/')
 # ----------------------------------------------------------------------------
@@ -293,18 +293,19 @@ class RootHandler(BaseHandler):
                                           'numeric-interval', 'code'):
                     ans[i] = ans[i][0]
-        # correct answered questions and logout
-        await self.testapp.correct_test(uid, ans)
+        # submit answered questions, correct
+        await self.testapp.submit_test(uid, ans)
         # show final grade and grades of other tests in the database
-        allgrades = self.testapp.get_student_grades_from_all_tests(uid)
+        # allgrades = self.testapp.get_student_grades_from_all_tests(uid)
+        grade = self.testapp.get_student_grade(uid)
+        self.render('grade.html', t=test)
         self.clear_cookie('perguntations_user')
-        self.render('grade.html', t=test, allgrades=allgrades)
         self.testapp.logout(uid)
         timeit_finish = timer()
-        logging.info('  correction took %fs', timeit_finish-timeit_start)
+        logging.info('  elapsed time: %fs', timeit_finish-timeit_start)
 # ----------------------------------------------------------------------------
@@ -41,67 +41,21 @@
 <div class="container">
   <div class="jumbotron">
     {% if t['state'] == 'FINISHED' %}
-      <h1>Resultado:
+      <h3>Resultado:
         <strong>{{ f'{round(t["grade"], 3)}' }}</strong>
-        valores na escala de {{t['scale'][0]}} a {{t['scale'][1]}}.
-      </h1>
-      <p>O seu teste foi correctamente entregue e a nota registada.</p>
-      <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para sair do teste</a></p>
+        valores na escala [{{t['scale'][0]}},{{t['scale'][1]}}].
+      </h3>
       {% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}
         <i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>
       {% end %}
+    {% elif t['state'] == 'SUBMITTED' %}
+      <h3>A prova foi submetida com sucesso. Vai ser corrigida mais tarde.</h3>
     {% elif t['state'] == 'QUIT' %}
-      <p>Foi registada a sua desistência da prova.</p>
+      <h3>Foi registada a sua desistência da prova.</h3>
     {% end %}
+    <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para terminar</a></p>
   </div> <!-- jumbotron -->
-
-  <div class="card">
-    <h5 class="card-header">
-        Histórico de resultados
-    </h5>
-    <table class="table table-condensed noleftmargin">
-      <thead>
-          <tr>
-              <th>Prova</th>
-              <th>Data</th>
-              <th>Hora</th>
-              <th>Nota</th>
-          </tr>
-      </thead>
-      <tbody>
-        {% for g in allgrades %}
-          <tr>
-            <td>{{g[0]}}</td>  <!-- teste -->
-            <td>{{g[2][:10]}}</td>  <!-- data -->
-            <td>{{g[2][11:19]}}</td>  <!-- hora -->
-            <td> <!-- progress column -->
-              <div class="progress" style="height: 20px;">
-                <div class="progress-bar
-                  {% if g[1] - t['scale'][0] < 0.5*(t['scale'][1] - t['scale'][0]) %}
-                      bg-danger
-                  {% elif g[1]  - t['scale'][0] < 0.75*(t['scale'][1] - t['scale'][0]) %}
-                      bg-warning
-                  {% else %}
-                      bg-success
-                  {% end %}
-                  "
-                  role="progressbar"
-                  aria-valuenow="{{ 100*(g[1] - t['scale'][0])/(t['scale'][1] - t['scale'][0]) }}"
-                  aria-valuemin="0"
-                  aria-valuemax="100"
-                  style="min-width: 2em; width: {{ 100*(g[1]-t['scale'][0])/(t['scale'][1]-t['scale'][0]) }}%;">
-
-                    {{ str(round(g[1], 1)) }}
-
-                </div> <!-- progress-bar -->
-              </div> <!-- progress -->
-            </td> <!-- progress column -->
-          </tr>
-        {% end %}
-      </tbody>
-    </table>
-  </div> <!-- panel -->
 </div> <!-- container -->
 </body>
 </html>
@@ -5,6 +5,7 @@ Test - instances of this class are individual tests
 # python standard library
 from datetime import datetime
 import logging
+from math import nan
 # Logger configuration
 logger = logging.getLogger(__name__)
@@ -17,11 +18,13 @@ class Test(dict):
     '''
     # ------------------------------------------------------------------------
-    # def __init__(self, d):
-    #     super().__init__(d)
+    def __init__(self, d):
+        super().__init__(d)
+        self['grade'] = nan
+        self['comment'] = ''
     # ------------------------------------------------------------------------
-    def register(self, student: dict) -> None:
+    def start(self, student: dict) -> None:
         '''
         Write student id in the test and register start time
         '''
@@ -29,7 +32,6 @@ class Test(dict):
         self['start_time'] = datetime.now()
         self['finish_time'] = None
         self['state'] = 'ACTIVE'
-        self['comment'] = ''
     # ------------------------------------------------------------------------
     def reset_answers(self) -> None:
@@ -43,44 +45,56 @@ class Test(dict):
         self['questions'][ref].set_answer(ans)
     # ------------------------------------------------------------------------
-    def update_answers(self, answers_dict) -> None:
+    def submit(self, answers_dict) -> None:
         '''
         Given a dictionary ans={'ref': 'some answer'} updates the answers of
         multiple questions in the test.
         Only affects the questions referred in the dictionary.
         '''
+        self['finish_time'] = datetime.now()
         for ref, ans in answers_dict.items():
             self['questions'][ref].set_answer(ans)
+        self['state'] = 'SUBMITTED'
     # ------------------------------------------------------------------------
-    async def correct(self) -> float:
+    async def correct_async(self) -> None:
         '''Corrects all the answers of the test and computes the final grade'''
-        self['finish_time'] = datetime.now()
-        self['state'] = 'FINISHED'
-
         grade = 0.0
         for question in self['questions']:
             await question.correct_async()
             grade += question['grade'] * question['points']
             logger.debug('Correcting %30s: %3g%%',
-                         question["ref"], question["grade"]*100)
+                         question['ref'], question['grade']*100)
+
+        # truncate to avoid negative final grade and adjust scale
+        self['grade'] = max(0.0, grade) + self['scale'][0]
+        self['state'] = 'CORRECTED'
+
+    # ------------------------------------------------------------------------
+    def correct(self) -> None:
+        '''Corrects all the answers of the test and computes the final grade'''
+        grade = 0.0
+        for question in self['questions']:
+            question.correct()
+            grade += question['grade'] * question['points']
+            logger.debug('Correcting %30s: %3g%%',
+                         question['ref'], question['grade']*100)
         # truncate to avoid negative final grade and adjust scale
         self['grade'] = max(0.0, grade) + self['scale'][0]
-        return self['grade']
+        self['state'] = 'CORRECTED'
     # ------------------------------------------------------------------------
-    def giveup(self) -> float:
+    def giveup(self) -> None:
         '''Test is marqued as QUIT and is not corrected'''
         self['finish_time'] = datetime.now()
         self['state'] = 'QUIT'
         self['grade'] = 0.0
-        logger.info('Student %s:  gave up.', self["student"]["number"])
-        return self['grade']
     # ------------------------------------------------------------------------
     def __str__(self) -> str:
-        return ('Test:\n'
-                f'  student:  {self.get("student", "--")}\n'
-                f'  start_time:  {self.get("start_time", "--")}\n'
-                f'  questions:  {", ".join(q["ref"] for q in self["questions"])}\n')
+        return '\n'.join([f'{k}: {v}' for k,v in self.items()])
+        # return ('Test:\n'
+        #         f'  student:  {self.get("student", "--")}\n'
+        #         f'  start_time:  {self.get("start_time", "--")}\n'
+        #         f'  questions:  {", ".join(q["ref"] for q in self["questions"])}\n')
@@ -46,6 +46,7 @@ class TestFactory(dict):
             'scale': None,
             'duration': 0,  # 0=infinite
             'autosubmit': False,
+            'autocorrect': True,
             'debug': False,
             'show_ref': False,
             })
@@ -300,7 +301,7 @@ class TestFactory(dict):
         # copy these from the test configuratoin to each test instance
         inherit = {'ref', 'title', 'database', 'answers_dir',
                    'questions_dir', 'files',
-                   'duration', 'autosubmit',
+                   'duration', 'autosubmit', 'autocorrect',
                    'scale', 'show_points',
                    'show_ref', 'debug', }
         # NOT INCLUDED:  testfile, allow_all, review
	@@ -41,7 +41,7 @@ class Test(Base):		@@ -41,7 +41,7 @@ class Test(Base):
41	ref = Column(String)	41	ref = Column(String)
42	title = Column(String)	42	title = Column(String)
43	grade = Column(Float)	43	grade = Column(Float)
44	- state = Column(String) # ACTIVE, FINISHED, QUIT, NULL	44	+ state = Column(String) # ACTIVE, SUBMITTED, CORRECTED, QUIT, NULL
45	comment = Column(String)	45	comment = Column(String)
46	starttime = Column(String)	46	starttime = Column(String)
47	finishtime = Column(String)	47	finishtime = Column(String)
	@@ -32,6 +32,44 @@ QDict = NewType('QDict', Dict[str, Any])		@@ -32,6 +32,44 @@ QDict = NewType('QDict', Dict[str, Any])
32	class QuestionException(Exception):	32	class QuestionException(Exception):
33	'''Exceptions raised in this module'''	33	'''Exceptions raised in this module'''
34		34
		35	+# FIXME if this works, use it below
		36	+def QuestionFrom(question: dict):
		37	+ types = {
		38	+ 'radio': QuestionRadio,
		39	+ 'checkbox': QuestionCheckbox,
		40	+ 'text': QuestionText,
		41	+ 'text-regex': QuestionTextRegex,
		42	+ 'numeric-interval': QuestionNumericInterval,
		43	+ 'textarea': QuestionTextArea,
		44	+ # 'code': QuestionCode,
		45	+ # -- informative panels --
		46	+ 'information': QuestionInformation,
		47	+ 'success': QuestionInformation,
		48	+ 'warning': QuestionInformation,
		49	+ 'alert': QuestionInformation,
		50	+ }
		51	+
		52	+ # Get class for this question type
		53	+ try:
		54	+ qclass = types[question['type']]
		55	+ except KeyError:
		56	+ logger.error('Invalid type "%s" in "%s"',
		57	+ question['type'], question['ref'])
		58	+ raise
		59	+
		60	+ # Finally create an instance of Question()
		61	+ try:
		62	+ qinstance = qclass(QDict(question))
		63	+ except QuestionException:
		64	+ logger.error('Error generating question "%s". See "%s/%s"',
		65	+ question['ref'],
		66	+ question['path'],
		67	+ question['filename'])
		68	+ raise
		69	+
		70	+ return qinstance
		71	+
		72	+
35		73
36	# ============================================================================	74	# ============================================================================
37	# Questions derived from Question are already instantiated and ready to be	75	# Questions derived from Question are already instantiated and ready to be
	@@ -590,101 +628,101 @@ class QuestionTextArea(Question):		@@ -590,101 +628,101 @@ class QuestionTextArea(Question):
590		628
591		629
592	# ============================================================================	630	# ============================================================================
593	-class QuestionCode(Question):
594	- '''
595	- Submits answer to a JOBE server to compile and run against the test cases.
596	- '''
597	-
598	- _outcomes = {
599	- 0: 'JOBE outcome: Successful run',
600	- 11: 'JOBE outcome: Compile error',
601	- 12: 'JOBE outcome: Runtime error',
602	- 13: 'JOBE outcome: Time limit exceeded',
603	- 15: 'JOBE outcome: Successful run',
604	- 17: 'JOBE outcome: Memory limit exceeded',
605	- 19: 'JOBE outcome: Illegal system call',
606	- 20: 'JOBE outcome: Internal error, please report',
607	- 21: 'JOBE outcome: Server overload',
608	- }
609	-
610	- # ------------------------------------------------------------------------
611	- def __init__(self, q: QDict) -> None:
612	- super().__init__(q)
613	-
614	- self.set_defaults(QDict({
615	- 'text': '',
616	- 'timeout': 5, # seconds
617	- 'server': '127.0.0.1', # JOBE server
618	- 'language': 'c',
619	- 'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
620	- }))
621	-
622	- # ------------------------------------------------------------------------
623	- def correct(self) -> None:
624	- super().correct()
625	-
626	- if self['answer'] is None:
627	- return
628	-
629	- # submit answer to JOBE server
630	- resource = '/jobe/index.php/restapi/runs/'
631	- headers = {"Content-type": "application/json; charset=utf-8",
632	- "Accept": "application/json"}
633	-
634	- for expected in self['correct']:
635	- data_json = json.dumps({
636	- 'run_spec' : {
637	- 'language_id': self['language'],
638	- 'sourcecode': self['answer'],
639	- 'input': expected.get('stdin', ''),
640	- },
641	- })
642	-
643	- try:
644	- connect = http.client.HTTPConnection(self['server'])
645	- connect.request(
646	- method='POST',
647	- url=resource,
648	- body=data_json,
649	- headers=headers
650	- )
651	- response = connect.getresponse()
652	- logger.debug('JOBE response status %d', response.status)
653	- if response.status != 204:
654	- content = response.read().decode('utf8')
655	- if content:
656	- result = json.loads(content)
657	- connect.close()
658	-
659	- except (HTTPError, ValueError):
660	- logger.error('HTTPError while connecting to JOBE server')
661	-
662	- try:
663	- outcome = result['outcome']
664	- except (NameError, TypeError, KeyError):
665	- logger.error('Bad result returned from JOBE server: %s', result)
666	- return
667	- logger.debug(self._outcomes[outcome])
668	-
669	-
670	-
671	- if result['cmpinfo']: # compiler errors and warnings
672	- self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
673	- self['grade'] = 0.0
674	- return
675	-
676	- if result['stdout'] != expected.get('stdout', ''):
677	- self['comments'] = 'O output gerado é diferente do esperado.' # FIXME mostrar porque?
678	- self['grade'] = 0.0
679	- return
680	-
681	- self['comments'] = 'Ok!'
682	- self['grade'] = 1.0
683	-	631	+# class QuestionCode(Question):
		632	+# '''
		633	+# Submits answer to a JOBE server to compile and run against the test cases.
		634	+# '''
		635	+
		636	+# _outcomes = {
		637	+# 0: 'JOBE outcome: Successful run',
		638	+# 11: 'JOBE outcome: Compile error',
		639	+# 12: 'JOBE outcome: Runtime error',
		640	+# 13: 'JOBE outcome: Time limit exceeded',
		641	+# 15: 'JOBE outcome: Successful run',
		642	+# 17: 'JOBE outcome: Memory limit exceeded',
		643	+# 19: 'JOBE outcome: Illegal system call',
		644	+# 20: 'JOBE outcome: Internal error, please report',
		645	+# 21: 'JOBE outcome: Server overload',
		646	+# }
		647	+
		648	+# # ------------------------------------------------------------------------
		649	+# def __init__(self, q: QDict) -> None:
		650	+# super().__init__(q)
		651	+
		652	+# self.set_defaults(QDict({
		653	+# 'text': '',
		654	+# 'timeout': 5, # seconds
		655	+# 'server': '127.0.0.1', # JOBE server
		656	+# 'language': 'c',
		657	+# 'correct': [{'stdin': '', 'stdout': '', 'stderr': '', 'args': ''}],
		658	+# }))
684		659
685	# ------------------------------------------------------------------------	660	# ------------------------------------------------------------------------
686	- async def correct_async(self) -> None:
687	- self.correct()	661	+ # def correct(self) -> None:
		662	+ # super().correct()
		663	+
		664	+ # if self['answer'] is None:
		665	+ # return
		666	+
		667	+ # # submit answer to JOBE server
		668	+ # resource = '/jobe/index.php/restapi/runs/'
		669	+ # headers = {"Content-type": "application/json; charset=utf-8",
		670	+ # "Accept": "application/json"}
		671	+
		672	+ # for expected in self['correct']:
		673	+ # data_json = json.dumps({
		674	+ # 'run_spec' : {
		675	+ # 'language_id': self['language'],
		676	+ # 'sourcecode': self['answer'],
		677	+ # 'input': expected.get('stdin', ''),
		678	+ # },
		679	+ # })
		680	+
		681	+ # try:
		682	+ # connect = http.client.HTTPConnection(self['server'])
		683	+ # connect.request(
		684	+ # method='POST',
		685	+ # url=resource,
		686	+ # body=data_json,
		687	+ # headers=headers
		688	+ # )
		689	+ # response = connect.getresponse()
		690	+ # logger.debug('JOBE response status %d', response.status)
		691	+ # if response.status != 204:
		692	+ # content = response.read().decode('utf8')
		693	+ # if content:
		694	+ # result = json.loads(content)
		695	+ # connect.close()
		696	+
		697	+ # except (HTTPError, ValueError):
		698	+ # logger.error('HTTPError while connecting to JOBE server')
		699	+
		700	+ # try:
		701	+ # outcome = result['outcome']
		702	+ # except (NameError, TypeError, KeyError):
		703	+ # logger.error('Bad result returned from JOBE server: %s', result)
		704	+ # return
		705	+ # logger.debug(self._outcomes[outcome])
		706	+
		707	+
		708	+
		709	+ # if result['cmpinfo']: # compiler errors and warnings
		710	+ # self['comments'] = f'Erros de compilação:\n{result["cmpinfo"]}'
		711	+ # self['grade'] = 0.0
		712	+ # return
		713	+
		714	+ # if result['stdout'] != expected.get('stdout', ''):
		715	+ # self['comments'] = 'O output gerado é diferente do esperado.' # FIXME mostrar porque?
		716	+ # self['grade'] = 0.0
		717	+ # return
		718	+
		719	+ # self['comments'] = 'Ok!'
		720	+ # self['grade'] = 1.0
		721	+
		722	+
		723	+ # # ------------------------------------------------------------------------
		724	+ # async def correct_async(self) -> None:
		725	+ # self.correct() # FIXME there is no async correction!!!
688		726
689		727
690	# out = run_script(	728	# out = run_script(
	@@ -731,7 +769,6 @@ class QuestionInformation(Question):		@@ -731,7 +769,6 @@ class QuestionInformation(Question):
731	super().correct()	769	super().correct()
732	self['grade'] = 1.0 # always "correct" but points should be zero!	770	self['grade'] = 1.0 # always "correct" but points should be zero!
733		771
734	-
735	# ============================================================================	772	# ============================================================================
736	class QFactory():	773	class QFactory():
737	'''	774	'''
	@@ -774,7 +811,7 @@ class QFactory():		@@ -774,7 +811,7 @@ class QFactory():
774	'text-regex': QuestionTextRegex,	811	'text-regex': QuestionTextRegex,
775	'numeric-interval': QuestionNumericInterval,	812	'numeric-interval': QuestionNumericInterval,
776	'textarea': QuestionTextArea,	813	'textarea': QuestionTextArea,
777	- 'code': QuestionCode,	814	+ # 'code': QuestionCode,
778	# -- informative panels --	815	# -- informative panels --
779	'information': QuestionInformation,	816	'information': QuestionInformation,
780	'success': QuestionInformation,	817	'success': QuestionInformation,
	@@ -41,67 +41,21 @@		@@ -41,67 +41,21 @@
41	<div class="container">	41	<div class="container">
42	<div class="jumbotron">	42	<div class="jumbotron">
43	{% if t['state'] == 'FINISHED' %}	43	{% if t['state'] == 'FINISHED' %}
44	- <h1>Resultado:	44	+ <h3>Resultado:
45	<strong>{{ f'{round(t["grade"], 3)}' }}</strong>	45	<strong>{{ f'{round(t["grade"], 3)}' }}</strong>
46	- valores na escala de {{t['scale'][0]}} a {{t['scale'][1]}}.
47	- </h1>
48	- <p>O seu teste foi correctamente entregue e a nota registada.</p>
49	- <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para sair do teste</a></p>	46	+ valores na escala [{{t['scale'][0]}},{{t['scale'][1]}}].
		47	+ </h3>
50	{% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}	48	{% if t['grade'] - t['scale'][0] >= 0.75*(t['scale'][1] - t['scale'][0]) %}
51	<i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>	49	<i class="fas fa-thumbs-up fa-5x text-success" aria-hidden="true"></i>
52	{% end %}	50	{% end %}
		51	+ {% elif t['state'] == 'SUBMITTED' %}
		52	+ <h3>A prova foi submetida com sucesso. Vai ser corrigida mais tarde.</h3>
53	{% elif t['state'] == 'QUIT' %}	53	{% elif t['state'] == 'QUIT' %}
54	- <p>Foi registada a sua desistência da prova.</p>	54	+ <h3>Foi registada a sua desistência da prova.</h3>
55	{% end %}	55	{% end %}
56		56
		57	+ <p><a href="/logout" class="btn btn-primary btn-lg active" role="button">Clique aqui para terminar</a></p>
57	</div> <!-- jumbotron -->	58	</div> <!-- jumbotron -->
58	-
59	- <div class="card">
60	- <h5 class="card-header">
61	- Histórico de resultados
62	- </h5>
63	- <table class="table table-condensed noleftmargin">
64	- <thead>
65	- <tr>
66	- <th>Prova</th>
67	- <th>Data</th>
68	- <th>Hora</th>
69	- <th>Nota</th>
70	- </tr>
71	- </thead>
72	- <tbody>
73	- {% for g in allgrades %}
74	- <tr>
75	- <td>{{g[0]}}</td> <!-- teste -->
76	- <td>{{g[2][:10]}}</td> <!-- data -->
77	- <td>{{g[2][11:19]}}</td> <!-- hora -->
78	- <td> <!-- progress column -->
79	- <div class="progress" style="height: 20px;">
80	- <div class="progress-bar
81	- {% if g[1] - t['scale'][0] < 0.5*(t['scale'][1] - t['scale'][0]) %}
82	- bg-danger
83	- {% elif g[1] - t['scale'][0] < 0.75*(t['scale'][1] - t['scale'][0]) %}
84	- bg-warning
85	- {% else %}
86	- bg-success
87	- {% end %}
88	- "
89	- role="progressbar"
90	- aria-valuenow="{{ 100*(g[1] - t['scale'][0])/(t['scale'][1] - t['scale'][0]) }}"
91	- aria-valuemin="0"
92	- aria-valuemax="100"
93	- style="min-width: 2em; width: {{ 100*(g[1]-t['scale'][0])/(t['scale'][1]-t['scale'][0]) }}%;">
94	-
95	- {{ str(round(g[1], 1)) }}
96	-
97	- </div> <!-- progress-bar -->
98	- </div> <!-- progress -->
99	- </td> <!-- progress column -->
100	- </tr>
101	- {% end %}
102	- </tbody>
103	- </table>
104	- </div> <!-- panel -->
105	</div> <!-- container -->	59	</div> <!-- container -->
106	</body>	60	</body>
107	</html>	61	</html>