Miguel Barão / aprendizations

Download as
Browse Code »

Commit a203d3ccc52a4cd6d46aeff2efc28ebd161b1f87

Authored by Miguel Barão
1 parent 9161ff75
Exists in master and in 1 other branch dev

- new http server redirects all traffic to a given (https) server. 

- documentation updates to python3.7.
- started to make changes to allow multiple configuration files.
Showing 6 changed files with 228 additions and 126 deletions   Show diff stats
README.md
  Diff comments   View file @a203d3c
... ... @@ -5,23 +5,23 @@
5 5  
6 6 Before installing the server, we will need to install python with some additional packages.
7 7  
8   -### Install python3.6 with sqlite3 support
  8 +### Install python3.7 with sqlite3 support
9 9  
10 10 This can be done using the system package management, downloaded from [http://www.python.org](), or compiled from sources.
11 11  
12 12 - Installing from the system package manager:
13 13 ```sh
14   - sudo port install python36 # MacOS
15   - sudo pkg install python36 py36-sqlite3 # FreeBSD
  14 + sudo port install python37 # MacOS
  15 + sudo pkg install python37 py37-sqlite3 # FreeBSD
16 16 sudo apt install ?not available yet? # Linux, install from source
17 17 ```
18 18 - Installing from source:
19 19 Download from [http://www.python.org]() and
20 20  
21 21 ```sh
22   - unxz Python-3.6.tar.xz
23   - tar xvf Python-3.6.tar
24   - cd Python-3.6
  22 + unxz Python-3.7.tar.xz
  23 + tar xvf Python-3.7.tar
  24 + cd Python-3.7
25 25 ./configure --prefix=$HOME/.local/bin
26 26 make && make install
27 27 ```
... ... @@ -33,7 +33,7 @@ This can be done using the system package management, downloaded from [http://ww
33 33 If the `pip` command is not yet installed, run
34 34  
35 35 ```sh
36   -python3.6 -m ensurepip --user
  36 +python3.7 -m ensurepip --user
37 37 ```
38 38  
39 39 This will install pip in your user account under `~/.local/bin`.
... ... @@ -54,15 +54,15 @@ pip3 install --user \
54 54 sqlalchemy \
55 55 pyyaml \
56 56 pygments \
57   - markdown \
  57 + mistune \
58 58 bcrypt \
59 59 networkx
60 60 ```
61 61  
62 62 These are usually installed under
63 63  
64   -- `~/.local/lib/python3.6/site-packages/` in Linux/FreeBSD.
65   -- `~/Library/python/3.6/lib/python/site-packages/` in MacOS.
  64 +- `~/.local/lib/python3.7/site-packages/` in Linux/FreeBSD.
  65 +- `~/Library/python/3.7/lib/python/site-packages/` in MacOS.
66 66  
67 67 ## Installation
68 68  
... ... @@ -82,18 +82,19 @@ The user data is maintained in a sqlite3 database file. We first need to create
82 82  
83 83 ```sh
84 84 cd aprendizations
85   -./initdb.py --help # for the available options
86 85 ./initdb.py # show current database or initialize empty if nonexisting
87 86 ./initdb.py inscricoes.csv # add students from CSV, passwords are the numbers
88   -./initdb.py --add 1184 "Ana Bola" # add new user (password=1184)
89   -./initdb.py --update 1184 --pw alibaba # update password
  87 +./initdb.py --add 1184 "Aladino da Silva" # add new user (default password=1184)
  88 +./initdb.py --update 1184 --pw alibaba # update password of given student
  89 +./initdb.py --help # for the available options
90 90 ```
91 91  
92 92 ### SSL Certificates
93 93  
94 94 We need certificates for https. Certificates can be self-signed or validated by a trusted authority.
95 95  
96   -Self-signed can be used for testing, but browsers will complain. LetsEncrypt issues trusted and free certificates, but the server must have a fixed IP and a registered domain name in the DNS (dynamic DNS does not work).
  96 +Self-signed can be used locally for development and testing, but browsers will complain.
  97 +LetsEncrypt issues trusted and free certificates, but the server must have a registered publicly accessible domain name.
97 98  
98 99 #### Selfsigned
99 100  
... ... @@ -109,7 +110,7 @@ openssl req -x509 -newkey rsa:4096 -keyout privkey.pem -out cert.pem -days 365 -
109 110 sudo pkg install py27-certbot # FreeBSD
110 111 ```
111 112  
112   -Shutdown the firewall and any server running. Then run the script to generate the certificate:
  113 +Shutdown the firewall and any web server that might be running. Then run the script to generate the certificate:
113 114  
114 115 ```sh
115 116 sudo service pf stop # disable pf firewall (FreeBSD)
... ... @@ -144,7 +145,7 @@ The application includes a small example in `demo/demo.yaml`. Run it with
144 145 ./serve.py demo/demo.yaml
145 146 ```
146 147  
147   -and open a browser at [https://127.0.0.1:8443](). If it everything looks good, check at the correct address `https://www.example.com` (requires port forward in the firewall).
  148 +and open a browser at [https://127.0.0.1:8443](). If it everything looks good, check at the correct address `https://www.example.com` (requires port forward in the firewall). The option `--debug` provides more verbose logging and might be useful during testing.
148 149  
149 150  
150 151 ### Firewall configuration
... ... @@ -178,6 +179,7 @@ pflog_logfile="/var/log/pflog"
178 179  
179 180 Reboot or `sudo service pf start`.
180 181  
  182 +
181 183 ## Troubleshooting
182 184  
183 185 To help with troubleshooting, use the option `--debug` when running the server. This will increase logs in the terminal and will present the python exception errors in the browser.
... ...
demo/demo.yaml
  Diff comments   View file @a203d3c
1 1 title: Example
2 2 database: students.db
  3 +path: ./demo
3 4  
4   -# default global values
  5 +# values applie to each topic, if undefined there
  6 +# default values are: file=question.yaml, shuffle=True, choose: all
5 7 file: questions.yaml
6 8 shuffle: True
7 9 choose: 6
8   -
9   -# path prefix applied to the topics
10   -path: ./demo
  10 +forgetting_factor: 0.99
11 11  
12 12 # Representation of the edges of the dependency graph.
13 13 # Example: A depends on B and C
... ...
factory.py
  Diff comments   View file @a203d3c
... ... @@ -38,6 +38,14 @@ from questions import QuestionInformation, QuestionRadio, QuestionCheckbox, Ques
38 38 logger = logging.getLogger(__name__)
39 39  
40 40  
  41 +# Topic Factory
  42 +# Generates a list
  43 +# class RunningTopic(object):
  44 +# def __init__(self, graph, topic):
  45 +
  46 +
  47 +
  48 +
41 49  
42 50 # ===========================================================================
43 51 # Question Factory
... ... @@ -59,7 +67,7 @@ class QFactory(object):
59 67 'success' : QuestionInformation,
60 68 }
61 69  
62   - def __init__(self, question_dict):
  70 + def __init__(self, question_dict={}):
63 71 self.question = question_dict
64 72  
65 73 # -----------------------------------------------------------------------
... ... @@ -86,7 +94,7 @@ class QFactory(object):
86 94 try:
87 95 qinstance = self._types[q['type']](q) # instance with correct class
88 96 except KeyError as e:
89   - logger.error(f'Failed to generate question "{q["ref"]}"')
  97 + logger.error(f'Unknown type "{q["type"]}" in question "{q["ref"]}"')
90 98 raise e
91 99 else:
92 100 return qinstance
... ...
http-redirect.py 0 → 100755
  Diff comments   View file @a203d3c
... ... @@ -0,0 +1,87 @@
  1 +#!/usr/bin/env python3
  2 +
  3 +# python standard library
  4 +from os import path
  5 +import sys
  6 +import argparse
  7 +import signal
  8 +
  9 +# user installed libraries
  10 +from tornado import ioloop, web, httpserver
  11 +
  12 +
  13 +# ============================================================================
  14 +# WebApplication - Tornado Web Server
  15 +# ============================================================================
  16 +class WebRedirectApplication(web.Application):
  17 + def __init__(self, target='https://localhost'):
  18 + handlers = [
  19 + (r'/*', RootHandler), # redirect to https
  20 + ]
  21 + super().__init__(handlers)
  22 + self.target = target
  23 +
  24 +# ============================================================================
  25 +# Handlers
  26 +# ============================================================================
  27 +class RootHandler(web.RequestHandler):
  28 + SUPPORTED_METHODS = ['GET']
  29 +
  30 + def get(self):
  31 + print('Redirecting...')
  32 + self.redirect(self.application.target)
  33 +
  34 +
  35 +# -------------------------------------------------------------------------
  36 +def signal_handler(signal, frame):
  37 + r = input(' --> Stop webserver? (yes/no) ').lower()
  38 + if r == 'yes':
  39 + ioloop.IOLoop.current().stop()
  40 + print('Webserver stopped.')
  41 + sys.exit(0)
  42 +
  43 +
  44 +# -------------------------------------------------------------------------
  45 +# Tornado web server
  46 +# ----------------------------------------------------------------------------
  47 +def main():
  48 + # --- Commandline argument parsing
  49 + argparser = argparse.ArgumentParser(
  50 + description='Redirection server. Any request to http will be redirected to a target server provided in the command line argument.')
  51 + argparser.add_argument('target', type=str,
  52 + help='Target https server address, e.g. https://www.example.com.')
  53 + argparser.add_argument('--port', type=int, default=8080,
  54 + help='Port to which this server will listen to, e.g. 8080')
  55 + arg = argparser.parse_args()
  56 +
  57 + # --- create web redirect application
  58 + try:
  59 + redirectapp = WebRedirectApplication(target=arg.target)
  60 + except Exception as e:
  61 + print('Failed to start web redirect application.')
  62 + raise e
  63 +
  64 + # --- create webserver
  65 + try:
  66 + http_server = httpserver.HTTPServer(redirectapp)
  67 + except Exception as e:
  68 + print('Failed to create HTTP server.')
  69 + raise e
  70 + else:
  71 + http_server.listen(8080)
  72 +
  73 + # --- run webserver
  74 + print(f'Redirecting port {arg.port} to {arg.target} (Ctrl-C to stop)')
  75 + signal.signal(signal.SIGINT, signal_handler)
  76 +
  77 + try:
  78 + ioloop.IOLoop.current().start() # running...
  79 + except Exception as e:
  80 + print('Redirection stopped!')
  81 + ioloop.IOLoop.current().stop()
  82 + raise e
  83 +
  84 +
  85 +# ----------------------------------------------------------------------------
  86 +if __name__ == "__main__":
  87 + main()
... ...
knowledge.py
  Diff comments   View file @a203d3c
... ... @@ -17,7 +17,7 @@ logger = logging.getLogger(__name__)
17 17 # kowledge state of each student....??
18 18 # Contains:
19 19 # state - dict of topics with state of unlocked topics
20   -# deps - dependency graph
  20 +# deps - access to dependency graph shared between students
21 21 # topic_sequence - list with the order of recommended topics
22 22 # ----------------------------------------------------------------------------
23 23 class StudentKnowledge(object):
... ... @@ -28,12 +28,11 @@ class StudentKnowledge(object):
28 28 self.deps = deps # dependency graph shared among students
29 29 self.state = state # {'topic': {'level':0.5, 'date': datetime}, ...}
30 30  
31   - self.update_topic_levels() # forgetting factor
  31 + self.update_topic_levels() # applies forgetting factor
32 32 self.topic_sequence = self.recommend_topic_sequence() # ['a', 'b', ...]
33   - self.unlock_topics() # whose dependencies have been done
  33 + self.unlock_topics() # whose dependencies have been completed
34 34 self.current_topic = None
35   - print(deps.graph)
36   - self.MAX_QUESTIONS = deps.graph['config'].get('choose', 10)
  35 + # self.MAX_QUESTIONS = deps.graph['config'].get('choose', None)
37 36  
38 37 # ------------------------------------------------------------------------
39 38 # Updates the proficiency levels of the topics, with forgetting factor
... ... @@ -67,18 +66,13 @@ class StudentKnowledge(object):
67 66  
68 67  
69 68 # ------------------------------------------------------------------------
70   - # Start a new topic. If not provided, gets a recommendation.
  69 + # Start a new topic.
71 70 # questions: list of generated questions to do in the topic
72 71 # current_question: the current question to be presented
73 72 # ------------------------------------------------------------------------
74   - def init_topic(self, topic=''):
  73 + def init_topic(self, topic):
75 74 logger.debug(f'StudentKnowledge.init_topic({topic})')
76 75  
77   - # maybe get topic recommendation
78   - if not topic:
79   - topic = self.get_recommended_topic()
80   - logger.debug(f'Recommended topic is {topic}')
81   -
82 76 # do not allow locked topics
83 77 if self.is_locked(topic):
84 78 logger.debug(f'Topic {topic} is locked')
... ... @@ -105,23 +99,6 @@ class StudentKnowledge(object):
105 99 # get first question
106 100 self.next_question()
107 101  
108   -
109   - # def init_learning(self, topic=''):
110   - # logger.debug(f'StudentKnowledge.init_learning({topic})')
111   -
112   - # if self.is_locked(topic):
113   - # logger.debug(f'Topic {topic} is locked')
114   - # return False
115   -
116   - # self.current_topic = topic
117   - # factory = self.deps.node[topic]['factory']
118   - # lesson = self.deps.node[topic]['lesson']
119   -
120   - # self.questions = [factory[qref].generate() for qref in lesson]
121   - # logger.debug(f'Total: {len(self.questions)} questions')
122   -
123   - # self.next_question_in_lesson()
124   -
125 102 # ------------------------------------------------------------------------
126 103 # The topic has finished and there are no more questions.
127 104 # The topic level is updated in state and unlocks are performed.
... ... @@ -204,6 +181,11 @@ class StudentKnowledge(object):
204 181 # pure functions of the state (no side effects)
205 182 # ========================================================================
206 183  
  184 + def get_max_questions(self, topic=None):
  185 + if topic is not None:
  186 + node = self.deps.nodes[topic]
  187 + max_questions = node.get('choose', len(node['questions']))
  188 +
207 189  
208 190 # ------------------------------------------------------------------------
209 191 # compute recommended sequence of topics ['a', 'b', ...]
... ... @@ -251,6 +233,6 @@ class StudentKnowledge(object):
251 233 # ------------------------------------------------------------------------
252 234 # Recommends a topic to practice/learn from the state.
253 235 # ------------------------------------------------------------------------
254   - def get_recommended_topic(self): # FIXME untested
255   - return min(self.state.items(), key=lambda x: x[1]['level'])[0]
  236 + # def get_recommended_topic(self): # FIXME untested
  237 + # return min(self.state.items(), key=lambda x: x[1]['level'])[0]
256 238  
... ...
learnapp.py
  Diff comments   View file @a203d3c
... ... @@ -57,12 +57,14 @@ class LearnApp(object):
57 57 session.close()
58 58  
59 59 # ------------------------------------------------------------------------
60   - def __init__(self, config_file):
61   - # state of online students
62   - self.online = {}
63   - config = load_yaml(config_file[0])
64   - self.db_setup(config['database']) # setup and check students
65   - self.deps = build_dependency_graph(config)
  60 + def __init__(self, config_files):
  61 + self.db_setup() # setup database and check students
  62 + self.online = dict() # online students
  63 +
  64 + self.deps = nx.DiGraph()
  65 + for c in config_files:
  66 + self.populate_graph(c)
  67 +
66 68 self.db_add_missing_topics(self.deps.nodes())
67 69  
68 70 # ------------------------------------------------------------------------
... ... @@ -190,7 +192,6 @@ class LearnApp(object):
190 192 loop = asyncio.get_running_loop()
191 193 try:
192 194 await loop.run_in_executor(None, self.online[uid]['state'].init_topic, topic)
193   - # self.online[uid]['state'].init_topic(topic)
194 195 except KeyError as e:
195 196 logger.warning(f'User "{uid}" tried to open nonexistent topic: "{topic}"')
196 197 raise e
... ... @@ -200,10 +201,10 @@ class LearnApp(object):
200 201 # ------------------------------------------------------------------------
201 202 # Fill db table 'Topic' with topics from the graph if not already there.
202 203 # ------------------------------------------------------------------------
203   - def db_add_missing_topics(self, nn):
  204 + def db_add_missing_topics(self, topics):
204 205 with self.db_session() as s:
205   - tt = [t[0] for t in s.query(Topic.id)] # db list of topics
206   - missing_topics = [Topic(id=n) for n in nn if n not in tt]
  206 + dbtopics = [t[0] for t in s.query(Topic.id)] # get topics from DB
  207 + missing_topics = [Topic(id=t) for t in topics if t not in dbtopics]
207 208 if missing_topics:
208 209 s.add_all(missing_topics)
209 210 logger.info(f'Added {len(missing_topics)} new topics to the database')
... ... @@ -211,7 +212,7 @@ class LearnApp(object):
211 212 # ------------------------------------------------------------------------
212 213 # setup and check database
213 214 # ------------------------------------------------------------------------
214   - def db_setup(self, db):
  215 + def db_setup(self, db='students.db'):
215 216 logger.info(f'Checking database "{db}":')
216 217 engine = create_engine(f'sqlite:///{db}', echo=False)
217 218 self.Session = sessionmaker(bind=engine)
... ... @@ -260,7 +261,7 @@ class LearnApp(object):
260 261  
261 262 # ------------------------------------------------------------------------
262 263 def get_title(self):
263   - return self.deps.graph['title']
  264 + return self.deps.graph.get('title', '') # FIXME
264 265  
265 266 # ------------------------------------------------------------------------
266 267 def get_topic_name(self, ref):
... ... @@ -278,64 +279,86 @@ class LearnApp(object):
278 279  
279 280  
280 281  
281   -# ============================================================================
282   -# Builds a digraph.
283   -#
284   -# First, topics such as `computer/mips/exceptions` are added as nodes
285   -# together with dependencies. Then, questions are loaded to a factory.
286   -#
287   -# g.graph['path'] base path where topic directories are located
288   -# g.graph['title'] title defined in the configuration YAML
289   -# g.graph['database'] sqlite3 database file to use
290   -#
291   -# Nodes are the topic references e.g. 'my/topic'
292   -# g.node['my/topic']['name'] name of the topic
293   -# g.node['my/topic']['questions'] list of question refs defined in YAML
294   -# g.node['my/topic']['factory'] dict with question factories
295   -#
296   -# Edges are obtained from the deps defined in the YAML file for each topic.
297   -# ----------------------------------------------------------------------------
298   -def build_dependency_graph(config={}):
299   - logger.info('Building graph and loading questions:')
300   -
301   - # create graph
302   - prefix = config.get('path', '.')
303   - title = config.get('title', '')
304   - database = config.get('database', 'students.db')
305   - g = nx.DiGraph(path=prefix, title=title, database=database, config=config)
306   -
307   - # iterate over topics and build graph
308   - topics = config.get('topics', {})
309   - tcount = qcount = 0 # topic and question counters
310   - for ref,attr in topics.items():
311   - g.add_node(ref)
312   - tnode = g.node[ref] # current node (topic)
313   -
314   - if isinstance(attr, dict):
315   - tnode['type'] = attr.get('type', 'topic')
316   - tnode['name'] = attr.get('name', ref)
317   - tnode['questions'] = attr.get('questions', [])
318   - g.add_edges_from((d,ref) for d in attr.get('deps', []))
319   -
320   - fullpath = path.expanduser(path.join(prefix, ref))
321   -
322   - # load questions
323   - filename = path.join(fullpath, 'questions.yaml')
324   - loaded_questions = load_yaml(filename, default=[]) # list
325   - if not tnode['questions']:
326   - # if questions not in configuration then load all, preserving order
327   - tnode['questions'] = [q.setdefault('ref', f'{ref}:{i}') for i,q in enumerate(loaded_questions)]
328   -
329   - # make questions factory (without repeating same question)
330   - tnode['factory'] = {}
331   - for q in loaded_questions:
332   - if q['ref'] in tnode['questions']:
333   - q['path'] = fullpath # fullpath added to each question
334   - tnode['factory'][q['ref']] = QFactory(q)
335   -
336   - logger.info(f'{len(tnode["questions"]):6} {ref}')
337   - qcount += len(tnode["questions"]) # count total questions
338   - tcount += 1
339   -
340   - logger.info(f'Total loaded: {tcount} topics, {qcount} questions')
341   - return g
  282 + # ============================================================================
  283 + # Populates a digraph.
  284 + #
  285 + # First, topics such as `computer/mips/exceptions` are added as nodes
  286 + # together with dependencies. Then, questions are loaded to a factory.
  287 + #
  288 + # g.graph['path'] base path where topic directories are located
  289 + # g.graph['title'] title defined in the configuration YAML
  290 + # g.graph['database'] sqlite3 database file to use
  291 + #
  292 + # Nodes are the topic references e.g. 'my/topic'
  293 + # g.node['my/topic']['name'] name of the topic
  294 + # g.node['my/topic']['questions'] list of question refs defined in YAML
  295 + # g.node['my/topic']['factory'] dict with question factories
  296 + #
  297 + # Edges are obtained from the deps defined in the YAML file for each topic.
  298 + # ----------------------------------------------------------------------------
  299 + def populate_graph(self, conffile):
  300 + logger.info(f'Loading {conffile} and populating graph:')
  301 + g = self.deps # the graph
  302 + config = load_yaml(conffile) # course configuration
  303 +
  304 + # set attributes of the graph
  305 + prefix = path.expanduser(config.get('path', '.'))
  306 + # title = config.get('title', '')
  307 + # database = config.get('database', 'students.db')
  308 +
  309 + # default attributes that apply to the topics
  310 + default_file = config.get('file', 'questions.yaml')
  311 + default_shuffle = config.get('shuffle', True)
  312 + default_choose = config.get('choose', 9999)
  313 + default_forgetting_factor = config.get('forgetting_factor', 1.0)
  314 +
  315 + # create graph
  316 + # g = nx.DiGraph(path=prefix, title=title, database=database, config=config)
  317 +
  318 +
  319 + # iterate over topics and populate graph
  320 + topics = config.get('topics', {})
  321 + tcount = qcount = 0 # topic and question counters
  322 + for tref, attr in topics.items():
  323 + if tref in g:
  324 + logger.error(f'--> Topic {tref} already exists. Skipped.')
  325 + continue
  326 +
  327 + # add topic to the graph
  328 + g.add_node(tref)
  329 + t = g.node[tref] # current topic node
  330 +
  331 + topicpath = path.join(prefix, tref)
  332 +
  333 + t['type'] = attr.get('type', 'topic')
  334 + t['name'] = attr.get('name', tref)
  335 + t['path'] = topicpath
  336 + t['file'] = attr.get('file', default_file)
  337 + t['shuffle'] = attr.get('shuffle', default_shuffle)
  338 + t['forgetting_factor'] = attr.get('forgetting_factor', default_forgetting_factor)
  339 + g.add_edges_from((d,tref) for d in attr.get('deps', []))
  340 +
  341 + # load questions as list of dicts
  342 + questions = load_yaml(path.join(topicpath, t['file']), default=[])
  343 +
  344 + # if questions are left undefined, include all.
  345 + # refs undefined in questions.yaml are set to topic:n
  346 + t['questions'] = attr.get('questions',
  347 + [q.setdefault('ref', f'{tref}:{i}') for i, q in enumerate(questions)])
  348 +
  349 + # topic will generate a certain amount of questions
  350 + t['choose'] = min(attr.get('choose', default_choose), len(t['questions']))
  351 +
  352 + # make questions factory (without repeating same question) FIXME move to somewhere else?
  353 + t['factory'] = {}
  354 + for q in questions:
  355 + if q['ref'] in t['questions']:
  356 + q['path'] = topicpath # fullpath added to each question
  357 + t['factory'][q['ref']] = QFactory(q)
  358 +
  359 + logger.info(f'{len(t["questions"]):6} {tref}')
  360 + qcount += len(t["questions"]) # count total questions
  361 + tcount += 1
  362 +
  363 + logger.info(f'Total loaded: {tcount} topics, {qcount} questions')
  364 +
... ...