...
 
Commits (117)
*.pyc
*/__pycache__/*
VENV/*
install/docker/gargantext_lib.tar.bz2
__pycache__
/static
.env
gargantext.ini
postgrest.conf
*.log
# Gargantext backend configuration and installation
## TL;DR
Gargantext main backend is written in Python 3.5. For a basic setup you need
an up-to-date version of `pip`, `pipenv`, a PostgreSQL (>= 9.5) database,
RabbitMQ (default message broker for asynchronous tasks) and
[PostgREST](https://postgrest.com/). See below for more details.
To setup a development environment and run backend servers in DEBUG mode:
git clone -b gargantext-light ssh://git@gitlab.iscpif.fr:20022/humanities/gargantext.git gargantext-light
cd gargantext-light
make setup
# At this point you should check log directories existence and write access
# knowing that servers are run as your current user. Path of the log files
# are configured in gargantext.ini and postgrest.conf (see section
# Configuration below).
pipenv run make start
By default Django test server is running at <http://localhost:8000>, and
PostgREST at <http://localhost:3000>.
Web server `uWSGI` is needed for production, behind any good enough HTTP
server, for example `nginx`.
## Requirements
External dependencies are listed below. Python ones are automatically handled
by `pipenv` and are mainly Django, Celery, SQLAlchemy and Alembic. See
`./Pipfile` for details about used versions.
### Up-to-date pip
On Debian-like distros, `pip` is not installed by default, if you didn't do it
already:
sudo apt-get install python3-pip
Pipenv (see below) needs an up-to-date version of pip, on a Debian-like just
`apt-get upgrade`, otherwise upgrade it like so:
pip install pip --user --upgrade
### Pipenv
You will need [pipenv][1] to easily get dependencies of Gargantext. It handles
python packages and takes care of the virtualenv and environment variables.
There are various ways to install `pipenv`, see its [documentation][2] for more
insights. Here is the straightforward way:
pip install pipenv --user --upgrade
[1]: https://github.com/kennethreitz/pipenv
[2]: https://docs.pipenv.org/
### PostgreSQL
Gargantext rely on [PostgreSQL](https://www.postgresql.org/) (>= 9.5) for data
persistence. To install it on a Debian-based OS:
sudo apt-get install postgresql-9.6 postgresql-client-9.6
To setup Gargantext database:
sudo -u postgres psql -c "CREATE USER gargantua PASSWORD '<pass>' CREATEROLE BYPASSRLS"
sudo -u postgres createdb -O gargantua gargandb
### Celery
[Celery](http://www.celeryproject.org/) is used to handle asynchronous tasks,
its installation is handled by `pipenv` so you don't need to take care of it,
just note that version 3.1 is used because `djcelery` is used for django admin
integration.
Celery 3.1 documentation: <http://docs.celeryproject.org/en/3.1/>
### RabbitMQ
[RabbitMQ](http://www.rabbitmq.com/) is the default[^2] message broker of
Celery, version 3.6 is enough. Installations instructions here:
<https://www.rabbitmq.com/download.html>.
To install it on Debian:
sudo apt-get install rabbitmq-server
[^2]: It it is possible to use another broker, but RabbitMQ is the only one
supported out of the box, plus it is really simple to deploy. See
<http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html>.
### PostgREST
See <https://postgrest.com/en/v4.1/tutorials/tut0.html#step-3-install-postgrest>
for installation instructions.
### uWSGI
[uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) is the web server used by
Gargantext in production. It is supposed to run behind an HTTP server such as
nginx.
To install it on Debian:
sudo apt-get install uwsgi
## Installation
To bootstrap Gargantext environment just cd into your local Gargantext repo and
do:
make setup
Or for production (without dev dependencies and without `DEBUG` mode):
make ENVIR=prod setup
If you want to specify custom paths for configuration files (by default
`gargantext.ini` and `postgrest.conf` in current directory), use
`GARGANTEXT_CONF` and `POSTGREST_CONF` environment variable. For example:
GARGANTEXT_CONF=/etc/gargantext/gargantext.ini \
POSTGREST_CONF=/etc/gargantext/postgrest.conf make ENVIR=prod setup
If everything is going well, you now have a clean virtualenv with every
packages you need to run Gargantext, and fresh configuration files.
You can now run any command by prefixing it with `pipenv run` or by first
entering the virtualenv with `pipenv shell`. If you use `pipenv shell`, don't
forget to leave the virtualenv (`exit` or `<Ctrl-D>`) and enter it again each
time you install or uninstall packages with `pipenv` or `pip`, to avoid weird
issues with your environment.
To run Gargantext backend servers do:
pipenv run make start
If you want to run production servers in a development environment (or dev
servers in prod) just specify environment like this:
# Run production servers regardless of current environment
pipenv run make ENVIR=prod start
# Don't forget to also set ENVIR at shutdown
pipenv run make ENVIR=prod stop
## Configuration
Configuration is located in three files: `.env` (environment variables),
`gargantext.ini` and `postgrest.conf` (these file names are default values,
they can be specified in `GARGANTEXT_CONF` and `POSTGREST_CONF` env variables).
These files are automatically generated by running `make setup` (or
`make ENVIR=prod setup`), which takes care of the tedious work.
To generate configuration again, one can run `./tools/mkconf.sh -f <target>`
where `<target>` can be `dev` or `prod`. Be careful, this script will change
Django secret key and PostgREST role password, so all current tokens and
sessions will be invalidated.
### Environment variables
Configured in `.env` (loaded by `pipenv` when running `pipenv run` or
`pipenv shell`).
* `DJANGO_SETTINGS_MODULE`: python file path of the django settings module
* `GARGANTEXT_CONF`: django backend and uwsgi configuration file path
* `POSTGREST_CONF`: postgrest configuration file path
### Django backend
Configured in `GARGANTEXT_CONF` (`gargantext.ini` by default) under `[django]`
section. You can override options listed below with environment variables. For
example, to start production servers with DEBUG mode disabled from a
development environment, you can do: `DEBUG=False make ENVIR=prod start`.
See `settings.py` (full path in `DJANGO_SETTINGS_MODULE`) to understand details
about how these options are used.
* `DEBUG`: `True` | `False`
* `SECRET_KEY`: arbitrary string used by django in various ways[^1], and to
generate JSON Web Tokens, so it MUST be reflected in PostgREST configuration
* `ALLOWED_HOSTS`: space separated list of allowed hosts
* `TIME_ZONE`: see <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones>
* `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASS`: database settings
* `LOG_FILE`: django backend log file path
* `LOG_LEVEL`: `DEBUG` | `INFO` | `WARNING` | `ERROR` | `CRITICAL`
* `LOG_FORMATTER`: `simple` | `verbose`, formatters can be added in settings.py
* `TESTSERVER_PIDFILE`: pidfile for the testserver (only used by startup script)
* `CELERYD_PID_FILE`: pidfile for celery main worker
* `CELERYD_LOG_FILE`: celery log file path
* `CELERYD_LOG_LEVEL`: `DEBUG` | `INFO` | `WARNING` | `ERROR` | `CRITICAL`
[^1]: Details here: <https://stackoverflow.com/questions/15170637/effects-of-changing-djangos-secret-key#answer-15383766>.
### uWSGI
Configured in `GARGANTEXT_CONF` (`gargantext.ini` by default) under `[uwsgi]`
section.
uWSGI (used to serve django backend in production) configuration is not very
well documented. Here is a comprehensive but very harsh reference of all its
options: <https://uwsgi-docs.readthedocs.io/en/latest/Options.html>.
Here are some general options you may need to change:
* `daemonize`: log file path
* `log-reopen`: a `true` value tells uWSGI to write a new file each day
* `pidfile`: pidfile of master process
### PostgREST
Configured in `POSTGREST_CONF` (`postgrest.conf` by default).
Here are Gargantext specific options used by our startup script (NB: these
options won't be taken into account by postgrest itself):
* `pidfile`: pidfile of postgrest process
* `logfile`: log file path
For general options, see <https://postgrest.com/en/v4.3/install.html#configuration>.
## Development
### Customize dev environment
To install specific packages without messing with dependencies, just use pip.
For example, to install ipython or bpython shells locally:
pipenv run pip install ipython
pipenv run pip install bpython
### Pylint
> Pylint is a tool that checks for errors in Python code, tries to enforce a
> coding standard and looks for code smells.
See `pylintrc` for configuration. Be aware that pylint is slow to do its work,
just be patient.
To check for errors:
pipenv run pylint gargantext
To get a full report with a dependency graph:
pipenv run pylint -r y gargantext
# To review a particular module pass its folder to pylint:
pipenv run pylint -r y gargantext/backend
CELERY_INIT=./tools/init.d/gargantext-celery
POSTGREST_INIT=./tools/init.d/gargantext-postgrest
ifeq ("$(ENVIR)", "prod")
PIPENV_ARGS=
BACKEND_INIT=./tools/init.d/gargantext-uwsgi
else
ENVIR=dev
PIPENV_ARGS=--dev
BACKEND_INIT=./tools/init.d/gargantext-testserver
endif
PY_VERSION='import sys; v=sys.version_info; print("{0}.{1}".format(*v))'
.PHONY: default
default:
@echo "Please specify a target."
@echo "To setup environment: make ENVIR=$(ENVIR) setup"
@echo "To start/stop/restart/reload/check: make ENVIR=$(ENVIR) {start|stop|restart|reload|check}"
.PHONY: setup
setup: venv conf migrate
.PHONY: venv
venv: envs
@echo "• Setup virtualenv with all dependencies..."
pipenv install $(PIPENV_ARGS)
@# Put current directory in python path to be able to import gargantext
@# from scripts in sub-directories (eg. alembic revisions)
@pwd > $$(pipenv --venv)/lib/python$$($$(pipenv --py) -c $(PY_VERSION))/site-packages/gargantext.pth
@echo
.PHONY: envs
envs:
@echo "• Setup django settings module and configuration path..."
./tools/mkenvs.sh
@echo
.PHONY: migrate
migrate:
@echo "• Migrate database to latest version..."
pipenv run ./manage.py migrate
pipenv run alembic upgrade head
@echo
.PHONY: conf
conf:
@echo "• Setup gargantext configuration..."
./tools/mkconf.sh $(ENVIR)
@echo
.PHONY: checkdebian
checkdebian:
@./tools/checkdebian.sh
.PHONY: checkpipenv
checkpipenv:
@./tools/checkpipenv.sh
.PHONY: checkstartup
checkstartup: checkdebian checkpipenv
.PHONY: start
start: checkstartup
@echo "• Start gargantext servers..."
@$(BACKEND_INIT) start
@$(CELERY_INIT) start
@$(POSTGREST_INIT) start
@echo
.PHONY: stop
stop: checkstartup
@echo "• Stop gargantext servers..."
@$(BACKEND_INIT) stop
@$(CELERY_INIT) stop
@$(POSTGREST_INIT) stop
@echo
.PHONY: restart
restart: checkstartup
@echo "• Restart gargantext servers..."
@$(BACKEND_INIT) restart
@$(CELERY_INIT) restart
@$(POSTGREST_INIT) restart
@echo
.PHONY: reload
reload: checkstartup
@echo "• Reload gargantext servers..."
@$(BACKEND_INIT) reload
@$(CELERY_INIT) force-reload
@$(POSTGREST_INIT) reload
@echo
.PHONY: check
check: checkstartup
@echo "• Check gargantext servers..."
@$(BACKEND_INIT) status || true
@$(CELERY_INIT) status || true
@$(POSTGREST_INIT) status || true
@echo
.PHONY: status
status: check
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[dev-packages]
pylint = "*"
pylint-celery = "*"
pylint-django = "*"
colorama = "*"
[packages]
Django = ">=1.11,<1.12"
dateutils = "*"
celery = "==3.1.25"
SQLAlchemy = "*"
"psycopg2" = "*"
SQLAlchemy-Utils = "*"
django-celery = "*"
djangorestframework = "*"
djangorestframework-jwt = "*"
python-decouple = "*"
alembic = "*"
[requires]
python_version = "3.5"
This diff is collapsed.
# API
Be more careful about authorizations.
cf. "ng-resource".
# Taggers
Path for data used by taggers should be defined in `gargantext.constants`.
# Database
# Sharing
Here follows a brief description of how sharing could be implemented.
## Database representation
The database representation of sharing can be distributed among 4 tables:
- `persons`, of which items represent either a user or a group
- `relationships` describes the relationships between persons (affiliation
of a user to a group, contact between two users, etc.)
- `nodes` contains the projects, corpora, documents, etc. to share (they shall
inherit the sharing properties from their parents)
- `permissions` stores the relations existing between the three previously
described above: it only consists of 2 foreign keys, plus an integer
between 1 and 3 representing the level of sharing and the start date
(when the sharing has been set) and the end date (when necessary, the time
at which sharing has been removed, `NULL` otherwise)
## Python code
The permission levels should be set in `gargantext.constants`, and defined as:
```python
PERMISSION_NONE = 0 # 0b0000
PERMISSION_READ = 1 # 0b0001
PERMISSION_WRITE = 3 # 0b0011
PERMISSION_OWNER = 7 # 0b0111
```
The requests to check for permissions (or add new ones) should not be rewritten
every time. They should be "hidden" within the models:
- `Person.owns(node)` returns a boolean
- `Person.can_read(node)` returns a boolean
- `Person.can_write(node)` returns a boolean
- `Person.give_right(node, permission)` gives a right to a given user
- `Person.remove_right(node, permission)` removes a right from a given user
- `Person.get_nodes(permission[, type])` returns an iterator on the list of
nodes on which the person has at least the given permission (optional
argument: type of requested node)
- `Node.get_persons(permission[, type])` returns an iterator on the list of
users who have at least the given permission on the node (optional argument:
type of requested persons, such as `USER` or `GROUP`)
## Example
Let's imagine the `persons` table contains the following data:
| id | type | username |
|----|-------|-----------|
| 1 | USER | David |
| 2 | GROUP | C.N.R.S. |
| 3 | USER | Alexandre |
| 4 | USER | Untel |
| 5 | GROUP | I.S.C. |
| 6 | USER | Bidule |
Assume "David" owns the groups "C.N.R.S." and "I.S.C.", "Alexandre" belongs to
the group "I.S.C.", with "Untel" and "Bidule" belonging to the group "C.N.R.S.".
"Alexandre" and "David" are in contact.
The `relationships` table then contains:
| person1_id | person2_id | type |
|------------|------------|---------|
| 1 | 2 | OWNER |
| 1 | 5 | OWNER |
| 3 | 2 | MEMBER |
| 4 | 5 | MEMBER |
| 6 | 5 | MEMBER |
| 1 | 3 | CONTACT |
The `nodes` table is populated as such:
| id | type | name |
|----|----------|----------------------|
| 12 | PROJECT | My super project |
| 13 | CORPUS | A given corpus |
| 13 | CORPUS | The corpus |
| 14 | DOCUMENT | Some document |
| 15 | DOCUMENT | Another document |
| 16 | DOCUMENT | Yet another document |
| 17 | DOCUMENT | Last document |
| 18 | PROJECT | Another project |
| 19 | PROJECT | That project |
If we want to express that "David" created "My super project" (and its children)
and wants everyone in "C.N.R.S." to be able to view it, but not access it,
`permissions` should contain:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
If "David" also wanted "Alexandre" (and no one else) to view and modify "The
corpus" (and its children), we would have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
| 3 | 13 | WRITE |
If "Alexandre" created "That project" and wants "Bidule" (and no one else) to be
able to view and modify it (and its children), the table should then have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 3 | 19 | OWNER |
| 6 | 19 | WRITE |
......@@ -4,21 +4,18 @@ from sqlalchemy import engine_from_config, pool
from logging.config import fileConfig
import re
# Add projet root directory in path and setup Django...
import os
# Setup django to be able to import gargantext
import django
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'gargantext.settings')
django.setup()
# ...to be able to import gargantext.
from gargantext import settings, models
from django.conf import settings
from gargantext import models
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
config.set_main_option("sqlalchemy.url", settings.DATABASES['default']['URL'])
config.set_main_option("sqlalchemy.url",
settings.DATABASES['default']['SECRET_URL'])
# Interpret the config file for Python logging.
# This line sets up loggers basically.
......
"""Use current_user_id() as default nodes.user_id
Revision ID: 291289b47bad
Revises: 4db5dcbe4bc7
Create Date: 2017-10-09 14:58:12.992106
"""
from alembic import op
import sqlalchemy as sa
import gargantext
# revision identifiers, used by Alembic.
revision = '291289b47bad'
down_revision = '4db5dcbe4bc7'
branch_labels = None
depends_on = None
def upgrade():
op.alter_column('nodes', 'user_id',
existing_type=sa.INTEGER(),
server_default=sa.text('current_user_id()'),
existing_nullable=False)
def downgrade():
op.alter_column('nodes', 'user_id',
existing_type=sa.INTEGER(),
server_default=None,
existing_nullable=False)
"""Bootstrap access control system
Revision ID: 4db5dcbe4bc7
Revises: 73304ae9f1fb
Create Date: 2017-10-06 17:23:27.765318
"""
from alembic import op
import sqlalchemy as sa
from gargantext.util.alembic import ReplaceableObject
# revision identifiers, used by Alembic.
revision = '4db5dcbe4bc7'
down_revision = '73304ae9f1fb'
branch_labels = None
depends_on = None
# Publicly exposed schema through PostgREST
api_schema = ReplaceableObject("api")
api_nodes_view = ReplaceableObject(
"api.nodes",
"SELECT id, typename AS type, user_id, parent_id, name, date AS created, hyperdata AS data, title_abstract FROM nodes")
# Mere mortals have 'gargantext' role, admin is 'gargantua'
gargantext_role = ReplaceableObject("gargantext", "NOLOGIN")
# PostgREST authentification system; could be used without PostgREST
authenticator_role = ReplaceableObject(
"authenticator",
"LOGIN NOINHERIT PASSWORD 'CHANGEME'")
anon_role = ReplaceableObject("anon", "NOLOGIN")
roles = [gargantext_role, authenticator_role, anon_role]
grants = [
('gargantext', 'gargantua'),
# Enable login through PostgREST auth system for gargantua, anon and
# gargantext
('gargantua, anon, gargantext', 'authenticator'),
# Basic privileges for gargantext role
('CREATE, USAGE ON SCHEMA api', 'gargantext'),
('SELECT ON nodes', 'gargantext'),
('UPDATE (parent_id, name, date, hyperdata) ON nodes', 'gargantext'),
('INSERT ON nodes', 'gargantext'),
('USAGE, SELECT ON SEQUENCE nodes_id_seq', 'gargantext'),
('DELETE ON nodes', 'gargantext'),
]
current_user_id_sp = ReplaceableObject(
"current_user_id()",
"""
-- Assuming JWT and claim.user_id is set to user.id at login
-- https://stackoverflow.com/questions/2082686/how-do-i-cast-a-string-to-integer-and-have-0-in-case-of-error-in-the-cast-with-p
RETURNS integer AS $$
DECLARE
user_id INTEGER NOT NULL DEFAULT 0;
BEGIN
BEGIN
user_id := current_setting('request.jwt.claim.user_id')::int;
EXCEPTION WHEN OTHERS THEN
RAISE NOTICE 'Invalid user_id: %. Check JWT generation.',
current_setting('request.jwt.claim.user_id', TRUE);
RETURN -1;
END;
RETURN user_id;
END;
$$ LANGUAGE plpgsql""")
stored_procedures = [current_user_id_sp]
is_owner = "COALESCE(current_user_id() = user_id, FALSE)"
is_parent_owner = "COALESCE(current_user_id() = (SELECT user_id FROM nodes n WHERE id = nodes.parent_id), FALSE)"
owner_select_policy = ReplaceableObject("owner_select", "nodes", "FOR SELECT USING (%s)" % is_owner)
owner_update_policy = ReplaceableObject("owner_update", "nodes", "FOR UPDATE USING (%s)" % is_owner)
owner_insert_policy = ReplaceableObject("owner_insert", "nodes", "FOR INSERT WITH CHECK (%s)" % is_parent_owner)
owner_delete_policy = ReplaceableObject("owner_delete", "nodes", "FOR DELETE USING (%s)" % is_parent_owner)
policies = [owner_select_policy, owner_update_policy, owner_insert_policy,
owner_delete_policy]
def upgrade():
op.create_schema(api_schema)
for role in roles:
op.create_role(role)
op.create_view(api_nodes_view)
for grant in grants:
op.execute('GRANT {} TO {}'.format(*grant))
op.execute("ALTER VIEW api.nodes OWNER TO gargantext")
op.execute("ALTER TABLE nodes ENABLE ROW LEVEL SECURITY")
for sp in stored_procedures:
op.create_sp(sp)
for policy in policies:
op.create_policy(policy)
def downgrade():
for policy in policies:
op.drop_policy(policy)
for sp in stored_procedures:
op.drop_sp(sp)
op.execute("ALTER TABLE nodes DISABLE ROW LEVEL SECURITY")
for grant in grants:
op.execute('REVOKE {} FROM {}'.format(*grant))
op.drop_view(api_nodes_view)
for role in roles:
op.drop_role(role)
op.drop_schema(api_schema)
......@@ -18,6 +18,12 @@ depends_on = None
def upgrade():
op.execute("UPDATE nodes SET date = CURRENT_TIMESTAMP WHERE date IS NULL")
op.execute("UPDATE nodes SET hyperdata = '{}'::jsonb WHERE hyperdata IS NULL")
op.execute("UPDATE nodes SET name = '' WHERE name IS NULL")
op.execute("DELETE FROM nodes WHERE typename IS NULL")
op.execute("DELETE FROM nodes WHERE user_id IS NULL")
op.alter_column('nodes', 'date',
existing_type=postgresql.TIMESTAMP(timezone=True),
server_default=sa.text('CURRENT_TIMESTAMP'),
......
"""Map database access control to django auth system
Revision ID: d77f0a598ad0
Revises: 291289b47bad
Create Date: 2017-10-12 15:23:40.481825
"""
from alembic import op
import sqlalchemy as sa
from gargantext.util.alembic import ReplaceableObject
# revision identifiers, used by Alembic.
revision = 'd77f0a598ad0'
down_revision = '291289b47bad'
branch_labels = None
depends_on = None
gargandmin_role = ReplaceableObject('gargandmin')
grants = [('gargandmin', 'authenticator'),
('gargantext', 'gargandmin')]
r = "COALESCE(current_setting('request.jwt.claim.role', TRUE), current_user)"
has_perm = "{} = 'gargantua' OR current_user_id() = user_id".format(r)
has_parent_perm = "{} = 'gargantua' OR current_user_id() = (SELECT user_id FROM nodes n WHERE id = nodes.parent_id)".format(r)
policies = {
'owner_select': ReplaceableObject("owner_select", "nodes", "FOR SELECT USING (%s)" % has_perm),
'owner_update': ReplaceableObject("owner_update", "nodes", "FOR UPDATE USING (%s)" % has_perm),
'owner_insert': ReplaceableObject("owner_insert", "nodes", "FOR INSERT WITH CHECK (%s)" % has_parent_perm),
'owner_delete': ReplaceableObject("owner_delete", "nodes", "FOR DELETE USING (%s)" % has_parent_perm),
}
def upgrade():
op.create_role(gargandmin_role)
for grant in grants:
op.execute('GRANT {} TO {}'.format(*grant))
for name, policy in policies.items():
op.replace_policy(policy, replaces='4db5dcbe4bc7.{}_policy'.format(name))
def downgrade():
for name, policy in policies.items():
op.replace_policy(policy, replace_with='4db5dcbe4bc7.{}_policy'.format(name))
for grant in grants:
op.execute('REVOKE {} FROM {}'.format(*grant))
op.drop_role(gargandmin_role)
{
"directory": "static/bower_components"
}
static/bower_components/
{
"globalstrict": true,
"globals": {
"angular": false,
"describe": false,
"it": false,
"expect": false,
"beforeEach": false,
"afterEach": false,
"module": false,
"inject": false
}
}
\ No newline at end of file
set tabstop=4
set shiftwidth=4
set expandtab
set softtabstop=4
# Gargantext Annotations web application
## Install dependencies
(2016-05-11) Instead of the previously "npm-installable" structure, all the libraries are now permanently installed outside of the app 'annotations', directly in the 'static' dir of the entire gargantext project.
Initial version specifications of the dependencies are:
- "angular": "~1.2.x",
- "angular-cookies": "~1.2.x",
- "angular-loader": "~1.2.x",
- "angular-resource": "~1.2.x",
- "bootstrap": "~3.x",
- "angular-cookies": "1.2",
- "bootstrap-select": "silviomoreto/bootstrap-select#~1.7.3"
- "underscore": "1.5.2"
These dependencies are imported via annotations/main.js (js) and main.html (css).
## Directory Layout
The layout is :
```
├── __init__.py
├── README.md
├── static
│   └── annotations
│   ├── activelists.js
│   ├── app.css
│   ├── app.js
│   ├── document.js
│   ├── highlight.js
│   ├── http.js
│   ├── keyword_tpl.html
│   ├── main.js
│   ├── ngramlist.js
│   └── utils.js
├── templates
│   └── annotations
│   └── main.html
├── urls.py
└── views.py
```
# Conception and workflow documentation
## TODO : à traduire en anglais
Cette API permet d'éditer les mots-clés miamlistés ou stoplistés associé à un document affiché dans un cadre d'une page web permettant de naviguer à travers un ensemble de document d'un corpus.
### Architecture
- Templates : Django et Angular.js ?
- Communication entre les modules : évènements Angular ($emit et $broadcast)
- Pas de routage entre différentes URL, car ici une seule vue principale basée sur le template django corpus.html
- Modèle d'abstraction de données : côté client (Angular Scopes) et côté serveur (Django Model et SQLAlchemy)
- Composants : TODO lister et décrire les composants client et serveur
- Structure de l'application : organisation du client et du serveur
- Style : Bootstrap et un thème spécifique choisi pour Gargantext
- Gestion des dépendances :
- bower, npm pour le développement web et les tests côté client
- pip requirements pour le côté serveur
## Quelles actions execute l'API ?
- afficher le titre, les auteurs, le résumé, la date de publication et le corps d'un document.
- lecture des mots-clés miamlistés associés à un document (dans le texte et hors du texte).
- lecture des mots-clés stoplistés associés à un document (dans le texte et hors du texte).
- lecture des documents ayant le plus de mots-clés miamlistés associés identiques pour afficher une liste de liens vers de nouveaux documents
- lecture du groupe de mots-clés auquel appartient un mot-clé (synonymes, différentes formes)
- modification du groupe de mots-clés auquel appartient un mot-clé donné
On désigne par mot-clé un NGram.
## Schéma de l'API
Liste des endpoints
### Lecture des données
- POST '^api/nodes/(\d+)/children/queries$' : liste des NGrams d'un document avec la possibilité de filtrer par NGrams
- GET '^api/nodes$' : liste des identifiants de mots-clés filtrés par type (NGram ou autre) pour un identifiant de parent (Document ou autre)
- GET '^api/nodes/(\d+)/ngrams$': liste des termes des mots-clés associés à un Document parent, filtrés par termes
- GET ^api/nodes/(\d+)/children/metadata$ : liste des metadata d'un Node, c'est-à-dire :
- pour un document : titre, auteur, etc
- pour un NGram : stoplisté ou miamlisté ?
### Écriture des données
TODO
## Workflow
Nous nous fixons sur cette documentation et spécification de l'API
- en parallèle : développement de l'API et prototypage de l'interface
- le prototypage de l'interface peut modifier l'API si besoin
### Spécifications des fondations de l'interface
- résolutions d'écran
- browsers
- langue: english only
- SEO: aucun ?
- collaboratif : oui, les modifications d'un autre utilisateurs seront notifiées à tous les utilisateurs présent sur le même corpus de documents
- fonctionne offline ?
### Working process
- follow board is updated regularly (https://trello.com/b/96ItkDBS/gargantext-miamlists-and-stoplists)[on Trello]
- calendrier prévisionnel: TODO
- interactions entre les acteurs: emails
- git, branches : branche "elias", `git pull --rebase origin master` réguliers
- prévision des revues de code et de l'interface : TODO
### Plateforme
- Python 3.4
- Django 1.6
- Postgresql 9.3 + HSTORE
- SQLAlchemy
- Bootstrap CSS
- Angular.js
### Outils de qualité de code
- pylint
- jshint (voir .jshintrc)
- indentations : 4 espaces (voir .lvimrc)
- nettoyage automatique des espaces en fin de ligne
## Tests
There are two kinds of tests possible : Unit tests and End to End tests.
- côté client : étudier karma.js et protractor
- définir la stratégie de tests : TODO
## Déploiement
- définir le processus de déploiement
- prévoir un système de monitoring des erreurs du serveur une fois en ligne
- Sentry ?
ANNOTATIONS
===========
2016-01
## Routines de manipulation de ngrams dans les listes
#### Trajectoire globale des actions choisies
1. angular: ngramlist.js (user input) or highlight.js (user menu controller)
2. angular: http.js configuration object
{ 'action': 'post', 'listId': miamlist_id, ..}
3. AJAX POST/DELETE
4. "API locale" (=> annotations.views)
5. DB insert/delete
Remarque:
Dans le code annotations d'Elias, il y a une "API locale" qui transmet les actions client vers le serveur.
=> l'interconnexion est configurée pour angular dans annotations/static/annotations/app.js qui lance son propre main sur la fenêtre en prenant les paramètres depuis l'url et en s'isolant de django
=> les routes amont sont définies pour django dans annotations.urls et reprises pour angular dans http.js
#### Par ex: l'étape AJAX pour suppression
`curl -XDELETE http://localhost:8000/annotations/lists/7129/ngrams/4497`
via annotations.views.NgramEdit.as_view())
#### ajout d'un ngram
```
curl -XPOST http://localhost:8000/annotations/lists/1866/ngrams/create \
-H "Content-Type: application/json" \
-d '{"text":"yooooooooo"}' > response_to_ngrams_create.html
```
## Points d'interaction côté client (GUI)
Add Ngram via dialog box :
- controller:
ngramlist.annotationsAppNgramList.controller('NgramInputController')
- effect:
1. NgramHttpService.post()
Add Ngram via select new + menu
- controller:
highlight.annotationsAppHighlight.controller('TextSelectionMenuController')