...
 
Commits (81)
*.pyc
*/__pycache__/*
VENV/*
install/docker/gargantext_lib.tar.bz2
__pycache__
/static
.env
gargantext.ini
postgrest.conf
This diff is collapsed.
# Gargantext installation and dev environment
## TL;DR
You need `pipenv`, an up-to-date version of `pip`, and a PostgreSQL (>= 9.5)
database. To setup development environment and run test server:
$ git clone -b gargantext-light ssh://git@gitlab.iscpif.fr:20022/humanities/gargantext.git gargantext-light
$ cd gargantext-light
$ make
$ pipenv run ./manage runserver
## Requirements
### Up-to-date pip
On Debian-like distros, `pip` is not installed by default, if you didn't do it
already:
$ sudo apt install python3-pip
Pipenv (see below) needs an up-to-date version of pip, on a Debian-like just
`apt upgrade`, otherwise upgrade it like so:
$ pip install pip --user --upgrade
### Pipenv
You will need [pipenv][1] to easily get dependencies of Gargantext.
It handles packages and takes care of the virtualenv and environment variables.
There are various ways to install `pipenv`, see its [documentation][2] for more
insights. Here is the straightforward way:
$ pip install pipenv --user
If you use `pipenv shell`, don't forget to leave the virtualenv (`exit` or
`<Ctrl-D>`) and enter it again each time you install or uninstall packages with
`pipenv` or `pip`, to avoid weird issues with your environment.
[1]: https://github.com/kennethreitz/pipenv
[2]: https://docs.pipenv.org/
## PostgreSQL
Gargantext rely on PostgreSQL (>= 9.5) for data persistence. To setup
database on a Debian-based OS:
$ sudo apt-get install postgresql-9.6 postgresql-client-9.6
$ sudo -u postgres psql -c "CREATE USER gargantua PASSWORD '<pass>' CREATEROLE BYPASSRLS"
$ sudo -u postgres createdb -O gargantua gargandb
## Installation
To bootstrap Gargantext environment just cd into your local Gargantext repo and
do:
$ make
Or for production (without dev dependencies and without DEBUG mode):
$ make TARGET=prod
To specify a path for configuration file (by default gargantext.ini in current
directory), use GARGANTEXT_CONF environment variable. For example:
$ GARGANTEXT_CONF=/etc/gargantext/gargantext.ini make TARGET=prod
You can specify path of PostgREST configuration the same way by setting
POSTGREST_CONF environment variable.
If everything is going well, you now have a clean virtualenv with every
packages you need to run Gargantext, and a fresh configuration file.
You can now run any command by prefixing it with `pipenv run` or by first
entering the virtualenv with `pipenv shell`. To run Gargantext django backend
test server you can do:
$ pipenv run ./manage.py runserver
## Customize dev environment
To install specific packages without messing with dependencies, just use pip.
For example, to install ipython or bpython shells locally:
$ pipenv run pip install ipython
$ pipenv run pip install bpython
.PHONY: gargantext venv envs migrate conf
ifeq ($(TARGET), "prod")
TARG=prod
PIPENV_ARGS=
else
TARG=dev
PIPENV_ARGS=--dev
endif
PY_VERSION='import sys; v=sys.version_info; print("{0}.{1}".format(*v))'
gargantext: venv conf migrate
venv: envs
@echo "• Setup virtualenv with all dependencies..."
pipenv install $(PIPENV_ARGS)
@# Put current directory in python path to be able to import gargantext
@# from scripts in sub-directories (eg. alembic revisions)
@pwd > $$(pipenv --venv)/lib/python$$($$(pipenv --py) -c $(PY_VERSION))/site-packages/gargantext.pth
@echo
envs:
@echo "• Setup django settings module and configuration path..."
./tools/mkenvs.sh
@echo
migrate:
@echo "• Migrate database to latest version..."
pipenv run ./manage.py migrate
pipenv run alembic upgrade head
@echo
conf:
@echo "• Setup gargantext configuration..."
./tools/mkconf.sh $(TARG)
@echo
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[dev-packages]
pylint = "*"
pylint-celery = "*"
pylint-django = "*"
colorama = "*"
[packages]
Django = ">=1.11,<1.12"
dateutils = "*"
celery = "==3.1.25"
SQLAlchemy = "*"
"psycopg2" = "*"
SQLAlchemy-Utils = "*"
django-celery = "*"
djangorestframework = "*"
djangorestframework-jwt = "*"
python-decouple = "*"
alembic = "*"
[requires]
python_version = "3.5"
This diff is collapsed.
......@@ -37,10 +37,7 @@ of security.
The core team at the origin of Gargantext is :
* David Chavalarias, principal investigator
* Alexandre Delanoë, project manager
* Samuel Castillo J., developer
* Romain Loth, developer
* Mathieu Rodic, developer
* Constance de Quatrebarbes, developer
* Simon Murail, developer
# Host institutions of this project
* CNRS labs ISC-PIF (Institut des Systèmes Complexes de Paris Île de France)
......
# API
Be more careful about authorizations.
cf. "ng-resource".
# Taggers
Path for data used by taggers should be defined in `gargantext.constants`.
# Database
# Sharing
Here follows a brief description of how sharing could be implemented.
## Database representation
The database representation of sharing can be distributed among 4 tables:
- `persons`, of which items represent either a user or a group
- `relationships` describes the relationships between persons (affiliation
of a user to a group, contact between two users, etc.)
- `nodes` contains the projects, corpora, documents, etc. to share (they shall
inherit the sharing properties from their parents)
- `permissions` stores the relations existing between the three previously
described above: it only consists of 2 foreign keys, plus an integer
between 1 and 3 representing the level of sharing and the start date
(when the sharing has been set) and the end date (when necessary, the time
at which sharing has been removed, `NULL` otherwise)
## Python code
The permission levels should be set in `gargantext.constants`, and defined as:
```python
PERMISSION_NONE = 0 # 0b0000
PERMISSION_READ = 1 # 0b0001
PERMISSION_WRITE = 3 # 0b0011
PERMISSION_OWNER = 7 # 0b0111
```
The requests to check for permissions (or add new ones) should not be rewritten
every time. They should be "hidden" within the models:
- `Person.owns(node)` returns a boolean
- `Person.can_read(node)` returns a boolean
- `Person.can_write(node)` returns a boolean
- `Person.give_right(node, permission)` gives a right to a given user
- `Person.remove_right(node, permission)` removes a right from a given user
- `Person.get_nodes(permission[, type])` returns an iterator on the list of
nodes on which the person has at least the given permission (optional
argument: type of requested node)
- `Node.get_persons(permission[, type])` returns an iterator on the list of
users who have at least the given permission on the node (optional argument:
type of requested persons, such as `USER` or `GROUP`)
## Example
Let's imagine the `persons` table contains the following data:
| id | type | username |
|----|-------|-----------|
| 1 | USER | David |
| 2 | GROUP | C.N.R.S. |
| 3 | USER | Alexandre |
| 4 | USER | Untel |
| 5 | GROUP | I.S.C. |
| 6 | USER | Bidule |
Assume "David" owns the groups "C.N.R.S." and "I.S.C.", "Alexandre" belongs to
the group "I.S.C.", with "Untel" and "Bidule" belonging to the group "C.N.R.S.".
"Alexandre" and "David" are in contact.
The `relationships` table then contains:
| person1_id | person2_id | type |
|------------|------------|---------|
| 1 | 2 | OWNER |
| 1 | 5 | OWNER |
| 3 | 2 | MEMBER |
| 4 | 5 | MEMBER |
| 6 | 5 | MEMBER |
| 1 | 3 | CONTACT |
The `nodes` table is populated as such:
| id | type | name |
|----|----------|----------------------|
| 12 | PROJECT | My super project |
| 13 | CORPUS | A given corpus |
| 13 | CORPUS | The corpus |
| 14 | DOCUMENT | Some document |
| 15 | DOCUMENT | Another document |
| 16 | DOCUMENT | Yet another document |
| 17 | DOCUMENT | Last document |
| 18 | PROJECT | Another project |
| 19 | PROJECT | That project |
If we want to express that "David" created "My super project" (and its children)
and wants everyone in "C.N.R.S." to be able to view it, but not access it,
`permissions` should contain:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
If "David" also wanted "Alexandre" (and no one else) to view and modify "The
corpus" (and its children), we would have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
| 3 | 13 | WRITE |
If "Alexandre" created "That project" and wants "Bidule" (and no one else) to be
able to view and modify it (and its children), the table should then have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 3 | 19 | OWNER |
| 6 | 19 | WRITE |
......@@ -2,7 +2,7 @@
[alembic]
# path to migration scripts
script_location = alembic
script_location = gargalembic
# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s
......
{
"directory": "static/bower_components"
}
static/bower_components/
{
"globalstrict": true,
"globals": {
"angular": false,
"describe": false,
"it": false,
"expect": false,
"beforeEach": false,
"afterEach": false,
"module": false,
"inject": false
}
}
\ No newline at end of file
set tabstop=4
set shiftwidth=4
set expandtab
set softtabstop=4
# Gargantext Annotations web application
## Install dependencies
(2016-05-11) Instead of the previously "npm-installable" structure, all the libraries are now permanently installed outside of the app 'annotations', directly in the 'static' dir of the entire gargantext project.
Initial version specifications of the dependencies are:
- "angular": "~1.2.x",
- "angular-cookies": "~1.2.x",
- "angular-loader": "~1.2.x",
- "angular-resource": "~1.2.x",
- "bootstrap": "~3.x",
- "angular-cookies": "1.2",
- "bootstrap-select": "silviomoreto/bootstrap-select#~1.7.3"
- "underscore": "1.5.2"
These dependencies are imported via annotations/main.js (js) and main.html (css).
## Directory Layout
The layout is :
```
├── __init__.py
├── README.md
├── static
│   └── annotations
│   ├── activelists.js
│   ├── app.css
│   ├── app.js
│   ├── document.js
│   ├── highlight.js
│   ├── http.js
│   ├── keyword_tpl.html
│   ├── main.js
│   ├── ngramlist.js
│   └── utils.js
├── templates
│   └── annotations
│   └── main.html
├── urls.py
└── views.py
```
# Conception and workflow documentation
## TODO : à traduire en anglais
Cette API permet d'éditer les mots-clés miamlistés ou stoplistés associé à un document affiché dans un cadre d'une page web permettant de naviguer à travers un ensemble de document d'un corpus.
### Architecture
- Templates : Django et Angular.js ?
- Communication entre les modules : évènements Angular ($emit et $broadcast)
- Pas de routage entre différentes URL, car ici une seule vue principale basée sur le template django corpus.html
- Modèle d'abstraction de données : côté client (Angular Scopes) et côté serveur (Django Model et SQLAlchemy)
- Composants : TODO lister et décrire les composants client et serveur
- Structure de l'application : organisation du client et du serveur
- Style : Bootstrap et un thème spécifique choisi pour Gargantext
- Gestion des dépendances :
- bower, npm pour le développement web et les tests côté client
- pip requirements pour le côté serveur
## Quelles actions execute l'API ?
- afficher le titre, les auteurs, le résumé, la date de publication et le corps d'un document.
- lecture des mots-clés miamlistés associés à un document (dans le texte et hors du texte).
- lecture des mots-clés stoplistés associés à un document (dans le texte et hors du texte).
- lecture des documents ayant le plus de mots-clés miamlistés associés identiques pour afficher une liste de liens vers de nouveaux documents
- lecture du groupe de mots-clés auquel appartient un mot-clé (synonymes, différentes formes)
- modification du groupe de mots-clés auquel appartient un mot-clé donné
On désigne par mot-clé un NGram.
## Schéma de l'API
Liste des endpoints
### Lecture des données
- POST '^api/nodes/(\d+)/children/queries$' : liste des NGrams d'un document avec la possibilité de filtrer par NGrams
- GET '^api/nodes$' : liste des identifiants de mots-clés filtrés par type (NGram ou autre) pour un identifiant de parent (Document ou autre)
- GET '^api/nodes/(\d+)/ngrams$': liste des termes des mots-clés associés à un Document parent, filtrés par termes
- GET ^api/nodes/(\d+)/children/metadata$ : liste des metadata d'un Node, c'est-à-dire :
- pour un document : titre, auteur, etc
- pour un NGram : stoplisté ou miamlisté ?
### Écriture des données
TODO
## Workflow
Nous nous fixons sur cette documentation et spécification de l'API
- en parallèle : développement de l'API et prototypage de l'interface
- le prototypage de l'interface peut modifier l'API si besoin
### Spécifications des fondations de l'interface
- résolutions d'écran
- browsers
- langue: english only
- SEO: aucun ?
- collaboratif : oui, les modifications d'un autre utilisateurs seront notifiées à tous les utilisateurs présent sur le même corpus de documents
- fonctionne offline ?
### Working process
- follow board is updated regularly (https://trello.com/b/96ItkDBS/gargantext-miamlists-and-stoplists)[on Trello]
- calendrier prévisionnel: TODO
- interactions entre les acteurs: emails
- git, branches : branche "elias", `git pull --rebase origin master` réguliers
- prévision des revues de code et de l'interface : TODO
### Plateforme
- Python 3.4
- Django 1.6
- Postgresql 9.3 + HSTORE
- SQLAlchemy
- Bootstrap CSS
- Angular.js
### Outils de qualité de code
- pylint
- jshint (voir .jshintrc)
- indentations : 4 espaces (voir .lvimrc)
- nettoyage automatique des espaces en fin de ligne
## Tests
There are two kinds of tests possible : Unit tests and End to End tests.
- côté client : étudier karma.js et protractor
- définir la stratégie de tests : TODO
## Déploiement
- définir le processus de déploiement
- prévoir un système de monitoring des erreurs du serveur une fois en ligne
- Sentry ?
ANNOTATIONS
===========
2016-01
## Routines de manipulation de ngrams dans les listes
#### Trajectoire globale des actions choisies
1. angular: ngramlist.js (user input) or highlight.js (user menu controller)
2. angular: http.js configuration object
{ 'action': 'post', 'listId': miamlist_id, ..}
3. AJAX POST/DELETE
4. "API locale" (=> annotations.views)
5. DB insert/delete
Remarque:
Dans le code annotations d'Elias, il y a une "API locale" qui transmet les actions client vers le serveur.
=> l'interconnexion est configurée pour angular dans annotations/static/annotations/app.js qui lance son propre main sur la fenêtre en prenant les paramètres depuis l'url et en s'isolant de django
=> les routes amont sont définies pour django dans annotations.urls et reprises pour angular dans http.js
#### Par ex: l'étape AJAX pour suppression
`curl -XDELETE http://localhost:8000/annotations/lists/7129/ngrams/4497`
via annotations.views.NgramEdit.as_view())
#### ajout d'un ngram
```
curl -XPOST http://localhost:8000/annotations/lists/1866/ngrams/create \
-H "Content-Type: application/json" \
-d '{"text":"yooooooooo"}' > response_to_ngrams_create.html
```
## Points d'interaction côté client (GUI)
Add Ngram via dialog box :
- controller:
ngramlist.annotationsAppNgramList.controller('NgramInputController')
- effect:
1. NgramHttpService.post()
Add Ngram via select new + menu
- controller:
highlight.annotationsAppHighlight.controller('TextSelectionMenuController')
1. toggleMenu (sets action = X)
2. onMenuClick
- effect:
1. NgramHttpService[action]
(function () {
'use strict';
var annotationsAppActiveLists = angular.module('annotationsAppActiveLists', []);
annotationsAppActiveLists.controller('ActiveListsController',
['$scope', '$rootScope', '$timeout',
function ($scope, $rootScope, $timeout) {
$scope.activeListsChange = function() {
var selected = $('.selectpicker option:selected').val();
var newActive = {};
$('.selectpicker option:selected').each(function(item, opt) {
// ex opt:
// <option id="list---748" value="MAINLIST">MAINLIST</option>
var id = opt.id.split("---", 2)[1];
newActive[id] = opt.value;
});
// ex: {745: "MAINLIST", 748: "MAPLIST"}
$rootScope.activeLists = newActive;
};
$rootScope.$watchCollection('activeLists', function (newValue, oldValue) {
if (newValue === undefined) return;
$timeout(function() {
$('.selectpicker').selectpicker('refresh');
});
});
// FIXME: est-ce qu'on ne pourrait pas directement utiliser lists
// au lieu de recopier dans allListsSelect ?
$rootScope.$watchCollection('lists', function (newValue, oldValue) {
if (newValue === undefined) return;
// reformat lists to allListsSelect
var allListsSelect = [];
// console.log($rootScope.lists)
angular.forEach($rootScope.lists, function(value, key) {
this.push({
'id': key,
'label': value
});
// initialize activeLists with the MAPLIST by default
if (value == 'MAPLIST') {
$rootScope.activeLists = {};
$rootScope.activeLists[key] = value;
}
}, allListsSelect);
$rootScope.allListsSelect = allListsSelect;
$timeout(function() {
$('.selectpicker').selectpicker();
$('.selectpicker').selectpicker('val', ['MAPLIST']);
});
});
}]);
})(window);
/* app css stylesheet */
/*
* Class names corresponding to server-side list names
* To display another list name, add a new class under this
*/
.MAPLIST {
color: black;
/* green */
background-color: rgba(23, 255, 189, .7);
/* background-color: rgba(60, 118, 61, .7); */
cursor: pointer;
}
.MAINLIST {
color: black;
/* orange */
background-color: orange;
/* background-color: rgba(60, 118, 61, 0.5); */
cursor: pointer;
}
.STOPLIST {
color: black;
/* grey */
background-color: rgba(169, 68, 66, 0.2);
cursor: pointer;
}
.FOCUS {
color: black;
font-weight: bold;
/* yellow */
background-color: rgba(255, 224, 101, 1);
cursor: pointer;
}
.inlay {
border-radius: .8em;
padding: 0 .5em;
margin: 0 .2em;
}
.words-panel > .tab-content {
font-size: 80%;
border-left: 1px solid #aaa;
border-right: 1px solid #aaa;
border-bottom: 1px solid #aaa;
padding: 1em ;
margin-right: 1px;
min-height: 22em ; /* needs to be bigger than .words-list */
}
#addfreengram {
padding: 3em .5em .2em .5em ;
}
.nav-pills.activelists >li>a {
padding:10px 1px;
border-top-left-radius: 4px;
border-top-right-radius: 4px;
border-bottom-left-radius: 0;
border-bottom-right-radius: 0;
border-top: 1px solid #ddd;
border-left: 1px solid #ddd;
border-right: 1px solid #ddd;
border-bottom: 1px solid #aaa;
}
.nav-pills.activelists > li.active > a,
.nav-pills.activelists > li.active > a:focus,
.nav-pills.activelists > li.active > a:hover {
background-color: #337ab7;
border-top: 1px solid #aaa;
border-left: 1px solid #aaa;
border-right: 1px solid #aaa;
border-bottom: none ;
}
.delete-keyword, .occurrences {
vertical-align: super;
font-size: 70%;
}
.delete-keyword {
cursor: pointer;
}
.center-block {
display: block;
margin-left: auto;
margin-right: auto;
}
.keyword-inline {
display: inline;
}
.keyword-inline:hover {
text-decoration: none;
}
.nav-tabs {
border-bottom: none;
}
.main-panel, .text-panel, .words-panel {
margin: 10px 0;
}
#annotationsApp {
min-width: 780px;
}
.words-panel {
min-width: 220px;
}
.text-panel {
overflow-y: auto;
min-width: 400px;
}
.words-list {
margin-bottom: 5px;
min-height: 17em ;
}
.keyword-text {
word-break: break-all;
}
.keyword-group-item {
display: inline-block;
float: left;
padding: 5px;
margin: .25em;
box-shadow: .2em .2em .1em rgba(0, 0, 0, .125);
}
.words-pagination {
margin: 5px 0;
}
.text-panel p, .text-panel h3 {
-webkit-transition: all 0.25s linear;
-moz-transition: all 0.25s linear;
-ms-transition: all 0.25s linear;
-o-transition: all 0.25s linear;
transition: all 0.25s linear;
}
/* this was used for the p or div that *contained* a selection */
/*.selection {
color: #aaa;
}*/
/* this is used for the selected text itself */
::selection {
color: black;
background-color: #aaa;
}
.noselection {
-webkit-touch-callout: none;
-webkit-user-select: none;
-khtml-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
}
.selection-menu {
display: none;
position: absolute;
color: #394141;
background: white;
font-size: 0.8em;
font-weight: 600;
box-shadow: 1px 1px 2px rgba(0, 0, 0, 0.5);
}
/*.selection-menu:before {
content: '';
position: absolute;
left: -10px;
top: 0px;
border-right: solid white 10px;
border-top: solid transparent 8px;
border-bottom: solid transparent 8px;
}*/
.selection-menu ul {
list-style: none;
margin: 0;
padding: 0;
}
.selection-menu li {
border-bottom: solid thin #CCC;
padding: 10px;
white-space: nowrap;
}
.selection-menu [class*="glyphicon"] {
min-width: 25px;
display: inline-block;
text-align: center;
border-right: solid thin #CCC;
margin-right: 5px;
}
.float-right {
float: right;
}
.spacer {
margin-top:40px;
}
<