Commit 0c4ca9a6 authored by delanoe's avatar delanoe

Merge branch 'refactoring' into refactoring-alex

parents b0ed631c 8a12394b
......@@ -15,3 +15,112 @@ Path for data used by taggers should be defined in `gargantext.constants`.
# Database
# Sharing
Here follows a brief description of how sharing could be implemented.
## Database representation
The database representation of sharing can be distributed among 4 tables:
- `persons`, of which items represent either a user or a group
- `relationships` describes the relationships between persons (affiliation
of a user to a group, contact between two users, etc.)
- `nodes` contains the projects, corpora, documents, etc. to share (they shall
inherit the sharing properties from their parents)
- `permissions` stores the relations existing between the three previously
described above: it only consists of 2 foreign keys, plus an integer
between 1 and 3 representing the level of sharing and the start date
(when the sharing has been set) and the end date (when necessary, the time
at which sharing has been removed, `NULL` otherwise)
## Python code
The permission levels should be set in `gargantext.constants`, and defined as:
```python
PERMISSION_NONE = 0 # 0b0000
PERMISSION_READ = 1 # 0b0001
PERMISSION_WRITE = 3 # 0b0011
PERMISSION_OWNER = 7 # 0b0111
```
The requests to check for permissions (or add new ones) should not be rewritten
every time. They should be "hidden" within the models:
- `Person.owns(node)` returns a boolean
- `Person.can_read(node)` returns a boolean
- `Person.can_write(node)` returns a boolean
- `Person.give_right(node, permission)` gives a right to a given user
- `Person.remove_right(node, permission)` removes a right from a given user
- `Person.get_nodes(permission[, type])` returns an iterator on the list of
nodes on which the person has at least the given permission (optional
argument: type of requested node)
- `Node.get_persons(permission[, type])` returns an iterator on the list of
users who have at least the given permission on the node (optional argument:
type of requested persons, such as `USER` or `GROUP`)
## Example
Let's imagine the `persons` table contains the following data:
| id | type | username |
|----|-------|-----------|
| 1 | USER | David |
| 2 | GROUP | C.N.R.S. |
| 3 | USER | Alexandre |
| 4 | USER | Untel |
| 5 | GROUP | I.S.C. |
| 6 | USER | Bidule |
Assume "David" owns the groups "C.N.R.S." and "I.S.C.", "Alexandre" belongs to
the group "I.S.C.", with "Untel" and "Bidule" belonging to the group "C.N.R.S.".
"Alexandre" and "David" are in contact.
The `relationships` table then contains:
| person1_id | person2_id | type |
|------------|------------|---------|
| 1 | 2 | OWNER |
| 1 | 5 | OWNER |
| 3 | 2 | MEMBER |
| 4 | 5 | MEMBER |
| 6 | 5 | MEMBER |
| 1 | 3 | CONTACT |
The `nodes` table is populated as such:
| id | type | name |
|----|----------|----------------------|
| 12 | PROJECT | My super project |
| 13 | CORPUS | A given corpus |
| 13 | CORPUS | The corpus |
| 14 | DOCUMENT | Some document |
| 15 | DOCUMENT | Another document |
| 16 | DOCUMENT | Yet another document |
| 17 | DOCUMENT | Last document |
| 18 | PROJECT | Another project |
| 19 | PROJECT | That project |
If we want to express that "David" created "My super project" (and its children)
and wants everyone in "C.N.R.S." to be able to view it, but not access it,
`permissions` should contain:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
If "David" also wanted "Alexandre" (and no one else) to view and modify "The
corpus" (and its children), we would have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 1 | 12 | OWNER |
| 2 | 12 | READ |
| 3 | 13 | WRITE |
If "Alexandre" created "That project" and wants "Bidule" (and no one else) to be
able to view and modify it (and its children), the table should then have:
| person_id | node_id | permission |
|-----------|---------|------------|
| 3 | 19 | OWNER |
| 6 | 19 | WRITE |
......@@ -51,6 +51,7 @@ INSTALLED_APPS = [
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'djcelery',
]
......
......@@ -10,7 +10,7 @@ from django.conf.urls import include, url
from django.contrib import admin
# import gargantext.views.api
import gargantext.views.api.urls
import gargantext.views.generated.urls
import gargantext.views.pages.urls
......@@ -18,5 +18,6 @@ import gargantext.views.pages.urls
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^generated/', include(gargantext.views.generated.urls)),
url(r'^api/', include(gargantext.views.api.urls)),
url(r'^', include(gargantext.views.pages.urls)),
]
......@@ -9,6 +9,8 @@ from urllib.parse import quote_plus as urlencode
from gargantext import settings
# authentication
def requires_auth(func):
"""Provides a decorator to force authentication on a given view.
Also passes the URL to redirect towards as a GET parameter.
......@@ -21,8 +23,57 @@ def requires_auth(func):
return _requires_auth
# download from a given URL
import urllib.request
def get(url):
response = urllib.request.urlopen(url)
html = response.read()
# retrieve GET parameters from a request
def get_parameters(request):
parameters = {}
print(request.GET)
print(request.GET._iterlists())
for key, value in request.GET._iterlists():
if key.endswith('[]'):
parameters[key[:-2]] = value
else:
parameters[key] = value[0]
return parameters
# REST
from rest_framework.views import APIView
# provide a JSON response
import json
import datetime
class JSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
return obj.isoformat()[:19] + 'Z'
elif isinstance(obj, (set, tuple)):
return list(obj)
else:
return super(self.__class__, self).default(obj)
json_encoder = JSONEncoder(indent=4)
def JsonHttpResponse(data, status=200):
return HttpResponse(
content = json_encoder.encode(data),
content_type = 'application/json; charset=utf-8',
status = status
)
# provide exceptions for JSON APIs
from rest_framework.exceptions import APIException
from rest_framework.exceptions import ValidationError as ValidationException
......@@ -36,11 +36,8 @@ from celery import shared_task
def scheduled_celery(func):
"""Provides a decorator to schedule a task with Celery.
"""
@shared_task
def _func(*args, **kwargs):
func(*args, **kwargs)
def go(*args, **kwargs):
_func.apply_async(args=args, kwargs=kwargs)
shared_task(func).apply_async(args=args, kwargs=kwargs)
return go
......
from gargantext.util.http import ValidationException
from datetime import datetime
__all__ = ['validate']
_types_names = {
bool: 'boolean',
int: 'integer',
float: 'float',
str: 'string',
dict: 'object',
list: 'array',
datetime: 'datetime',
}
def validate(value, expected, path='input'):
# Is the expected type respected?
if 'type' in expected:
expected_type = expected['type']
if not isinstance(value, expected_type):
if expected_type in (bool, int, float, str, datetime, ):
try:
if expected_type == bool:
value = value not in {0, 0.0, '', '0', 'false'}
elif expected_type == datetime:
value = value + '2000-01-01T00:00:00Z'[len(value):]
value = datetime.strptime(value, '%Y-%m-%dT%H:%M:%SZ')
else:
value = expected_type(value)
except ValueError:
raise ValidationException('%s should be a JSON %s, but could not be parsed as such' % (path, _types_names[expected_type], ))
else:
raise ValidationException('%s should be a JSON %s' % (path, _types_names[expected_type], ))
else:
expected_type = type(value)
# Is the value in the expected range?
if 'range' in expected:
expected_range = expected['range']
if isinstance(expected_range, tuple):
if expected_type in (int, float):
tested_value = value
tested_name = 'value'
elif expected_type in (str, list):
tested_value = len(value)
tested_name = 'length'
if tested_value < expected_range[0]:
raise ValidationException('%s should have a minimum %s of %d' % (path, tested_name, expected_range[0], ))
if len(expected_range) > 1 and tested_value > expected_range[1]:
raise ValidationException('%s should have a maximum %s of %d' % (path, tested_name, expected_range[1], ))
elif isinstance(expected_range, (list, set, )) and value not in expected_range:
expected_values = expected_range if isinstance(expected_range, list) else expected_range
expected_values = [str(value) for value in expected_values if isinstance(value, expected_type)]
if len(expected_values) < 16:
expected_values_str = '", "'.join(expected_values)
expected_values_str = '"' + expected_values_str + '"'
else:
expected_values_str = '", "'.join(expected_values[:16])
expected_values_str = '"' + expected_values_str + '"...'
raise ValidationException('%s should take one of the following values: %s' % (path, expected_values_str, ))
# Do we have to translate through a dictionary?
if 'translate' in expected:
translate = expected['translate']
if callable(translate):
value = translate(value)
if value is None and expected.get('required', False):
raise ValidationException('%s has been given an invalid value' % (path, ))
return value
try:
value = expected['translate'][value]
except KeyError:
if expected.get('translate_fallback_keep', False):
return value
if expected.get('required', False):
raise ValidationException('%s has been given an invalid value' % (path, ))
else:
return expected.get('default', value)
# Are we handling an iterable?
if expected_type in (list, dict):
if 'items' in expected:
expected_items = expected['items']
if expected_type == list:
for i, element in enumerate(value):
value[i] = validate(element, expected_items, '%s[%d]' % (path, i, ))
elif expected_type == dict:
if expected_items:
for key in value:
if key not in expected_items:
raise ValidationException('%s should not have a "%s" key.' % (path, key, ))
for expected_key, expected_value in expected_items.items():
if expected_key in value:
value[expected_key] = validate(value[expected_key], expected_value, '%s["%s"]' % (path, expected_key, ))
elif 'required' in expected_value and expected_value['required']:
raise ValidationException('%s should have a "%s" key.' % (path, expected_key, ))
elif 'default' in expected_value:
value[expected_key] = expected_value['default']
# Let's return the proper value!
return value
from gargantext.util.http import *
from gargantext.util.db import *
from gargantext.models import *
from gargantext.constants import *
from gargantext.util.validation import validate
class NodesList(APIView):
_fields = ['id', 'parent_id', 'name', 'typename', 'hyperdata']
_types = NODETYPES
def _query(self, request):
# parameters validation
parameters = get_parameters(request)
parameters = validate(parameters, {'type': dict, 'items': {
'pagination_limit': {'type': int, 'default': 10},
'pagination_offset': {'type': int, 'default': 0},
'pagination_offset': {'type': int, 'default': 0},
'fields': {'type': list, 'default': self._fields, 'items': {
'type': str, 'range': self._fields,
}},
# optional filtering parameters
'type': {'type': list, 'default': self._types, 'required': False, 'items': {
'type': str, 'range': self._types,
}},
'parent_id': {'type': int, 'required': False},
}})
# start the query
query = session.query(*tuple(
getattr(Node, field) for field in parameters['fields']
))
# filter by type
if 'type' in parameters:
query = query.filter(Node.typename.in_(parameters['type']))
# filter by parent
if 'parent_id' in parameters:
query = query.filter(Node.parent_id == parameters['parent_id'])
# paginate the query
count = query.count()
if parameters['pagination_limit'] == -1:
query = query[parameters['pagination_offset']:]
else:
query = query[
parameters['pagination_offset'] :
parameters['pagination_limit']
]
# return the result!
return parameters, query, count
def get(self, request):
"""Displays the list of nodes corresponding to the query.
"""
parameters, query, count = self._query(request)
return JsonHttpResponse({
'parameters': parameters,
'count': count,
'records': [dict(zip(parameters['fields'], node)) for node in query]
})
def delete(self, request):
"""Removes the list of nodes corresponding to the query.
WARNING! THIS IS TOTALLY UNTESTED!!!!!
"""
parameters, query, count = self._query(request)
for node in query:
node.delete()
session.commit()
return JsonHttpResponse({
'parameters': parameters,
'count': count,
}, 200)
from django.conf.urls import url
# from . import main
from . import nodes
urlpatterns = [
# url(r'^$', main.home),
url(r'^nodes$', nodes.NodesList.as_view()),
]
from gargantext.util.http import *
from gargantext.util.db import *
from gargantext.util.db_cache import cache
from gargantext.models import *
from gargantext.constants import *
from gargantext.settings import *
from datetime import datetime
def _get_user_project_corpus(request, project_id, corpus_id):
"""Helper method to get a corpus, knowing the project's and corpus' ID.
Raises HTTP errors when parameters (user, IDs...) are invalid.
"""
user = cache.User[request.user.username]
project = session.query(Node).filter(Node.id == project_id).first()
corpus = session.query(Node).filter(Node.id == corpus_id).filter(Node.parent_id == project_id).first()
if corpus is None:
raise Http404()
if not user.owns(corpus):
raise HttpResponseForbidden()
return user, project, corpus
@requires_auth
def corpus(request, project_id, corpus_id):
user, project, corpus = _get_user_project_corpus(request, project_id, corpus_id)
# response!
return render(
template_name = 'pages/corpora/corpus.html',
request = request,
context = {
'debug': DEBUG,
'user': user,
'date': datetime.now(),
'project': project,
'corpus': corpus,
# 'processing': processing,
# 'number': number,
'view': 'documents'
},
)
@requires_auth
def chart(request, project_id, corpus_id):
user, project, corpus = _get_user_project_corpus(request, project_id, corpus_id)
from django.conf.urls import url
from . import main, auth, projects
from . import main, auth
from . import projects, corpora
urlpatterns = [
......@@ -14,8 +15,12 @@ urlpatterns = [
url(r'^auth/login/?$', auth.login),
url(r'^auth/logout/?$', auth.logout),
# overview on projects
# projects
url(r'^projects/?$', projects.overview),
url(r'^projects/(\d+)/?$', projects.project),
# corpora
url(r'^projects/(\d+)/corpora/(\d+)?$', corpora.corpus),
url(r'^projects/(\d+)/corpora/(\d+)/chart?$', corpora.chart),
]
......@@ -10,6 +10,7 @@ dateparser==0.3.2
django-celery==3.1.17
django-pgfields==1.4.4
django-pgjsonb==0.0.16
djangorestframework==3.3.2
html5lib==0.9999999
jdatetime==1.7.2
kombu==3.0.33
......
.dc-chart {
float: left;
}
.dc-chart rect.bar {
stroke: none;
fill: steelblue;
}
.dc-chart rect.bar:hover {
fill-opacity: .5;
}
.dc-chart rect.stack1 {
stroke: none;
fill: red;
}
.dc-chart rect.stack2 {
stroke: none;
fill: green;
}
.dc-chart rect.deselected {
stroke: none;
fill: #ccc;
}
.dc-chart .sub .bar {
stroke: none;
fill: #ccc;
}
.dc-chart .pie-slice {
fill: white;
font-size: 12px;
cursor: pointer;
}
.dc-chart .pie-slice :hover{
fill-opacity: .8;
}
.dc-chart .selected path{
stroke-width: 3;
stroke: #ccc;
fill-opacity: 1;
}
.dc-chart .deselected path{
strok: none;
fill-opacity: .5;
fill: #ccc;
}
.dc-chart .axis path, .axis line {
fill: none;
stroke: #000;
shape-rendering: crispEdges;
}
.dc-chart .axis text {
font: 10px sans-serif;
}
.dc-chart .grid-line line {
fill: none;
stroke: #ccc;
opacity: .5;
shape-rendering: crispEdges;
}
.dc-chart .brush rect.background {
z-index: -999;
}
.dc-chart .brush rect.extent {
fill: steelblue;
fill-opacity: .125;
}
.dc-chart .brush .resize path {
fill: #eee;
stroke: #666;
}
.dc-chart path.line {
fill: none;
stroke: steelblue;
stroke-width: 1.5px;
}
.dc-chart circle.dot{
fill: steelblue;
}
.dc-chart g.stack1 path.line {
stroke: green;
}
.dc-chart g.stack1 circle.dot{
fill: green;
}
.dc-chart g.stack2 path.line {
stroke: red;
}
.dc-chart g.stack2 circle.dot{
fill: red;
}
.dc-chart g.dc-tooltip path{
fill: none;
stroke: grey;
stroke-opacity: .8;
}
.dc-chart path.area {
fill: steelblue;
fill-opacity: .3;
stroke: none;
}
.dc-chart g.stack1 path.area {
fill: green;
}
.dc-chart g.stack2 path.area {
fill: red;
}
.dc-chart .node {
font-size: 0.7em;
cursor: pointer;
}
.dc-chart .node :hover{
fill-opacity: .8;
}
.dc-chart .selected circle {
stroke-width: 3;
stroke: #ccc;
fill-opacity: 1;
}
.dc-chart .deselected circle {
strok: none;
fill-opacity: .5;
fill: #ccc;
}
.dc-chart .bubble {
stroke: none;
fill-opacity: 0.6;
}
.dc-data-count {
float: right;
margin-top: 15px;
margin-right: 15px;
}
.dc-data-count .filter-count {
color: #3182bd;
font-weight: bold;
}
.dc-data-count .total-count {
color: #3182bd;
font-weight: bold;
}
.dc-data-table {}
.dc-chart g.state{
cursor: pointer;
}
.dc-chart g.state :hover{
fill-opacity: .8;
}
.dc-chart g.state path {
stroke: white;
}
.dc-chart g.selected path {
}
.dc-chart g.deselected path {
fill: grey;
}
.dc-chart g.selected text {
}
.dc-chart g.deselected text {
display: none;
}
.dc-chart g.county path {
stroke: white;
fill: none;
}
.dc-chart g.debug rect{
fill: blue;
fill-opacity: .2;
}
.dc-chart g.row rect {
fill-opacity: 0.8;
cursor: pointer;
}
.dc-chart g.row rect:hover {
fill-opacity: 0.6;
}
.dc-chart g.row text {
fill: white;
font-size: 12px;
}
/*
* jQuery Dynatable plugin 0.3.1
*
* Copyright (c) 2014 Steve Schwartz (JangoSteve)
*
* Dual licensed under the AGPL and Proprietary licenses:
* http://www.dynatable.com/license/
*
* Date: Tue Jan 02 2014
*/
th {
background: #bd2525;
}
th a {
color: #fff;
}
th a:hover {
color: #fff;
text-decoration: underline;
}
.dynatable-search {