Skip to content

gargantext
gargantext
Commit ea1172a4 authored by delanoe's avatar delanoe
Browse files
Options

Merge branch 'testing' into stable-merge

parents fd2cbb52 8a68083e
Hide whitespace changes
Inline Side-by-side
Showing with 1390 additions and 186 deletions
docs/architecture.md 0 → 100644
View file @ ea1172a4
# Definitions and notation for the documentation (!= python notation)
## Node
The table (nodes) is a list of nodes: [Node]
Each Node has:
- a typename
- a parent_id
- a name
### Each Node has a parent_id
Node A
├── Node B
└── Node C
If Node A is Parent of Node B and Node C
then NodeA.id == NodeB.parent_id == NodeC.parent_id.
### Each Node has a typename
Notation: Node[foo](bar) is a Node of typename "foo" and with name "bar".
Then:
- Then Node[project] is a project.
- Then Node[corpus] is a corpus.
- Then Node[document] is a document.
### Each Node as a typename and a parent
Node[user](name)
├── Node[project](myProject1)
│   ├── Node[corpus](myCorpus1)
│   ├── Node[corpus](myCorpus2)
│   └── Node[corpus](myCorpus3)
└── Node[project](myProject2)
/!\ 3 way to manage rights of the Node:
1) Then Node[User] is a folder containing all User projects and corpus and
documents (i.e. Node[user] is the parent_id of the children).
2) Each node as a user_id (mainly used today)
3) Right management for the groups (implemented already but not
used since not connected to the frontend).
## Global Parameters
Global User is Gargantua (Node with typename User).
This node is the parent of the others Nodes for parameters.
Node[user](gargantua) (gargantua.id == Node[user].user_id)
├── Node[TFIDF-Global](global) : without group
│   ├── Node[tfidf](database1)
│   ├── Node[tfidf](database2)
│   └── Node[tfidf](database2)
└── Node[anotherMetric](global)
## NodeNgram
NodeNgram is a relation of a Node with a ngram:
- document and ngrams
- metrics and ngrams (position of the node metrics indicates the
context)
# Community Parameters
# User Parameters
docs/index.md
View file @ ea1172a4
...@@ -8,6 +8,9 @@ Gargantext is a web plateform to explore your corpora using text-mining[...](abo ...@@ -8,6 +8,9 @@ Gargantext is a web plateform to explore your corpora using text-mining[...](abo
* [Take a tour](demo.md) of the different features offered by Gargantext * [Take a tour](demo.md) of the different features offered by Gargantext
## Architecture
* [Architecture](architecture.md) Architecture of Gargantext
##Need some help? ##Need some help?
Ask the community at: Ask the community at:
......
docs/manual_install.md deleted 100644 → 0
View file @ fd2cbb52
* Create user gargantua
Main user of Gargantext is Gargantua (role of Pantagruel soon)!
``` bash
sudo adduser --disabled-password --gecos "" gargantua
```
* Create the directories you need
here for the example gargantext package will be installed in /srv/
``` bash
for dir in "/srv/gargantext"
"/srv/gargantext_lib"
"/srv/gargantext_static"
"/srv/gargantext_media"
"/srv/env_3-5"; do
sudo mkdir -p $dir ;
sudo chown gargantua:gargantua $dir ;
done
```
You should see:
```bash
$tree /srv
/srv
├── gargantext
├── gargantext_lib
├── gargantext_media
│   └── srv
│   └── env_3-5
└── gargantext_static
```
* Get the main libraries
Download uncompress and make main user access to it.
PLease, Be patient due to the size of the packages libraries (27GO)
this step can be long....
``` bash
wget http://dl.gargantext.org/gargantext_lib.tar.bz2 \
&& tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib \
&& sudo chown -R gargantua:gargantua /srv/gargantext_lib \
&& echo "Libs installed"
```
* Get the source code of Gargantext
by cloning the repository of gargantext
``` bash
git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
&& cd /srv/gargantext \
&& git fetch origin refactoring \
&& git checkout refactoring \
```
TODO(soon): git clone https://gogs.iscpif.fr/gargantext.git
See the [next steps of installation procedure](install.md#Install)
docs/manual_install.md 0 → 120000
View file @ ea1172a4
tools/manual_install.md
\ No newline at end of file
gargantext/constants.py
View file @ ea1172a4
...@@ -240,7 +240,7 @@ RESOURCETYPES = [ ...@@ -240,7 +240,7 @@ RESOURCETYPES = [
'crawler': None, 'crawler': None,
}, },
{ "type": 9, { "type": 9,
"name": 'SCOAP [CRAWLER/XML]', "name": 'SCOAP [API/XML]',
"parser": "CernParser", "parser": "CernParser",
"format": 'MARC21', "format": 'MARC21',
'file_formats':["zip","xml"], 'file_formats':["zip","xml"],
...@@ -255,7 +255,7 @@ RESOURCETYPES = [ ...@@ -255,7 +255,7 @@ RESOURCETYPES = [
# }, # },
# #
{ "type": 10, { "type": 10,
"name": 'REPEC [CRAWLER]', "name": 'REPEC [MULTIVAC API]',
"parser": "MultivacParser", "parser": "MultivacParser",
"format": 'JSON', "format": 'JSON',
'file_formats':["zip","json"], 'file_formats':["zip","json"],
...@@ -263,13 +263,21 @@ RESOURCETYPES = [ ...@@ -263,13 +263,21 @@ RESOURCETYPES = [
}, },
{ "type": 11, { "type": 11,
"name": 'HAL [CRAWLER]', "name": 'HAL [API]',
"parser": "HalParser", "parser": "HalParser",
"format": 'JSON', "format": 'JSON',
'file_formats':["zip","json"], 'file_formats':["zip","json"],
"crawler": "HalCrawler", "crawler": "HalCrawler",
}, },
{ "type": 12,
"name": 'ISIDORE [SPARQLE API /!\ BETA]',
"parser": "IsidoreParser",
"format": 'JSON',
'file_formats':["zip","json"],
"crawler": "IsidoreCrawler",
},
] ]
#shortcut for resources declaration in template #shortcut for resources declaration in template
PARSERS = [(n["type"],n["name"]) for n in RESOURCETYPES if n["parser"] is not None] PARSERS = [(n["type"],n["name"]) for n in RESOURCETYPES if n["parser"] is not None]
......
gargantext/util/crawlers/ISIDORE.py 0 → 100644
View file @ ea1172a4
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# ****************************
# **** ISIDORE Scrapper ***
# ****************************
# CNRS COPYRIGHTS
# SEE LEGAL LICENCE OF GARGANTEXT.ORG
from ._Crawler import *
import json
from gargantext.constants import UPLOAD_DIRECTORY
from math import trunc
from gargantext.util.files import save
from gargantext.util.crawlers.sparql.bool2sparql import bool2sparql, isidore
class IsidoreCrawler(Crawler):
''' ISIDORE SPARQL API CLIENT'''
def __init__(self):
# Main EndPoints
self.BASE_URL = "https://www.rechercheisidore.fr"
self.API_URL = "sparql"
# Final EndPoints
# TODO : Change endpoint according type of database
self.URL = self.BASE_URL + "/" + self.API_URL
self.status = []
def __format_query__(self, query=None, count=False, offset=None, limit=None):
'''formating the query'''
return (bool2sparql(query, count=count, offset=offset, limit=limit))
def _get(self, query, offset=0, limit=None, lang=None):
'''Parameters to download data'''
isidore(query, count=False, offset=offset, limit=limit)
def scan_results(self, query):
'''
scan_results : Returns the number of results
Query String -> Int
'''
self.results_nb = [n for n in isidore(query, count=True)][0]
return self.results_nb
def download(self, query):
downloaded = False
self.status.append("fetching results")
corpus = []
limit = 1000
self.query_max = self.scan_results(query)
print("self.query_max : %s" % self.query_max)
if self.query_max > QUERY_SIZE_N_MAX:
msg = "Invalid sample size N = %i (max = %i)" % ( self.query_max
, QUERY_SIZE_N_MAX
)
print("WARNING (scrap: ISIDORE d/l ): " , msg)
self.query_max = QUERY_SIZE_N_MAX
for offset in range(0, self.query_max, limit):
print("Downloading result %s to %s" % (offset, self.query_max))
for doc in isidore(query, offset=offset, limit=limit) :
corpus.append(doc)
self.path = save( json.dumps(corpus).encode("utf-8")
, name='ISIDORE.json'
, basedir=UPLOAD_DIRECTORY
)
downloaded = True
return downloaded
gargantext/util/crawlers/sparql/bool2sparql.py 0 → 100644
View file @ ea1172a4
import subprocess
import re
from .sparql import Service
#from sparql import Service
def bool2sparql(rawQuery, count=False, offset=None, limit=None):
"""
bool2sparql :: String -> Bool -> Int -> String
Translate a boolean query into a Sparql request
You need to build bool2sparql binaries before
See: https://github.com/delanoe/bool2sparql
"""
query = re.sub("\"", "\'", rawQuery)
bashCommand = ["/srv/gargantext/gargantext/util/crawlers/sparql/bool2sparql-exe","-q",query]
if count is True :
bashCommand.append("-c")
else :
if offset is not None :
for command in ["--offset", str(offset)] :
bashCommand.append(command)
if limit is not None :
for command in ["--limit", str(limit)] :
bashCommand.append(command)
process = subprocess.Popen(bashCommand, stdout=subprocess.PIPE)
output, error = process.communicate()
if error is not None :
raise(error)
else :
print(output)
return(output.decode("utf-8"))
def isidore(query, count=False, offset=None, limit=None):
"""
isidore :: String -> Bool -> Int -> Either (Dict String) Int
use sparql-client either to search or to scan
"""
query = bool2sparql(query, count=count, offset=offset, limit=limit)
go = Service("https://www.rechercheisidore.fr/sparql/", "utf-8", "GET")
results = go.query(query)
if count is False:
for r in results:
doc = dict()
doc_values = dict()
doc["url"], doc["title"], doc["date"], doc["abstract"], doc["source"] = r
for k in doc.keys():
doc_values[k] = doc[k].value
yield(doc_values)
else :
count = []
for r in results:
n, = r
count.append(int(n.value))
yield count[0]
def test():
query = "delanoe"
limit = 100
offset = 10
for d in isidore(query, offset=offset, limit=limit):
print(d["date"])
#print([n for n in isidore(query, count=True)])
if __name__ == '__main__':
test()
gargantext/util/crawlers/sparql/sparql.py 0 → 100644
View file @ ea1172a4
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# The contents of this file are subject to the Mozilla Public
# License Version 1.1 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of
# the License at http://www.mozilla.org/MPL/
#
# Software distributed under the License is distributed on an "AS
# IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
# implied. See the License for the specific language governing
# rights and limitations under the License.
#
# The Original Code is "SPARQL Client"
#
# The Initial Owner of the Original Code is European Environment
# Agency (EEA). Portions created by Eau de Web are
# Copyright (C) 2011 by European Environment Agency. All
# Rights Reserved.
#
# Contributor(s):
# Søren Roug, EEA
# Alex Morega, Eau de Web
# David Bătrânu, Eau de Web
"""
The `sparql` module can be invoked in several different ways. To quickly run a
query use :func:`query`. Results are encapsulated in a
:class:`_ResultsParser` instance::
>>> result = sparql.query(endpoint, query)
>>> for row in result:
>>> print row
Command-line use
----------------
::
sparql.py [-i] endpoint
-i Interactive mode
If interactive mode is enabled, the program reads queries from the console
and then executes them. Use a double line (two 'enters') to separate queries.
Otherwise, the query is read from standard input.
"""
from base64 import encodestring
#from string import replace
from urllib.parse import urlencode
from xml.dom import pulldom
import copy
import decimal
import re
import tempfile
import eventlet
import urllib.request as urllib2
try:
__version__ = open('version.txt').read().strip()
except Exception:
__version__ = "2.6"
USER_AGENT = "sparql-client/%s +http://www.eionet.europa.eu/software/sparql-client/" % __version__
CONTENT_TYPE = {
'turtle' : "application/turtle" ,
'n3' :"application/n3",
'rdfxml' : "application/rdf+xml" ,
'ntriples' : "application/n-triples" ,
'xml' : "application/xml"
}
RESULTS_TYPES = {
'xml' : "application/sparql-results+xml" ,
'json' : "application/sparql-results+json"
}
# The purpose of this construction is to use shared strings when
# they have the same value. This way comparisons can happen on the
# memory address rather than looping through the content.
XSD_STRING = 'http://www.w3.org/2001/XMLSchema#string'
XSD_INT = 'http://www.w3.org/2001/XMLSchema#int'
XSD_LONG = 'http://www.w3.org/2001/XMLSchema#long'
XSD_DOUBLE = 'http://www.w3.org/2001/XMLSchema#double'
XSD_FLOAT = 'http://www.w3.org/2001/XMLSchema#float'
XSD_INTEGER = 'http://www.w3.org/2001/XMLSchema#integer'
XSD_DECIMAL = 'http://www.w3.org/2001/XMLSchema#decimal'
XSD_DATETIME = 'http://www.w3.org/2001/XMLSchema#dateTime'
XSD_DATE = 'http://www.w3.org/2001/XMLSchema#date'
XSD_TIME = 'http://www.w3.org/2001/XMLSchema#time'
XSD_BOOLEAN = 'http://www.w3.org/2001/XMLSchema#boolean'
datatype_dict = {
'': '',
XSD_STRING : XSD_STRING,
XSD_INT : XSD_INT,
XSD_LONG : XSD_LONG,
XSD_DOUBLE : XSD_DOUBLE,
XSD_FLOAT : XSD_FLOAT,
XSD_INTEGER : XSD_INTEGER,
XSD_DECIMAL : XSD_DECIMAL,
XSD_DATETIME : XSD_DATETIME,
XSD_DATE : XSD_DATE,
XSD_TIME : XSD_TIME,
XSD_BOOLEAN : XSD_BOOLEAN
}
# allow import from RestrictedPython
__allow_access_to_unprotected_subobjects__ = {'Datatype': 1, 'unpack_row': 1,
'RDFTerm': 1, 'IRI': 1, 'Literal': 1, 'BlankNode': 1}
def Datatype(value):
"""
Replace the string with a shared string.
intern() only works for plain strings - not unicode.
We make it look like a class, because it conceptually could be.
"""
if value==None:
r = None
elif value in datatype_dict :
r = datatype_dict[value]
else:
r = datatype_dict[value] = value
return r
class RDFTerm(object):
"""
Super class containing methods to override. :class:`IRI`,
:class:`Literal` and :class:`BlankNode` all inherit from :class:`RDFTerm`.
"""
__allow_access_to_unprotected_subobjects__ = {'n3': 1}
def __str__(self):
return str(self.value)
def __unicode__(self):
return self.value
def n3(self):
""" Return a Notation3 representation of this term. """
# To override
# See N-Triples syntax: http://www.w3.org/TR/rdf-testcases/#ntriples
raise NotImplementedError("Subclasses of RDFTerm must implement `n3`")
def __repr__(self):
return '<%s %s>' % (type(self).__name__, self.n3())
class IRI(RDFTerm):
""" An RDF resource. """
def __init__(self, value):
self.value = value
def __str__(self):
return self.value.encode("unicode-escape")
def __eq__(self, other):
if type(self) != type(other):
return False
if self.value == other.value: return True
return False
def n3(self):
return '<%s>' % self.value
_n3_quote_char = re.compile(r'[^ -~]|["\\]')
_n3_quote_map = {
'"': '\\"',
'\n': '\\n',
'\t': '\\t',
'\\': '\\\\',
}
def _n3_quote(string):
def escape(m):
ch = m.group()
if ch in _n3_quote_map:
return _n3_quote_map[ch]
else:
return "\\u%04x" % ord(ch)
return '"' + _n3_quote_char.sub(escape, string) + '"'
class Literal(RDFTerm):
"""
Literals. These can take a data type or a language code.
"""
def __init__(self, value, datatype=None, lang=None):
self.value = value
self.lang = lang
self.datatype = datatype
def __eq__(self, other):
if type(self) != type(other):
return False
elif (self.value == other.value and
self.lang == other.lang and
self.datatype == other.datatype):
return True
else:
return False
def n3(self):
n3_value = _n3_quote(self.value)
if self.datatype is not None:
n3_value += '^^<%s>' % self.datatype
if self.lang is not None:
n3_value += '@' + self.lang
return n3_value
class BlankNode(RDFTerm):
""" Blank node. Similar to `IRI` but lacks a stable identifier. """
def __init__(self, value):
self.value = value
def __eq__(self, other):
if type(self) != type(other):
return False
if self.value == other.value:
return True
return False
def n3(self):
return '_:%s' % self.value
_n3parser_lang = re.compile(r'@(?P<lang>\w+)$')
_n3parser_datatype = re.compile(r'\^\^<(?P<datatype>[^\^"\'>]+)>$')
def parse_n3_term(src):
"""
Parse a Notation3 value into a RDFTerm object (IRI or Literal).
This parser understands IRIs and quoted strings; basic non-string types
(integers, decimals, booleans, etc) are not supported yet.
"""
if src.startswith('<'):
# `src` is an IRI
if not src.endswith('>'):
raise ValueError
value = src[1:-1]
if '<' in value or '>' in value:
raise ValueError
return IRI(value)
else:
datatype_match = _n3parser_datatype.search(src)
if datatype_match is not None:
datatype = datatype_match.group('datatype')
src = _n3parser_datatype.sub('', src)
else:
datatype = None
lang_match = _n3parser_lang.search(src)
if lang_match is not None:
lang = lang_match.group('lang')
src = _n3parser_lang.sub('', src)
else:
lang = None
# Python literals syntax is mostly compatible with N3.
# We don't execute the code, just turn it into an AST.
# try:
# ast = compiler.parse("value = u" + src)
# except:
# raise ValueError
# Don't allow any extra tokens in the AST
# if len(ast.node.getChildNodes()) != 1:
# raise ValueError
# assign_node = ast.node.getChildNodes()[0]
# if len(assign_node.getChildNodes()) != 2:
# raise ValueError
# value_node = assign_node.getChildNodes()[1]
# if value_node.getChildNodes():
# raise ValueError
# if value_node.__class__ != compiler.ast.Const:
# raise ValueError
# value = value_node.value
#
# if type(value) is not unicode:
# raise ValueError
#
return Literal(value, datatype, lang)
#########################################
#
# _ServiceMixin
#
#########################################
class _ServiceMixin(object):
def __init__(self, endpoint, method = "POST"):
self._method = method
self.endpoint = endpoint
self._default_graphs = []
self._named_graphs = []
self._prefix_map = {}
self._headers_map = {}
self._headers_map['Accept'] = RESULTS_TYPES['xml']
self._headers_map['User-Agent'] = USER_AGENT
def _setMethod(self, method):
if method in ("GET", "POST"):
self._method = method
else: raise ValueError("Only GET or POST is allowed")
def _getMethod(self):
return self._method
method = property(_getMethod, _setMethod)
def addDefaultGraph(self, g):
self._default_graphs.append(g)
def defaultGraphs(self):
return self._default_graphs
def addNamedGraph(self, g):
self._named_graphs.append(g)
def namedGraphs(self):
return self._named_graphs
def setPrefix(self, prefix, uri):
self._prefix_map[prefix] = uri
def prefixes(self):
return self._prefix_map
def headers(self):
return self._headers_map
#########################################
#
# Service
#
#########################################
class Service(_ServiceMixin):
"""
This is the main entry to the library.
The user creates a :class:`Service`, then sends a query to it.
If we want to have persistent connections, then open them here.
"""
def __init__(self, endpoint, qs_encoding = "utf-8", method = "POST"):
_ServiceMixin.__init__(self, endpoint, method)
self.qs_encoding = qs_encoding
def createQuery(self):
q = _Query(self)
q._default_graphs = copy.deepcopy(self._default_graphs)
q._headers_map = copy.deepcopy(self._headers_map)
q._named_graphs = copy.deepcopy(self._named_graphs)
q._prefix_map = copy.deepcopy(self._prefix_map)
return q
def query(self, query, timeout = 0):
q = self.createQuery()
return q.query(query, timeout)
def authenticate(self, username, password):
self._headers_map['Authorization'] = "Basic %s" % encodestring("%s:%s" % (username, password)).replace("\012", "")
def _parseBoolean(val):
if val.lower() in ('true', '1'):
return True
else:
return False
# XMLSchema types and cast functions
_types = {
XSD_INT: int,
XSD_LONG: int,
XSD_DOUBLE: float,
XSD_FLOAT: float,
XSD_INTEGER: int, # INTEGER is a DECIMAL, but Python `int` has no size
# limit, so it's safe to use
XSD_DECIMAL: decimal.Decimal,
XSD_BOOLEAN: _parseBoolean,
}
try:
import dateutil.parser
_types[XSD_DATETIME] = dateutil.parser.parse
_types[XSD_DATE] = lambda v: dateutil.parser.parse(v).date()
_types[XSD_TIME] = lambda v: dateutil.parser.parse(v).time()
except ImportError:
pass
def unpack_row(row, convert=None, convert_type={}):
"""
Convert values in the given row from :class:`RDFTerm` objects to plain
Python values: :class:`IRI` is converted to a unicode string containing
the IRI value; :class:`BlankNode` is converted to a unicode string with
the BNode's identifier, and :class:`Literal` is converted based on its
XSD datatype.
The library knows about common XSD types (STRING becomes :class:`unicode`,
INTEGER and LONG become :class:`int`, DOUBLE and FLOAT become
:class:`float`, DECIMAL becomes :class:`~decimal.Decimal`, BOOLEAN becomes
:class:`bool`). If the `python-dateutil` library is found, then DATE,
TIME and DATETIME are converted to :class:`~datetime.date`,
:class:`~datetime.time` and :class:`~datetime.datetime` respectively. For
other conversions, an extra argument `convert` may be passed. It should be
a callable accepting two arguments: the serialized value as a
:class:`unicode` object, and the XSD datatype.
"""
out = []
known_types = dict(_types)
known_types.update(convert_type)
for item in row:
if item is None:
value = None
elif isinstance(item, Literal):
if item.datatype in known_types:
to_python = known_types[item.datatype]
value = to_python(item.value)
elif convert is not None:
value = convert(item.value, item.datatype)
else:
value = item.value
else:
value = item.value
out.append(value)
return out
#########################################
#
# _Query
#
#########################################
class _Query(_ServiceMixin):
def __init__(self, service):
_ServiceMixin.__init__(self, service.endpoint, service.method)
def _build_request(self, query):
if self.method == "GET":
if '?' in self.endpoint:
separator = '&'
else:
separator = '?'
uri = self.endpoint.strip() + separator + str(query)
#print("endpoint", self.endpoint.strip())
#print("separator", separator)
#print("query", query.strip())
#print(uri)
return urllib2.Request(uri)
else:
uri = self.endpoint.strip()
return urllib2.Request(uri, data=urlencode(query))
def _get_response(self, opener, request, buf, timeout=None):
try:
response = opener.open(request, timeout=timeout)
response_code = response.getcode()
if response_code != 200:
buf.seek(0)
ret = buf.read()
buf.close()
raise (SparqlException(response_code, ret))
else:
return response
except Exception as error:
raise (SparqlException('Error', error))
def _read_response(self, response, buf, timeout):
if timeout > 0:
with eventlet.timeout.Timeout(timeout):
try:
buf.write(response.read())
except eventlet.timeout.Timeout as error:
raise SparqlException('Timeout', repr(error))
else:
buf.write(response.read())
def _request(self, statement, timeout=0):
"""
Builds the query string, then opens a connection to the endpoint
and returns the file descriptor.
"""
query = self._queryString(statement)
buf = tempfile.NamedTemporaryFile()
opener = urllib2.build_opener()
opener.addheaders = self.headers().items()
request = self._build_request(query)
response = self._get_response(opener, request, buf,
timeout if timeout > 0 else None)
self._read_response(response, buf, timeout)
buf.seek(0)
return buf
def query(self, statement, timeout=0):
"""
Sends the request and starts the parser on the response.
"""
response = self._request(statement, timeout)
return _ResultsParser(response)
def _queryString(self, statement):
"""
Creates the REST query string from the statement and graphs.
"""
args = []
# refs #72876 removing the replace of newline to allow the comments in sparql queries
#statement = statement.replace("\n", " ").encode('utf-8')
pref = ' '.join(["PREFIX %s: <%s> " % (p, self._prefix_map[p]) for p in self._prefix_map])
statement = str(pref) + str(statement)
args.append(('query', statement))
for uri in self.defaultGraphs():
args.append(('default-graph-uri', uri))
for uri in self.namedGraphs():
args.append(('named-graph-uri', uri))
return urlencode(args)
class _ResultsParser(object):
"""
Parse the XML result.
"""
__allow_access_to_unprotected_subobjects__ = {'fetchone': 1,
'fetchmany': 1, 'fetchall': 1, 'hasresult': 1, 'variables': 1}
def __init__(self, fp):
self.__fp = fp
self._vals = []
self._hasResult = None
self.variables = []
self._fetchhead()
def __del__(self):
self.__fp.close()
def _fetchhead(self):
"""
Fetches the head information. If there are no variables in the
<head>, then we also fetch the boolean result.
"""
self.events = pulldom.parse(self.__fp)
for (event, node) in self.events:
if event == pulldom.START_ELEMENT:
if node.tagName == 'variable':
self.variables.append(node.attributes['name'].value)
elif node.tagName == 'boolean':
self.events.expandNode(node)
self._hasResult = (node.firstChild.data == 'true')
elif node.tagName == 'result':
return # We should not arrive here
elif event == pulldom.END_ELEMENT:
if node.tagName == 'head' and self.variables:
return
elif node.tagName == 'sparql':
return
def hasresult(self):
"""
ASK queries are used to test if a query would have a result. If the
query is an ASK query there won't be an actual result, and
:func:`fetchone` will return nothing. Instead, this method can be
called to check the result from the ASK query.
If the query is a SELECT statement, then the return value of
:func:`hasresult` is `None`, as the XML result format doesn't tell you
if there are any rows in the result until you have read the first one.
"""
return self._hasResult
def __iter__(self):
""" Synonim for :func:`fetchone`. """
return self.fetchone()
def fetchone(self):
""" Fetches the next set of rows of a query result, returning a list.
An empty list is returned when no more rows are available.
If the query was an ASK request, then an empty list is returned as
there are no rows available.
"""
idx = -1
for (event, node) in self.events:
if event == pulldom.START_ELEMENT:
if node.tagName == 'result':
self._vals = [None] * len(self.variables)
elif node.tagName == 'binding':
idx = self.variables.index(node.attributes['name'].value)
elif node.tagName == 'uri':
self.events.expandNode(node)
data = ''.join(t.data for t in node.childNodes)
self._vals[idx] = IRI(data)
elif node.tagName == 'literal':
self.events.expandNode(node)
data = ''.join(t.data for t in node.childNodes)
lang = node.getAttribute('xml:lang') or None
datatype = Datatype(node.getAttribute('datatype')) or None
self._vals[idx] = Literal(data, datatype, lang)
elif node.tagName == 'bnode':
self.events.expandNode(node)
data = ''.join(t.data for t in node.childNodes)
self._vals[idx] = BlankNode(data)
elif event == pulldom.END_ELEMENT:
if node.tagName == 'result':
#print "rtn:", len(self._vals), self._vals
yield tuple(self._vals)
def fetchall(self):
""" Loop through the result to build up a list of all rows.
Patterned after DB-API 2.0.
"""
result = []
for row in self.fetchone():
result.append(row)
return result
def fetchmany(self, num):
result = []
for row in self.fetchone():
result.append(row)
num -= 1
if num <= 0: return result
return result
def query(endpoint, query, timeout = 0, qs_encoding = "utf-8", method = "POST"):
"""
Convenient method to execute a query. Exactly equivalent to::
sparql.Service(endpoint).query(query)
"""
s = Service(endpoint, qs_encoding, method)
return s.query(query, timeout)
def _interactive(endpoint):
while True:
try:
lines = []
while True:
next = raw_input()
if not next:
break
else:
lines.append(next)
if lines:
sys.stdout.write("Querying...")
result = query(endpoint, " ".join(lines))
sys.stdout.write(" done\n")
for row in result.fetchone():
print ("\t".join(row))
print
lines = []
except Exception as e:
sys.stderr.write(str(e))
class SparqlException(Exception):
""" Sparql Exceptions """
def __init__(self, code, message):
self.code = code
self.message = message
if __name__ == '__main__':
import sys
import codecs
from optparse import OptionParser
try:
c = codecs.getwriter(sys.stdout.encoding)
except:
c = codecs.getwriter('ascii')
sys.stdout = c(sys.stdout, 'replace')
parser = OptionParser(usage="%prog [-i] endpoint",
version="%prog " + str(__version__))
parser.add_option("-i", dest="interactive", action="store_true",
help="Enables interactive mode")
(options, args) = parser.parse_args()
if len(args) != 1:
parser.error("Endpoint must be specified")
endpoint = args[0]
if options.interactive:
_interactive(endpoint)
q = sys.stdin.read()
try:
result = query(endpoint, q)
for row in result.fetchone():
print ("\t".join(row))
except SparqlException as e:
faultString = e.message
print >>sys.stderr, faultString
gargantext/util/parsers/HAL.py
View file @ ea1172a4
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
# **************************** # ****************************
# **** HAL Parser *** # **** HAL Parser ***
# **************************** # ****************************
# CNRS COPYRIGHTS # CNRS COPYRIGHTS 2017
# SEE LEGAL LICENCE OF GARGANTEXT.ORG # SEE LEGAL LICENCE OF GARGANTEXT.ORG
from ._Parser import Parser from ._Parser import Parser
......
gargantext/util/parsers/ISIDORE.py 0 → 100644
View file @ ea1172a4
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# ****************************
# **** ISIDORE Parser ***
# ****************************
# CNRS COPYRIGHTS
# SEE LEGAL LICENCE OF GARGANTEXT.ORG
from ._Parser import Parser
from datetime import datetime
import json
class IsidoreParser(Parser):
def parse(self, filebuf):
'''
parse :: FileBuff -> [Hyperdata]
'''
contents = filebuf.read().decode("UTF-8")
data = json.loads(contents)
filebuf.close()
json_docs = data
hyperdata_list = []
hyperdata_path = { "title" : "title"
, "abstract" : "abstract"
, "authors" : "authors"
, "url" : "url"
, "source" : "source"
}
uniq_id = set()
for doc in json_docs:
hyperdata = {}
for key, path in hyperdata_path.items():
hyperdata[key] = doc.get(path, "")
if hyperdata["url"] not in uniq_id:
# Removing the duplicates implicitly
uniq_id.add(hyperdata["url"])
# Source is the Journal Name
hyperdata["source"] = doc.get("source", "ISIDORE Database")
# Working on the date
maybeDate = doc.get("date" , None)
if maybeDate is None:
date = datetime.now()
else:
try :
# Model of date: 1958-01-01T00:00:00
date = datetime.strptime(maybeDate, '%Y-%m-%dT%H:%M:%S')
except :
print("FIX DATE ISIDORE please >%s<" % maybeDate)
date = datetime.now()
hyperdata["publication_date"] = date
hyperdata["publication_year"] = str(date.year)
hyperdata["publication_month"] = str(date.month)
hyperdata["publication_day"] = str(date.day)
hyperdata_list.append(hyperdata)
return hyperdata_list
gargantext/util/toolchain/parsing.py
View file @ ea1172a4
...@@ -175,7 +175,6 @@ def parse(corpus): ...@@ -175,7 +175,6 @@ def parse(corpus):
hyperdata = hyperdata, hyperdata = hyperdata,
) )
session.add(document) session.add(document)
session.commit()
documents_count += 1 documents_count += 1
if pending_add_error_stats: if pending_add_error_stats:
...@@ -190,6 +189,9 @@ def parse(corpus): ...@@ -190,6 +189,9 @@ def parse(corpus):
session.add(corpus) session.add(corpus)
session.commit() session.commit()
# Commit any pending document
session.commit()
# update info about the resource # update info about the resource
resource['extracted'] = True resource['extracted'] = True
#print( "resource n°",i, ":", d, "docs inside this file") #print( "resource n°",i, ":", d, "docs inside this file")
......
install/gargamelle/Debian.sh
View file @ ea1172a4
#!/bin/bash
### Update and install base dependencies ### Update and install base dependencies
echo "############ DEBIAN LIBS ###############" echo "############ DEBIAN LIBS ###############"
apt-get update && \ apt-get update && \
...@@ -32,26 +34,26 @@ update-locale LC_ALL=fr_FR.UTF-8 ...@@ -32,26 +34,26 @@ update-locale LC_ALL=fr_FR.UTF-8
libxml2-dev xml-core libgfortran-6-dev \ libxml2-dev xml-core libgfortran-6-dev \
libpq-dev \ libpq-dev \
python3.5 \ python3.5 \
python3-dev \ python3.5-dev \
python3-six python3-numpy python3-setuptools \ python3-six python3-numpy python3-setuptools \
python3-numexpr \ python3-numexpr \
python3-pip \ python3-pip \
libxml2-dev libxslt-dev zlib1g-dev libxml2-dev libxslt-dev zlib1g-dev libigraph0-dev
#libxslt1-dev #libxslt1-dev
UPDATE AND CLEAN # UPDATE AND CLEAN
apt-get update && apt-get autoclean apt-get update && apt-get autoclean
#NB: removing /var/lib will avoid to significantly fill up your /var/ folder on your native system #NB: removing /var/lib will avoid to significantly fill up your /var/ folder on your native system
######################################################################## ########################################################################
### PYTHON ENVIRONNEMENT (as ROOT) ### PYTHON ENVIRONNEMENT (as ROOT)
######################################################################## ########################################################################
#adduser --disabled-password --gecos "" gargantua #adduser --disabled-password --gecos "" gargantua
cd /srv/ cd /srv/
pip3 install virtualenv pip3 install virtualenv
virtualenv /srv/env_3-5 virtualenv /srv/env_3-5 -p /usr/bin/python3.5
echo 'alias venv="source /srv/env_3-5/bin/activate"' >> ~/.bashrc echo 'alias venv="source /srv/env_3-5/bin/activate"' >> ~/.bashrc
# CONFIG FILES # CONFIG FILES
...@@ -60,9 +62,9 @@ update-locale LC_ALL=fr_FR.UTF-8 ...@@ -60,9 +62,9 @@ update-locale LC_ALL=fr_FR.UTF-8
source /srv/env_3-5/bin/activate && pip3 install -r /srv/gargantext/install/gargamelle/requirements.txt && \ source /srv/env_3-5/bin/activate && pip3 install -r /srv/gargantext/install/gargamelle/requirements.txt && \
pip3 install git+https://github.com/zzzeek/sqlalchemy.git@rel_1_1 && \ pip3 install git+https://github.com/zzzeek/sqlalchemy.git@rel_1_1 && \
python3 -m nltk.downloader averaged_perceptron_tagger -d /usr/local/share/nltk_data python3 -m nltk.downloader averaged_perceptron_tagger -d /usr/local/share/nltk_data
chown gargantua:gargantua -R /srv/env_3-5 chown gargantua:gargantua -R /srv/env_3-5
####################################################################### #######################################################################
## POSTGRESQL DATA (as ROOT) ## POSTGRESQL DATA (as ROOT)
####################################################################### #######################################################################
......
install/gargamelle/django_configure.sh
View file @ ea1172a4
...@@ -14,7 +14,7 @@ echo "::::: DJANGO :::::" ...@@ -14,7 +14,7 @@ echo "::::: DJANGO :::::"
/bin/su gargantua -c 'source /env_3-5/bin/activate &&\ su gargantua -c 'source /srv/env_3-5/bin/activate &&\
echo "Activated env" &&\ echo "Activated env" &&\
/srv/gargantext/manage.py makemigrations &&\ /srv/gargantext/manage.py makemigrations &&\
/srv/gargantext/manage.py migrate && \ /srv/gargantext/manage.py migrate && \
...@@ -24,4 +24,4 @@ echo "::::: DJANGO :::::" ...@@ -24,4 +24,4 @@ echo "::::: DJANGO :::::"
/srv/gargantext/dbmigrate.py && \ /srv/gargantext/dbmigrate.py && \
/srv/gargantext/manage.py createsuperuser' /srv/gargantext/manage.py createsuperuser'
/usr/sbin/service postgresql stop service postgresql stop
install/gargamelle/nginx.config 0 → 100644
View file @ ea1172a4
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean
# file but keep this around for reference. Or just disable in sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
# the upstream component nginx needs to connect to
upstream gargantext {
server unix:///tmp/gargantext.sock; # for a file socket
#server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
client_max_body_size 800M;
client_body_timeout 12;
client_header_timeout 12;
keepalive_timeout 15;
send_timeout 10;
root /var/www/html;
# Add index.php to the list if you are using PHP
#index index.html index.htm index.nginx-debian.html;
server_name _ stable.gargantext.org gargantext.org ;
# Django media
location /media {
alias /var/www/gargantext/media; # your Django project's media files - amend as required
}
location /static {
alias /srv/gargantext_static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass gargantext;
include uwsgi_params;
}
#access_log off;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
}
server {
listen 80 ;
listen [::]:80;
server_name dl.gargantext.org ;
error_page 404 /index.html;
location / {
root /var/www/dl ;
proxy_set_header Host $host;
proxy_buffering off;
}
access_log /var/log/nginx/dl.gargantext.org-access.log;
error_log /var/log/nginx/dl.gargantext.org-error.log;
}
install/gargamelle/requirements.txt
View file @ ea1172a4
# try bottleneck # try bottleneck
eventlet==0.20.1
amqp==1.4.9 amqp==1.4.9
anyjson==0.3.3 anyjson==0.3.3
billiard==3.3.0.23 billiard==3.3.0.23
......
moissonneurs/isidore.py 0 → 100644
View file @ ea1172a4
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# ****************************
# ***** ISIDORE Crawler *****
# ****************************
RESOURCE_TYPE_ISIDORE = 12
from django.shortcuts import redirect, render
from django.http import Http404, HttpResponseRedirect, HttpResponseForbidden
from gargantext.constants import get_resource, load_crawler, QUERY_SIZE_N_MAX
from gargantext.models.nodes import Node
from gargantext.util.db import session
from gargantext.util.db_cache import cache
from gargantext.util.http import JsonHttpResponse
from gargantext.util.scheduling import scheduled
from gargantext.util.toolchain import parse_extract_indexhyperdata
def query( request):
'''get GlobalResults()'''
if request.method == "POST":
query = request.POST["query"]
source = get_resource(RESOURCE_TYPE_ISIDORE)
if source["crawler"] is not None:
crawlerbot = load_crawler(source)()
#old raw way to get results_nb
results = crawlerbot.scan_results(query)
#ids = crawlerbot.get_ids(query)
return JsonHttpResponse({"results_nb":crawlerbot.results_nb})
def save(request, project_id):
'''save'''
if request.method == "POST":
query = request.POST.get("query")
try:
N = int(request.POST.get("N"))
except:
N = 0
print(query, N)
#for next time
#ids = request.POST["ids"]
source = get_resource(RESOURCE_TYPE_ISIDORE)
if N == 0:
raise Http404()
if N > QUERY_SIZE_N_MAX:
N = QUERY_SIZE_N_MAX
try:
project_id = int(project_id)
except ValueError:
raise Http404()
# do we have a valid project?
project = session.query( Node ).filter(Node.id == project_id).first()
if project is None:
raise Http404()
user = cache.User[request.user.id]
if not user.owns(project):
return HttpResponseForbidden()
# corpus node instanciation as a Django model
corpus = Node(
name = query,
user_id = request.user.id,
parent_id = project_id,
typename = 'CORPUS',
hyperdata = { "action" : "Scrapping data"
, "language_id" : "fr"
}
)
#download_file
crawler_bot = load_crawler(source)()
#for now no way to force downloading X records
#the long running command
filename = crawler_bot.download(query)
corpus.add_resource(
type = source["type"]
#, name = source["name"]
, path = crawler_bot.path
)
session.add(corpus)
session.commit()
#corpus_id = corpus.id
try:
scheduled(parse_extract_indexhyperdata)(corpus.id)
except Exception as error:
print('WORKFLOW ERROR')
print(error)
try:
print_tb(error.__traceback__)
except:
pass
# IMPORTANT ---------------------------------
# sanitize session after interrupted transact
session.rollback()
# --------------------------------------------
return render(
template_name = 'pages/projects/wait.html',
request = request,
context = {
'user' : request.user,
'project': project,
},
)
data = [query_string,query,N]
print(data)
return JsonHttpResponse(data)
moissonneurs/urls.py
View file @ ea1172a4
...@@ -10,19 +10,15 @@ ...@@ -10,19 +10,15 @@
# moissonneurs == getting data from external databases # moissonneurs == getting data from external databases
# Available databases :
## Pubmed
## IsTex,
## CERN
from django.conf.urls import url from django.conf.urls import url
# Available databases :
import moissonneurs.pubmed as pubmed import moissonneurs.pubmed as pubmed
import moissonneurs.istex as istex import moissonneurs.istex as istex
import moissonneurs.cern as cern import moissonneurs.cern as cern
import moissonneurs.multivac as multivac import moissonneurs.multivac as multivac
import moissonneurs.hal as hal import moissonneurs.hal as hal
import moissonneurs.isidore as isidore
# TODO : ISIDORE # TODO : ISIDORE
...@@ -42,7 +38,7 @@ urlpatterns = [ url(r'^pubmed/query$' , pubmed.query ) ...@@ -42,7 +38,7 @@ urlpatterns = [ url(r'^pubmed/query$' , pubmed.query )
, url(r'^hal/query$' , hal.query ) , url(r'^hal/query$' , hal.query )
, url(r'^hal/save/(\d+)' , hal.save ) , url(r'^hal/save/(\d+)' , hal.save )
#, url(r'^isidore/query$' , isidore.query ) , url(r'^isidore/query$' , isidore.query )
#, url(r'^isidore/save/(\d+)' , isidore.save ) , url(r'^isidore/save/(\d+)' , isidore.save )
] ]
templates/pages/menu.html
View file @ ea1172a4
...@@ -367,7 +367,7 @@ ...@@ -367,7 +367,7 @@
<p> <p>
Gargantext Gargantext
<span class="glyphicon glyphicon-registration-mark" aria-hidden="true"></span> <span class="glyphicon glyphicon-registration-mark" aria-hidden="true"></span>
, version 3.0.6.8, , version 3.0.6.9.4,
<a href="http://www.cnrs.fr" target="blank" title="Institution that enables this project."> <a href="http://www.cnrs.fr" target="blank" title="Institution that enables this project.">
Copyrights Copyrights
<span class="glyphicon glyphicon-copyright-mark" aria-hidden="true"></span> <span class="glyphicon glyphicon-copyright-mark" aria-hidden="true"></span>
......
templates/pages/projects/overview.html
View file @ ea1172a4
...@@ -41,39 +41,42 @@ ...@@ -41,39 +41,42 @@
<div class="container theme-showcase" role="main"> <div class="container theme-showcase" role="main">
<div class="jumbotron"> <div class="jumbotron">
<div class="row"> <div class="row">
<div class="col-md-4"> <div class="col-md-4">
<h1> <h1>
<span class="glyphicon glyphicon-home" aria-hidden="true"></span> <span class="glyphicon glyphicon-home" aria-hidden="true"></span>
Projects Projects
</h1> </h1>
</div> </div>
<div class="col-md-3"></div> <div class="col-md-3"></div>
<div class="col-md-5"> <div class="col-md-5">
<p id="project" class="help"> <p id="project" class="help">
<br> <br>
<button id="add" type="button" class="btn btn-primary btn-lg help" data-container="body" data-toggle="popover" data-placement="bottom"> <button id="add" type="button" class="btn btn-primary btn-lg help" data-container="body" data-toggle="popover" data-placement="bottom">
<span class="glyphicon glyphicon-plus" aria-hidden="true"></span> <span class="glyphicon glyphicon-plus" aria-hidden="true"></span>
Add a new project Add a new project
</button> </button>
<div id="popover-content" class="hide"> <div id="popover-content" class="hide">
<div id="createForm" class="form-group"> <form>
{% csrf_token %} <div id="createForm" class="form-group">
<div id="status-form" class="collapse"> {% csrf_token %}
</div> <div id="status-form" class="collapse"></div>
<div class="row inline">
<label class="col-lg-3" for="inputName" ><span class="pull-right">Name:</span></label> <div class="row inline">
<input class="col-lg-8" type="text" id="inputName" class="form-control"> <label class="col-lg-3" for="inputName" ><span class="pull-right">Name:</span></label>
</div> <input class="col-lg-8" type="text" id="inputName" class="form-control">
</div>
<div class="row inline">
<div class="col-lg-3"></div> <div class="row inline">
<button id="createProject" class="btn btn-primary btn-sm col-lg-8 push-left">Add Project</button> <div class="col-lg-3"></div>
<div class="col-lg-2"></div> <button id="createProject" class="btn btn-primary btn-sm col-lg-8 push-left">Add Project</button>
<div class="col-lg-2"></div>
</div>
</div>
</form>
</div> </div>
</div> </p>
</div> </div>
</p>
</div> </div>
</div> </div>
</div> </div>
...@@ -87,7 +90,7 @@ ...@@ -87,7 +90,7 @@
</div> </div>
<!-- CHECKBOX EDITION --> <!-- CHECKBOX EDITION -->
<!-- <!--
<div class="row collapse" id="editor"> <div class="row collapse" id="editor">
<button title="delete selected project" type="button" class="btn btn-danger" id="delete"> <button title="delete selected project" type="button" class="btn btn-danger" id="delete">
<span class="glyphicon glyphicon-trash " aria-hidden="true" ></span> <span class="glyphicon glyphicon-trash " aria-hidden="true" ></span>
...@@ -98,9 +101,8 @@ ...@@ -98,9 +101,8 @@
<!-- <button type="button" class="btn btn-info" id="recalculate"> <!-- <button type="button" class="btn btn-info" id="recalculate">
<span class="glyphicon glyphicon-refresh " aria-hidden="true" onclick="recalculateProjects()"></span> <span class="glyphicon glyphicon-refresh " aria-hidden="true" onclick="recalculateProjects()"></span>
</button> </button>
-->
</div> </div>
-->
<br /> <br />
......
templates/pages/projects/project.html
View file @ ea1172a4
...@@ -675,7 +675,7 @@ ...@@ -675,7 +675,7 @@
$("#submit_thing").prop('disabled' , false) $("#submit_thing").prop('disabled' , false)
//$("#submit_thing").attr('onclick', testCERN(query, N)); //$("#submit_thing").attr('onclick', testCERN(query, N));
$("#submit_thing").on("click", function(){ $("#submit_thing").on("click", function(){
saveMultivac(pubmedquery, N); saveMultivac(pubmedquery, N, "/moissonneurs/multivac/save/");
//$("#submit_thing").onclick() //$("#submit_thing").onclick()
})} })}
//(N > {{query_size}}) //(N > {{query_size}})
...@@ -684,7 +684,7 @@ ...@@ -684,7 +684,7 @@
$('#submit_thing').prop('disabled', false); $('#submit_thing').prop('disabled', false);
$("#submit_thing").html("Processing a sample file") $("#submit_thing").html("Processing a sample file")
$("#submit_thing").on("click", function(){ $("#submit_thing").on("click", function(){
saveMultivac(pubmedquery, N); saveMultivac(pubmedquery, N,"/moissonneurs/multivac/save/" );
//$("#submit_thing").onclick() //$("#submit_thing").onclick()
})} })}
} }
...@@ -708,7 +708,6 @@ ...@@ -708,7 +708,6 @@
//HAL = 11 //HAL = 11
if (SourceTypeId == "11"){ if (SourceTypeId == "11"){
$.ajax({ $.ajax({
// contentType: "application/json", // contentType: "application/json",
...@@ -736,7 +735,7 @@ ...@@ -736,7 +735,7 @@
$("#submit_thing").prop('disabled' , false) $("#submit_thing").prop('disabled' , false)
//$("#submit_thing").attr('onclick', testCERN(query, N)); //$("#submit_thing").attr('onclick', testCERN(query, N));
$("#submit_thing").on("click", function(){ $("#submit_thing").on("click", function(){
saveALL(pubmedquery, N); save(pubmedquery, N, "/moissonneurs/hal/save/");
//$("#submit_thing").onclick() //$("#submit_thing").onclick()
})} })}
//(N > {{query_size}}) //(N > {{query_size}})
...@@ -745,7 +744,7 @@ ...@@ -745,7 +744,7 @@
$('#submit_thing').prop('disabled', false); $('#submit_thing').prop('disabled', false);
$("#submit_thing").html("Processing a sample file") $("#submit_thing").html("Processing a sample file")
$("#submit_thing").on("click", function(){ $("#submit_thing").on("click", function(){
saveALL(pubmedquery, N); save(pubmedquery, N, "/moissonneurs/hal/save/");
//$("#submit_thing").onclick() //$("#submit_thing").onclick()
})} })}
} }
...@@ -768,6 +767,69 @@ ...@@ -768,6 +767,69 @@
} }
//HAL = 12
if (SourceTypeId == "12"){
$.ajax({
// contentType: "application/json",
url: window.location.origin+"/moissonneurs/isidore/query",
data: formData,
type: 'POST',
beforeSend: function(xhr) {
xhr.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
},
success: function(data) {
console.log(data)
console.log("SUCCESS")
console.log("enabling "+"#"+value.id)
// $("#"+value.id).attr('onclick','getGlobalResults(this);');
$("#submit_thing").prop('disabled' , false)
//$("#submit_thing").html("Process a {{ query_size }} sample!")
N = data["results_nb"]
if(N > 0) {
if (N <= {{query_size}}){
$("#theresults").html("<i> <b>"+pubmedquery+"</b>: "+N+" publications </i><br>")
$("#submit_thing").html("Download!")
$("#submit_thing").prop('disabled' , false)
//$("#submit_thing").attr('onclick', testCERN(query, N));
$("#submit_thing").on("click", function(){
save(pubmedquery, N, "/moissonneurs/isidore/save/");
//$("#submit_thing").onclick()
})}
//(N > {{query_size}})
else {
$("#theresults").html("<i> <b>"+pubmedquery+"</b>: "+N+" publications </i><br>")
$('#submit_thing').prop('disabled', false);
$("#submit_thing").html("Processing a sample file")
$("#submit_thing").on("click", function(){
save(pubmedquery, N, "/moissonneurs/isidore/save/");
//$("#submit_thing").onclick()
})}
}
else {
$("#theresults").html("<i> <b>"+pubmedquery+"</b>: No results!.</i><br>")
if(data[0]==false)
$("#theresults").html(theType +" connection error!</i><br>")
$('#submit_thing').prop('disabled', true);
}
},
error: function(result) {
$("#theresults").html(theType +" connection error</i><br>")
$('#submit_thing').prop('disabled', true);
}
});
}
} }
// CSS events for selecting one Radio-Input // CSS events for selecting one Radio-Input
...@@ -819,6 +881,7 @@ ...@@ -819,6 +881,7 @@
|| selectedId == "9" || selectedId == "9"
|| selectedId == "10" || selectedId == "10"
|| selectedId == "11" || selectedId == "11"
|| selectedId == "12"
) { ) {
console.log("show the button for: " + selectedId) console.log("show the button for: " + selectedId)
$("#div-fileornot").css("visibility", "visible"); $("#div-fileornot").css("visibility", "visible");
...@@ -1001,7 +1064,7 @@ ...@@ -1001,7 +1064,7 @@
}); });
} }
function saveALL(query, N){ function save(query, N, urlGarg){
console.log("In Gargantext") console.log("In Gargantext")
if(!query || query=="") return; if(!query || query=="") return;
...@@ -1016,7 +1079,7 @@ ...@@ -1016,7 +1079,7 @@
console.log(data) console.log(data)
$.ajax({ $.ajax({
dataType: 'json', dataType: 'json',
url: window.location.origin+"/moissonneurs/hal/save/"+projectid, url: window.location.origin + urlGarg + projectid,
data: data, data: data,
type: 'POST', type: 'POST',
beforeSend: function(xhr) { beforeSend: function(xhr) {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment