Skip to content

gargantext
gargantext
Commit 72b8bdc4 authored by delanoe's avatar delanoe
Browse files
Options

Merge branch 'install' into unstable-install

parents 020485c9 5f2cbd8c
Hide whitespace changes
Inline Side-by-side
Showing with 427 additions and 211 deletions
docs/install.md
View file @ 72b8bdc4
#Install Instructions for Gargantext (CNRS):
## Get the source code
by cloning gargantext into /srv/gargantext
``` bash
git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
&& cd /srv/gargantext \
&& git fetch origin stable \
&& git checkout stable \
```
#Install Instructions for Gargamelle:
Gargamelle is the gargantext plateforme toolbox it is a full plateform system
with minimal modules
First you need to get the source code to install it
The folder will be /srv/gargantext:
* docs containes all informations on gargantext
/srv/gargantext/docs/
......@@ -20,151 +13,49 @@ The folder will be /srv/gargantext:
Help needed ?
See [http://gargantext.org/about](http://gargantext.org/about) and [tools](./contribution_guide.md) for the community
Two installation procedure are provided:
1. Semi-automatic installation [EASY]
2. Step by step installation [ADVANCED]
Here only semi-automatic installation is covered checkout [manual_install](manual_install.md)
to follow step by step procedure
##Prerequisites
## Init Setup
## Install
## Run
--------------------
# Semi automatic installation
All the procedure files are located into /srv/garantext/install/
``` bash
user@computer:$ cd /srv/garantext/install/
```
## Prerequisites
* A Debian based OS >= [FIXME]
* At least 35GO in /srv/ [FIXME]
todo: reduce the size of gargantext lib
todo: remove lib once docker is configured
! tip: if you have enought space for the full package you can:
* resize your partition
* make a simlink on gargantext_lib
##Init Setup
Prepare your environnement and make the initial setup.
This initial step creates a user for gargantext plateform along with dowloading additionnal libs and files.
It also install docker and build the docker image and build the gargantext box
``` bash
user@computer:/srv/garantext/install/$ .init.sh
```
### Install
Once the init step is done
* Enter into the docker environnement
Inside folder /srv/garantext/install/
enter the gargantext image
``` bash
user@computer:/srv/garantext/install/$ .docker/enterGargantextImage
```
go to the installation folder
``` bash
root@dockerimage8989809:$ cd /srv/gargantext/install/
```
[ICI] Tester si les config de postgresql et python sont faits en amont à la création du docker file
* Install Python environment
``` bash
root@dockerimage8989809:/srv/garantext/install/$ python/configure
```
* Configure PostgreSql
Inside the docker image, execute as root:
``` bash
root@computer:/srv/garantext/install/$ postgres/configure
```
[Si OK ] enlever ses lignes
## Get the source code
Install Gargantext server
by cloning gargantext into /srv/gargantext
* Configure the database
Inside the docker container:
``` bash
service postgresql start
#su gargantua
#activate the virtualenv
source /srv/env_3-5/bin/activate
```
You have entered the virtualenv as shown with (env_3-5)
``` bash
(env_3-5) $ python /srv/gargantext/dbmigrate.py
(env_3-5) $ /srv/gargantext/manage.py makemigrations
(env_3-5) $ /srv/gargantext/manage.py migrate
(env_3-5) $ python /srv/gargantext/dbmigrate.py
#will create tables and not hyperdata_nodes
(env_3-5) $ python /srv/gargantext/dbmigrate.py
#will create table hyperdata_nodes
#launch first time the server to create first user
(env_3-5) $ /srv/gargantext/manage.py runserver 0.0.0.0:8000
(env_3-5) $ /srv/gargantext/init_accounts.py /srv/gargantext/install/init/account.csv
```
FIXME: dbmigrate need to launched several times since tables are
ordered with alphabetical order (and not dependencies order)
* Exit the docker
```
exit (or Ctrl+D)
git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
&& cd /srv/gargantext \
&& git fetch origin stable \
&& git checkout stable \
```
## Install
``` bash
# go into the directory
user@computer: cd /srv/gargantext/
#git inside installation folder
user@computer: cd /install
#execute the installation
user@computer: ./install
```
During installation an admin account for gargantext will be created by asking you a username and a password
Remember it to accès to the Gargantext plateform
## Run Gargantext
## Run
Once you proceed to installation Gargantext plateforme will be available at localhost:8000
by running the run executable file
``` bash
# go into the directory
user@computer: cd /srv/gargantext/
#git inside installation folder
user@computer: cd /install
#execute the installation
user@computer: ./run
#type ctrl+d to exit or exit; command
```
Enter the docker container:
``` bash
/srv/gargantext/install/docker/enterGargantextImage
```
Inside the docker container:
``` bash
#start Database (postgresql)
service postgresql start
#change to user
su gargantua
#activate the virtualenv
source /srv/env_3-5/bin/activate
#go to gargantext srv
(env_3-5) $ cd /srv/gargantext/
#run the server
(env_3-5) $ /manage.py runserver 0.0.0.0:8000
```
Keep it open and outside the docker launch browser
Then open up a chromium browser and go to localhost:8000
Click on "Enter Gargantext"
Login in with you created username and pasword
``` bash
chromium http://127.0.0.1:8000/
```
Enjoy! ;)
* Click on Test Gargantext
```
Login : gargantua
Password : autnagrag
```
Enjoy :)
See [User Guide](/demo/tuto.md) for quick usage example
gargantext/constants.py
View file @ 72b8bdc4
......@@ -115,17 +115,19 @@ INDEXED_HYPERDATA = {
}
from gargantext.util.taggers import FrenchMeltTagger, TurboTagger
#from gargantext.util.taggers import FrenchMeltTagger, TurboTagger
from gargantext.util.taggers import NltkTagger
LANGUAGES = {
'en': {
#'tagger': EnglishMeltTagger,
'tagger': TurboTagger,
#'tagger': NltkTagger,
#'tagger': TurboTagger,
'tagger': NltkTagger,
},
'fr': {
'tagger': FrenchMeltTagger,
# 'tagger': TreeTagger,
#'tagger': FrenchMeltTagger,
#'tagger': TreeTagger,
'tagger': NltkTagger,
},
}
......
gargantext/util/crawlers/__init__.py 0 → 100644
View file @ 72b8bdc4
import importlib
from gargantext.constants import RESOURCETYPES
from gargantext.settings import DEBUG
#if DEBUG: print("Loading available Crawlers")
base_parser = "gargantext.util.crawlers"
for resource in RESOURCETYPES:
if resource["crawler"] is not None:
try:
name =resource["crawler"]
#crawler is type basename+"Crawler"
filename = name.replace("Crawler", "").lower()
module = base_parser+".%s" %(filename)
importlib.import_module(module,name, locals(), globals())
#if DEBUG: print("\t-", name)
except Exception as e:
print("Check constants.py RESOURCETYPES declaration %s \nCRAWLER %s is not available for %s" %(str(e), resource["crawler"], resource["name"]))
#initial import
#from .cern import CernCrawler
#from .istex import ISTexCrawler
#from .pubmed import PubmedCrawler
gargantext/util/taggers/NltkTagger.py
View file @ 72b8bdc4
#!/usr/bin/python env
from ._Tagger import Tagger
import nltk
#from nltk import pos_tag
from nltk.tag.perceptron import PerceptronTagger
class NltkTagger(Tagger):
'''Require maxtree'''
#import nltk
def __init__(self, *args, **kwargs):
self.tagr = PerceptronTagger()
super(self.__class__, self).__init__(*args, **kwargs)
#~ def __start__(self):
#~ self.tagr = PerceptronTagger()
def tag_tokens(self, tokens, single=True):
return nltk.pos_tag(tokens)
return self.tagr.tag(tokens)
gargantext/util/toolchain/metric_specificity.py 0 → 100644
View file @ 72b8bdc4
"""
Computes a specificity metric from the ngram cooccurrence matrix.
+ SAVE => WeightedList => NodeNgram
"""
from gargantext.models import Node, Ngram, NodeNgram, NodeNgramNgram
from gargantext.util.db import session, aliased, func, bulk_insert
from gargantext.util.lists import WeightedList
from collections import defaultdict
from pandas import DataFrame
import pandas as pd
def compute_specificity(corpus, cooc_id=None, cooc_matrix=None, overwrite_id = None):
'''
Compute the specificity, simple calculus.
Parameters:
- cooc_id: mandatory id of a cooccurrences node to use as base
- overwrite_id: optional preexisting specificity node to overwrite
'''
matrix = defaultdict(lambda : defaultdict(float))
if cooc_id == None and cooc_matrix == None:
raise TypeError("compute_specificity: needs a cooc_id or cooc_matrix param")
elif cooc_id:
cooccurrences = (session.query(NodeNgramNgram)
.filter(NodeNgramNgram.node_id==cooc_id)
)
# no filtering: cooc already filtered on mainlist_id at creation
for cooccurrence in cooccurrences:
matrix[cooccurrence.ngram1_id][cooccurrence.ngram2_id] = cooccurrence.weight
matrix[cooccurrence.ngram2_id][cooccurrence.ngram1_id] = cooccurrence.weight
elif cooc_matrix:
# copy WeightedMatrix into local matrix structure
for (ngram1_id, ngram2_id) in cooc_matrix.items:
w = cooc_matrix.items[(ngram1_id, ngram2_id)]
matrix[ngram1_id][ngram2_id] = w
nb_ngrams = len(matrix)
print("SPECIFICITY: computing on %i ngrams" % nb_ngrams)
x = DataFrame(matrix).fillna(0)
# proba (x/y) ( <= on divise chaque ligne par son total)
x = x / x.sum(axis=1)
# vectorisation
# d:Matrix => v: Vector (len = nb_ngrams)
# v = d.sum(axis=1) (- lui-même)
xs = x.sum(axis=1) - x
ys = x.sum(axis=0) - x
# top inclus ou exclus
#n = ( xs + ys) / (2 * (x.shape[0] - 1))
# top generic or specific (asc is spec, desc is generic)
v = ( xs - ys) / ( 2 * (x.shape[0] - 1))
## d ##
#######
# Grenelle biodiversité kilomètres site élus île
# Grenelle 0 0 4 0 0 0
# biodiversité 0 0 0 0 4 0
# kilomètres 4 0 0 0 4 0
# site 0 0 0 0 4 6
# élus 0 4 4 4 0 0
# île 0 0 0 6 0 0
## d.sum(axis=1) ##
###################
# Grenelle 4
# biodiversité 4
# kilomètres 8
# site 10
# élus 12
# île 6
# résultat temporaire
# -------------------
# pour l'instant on va utiliser les sommes en ligne comme ranking de spécificité
# (**même** ordre qu'avec la formule d'avant le refactoring mais calcul + simple)
# TODO analyser la cohérence math ET sem de cet indicateur
#v.sort_values(inplace=True)
# [ ('biodiversité' , 0.333 ),
# ('Grenelle' , 0.5 ),
# ('île' , 0.599 ),
# ('kilomètres' , 1.333 ),
# ('site' , 1.333 ),
# ('élus' , 1.899 ) ]
# ----------------
# specificity node
if overwrite_id:
# overwrite pre-existing id
the_id = overwrite_id
session.query(NodeNgram).filter(NodeNgram.node_id==the_id).delete()
session.commit()
else:
specnode = corpus.add_child(
typename = "SPECIFICITY",
name = "Specif (in:%s)" % corpus.id
)
session.add(specnode)
session.commit()
the_id = specnode.id
# print(v)
pd.options.display.float_format = '${:,.2f}'.format
if not v.empty:
data = WeightedList(
zip( v.index.tolist()
, v.values.tolist()[0]
)
)
data.save(the_id)
else:
print("WARNING: had no terms in COOCS => empty SPECIFICITY node")
return(the_id)
gargantext/util/toolchain/ngrams_extraction.py
View file @ 72b8bdc4
......@@ -48,7 +48,7 @@ def extract_ngrams(corpus, keys=('title', 'abstract', ), do_subngrams = DEFAULT_
ngrams_data = set()
# extract ngrams
resource_type_index = corpus.resources()[0]['type']
documents_count = 0
resource_type = RESOURCETYPES[resource_type_index]
default_language_iso2 = resource_type['default_language']
for documents_count, document in enumerate(corpus.children('DOCUMENT')):
......
install/01-setup deleted 100755 → 0
View file @ 020485c9
#!/bin/bash
#name:01-setup
#TODO clone the repo into /srv/gargantext/ and reduce the different steps
#git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
# && cd /srv/gargantext \
# && git fetch origin stable \
# && git checkout stable \
echo "****************SETUP**************************";
for dir in "/srv/gargantext_lib" "/srv/gargantext_static" "/srv/gargantext_media"; do
sudo mkdir -p $dir ;
sudo chown gargantua:gargantua $dir ;
done;
#here gargantext_lib.tar.bz2 inside this directory
#sudo wget http://dl.gargantext.org/gargantext_lib.tar.bz2 && \
sudo tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib \
&& sudo chown -R gargantua:gargantua /srv/gargantext_lib \
&& echo ":::::::::::::::::Done::::::::::::::::::::::::::";
install/README.md
View file @ 72b8bdc4
# Install
#Install Instructions for Gargamelle:
Gargamelle is a developpement environemment to install configure and run Gargantext plateform.
Gargamelle is the gargantext plateforme toolbox it is a full plateform system
with minimal modules
## Prerequisites
First you need to get the source code to install it
The folder will be /srv/gargantext:
* docs containes all informations on gargantext
/srv/gargantext/docs/
* install contains all the installation files
/srv/gargantext/install/
* MaC or Debian based OS
* Git
* Docker engine
Help needed ?
See [http://gargantext.org/about](http://gargantext.org/about) and [tools](./contribution_guide.md) for the community
## Installation Procedure
## Get the source code
* Clone the repository
If you want to contribute see our [git workflow procedure](../tools/git.md)
clone the repo and switch to your own branch named with username-branchname such as username-unstable for example
Inside the repository (install)
you will find the setup config and run procedure files
cd gargantext/install/
by cloning gargantext into /srv/gargantext
``` bash
git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
&& cd /srv/gargantext \
&& git fetch origin stable \
&& git checkout stable \
```
./01-setup
./02-config #here you will have to choose a username & a password for gargantext
./03-run
```
Open your browser Chrome perferably at localhost:8000
click into enter and login!
## Install
``` bash
# go into the directory
user@computer: cd /srv/gargantext/
#git inside installation folder
user@computer: cd /install
#execute the installation
user@computer: ./install
```
During installation an admin account for gargantext will be created by asking you a username and a password
Remember it to accès to the Gargantext plateform
Enjoy ;)!
## Run
Once you proceed to installation Gargantext plateforme will be available at localhost:8000
by running the run executable file
``` bash
# go into the directory
user@computer: cd /srv/gargantext/
#git inside installation folder
user@computer: cd /install
#execute the installation
user@computer: ./run
#type ctrl+d to exit or exit; command
```
Then open up a chromium browser and go to localhost:8000
Click on "Enter Gargantext"
Login in with you created username and pasword
Enjoy! ;)
install/gargamelle/.01-setup 0 → 100755
View file @ 72b8bdc4
#!/bin/bash
#name:01-setup
#TODO clone the repo into /srv/gargantext/ and reduce the different steps
#git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
# && cd /srv/gargantext \
# && git fetch origin stable \
# && git checkout stable
function clone_repo {
echo "* CLONE"
#echo "/home/"$USER"/ssh_keys/"$USER;
read -e -r -p "Please provide you ssh keys full directory:" response;
ssh-agent bash -c 'ssh-add ' $response '';
git clone ssh://gitolite@delanoe.org:1979/gargantext -o /srv/gargantext;
cd /srv/gargantext
git fetch origin stable
git pull
git checkout -b $USER'-stable';
echo "Switching to your own branch";
}
function download_lib {
echo "* DOWNLOAD additionnal lib"
echo "(May take a while, be patient or come back later when finished;)"
sudo wget -P install/ http://dl.gargantext.org/gargantext_lib.tar.bz2
#here gargantext_lib.tar.bz2 inside this directory
}
function uncompress_lib {
cd /srv/gargantext/install/
echo "* UNCOMPRESSING additionnal lib"
sudo tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib
sudo chown -R gargantua:gargantua /srv/gargantext_lib
}
#~ read -r -p "Do you want to clone the project? [y/N] " response
#~ case $response in
#~ [yY][eE][sS]|[yY])
#~ clone_repo
#~ ;;
#~ *)
#~ cd /srv/gargantext/
#~ ;;
#~ esac
echo "::: SETUP ENV :::";
for dir in "/srv/gargantext_lib" "/srv/gargantext_static" "/srv/gargantext_media"; do
sudo mkdir -p $dir ;
sudo chown gargantua:gargantua $dir ;
done;
#here gargantext_lib.tar.bz2 inside this directory
#sudo wget http://dl.gargantext.org/gargantext_lib.tar.bz2 && \
#sudo tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib \
#&& sudo chown -R gargantua:gargantua /srv/gargantext_lib \
#&& echo ":::::::::::::::::Done::::::::::::::::::::::::::";
echo ':::: BUILD ::::'
sudo docker build -t gargamelle:latest ./gargamelle
echo ':::: CONFIGURE ::::'
sudo docker run \
-v /srv/:/srv/ \
-p 8000:8000 \
-p 5432 \
-it gargamelle:latest \
/bin/bash -c "./psql_configure.sh; ./django_configure.sh ; exit"
sudo docker rm -f `docker ps -a | grep -v CONTAINER | awk '{print $1 }'`
install/02-config → install/gargamelle/.02-config
View file @ 72b8bdc4
#!/bin/bash
#configure the base image gargamelle
#echo '::::: BUILD :::::'
#echo ':::: BUILD ::::'
sudo docker build -t gargamelle:latest ./gargamelle
echo '::::: CONFIG :::::'
echo ':::: CONFIGURE ::::'
sudo docker run \
......@@ -16,4 +13,3 @@ sudo docker run \
/bin/bash -c "./psql_configure.sh; ./django_configure.sh ; exit"
sudo docker rm -f `docker ps -a | grep -v CONTAINER | awk '{print $1 }'`
install/gargamelle/Dockerfile
View file @ 72b8bdc4
......@@ -64,9 +64,10 @@ ADD requirements.txt /
ADD psql_configure.sh /
ADD django_configure.sh /
RUN . /env_3-5/bin/activate; pip3 install -r requirements.txt; \
pip3 install git+https://github.com/zzzeek/sqlalchemy.git@rel_1_1; \
python3 -m nltk.downloader averaged_perceptron_tagger;
RUN . /env_3-5/bin/activate && pip3 install -r requirements.txt && \
pip3 install git+https://github.com/zzzeek/sqlalchemy.git@rel_1_1 &&\
python3 -m nltk.downloader averaged_perceptron_tagger -d /usr/local/share/nltk_data;
# nltk.data.path.append('path_to_nltk_data')
RUN chown gargantua:gargantua -R /env_3-5
......
install/gargamelle/debug 0 → 100644
View file @ 72b8bdc4
#!/bin/bash
sudo docker run \
-v /srv/:/srv/\
-p 8000:8000 \
-p 5432 \
-it gargamelle:latest \
/bin/bash -c "service postgresql start; /bin/su gargantua -c 'source /env_3-5/bin/activate && /srv/gargantext/manage.py runserver 0.0.0.0:8000' && bin/bash"
sudo docker rm -f `docker ps -a | grep -v CONTAINER | awk '{print $1 }'`
install/gargamelle/django_configure.sh
View file @ 72b8bdc4
......@@ -24,7 +24,5 @@ echo "::::: DJANGO :::::"
./srv/gargantext/dbmigrate.py && \
./srv/gargantext/manage.py createsuperuser'
/usr/sbin/service postgresql stop
install/gargamelle/psql_configure.sh
View file @ 72b8bdc4
......@@ -13,6 +13,7 @@ su postgres -c 'pg_dropcluster 9.4 main --stop'
#done in docker but redoing it
rm -rf /srv/gargandata && mkdir /srv/gargandata && chown postgres:postgres /srv/gargandata
su postgres -c '/usr/lib/postgresql/9.5/bin/initdb -D /srv/gargandata/'
#su postgres -c '/usr/lib/postgresql/9.5/bin/pg_ctl -D /srv/gargandata/ -l journal_applicatif start'
su postgres -c '/usr/lib/postgresql/9.5/bin/pg_ctl -D /srv/gargandata/ -l journal_applicatif start'
su postgres -c 'pg_createcluster -D /srv/gargandata 9.5 main '
......@@ -25,5 +26,4 @@ su postgres -c "psql -c \"CREATE user gargantua WITH PASSWORD 'C8kdcUrAQy66U'\""
su postgres -c "createdb -O gargantua gargandb"
echo "Postgres configured"
#service postgresql stop
install/install 0 → 100755
View file @ 72b8bdc4
#!/bin/bash
#name:01-setup
#TODO clone the repo into /srv/gargantext/ and reduce the different steps
#git clone ssh://gitolite@delanoe.org:1979/gargantext /srv/gargantext \
# && cd /srv/gargantext \
# && git fetch origin stable \
# && git checkout stable
function clone_repo {
echo "* CLONE"
#echo "/home/"$USER"/ssh_keys/"$USER;
read -e -r -p "Please provide you ssh keys full directory:" response;
ssh-agent bash -c 'ssh-add ' $response '';
git clone ssh://gitolite@delanoe.org:1979/gargantext -o /srv/gargantext;
cd /srv/gargantext
git fetch origin stable
git pull
git checkout -b $USER'-stable';
echo "Switching to your own branch";
}
function download_lib {
echo "* DOWNLOAD additionnal lib"
echo "(May take a while, be patient or come back later when finished;)"
sudo wget -P install/ http://dl.gargantext.org/gargantext_lib.tar.bz2
#here gargantext_lib.tar.bz2 inside this directory
}
function uncompress_lib {
cd /srv/gargantext/install/
echo "* UNCOMPRESSING additionnal lib"
sudo tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib
sudo chown -R gargantua:gargantua /srv/gargantext_lib
}
#~ read -r -p "Do you want to clone the project? [y/N] " response
#~ case $response in
#~ [yY][eE][sS]|[yY])
#~ clone_repo
#~ ;;
#~ *)
#~ cd /srv/gargantext/
#~ ;;
#~ esac
echo "::: SETUP ENV :::";
for dir in "/srv/gargantext_lib" "/srv/gargantext_static" "/srv/gargantext_media"; do
sudo mkdir -p $dir ;
sudo chown gargantua:gargantua $dir ;
done;
#here gargantext_lib.tar.bz2 inside this directory
#sudo wget http://dl.gargantext.org/gargantext_lib.tar.bz2 && \
#sudo tar xvjf gargantext_lib.tar.bz2 -o /srv/gargantext_lib \
#&& sudo chown -R gargantua:gargantua /srv/gargantext_lib \
#&& echo ":::::::::::::::::Done::::::::::::::::::::::::::";
echo ':::: BUILD ::::'
sudo docker build -t gargamelle:latest ./gargamelle
echo ':::: CONFIGURE ::::'
sudo docker run \
-v /srv/:/srv/ \
-p 8000:8000 \
-p 5432 \
-it gargamelle:latest \
/bin/bash -c "./psql_configure.sh; ./django_configure.sh ; exit"
sudo docker rm -f `docker ps -a | grep -v CONTAINER | awk '{print $1 }'`
install/run 0 → 100755
View file @ 72b8bdc4
#!/bin/bash
sudo docker run \
-v /srv/:/srv/\
-p 8000:8000 \
-p 5432 \
-it gargamelle:latest \
/bin/bash -c "service postgresql start; /bin/su gargantua -c 'source /env_3-5/bin/activate && /srv/gargantext/manage.py runserver 0.0.0.0:8000'"
sudo docker rm -f `docker ps -a | grep -v CONTAINER | awk '{print $1 }'`
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment