Skip to content

haskell-gargantext
haskell-gargantext
Verified Commit cff88cad authored by Przemyslaw Kaminski's avatar Przemyslaw Kaminski
Browse files
Options

Merge branch 'dev' into 175-dev-doc-table-count-temp

parents 19a73c11 24c56bae
Expand all Show whitespace changes
Inline Side-by-side
Showing with 369 additions and 131 deletions
CHANGELOG.md
View file @ cff88cad
## Version 0.0.6.9.4.5
* [BACK][CLEAN] Switch to Hackage version of hstatistics
* [BACK][CLEAN] Drop unused servant-static-th
* [FRONT][CLEAN] README
* [FIX][CRALER] ArXiv API
## Version 0.0.6.9.4.4
* [BACK][Add optional Accelerate dependency on Darwin for pkgs.nix](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/merge_requests/128)
* [BACK][Crawlers] pubmed clean unused deps removed
* [BACK][CLEAN][Explore cutting down forks/extra-deps of libraries (#180)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/180)
* [BACK][DOC] README
## Version 0.0.6.9.4.3
* [BACK][DOC][Welcome: Door To enter the project (#177)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/177)
* [FRONT][DOC][Door to entry: Welcome! (#269)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/269)
* [BACK][REFACT][Doc Table: count (#175)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/175)
## Version 0.0.6.9.4.2
* [FRONT][FIX][Regression : on graph (#496)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/496)
......
README.md
View file @ cff88cad
<div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/images/logo.png"></div>
&nbsp;
# Gargantext with Haskell (Backend instance)
## About the project
![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=debian&color=6586c8)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
#### Table of Contents
1. [About the project](#about)
2. [Example2](#example2)
3. [Third Example](#third-example)
4. [Fourth Example](#fourth-examplehttpwwwfourthexamplecom)
## About the project <a name="about"></a>
GarganText is a collaborative web-decentralized-based macro-service
platform for the exploration of unstructured texts. It combines tools
from natural language processing, text-data-mining tricks, complex
networks analysis algorithms and interactive data visualization tools to
pave the way toward new kinds of interactions with your digital corpora.
from natural language processing, text-data-mining bricks, complex
networks analysis algorithms and interactive data visualization tools
to pave the way toward new kinds of interactions with your textual and
digital corpora.
This software is free software, developed and offered by the CNRS
Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
This software is free (as "Libre" in French) software, developed by the
CNRS Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
partners.
GarganText Project: this repo builds the
backend for the frontend server built by
GarganText Project: this repo builds the backend for the frontend server built by
[backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
## Installation
Disclaimer: this project is still in development, this is work in
progress. Please report and improve this documentation if you encounter issues.
Disclaimer: since this project is still in development, this document
remains in progress. Please report and improve this documentation if you
encounter any issues.
### Stack setup
### Prerequisite
You need to install stack first:
Clone the project.
```shell
git clone https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git
cd haskell-gargantext
```
### 1. Install Stack
Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
```shell
curl -sSL https://get.haskellstack.org/ | sh
......@@ -33,152 +52,85 @@ curl -sSL https://get.haskellstack.org/ | sh
Verify the installation is complete with
```shell
stack --version
Version 2.9.1
```
### With Nix setup
### 2. Install Nix
First install [nix](https://nixos.org/guides/install-nix.html):
Install [Nix](https://nixos.org/download.html):
```shell
curl -sSL https://nixos.org/nix/install | sh
$ sh <(curl -L https://nixos.org/nix/install) --daemon
```
Verify the installation is complete
Verify the installation is complete with
```shell
$ nix-env
nix-env (Nix) 2.3.12
```
And just build:
``` sh
stack --nix build --fast
$ nix-env --version
nix-env (Nix) 2.12.0
```
### Build Core Code
> **NOTE INFO (upgrade/downgrade if needed)**
> Gargantext works with Nix 2.12.0 (older version than current 2.13.2). To downgrade your Nix version:
> `nix-channel --update; nix-env -iA nixpkgs.nixVersions.nix_2_12 nixpkgs.cacert; systemctl daemon-reload; systemctl restart nix-daemon`
> Upgrading Nix: https://nixos.org/manual/nix/unstable/installation/upgrading.html
### 3. Build Core Code
NOTE: Default build (with optimizations) requires large amounts of RAM
(16GB at least). To avoid heavy compilation times and swapping out your
machine, it is recommended to `stack build` with the `--fast-` flag,
machine, it is recommended to `stack build` with the `--fast` flag,
i.e.:
``` sh
stack --nix build --fast
```
or
``` sh
stack --docker build --fast
```
#### Docker
``` sh
curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
```
#### Debian
``` sh
curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/debian/install | sh
```
#### Ubuntu
``` sh
curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/ubuntu/install | sh
```
If the build is finishing without error, you are ready to launch
GarganText! See next step.
### Add dependencies
&nbsp;
&nbsp;
&nbsp;
&nbsp;
&nbsp;
1. CoreNLP is needed (EN and FR); This dependency will not be needed soon.
``` sh
./devops/install-corenlp
```
### Launch Gargantext (It should have been initialized first)
Run
haskell-gargantext/devops/docker$ docker-compose up
Then
``` sh
stack --nix install
~/.local/bin/gargantext-init "gargantext.ini"
```
Or for Docker env, first create the appropriate image:
``` sh
cd devops/docker
docker build -t cgenie/stack-build:lts-18.12-garg .
```
then run:
``` sh
stack --docker exec gargantext-init -- gargantext.ini
### Initialization
#### Docker
Run PostgreSQL first:
Docker-compose will configure your database and some NLP bricks (such as CoreNLP):
``` sh
# If docker is not installed:
# curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
cd devops/docker
docker-compose up
docker compose up
```
Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
#### Gargantext
##### Fix the passwords
Change the passwords in gargantext.ini_toModify then move it:
``` sh
cp gargantext.ini_toModify gargantext.ini
```
(`.gitignore` avoids adding this file to the repository by mistake)
##### Run Gargantext
Users have to be created first (`user1` is created as instance):
Then install:
``` sh
stack --nix install
~/.local/bin/gargantext-init "gargantext.ini"
```
For Docker env, first create the appropriate image:
``` sh
cd devops/docker
docker build -t cgenie/stack-build:lts-18.12-garg .
```
then run:
Copy the configuration file:
``` sh
stack --docker exec gargantext-init -- gargantext.ini
cp gargantext.ini_toModify gargantext.ini
```
Do not worry, `.gitignore` avoids adding this file to the repository by
mistake, then you can change the passwords in gargantext.ini safely.
### Importing data
You can import some data with:
Users have to be created first (`user1` is created as instance):
``` sh
docker run --rm -it -p 9000:9000 cgenie/corenlp-garg
stack exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
~/.local/bin/gargantext-init "gargantext.ini"
```
### Nix
It is also possible to build everything with [Nix](https://nixos.org/) instead of Docker:
Launch GarganText:
``` sh
stack --nix build
stack --nix exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
stack --nix exec gargantext-server -- --ini gargantext.ini --run Prod
```
## Use Cases
### Multi-User with Graphical User Interface (Server Mode)
......@@ -212,7 +164,7 @@ cat repos/repo.cbor.v5 | stack --nix exec gargantext-cbor2json | jq .
To build documentation, run:
```sh
stack --docker build --haddock --no-haddock-deps --fast
stack --nix build --haddock --no-haddock-deps --fast
```
(in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
......
bin/track_haskell_deps.hs 0 → 100755
View file @ cff88cad
This diff is collapsed.
gargantext.cabal
View file @ cff88cad
......@@ -5,7 +5,7 @@ cabal-version: 1.12
-- see: https://github.com/sol/hpack
name: gargantext
version: 0.0.6.9.4.2
version: 0.0.6.9.4.5
synopsis: Search, map, share
description: Please see README.md
category: Data
......@@ -76,6 +76,7 @@ library
Gargantext.Core.Viz.Graph.Index
Gargantext.Core.Viz.Graph.Tools
Gargantext.Core.Viz.Graph.Tools.IGraph
Gargantext.Core.Viz.Graph.Types
Gargantext.Core.Viz.Phylo
Gargantext.Core.Viz.Phylo.API
Gargantext.Core.Viz.Phylo.API.Tools
......@@ -234,7 +235,6 @@ library
Gargantext.Core.Viz.Graph.Legend
Gargantext.Core.Viz.Graph.PatriciaTreeTypes
Gargantext.Core.Viz.Graph.Tools.Infomap
Gargantext.Core.Viz.Graph.Types
Gargantext.Core.Viz.Graph.Utils
Gargantext.Core.Viz.LegacyPhylo
Gargantext.Core.Viz.Phylo.Example
......@@ -431,7 +431,6 @@ library
, matrix
, monad-control
, monad-logger
, monad-logger-aeson
, morpheus-graphql
, morpheus-graphql-app
, morpheus-graphql-core
......@@ -480,7 +479,6 @@ library
, servant-mock
, servant-multipart
, servant-server
, servant-static-th
, servant-swagger
, servant-swagger-ui
, servant-xml
......
nix/pkgs.nix
View file @ cff88cad
......@@ -32,7 +32,9 @@ rec {
icu
graphviz
llvm_9
];
] ++ ( lib.optionals stdenv.isDarwin [
darwin.apple_sdk.frameworks.Accelerate
]);
libPaths = pkgs.lib.makeLibraryPath nonhsBuildInputs;
shellHook = ''
export LD_LIBRARY_PATH="${pkgs.gfortran7.cc.lib}:${libPaths}:$LD_LIBRARY_PATH"
......
package.yaml
View file @ cff88cad
......@@ -6,7 +6,7 @@ name: gargantext
# | | | +----- Layers * : New versions with API additions
# | | | | +--- Layers * : New versions without API breaking changes
# | | | | |
version: '0.0.6.9.4.2'
version: '0.0.6.9.4.5'
synopsis: Search, map, share
description: Please see README.md
category: Data
......@@ -101,6 +101,7 @@ library:
- Gargantext.Core.Viz.Graph.Index
- Gargantext.Core.Viz.Graph.Tools
- Gargantext.Core.Viz.Graph.Tools.IGraph
- Gargantext.Core.Viz.Graph.Types
- Gargantext.Core.Viz.Phylo
- Gargantext.Core.Viz.Phylo.API
- Gargantext.Core.Viz.Phylo.API.Tools
......@@ -215,7 +216,6 @@ library:
- matrix
- monad-control
- monad-logger
- monad-logger-aeson
- morpheus-graphql
- morpheus-graphql-app
- morpheus-graphql-core
......@@ -264,7 +264,6 @@ library:
- servant-mock
- servant-multipart
- servant-server
- servant-static-th
- servant-swagger
- servant-swagger-ui
- servant-xml
......
src-test/Graph/Clustering.hs
View file @ cff88cad
......@@ -13,7 +13,8 @@ Portability : POSIX
module Graph.Clustering where
import Gargantext.API.Ngrams.Types (NgramsTerm(..))
import Gargantext.Core.Viz.Graph (Graph(..), Strength(..))
import Gargantext.Core.Viz.Graph
import Gargantext.Core.Viz.Graph.Types
import Gargantext.Core.Viz.Graph.Tools (doSimilarityMap)
import Gargantext.Core.Viz.Graph.Tools.IGraph (spinglass)
import Gargantext.Prelude
......
stack.yaml
View file @ cff88cad
......@@ -42,10 +42,10 @@ extra-deps:
commit: 10a416b9f6c443866b36479c3441ebb3bcdeb7ef
- git: https://gitlab.iscpif.fr/gargantext/hlcm.git
commit: 6f0595d2421005837d59151a8b26eee83ebb67b5
- git: https://github.com/delanoe/hstatistics.git
commit: 90eef7604bb230644c2246eccd094d7bfefcb135
- git: https://github.com/paulrzcz/HSvm.git
commit: 3fe28b683aba5ddf05e3b5f8eced0bd05c5a29f9
- git: https://github.com/delanoe/hstatistics.git
commit: 90eef7604bb230644c2246eccd094d7bfefcb135
# API libs
- git: https://github.com/haskell-servant/servant.git
......@@ -58,8 +58,6 @@ extra-deps:
- servant-auth/servant-auth/
- servant-auth/servant-auth-client/
- servant-auth/servant-auth-server/
- git: https://github.com/delanoe/servant-static-th.git
commit: 8cb8aaf2962ad44d319fcea48442e4397b3c49e8
- git: https://github.com/alpmestan/ekg-json.git
commit: fd7e5d7325939103cd87d0dc592faf644160341c
......@@ -74,8 +72,7 @@ extra-deps:
# External Data API connectors
- git: https://gitlab.iscpif.fr/gargantext/crawlers/pubmed.git
commit: 31cb4d28dcb5d17274cede5e67b2a01914379129
#commit: 364885c891cbadcd4d8a623d2e41394b09f653aa
commit: 4ade495751eaf31d3ca1ac8b0ae13d3538c6e18c
- git: https://gitlab.iscpif.fr/gargantext/crawlers/istex.git
commit: a34bb341236d82cf3d488210bc1d8448a98f5808
- git: https://gitlab.iscpif.fr/gargantext/crawlers/hal.git
......@@ -83,7 +80,7 @@ extra-deps:
- git: https://gitlab.iscpif.fr/gargantext/crawlers/isidore.git
commit: 3db385e767d2100d8abe900833c6e7de3ac55e1b
- git: https://gitlab.iscpif.fr/gargantext/crawlers/arxiv-api.git
commit: a2d78abeaec9315be765b90d5e51a4a50c48e7b8
commit: 2d7e5753cbbce248b860b571a0e9885415c846f7
#- git: https://gitlab.iscpif.fr/cgenie/arxiv-api.git
#- arxiv-0.0.3@sha256:02de1114091d11f1f3ab401d104d125ad4301260806feb7f63b3dcefc7db88cf,1588
......@@ -178,3 +175,4 @@ ghc-options:
hmatrix: -O2 -fsimpl-tick-factor=10000 -fdicts-cheap -fdicts-strict -flate-dmd-anal -fno-state-hack
sparse-linear: -O2 -fsimpl-tick-factor=10000 -fdicts-cheap -fdicts-strict -flate-dmd-anal -fno-state-hack
gargantext-graph: -O2 -fsimpl-tick-factor=10000 -fdicts-cheap -fdicts-strict -flate-dmd-anal -fno-state-hack
"$locals": -fwrite-ide-info -hiedir=".stack-work/hiedb"
weeder.dhall 0 → 100644
View file @ cff88cad
{ roots = [ "^Main\\.main\$"
, "^Paths_.*"
], type-class-roots = True }
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment