README.md 8.83 KB
Newer Older
Fabien Manière's avatar
Fabien Manière committed
1
<div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/raw/master/images/logo.png"></div>
fabien's avatar
fabien committed
2 3

&nbsp;
4
# Gargantext with Haskell (Backend instance)
5

6
![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=nixos&color=6586c8)&nbsp;&nbsp;![Cabal](https://img.shields.io/badge/Tools-Cabal-informational?style=flat&logo=cabal&color=567dd9)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=stack&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
fabien's avatar
fabien committed
7

Fabien Manière's avatar
Fabien Manière committed
8 9
#### Table of Contents
1. [About the project](#about)
Fabien Manière's avatar
Fabien Manière committed
10 11 12 13 14 15
2. [Installation](#install)
3. [Initialization](#init)
4. [Launch & develop GarganText](#launch)
5. [Uses cases](#use-cases)
6. [GraphQL](#graphql)
7. [PostgreSQL](#postgresql)
Fabien Manière's avatar
Fabien Manière committed
16

Fabien Manière's avatar
Fabien Manière committed
17
## About the project <a name="about"></a>
18

19
GarganText is a collaborative web-decentralized-based macro-service platform for the exploration of unstructured texts. It combines tools from natural language processing, text-data-mining bricks, complex networks analysis algorithms and interactive data visualization tools to pave the way toward new kinds of interactions with your textual and digital corpora.
20

21
This software is free (as "Libre" in French) software, developed by the CNRS Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its partners.
22

23
GarganText Project: this repo builds the backend for the frontend server built by [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
24 25


Fabien Manière's avatar
Fabien Manière committed
26
## Installation <a name="install"></a>
27

28
Disclaimer: since this project is still in development, this document remains in progress. Please report and improve this documentation if you encounter any issues.
29

Fabien Manière's avatar
Fabien Manière committed
30 31 32 33 34 35 36 37 38 39
#### Prerequisites

- Install:
  - git (https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
  - curl (https://everything.curl.dev/get)
- Clone the project.
  ```shell
  git clone https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git
  cd haskell-gargantext
  ```
40
#### 1. Installation
41

42 43
This project can be built with either Stack or Cabal. For historical reasons, we generate a `cabal.project` from the `stack.yaml`, and we do not commit the former to the repo, to have a single "source of truth".
However, it's always possible to generate a `cabal.project` thanks to [stack2cabal](https://hackage.haskell.org/package/stack2cabal).
44

45
#### Install Nix 
46

47
Gargantext requires [Nix](https://github.com/NixOS/nix) to provide system dependencies (for example, C libraries), but its use is limited to that. In order to install [Nix](https://nixos.org/download.html):
48 49

```shell
50
sh <(curl -L https://nixos.org/nix/install) --daemon
51 52 53 54
```

Verify the installation is complete with
```shell
55
nix-env --version
Fabien Manière's avatar
Fabien Manière committed
56
nix-env (Nix) 2.16.0
57 58
```

59
**Important:** Before building the project with either `stack` or `cabal` you need to be in the correct Nix shell, which will fetch all the required system dependencies. To do so, just type:
60

61 62 63 64 65 66
```shell
nix-shell
```

This will take a bit of time the first time.

Fabien Manière's avatar
Fabien Manière committed
67
#### Build: choose cabal (new) or stack (old)
68

Fabien Manière's avatar
Fabien Manière committed
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91
#### With Cabal (recommanded)

First, into `nix-shell`:
```shell
cabal update
cabal install
```


Once you have a valid version of `cabal`, building requires generating a valid `cabal.project`. This can be done by installing `stack2cabal`:

```shell
cabal v2-install stack2cabal-1.0.14
```

And finally:

```shell
stack2cabal --no-run-hpack -p '2023-06-25'
cabal v2-build
```

#### With Stack
92 93

Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
94 95

```shell
96
curl -sSL https://get.haskellstack.org/ | sh
97 98
```

Alexandre Delanoë's avatar
Alexandre Delanoë committed
99
Verify the installation is complete with
100
```shell
101 102
stack --version
Version 2.9.1
103 104
```

105
NOTE: Default build (with optimizations) requires large amounts of RAM (16GB at least). To avoid heavy compilation times and swapping out your machine, it is recommended to `stack build` with the `--fast` flag, i.e.:
106

107 108
```shell
stack build --fast
109
```
Fabien Manière's avatar
Fabien Manière committed
110 111 112



113 114 115 116 117 118 119 120
#### Keeping the cabal.project updated with stack.yaml

Simply run:

```shell
./bin/update-cabal-project
```

Fabien Manière's avatar
Fabien Manière committed
121
## Initialization <a name="init"></a>
Alexandre Delanoë's avatar
Alexandre Delanoë committed
122

Fabien Manière's avatar
Fabien Manière committed
123
#### 1. Docker-compose will configure your database and some NLP bricks (such as CoreNLP):
124 125

``` sh
Alexandre Delanoë's avatar
Alexandre Delanoë committed
126 127
# If docker is not installed:
# curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
128
cd devops/docker
Alexandre Delanoë's avatar
Alexandre Delanoë committed
129
docker compose up
130 131 132
```
Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).

133
##### (Optional) If using stack, then install:
134
``` sh
135
stack install
136 137
```

138
#### 2. Copy the configuration file:
139
``` sh
Alexandre Delanoë's avatar
Alexandre Delanoë committed
140
cp gargantext.ini_toModify gargantext.ini
141
```
Fabien Manière's avatar
Fabien Manière committed
142
> Do not worry, `.gitignore` avoids adding this file to the repository by mistake, then you can change the passwords in gargantext.ini safely.
143

144
#### 3. A user have to be created first as instance:
145
``` sh
Alexandre Delanoë's avatar
Alexandre Delanoë committed
146
~/.local/bin/gargantext-init "gargantext.ini"
147
```
Fabien Manière's avatar
Fabien Manière committed
148
Now, `user1` is created with password `1resu`
149

150
#### 4. Clone FRONTEND repository:
Fabien Manière's avatar
Fabien Manière committed
151 152 153 154 155 156

From the Backend root folder (haskell-gargantext):

```shell
git clone ssh://git@gitlab.iscpif.fr:20022/gargantext/purescript-gargantext.git
```
Fabien Manière's avatar
Fabien Manière committed
157
&nbsp;
Fabien Manière's avatar
Fabien Manière committed
158

Fabien Manière's avatar
Fabien Manière committed
159
## Launch & develop GarganText <a name="launch"></a>
Fabien Manière's avatar
Fabien Manière committed
160

161 162 163
>  **Note:** here, the method with Cabal is used as default


164
From the Backend root folder (haskell-gargantext):
Fabien Manière's avatar
Fabien Manière committed
165 166

``` shell
167
./start
168 169 170
# The start script runs following commands:
# - `docker compose up` to run the Docker for postgresql from devops/docker folder
# - `cabal run gargantext-server -- --ini gargantext.ini --run Prod` to run other services through `nix-shell`
Fabien Manière's avatar
Fabien Manière committed
171 172
```

173
For frontend development and compilation, see the [Frontend Readme.md](https://gitlab.iscpif.fr/gargantext/purescript-gargantext#dev)
174

Alexandre Delanoë's avatar
Alexandre Delanoë committed
175

Fabien Manière's avatar
Fabien Manière committed
176
## Use Cases <a name="use-cases"></a>
177

178
### Multi-User with Graphical User Interface (Server Mode)
179

180
``` sh
181
~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
182 183
```

Fabien Manière's avatar
Fabien Manière committed
184
Then you can log in with `user1` / `1resu`
185 186 187 188 189 190


### Command Line Mode tools

#### Simple cooccurrences computation and indexation from a list of Ngrams

191
``` sh
192
stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
193
```
194

195 196
### Analyzing the ngrams table repo

197
We store the repository in directory `repos` in the [CBOR](https://cbor.io/) file format. To decode it to JSON and analyze, say, using [jq](https://shapeshed.com/jq-json/), use the following command:
198 199

``` sh
200
cat repos/repo.cbor.v5 | stack exec gargantext-cbor2json | jq .
201
```
202 203 204 205 206
### Documentation

To build documentation, run:

```sh
207
stack build --haddock --no-haddock-deps --fast
208 209
```

210 211
(in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).

Fabien Manière's avatar
Fabien Manière committed
212
## GraphQL <a name="graphql"></a>
213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244

Some introspection information.

Playground is located at http://localhost:8008/gql

### List all GraphQL types in the Playground

```
{
  __schema {
    types {
      name
    }
  }
}
```

### List details about a type in GraphQL

```
{
  __type(name:"User") {
  	fields {
    	name
      description
      type {
        name
      }
  	}
	}
}
```
Fabien Manière's avatar
Fabien Manière committed
245
## PostgreSQL <a name="pgsql"></a>
246 247

### Upgrading using Docker
248 249 250 251 252 253 254 255

https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/

To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
```sh
docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
```

256
Then, shut down the container, replace `image` section in `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
257 258 259 260 261 262
```sh
docker-compose rm postgres
docker volume rm docker_garg-pgdata
```

Now, start the container and execute:
263 264 265 266 267 268 269 270
```sh
# need to drop the empty DB first, since schema will be created when restoring the dump
docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
# recreate the db, but empty with no schema
docker exec -i <new-container-id> createdb -U gargantua gargandbV5
# now we can restore the dump
docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
```
271 272 273

### Upgrading using 

274
There is a solution using pgupgrade_cluster but you need to manage the clusters version 14 and 13. Hence here is a simple solution to upgrade.
275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296

First save your data:
```
sudo su postgres
pg_dumpall > gargandb.dump
```

Upgrade postgresql:
```
sudo apt install postgresql-server-14 postgresql-client-14
sudo apt remove --purge postgresql-13
```
Restore your data:
```
sudo su postgres
psql < gargandb.dump
```

Maybe you need to restore the gargantua password
```
ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
```
297
Maybe you need to change the port to 5433 for database connection in your gargantext.ini file.
298 299 300 301