Merge branch 'dev-README-update-with-postgres-system-config' into 'dev'

README: add info about system postgresql configuration

See merge request !403

README.md

 <div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/raw/master/images/logo.png"></div>
 
 &nbsp;
+
 # Gargantext with Haskell (Backend instance)
 
 ![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=nixos&color=6586c8)&nbsp;&nbsp;![Cabal](https://img.shields.io/badge/Tools-Cabal-informational?style=flat&logo=cabal&color=567dd9)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=stack&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
 
 #### Table of Contents
+
 1. [About the project](#about)
 2. [Installation](#install)
 3. [Use cases](#use-cases)
@@ -23,7 +25,6 @@ This software is free (as "Libre" in French) software, developed by the CNRS Com
 
 GarganText Project: this repo builds the backend for the frontend server built by [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
 
-
 ## Installation <a name="install"></a>
 
 Disclaimer: since this project is still in development, this document remains in progress. Please report and improve this documentation if you encounter any issues.
@@ -31,6 +32,7 @@ Disclaimer: since this project is still in development, this document remains in
 ### Prerequisites
 
 You need to have the following installed:
+
 * [GHCup](https://www.haskell.org/ghcup/)
 * [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
 * [Nix](https://nixos.org/download/)
@@ -39,6 +41,7 @@ You need to have the following installed:
 You will need to add yourself as a member of the `docker` group for Docker
 Compose to work (replace `myusername` with your username in the following
 command):
+
 ```bash
 sudo usermod -a -G docker myusername
 ```
@@ -57,14 +60,17 @@ cd ..
 #### Building the Backend
 
 Run the install script from inside the `haskell-gargantext/` directory:
+
 ```bash
 ./bin/install
 ```
+
 This will build Gargantext, run the tests, and build the documentation.
 
 #### Building the Frontend
 
 Move into the `purescript-gargantext/` directory, then run the install script:
+
 ```bash
 cd purescript-gargantext
 ./bin/install
@@ -83,6 +89,7 @@ and fill in the missing fields.
 **NOTE** If you had the `gargantext.ini` file before, you can automatically
 convert it into a file `gargantext-settings.toml` by running the following
 from a Nix shell:
+
 ```shell
 n$ ./bin/cli ini
 ```
@@ -90,6 +97,7 @@ n$ ./bin/cli ini
 ### Launching and Initializing the Database
 
 Launch the docker image from inside the dedicated directory:
+
 ```bash
 cd devops/docker/
 docker compose up
@@ -97,6 +105,7 @@ docker compose up
 
 Then initialize the database using the dedicated command: from inside the
 `haskell-gargantext/` directory, run
+
 ```bash
 ./bin/cli init
 ```
@@ -106,10 +115,13 @@ And provide a name and a password for the master user as requested.
 ### Launching CoreNLP
 
 CoreNLP can be started with nix:
+
 ```shell
 nix-shell --run startCoreNLPServer.sh
 ```
+
 By default, this starts on port `9000`. If you want a different port, then do:
+
 ```shell
 nix-shell --run 'startCoreNLPServer.sh -p 9001'
 ```
@@ -117,6 +129,7 @@ nix-shell --run 'startCoreNLPServer.sh -p 9001'
 ### Running Gargantext
 
 From inside the `haskell-gargantext/` directory, run
+
 ```bash
 ./bin/run
 ```
@@ -151,6 +164,7 @@ We store the repository in directory `repos` in the [CBOR](https://cbor.io/) fil
 ``` shell
 $ cat repos/repo.cbor.v5 | stack exec gargantext-cbor2json | jq .
 ```
+
 ### Documentation
 
 To build documentation, run:
@@ -161,7 +175,6 @@ $ stack build --haddock --no-haddock-deps --fast
 
 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 
-
 ## Notes for developers <a name="develop"></a>
 
 ### Developer Documentation
@@ -178,15 +191,15 @@ Upgrading packages can be a pain sometimes, with cabal.
 
 Here are some tips:
 
-- Manually remove entries from your `cabal.project.freeze` to make the
+* Manually remove entries from your `cabal.project.freeze` to make the
   build a bit more "elastic";
-- Lock the hackage-index state in the `cabal.project`, so that the
+* Lock the hackage-index state in the `cabal.project`, so that the
   solver won't try to pull newer dependencies;
-- Specify constraints you want directly when building like `cabal
+* Specify constraints you want directly when building like `cabal
   v2-build --constraint tasty==x.y.z.w`
-- Generate another `.freeze` with `cabal v2-freeze` once you got the
+* Generate another `.freeze` with `cabal v2-freeze` once you got the
   new build to compile (this is good for small, incremental upgrades)
-- Bounds in `.cabal` are definitely respected, but ofc the `.freeze`
+* Bounds in `.cabal` are definitely respected, but ofc the `.freeze`
   takes priority, so you want to maybe use `cabal gen-bounds` when
   your `.freeze` still exists, remove the file, try again.
 
@@ -198,9 +211,11 @@ the `.cabal` file versions.
 Occasionally, you can get issues with the `allow-newer: *` constraint
 from `cabal.project`. E.g. when I was building with GHC 9.4.7, I had
 errors with `hashable-1.5.0`. The solution is:
+
 ```shell
 cabal v2-build --constraint hashable==1.4.3.0
 ```
+
 (we don't depend on `hashable` directly, but `allow-newer: *` is so
 liberal that a package that is too new is used).
 
@@ -209,6 +224,7 @@ Overall, it's preferred to specify strict constraints in
 to have an idea what works.
 
 If you want to see the detailed build info for a given dependency:
+
 ```shell
 cabal v2-build -v servant-server
 ```
@@ -229,15 +245,19 @@ n$ cabal v2-test --test-show-details=streaming
 ```
 
 Or, from "outside":
+
 ```shell
 $ nix-shell --run "cabal v2-test --test-show-details=streaming"
 ```
 
 If you want to run particular tests, use (for Tasty):
+
 ```shell
 cabal v2-test garg-test-tasty --test-show-details=streaming --test-option=--pattern='/job status update and tracking/
 ```
+
 or (for Hspec):
+
 ```shell
 cabal v2-test garg-test-hspec --test-show-details=streaming --test-option=--match='/Dispatcher, Central Exchange, WebSockets/'
 ```
@@ -247,16 +267,20 @@ cabal v2-test garg-test-hspec --test-show-details=streaming --test-option=--matc
 The CI is on gitlab and is controlled by the `.gitlab-ci.yml` file.
 
 The Docker image that is used can be built with:
+
 ```shell
 podman build -t cgenie/gargantext:9.6.6 --pull -f ./devops/docker/Dockerfile --format=docker .
 ```
+
 or
+
 ```shell
 docker build -t cgenie/gargantext:9.6.6 --pull -f ./devops/docker/Dockerfile .
 ```
 
 NOTE: if podman/docker complains about not enough space, it's probably
 about tmpfs. In that case you can set
+
 ```shell
 mkdir _build
 export TMPDIR=$(pwd)/_build
@@ -269,14 +293,14 @@ When a development is needed on libraries (for instance, the HAL crawler in http
 
 1. Ongoing devlopment (on local repo):
    1. In `cabal.project`:
-      - add `../hal` to `packages:`
-      - turn off (temporarily) the `hal` in `source-repository-package` 
+      * add `../hal` to `packages:`
+      * turn off (temporarily) the `hal` in `source-repository-package`
    2. When changes work and tests are OK, commit in repo `hal`
 2. When changes are commited / merged:
    1. Get the hash id, and edit `cabal.project` with the **new commit id**
    2. run `nix-shell --run ./bin/update-project-dependencies`
-      - get an error that sha256 don't match, so update the `./bin/update-project-dependencies` with new sha256 hash
-      - run again `nix-shell --run ./bin/update-project-dependencies` (to make sure it's a fixed point now)
+      * get an error that sha256 don't match, so update the `./bin/update-project-dependencies` with new sha256 hash
+      * run again `nix-shell --run ./bin/update-project-dependencies` (to make sure it's a fixed point now)
 
 > Note: without `stack.yaml` we would have to only fix `cabal.project` -> `source-repository-package` commit id. Sha256 is there to make sure CI reruns the tests.
 
@@ -313,24 +337,49 @@ Playground is located at http://localhost:8008/gql
 	}
 }
 ```
+
 ### PostgreSQL <a name="pgsql"></a>
 
+#### Setting up system PostgreSQL (no Docker)
+
+If you use system PostgreSQL, make sure to:
+
+1. install `postgresql-contrib` package as well (with official postgres extensions such as `hstore`, `pgcrypto`, `fts` etc)
+2. create `gargantua` user: `CREATE USER gargantua WITH ENCRYPTED PASSWORD 'xxxx'` (as `postgres`)
+3. grant `CREATEDB` permissions (needed for`gargandb` and `pgmq`):`ALTER USER gargantua CREATEDB` (as `postgres`)
+   (If you don't want to grant such a permission, just create `gargandb` and `pgmq` databases and make `gargantua` their owner).
+   Also, note that `pgmq` works in a separate `pgmq` schema, so if you want, you can just use the `gargandb` database for it.
+4. alter `pg_hba.conf` to have something like this:
+
+   ```conf
+   host    all             all             127.0.0.1/32            md5
+   host    all             all             ::1/128                 md5
+   ```
+
+5. make sure you can log in: `psql -h localhost -U gargantua -d gargandb -W`
+6. apply the schema: `psql -h localhost -U gargantua -d gargandb -W < devops/docker/schema.sql`
+7. modify `gargantext-settings.toml` in sections `[database]` and `[worker.database]` according to your setup
+8. you want to create the `gargantua` master user: `cabal v2-run gargantext -- init`
+
 #### Upgrading using Docker
 
 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 
 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
+
 ```shell
 $ docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 ```
 
 Then, shut down the container, replace `image` section in `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
+
 ```shell
 $ docker-compose rm postgres
 $ docker volume rm docker_garg-pgdata
 ```
 
 Now, start the container and execute:
+
 ```shell
 $ # need to drop the empty DB first, since schema will be created when restoring the dump
 $ docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
@@ -340,28 +389,33 @@ $ # now we can restore the dump
 $ docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 ```
 
-#### Upgrading using 
+#### Upgrading using
 
 There is a solution using pgupgrade_cluster but you need to manage the clusters version 14 and 13. Hence here is a simple solution to upgrade.
 
 First save your data:
+
 ```shell
 $ sudo su postgres
 $ pg_dumpall > gargandb.dump
 ```
 
 Upgrade postgresql:
+
 ```shell
 $ sudo apt install postgresql-server-14 postgresql-client-14
 $ sudo apt remove --purge postgresql-13
 ```
+
 Restore your data:
+
 ```shell
 $ sudo su postgres
 $ psql < gargandb.dump
 ```
 
 Maybe you need to restore the gargantua password
+
 ```shell
 $ ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext-settings.toml'
 ```
@@ -399,15 +453,19 @@ cabal v2-install gargantext:exe:gargantext
 ```
 
 For tasty, if you want to run specific test (via patterns), use:
+
 ```shell
 cabal v2-run garg-test-tasty -- -p '/Ngrams/
 ```
+
 For integration tests, do:
+
 ```shell
 cabal v2-test garg-test-hspec --test-show-details=streaming --test-option=--match='/some pattern/'
 ```
 
 You could also use [ghciwatch](https://mercurytechnologies.github.io/ghciwatch/) for testsing:
+
 ```shell
 ghciwatch --command "cabal v2-repl garg-test-tasty" --after-startup-ghci ':set args "--pattern" "/Ngrams/"' --after-startup-ghci "Main.main" --after-reload-ghci "Main.main" --watch src --watch test
 ```
@@ -428,18 +486,22 @@ cabal v2-test garg-test-tasty --test-show-details=streaming --flags 'test-crypto
 ## Steps to configure the worker
 
 1. First create our Worker
+
 ```bash
 createdb gargantext_pgmq
 psql -c "ALTER DATABASE \"gargantext_pgmq\" OWNER TO \"gargantua\""
 ```
+
 2. Then configure you gargantext-settings.toml (see next section)
 
 3. Finally launch the worker
+
 ```bash
 nix-shell --run "cabal v2-run gargantext -- worker run --name default"
 ```
 
 Or launch all worker definitions at once:
+
 ```bash
 nix-shell --run "cabal v2-run gargantext -- worker run-all"
 ```
@@ -447,6 +509,7 @@ nix-shell --run "cabal v2-run gargantext -- worker run-all"
 ## Configuration
 
 Edit your `gargantext-settings.toml` file and add this section:
+
 ```toml
 [worker]
 
@@ -492,11 +555,13 @@ exchange`, the `dispatcher` and the `websocket server`.
 
 See `gargantex-settings.toml_toModify`. Overall, the config for
 notifications is quite simple:
+
 ```toml
 [notifications]
 central-exchange = { bind = "tcp://*:5560", connect = "tcp://localhost:5560" }
 dispatcher = { bind = "tcp://*:5561", connect = "tcp://localhost:5561" }
 ```
+
 The `bind`/`connect` strings are just forwarded to nanomsg, hence we
 could switch `tcp` to `ipc` (but this limits us to a single machine)
 or `inproc` (this limits us to inter-process communication).