New cabal2stack scaffolding

.gitlab-ci.yml

@@ -11,8 +11,8 @@ variables:
   CACHE_COMPRESSION_LEVEL: "fast"
 
 stages:
-  - stack
   - cabal
+  - stack
   - bench
   - test
 
@@ -37,7 +37,7 @@ cabal:
       - .cabal/
     policy: pull-push
   script:
-    - nix-shell --run "./bin/update-cabal-project $CABAL_STORE_DIR && cabal --store-dir=$CABAL_STORE_DIR v2-build --ghc-options='-O2 -fclear-plugins'"
+    - nix-shell --run "./bin/update-project-dependencies $CABAL_STORE_DIR && cabal --store-dir=$CABAL_STORE_DIR v2-build --ghc-options='-O2 -fclear-plugins'"
   allow_failure: false
 
 bench:
@@ -49,7 +49,7 @@ bench:
       - .cabal/
     policy: pull-push
   script:
-    - nix-shell --run "./bin/update-cabal-project $CABAL_STORE_DIR && cabal --store-dir=$CABAL_STORE_DIR v2-bench --ghc-options='-O2 -fclear-plugins'"
+    - nix-shell --run "./bin/update-project-dependencies $CABAL_STORE_DIR && cabal --store-dir=$CABAL_STORE_DIR v2-bench --ghc-options='-O2 -fclear-plugins'"
   allow_failure: false
 
 test:
@@ -75,7 +75,7 @@ test:
       echo $CABAL
       echo $TEST_NIX_PATH
       git config --global --add safe.directory '*'
-      nix-shell --run "./bin/update-cabal-project $CABAL_STORE_DIR"
+      nix-shell --run "./bin/update-project-dependencies $CABAL_STORE_DIR"
       mkdir -p /root/.cache/cabal/logs
       chown -R test:test /root/.cache/cabal/logs/
       chown -R test:test /root/.cache/cabal/packages/hackage.haskell.org/

README.md

@@ -39,12 +39,16 @@ Disclaimer: since this project is still in development, this document remains in
   ```
 ### Installation
 
-This project can be built with either Stack or Cabal. For historical reasons, we generate a `cabal.project` from the `stack.yaml`, and we do not commit the former to the repo, to have a single "source of truth".
-However, it's always possible to generate a `cabal.project` thanks to [stack2cabal](https://hackage.haskell.org/package/stack2cabal).
+This project can be built with either Stack or Cabal. We keep up-to-date the `cabal.project` (which allows us
+to build with `cabal` by default) but we support `stack` thanks to thanks to
+[cabal2stack](https://github.com/iconnect/cabal2stack), which allows us to generate a valid `stack.yaml` from
+a `cabal.project`. Due to the fact gargantext requires a particular set of system dependencies (C++ libraries,
+toolchains, etc) we use [nix](https://nixos.org/) to setup an environment with all the required system
+dependencies, in a sandboxed and isolated fashion.
 
 #### Install Nix 
 
-Gargantext requires [Nix](https://github.com/NixOS/nix) to provide system dependencies (for example, C libraries), but its use is limited to that. In order to install [Nix](https://nixos.org/download.html):
+As said, Gargantext requires [Nix](https://github.com/NixOS/nix) to provide system dependencies (for example, C libraries), but its use is limited to that. In order to install [Nix](https://nixos.org/download.html):
 
 ```shell
 sh <(curl -L https://nixos.org/nix/install) --daemon
@@ -62,7 +66,7 @@ nix-env (Nix) 2.19.2
 nix-shell
 ```
 
-This will take a bit of time the first time.
+This will take a bit of time as it has to download/build the dependencies, but this will be needed only the first time.
 
 ### Build: choose cabal (new) or stack (old)
 
@@ -74,6 +78,13 @@ cabal update
 cabal install
 ```
 
+Alternatively, if you want to run the command "from the outside", in your current shell:
+
+```
+nix-shell --run "cabal update"
+nix-shell --run "cabal install"
+```
+
 #### With Stack
 
 Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
@@ -95,28 +106,32 @@ stack build --fast
 ```
 
 
-#### Keeping the cabal.project updated with stack.yaml
+#### Keeping the stack.yaml updated with the cabal.project
 
-(Section for Developers using cabal only)
+(Section for Developers using stack only)
 
-Once you have a valid version of `cabal`, building requires generating a valid `cabal.project`. This can be done by installing `stack2cabal`:
+Once you have a valid version of `stack`, building requires generating a valid `stack.yaml`.
+This can be obtained by installing `cabal2stack`:
 
 ```shell
-cabal v2-install --index-state="2023-12-10T10:34:46Z" --constraint 'Cabal==3.6.3.0' stack2cabal-1.0.14 --overwrite-policy=always
+git clone https://github.com/iconnect/cabal2stack.git
+cd cabal2stack
 ```
 
+Then, depending on what build system you are using, either build with `cabal install --overwrite-policy=always` or `stack install`.
+
 And finally:
 
 ```shell
-stack2cabal --no-run-hpack -p '2023-06-25'
-cabal v2-build
+cabal2stack --system-ghc --allow-newer --resolver lts-21.17 --resolver-file devops/stack/lts-21.17.yaml -o stack.yaml
+stack build
 ```
 
-
-Simply run:
+The good news is that you don't have to do all of this manually; during development, after modifying the
+`cabal.project`, it's enough to do:
 
 ```shell
-./bin/update-cabal-project
+./bin/update-project-dependencies
 ```
 
 ## Initialization <a name="init"></a>
@@ -184,10 +199,10 @@ When a devlopment is needed on libraries (for instance, the HAL crawler in https
       - turn off (temporarily) the `hal` in `source-repository-package` 
    2. When changes work and tests are OK, commit in repo `hal`
 2. When changes are commited / merged:
-   1. Get the hash id, and edit `stack.yaml` with the **new commit id**
-   2. run `./bin/update-cabal-project`
-      - get an error that sha256 don't match, so update the `./bin/update-cabal-project` with new sha256 hash
-      - run again `./bin/update-cabal-project` (to make sure it's a fixed point now)
+   1. Get the hash id, and edit `cabal.project` with the **new commit id**
+   2. run `./bin/update-project-dependencies`
+      - get an error that sha256 don't match, so update the `./bin/update-project-dependencies` with new sha256 hash
+      - run again `./bin/update-project-dependencies` (to make sure it's a fixed point now)
 
 > Note: without `stack.yaml` we would have to only fix `cabal.project` -> `source-repository-package` commit id. Sha256 is there to make sure CI reruns the tests.
 

bin/update-cabal-project

-#!/usr/bin/env bash
-
-set -euxo pipefail
-
-DEFAULT_STORE=$HOME/.cabal
-STORE_DIR="${1:-$DEFAULT_STORE}"
-
-# README!
-# Every time you modify the `stack.yaml` and as result the relevant `cabal.project`
-# changes, you have to make sure to update the `expected_cabal_project_hash` and
-# `expected_cabal_project_freeze_hash` with the
-# `sha256sum` result calculated on the `cabal.project` and `cabal.project.freeze`.
-# This ensures the files stay deterministic so that CI cache can kick in.
-expected_cabal_project_hash="1e4d40d48546606fba0ce0eaae9f2799c57d8ce97c4425940f3a535c4f628a8a"
-expected_cabal_project_freeze_hash="bd90562207ea42857402eced5836b7a23024cc16034bf150968dbf373d2e243c"
-
-cabal --store-dir=$STORE_DIR v2-update 'hackage.haskell.org,2023-12-10T10:34:46Z'
-
-# Install stack2cabal if it can't be found.
-if ! stack2cabal --help &> /dev/null
-then
-    echo "stack2cabal could not be found"
-    cabal --store-dir=$STORE_DIR v2-install --index-state="2023-12-10T10:34:46Z" --constraint 'Cabal==3.6.3.0' stack2cabal-1.0.14 --overwrite-policy=always
-fi
-stack2cabal --no-run-hpack -p '2023-12-10 10:34:46'
-
-actual_cabal_project_hash=$(sha256sum cabal.project | awk '{printf "%s",$1}')
-actual_cabal_project_freeze_hash=$(sha256sum cabal.project.freeze | awk '{printf "%s",$1}')
-if [[ $actual_cabal_project_hash != $expected_cabal_project_hash ]]; then
-  echo "ERROR! hash mismatch between expected cabal.project and the one computed by stack2cabal."
-  exit 1
-else
-  echo "cabal.project updated successfully."
-fi
-if [[ $actual_cabal_project_freeze_hash != $expected_cabal_project_freeze_hash ]]; then
-  echo "ERROR! hash mismatch between expected cabal.project.freeze and the one computed by stack2cabal."
-  exit 1
-else
-  echo "cabal.project.freeze updated successfully."
-fi

bin/update-project-dependencies

+#!/usr/bin/env bash
+
+set -euxo pipefail
+
+DEFAULT_STORE=$HOME/.cabal
+STORE_DIR="${1:-$DEFAULT_STORE}"
+INDEX_STATE="2023-12-10T10:34:46Z"
+
+# README!
+# Every time you modify the `cabal.project`, you have to make sure to update
+# the `expected_cabal_project_hash` and `expected_cabal_project_freeze_hash`
+# with the `sha256sum` result calculated on the `cabal.project` and
+# `cabal.project.freeze`. This ensures the files stay deterministic so that CI
+# cache can kick in.
+expected_cabal_project_hash="1e4d40d48546606fba0ce0eaae9f2799c57d8ce97c4425940f3a535c4f628a8a"
+expected_cabal_project_freeze_hash="2c13034bdeaeaece6c81362ef047c3102782b4fbf4fd7670bb677bd1ac3b0151"
+
+cabal --store-dir=$STORE_DIR v2-update "hackage.haskell.org,${INDEX_STATE}"
+
+# Install cabal2stack if it can't be found.
+if ! cabal2stack --help &> /dev/null
+then
+    echo "cabal2stack could not be found"
+    CURDIR=$PWD
+    git clone https://github.com/iconnect/cabal2stack.git cabal2stack-installer
+    cd cabal2stack-installer
+    cabal --store-dir=$STORE_DIR v2-install --index-state="${INDEX_STATE}" --overwrite-policy=always
+    cd $CURDIR
+    rm -rf cabal2stack-installer
+fi
+
+cabal --store-dir=$STORE_DIR v2-build --dry-run
+cabal2stack --system-ghc --allow-newer --resolver lts-21.17 --resolver-file devops/stack/lts-21.17.yaml -o stack.yaml
+cabal --store-dir=$STORE_DIR v2-freeze
+
+# Run 'sed' to remove the constraint for 'gargantext', as it doesn't make sense and
+# for the test we need to run this with a different flag.
+echo -e "\e[33mPatching cabal.project.freeze to remove redundant constraint on gargantext\e[0m"
+sed -i '/^ *gargantext/d' cabal.project.freeze
+
+actual_cabal_project_hash=$(sha256sum cabal.project | awk '{printf "%s",$1}')
+actual_cabal_project_freeze_hash=$(sha256sum cabal.project.freeze | awk '{printf "%s",$1}')
+if [[ $actual_cabal_project_hash != $expected_cabal_project_hash ]]; then
+  echo -e "\e[31mERROR! hash mismatch between expected cabal.project and the one computed by cabal2stack.\e[0m"
+  echo -e "\e[33mPlease update the hashes inside the './bin/update-project-dependencies' file.\e[0m"
+  exit 1
+else
+  echo -e "\e[32mstack.yaml updated successfully.\e[0m"
+fi
+if [[ $actual_cabal_project_freeze_hash != $expected_cabal_project_freeze_hash ]]; then
+  echo -e "\e[31mERROR! hash mismatch between expected cabal.project.freeze and the one computed by cabal2stack.\e[0m"
+  echo -e "\e[33mPlease update the hashes inside the './bin/update-project-dependencies' file.\e[0m"
+  exit 1
+else
+  echo -e "\e[32mcabal.project.freeze updated successfully.\e[0m"
+fi