Evaluating build systems: `nix`+`cabal.project.freeze` vs. `nixpkgs.haskellPackages` vs. `nix-build-cabal-project` vs. `haskell.nix`

This issue springs from my three attempts at writing a flake.nix for gargantext.

Nix flakes is an experimental feature of the Nix package manager. Flakes was introduced with Nix 2.4 (see release notes).

Flakes is a feature of managing Nix packages to simplify usability and improve reproducibility of Nix installations. Flakes manages dependencies between Nix expressions, which are the primary protocols for specifying packages. Flakes implements these protocols in a consistent schema with a common set of policies for managing packages.

https://nixos.wiki/wiki/Flakes

Motivation

Lately, you've likely read me being noticeably wary of some dependencies pulled by gargantext, especially ad-hoc ones or patched ones. This wariness sprung out from my trying to rule all dependencies with the best tool I currently know to wield for clarifying and managing dependencies: flake.nix.

Yet I'd like to stress that the most important problem I'd like to face here is not advocating for one flavor of the nix build system over another, both still have pros and cons, but advocating for managing all Haskell dependencies with nix instead of using cabal-install, since, as I shall explain below, using nix is better for the concern I personally often cherish the most: software correctness.

It just turns out that managing all Haskell dependencies with nix is by far the most difficult obstacle to overcome to be able to unlock the pros of a flake.nix. Therefore, since I've just rebased my previous iteration of flake.nix onto the lastest dev and nixpkgs-23.11, I thought it would be a good time to explicit more thoroughly the pros and cons of a nixpkgs.haskellPackages-based flake.nix with respect to any other legitimate software quality concerns.

To proceed in a methodological and exhaustive way I thought I would first let me guided by the generic concerns listed in the ISO/IEC 25010 Software Quality Model, staying narrowly on the topic of each of them, but that should not refrain you from raising (in the discussion below or out-of-band) any other concern that you may have, be it related to software quality or out-of-code concerns.

Again let me stress that except for the narrow concern of "correctness", I am not pretending that using flake.nix, and more precisely the flake.nix based upon nixpkgs.haskellPackages I'm proposing in !258 is the best way to go, nor that I've assessed correctly the impacts it could have on those other concerns. In the end, I am happy to leave that to you ;)

Meta remark

Sorry, I am aware this is too verbose, but this is the best I can do to convince others, and to force myself to recognize and maybe address concerns I'd rather not. I will modify it in place and post a brief resume if need be, but if that proves to be too hard or confusing, I may switch to a standalone Markdown document, trackable as easily as any other piece of code.

Functional Suitability

Functional Completeness

As could an improved nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system can manage all dependencies (Haskell and non-Haskell), but the latter enforces this when used to build the resulting gargantext package (instead of just the development shells).

For the record, the current nix+cabal.project.freeze build system only manages with nix the executables ones: ghc947, cabal_install_3_10_1_0, haskellPackages.alex, haskellPackages.happy and haskellPackages.pretty-show.

Functional Correctness

Both a nix+cabal.project.freeze build system or a nixpkgs.haskellPackages-based build system enforce the precise pinning of all inputs which improves the delivery of precise outputs which is essential to correctness.

Side note: cabal-install's is in Haskell, and nix's in C++, but the former is likely less battle-tested than the latter.

However, contrary to the nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system forces to specify some expectations about the outputs of all the dependencies, and most notably what to expect from running their tests (either doCheck or dontCheck or which precise tests must not be run).

With the current nix+cabal.project.freeze build system, the Haskell dependencies are managed with cabal-install which does not run their test-suites systematically: cabal-install's run-tests defaults to False, so to run the test-suites of the whole closure of Haskell dependencies I guess one could use a custom --config-file= containing run-tests: True, yet that :

would not be retro-active on already installed Haskell packages,
would either force every installing user to run the test-suites or make it optional (hence forgettable),
and, would not give a mean to specify which test-suites, let alone specific tests, are expected to fail at that point.

IMHO both systematic testing of dependencies and specifying failing tests, must be done, either with nix+cabal.project.freeze+run-tests or nixpkgs.haskellPackages. But doing so means writing Nix code to exhaustively build, test and fix Haskell packages, which was the most difficult part of writing a nixpkgs.haskellPackages-based flake.nix for gargantext anyway.

As a motivating example demonstrating that the problem is not only hypothetic but actually occuring, doing such a systematic checking while writing the demo nixpkgs.haskellPackages-based flake.nix is what enabled the uncovering of a few failing tests in our own text16-compat which may or may not cause an other failing test (/Unicode and Regex Test/) in duckling which was not failing 3 months ago where text-2 and text16-compat were not yet used.

Functional Appropriateness

As could do an improved nix+cabal.project.freeze build system, a flake.nix build system gathers in a single tool (nix) all the tasks and objectives of the current nix+cabal.project.freeze build system and more:

providing development shells:
- nix -L develop
- nix -L develop .#prof-trace (alternative development shell)
- nix -L develop .#haskellPackages.boolexpr (development shell for a specific dependency)
- …
packages:
- nix -L build .#gargantext (building the gargantext package)
- nix -L build .#haskellPackages.boolexpr (building only a specific dependency)
- …
source code formatters (eg. with the help of pre-commit-hooks).
applications (build or tests scripts or actual end-users programs).
Nixpkgs overlays (propagated modification of a dependency).
NixOS configurations.
checks.
…

cabal-install remains responsible to build Haskell packages (mostly to invoke necessary ghc calls with the right parameters) under the hood, but it no longer selects nor fetches Haskell dependencies.

Reliability

Maturity

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system is less mature, it's actually still guarded behind a feature flag, usually set in ~/.config/nix/nix.conf or /etc/nix/nix.conf:

experimental-features = nix-command flakes

See this June 2023 discussion: Why are flakes still experimental?:

Note that there is also a bit of controversy around flakes. The feature has been implemented without going through the RFC process, lacking input from community. As a result part of the community rejected flakes.
Many also feel that it is a too big monolithic change to the Nix language, parts of which could just as well be implemented outside of Nix (see e.g. niv for managing and pinning sources) and promoting one true way will stifle innovation. Having less controversial features like nix-command and evaluation caching tied to flakes is also considered unfortunate.
However, there is currently a new RFC to try to resolve this situation: https://github.com/NixOS/rfcs/pull/136 (merged)

Availability

Contrary to the nix+cabal.project.freeze build system, a flake.nix build system is only available since nix-2.4 (2021-11-01).

Fault Tolerance

I currently see no visible difference on that concern.

Recoverability

Contrary to a nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system handles Haskell-dependencies, and as all nix dependency by default they're built in a temporary sandbox, meaning that when a dependency fails to build or pass its test-suite(s), it has to be fixed or worked-around, which in both case means to be rebuilt from the beginning. The same applies when patching a dependency, though there is work to support incremental builds, it is not yet morally-supported by upstream's nix.

Performance Efficiency

Time Behavior

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system has an evaluation cache in ~/.cache/nix/ for flake outputs, speeding up significantly the spawning of development shells for instance. Meaning also that if a build fails, re-running it without modifying anything from what it depends, will fail without even trying to rebuild, hence adding --show-trace to get a nix stack trace will not work without modifying the evaluated nix expression. It's not a problem, it's just something to know.

In both cases, using nix to manage both Haskell and non-Haskell dependencies, enables to fetch the compiled outputs from https://cache.nixos.org (or a custom cache (possibly populated by the CI) specific to gargantext) instead of having every integrator-user recompile Haskell dependencies when installing.

Resource Utilization

Contrary to the nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system uses more disk space as it always copies every changed input to the Nix store. By making it more easy to update inputs, (especially nixpkgs), it will also:

download more versions of them, which can be mitigated by running nix flake lock --override-input nixpkgs github:NixOS/nixpkgs/<a commit-revision-already-downloaded-previously> to the expense of reproductibility,
keep more versions of them in the Nix store, which can be mitigated by (manually) running nix-collect-garbage.

Moreover, when the outputs are registered as a nix garbage-collector root (eg. as a result symlink when using nix build, or when using the direnv integration), those roots (listable with nix-store --gc --print-roots) may be a pain to remove by hand. So when using direnv, I recommend putting the following in ~/.config/direnv/direnvrc to gather all the GC roots in a single directory where it's easier to remove them when we no longer need them:

: ${XDG_CACHE_HOME:=$HOME/.cache}
declare -A direnv_layout_dirs
direnv_layout_dir() {
    echo "${direnv_layout_dirs[$PWD]:=$(
        echo -n "$XDG_CACHE_HOME"/direnv/layouts/${PWD##*/}-
        echo -n "$PWD" | shasum | cut -d ' ' -f 1
    )}"
}

The impact on the CI should also be assessed, for instance if it's using Docker containers this could likely be replaced by NixOS VM, which could use much less disk space because they would share dependencies at the package level, not the container level.

Besides, since flakes compose, their inputs may proliferate in cascade, giving rise to many different version of a same input (eg. nixpkgs) being pulled. This must be prevented by using the follows attribute of inputs. See the "Interoperability" section below.

Capacity

Contrary to a nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system limits the inputs, when building an output package, no unspecified input can be accessed.

Usability

Appropriateness Recognizability

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system cannot hardly be considered an ad-hoc architecture it follows an established schema, making it easier to organize and document goals in a way familiar to other Nix flakes users,

There's also nix flake show to print what's implemented on what system, but in simple cases it's enough and easier to just read flake.nix.

Learnability

From knowing the current nix+cabal.project.freeze build system, there are mainly two things to learn:

The flake.nix schema used to organize inputs and outputs in a consistent and coherent Nix expression. That is quite simple and should not cause pain. Besides, learning it is an amortizable cost as more and more projects are using it to manage package inputs and outputs. However the flake.nix way is more recent, hence may not be the way presented in online documentations and other resources.
The Nixpkgs Haskell infrastructure, used to replace cabal-install's dependencies management. Though it's already used by current nix+cabal.project.freeze build system for specific packages (the executable ones), using a nixpkgs.haskellPackages-based build system would require a more thorough understanding of how to do things usually disabled (tests) or done with Git (patching) or cabal.project (pinning). However I think that the proposed flake.nix already has examples to learn from to solve most of the problems that may arise.

Operability

User Error Protection

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system gives the ability to update all inputs in one command (nix flake update), which helps a lot to prevent forgetting about updating a pinned forked repository, and warns when an external upstream has been updated (except when the input is pointing to an ad-hoc and unmaintained branch instead of main/master, that's why I prefer to put yet-unmerged change in patches/ and apply them on top of upstream's main with mkDerivation's patches or lib's applyPatches, which will raise an error when upstream's has merged it or has broken it).

User Interface Aesthetics

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system does not need ad-hoc scripts or manually fetching and modifying hardcoded commits in cabal.project's source-repository-package to update dependencies, the nix flake subcommand automates all that, yet let us able to pin them manually if need be. It is a unified way of managing dependencies, removing the responsability to select, pin and update dependencies from underlying build tools like cabal-install (which only keeps the task of building the Haskell code).

Accessibility

Contrary to a nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system is not commonly used by other developers, and it's not clear for me whether or not most nix users are already used to flake.nix, which are only 3 years old and still require to enable a feature flag in ~/.config/nix/nix.conf or /etc/nix/nix.conf. Meaning that any basic usage of a flake.nix that would come to be required must be carefuly documented in gargantext's own documentation.

Security

Confidentiality

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system forcibly copies every input into the Nix store. Because the Nix store is readable by all Unix users, this can be problematic on shared computers if secrets are used in inputs.

Integrity

I do not see any difference for that concern.

Non-repudiation

I do not see any difference for that concern.

Accountability

I do not see any difference for that concern.

Authenticity

I do not see any difference for that concern.

Compatibility

Co-existence

A nix+cabal.project.freeze build system can co-exist with a nixpkgs.haskellPackages-based flake.nix build system, but this would be a duplication of efforts to support both of them beyond a demo.

Interoperability

Contrary to the nix+cabal.project.freeze build system, a flake.nix build system enables users to change inputs without modifying gargantext's flake.nix using the follows attribute.

For instance this is how we can use a single version of nixpkgs instead of two (gargantext's and pre-commit-hooks'):

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
  pre-commit-hooks.url = "github:cachix/pre-commit-hooks.nix";
  pre-commit-hooks.inputs.nixpkgs.follows = "nixpkgs";
};

The same could be done by someone importing gargantext's flake.nix to run its own instance while using its own version of nixpkgs instead of the one pinned by gargantext's flake.nix:

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
  gargantext.url = "git+https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git";
  gargantext.inputs.nixpkgs.follows = "nixpkgs";
};

Maintainability

Modularity

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system can more easily pin, update and fetch other flake.nix which can themselves provide not only packages, but all sort of integrations like overlays or shells. See for example pre-commit-hooks's flake.nix, which provides facilities to integrate itself into the development shell.

But the most important use of that may be by gargantext's own users which instead of publishing their data and mentionning gargantext's version used to analyse them, could just publish their own flake.nix pinning gargantext's flake.nix for a better and easier reproducibility of science results.

The same could be done with a nix+cabal.project.freeze build system, but flake.nix makes it much easier and more manageable.

Reusability

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system is designed to compose with other flake.nix, which can load other flakes as input and easily reuse any code made available in its Flake schema.

This could help the next step of nixifying gargantext: writing a NixOS module into flake.nix alongside the packaging.

Analysability

Contrary to a nix+cabal.project.freeze build system, a nixpkgs.haskellPackages-based build system also manages all Haskell dependencies, which, in the context of benchmarking, profiling, or testing, make it easier to apply custom compile flags to the whole package closure, and to switch between packages and shells enabling different sets of flags.

Modifiability

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system may render the dependencies' code harder to modify since (unless --impure is passed to nix by the developer), all files necessary to build an output must be put inside Git's index (ie. git add).

Eg. it you add a patch in patches/ and add it to a package's patches= attribute in flake.nix, the build will fail to find the patch unless you've previously git add patches/path/to/the.patch, as nix build packages in a sandbox only able to reach what is in the Nix store, and when copying the self input (your working tree) only what has been put into your Git repository's index will be copied there. Side note, once git add patches/path/to/the.patch has been done for one version of the patch, it's not necessary to do it again while iterating.

Moreover like when using git submodules, modifying dependencies requires to update them (in flake.lock, with nix flake lock --update $input), which can be problematic if the dependency modification has not yet been published.

That can be mitigated with a temporary :

$ nix flake lock --override-input some-input git+file:///local/path/to/some-input?ref=main

Testability

Besides the correctness concern of Haskell dependencies studied above, there is also the possiblity to provide checks in the flake.nix:

$ nix flake check .#gargantext

Those checks could spawn multiple NixOS-based VM for running them in a more controlled environment.

Portability

Adaptability

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system is not familiar to any other gargantext developers but me so far, and I'm no darwin user.

Yet, flake.nix provides a principled way to do adapt things to differents systems, because both packages and shells are only reachable by first specifying the system they should be built for.

Installability

Contrary to a nix+cabal.project.freeze build system, a flake.nix build system is easier to install, as it takes care of installing all inputs (except for the kernel), one could just do:

$ nix run git+https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git?ref=dev#gargantext

and that would download the whole closure of latest dev branch of gargantext, build what's not in the cache and execute the application named "gargantext" in flake.nix.

Whichever the build system chosen, I'd like to stress the importance of using development shells to provide development tools which have a strong compatiblity constraint. It would prevent debugging a mismatch between GHC and HLS like what happened to @cgenie recently.

Replaceability

The actionable alternatives are:

The current nix+cabal.project.freeze.
A flake.nix based upon nixpkgs.haskellPackages like the current !258
A flake.nix based upon fgaz's nix-build-cabal-project, which boils down to just a thin wrapper around cabal-install. Meaning it does not package nor cache each individual Haskell dependencies into Nix, it bundles them all in a single package, which is helpful for deployment, no so helpful for development. If cabal-install is setup to run-tests and we find a way to override run-tests per-package, it can also check test-suites.
A flake.nix based upon IOHK's haskell.nix, which produces Nix expressions for all of Hackage. However it does not put their outputs on the default Nix cache at https://cache.nixos.org but on IOHK's Nix cache at hydra.iohk.io.
"1. + 2." but this 2. would not use the exact same package versions than to 1.
"1. + 3."
"1. + 4."
guix's haskell-build-system, but I doubt it provides the same level of integration of Haskell than 2. or 4.

This issue springs from my three attempts at writing a `flake.nix` for gargantext.

> Nix flakes is an experimental feature of the Nix package manager. Flakes was introduced with Nix 2.4 (see release notes).
>
> Flakes is a feature of managing Nix packages to simplify usability and improve reproducibility of Nix installations. Flakes manages dependencies between Nix expressions, which are the primary protocols for specifying packages. Flakes implements these protocols in a consistent schema with a common set of policies for managing packages.

https://nixos.wiki/wiki/Flakes

# Motivation
Lately, you've likely read me being noticeably wary of some dependencies
pulled by gargantext, especially ad-hoc ones or patched ones.
This wariness sprung out from my trying to rule all dependencies
with the best tool I currently know to wield
for clarifying and managing dependencies: `flake.nix`.

Yet I'd like to stress that the most important problem I'd like to face
here is not advocating for one flavor of the `nix` build system over another,
both still have pros and cons, but advocating for managing all Haskell dependencies
with `nix` instead of using `cabal-install`,
since, as I shall explain below, using `nix` is better for the concern
I personally often cherish the most: software correctness.

It just turns out that managing all Haskell dependencies
with `nix` is by far the most difficult obstacle to overcome
to be able to unlock the pros of a `flake.nix`.
Therefore, since I've just rebased my previous iteration of `flake.nix`
onto the lastest `dev` and `nixpkgs-23.11`,
I thought it would be a good time to explicit more thoroughly
the pros and cons of a [`nixpkgs.haskellPackages`](https://haskell4nix.readthedocs.io/search.html)-based `flake.nix`
with respect to any other legitimate software quality concerns.

To proceed in a methodological and exhaustive way I thought I would first let me guided by the [generic concerns](https://blog.codacy.com/iso-25010-software-quality-model) listed in the [ISO/IEC 25010 Software Quality Model](https://www.evs.ee/en/search?query=25010%3A2011&Search=Search), staying narrowly on the topic of each of them,
but that should not refrain you from raising (in the discussion below or out-of-band)
any other concern that you may have, be it related to software quality or out-of-code concerns.

Again let me stress that except for the narrow concern of "correctness",
I am not pretending that using `flake.nix`,
and more precisely *the* `flake.nix` based upon [`nixpkgs.haskellPackages`](https://haskell4nix.readthedocs.io/search.html)
I'm proposing in https://gitlab.iscpif.fr/gargantext/haskell-gargantext/merge_requests/258 is the best way to go,
nor that I've assessed correctly the impacts it could have on those other concerns.
In the end, I am happy to leave that to you ;)

#### Meta remark
Sorry, I am aware this is too verbose, but this is the best I can do to convince others,
and to force myself to recognize and maybe address concerns I'd rather not.
I will modify it in place and post a brief resume if need be,
but if that proves to be too hard or confusing,
I may switch to a standalone Markdown document,
trackable as easily as any other piece of code.

## Functional Suitability
### Functional Completeness

As could an improved `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
can manage all dependencies (Haskell and non-Haskell),
but the latter enforces this when used to build the resulting gargantext package
(instead of just the development shells).

For the record, the current `nix`+`cabal.project.freeze` build system only manages with `nix` the executables ones: `ghc947`, `cabal_install_3_10_1_0`, `haskellPackages.alex`, `haskellPackages.happy` and `haskellPackages.pretty-show`.

### Functional Correctness

Both a `nix`+`cabal.project.freeze` build system or a `nixpkgs.haskellPackages`-based build system
enforce the precise pinning of **all** inputs which improves the delivery of precise outputs which is essential to correctness.

Side note: `cabal-install`'s is in Haskell, and `nix`'s in C++, but the former is likely less battle-tested than the latter.

However, contrary to the `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
forces to specify some expectations about the outputs of all the dependencies,
and most notably what to expect from running their tests (either `doCheck` or `dontCheck` or which precise tests must not be run).

With the current `nix`+`cabal.project.freeze` build system,
the Haskell dependencies are managed with `cabal-install`
which does not run their `test-suite`s systematically:
[`cabal-install`'s `run-tests` defaults to `False`](https://cabal.readthedocs.io/en/latest/cabal-project-description-file.html#cfg-field-run-tests),
so to run the `test-suite`s of the whole closure of Haskell dependencies
I guess one could use a custom `--config-file=` containing `run-tests: True`,
yet that :
- would not be retro-active on already installed Haskell packages,
- would either force every installing user to run the `test-suite`s
  or make it optional (hence forgettable),
- and, would not give a mean to specify which `test-suite`s, let alone specific tests,
  are expected to fail at that point.

IMHO both systematic testing of dependencies and specifying failing tests,
*must* be done, either with `nix`+`cabal.project.freeze`+`run-tests` or `nixpkgs.haskellPackages`.
But doing so means writing Nix code to exhaustively build, test and fix Haskell packages,
which was the most difficult part of writing a `nixpkgs.haskellPackages`-based `flake.nix` for gargantext anyway.

As a motivating example demonstrating that the problem is not only hypothetic
but actually occuring, doing such a systematic checking while writing the demo `nixpkgs.haskellPackages`-based `flake.nix`
is what enabled the uncovering of a few [failing tests](https://github.com/adinapoli/text16-compat/issues/1) in our own `text16-compat`
which may or may not cause an other failing test (`/Unicode and Regex Test/`)
in `duckling` which was not failing 3 months ago where `text-2` and `text16-compat` were not yet used.

### Functional Appropriateness

As could do an improved `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
gathers in a single tool (`nix`) all the tasks
and objectives of the current `nix`+`cabal.project.freeze` build system and more:
- providing development shell**s**:
  - `nix -L develop`
  - `nix -L develop .#prof-trace` (alternative development shell)
  - `nix -L develop .#haskellPackages.boolexpr` (development shell for a specific dependency)
  - …
- package**s**:
  - `nix -L build .#gargantext` (building the gargantext package)
  - `nix -L build .#haskellPackages.boolexpr` (building only a specific dependency)
  - …
- source code formatters (eg. with the help of [pre-commit-hooks](https://github.com/cachix/pre-commit-hooks.nix)).
- applications (build or tests scripts or actual end-users programs).
- Nixpkgs overlays (propagated modification of a dependency).
- NixOS configurations.
- checks.
- …

`cabal-install` remains responsible to build Haskell packages
(mostly to invoke necessary `ghc` calls with the right parameters) under the hood,
but it no longer selects nor fetches Haskell dependencies.

## Reliability

### Maturity

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
is less mature, it's actually still guarded behind a feature flag,
usually set in `~/.config/nix/nix.conf` or `/etc/nix/nix.conf`:

```
experimental-features = nix-command flakes
```
See this June 2023 discussion: [Why are flakes still experimental?](https://discourse.nixos.org/t/why-are-flakes-still-experimental/29317):

> Note that there is also a bit of controversy around flakes. The feature has been implemented without going through the RFC process, lacking input from community. As a result part of the community rejected flakes.  
> Many also feel that it is a too big monolithic change to the Nix language, parts of which could just as well be implemented outside of Nix (see e.g. niv for managing and pinning sources) and promoting one true way will stifle innovation. Having less controversial features like nix-command and evaluation caching tied to flakes is also considered unfortunate.  
> However, there is currently a new RFC to try to resolve this situation:
> https://github.com/NixOS/rfcs/pull/136 (merged)

### Availability

Contrary to the `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
is only available since [nix-2.4 (2021-11-01)](https://nixos.org/manual/nix/unstable/release-notes/rl-2.4.html).

### Fault Tolerance

I currently see no visible difference on that concern.

### Recoverability

Contrary to a `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
handles Haskell-dependencies, and as all nix dependency by default they're built in a temporary sandbox,
meaning that when a dependency fails to build or pass its test-suite(s),
it has to be fixed or worked-around, which in both case means to be rebuilt from the beginning.
The same applies when patching a dependency, though there is [work to support incremental builds](https://github.com/NixOS/nixpkgs/pull/204020), it is not [yet morally-supported by upstream's nix](https://github.com/NixOS/nix/pull/7362#issuecomment-1564251964).

## Performance Efficiency

### Time Behavior

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
has an evaluation cache in `~/.cache/nix/` for flake outputs,
[speeding up significantly](https://nixos.wiki/wiki/Flakes#Super_fast_nix-shell) the spawning of development shells for instance. Meaning also that if a build fails, re-running it without modifying anything from what it depends, will fail without even trying to rebuild, hence adding `--show-trace` to get a nix stack trace will not work without modifying the evaluated nix expression. It's not a problem, it's just something to know.

In both cases, using `nix` to manage both Haskell and non-Haskell dependencies, enables to fetch the compiled outputs from https://cache.nixos.org (or a [custom cache](http://cachix.org/) (possibly populated by the CI) specific to gargantext) instead of having every integrator-user recompile Haskell dependencies when installing.

### Resource Utilization

Contrary to the `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
uses more disk space as it always copies every changed input to the Nix store.
By making it more easy to update inputs, (especially `nixpkgs`),
it will also:
- download more versions of them,
which can be mitigated by running `nix flake lock --override-input nixpkgs github:NixOS/nixpkgs/<a commit-revision-already-downloaded-previously>` to the expense of reproductibility,
- keep more versions of them in the Nix store,
which can be mitigated by (manually) running `nix-collect-garbage`.

Moreover, when the outputs are registered as a nix garbage-collector root
(eg. as a `result` symlink when using `nix build`, or when using the `direnv` integration),
those roots (listable with `nix-store --gc --print-roots`) may be a pain to remove by hand.
So when using `direnv`, I recommend putting the following
in `~/.config/direnv/direnvrc` to gather all the GC roots in a single directory
where it's easier to remove them when we no longer need them:
```bash
: ${XDG_CACHE_HOME:=$HOME/.cache}
declare -A direnv_layout_dirs
direnv_layout_dir() {
    echo "${direnv_layout_dirs[$PWD]:=$(
        echo -n "$XDG_CACHE_HOME"/direnv/layouts/${PWD##*/}-
        echo -n "$PWD" | shasum | cut -d ' ' -f 1
    )}"
}
```

The impact on the CI should also be assessed,
for instance if it's using Docker containers this could likely be replaced by NixOS VM,
which could use much less disk space because they would share dependencies at the package level,
not the container level.

Besides, since flakes compose, their inputs may proliferate in cascade, giving rise to many different version of a same input (eg. `nixpkgs`) being pulled. This must be prevented by using the `follows` attribute of inputs. See the "Interoperability" section below.

### Capacity

Contrary to a `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
limits the inputs, when building an output package, no unspecified input can be accessed.

## Usability

### Appropriateness Recognizability

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
cannot hardly be considered an ad-hoc architecture
it follows an [established schema](https://nixos.wiki/wiki/Flakes#Flake_schema),
making it easier to organize and document goals
in a way familiar to other Nix flakes users,

There's also `nix flake show` to print what's implemented on what system,
but in simple cases it's enough and easier to just read `flake.nix`.

### Learnability

From knowing the current `nix`+`cabal.project.freeze` build system, there are mainly two things to learn:
- The [`flake.nix` schema](https://nixos.wiki/wiki/Flakes#Flake_schema) used to organize inputs and outputs in a consistent and coherent Nix expression. That is quite simple and should not cause pain. Besides, learning it is an amortizable cost as more and more projects are using it to manage package inputs and outputs. However the `flake.nix` way is more recent, hence may not be the way presented in online documentations and other resources.

- The [Nixpkgs Haskell infrastructure](https://haskell4nix.readthedocs.io/search.html), used to replace `cabal-install`'s dependencies management. Though it's already used by current `nix`+`cabal.project.freeze` build system for specific packages (the executable ones), using a `nixpkgs.haskellPackages`-based build system would require a more thorough understanding of how to do things usually disabled (tests) or done with `Git` (patching) or `cabal.project` (pinning). However I think that the proposed `flake.nix` already has examples to learn from to solve most of the problems that may arise.

### Operability

### User Error Protection

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
gives the ability to update all inputs in one command (`nix flake update`),
which helps a lot to prevent forgetting about updating a pinned forked repository,
and warns when an external upstream has been updated
(except when the input is pointing to an ad-hoc and unmaintained branch instead of `main`/`master`,
that's why I prefer to put yet-unmerged change in `patches/`
and apply them on top of upstream's `main` with `mkDerivation`'s `patches` or `lib`'s `applyPatches`,
which will raise an error when upstream's has merged it or has broken it).

### User Interface Aesthetics

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
does not need ad-hoc scripts or manually fetching and modifying hardcoded commits in `cabal.project`'s `source-repository-package`
to update dependencies, the `nix flake` subcommand automates all that,
yet let us able to pin them manually if need be.
It is a unified way of managing dependencies, removing the responsability to select, pin and update dependencies
from underlying build tools like `cabal-install`
(which only keeps the task of building the Haskell code).

### Accessibility

Contrary to a `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
is not commonly used by other developers, and it's not clear for me whether or not
most nix users are already used to `flake.nix`, which are only 3 years old
and still require to enable a feature flag in `~/.config/nix/nix.conf` or `/etc/nix/nix.conf`.
Meaning that any basic usage of a `flake.nix` that would come to be required
must be carefuly documented in gargantext's own documentation.

## Security

### Confidentiality

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
forcibly copies every input into the Nix store.
Because the Nix store is readable by all Unix users,
this can be problematic on shared computers if secrets are used in inputs.

### Integrity

I do not see any difference for that concern.

### Non-repudiation

I do not see any difference for that concern.

### Accountability

I do not see any difference for that concern.

### Authenticity

I do not see any difference for that concern.

## Compatibility

### Co-existence

A `nix`+`cabal.project.freeze` build system can co-exist with a `nixpkgs.haskellPackages`-based `flake.nix` build system,
but this would be a duplication of efforts to support both of them beyond a demo.

### Interoperability

Contrary to the `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
enables users to change inputs without modifying gargantext's `flake.nix`
using the `follows` attribute.

For instance this is how we can use a single version of `nixpkgs`
instead of two (gargantext's and pre-commit-hooks'):
```
inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
  pre-commit-hooks.url = "github:cachix/pre-commit-hooks.nix";
  pre-commit-hooks.inputs.nixpkgs.follows = "nixpkgs";
};
```

The same could be done by someone importing gargantext's `flake.nix`
to run its own instance while using its own version of `nixpkgs`
instead of the one pinned by gargantext's `flake.nix`:

```
inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
  gargantext.url = "git+https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git";
  gargantext.inputs.nixpkgs.follows = "nixpkgs";
};
```

## Maintainability

### Modularity

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
can more easily pin, update and fetch other `flake.nix` which can themselves provide
not only packages, but all sort of integrations like overlays or shells.
See for example `pre-commit-hooks`'s `flake.nix`,
which provides facilities to integrate itself into the development shell.

But the most important use of that may be by gargantext's own users
which instead of publishing their data and mentionning gargantext's version used to analyse them,
could just publish their own `flake.nix` pinning gargantext's `flake.nix`
for a better and easier reproducibility of science results.

The same could be done with a `nix`+`cabal.project.freeze` build system,
but `flake.nix` makes it much easier and more manageable.

### Reusability

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
is designed to compose with other `flake.nix`, which can load other flakes as input
and easily reuse any code made available in its Flake schema.

This could help the next step of nixifying gargantext:
writing a NixOS module into `flake.nix` alongside the packaging.

### Analysability

Contrary to a `nix`+`cabal.project.freeze` build system, a `nixpkgs.haskellPackages`-based build system
also manages all Haskell dependencies, which, in the context of benchmarking, profiling, or testing,
make it easier to apply custom compile flags
to the whole package closure, and to switch between packages and shells
enabling different sets of flags.

### Modifiability

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
may render the dependencies' code harder to modify
since (unless `--impure` is passed to `nix` by the developer),
all files necessary to build an output must be put inside Git's index (ie. `git add`).

Eg. it you add a patch in `patches/` and add it to a package's `patches=` attribute in `flake.nix`,
the build will fail to find the patch unless you've previously `git add patches/path/to/the.patch`,
as nix build packages in a sandbox only able to reach what is in the Nix store,
and when copying the `self` input (your working tree) only what has been put
into your Git repository's index will be copied there.
Side note, once `git add patches/path/to/the.patch` has been done for one version of the patch,
it's not necessary to do it again while iterating.

Moreover like when using git submodules, modifying dependencies requires to update them (in `flake.lock`, with `nix flake lock --update $input`), which can be problematic if the dependency modification has not yet been published.

That can be mitigated with a temporary :
```
$ nix flake lock --override-input some-input git+file:///local/path/to/some-input?ref=main
```

### Testability

Besides the correctness concern of Haskell dependencies studied above,
there is also the possiblity to provide checks in the `flake.nix`:
```
$ nix flake check .#gargantext
```
Those checks could spawn multiple NixOS-based VM for running them in a more controlled environment.

## Portability

### Adaptability

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
is not familiar to any other gargantext developers but me so far, and I'm no `darwin` user.

Yet, `flake.nix` provides a principled way to do adapt things to differents systems,
because both packages and shells are only reachable by first specifying
the system they should be built for.

### Installability

Contrary to a `nix`+`cabal.project.freeze` build system, a `flake.nix` build system
is easier to install, as it takes care of installing all inputs (except for the kernel),
one could just do:
```
$ nix run git+https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git?ref=dev#gargantext
```
and that would download the whole closure of latest `dev` branch of gargantext,
build what's not in the cache and execute the application named "gargantext" in `flake.nix`.

Whichever the build system chosen,
I'd like to stress the importance of using development shells
to provide development tools which have a strong compatiblity constraint.
It would prevent debugging a mismatch between GHC and HLS
like what happened to @cgenie recently.

### Replaceability

The actionable alternatives are:
1. The current `nix`+`cabal.project.freeze`.
2. A `flake.nix` based upon `nixpkgs.haskellPackages` like the current !258
3. A `flake.nix` based upon fgaz's [nix-build-cabal-project](https://git.sr.ht/\~fgaz/nix-build-cabal-project), which boils down to just a [thin wrapper](https://git.sr.ht/\~fgaz/nix-build-cabal-project/tree/877a08434ce1db4c677c41c56519f85f6e12ebe5/item/default.nix#L91) around `cabal-install`. Meaning it does not package nor cache each individual Haskell dependencies into Nix, it bundles them all in a single package, which is helpful for deployment, no so helpful for development. If `cabal-install` is setup to `run-tests` and we find a way to override `run-tests` per-package, it can also check `test-suite`s.
4. A `flake.nix` based upon IOHK's [haskell.nix](https://input-output-hk.github.io/haskell.nix), which produces Nix expressions for all of Hackage. However it does not put their outputs on the default Nix cache at https://cache.nixos.org but on [IOHK's Nix cache at hydra.iohk.io](https://input-output-hk.github.io/haskell.nix/tutorials/getting-started.html#setting-up-the-binary-cache).
5. "1. + 2." but this 2. would not use the exact same package versions than to 1.
6. "1. + 3."
7. "1. + 4."
8. `guix`'s [`haskell-build-system`](https://guix.gnu.org/manual/en/html_node/Build-Systems.html), but I doubt it provides the same level of integration of Haskell than 2. or 4.

Edited Mar 05, 2024 by Julien Moutinho