Skip to content

haskell-gargantext
haskell-gargantext
Commit e9301e4e authored by Alexandre Delanoë's avatar Alexandre Delanoë
Browse files
Options

Merge remote-tracking branch 'origin/issue-436-dockerfile-and-corenlp' into dev

parents 0ab82ad5 baf97f4f
Hide whitespace changes
Inline Side-by-side
Showing with 276 additions and 225 deletions
.gitlab-ci.yml
View file @ e9301e4e
# Optimising CI speed by using tips from https://blog.nimbleways.com/let-s-make-faster-gitlab-ci-cd-pipelines/
#image: cgenie/gargantext:9.4.8
image: adinapoli/gargantext:v3.5
image: cgenie/gargantext:9.6.6
variables:
STACK_ROOT: "${CI_PROJECT_DIR}/.stack-root"
STACK_OPTS: "--system-ghc"
STORE_DIR: "${CI_PROJECT_DIR}/.cabal"
CABAL_DIR: "${CI_PROJECT_DIR}/.cabal"
CORENLP: "4.5.4"
FF_USE_FASTZIP: "true"
ARTIFACT_COMPRESSION_LEVEL: "fast"
CACHE_COMPRESSION_LEVEL: "fast"
......@@ -95,17 +93,11 @@ test:
nix-shell --run "./bin/update-project-dependencies $STORE_DIR"
mkdir -p /root/.cache/cabal/logs
chown -R test:test /root/.cache/cabal/logs/
chown -R test:test /root/.cache/cabal/packages/hackage.haskell.org/
chown -R test:test "$TEST_TMPDIR"
mkdir -p /builds/gargantext/haskell-gargantext/devops/coreNLP/stanford-corenlp-current
cp -R /root/devops/coreNLP/stanford-corenlp-${CORENLP}/* /builds/gargantext/haskell-gargantext/devops/coreNLP/stanford-corenlp-current/
nix-shell --run "chown -R test:test /root/.config/ && su -m test -c \"export PATH=$PATH:$TEST_NIX_PATH && export TMPDIR=$TEST_TMPDIR && cd /builds/gargantext/haskell-gargantext; $CABAL --store-dir=$STORE_DIR v2-test --test-show-details=streaming --flags 'test-crypto no-phylo-debug-logs' --ghc-options='-O0 -fclear-plugins'\""
chown -R root:root dist-newstyle/
chown -R root:root /root/
chown -R root:root $STORE_DIR
chown -R root:root /root/.cache/cabal/logs/
chown -R root:root /root/.cache/cabal/packages/hackage.haskell.org/
chown -Rh root:root /builds/gargantext/haskell-gargantext/devops/coreNLP/stanford-corenlp-current
README.md
View file @ e9301e4e
......@@ -103,6 +103,17 @@ Then initialize the database using the dedicated command: from inside the
And provide a name and a password for the master user as requested.
### Launching CoreNLP
CoreNLP can be started with nix:
```shell
nix-shell --run startCoreNLPServer.sh
```
By default, this starts on port `9000`. If you want a different port, then do:
```shell
nix-shell --run 'startCoreNLPServer.sh -p 9001'
```
### Running Gargantext
From inside the `haskell-gargantext/` directory, run
......@@ -237,11 +248,11 @@ The CI is on gitlab and is controlled by the `.gitlab-ci.yml` file.
The Docker image that is used can be built with:
```shell
podman build -t cgenie/gargantext:9.4.8 -f ./devops/docker/Dockerfile --format=docker .
podman build -t cgenie/gargantext:9.6.6 --pull -f ./devops/docker/Dockerfile --format=docker .
```
or
```shell
docker build -t cgenie/gargantext:9.4.8 -f ./devops/docker/Dockerfile .
docker build -t cgenie/gargantext:9.6.6 --pull -f ./devops/docker/Dockerfile .
```
NOTE: if podman/docker complains about not enough space, it's probably
......@@ -263,9 +274,9 @@ When a development is needed on libraries (for instance, the HAL crawler in http
2. When changes work and tests are OK, commit in repo `hal`
2. When changes are commited / merged:
1. Get the hash id, and edit `cabal.project` with the **new commit id**
2. run `./bin/update-project-dependencies`
2. run `nix-shell --run ./bin/update-project-dependencies`
- get an error that sha256 don't match, so update the `./bin/update-project-dependencies` with new sha256 hash
- run again `./bin/update-project-dependencies` (to make sure it's a fixed point now)
- run again `nix-shell --run ./bin/update-project-dependencies` (to make sure it's a fixed point now)
> Note: without `stack.yaml` we would have to only fix `cabal.project` -> `source-repository-package` commit id. Sha256 is there to make sure CI reruns the tests.
......
bin/install-cabal2stack deleted 100755 → 0
View file @ 0ab82ad5
#!/usr/bin/env bash
set -euxo pipefail
current_dir=$(basename "$PWD")
if [ "$current_dir" == "bin" ]; then
source ./setup-ci-environment
else
source ./bin/setup-ci-environment
fi
cabal --store-dir=$STORE_DIR v2-update "hackage.haskell.org,${INDEX_STATE}"
# Install cabal2stack if it can't be found.
if ! cabal2stack --help &> /dev/null
then
echo "cabal2stack could not be found"
CURDIR=$PWD
git clone https://github.com/iconnect/cabal2stack.git cabal2stack-installer
cd cabal2stack-installer
cabal --store-dir=$STORE_DIR v2-install --allow-newer --index-state="${INDEX_STATE}" --overwrite-policy=always
cd $CURDIR
rm -rf cabal2stack-installer
fi
bin/update-project-dependencies
View file @ e9301e4e
......@@ -6,10 +6,8 @@ current_dir=$(basename "$PWD")
if [ "$current_dir" == "bin" ]; then
source ./setup-ci-environment
./install-cabal2stack
else
source ./bin/setup-ci-environment
./bin/install-cabal2stack
fi
# README!
......
devops/coreNLP/Dockerfile deleted 100644 → 0
View file @ 0ab82ad5
FROM openjdk
#ADD home/debian/CoreNLP /CoreNLP
ADD stanford-corenlp-current /CoreNLP
WORKDIR /CoreNLP
CMD ./startServer.sh
devops/coreNLP/build.sh deleted 100755 → 0
View file @ 0ab82ad5
#!/bin/bash
# releases are here:
# https://stanfordnlp.github.io/CoreNLP/history.html
VERSION=4.5.4
FILE="stanford-corenlp-${VERSION}.zip"
DIR_V="stanford-corenlp-${VERSION}"
DIR="stanford-corenlp-current"
URL="http://nlp.stanford.edu/software/${FILE}"
[ ! -f ${FILE} ] && echo 'Fetching file' && wget ${URL} -O ${FILE}
[ ! -d ${DIR_V} ] && echo 'Unzipping file' && unzip ./${FILE}
[ ! -L ${DIR} ] && echo "Symlinking ${DIR_V} -> ${DIR}" && ln -s ${DIR_V} ${DIR}
[ ! -f ${DIR}/startServer.sh ] && echo "Copying startServer.sh" && cp ./startServer.sh ${DIR}/
echo "You can now build with: docker build -t cgenie/corenlp-garg:${VERSION}" --pull .
devops/coreNLP/startServer.sh deleted 100755 → 0
View file @ 0ab82ad5
#!/bin/sh
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
devops/docker/Dockerfile
View file @ e9301e4e
......@@ -3,16 +3,6 @@ FROM ubuntu:noble
## NOTA BENE: In order for this to be built successfully, you have to run ./devops/coreNLP/build.sh first.
ARG DEBIAN_FRONTEND=noninteractive
ARG GHC=9.6.6
ARG CORENLP=4.5.4
ARG CORE
COPY ./shell.nix /builds/gargantext/shell.nix
COPY ./nix/pkgs.nix /builds/gargantext/nix/pkgs.nix
COPY ./nix/pinned-25.05.nix /builds/gargantext/nix/pinned-25.05.nix
COPY ./devops/coreNLP/build.sh /root/devops/coreNLP/build.sh
COPY ./devops/coreNLP/startServer.sh /root/devops/coreNLP/startServer.sh
COPY ./bin/setup-ci-environment /builds/gargantext/bin/setup-ci-environment
COPY ./bin/install-cabal2stack /builds/gargantext/bin/install-cabal2stack
ENV TZ=Europe/Rome
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
......@@ -25,32 +15,15 @@ ENV PATH=/root/.local/bin:$PATH
RUN apt-get update && \
apt-get install --no-install-recommends -y \
apt-transport-https \
autoconf \
automake \
build-essential \
ca-certificates \
curl \
gcc \
git \
gnupg2 \
libffi-dev \
libffi8 \
libgmp-dev \
libgmp10 \
libncurses-dev \
libncurses6 \
libnuma-dev \
libtinfo6 \
locales \
lsb-release \
software-properties-common \
strace \
sudo \
wget \
vim \
xz-utils \
zlib1g-dev \
openjdk-21-jdk \
#zlib1g-dev \
unzip && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
mkdir -m 0755 /nix && groupadd -r nixbld && chown root /nix && \
......@@ -60,16 +33,21 @@ RUN apt-get update && \
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN cd /root/devops/coreNLP; ./build.sh && \
set -o pipefail && \
RUN set -o pipefail && \
locale-gen en_US.UTF-8 && \
bash <(curl -L https://releases.nixos.org/nix/nix-2.26.2/install) --no-daemon && \
locale-gen en_US.UTF-8 && chown root -R /nix && \
chown root -R /nix && \
. "$HOME/.nix-profile/etc/profile.d/nix.sh" && \
mkdir -p "/builds/gargantext/" && chmod 777 -R "/builds/gargantext" && \
echo "source $HOME/.nix-profile/etc/profile.d/nix.sh" >> "$HOME/.bashrc" && \
echo `which nix-env` && \
. $HOME/.bashrc && nix-env --version && \
cd /builds/gargantext && nix-shell --run "./bin/install-cabal2stack"
. $HOME/.bashrc && nix-env --version
# We want to cache nix artifacts in the Dockerfile to improve CI speed
COPY ./shell.nix /nix-ci-build/
COPY ./nix /nix-ci-build/
RUN set -o pipefail && \
pushd /nix-ci-build/ && nix-build shell.nix && popd
WORKDIR "/builds/gargantext/"
devops/docker/docker-compose.yaml
View file @ e9301e4e
......@@ -61,12 +61,6 @@ services:
# volumes:
# - pgadmin:/var/lib/pgadmin
corenlp:
#image: 'cgenie/corenlp-garg:latest'
image: 'cgenie/corenlp-garg:4.5.4'
ports:
- 9000:9000
# johnsnownlp:
# image: 'johnsnowlabs/nlp-server:latest'
# volumes:
......
devops/install-corenlp deleted 100644 → 0
View file @ 0ab82ad5
#!/bin/bash
if [ ! -d coreNLP ]; then
mkdir -v coreNLP
fi
pushd coreNLP
wget https://dl.gargantext.org/coreNLP.tar.bz2
tar xvjf coreNLP.tar.bz2
pushd home/debian/CoreNLP
./startServer.sh
nix/cabal2stack.nix 0 → 100644
View file @ e9301e4e
{ lib,
ghc,
haskellPackages,
fetchFromGitHub }:
let
src = fetchFromGitHub {
repo = "cabal2stack";
owner = "iconnect";
rev = "e4960683f518ab4c964e7646706fe2a7e1bf751a";
hash = "sha256-KE9VUXFy9QfRmu/+DFcgxV/E6oPBAR7hRaFrSt93eeY=";
};
in
with haskellPackages;
mkDerivation {
inherit src;
pname = "cabal2stack";
version = "0";
isExecutable = true;
executableHaskellDepends = [ base
aeson
cabal-plan
HsYAML
HsYAML-aeson
optics-core
optics-extra
optparse-applicative
transformers ];
patches = [
./patches/cabal2stack.patch
];
license = lib.licenses.bsd3;
mainProgram = "cabal2stack";
}
nix/corenlp.nix 0 → 100644
View file @ e9301e4e
# https://nixos.wiki/wiki/Java
{ fetchzip,
makeWrapper,
stdenv,
writeShellScript,
jre,
version ? "4.5.8",
hash ? "sha256-HZHNyw+LrxAN4OLueAhyxRG2onPWNMm5MwJDLLzsE2M="
}:
stdenv.mkDerivation (finalAttrs:
let
startServer = writeShellScript "startCoreNLPServer.sh" ''
set -x
PORT=9000
while getopts ':p:h' opt; do
case $opt in
(p) PORT=$OPTARG;;
(h) echo "$(basename $0) [-p 9000]"
exit 0
;;
esac
done
shift "$((OPTIND - 1))"
${jre}/bin/java -mx4g -cp "$CORENLP_PATH/*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port $PORT -timeout 15000 "$@"
'';
in
{
name = "corenlp";
inherit version;
src = fetchzip {
inherit hash;
url = "http://nlp.stanford.edu/software/stanford-corenlp-${finalAttrs.version}.zip";
};
buildInputs = [
jre
];
nativeBuildInputs = [
makeWrapper
];
phases = [ "unpackPhase" "installPhase" ];
installPhase = ''
runHook preInstall
mkdir -p $out/bin
mkdir -p $out/share/corenlp
cp -r . $out/share/corenlp
makeWrapper ${startServer} $out/bin/startCoreNLPServer.sh \
--set CORENLP_PATH "$out/share/corenlp"
runHook postInstall
'';
}
)
nix/graphviz.nix 0 → 100644
View file @ e9301e4e
{ fetchFromGitLab,
fetchpatch,
graphviz }:
let
graphviz_dev = graphviz.overrideAttrs (finalAttrs: previousAttrs: {
version = "11.0.0~dev";
src = fetchFromGitLab {
owner = "graphviz";
repo = "graphviz";
rev = "f3ec849249ef9cb824feb7f97449d7159e1dcb4e"; # head as of 2024-03-25, see gargantext#329
hash = "sha256-s86IqWz6zeKbcRqpV3cVQBVviHbhUSX1U8GVuJBfjC4=";
};
});
in
graphviz_dev.overrideAttrs
(finalAttrs: previousAttrs: {
# Increase the YY_BUF_SIZE, see https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/290#note_9015
patches = [
(fetchpatch {
url = "https://gist.githubusercontent.com/adinapoli/e93ca7b1d714d27f4af537716b03e3bb/raw/b9cc297c3465878da2d18ee92a3f9b8273923493/graphviz-yy-buf-size.patch";
sha256 = "sha256-8Q3tf37iYaPV50P+Vf/n263ordECiu5eKwONCy3ynV8=";
})
];
})
nix/igraph.nix 0 → 100644
View file @ e9301e4e
{ fetchFromGitHub,
igraph,
arpack,
blas,
glpk,
gmp,
lapack,
libxml2,
nanomsg,
plfit,
llvmPackages,
version ? "0.10.4",
hash ? "sha256-LsTOxUktGZcp46Ec9QH3+9C+VADMYTZZCjKF1gp36xk=" }:
igraph.overrideAttrs (finalAttrs: previousAttrs: {
inherit version;
nativeBuildInputs = previousAttrs.nativeBuildInputs;
src = fetchFromGitHub {
inherit hash;
owner = "igraph";
repo = "igraph";
rev = version;
};
postPatch = ''
echo "${version}" > IGRAPH_VERSION
'';
outputs = [ "dev" "out" "doc" ];
buildInputs = [
arpack
blas
glpk
gmp
lapack
libxml2
nanomsg
plfit
llvmPackages.openmp
];
cmakeFlags = [
"-DIGRAPH_USE_INTERNAL_BLAS=OFF"
"-DIGRAPH_USE_INTERNAL_LAPACK=OFF"
"-DIGRAPH_USE_INTERNAL_ARPACK=OFF"
"-DIGRAPH_USE_INTERNAL_GLPK=OFF"
"-DIGRAPH_USE_INTERNAL_GMP=OFF"
"-DIGRAPH_USE_INTERNAL_PLFIT=OFF"
"-DIGRAPH_GLPK_SUPPORT=ON"
"-DIGRAPH_GRAPHML_SUPPORT=OFF"
"-DIGRAPH_OPENMP_SUPPORT=ON"
"-DIGRAPH_ENABLE_LTO=AUTO"
"-DIGRAPH_ENABLE_TLS=ON"
"-DBUILD_SHARED_LIBS=ON"
"-DCMAKE_INSTALL_PREFIX=${placeholder "out"}"
"-DCMAKE_INSTALL_LIBDIR=${placeholder "out"}/lib"
"-DCMAKE_INSTALL_DATADIR=${placeholder "out"}/share"
];
doCheck = false;
postInstall = ''
mkdir -p "$out/share"
cp -r doc "$out/share"
'';
postFixup = previousAttrs.postFixup + ''
echo "Copying files where they belong .."
CUR_DIR=$PWD
cd "$dev/include/igraph" && cp *.h ../
cd $CUR_DIR
'';
})
nix/patches/cabal2stack.patch 0 → 100644
View file @ e9301e4e
diff --git i/cabal2stack.cabal w/cabal2stack.cabal
index 69767a2..92c4895 100644
--- i/cabal2stack.cabal
+++ w/cabal2stack.cabal
@@ -26,14 +26,14 @@ executable cabal2stack
-- boot dependencies
build-depends:
- , base >=4.12 && <4.18
+ , base >=4.12 && <5
, bytestring ^>=0.10.8.2 || ^>=0.11.3.0
, containers ^>=0.6.0.1
, directory ^>=1.3.3.0
, filepath ^>=1.4.2.1
, process ^>=1.6.5.0
, text >=1.2.3.0 && <2.1
- , transformers ^>=0.5.6.2
+ , transformers >=0.5.6.2 && < 0.7
-- other dependencies
build-depends:
@@ -43,4 +43,4 @@ executable cabal2stack
, HsYAML-aeson ^>=0.2.0.1
, optics-core ^>=0.4
, optics-extra ^>=0.4
- , optparse-applicative ^>=0.17.0.0
+ , optparse-applicative >=0.17.0.0 && < 0.20
nix/pkgs.nix
View file @ e9301e4e
......@@ -6,88 +6,12 @@ rec {
inherit pkgs;
ghc966 = pkgs.haskell.compiler.ghc966;
cabal_install = pkgs.haskell.lib.compose.justStaticExecutables pkgs.haskell.packages.ghc966.cabal-install;
graphviz_dev = pkgs.graphviz.overrideAttrs (finalAttrs: previousAttrs: {
version = "11.0.0~dev";
src = pkgs.fetchFromGitLab {
owner = "graphviz";
repo = "graphviz";
rev = "f3ec849249ef9cb824feb7f97449d7159e1dcb4e"; # head as of 2024-03-25, see gargantext#329
hash = "sha256-s86IqWz6zeKbcRqpV3cVQBVviHbhUSX1U8GVuJBfjC4=";
};
});
graphviz = graphviz_dev.overrideAttrs (finalAttrs: previousAttrs: {
# Increase the YY_BUF_SIZE, see https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/290#note_9015
patches = [
(pkgs.fetchpatch {
url = "https://gist.githubusercontent.com/adinapoli/e93ca7b1d714d27f4af537716b03e3bb/raw/b9cc297c3465878da2d18ee92a3f9b8273923493/graphviz-yy-buf-size.patch";
sha256 = "sha256-8Q3tf37iYaPV50P+Vf/n263ordECiu5eKwONCy3ynV8=";
})
];
});
igraph_0_10_4 = pkgs.igraph.overrideAttrs (finalAttrs: previousAttrs: {
version = "0.10.4";
nativeBuildInputs = previousAttrs.nativeBuildInputs;
src = pkgs.fetchFromGitHub {
owner = "igraph";
repo = "igraph";
rev = "0.10.4";
hash = "sha256-LsTOxUktGZcp46Ec9QH3+9C+VADMYTZZCjKF1gp36xk=";
};
postPatch = ''
echo "0.10.4" > IGRAPH_VERSION
'';
outputs = [ "dev" "out" "doc" ];
buildInputs = [
pkgs.arpack
pkgs.blas
pkgs.glpk
pkgs.gmp
pkgs.lapack
pkgs.libxml2
pkgs.nanomsg
pkgs.plfit
pkgs.llvmPackages.openmp
];
cmakeFlags = [
"-DIGRAPH_USE_INTERNAL_BLAS=OFF"
"-DIGRAPH_USE_INTERNAL_LAPACK=OFF"
"-DIGRAPH_USE_INTERNAL_ARPACK=OFF"
"-DIGRAPH_USE_INTERNAL_GLPK=OFF"
"-DIGRAPH_USE_INTERNAL_GMP=OFF"
"-DIGRAPH_USE_INTERNAL_PLFIT=OFF"
"-DIGRAPH_GLPK_SUPPORT=ON"
"-DIGRAPH_GRAPHML_SUPPORT=OFF"
"-DIGRAPH_OPENMP_SUPPORT=ON"
"-DIGRAPH_ENABLE_LTO=AUTO"
"-DIGRAPH_ENABLE_TLS=ON"
"-DBUILD_SHARED_LIBS=ON"
"-DCMAKE_INSTALL_PREFIX=${placeholder "out"}"
"-DCMAKE_INSTALL_LIBDIR=${placeholder "out"}/lib"
"-DCMAKE_INSTALL_DATADIR=${placeholder "out"}/share"
];
doCheck = false;
postInstall = ''
mkdir -p "$out/share"
cp -r doc "$out/share"
'';
postFixup = previousAttrs.postFixup + ''
echo "Copying files where they belong .."
CUR_DIR=$PWD
cd "$dev/include/igraph" && cp *.h ../
cd $CUR_DIR
'';
});
graphviz = pkgs.callPackage ./graphviz.nix {};
igraph_0_10_4 = pkgs.callPackage ./igraph.nix {};
corenlp = pkgs.callPackage ./corenlp.nix { }; # 4.5.8
cabal2stack = pkgs.callPackage ./cabal2stack.nix { ghc = ghc966; };
hsBuildInputs = [
ghc966
cabal_install
......@@ -96,34 +20,36 @@ rec {
pkgs.haskellPackages.pretty-show
];
nonhsBuildInputs = with pkgs; [
#haskell-language-server
blas
bzip2
cabal2stack
corenlp
curl
czmq
docker-compose
expat
gfortran
git
gmp
graphviz
gsl
#haskell-language-server
hlint
libffi
icu
igraph_0_10_4
jre
lapack
xz
libffi
libpqxx
libsodium
nanomsg
nil # nix language server
pcre
pkg-config
postgresql
stdenv.cc.cc
xz
zlib
blas
gfortran
expat
icu
graphviz
gcc13
igraph_0_10_4
libpqxx
libsodium
nanomsg
zeromq
curl
] ++ ( lib.optionals stdenv.isDarwin [
darwin.apple_sdk.frameworks.Accelerate
]);
......
test/drivers/hspec/Main.hs
View file @ e9301e4e
......@@ -20,8 +20,8 @@ startCoreNLPServer :: IO ProcessHandle
startCoreNLPServer = do
putText "calling start core nlp"
devNull <- openFile "/dev/null" WriteMode
let p = proc "./startServer.sh" []
(_, _, _, hdl) <- (createProcess $ p { cwd = Just "devops/coreNLP/stanford-corenlp-current"
let p = proc "startCoreNLPServer.sh" []
(_, _, _, hdl) <- (createProcess $ p { cwd = Nothing
-- NOTE(adn) Issue #451, this one has to stay disabled, because if we
-- turn it on, despite the confusing documentation on the `process` library
-- it will cause the Haskell RTS to completely ignore the Ctrl^c and instead
......@@ -34,10 +34,7 @@ startCoreNLPServer = do
, std_err = UseHandle devNull
}) `catch` \e -> case e of
_ | True <- "does not exist" `isInfixOf` (T.pack . show @SomeException $ e)
-> fail $ "Cannot execute the 'startServer.sh' script. If this is the " <>
"first time you are running the tests, you have to run " <>
"cd devops/coreNLP && ./build.sh first. You have to run it only once, " <>
"and then you are good to go for the time being."
-> fail $ "Cannot execute the 'startCoreNLPServer.sh' script. Make sure you are in a nix environment."
| otherwise -> throwIO e
pure hdl
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment