Skip to content

haskell-gargantext
haskell-gargantext
Verified Commit 29aee119 authored by Przemyslaw Kaminski's avatar Przemyslaw Kaminski
Browse files
Options

Merge branch 'dev' into 184-dev-add-support-for-multiple-languages-in-ini-file

parents 04533163 c55902b9
Pipeline #3884 failed with stage
    in 30 minutes and 16 seconds
    Hide whitespace changes
    Inline Side-by-side
    Showing with 115 additions and 34 deletions
    CHANGELOG.md
    View file @ 29aee119
    ## Version 0.0.6.9.8.6.2
    * [BACK][FIX][Progress Bar (#192)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/192)
    ## Version 0.0.6.9.8.6.1
    * [FRONT][ADM][ Do not use `master` as a revision for `easy-ps.nix`](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/merge_requests/339)
    ## Version 0.0.6.9.8.6
    * [BACK][DOC][Welcome: Door To enter the project (#177)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/177)
    * [BACK][FEAT] Adding more redundancies of Text in FrameWrite parsing (Notes)
    * [BACK][FIX] Clean code for documentation haddock
    * [BACK][FIX][Doc Table: count (#175)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/175)
    * [BACK][FIX][Error WOS import](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/196)
    * [BACK][FIX][List upload does not work any more (#191)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/191)
    * [BACK][FIX][NLP parsing] Removing urls which are untokenable for Spacy
    * [BACK][FIX][query pubmed, arxiv, istex, HAL not working anymore (#193)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/193)
    * [FRONT][DOC][[Graph Toolbar] Add tooltips/legends on buttons (#529)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/529)
    * [FRONT][FIX][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    ## Version 0.0.6.9.8.5.1
    * [FRONT][FIX][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    ## Version 0.0.6.9.8.5
    * [BACK][REFACT][JOBs' API (#185)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/185)
    ## Version 0.0.6.9.8.4
    * [FRONT][FIX][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    * [BACK][FIX][query pubmed, arxiv, istex, HAL not working anymore (#193)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/193)
    * [BACK][FIX] Username could be an email
    ## Version 0.0.6.9.8.3
    * [BACK][REFACT][JOBs' API (#185)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/185)
    * [FRONT][CSS][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    * [BACK][FEAT][Adding New Language NLP tools through gargantext.ini (postagging case) (#184)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/184)
    ## Version 0.0.6.9.8.2.2
    * [FRONT][CSS][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    ## Version 0.0.6.9.8.2.1
    * [BACK][FIX][FrameCalc POST method not allowed any more (#190)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/190) (cherry-pick to testing)
    * [BACK][OPTIM] Quality tested and optimized Order 2 (with recursive clustering)
    ## Version 0.0.6.9.8.2
    * [BACK][OPTIM] Back to Distri Similarity (2)
    ## Version 0.0.6.9.8.1
    * [BACK][FEAT/ADM][Adding New Language NLP tools through gargantext.ini (postagging case) (#184)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/184)
    * UPGRADE: update your gargantext ini file with the [nlp] section check the gargantext.ini_toModify file as example
    * [BACK][SPACY] NLP server tested with FR lang
    ## Version 0.0.6.9.8
    * [BACK][FRONT][FEAT] Phylo 1 click
    ## Version 0.0.6.9.7.6
    * [BACK][UPGRADE] Igraph bindings
    * [BACK][NIX][Adapt nix configuration of haskell-igraph (#187)](https://gitlab.iscpif.fr/gargantext/haskell-gargantext/issues/187)
    * [FRONT][CLEAN] Node Graph, one option only
    ## Version 0.0.6.9.7.5
    * [FRONT][CSS][Improve global theme (#509)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/509)
    * [FRONT][CSS][FA send button not available (#517)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/517)
    ## Version 0.0.6.9.7.4
    * [FRONT][FIX] Node Settings : Main nodes (Private, Shared, Public) can not be deleted or created.
    ## Version 0.0.6.9.7.3 ## Version 0.0.6.9.7.3
    * [FRONT][FIX][[hackathon] GraphExplorer fixes (#489)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/489) * [FRONT][FIX][[hackathon] GraphExplorer fixes (#489)](https://gitlab.iscpif.fr/gargantext/purescript-gargantext/issues/489)
    ......
    gargantext.cabal
    View file @ 29aee119
    ...@@ -5,7 +5,7 @@ cabal-version: 1.12 ...@@ -5,7 +5,7 @@ cabal-version: 1.12
    -- see: https://github.com/sol/hpack -- see: https://github.com/sol/hpack
    name: gargantext name: gargantext
    version: 0.0.6.9.8.6 version: 0.0.6.9.8.6.2
    synopsis: Search, map, share synopsis: Search, map, share
    description: Please see README.md description: Please see README.md
    category: Data category: Data
    ......
    package.yaml
    View file @ 29aee119
    ...@@ -6,7 +6,7 @@ name: gargantext ...@@ -6,7 +6,7 @@ name: gargantext
    # | | | +----- Layers * : New versions with API additions # | | | +----- Layers * : New versions with API additions
    # | | | | +--- Layers * : New versions without API breaking changes # | | | | +--- Layers * : New versions without API breaking changes
    # | | | | | # | | | | |
    version: '0.0.6.9.8.6' version: '0.0.6.9.8.6.2'
    synopsis: Search, map, share synopsis: Search, map, share
    description: Please see README.md description: Please see README.md
    category: Data category: Data
    ......
    src/Gargantext/API/Admin/EnvTypes.hs
    View file @ 29aee119
    ...@@ -18,11 +18,11 @@ module Gargantext.API.Admin.EnvTypes ( ...@@ -18,11 +18,11 @@ module Gargantext.API.Admin.EnvTypes (
    , ConcreteJobHandle -- opaque , ConcreteJobHandle -- opaque
    ) where ) where
    import Control.Lens hiding ((:>)) import Control.Lens hiding ((:<))
    import Control.Monad.Except import Control.Monad.Except
    import Control.Monad.Reader import Control.Monad.Reader
    import Data.Pool (Pool) import Data.Pool (Pool)
    import Data.Sequence (Seq, ViewR(..), viewr) import Data.Sequence (Seq, ViewL(..), viewl)
    import Database.PostgreSQL.Simple (Connection) import Database.PostgreSQL.Simple (Connection)
    import GHC.Generics (Generic) import GHC.Generics (Generic)
    import Network.HTTP.Client (Manager) import Network.HTTP.Client (Manager)
    ...@@ -159,12 +159,12 @@ instance Jobs.MonadJobStatus (GargM Env err) where ...@@ -159,12 +159,12 @@ instance Jobs.MonadJobStatus (GargM Env err) where
    Just j -> case jTask j of Just j -> case jTask j of
    QueuedJ _ -> pure noJobLog QueuedJ _ -> pure noJobLog
    RunningJ rj -> liftIO (rjGetLog rj) <&> RunningJ rj -> liftIO (rjGetLog rj) <&>
    \lgs -> case viewr lgs of \lgs -> case viewl lgs of
    EmptyR -> noJobLog EmptyL -> noJobLog
    _ :> l -> l l :< _ -> l
    DoneJ lgs _ -> pure $ case viewr lgs of DoneJ lgs _ -> pure $ case viewl lgs of
    EmptyR -> noJobLog EmptyL -> noJobLog
    _ :> l -> l l :< _ -> l
    withTracer extraLogger (JobHandle jId logger) n = n (JobHandle jId (\w -> logger w >> liftIO (extraLogger w))) withTracer extraLogger (JobHandle jId logger) n = n (JobHandle jId (\w -> logger w >> liftIO (extraLogger w)))
    ......
    src/Gargantext/API/Node/Corpus/New.hs
    View file @ 29aee119
    ...@@ -20,6 +20,7 @@ module Gargantext.API.Node.Corpus.New ...@@ -20,6 +20,7 @@ module Gargantext.API.Node.Corpus.New
    import Conduit import Conduit
    import Control.Lens hiding (elements, Empty) import Control.Lens hiding (elements, Empty)
    import Control.Monad
    import Data.Aeson import Data.Aeson
    import Data.Aeson.TH (deriveJSON) import Data.Aeson.TH (deriveJSON)
    import qualified Data.ByteString.Base64 as BSB64 import qualified Data.ByteString.Base64 as BSB64
    ...@@ -193,7 +194,6 @@ addToCorpusWithQuery user cid (WithQuery { _wq_query = q ...@@ -193,7 +194,6 @@ addToCorpusWithQuery user cid (WithQuery { _wq_query = q
    , _wq_lang = l , _wq_lang = l
    , _wq_flowListWith = flw }) maybeLimit jobHandle = do , _wq_flowListWith = flw }) maybeLimit jobHandle = do
    -- TODO ... -- TODO ...
    markStarted 3 jobHandle
    -- printDebug "[addToCorpusWithQuery] (cid, dbs)" (cid, dbs) -- printDebug "[addToCorpusWithQuery] (cid, dbs)" (cid, dbs)
    -- printDebug "[addToCorpusWithQuery] datafield" datafield -- printDebug "[addToCorpusWithQuery] datafield" datafield
    -- printDebug "[addToCorpusWithQuery] flowListWith" flw -- printDebug "[addToCorpusWithQuery] flowListWith" flw
    ...@@ -202,49 +202,40 @@ addToCorpusWithQuery user cid (WithQuery { _wq_query = q ...@@ -202,49 +202,40 @@ addToCorpusWithQuery user cid (WithQuery { _wq_query = q
    Just Web -> do Just Web -> do
    -- printDebug "[addToCorpusWithQuery] processing web request" datafield -- printDebug "[addToCorpusWithQuery] processing web request" datafield
    markStarted 1 jobHandle
    _ <- triggerSearxSearch user cid q l jobHandle _ <- triggerSearxSearch user cid q l jobHandle
    markComplete jobHandle markComplete jobHandle
    _ -> do _ -> do
    markStarted 3 jobHandle
    -- TODO add cid -- TODO add cid
    -- TODO if cid is folder -> create Corpus -- TODO if cid is folder -> create Corpus
    -- if cid is corpus -> add to corpus -- if cid is corpus -> add to corpus
    -- if cid is root -> create corpus in Private -- if cid is root -> create corpus in Private
    -- printDebug "[G.A.N.C.New] getDataText with query" q -- printDebug "[G.A.N.C.New] getDataText with query" q
    databaseOrigin <- database2origin dbs db <- database2origin dbs
    eTxts <- mapM (\db -> getDataText db (Multi l) q maybeLimit) [databaseOrigin] eTxt <- getDataText db (Multi l) q maybeLimit
    let lTxts = lefts eTxts
    -- printDebug "[G.A.N.C.New] lTxts" lTxts -- printDebug "[G.A.N.C.New] lTxts" lTxts
    case lTxts of case eTxt of
    [] -> do Right txt -> do
    let txts = rights eTxts
    -- TODO Sum lenghts of each txt elements -- TODO Sum lenghts of each txt elements
    -- NOTE(adinapoli) Some other weird arithmetic to have the markProgress 1 jobHandle
    -- following 'JobLog' as output:
    -- JobLog
    -- { _scst_succeeded = Just 2
    -- , _scst_failed = Just 0
    -- , _scst_remaining = Just $ 1 + length txts
    -- , _scst_events = Just []
    -- }
    markStarted (3 + length txts) jobHandle
    markProgress 2 jobHandle
    _cids <- mapM (\txt -> do void $ flowDataText user txt (Multi l) cid (Just flw) jobHandle
    flowDataText user txt (Multi l) cid (Just flw) jobHandle) txts
    -- printDebug "corpus id" cids -- printDebug "corpus id" cids
    -- printDebug "sending email" ("xxxxxxxxxxxxxxxxxxxxx" :: Text) -- printDebug "sending email" ("xxxxxxxxxxxxxxxxxxxxx" :: Text)
    sendMail user sendMail user
    -- TODO ... -- TODO ...
    markComplete jobHandle markComplete jobHandle
    (err:_) -> do Left err -> do
    -- printDebug "Error: " err -- printDebug "Error: " err
    markFailure 1 (Just $ T.pack (show err)) jobHandle markFailed (Just $ T.pack (show err)) jobHandle
    type AddWithForm = Summary "Add with FormUrlEncoded to corpus endpoint" type AddWithForm = Summary "Add with FormUrlEncoded to corpus endpoint"
    :> "corpus" :> "corpus"
    ......
    src/Gargantext/Utils/Jobs/Internal.hs
    View file @ 29aee119
    ...@@ -118,6 +118,15 @@ pollJob limit offset jid je = do ...@@ -118,6 +118,15 @@ pollJob limit offset jid je = do
    let st = either (const SJ.IsFailure) (const SJ.IsFinished) r let st = either (const SJ.IsFailure) (const SJ.IsFinished) r
    me = either (Just . T.pack . show) (const Nothing) r me = either (Just . T.pack . show) (const Nothing) r
    in pure (ls, st, me) in pure (ls, st, me)
    -- /NOTE/: We need to be careful with the ordering of the logs here:
    -- we want to return the logs ordered from the newest to the oldest,
    -- because the API will use 'limit' to show only the newest ones,
    -- taking 'limit' of them from the front of the list.
    --
    -- Due to the fact we do not force any 'Ord' constraint on an 'event' type,
    -- and it would be inefficient to reverse the list here, it's important
    -- that the concrete implementation of 'rjGetLog' returns the logs in the
    -- correct order.
    pure $ SJ.jobStatus jid limit offset (toList logs) status merr pure $ SJ.jobStatus jid limit offset (toList logs) status merr
    waitJob waitJob
    ...@@ -176,4 +185,5 @@ killJob t limit offset jid je = do ...@@ -176,4 +185,5 @@ killJob t limit offset jid je = do
    me = either (Just . T.pack . show) (const Nothing) r me = either (Just . T.pack . show) (const Nothing) r
    removeJob False t jid removeJob False t jid
    pure (lgs, st, me) pure (lgs, st, me)
    -- /NOTE/: Same proviso as in 'pollJob' applies here.
    pure $ SJ.jobStatus jid limit offset (toList logs) status merr pure $ SJ.jobStatus jid limit offset (toList logs) status merr
    src/Gargantext/Utils/Jobs/Map.hs
    View file @ 29aee119
    ...@@ -13,7 +13,6 @@ module Gargantext.Utils.Jobs.Map ( ...@@ -13,7 +13,6 @@ module Gargantext.Utils.Jobs.Map (
    , newJobMap , newJobMap
    , lookupJob , lookupJob
    , gcThread , gcThread
    , jobLog
    , addJobEntry , addJobEntry
    , deleteJob , deleteJob
    , runJob , runJob
    ...@@ -117,8 +116,10 @@ gcThread js (JobMap mvar) = go ...@@ -117,8 +116,10 @@ gcThread js (JobMap mvar) = go
    _ -> False _ -> False
    -- | Make a 'Logger' that 'mappend's monoidal values in a 'TVar'. -- | Make a 'Logger' that 'mappend's monoidal values in a 'TVar'.
    -- /IMPORTANT/: The new value is appended in front. The ordering is important later on
    -- when consuming logs from the API (see for example 'pollJob').
    jobLog :: Semigroup w => TVar w -> Logger w -- w -> IO () jobLog :: Semigroup w => TVar w -> Logger w -- w -> IO ()
    jobLog logvar = \w -> atomically $ modifyTVar' logvar (\old_w -> old_w <> w) jobLog logvar = \w -> atomically $ modifyTVar' logvar (\old_w -> w <> old_w)
    -- | Generating new 'JobEntry's. -- | Generating new 'JobEntry's.
    addJobEntry addJobEntry
    ......
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment