Skip to content

haskell-gargantext
haskell-gargantext
Commit 15c6b984 authored by Alfredo Di Napoli's avatar Alfredo Di Napoli
Browse files
Options

Add gargantext-phylo-profile executable 

It also adds a semi-interesting phylo.
parent 988f1c3f
Hide whitespace changes
Inline Side-by-side
Showing with 1233 additions and 89 deletions
.gitignore
View file @ 15c6b984
......@@ -38,3 +38,6 @@ tmp*repo*json
data
devops/docker/js-cache
cabal.project.local
gargantext_profile_out.dot
bench-data/phylo/GarganText_DocsList-nodeId-185487.csv 0 → 100644
View file @ 15c6b984
This source diff could not be displayed because it is too large. You can view the blob instead.
bench-data/phylo/GarganText_NgramsList-185488.csv 0 → 100644
View file @ 15c6b984
status label forms
map %
map % ci
candidate 17α-ethinylestradiol
map 17β-estradiol
candidate 2-ethylhexyl
candidate 4-tert-octylphenol
candidate absence
candidate action
candidate activation
candidate activities activity
candidate activity
map addition
map additional studies
map adsorption
map adult zebrafish
candidate adults
map adverse effect
map adverse effects adverse effect
candidate adverse health effects
map aim
map aim was
candidate algae
map alteration
map alterations alteration
candidate alternatives
map analogs
candidate analogues
map analysis
map androgen receptor
map animal animals
map animal models
map animals
map apoptosis
candidate application applications
candidate applications
candidate aquatic ecosystems
map aquatic environment
candidate aquatic plants
candidate ar
candidate area
candidate areas area
map article is protected
candidate assay
candidate assays assay
candidate association
candidate associations association
map balloon pulmonary angioplasty
candidate basis
map better understanding
candidate beverages
candidate binding affinity
map bioaccumulation potential
candidate biological effects
map birth outcomes
map birth weight
map bisphenol bisphenols
map bisphenol a
map bisphenol a action
map bisphenol a analogues
candidate bisphenol a exposure
candidate bisphenol a.
map bisphenol af
map bisphenol analogues
map bisphenol b
candidate bisphenol compounds
map bisphenol f
map bisphenol s bisphenol-s
candidate bisphenol-a
map bisphenol-s
map bisphenols
candidate bnct
map body weight
map boron neutron capture therapy
map bp bps
map bpa
candidate bpa analogues
map bpa caused
map bpa degradation
candidate bpa did
map bpa disrupts
map bpa exposure
map bpa exposure was
candidate bpa group
candidate bpa had
map bpa increased
map bpa is
candidate bpa leads
candidate bpa replacements
candidate bpa stimulated
map bpa treatment
map bpa was bpa is
candidate bpa was determined
candidate bpa were found
candidate bpa-bnct
candidate bpa-exposed rats
map bpaf
map bpaf was
map bpb
map bpf
map bps
map breast cancer
map c-bisphenol a
candidate caffeine
candidate carbon nanotubes
map case
map cash register receipts
candidate cat
candidate catalase
candidate cck-8
candidate cd
map cell counting kit-8
map cell proliferation
map cell viability
candidate changes
map chemical chemicals
candidate chemical analysis
map chemical exposure
map chemical oxygen demand
map chemicals
candidate chemicals were
candidate children
candidate china
candidate chromosomes
map chronic exposure
map chronic thromboembolic pulmonary hypertension
candidate cm
candidate co
candidate cod
map coexistence
map color developer
candidate combination
candidate common sources
candidate comparison
candidate complications
map compounds
map concentration
map concentration-dependent manner
map concentrations concentration
map concern
map concerns concern
map conclusion
candidate consumer products
candidate contamination
candidate contrast
map control
map control group control groups
map control groups
map controls control
map copyright
candidate cr
map critical period
map critical windows
candidate cross-sectional study
candidate crucial role
candidate cteph
candidate cu
candidate current knowledge
map cyp11a1
map danio rerio
map data
map day
map days day
map dbp
candidate decrease decreases
candidate decreases
candidate degradation
map degradation rate
candidate dehp
candidate deleterious effects
map delivery
map dermal absorption
map dermal exposure
candidate desorption
map desorption time
map detection
candidate detection limit
map determination
map development
map developmental periods
candidate diabetes
map dibutyl phthalate
candidate dietary exposure
candidate different effects
map different products
candidate different regions
map distribution distributions
map distributions
candidate dna damage
map dna methylation
candidate dose
map dose-limiting tissue
candidate doses dose
map drinking water
map e1
map e2
map ecological and human health risk
map edc
map edcs edc
candidate ee2
map effect
map effects effect
map effects are effects were
map effects were
candidate efsa
candidate embryos
map emt
map emt process
candidate endocrine disrupter
candidate endocrine disruption
map endocrine disruptor
map endocrine disruptors endocrine disruptor
map endocrine system
candidate endocrine-disrupting chemical
candidate environment
candidate environmental behaviors
candidate environmental contaminant
candidate environmental endocrine disruptors
map environmental exposure
map environmental remediation
map environmental toxicants
candidate epidemiological studies
map epoxy resins
map er
map er-dependent pathway
candidate erk1/2
map ers er
map estriol
candidate estrogen
map estrogen receptor
map estrogen receptors estrogen receptor
map estrogenic activity
candidate estrogenic properties
candidate estrogens estrogen
map estrone
candidate europe
map european food safety authority
candidate evaluation
candidate evidence
candidate evidence indicates
candidate experimental data
map exposure
map exposure was
map exposures exposure
map expression expressions
map expressions
candidate extent
map extraction
candidate fact
map females
map fetal development
map findings indicate
candidate findings suggest
map first time
candidate fish
map flow rate
map food foods
candidate food packaging
map foods
candidate formation
map fourier transform infrared spectroscopy
candidate freshwater
map freundlich model
candidate ftir
candidate function
map g creatinine
candidate g protein-coupled estrogen receptor
map gene expression
map general population
candidate gestation
candidate glucose homeostasis
map glutathione peroxidase
candidate good linear relationship
map group
map groups group
candidate growth
candidate h
map head circumference
map health effects
candidate high expression
candidate high risk
candidate high sensitivity
candidate high-resolution transmission electron microscopy
map higher bpa levels
candidate higher concentrations
map higher rates
map hippocampus
candidate hong kong
map hormonal regulation
candidate hours
candidate hplc-uv
candidate hrtem
map human exposure
map human health
candidate human plasma
candidate humans
map humans are
map humic acid
map humic acids humic acid
map hydroxyl radicals
candidate hypothalamus
candidate ici
candidate il-6
candidate immune system
candidate impact
candidate impacts impact
candidate impairment
map implantation failure
candidate important role
candidate increase
map induction
candidate influence
candidate inhibition
map initial concentration
map insulin resistance
candidate interaction interactions
candidate interactions
candidate involvement
map john wiley
map juvenile sprague-dawley rats
candidate key role
candidate kg
candidate kinetics
map l
candidate lack
candidate learning
candidate length
map level
map levels level
map limit
candidate limited number
map limits limit
candidate liver
map lod
map lods lod
candidate loq
map low concentrations
candidate low-dose bisphenol a
map low-dose bpa
map low-dose exposure
map ltd.
map major source
map male
map male mice
map male offspring
map males male
candidate manufacture
candidate mapk
candidate mass spectrometry
candidate maternal diet
map matrix effect
map mcf-7 breast cancer cells
map mcf-7 cells
candidate mechanism
candidate mechanisms mechanism
candidate method
map method was
map method was applied
candidate methods method
map mg g
candidate mg kg
candidate mg l
candidate mg/kg
candidate mg/l
candidate mice
map migration
candidate min
map mitogen-activated protein kinase
candidate ml
map mobile phase
candidate modulation
map molecular docking
candidate molecular interactions
candidate motility
map mrna expression
map myelin basic protein
map n
candidate naphthalene
candidate natural waters
map negative effect
candidate neuroendocrine disruption
map new mechanism
candidate ng/l
candidate ni
map nonylphenol
candidate np
candidate number
candidate obesity
candidate objective
candidate objective was
candidate occupational exposure
map occurrence
candidate onset
candidate order
map organic contaminants
map organic pollutants
candidate organisms
map ovarian reserve
map oxidative stress
candidate p
candidate papillary thyroid carcinoma
candidate parabens
candidate part
candidate participants
candidate pathophysiology
candidate pb
candidate pergafast
map perinatal exposure
candidate pfoa
candidate pfos
candidate ph
candidate phenol phenols
candidate phenolic compounds
candidate phenols
candidate phosphate
candidate phosphorylation
map photocatalytic mechanism
map photocatalytic properties
candidate photoluminescence
map phthalate
map phthalates phthalate
candidate physical activity
map physicochemical properties
candidate pl
map plastic industry
map plastic production
candidate plastics
candidate pnd
candidate pnd90
candidate pollutants
map polycarbonate plastic
map polycarbonate plastics polycarbonate plastic
map positive effect
candidate possible effects
map postnatal day
candidate potential
candidate potential applications
candidate potential effects
candidate potential mechanism
candidate potential risk
map potential sources
map pregnancy
candidate preliminary results
candidate prenatal bisphenol a
map prenatal exposure
map presence
map present study
map present study is present study was
map present study was
candidate previous study
map pro-inflammatory cytokines
candidate production
candidate products
candidate proliferation
map protein expression
map protein levels
candidate pseudo-second-order kinetic model
map pubertal development
candidate puberty
candidate quantification
map r
map range
map ranges range
map rat offspring
map reactive oxygen species
candidate regulation
map relation
map relative standard deviations
candidate release
candidate relevant levels
map removal
map removal efficiencies
map removal efficiency removal efficiencies
candidate reproductive development
map reproductive function
map reproductive hormones
candidate response
map result
map results result
candidate results demonstrated
map results indicate
map results indicated results indicate
candidate results revealed
map results show
map results showed results show
map results suggest
candidate review
map rights reserved
map risk
map risk assessment
map risks risk
map role
map ros
candidate s/n
map sample samples
map samples
candidate samples were
map sampling sites showed
candidate scanning electron microscope
map scanning electron microscopy
map sediment
map sediments sediment
candidate sem
map sensitivities
map sensitivity sensitivities
candidate series
candidate serum
map signal-to-noise ratio
map significant decrease
candidate significant differences
map significant increase
map simultaneous determination
map simultaneous removal
candidate soil
map sons
map sorbent
map source
map sources source
map spatial distribution
map spatial memory
candidate species
map stem cells
map structural analogues
candidate structure
map studies study
candidate studies showed
map study
map study aims
map study evaluated
map study investigated
candidate study suggests
map study was
map superoxide dismutase
candidate surface
map surface water surface waters
map surface waters
candidate susceptibility
map synaptic plasticity
map synthesis
candidate t
map taihu lake
candidate tcs
candidate testis
map tetrabromobisphenol a
map thermal paper
map thermal paper contains
candidate thermal paper receipts
map toxicity
map transcriptional activity
candidate treatment
map triclosan
candidate type types
candidate types
candidate ubiquitous endocrine-disrupting chemical
map urinary levels
map urine samples
map urine samples collected
candidate use
map utero development
map utero windows
candidate value
candidate values value
map vi
map visible light irradiation
map wastewater
map water
candidate water sources
candidate water treatment
candidate week weeks
candidate weeks
map wide range
map women
candidate work
map work was
candidate wt %
map x-ray diffraction
candidate xrd
candidate young children
candidate zebrafish
map zebrafish embryos
map μg
map μg kg
candidate μgl
bench-data/phylo/bpa-config.json 0 → 100644
View file @ 15c6b984
{
"corpusPath" : "Gargantext_DocsList-nodeId-185487.csv",
"listPath" : "Gargantext_NgramsList-185488.csv",
"outputPath" : "data",
"corpusParser" : {
"tag" : "Csv",
"_csv_limit" : 1500000
},
"listParser" : "V3",
"phyloName" : "bpa",
"phyloScale" : 2,
"similarity" : {
"tag" : "WeightedLogJaccard",
"_wlj_sensibility" : 0.5,
"_wlj_minSharedNgrams" : 1
},
"seaElevation" : {
"tag" : "Evolving",
"_evol_neighborhood" : true
},
"defaultMode" : false,
"findAncestors" : true,
"phyloSynchrony" : {
"tag" : "ByProximityThreshold",
"_bpt_threshold" : 0.6,
"_bpt_sensibility" : 0,
"_bpt_scope" : "AllBranches",
"_bpt_strategy" : "MergeAllGroups"
},
"phyloQuality" : {
"tag" : "Quality",
"_qua_granularity" : 0.1,
"_qua_minBranch" : 2
},
"timeUnit" : {
"tag" : "Week",
"_week_period" : 4,
"_week_step" : 2,
"_week_matchingFrame" : 5
},
"clique" : {
"tag" : "Fis",
"_fis_support" : 3,
"_fis_size" : 1
},
"exportLabel" : [
{
"tag" : "BranchLabel",
"_branch_labelTagger" : "MostEmergentTfIdf",
"_branch_labelSize" : 2
},
{
"tag" : "GroupLabel",
"_group_labelTagger" : "MostEmergentInclusive",
"_group_labelSize" : 2
}
],
"exportSort" : {
"tag" : "ByHierarchy",
"_sort_order" : "Desc"
},
"exportFilter" : [
{
"tag" : "ByBranchSize",
"_branch_size" : 2
}
]
}
bin/gargantext-phylo-profile/Main.hs 0 → 100644
View file @ 15c6b984
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Common
import Data.Aeson
import Data.List (nub)
import Gargantext.Core.Viz.Phylo
import Gargantext.Core.Viz.Phylo.API.Tools
import Gargantext.Core.Viz.Phylo.PhyloExport (toPhyloExport, dotToFile)
import Gargantext.Core.Viz.Phylo.PhyloMaker (toPhylo, toPhyloWithoutLink)
import Gargantext.Core.Viz.Phylo.PhyloTools (printIOMsg, printIOComment, setConfig, toPeriods, getTimePeriod, getTimeStep)
import GHC.IO.Encoding
import GHC.Stack
import Paths_gargantext
import Prelude
import qualified Data.Text as T
import Shelly
import System.Directory
--------------
-- | Main | --
--------------
phyloConfig :: FilePath -> PhyloConfig
phyloConfig outdir = PhyloConfig {
corpusPath = "corpus.csv"
, listPath = "list.csv"
, outputPath = outdir
, corpusParser = Csv {_csv_limit = 150000}
, listParser = V4
, phyloName = "phylo_profile_test"
, phyloScale = 2
, similarity = WeightedLogJaccard {_wlj_sensibility = 0.5, _wlj_minSharedNgrams = 2}
, seaElevation = Constante {_cons_start = 0.1, _cons_gap = 0.1}
, defaultMode = True
, findAncestors = True
, phyloSynchrony = ByProximityThreshold {_bpt_threshold = 0.5, _bpt_sensibility = 0.0, _bpt_scope = AllBranches, _bpt_strategy = MergeAllGroups}
, phyloQuality = Quality {_qua_granularity = 0.8, _qua_minBranch = 3}
, timeUnit = Year {_year_period = 3, _year_step = 1, _year_matchingFrame = 5}
, clique = MaxClique {_mcl_size = 5, _mcl_threshold = 1.0e-4, _mcl_filter = ByThreshold}
, exportLabel = [ BranchLabel {_branch_labelTagger = MostEmergentTfIdf, _branch_labelSize = 2}
, GroupLabel {_group_labelTagger = MostEmergentInclusive, _group_labelSize = 2}
]
, exportSort = ByHierarchy {_sort_order = Desc}
, exportFilter = [ByBranchSize {_branch_size = 3.0}]
}
main :: HasCallStack => IO ()
main = do
shelly $ escaping False $ withTmpDir $ \tdir -> do
curDir <- pwd
let output = curDir <> "/" <> "gargantext_profile_out.dot"
chdir tdir $ do
liftIO $ setLocaleEncoding utf8
bpaConfig <- liftIO $ getDataFileName "bench-data/phylo/bpa-config.json"
corpusPath' <- liftIO $ getDataFileName "bench-data/phylo/GarganText_DocsList-nodeId-185487.csv"
listPath' <- liftIO $ getDataFileName "bench-data/phylo/GarganText_NgramsList-185488.csv"
(Right config) <- fmap (\pcfg -> pcfg { outputPath = tdir
, corpusPath = corpusPath'
, listPath = listPath'
}) <$> liftIO (eitherDecodeFileStrict' bpaConfig)
mapList <- liftIO $ fileToList (listParser config) (listPath config)
corpus <- liftIO $ if (defaultMode config)
then fileToDocsDefault (corpusParser config) (corpusPath config) [Year 3 1 5,Month 3 1 5,Week 4 2 5] mapList
else fileToDocsAdvanced (corpusParser config) (corpusPath config) (timeUnit config) mapList
liftIO $ do
printIOComment (show (length corpus) <> " parsed docs from the corpus")
printIOComment (show (length $ nub $ concat $ map text corpus) <> " Size ngs_coterms")
printIOComment (show (length mapList) <> " Size ngs_terms List Map Ngrams")
printIOMsg "Reconstruct the phylo"
-- check the existing backup files
let backupPhyloWithoutLink = (outputPath config) <> "backupPhyloWithoutLink_" <> (configToSha BackupPhyloWithoutLink config) <> ".json"
let backupPhylo = (outputPath config) <> "backupPhylo_" <> (configToSha BackupPhylo config) <> ".json"
phyloWithoutLinkExists <- doesFileExist backupPhyloWithoutLink
phyloExists <- doesFileExist backupPhylo
-- reconstruct the phylo
phylo <- if phyloExists
then do
printIOMsg "Reconstruct the phylo from an existing file"
readPhylo backupPhylo
else do
if phyloWithoutLinkExists
then do
printIOMsg "Reconstruct the phylo from an existing file without links"
phyloWithoutLink <- readPhylo backupPhyloWithoutLink
writePhylo backupPhyloWithoutLink phyloWithoutLink
pure $ toPhylo (setConfig config phyloWithoutLink)
else do
printIOMsg "Reconstruct the phylo from scratch"
phyloWithoutLink <- pure $ toPhyloWithoutLink corpus config
writePhylo backupPhyloWithoutLink phyloWithoutLink
pure $ toPhylo (setConfig config phyloWithoutLink)
writePhylo backupPhylo phylo
printIOMsg "End of reconstruction, start the export"
let dot = toPhyloExport (setConfig config phylo)
dotToFile output dot
echo "Done."
bin/gargantext-phylo/Main.hs
View file @ 15c6b984
......@@ -46,8 +46,7 @@ import Gargantext.Database.Schema.Ngrams (NgramsType(..))
import Gargantext.Prelude hiding (hash, replace)
import Prelude qualified
import System.Directory (listDirectory,doesFileExist)
data Backup = BackupPhyloWithoutLink | BackupPhylo deriving (Show)
import Common
---------------
-- | Tools | --
......@@ -145,87 +144,6 @@ fileToDocsDefault parser path timeUnits lst =
-- dans le phyloMaker si default est true alors dans le setDefault ou pense à utiliser la TimeUnit de la conf
---------------
-- | Label | --
---------------
-- Config time parameters to label
timeToLabel :: PhyloConfig -> [Char]
timeToLabel config = case (timeUnit config) of
Epoch p s f -> ("time_epochs" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Year p s f -> ("time_years" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Month p s f -> ("time_months" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Week p s f -> ("time_weeks" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Day p s f -> ("time_days" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
seaToLabel :: PhyloConfig -> [Char]
seaToLabel config = case (seaElevation config) of
Constante start step -> ("sea_cst_" <> (show start) <> "_" <> (show step))
Adaptative granularity -> ("sea_adapt" <> (show granularity))
Evolving _ -> ("sea_evolv")
sensToLabel :: PhyloConfig -> [Char]
sensToLabel config = case (similarity config) of
Hamming _ _ -> Prelude.error "sensToLabel: unimplemented"
WeightedLogJaccard s _ -> ("WeightedLogJaccard_" <> show s)
WeightedLogSim s _ -> ( "WeightedLogSim-sens_" <> show s)
cliqueToLabel :: PhyloConfig -> [Char]
cliqueToLabel config = case (clique config) of
Fis s s' -> "fis_" <> (show s) <> "_" <> (show s')
MaxClique s t f -> "clique_" <> (show s)<> "_" <> (show f)<> "_" <> (show t)
syncToLabel :: PhyloConfig -> [Char]
syncToLabel config = case (phyloSynchrony config) of
ByProximityThreshold scl sync_sens scope _ -> ("scale_" <> (show scope) <> "_" <> (show sync_sens) <> "_" <> (show scl))
ByProximityDistribution _ _ -> "syncToLabel: unimplemented"
qualToConfig :: PhyloConfig -> [Char]
qualToConfig config = case (phyloQuality config) of
Quality g m -> "quality_" <> (show g) <> "_" <> (show m)
-- To set up the export file's label from the configuration
configToLabel :: PhyloConfig -> [Char]
configToLabel config = outputPath config
<> (unpack $ phyloName config)
<> "-" <> (timeToLabel config)
<> "-scale_" <> (show (phyloScale config))
<> "-" <> (seaToLabel config)
<> "-" <> (sensToLabel config)
<> "-" <> (cliqueToLabel config)
<> "-level_" <> (show (_qua_granularity $ phyloQuality config))
<> "-" <> (syncToLabel config)
<> ".dot"
-- To write a sha256 from a set of config's parameters
configToSha :: Backup -> PhyloConfig -> [Char]
configToSha stage config = unpack
$ replace "/" "-"
$ T.pack (show (hash $ C8.pack label))
where
label :: [Char]
label = case stage of
BackupPhyloWithoutLink -> (corpusPath config)
<> (listPath config)
<> (timeToLabel config)
<> (cliqueToLabel config)
BackupPhylo -> (corpusPath config)
<> (listPath config)
<> (timeToLabel config)
<> (cliqueToLabel config)
<> (sensToLabel config)
<> (seaToLabel config)
<> (syncToLabel config)
<> (qualToConfig config)
<> (show (phyloScale config))
readListV4 :: [Char] -> IO NgramsList
readListV4 path = do
......
bin/gargantext-phylo/Phylo/Common.hs 0 → 100644
View file @ 15c6b984
{-# LANGUAGE OverloadedStrings #-}
module Common where
import Control.Concurrent.Async (mapConcurrently)
import Crypto.Hash.SHA256 (hash)
import Data.Aeson
import Data.ByteString.Char8 qualified as C8
import Data.List (nub, tail)
import Data.List.Split
import Data.Maybe (fromJust)
import Data.Text (unpack, replace, pack)
import Data.Text qualified as T
import Data.Vector qualified as Vector
import Gargantext.API.Ngrams.Prelude (toTermList)
import Gargantext.API.Ngrams.Types
import Gargantext.Core.Text.Context (TermList)
import Gargantext.Core.Text.Corpus.Parsers (FileFormat(..), FileType(..), parseFile)
import Gargantext.Core.Text.Corpus.Parsers.CSV (csv_title, csv_abstract, csv_publication_year, csv_publication_month, csv_publication_day, csv'_source, csv'_title, csv'_abstract, csv'_publication_year, csv'_publication_month, csv'_publication_day, csv'_weight)
import Gargantext.Core.Text.Corpus.Parsers.CSV qualified as Csv
import Gargantext.Core.Text.List.Formats.CSV (csvMapTermList)
import Gargantext.Core.Text.Terms.WithList (Patterns, buildPatterns, extractTermsWithList)
import Gargantext.Core.Types.Main (ListType(..))
import Gargantext.Core.Viz.Phylo
import Gargantext.Core.Viz.Phylo.API.Tools
import Gargantext.Core.Viz.Phylo.PhyloTools (toPeriods, getTimePeriod, getTimeStep)
import Gargantext.Database.Admin.Types.Hyperdata (HyperdataDocument(..))
import Gargantext.Database.Schema.Ngrams (NgramsType(..))
import Gargantext.Prelude hiding (hash, replace)
import Prelude qualified
import System.Directory (listDirectory)
data Backup = BackupPhyloWithoutLink | BackupPhylo deriving (Show)
---------------
-- | Tools | --
---------------
-- | To get all the files in a directory or just a file
getFilesFromPath :: FilePath -> IO [FilePath]
getFilesFromPath path = do
if (isSuffixOf "/" path)
then (listDirectory path)
else return [path]
----------------
-- | Parser | --
----------------
-- | To filter the Ngrams of a document based on the termList
termsInText :: Patterns -> Text -> [Text]
termsInText pats txt = nub $ concat $ map (map unwords) $ extractTermsWithList pats txt
-- | To transform a Wos file (or [file]) into a list of Docs
wosToDocs :: Int -> Patterns -> TimeUnit -> FilePath -> IO [Document]
wosToDocs limit patterns time path = do
files <- getFilesFromPath path
take limit
<$> map (\d -> let title = fromJust $ _hd_title d
abstr = if (isJust $ _hd_abstract d)
then fromJust $ _hd_abstract d
else ""
in Document (toPhyloDate
(fromIntegral $ fromJust $ _hd_publication_year d)
(fromJust $ _hd_publication_month d)
(fromJust $ _hd_publication_day d) time)
(toPhyloDate'
(fromIntegral $ fromJust $ _hd_publication_year d)
(fromJust $ _hd_publication_month d)
(fromJust $ _hd_publication_day d) time)
(termsInText patterns $ title <> " " <> abstr) Nothing [] time)
<$> concat
<$> mapConcurrently (\file ->
filter (\d -> (isJust $ _hd_publication_year d)
&& (isJust $ _hd_title d))
<$> fromRight [] <$> parseFile WOS Plain (path <> file) ) files
-- To transform a Csv file into a list of Document
csvToDocs :: CorpusParser -> Patterns -> TimeUnit -> FilePath -> IO [Document]
csvToDocs parser patterns time path =
case parser of
Wos _ -> Prelude.error "csvToDocs: unimplemented"
Csv limit -> Vector.toList
<$> Vector.take limit
<$> Vector.map (\row -> Document (toPhyloDate (Csv.fromMIntOrDec Csv.defaultYear $ csv_publication_year row) (fromMaybe Csv.defaultMonth $ csv_publication_month row) (fromMaybe Csv.defaultDay $ csv_publication_day row) time)
(toPhyloDate' (Csv.fromMIntOrDec Csv.defaultYear $ csv_publication_year row) (fromMaybe Csv.defaultMonth $ csv_publication_month row) (fromMaybe Csv.defaultDay $ csv_publication_day row) time)
(termsInText patterns $ (csv_title row) <> " " <> (csv_abstract row))
Nothing
[]
time
) <$> snd <$> either (\err -> panicTrace $ "CSV error" <> (show err)) identity <$> Csv.readCSVFile path
Csv' limit -> Vector.toList
<$> Vector.take limit
<$> Vector.map (\row -> Document (toPhyloDate (csv'_publication_year row) (csv'_publication_month row) (csv'_publication_day row) time)
(toPhyloDate' (csv'_publication_year row) (csv'_publication_month row) (csv'_publication_day row) time)
(termsInText patterns $ (csv'_title row) <> " " <> (csv'_abstract row))
(Just $ csv'_weight row)
(map (T.strip . pack) $ splitOn ";" (unpack $ (csv'_source row)))
time
) <$> snd <$> Csv.readWeightedCsv path
-- To parse a file into a list of Document
fileToDocsAdvanced :: CorpusParser -> FilePath -> TimeUnit -> TermList -> IO [Document]
fileToDocsAdvanced parser path time lst = do
let patterns = buildPatterns lst
case parser of
Wos limit -> wosToDocs limit patterns time path
Csv _ -> csvToDocs parser patterns time path
Csv' _ -> csvToDocs parser patterns time path
fileToDocsDefault :: CorpusParser -> FilePath -> [TimeUnit] -> TermList -> IO [Document]
fileToDocsDefault parser path timeUnits lst =
if length timeUnits > 0
then
do
let timeUnit = (head' "fileToDocsDefault" timeUnits)
docs <- fileToDocsAdvanced parser path timeUnit lst
let periods = toPeriods (sort $ nub $ map date docs) (getTimePeriod timeUnit) (getTimeStep timeUnit)
if (length periods < 3)
then fileToDocsDefault parser path (tail timeUnits) lst
else pure docs
else panicTrace "this corpus is incompatible with the phylomemy reconstruction"
-- on passe à passer la time unit dans la conf envoyé au phyloMaker
-- dans le phyloMaker si default est true alors dans le setDefault ou pense à utiliser la TimeUnit de la conf
readListV4 :: [Char] -> IO NgramsList
readListV4 path = do
listJson <- (eitherDecode <$> readJson path) :: IO (Either Prelude.String NgramsList)
case listJson of
Left err -> do
putStrLn err
Prelude.error "readListV4 unimplemented"
Right listV4 -> pure listV4
fileToList :: ListParser -> FilePath -> IO TermList
fileToList parser path =
case parser of
V3 -> csvMapTermList path
V4 -> fromJust
<$> toTermList MapTerm NgramsTerms
<$> readListV4 path
---------------
-- | Label | --
---------------
-- Config time parameters to label
timeToLabel :: PhyloConfig -> [Char]
timeToLabel config = case (timeUnit config) of
Epoch p s f -> ("time_epochs" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Year p s f -> ("time_years" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Month p s f -> ("time_months" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Week p s f -> ("time_weeks" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
Day p s f -> ("time_days" <> "_" <> (show p) <> "_" <> (show s) <> "_" <> (show f))
seaToLabel :: PhyloConfig -> [Char]
seaToLabel config = case (seaElevation config) of
Constante start step -> ("sea_cst_" <> (show start) <> "_" <> (show step))
Adaptative granularity -> ("sea_adapt" <> (show granularity))
Evolving _ -> ("sea_evolv")
sensToLabel :: PhyloConfig -> [Char]
sensToLabel config = case (similarity config) of
Hamming _ _ -> Prelude.error "sensToLabel: unimplemented"
WeightedLogJaccard s _ -> ("WeightedLogJaccard_" <> show s)
WeightedLogSim s _ -> ( "WeightedLogSim-sens_" <> show s)
cliqueToLabel :: PhyloConfig -> [Char]
cliqueToLabel config = case (clique config) of
Fis s s' -> "fis_" <> (show s) <> "_" <> (show s')
MaxClique s t f -> "clique_" <> (show s)<> "_" <> (show f)<> "_" <> (show t)
syncToLabel :: PhyloConfig -> [Char]
syncToLabel config = case (phyloSynchrony config) of
ByProximityThreshold scl sync_sens scope _ -> ("scale_" <> (show scope) <> "_" <> (show sync_sens) <> "_" <> (show scl))
ByProximityDistribution _ _ -> "syncToLabel: unimplemented"
qualToConfig :: PhyloConfig -> [Char]
qualToConfig config = case (phyloQuality config) of
Quality g m -> "quality_" <> (show g) <> "_" <> (show m)
-- To set up the export file's label from the configuration
configToLabel :: PhyloConfig -> [Char]
configToLabel config = outputPath config
<> (unpack $ phyloName config)
<> "-" <> (timeToLabel config)
<> "-scale_" <> (show (phyloScale config))
<> "-" <> (seaToLabel config)
<> "-" <> (sensToLabel config)
<> "-" <> (cliqueToLabel config)
<> "-level_" <> (show (_qua_granularity $ phyloQuality config))
<> "-" <> (syncToLabel config)
<> ".dot"
-- To write a sha256 from a set of config's parameters
configToSha :: Backup -> PhyloConfig -> [Char]
configToSha stage config = unpack
$ replace "/" "-"
$ T.pack (show (hash $ C8.pack label))
where
label :: [Char]
label = case stage of
BackupPhyloWithoutLink -> (corpusPath config)
<> (listPath config)
<> (timeToLabel config)
<> (cliqueToLabel config)
BackupPhylo -> (corpusPath config)
<> (listPath config)
<> (timeToLabel config)
<> (cliqueToLabel config)
<> (sensToLabel config)
<> (seaToLabel config)
<> (syncToLabel config)
<> (qualToConfig config)
<> (show (phyloScale config))
gargantext.cabal
View file @ 15c6b984
......@@ -17,6 +17,9 @@ license: AGPL-3
license-file: LICENSE
build-type: Simple
data-files:
bench-data/phylo/bpa-config.json
bench-data/phylo/GarganText_DocsList-nodeId-185487.csv
bench-data/phylo/GarganText_NgramsList-185488.csv
bench-data/phylo/issue-290.json
bench-data/phylo/issue-290-small.json
devops/postgres/extensions.sql
......@@ -846,8 +849,9 @@ executable gargantext-phylo
main-is: Main.hs
other-modules:
Paths_gargantext
Common
hs-source-dirs:
bin/gargantext-phylo
bin/gargantext-phylo bin/gargantext-phylo/Phylo
default-extensions:
DataKinds
DeriveGeneric
......@@ -1199,3 +1203,27 @@ benchmark garg-bench
ghc-options: "-with-rtsopts=-T -A32m"
if impl(ghc >= 8.6)
ghc-options: "-with-rtsopts=--nonmoving-gc"
executable gargantext-phylo-profile
main-is: Main.hs
other-modules:
Paths_gargantext
Common
hs-source-dirs:
bin/gargantext-phylo-profile bin/gargantext-phylo/Phylo
default-extensions: GHC2021
ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N -O2 -Wmissing-signatures
build-depends:
base
, bytestring
, gargantext
, gargantext-prelude
, shelly
, text
, async
, cryptohash
, aeson
, split
, vector
, directory
default-language: Haskell2010
src/Gargantext/Core/Text/List/Formats/CSV.hs
View file @ 15c6b984
......@@ -64,10 +64,11 @@ instance ToNamedRecord CsvList where
]
------------------------------------------------------------------------
instance FromField CsvListType where
parseField "map" = pure CsvMap
parseField "main" = pure CsvCandidate
parseField "stop" = pure CsvStop
parseField _ = mzero
parseField "map" = pure CsvMap
parseField "main" = pure CsvCandidate
parseField "candidate" = pure CsvCandidate -- backward compat
parseField "stop" = pure CsvStop
parseField _ = mzero
instance ToField CsvListType where
toField CsvMap = "map"
......
src/Gargantext/Core/Viz/Phylo/API/Tools.hs
View file @ 15c6b984
......@@ -195,7 +195,7 @@ toPhyloDate' y m d _ = pack $ showGregorian $ fromGregorian (toIntege
-- Utils
writePhylo :: [Char] -> Phylo -> IO ()
writePhylo :: HasCallStack => [Char] -> Phylo -> IO ()
writePhylo path phylo = Lazy.writeFile path $ encode phylo
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment