Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
haskell-gargantext
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
175
Issues
175
List
Board
Labels
Milestones
Merge Requests
10
Merge Requests
10
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
gargantext
haskell-gargantext
Commits
d97e2510
Commit
d97e2510
authored
Apr 29, 2021
by
Alexandre Delanoë
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
[FIX] testing (WIP)
parent
9f5df649
Pipeline
#1455
failed with stage
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
14 additions
and
6 deletions
+14
-6
List.hs
src/Gargantext/API/Ngrams/List.hs
+14
-6
No files found.
src/Gargantext/API/Ngrams/List.hs
View file @
d97e2510
...
@@ -39,7 +39,7 @@ import Gargantext.Database.Action.Metrics.NgramsByNode (getOccByNgramsOnlyFast')
...
@@ -39,7 +39,7 @@ import Gargantext.Database.Action.Metrics.NgramsByNode (getOccByNgramsOnlyFast')
import
Gargantext.Database.Admin.Types.Hyperdata.Document
import
Gargantext.Database.Admin.Types.Hyperdata.Document
import
Gargantext.Database.Admin.Types.Node
import
Gargantext.Database.Admin.Types.Node
import
Gargantext.Database.Query.Table.Ngrams
(
insertNgrams
)
import
Gargantext.Database.Query.Table.Ngrams
(
insertNgrams
)
import
Gargantext.Database.Query.Table.Node
(
getDocumentsWithParentId
)
import
Gargantext.Database.Query.Table.Node
Node
(
selectDocNodes
)
import
Gargantext.Database.Schema.Ngrams
import
Gargantext.Database.Schema.Ngrams
import
Gargantext.Database.Schema.Node
import
Gargantext.Database.Schema.Node
import
Gargantext.Database.Types
(
Indexed
(
..
))
import
Gargantext.Database.Types
(
Indexed
(
..
))
...
@@ -114,12 +114,16 @@ reIndexWith cId lId nt lts = do
...
@@ -114,12 +114,16 @@ reIndexWith cId lId nt lts = do
<$>
HashMap
.
toList
<$>
HashMap
.
toList
<$>
getTermsWith
identity
[
lId
]
nt
lts
<$>
getTermsWith
identity
[
lId
]
nt
lts
--
printDebug "ts" ts
--
printDebug "ts" ts
-- Taking the ngrams with 0 occurrences only (orphans)
-- Taking the ngrams with 0 occurrences only (orphans)
orphans
<-
HashMap
.
keys
occs
<-
getOccByNgramsOnlyFast'
cId
lId
nt
ts
<$>
HashMap
.
filter
(
==
0
)
<$>
getOccByNgramsOnlyFast'
cId
lId
nt
ts
let
orphans
=
List
.
concat
$
map
(
\
t
->
case
HashMap
.
lookup
t
occs
of
Nothing
->
[
t
]
Just
n
->
if
n
==
1
then
[
t
]
else
[ ]
)
ts
-- Getting the Id of orphan ngrams
-- Getting the Id of orphan ngrams
mapTextNgramsId
<-
insertNgrams
(
map
(
text2ngrams
.
unNgramsTerm
)
orphans
)
mapTextNgramsId
<-
insertNgrams
(
map
(
text2ngrams
.
unNgramsTerm
)
orphans
)
...
@@ -127,7 +131,9 @@ reIndexWith cId lId nt lts = do
...
@@ -127,7 +131,9 @@ reIndexWith cId lId nt lts = do
printDebug
"orphans"
orphans
printDebug
"orphans"
orphans
-- Get all documents of the corpus
-- Get all documents of the corpus
docs
<-
getDocumentsWithParentId
cId
docs
<-
selectDocNodes
cId
printDebug
"docs length"
(
List
.
length
docs
)
-- Checking Text documents where orphans match
-- Checking Text documents where orphans match
-- TODO Tests here
-- TODO Tests here
...
@@ -143,6 +149,8 @@ reIndexWith cId lId nt lts = do
...
@@ -143,6 +149,8 @@ reIndexWith cId lId nt lts = do
(
List
.
cycle
[
Map
.
fromList
$
[(
nt
,
Map
.
singleton
(
doc
^.
node_id
)
1
)]])
(
List
.
cycle
[
Map
.
fromList
$
[(
nt
,
Map
.
singleton
(
doc
^.
node_id
)
1
)]])
)
docs
)
docs
printDebug
"ngramsByDoc"
ngramsByDoc
-- Saving the indexation in database
-- Saving the indexation in database
_
<-
insertDocNgrams
lId
(
HashMap
.
fromList
_
<-
insertDocNgrams
lId
(
HashMap
.
fromList
$
catMaybes
$
catMaybes
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment