Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
haskell-gargantext
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
153
Issues
153
List
Board
Labels
Milestones
Merge Requests
9
Merge Requests
9
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
gargantext
haskell-gargantext
Commits
e4b72739
Commit
e4b72739
authored
Oct 08, 2021
by
Przemyslaw Kaminski
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
[zip upload] some more work on zip file
parent
52c7f251
Pipeline
#1949
failed with stage
in 32 minutes and 32 seconds
Changes
2
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
15 additions
and
7 deletions
+15
-7
New.hs
src/Gargantext/API/Node/Corpus/New.hs
+8
-4
Parsers.hs
src/Gargantext/Core/Text/Corpus/Parsers.hs
+7
-3
No files found.
src/Gargantext/API/Node/Corpus/New.hs
View file @
e4b72739
...
...
@@ -37,7 +37,7 @@ import Test.QuickCheck.Arbitrary
import
Gargantext.Prelude
import
Gargantext.API.Admin.Orchestrator.Types
(
JobLog
(
..
),
AsyncJobs
)
import
Gargantext.API.Admin.Orchestrator.Types
(
JobLog
(
..
),
AsyncJobs
,
ScraperEvent
(
..
),
scst_events
)
import
Gargantext.API.Admin.Types
(
HasSettings
)
import
Gargantext.API.Job
(
jobLogSuccess
,
jobLogFailTotal
)
import
Gargantext.API.Node.Corpus.New.File
...
...
@@ -258,7 +258,7 @@ addToCorpusWithForm user cid (NewWithForm ft d l _n) logStatus jobLog = do
WOS
->
Parser
.
parseFormat
Parser
.
WOS
PresseRIS
->
Parser
.
parseFormat
Parser
.
RisPresse
ZIP
->
Parser
.
parseFormat
Parser
.
ZIP
-- TODO granularity of the logStatus
eDocs
<-
liftBase
$
parse
$
cs
d
case
eDocs
of
...
...
@@ -283,9 +283,13 @@ addToCorpusWithForm user cid (NewWithForm ft d l _n) logStatus jobLog = do
logStatus
jobLog3
pure
$
jobLog3
Left
e
->
do
printDebug
"Error"
e
printDebug
"[addToCorpusWithForm] parse error"
e
let
evt
=
ScraperEvent
{
_scev_message
=
Just
$
T
.
pack
e
,
_scev_level
=
Just
"ERROR"
,
_scev_date
=
Nothing
}
logStatus
jobLogE
logStatus
$
over
(
scst_events
.
_Just
)
(
\
evt'
->
evt'
<>
[
evt
])
jobLogE
pure
jobLogE
where
jobLog2
=
jobLogSuccess
jobLog
...
...
src/Gargantext/Core/Text/Corpus/Parsers.hs
View file @
e4b72739
...
...
@@ -43,6 +43,7 @@ import qualified Data.ByteString.Lazy as DBL
import
qualified
Data.Map
as
DM
import
qualified
Data.Text
as
DT
import
qualified
Prelude
as
Prelude
import
System.IO.Temp
(
emptySystemTempFile
)
import
Gargantext.Core
(
Lang
(
..
))
import
Gargantext.Database.Admin.Types.Hyperdata
(
HyperdataDocument
(
..
))
...
...
@@ -95,9 +96,12 @@ parseFormat WOS bs = do
$
partitionEithers
$
[
runParser'
WOS
bs
]
pure
$
Right
docs
parseFormat
ZIP
_bs
=
do
printDebug
"[parseFormat]"
ZIP
pure
$
Left
"Not implemented for ZIP"
parseFormat
ZIP
bs
=
do
path
<-
emptySystemTempFile
"parsed-zip"
DB
.
writeFile
path
bs
parsedZip
<-
withArchive
path
$
do
DM
.
keys
<$>
getEntries
pure
$
Left
$
"Not implemented for ZIP, parsedZip"
<>
show
parsedZip
parseFormat
_
_
=
undefined
-- | Parse file into documents
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment