Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
haskell-gargantext
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
195
Issues
195
List
Board
Labels
Milestones
Merge Requests
12
Merge Requests
12
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
gargantext
haskell-gargantext
Commits
9b4bef67
Verified
Commit
9b4bef67
authored
Aug 08, 2023
by
Przemyslaw Kaminski
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
[corenlp] better debugging for corenlp errors
Also, some openalex fixes.
parent
5d0461e3
Pipeline
#4493
failed with stages
in 10 minutes and 13 seconds
Changes
3
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
15 additions
and
7 deletions
+15
-7
cabal.project
cabal.project
+1
-1
OpenAlex.hs
src/Gargantext/Core/Text/Corpus/API/OpenAlex.hs
+1
-1
PosTagging.hs
src/Gargantext/Core/Text/Terms/Multi/PosTagging.hs
+13
-5
No files found.
cabal.project
View file @
9b4bef67
...
@@ -99,7 +99,7 @@ source-repository-package
...
@@ -99,7 +99,7 @@ source-repository-package
source
-
repository
-
package
source
-
repository
-
package
type
:
git
type
:
git
location
:
https
://
gitlab
.
iscpif
.
fr
/
gargantext
/
crawlers
/
openalex
.
git
location
:
https
://
gitlab
.
iscpif
.
fr
/
gargantext
/
crawlers
/
openalex
.
git
tag
:
dab07cb89e8ab8eaaff8619f5e21d944d9c526ab
tag
:
1
cf872fb3bd0e3e44af31247833c4b6bb7d0dca5
source
-
repository
-
package
source
-
repository
-
package
type
:
git
type
:
git
...
...
src/Gargantext/Core/Text/Corpus/API/OpenAlex.hs
View file @
9b4bef67
...
@@ -26,7 +26,7 @@ get :: Text
...
@@ -26,7 +26,7 @@ get :: Text
->
Maybe
Limit
->
Maybe
Limit
->
IO
(
Either
ClientError
(
Maybe
Integer
,
ConduitT
()
HyperdataDocument
IO
()
))
->
IO
(
Either
ClientError
(
Maybe
Integer
,
ConduitT
()
HyperdataDocument
IO
()
))
get
_email
q
lang
mLimit
=
do
get
_email
q
lang
mLimit
=
do
let
limit
=
getLimit
$
fromMaybe
1000
0
mLimit
let
limit
=
getLimit
$
fromMaybe
1000
mLimit
let
mFilter
=
(
\
l
->
"language:"
<>
l
)
<$>
toISO639Lang
lang
let
mFilter
=
(
\
l
->
"language:"
<>
l
)
<$>
toISO639Lang
lang
eRes
<-
OA
.
fetchWorksC
Nothing
mFilter
$
Just
$
Corpus
.
getRawQuery
q
eRes
<-
OA
.
fetchWorksC
Nothing
mFilter
$
Just
$
Corpus
.
getRawQuery
q
pure
$
(
\
(
len
,
docsC
)
->
(
len
,
docsC
.|
takeC
limit
.|
mapC
toDoc
))
<$>
eRes
pure
$
(
\
(
len
,
docsC
)
->
(
len
,
docsC
.|
takeC
limit
.|
mapC
toDoc
))
<$>
eRes
...
...
src/Gargantext/Core/Text/Terms/Multi/PosTagging.hs
View file @
9b4bef67
...
@@ -22,11 +22,13 @@ Source: https://en.wikipedia.org/wiki/Part-of-speech_tagging
...
@@ -22,11 +22,13 @@ Source: https://en.wikipedia.org/wiki/Part-of-speech_tagging
{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE TypeOperators #-}
module
Gargantext.Core.Text.Terms.Multi.PosTagging
module
Gargantext.Core.Text.Terms.Multi.PosTagging
where
where
import
Control.Exception
(
catch
,
throwIO
)
import
Data.Aeson
import
Data.Aeson
import
Data.ByteString.Lazy.Char8
qualified
as
BSL
import
Data.ByteString.Lazy.Internal
(
ByteString
)
import
Data.ByteString.Lazy.Internal
(
ByteString
)
import
Data.Map
qualified
as
Map
import
Data.Set
(
fromList
)
import
Data.Set
(
fromList
)
import
Data.Text
(
Text
,
splitOn
,
pack
,
toLower
)
import
Data.Text
(
Text
,
splitOn
,
pack
,
toLower
)
import
Gargantext.Core
(
Lang
(
..
))
import
Gargantext.Core
(
Lang
(
..
))
...
@@ -35,8 +37,6 @@ import Gargantext.Core.Types
...
@@ -35,8 +37,6 @@ import Gargantext.Core.Types
import
Gargantext.Prelude
import
Gargantext.Prelude
import
Network.HTTP.Simple
import
Network.HTTP.Simple
import
Network.URI
(
URI
(
..
))
import
Network.URI
(
URI
(
..
))
import
qualified
Data.ByteString.Lazy.Char8
as
BSL
import
qualified
Data.Map
as
Map
-- import qualified Gargantext.Utils.SpacyNLP as SpacyNLP
-- import qualified Gargantext.Utils.SpacyNLP as SpacyNLP
...
@@ -82,7 +82,15 @@ corenlp' uri lang txt = do
...
@@ -82,7 +82,15 @@ corenlp' uri lang txt = do
req
<-
parseRequest
$
req
<-
parseRequest
$
"POST "
<>
show
(
uri
{
uriQuery
=
"?properties="
<>
(
BSL
.
unpack
$
encode
$
toJSON
$
Map
.
fromList
properties
)
})
"POST "
<>
show
(
uri
{
uriQuery
=
"?properties="
<>
(
BSL
.
unpack
$
encode
$
toJSON
$
Map
.
fromList
properties
)
})
-- curl -XPOST 'http://localhost:9000/?properties=%7B%22annotators%22:%20%22tokenize,ssplit,pos,ner%22,%20%22outputFormat%22:%20%22json%22%7D' -d 'hello world, hello' | jq .
-- curl -XPOST 'http://localhost:9000/?properties=%7B%22annotators%22:%20%22tokenize,ssplit,pos,ner%22,%20%22outputFormat%22:%20%22json%22%7D' -d 'hello world, hello' | jq .
httpJSON
$
setRequestBodyLBS
(
cs
txt
)
req
-- printDebug "[corenlp] sending body" $ (cs txt :: ByteString)
catch
(
httpJSON
$
setRequestBodyLBS
(
cs
txt
)
req
)
$
\
e
->
case
e
of
JSONParseException
_req
res
_err
->
do
let
body
=
getResponseBody
res
printDebug
"[corenlp'] request text"
(
cs
txt
::
ByteString
)
printDebug
"[corenlp'] response body (error)"
body
throwIO
e
JSONConversionException
_req
_res
_err
->
throwIO
e
where
where
properties_
::
[(
Text
,
Text
)]
properties_
::
[(
Text
,
Text
)]
properties_
=
case
lang
of
properties_
=
case
lang
of
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment