Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
gargantext
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
humanities
gargantext
Commits
52a7400c
Commit
52a7400c
authored
Jul 29, 2016
by
c24b
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Resource docs added
parent
be9395c3
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
66 additions
and
2 deletions
+66
-2
contribution.md
docs/contribution.md
+1
-2
resource.md
docs/resource.md
+65
-0
No files found.
docs/contribution.md
View file @
52a7400c
...
...
@@ -28,6 +28,5 @@ see [install procedure](install.md)
2.
Create a new branch
<username>
-refactoring
3.
Run the gargantext-box
4.
Code
5.
Test
5.
Test
6.
Commit
docs/resource.md
0 → 100644
View file @
52a7400c
#resources
Adding a new source into Gargantext requires a previous declaration
of the source inside constants.py
```
python
RESOURCETYPES
=
[
{
"type"
:
9
,
#give a unique type int
"name"
:
'SCOAP [XML]'
,
#resource name as proposed into the add corpus FORM [generic format]
"parser"
:
"CernParser"
,
#name of the new parser class inside a CERN.py file (set to None if not implemented)
"format"
:
'MARC21'
,
#specific format
'file_formats'
:[
"zip"
,
"xml"
],
# accepted file format
"crawler"
:
"CernCrawler"
,
#name of the new crawler class inside a CERN.py file (set to None if no Crawler implemented)
'default_languages'
:
[
'en'
,
'fr'
],
#supported defaut languages of the source
},
...
]
```
## adding a new parser
Once you declared your new parser inside constants.py
add your new crawler file into /srv/gargantext/utils/parsers/
following this naming convention:
*
Filename must be in uppercase without the Crawler mention.
eg. MailParser => MAIL.py
*
Inside this file the Parser must be called following the exact typo declared as parser in constants.py
*
Your new crawler shall inherit from baseclasse Parser and provide a parse(filebuffer) method
```
python
#!/usr/bin/python3 env
#filename:/srv/gargantext/util/parser/MAIL.py:
from
._Parser
import
Parser
class
MailParser
(
Parser
):
def
parse
(
self
,
file
):
...
```
## adding a new crawler
Once you declared your new parser inside constants.py
add your new crawler file into /srv/gargantext/utils/parsers/
following this naming convention:
*
Filename must be in uppercase without the Crawler mention.
eg. MailCrawler => MAIL.py
*
Inside this file the Crawler must be called following the exact typo declared as crawler in constants.py
*
Your new crawler shall inherit from baseclasse Crawler and provide three method:
*
scan_results => ids
*
sample = > yes/no
*
fetch
```
python
#!/usr/bin/python3 env
#filename:/srv/gargantext/util/crawler/MAIL.py:
from
._Crawler
import
Crawler
class
MailCrawler
(
Crawler
):
def
scan_results
(
self
,
query
):
...
self
.
ids
=
set
()
def
sample
(
self
,
results_nb
):
...
def
fetch
(
self
,
ids
):
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment