The idea is to improve our async job workers, taking some ideas from the celery project: https://docs.celeryq.dev/en/stable/

• persist jobs in the database: i.e. create a jobs db table, with job type, job params, job state, possibly job output/error traceback as columns
• create a separate program to run jobs whatever they are in the DB. This way our API will only serve requests back and forth. The worker will run all the hard work. This allows us to, say, have a single API endpoint, and multiple worker services even on different machines.
• optionally we could then separate the workers to accept only specific tasks. This way we can create higher-priority queues. So slow jobs, like crawling, will be run by one type of worker, while faster ones, like password reminder etc will be run by other workers. This improves responsiveness to the users.
• fix the issue when someone runs a background job and then refreshes the page, and the frontend doesn't have it anymore. This isn't directly connected to the above rewrite, though I think it will make querying for jobs easier: instead of the frontend having to provide job id, we could ask the API to, say, list specific jobs for a node (e.g. does this corpus any jobs, like search queries running?)

With more and more people using gargantext, I think the above makes sense. I have worked with architectures like above and I think it's worth it.