Consider reducing or removing throttling in the dispatcher as it's confusing for the user (#501) · Issues · gargantext / haskell-gargantext

Consider reducing or removing throttling in the dispatcher as it's confusing for the user

While working on #498, I couldn't understand why I wasn't getting any notification on the frontend when my ngrams import formed a loop: I could see that the server emitted the relevant log error message pretty much immediately, but nothing on the frontend.

Then, by accident, I waiting more than one minute and the notification eventually appeared.

This is due to the following code inside dispatcherListener:

    Async.withAsync (throttle (_nc_dispatcher_throttle_ms * 1000) throttleTChan
                     (sendDataMessageThrottled nc log_cfg)) $ \_ -> do

Under normal circumstances this would be around 8 minutes, because the default value in gargantext-settings.toml is 500 (ms) and here we multiply by 1000. I think this is way too much, and completely confusing for users.

@cgenie Questions:

Why do we need throttling at all? I understand it could be a way to implement some form of backpressure, but surely we can do that with other heuristics, like looking at the connected clients or the number of messages waiting to be dispatched?
Can we reduce this number? I think we should remove the 1000 multiplier. I feel that 500ms is always pretty generous -- can we go lower than that?

Then, by accident, I waiting more than one minute and the notification _eventually_ appeared.

This is due to the following code inside `dispatcherListener`:

```
    Async.withAsync (throttle (_nc_dispatcher_throttle_ms * 1000) throttleTChan
                     (sendDataMessageThrottled nc log_cfg)) $ \_ -> do

```

Under normal circumstances this would be around 8 minutes, because the default value in `gargantext-settings.toml` is 500 (ms) and here we multiply by 1000. I think this is _way_ too much, and completely confusing for users.

@cgenie Questions:

1. Why do we need throttling at all? I understand it could be a way to implement some form of backpressure, but surely we can do that with other heuristics, like looking at the connected clients or the number of messages waiting to be dispatched?
2. Can we reduce this number? I think we should remove the `1000` multiplier. I feel that 500ms is always pretty generous -- can we go lower than that?