- 12 May, 2025 1 commit
-
-
Alfredo Di Napoli authored
This gigantic commit ports the existing DB operations in GGTX to use the transactional API, meaning that we can now compose DB operations and they will all run in the same Postgres transaction using the same connection, which will eliminate those class of bugs where concurrent DB access might result in an inconsistent state. On top of that, we simplify some parts of the API, for which a summary is given below: 1. The `NodeStoryEnv` management has been greatly simplified; in the new API we don't need an external connection pool to be passed and we don't have to pass IO actions, we can just pass DB operations, therefore we can greatly simplify the API to just pass mostly pure values; 2. Due to the fact that our `DBTx` monad can't do arbitrary IO (which is a good thing) we cannot fire Central Exchange notifications immediately. Rather that happens now is that we collect the `CEMessage` to be sent and we fire them in the relevant concrete monad after we finished with the DB transaction. This means that in principle there would be a small delay between the DB operation taking place and the notification firing but in practice the latency should be negligible and bear in mind this is typically what we want: if we have a long DB Tx that triggers an error in the middle we don't want to be sending out CE messages prematurely if the overall operation didn't succeed! 3. There are still a few places in the codebase where we couldn't make things fully compositional with regards to the DBTx API, because we had Servant handlers which had DB operations mixed with other IO effectful computations (or other things like the notification from the `MonadJobStatus`). For now we are splitting these functions by manually running the partial DB operations, and while this is not ideal it can be fixed in subsequent merge requests. 4. The `WorkerEnv` doesn't use `IOException` as its `MonadError` anymore, as for consistency we can just use `BackendInternalError` by adding a `InternalWorkerError` data constructor accepting the `IOException` triggered by the Worker monad. More testing is needed, with particular attention to performance (regression) but this should hopefully offer a decent baseline.
-
- 29 Jan, 2025 1 commit
-
-
Przemyslaw Kaminski authored
commit b4755ad5 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Wed Jan 29 11:41:32 2025 +0100 Code review, part II This commit splits the /export (renaming it to just remote) and tuck it under the /node hierarchy. The import also lives tucked in the /node. commit 483bd3e5 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Tue Jan 28 09:34:14 2025 +0100 Code review feedback * Rename `exampleS` into `exampleSchema`; * Revert commit about the public keys & co; commit 18d207f0 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Tue Jan 28 09:50:56 2025 +0100 Revert "Add _env_remote_transfer_keys field" This reverts commit 3ea32b50. commit 9cc5159a Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 20 10:00:53 2025 +0100 Support transfering of notes commit 1fe60d75 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Fri Jan 17 08:37:53 2025 +0100 Refactor exporting and transfering of nodes commit b39c1805 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Thu Jan 16 09:54:38 2025 +0100 Preliminary work to transfer notes commit b2f7a9a8 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 13 15:25:34 2025 +0100 Chunks the insertion of remote docs commit 0d4e0554 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 13 14:43:16 2025 +0100 Move terms updating to separate job as well commit c62480c7 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 13 12:30:19 2025 +0100 Proper support for importing documents commit 6019587c Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 15:44:13 2025 +0100 Initial support for importing ngrams commit 842b3d36 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 15:07:21 2025 +0100 Support exporting docs and ngrams (but not importing them yet) commit 98708c2e Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 12:32:17 2025 +0100 Support exporting of tree hierarchies (with proviso) Exporting a corpus works, as it also exports its children, but for example the docs and terms nodes do not have any associated content. This is because those are stored in separate DB tables, and we need to find a way to export those as well. commit c248eaf1 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 11:50:14 2025 +0100 Support trees of export nodes (to be tested) commit dd2049aa Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 11:23:30 2025 +0100 Add getNodes function to Database.Query.Table.Node commit c429cbb1 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 10:39:37 2025 +0100 Restrict export of nodes to only a few types commit c648699e Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Jan 6 09:27:28 2025 +0100 Update deps again (after rebase) commit 7337820e Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 16:07:49 2024 +0100 Basic Remote API testing commit 1eb59c52 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 15:31:47 2024 +0100 Barebone (non-streaming) storage of nodes commit be5e9faf Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 14:20:16 2024 +0100 Send serialised nodes instead of dummy strings commit aff15b60 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 12:07:49 2024 +0100 Remove redundant test imports commit 6d776767 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 12:01:55 2024 +0100 Bolt-on ownership check for /remote/export commit 58d9fcb0 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 16 11:08:11 2024 +0100 Proper error handling for remote import and export handlers commit 23a06d28 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 9 15:37:29 2024 +0100 Update project deps commit d5096e40 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 2 17:05:04 2024 +0100 Make a start on the remote (streaming) endpoints It typechecks but it exchange only a very simple string and it prints it. commit 3ea32b50 Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com> Date: Mon Dec 2 11:26:11 2024 +0100 Add _env_remote_transfer_keys field This adds a new randomly-generated pair of (PublicKey, PrivateKey) to be later used to send messages between instances. It also: * Returns a remote transfer pub key inside an AuthResponse * Adds pubKey roundtrip test
-