1. 12 May, 2025 1 commit
    • Alfredo Di Napoli's avatar
      Port DB operations to transactional API · 53512f89
      Alfredo Di Napoli authored
      This gigantic commit ports the existing DB operations in GGTX to use the
      transactional API, meaning that we can now compose DB operations and
      they will all run in the same Postgres transaction using the same
      connection, which will eliminate those class of bugs where concurrent DB
      access might result in an inconsistent state.
      
      On top of that, we simplify some parts of the API, for which a summary
      is given below:
      
      1. The `NodeStoryEnv` management has been greatly simplified; in the new
         API we don't need an external connection pool to be passed and we
         don't have to pass IO actions, we can just pass DB operations,
         therefore we can greatly simplify the API to just pass mostly pure
         values;
      
      2. Due to the fact that our `DBTx` monad can't do arbitrary IO (which is
         a good thing) we cannot fire Central Exchange notifications
         immediately. Rather that happens now is that we collect the
         `CEMessage` to be sent and we fire them in the relevant concrete
         monad after we finished with the DB transaction. This means that in
         principle there would be a small delay between the DB operation
         taking place and the notification firing but in practice the latency
         should be negligible and bear in mind this is typically what we want:
         if we have a long DB Tx that triggers an error in the middle we don't
         want to be sending out CE messages prematurely if the overall
         operation didn't succeed!
      
      3. There are still a few places in the codebase where we couldn't make
         things fully compositional with regards to the DBTx API, because we
         had Servant handlers which had DB operations mixed with other IO
         effectful computations (or other things like the notification from
         the `MonadJobStatus`). For now we are splitting these functions by
         manually running the partial DB operations, and while this is not
         ideal it can be fixed in subsequent merge requests.
      
      4. The `WorkerEnv` doesn't use `IOException` as its `MonadError`
         anymore, as for consistency we can just use `BackendInternalError` by
         adding a `InternalWorkerError` data constructor accepting the
         `IOException` triggered by the Worker monad.
      
      More testing is needed, with particular attention to performance
      (regression) but this should hopefully offer a decent baseline.
      53512f89
  2. 29 Jan, 2025 1 commit
    • Przemyslaw Kaminski's avatar
      Squashed commit of the following: · 942e663f
      Przemyslaw Kaminski authored
      commit b4755ad5
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Wed Jan 29 11:41:32 2025 +0100
      
          Code review, part II
      
          This commit splits the /export (renaming it to just remote) and tuck it
          under the /node hierarchy. The import also lives tucked in the /node.
      
      commit 483bd3e5
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Tue Jan 28 09:34:14 2025 +0100
      
          Code review feedback
      
          * Rename `exampleS` into `exampleSchema`;
          * Revert commit about the public keys & co;
      
      commit 18d207f0
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Tue Jan 28 09:50:56 2025 +0100
      
          Revert "Add _env_remote_transfer_keys field"
      
          This reverts commit 3ea32b50.
      
      commit 9cc5159a
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 20 10:00:53 2025 +0100
      
          Support transfering of notes
      
      commit 1fe60d75
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Fri Jan 17 08:37:53 2025 +0100
      
          Refactor exporting and transfering of nodes
      
      commit b39c1805
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Thu Jan 16 09:54:38 2025 +0100
      
          Preliminary work to transfer notes
      
      commit b2f7a9a8
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 13 15:25:34 2025 +0100
      
          Chunks the insertion of remote docs
      
      commit 0d4e0554
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 13 14:43:16 2025 +0100
      
          Move terms updating to separate job as well
      
      commit c62480c7
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 13 12:30:19 2025 +0100
      
          Proper support for importing documents
      
      commit 6019587c
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 15:44:13 2025 +0100
      
          Initial support for importing ngrams
      
      commit 842b3d36
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 15:07:21 2025 +0100
      
          Support exporting docs and ngrams (but not importing them yet)
      
      commit 98708c2e
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 12:32:17 2025 +0100
      
          Support exporting of tree hierarchies (with proviso)
      
          Exporting a corpus works, as it also exports its children, but for
          example the docs and terms nodes do not have any associated content.
          This is because those are stored in separate DB tables, and we need
          to find a way to export those as well.
      
      commit c248eaf1
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 11:50:14 2025 +0100
      
          Support trees of export nodes (to be tested)
      
      commit dd2049aa
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 11:23:30 2025 +0100
      
          Add getNodes function to Database.Query.Table.Node
      
      commit c429cbb1
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 10:39:37 2025 +0100
      
          Restrict export of nodes to only a few types
      
      commit c648699e
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Jan 6 09:27:28 2025 +0100
      
          Update deps again (after rebase)
      
      commit 7337820e
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 16:07:49 2024 +0100
      
          Basic Remote API testing
      
      commit 1eb59c52
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 15:31:47 2024 +0100
      
          Barebone (non-streaming) storage of nodes
      
      commit be5e9faf
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 14:20:16 2024 +0100
      
          Send serialised nodes instead of dummy strings
      
      commit aff15b60
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 12:07:49 2024 +0100
      
          Remove redundant test imports
      
      commit 6d776767
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 12:01:55 2024 +0100
      
          Bolt-on ownership check for /remote/export
      
      commit 58d9fcb0
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 16 11:08:11 2024 +0100
      
          Proper error handling for remote import and export handlers
      
      commit 23a06d28
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 9 15:37:29 2024 +0100
      
          Update project deps
      
      commit d5096e40
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 2 17:05:04 2024 +0100
      
          Make a start on the remote (streaming) endpoints
      
          It typechecks but it exchange only a very simple string and
          it prints it.
      
      commit 3ea32b50
      Author: Alfredo Di Napoli <alfredo.dinapoli@gmail.com>
      Date:   Mon Dec 2 11:26:11 2024 +0100
      
          Add _env_remote_transfer_keys field
      
          This adds a new randomly-generated pair of (PublicKey, PrivateKey) to be
          later used to send messages between instances.
      
          It also:
      
          * Returns a remote transfer pub key inside an AuthResponse
          * Adds pubKey roundtrip test
      942e663f