Turns out the issue was also at parent getting rerenders.
So the trick is that I added an empty div with ref for parent node and,
next to it, the Graph component pointing to that ref.
Graph component calls react's usePortal with that parent ref and a small
'createElement' div to hold sigma canvases.