Clojurians Log v2
datahike
And one more question... maybe I do not understand something, maybe tiered storage is still too young for production usage. Anyway, I did not succeed with many configurations. For example current attempt: JDBC "h2" frontend and GCS backend. JDBC store returns Store, but GCS store returns async channel... sure it fails. Is somewhere an option to fix it? Or there are only limited combinations of stores? I tried some things but without forks I don't see any option
I have not worked with GCS yet, @pat knows this better. There might be an async mismatch somewhere. The tiered store itself is not that complicated, it probably reveals and underlying inconsistency. Do you have a snippet to reproduce or contextualize?
Just such :store config fails on first connection attempt:
{:backend :tiered
:id #prop :michelada.test.database.id
:frontend-config {:backend :jdbc
:dbtype "h2"
:user "sasha"
:password ""
:id #prop :michelada.test.database.id}
:backend-config {:backend :gcs
:location "europe-central2"
:bucket "my-bucket"
:store-path "datahike"
:project-id "michelada"
:id #prop :michelada.test.database.id}
:write-policy :write-behind
:read-policy :frontend-first}I am going to apply this patch and bump the konserve version with the datahike release. Lmk if you see any problems: https://github.com/replikativ/konserve/pull/137/changes
The only issue here is that error handling is gone, i.e. when the backend store fails you won't see it. But we can readd it.
Not sure this is your error though, I think this might be fine. I will just push the datahike release for now. Might be a GCS bug you experience.
Hm... your fix does not look to help with that problem. It fails actually here:
PAssocSerializers
(-assoc-serializers [this serializers]
(clojure.core/assoc this
:frontend-store (-assoc-serializers (:frontend-store this) serializers)
:backend-store (-assoc-serializers (:backend-store this) serializers)))(:frontend store) actually satisfies PAssocSerializers (it is DefaultStore or something like that), but (:backend-store this) in this moment is clojure.core.async.impl.channels.ManyToManyChannel and it crashes.
I do not think you have to apply that changes if everything about that read/write operations is workingMaybe really GCS problem...
It already happens on store connection, right?
Yes, d/connect fails
Right
It helps me a lot if you can give me a piece of code or sketch to reproduce first. I think it is super cool that you explore this, I need the branching features, too, and am almost ready to push this programming model. But it takes quite some time to reproduce here, trying to do so now with Claude.
I am also happy to discuss what you need in more detail. Yggdrasil is an attempt to unify this, because doing this only for Datahike is not nearly as powerful.
I think I found the bug in konserve-gcs.
Oh great! Btw I have some notes, what is not convenient for me, but I have to summarize them with proper examples etc. But I do not have enough time for it. I'll try soon
> https://github.com/replikativ/konserve-gcs/pull/1 Look's like it could help. I didn't tried exactly here... but now I see it makes sense
Should be released in a moment as 0.1.12
Lmk when you run into more problems.
Huh... okay I will, sure