Fork me on GitHub
#datahike
<
2024-02-15
>
octahedrion10:02:09

is the dynamodb storage backend for Datahike reasonably up to date ?

timo10:02:51

which one?

octahedrion11:02:23

there's more than one ? any

timo11:02:24

I just need to know which one you are referring to so I can take a look

octahedrion11:02:44

ok it looks like this is the only one it has https://github.com/csm/konserve-ddb

timo11:02:56

that one hasn't seen updates for 5 years so no. Datahike has seen a lot of development in recent years. Everything older than a year won't work with latest datahike version I assume

whilo20:02:24

If there is interest, it would not be hard to create one.

whilo20:02:39

The main limitation this one still has is that it does not use an async client as well when the API would prefer it, but just wraps sync calls in go blocks. That is totally fine though to get started.

whilo20:02:01

@U0CKDHF4L Would you be interested in helping with that?

whilo20:02:45

Maybe the S3 backend would also work, it depends mostly on your latency requirements I think.

whilo20:02:05

Would your backend run in a AWS data center?

octahedrion07:02:07

why S3 for dynamodb ? edit: oh you mean as an alternative would https://github.com/taoensso/faraday be of use ?

whilo08:02:36

that would do, although direct java API calls should also be fine. whatever is simple to pick up and reliable

whilo08:02:45

(and has no significant overhead)

whilo08:02:12

a backend does not require a lot of code, so we can also iterate on it if needed

octahedrion08:02:13

what does it need to store and retrieve ? is it just [e a v t op] ?

whilo08:02:48

just blobs

whilo08:02:05

key column which will be uuids as strings and a blob it will assign to it

whilo08:02:30

so there will be a blob column, but normally dynamo adds columns automatically as far as i understand

whilo08:02:55

these will hold tree fragments of the indices, which then contain the datoms

octahedrion08:02:51

ohhhh yessss I remember

whilo08:02:18

this is how it looks like for SQL if that helps https://github.com/replikativ/konserve-jdbc

whilo09:02:27

@U0CKDHF4L can you contextualize how you would like to use dynamodb?

octahedrion09:02:28

I don't have a preference really I just wondered. If there's an existing AWS backend that does the job then that's fine

octahedrion14:02:39

(unless there's some advantage to dynamodb)

whilo18:02:56

you pay more, but it will be faster

whilo18:02:27

the first thing to improve though is to add proper async support to konserve-s3, it will reduce latency on S3

whilo18:02:42

at least when you transact bigger batches of datoms

octahedrion08:02:20

also dynamodb comes by default with aws services doesn't it ?

whilo18:02:38

what do you mean with aws services?

octahedrion05:02:08

that you can access a dynamodb from anywhere by default without having to install it I thought

octahedrion05:02:29

(I'm not that familiar with aws)

whilo05:02:44

yes, you can

whilo05:02:52

it is offered as a service