Fork me on GitHub
#datascript
<
2021-09-02
>
oly09:09:26

Hi, I have an application which is using datascript which is having performance issues, the querys all work and are fast how ever the initial transact of data from an api is very slow and locks the webpage anyone have advise on improvements or optimisations of this step ?

ribelo10:09:56

Collect as many changes as possible in one transaction

ribelo10:09:44

And best of all, stop using datascript if you need performance

ribelo10:09:09

Every now and then the same problem appears here and there and the only solution is to use other things

oly11:09:46

I do actually do everything in one transact its a single api that pulls all data dumps it in allows you to query locally

oly11:09:45

I have wondered about doing the data load in a web worker as a way to stop the blocking

oly11:09:35

obviously I could use something else but that would require a lot of changes, just hoped there might be options or things that can be done that could improve that initial load

ribelo11:09:59

I don't know what you specifically use and what you need, but you still have compatible options

oly11:09:56

that is an option switching to one of the others, might end up doing that need to investigate how easy that might be

Linus Ericsson19:09:51

Would it be faster to download the ready made database from API? datascript->transit. If the data is mostly static, you could probably cache the similar parts on the server side and just add the individual data to that.

Linus Ericsson19:09:37

Aaand, make sure to use transit as serialization format if you have a lot of data that the cljs-client should read, since it is quite slow to parse edn on the client side, in comparision

Huahai13:09:03

Maybe don’t do initial transact, do bulk datom load instead?

Huahai13:09:47

transact is expensive, it does lots of reads and checks. For initial data load, you don’t need any of that

oly15:09:16

hum, @UQY3M3F6D so basically create the db server side with datascript then ship it over transit straight into the empty datascript atom ?

oly15:09:20

@U0A74MRCJ I like the idea how does one go about creating and adding the datoms directly any docs / info any where on how you do that ? is there an alternative to transact to insert a datom ?

Huahai15:09:15

the core.cljc namespace is the public API of datascript, take a look

Huahai15:09:30

at that code

oly15:09:20

okay cheers I will do 🙂

raspasov08:09:53

conn-from-datoms has worked for me over the past few months. Yes, DataScript is not the fastest but if you’re careful with not transacting/querying too often or too much data at once, it can be OK.

oly11:09:41

okay thanks for the tips looking into a few avenues

metasoarous19:09:43

In my experience, using a transit encoding of the datascript db is a significant improvement, but not by orders of magnitude, so will only get you so far. It still has to create the indexes and stuff, so there's "work" to be done.

metasoarous19:09:35

I'd recommend looking at https://datahike.io/. It's forked from datascript, but adds a number of features missing from datascript but found in datomic (tuples, history, etc). Most notably though, it's implemented durability on top of IndexedDB, so you can save/query data locally and not have to recreate an in memory database structure every time you boot up the application.

metasoarous19:09:38

Of course, querying from durable storage on the JS runtime implies an async api, and they've implemented this using core.async. So things will be a little different, and it's worth noting that this functionality may technically still be in beta, but given all that datahike has to offer (especially in terms of mirrored server/client functionality), I think it's a pretty attractive solution.

metasoarous19:09:37

There's a #datahike channel here, but folks seem to be more active on the discord, which you can find a link to on the website: https://datahike.io