Fork me on GitHub
#datascript
<
2016-10-24
>
luxbock14:10:38

I'd like to use Datascript for an offline-only web-app, where my plan is to generate a Datascript DB from Clojure first and then serializing it so that it's available in the CLJS part of the app

luxbock14:10:51

however I realized that the amount of datoms I would have is quite large, probably to the tune of 100k

luxbock14:10:29

so I was wondering how many datoms I should expect Datascript to handle without running into performance issues

luxbock14:10:51

the data is basically a list of records that I want to then display based on some filters which are constructed via the UI

misha14:10:11

it depends on: - user's device - complexity of the queries (sometimes it is much more efficient to use indexes directly, but it might not be enough)

dm314:10:15

it's hard to say - you're better off generating the data manually (e.g. using test.check) and looking at it

dm314:10:23

+ what @misha said

misha15:10:21

also your domain might allow you to split data set into several databases, which might give you some performance boost too

luxbock15:10:00

Thanks, the user device is desktop Chrome

luxbock15:10:37

I might just end up using a vector of maps, but I will benchmark Datascript first

Niki17:10:59

> DataScript is in different category, so expect different tradeoffs: query speed depends on the size of result set, you need to sort clasuses to have smaller joins, accessing entity properties is not free given its id, etc. As a benefit, you gain ability to query dataset for different projections, forward and reverse reference lookups, joins between different sets, etc. And direct index lookup (datascript.core/datoms) is still fast and comparable to lookup in a map (at least comparable, think binary lookup vs hashtable lookup, logarithm vs constant). Queries do much more than that.

Niki17:10:22

Database itself can handle 100k datoms no problem

Niki17:10:36

you’ll hit perf issues if you select many datoms at once

Niki17:10:00

including queries where intermediate sets contains lots of datoms

Niki17:10:04

direct index lookups are still fast no matter what (they are lazy walks over existing data structures, they don’t create any new arrays/data structures)

Niki17:10:33

one thing to watch out for is initial load of DB into memory

Niki18:10:06

currently, DataScript can’t work with parts of the index stored on a disk/local storage/.... It must all be in memory

Niki18:10:24

so every time your app starts you’ll have to read whole DB from your persistent storage

Niki18:10:44

measure that performance first and then decide if it’s suitable for you

Niki18:10:01

better use datascript-transit (it’s a lib) for serialization, avoid EDN or manual transact! of datoms. Perf difference might be 10×

luxbock21:10:36

@tonsky thanks those are some very good points