Fork me on GitHub
#datascript
<
2021-04-02
>
Tom H.13:04:26

Athens Research uses posh but it looks like they’re having performance issues https://github.com/athensresearch/athens/pull/784

🏛️ 3
lilactown16:04:19

it's unfortunate that datascript does get unbearably slow at large amounts of data

Saikyun08:04:18

ah, I see. that's too bad

Saikyun11:04:50

thanks @tomisme for the issue btw. I tried a bit of posh and datascript yesterday, and it seems like posh has trouble with certain queries. it seems like the athens work would solve that (by just using datascript queries).

Tom H.12:04:10

no worries 🙂

Tom H.12:04:06

I’m on the hunt for a performant path from datascript/datahike -> reagent

lilactown15:04:13

the bottleneck in these situations always seems to be datascript/datahike

lilactown15:04:17

the power and flexibility that they give you is amazing, but when you have a 16ms budget on the UI thread, you don't always want to spend the bulk of it getting data out of the client side db

💯 3
Saikyun07:04:23

@tomisme I'm guessing one simply has to try with the dataset one is needing. that's what I'm gonna do at least 🙂

👍 3
Saikyun07:04:42

@lilactown sounds reasonable. in my case, it seems that using datascript will lead to me doing a bit fewer recalculations. at least to my understanding, the athens solution above doesn't cause rerenders e.g. when filtering away results of transactions.

Saikyun07:04:04

also, the datasets I'm using aren't that big. we'll see how it goes

Richie14:04:12

(cljs.pprint/pprint
 (let [conn (ds/create-conn)]
   (ds/transact! conn [{:db/id -1
                        :name "Fred"
                        :friend {:db/id -2
                                 :name "Rich"
                                 }}])
   (ds/datoms @conn :eavt)))
Shouldn’t this create two datoms? I expect that it would resolve the temp id -2 to a new id as well as -1. Instead I get:
(#datascript/Datom [1 :friend {:db/id -2, :name "Rich"} 536870913 true]
 #datascript/Datom [1 :name "Fred" 536870913 true])

Richie15:04:25

Here’s the return from ds/transact!

{:db-before {},
 :db-after {1 :friend, 1 :name},
 :tx-data
 [#datascript/Datom [1 :name "Fred" 536870913 true]
  #datascript/Datom [1 :friend {:db/id -2, :name "Rich"} 536870913 true]],
 :tempids {-1 1, :db/current-tx 536870913}, 
 :tx-meta nil}

Richie15:04:49

From the docs on transact! :

; ref attributes can be specified as nested map, that will create nested entity as well
      (transact! conn [{:db/id  -1
                        :name   \"Oleg\"
                        :friend {:db/id -2
                                 :name \"Sergey\"}])

huxley15:04:21

:friend must be declared as :db.type/ref

Richie15:04:09

Ok, thanks.

Richie15:04:51

Thanks for setting me on the right track. I didn’t know I needed a schema. I haven’t figured out how to pass it in though. I’ve tried passing it on creaton:

(ds/create-conn {:schema
                             [{:db/ident :friend
                                :db/valueType :db.type/ref
                               :db/cardinality :db.cardinality/one}]})
And I’ve tried passing it to transact!
(ds/transact! conn [ {:db/ident :friend
                         :db/valueType :db.type/ref
                         :db/cardinality :db.cardinality/one}])

Richie16:04:23

Oh.

(cljs.pprint/pprint
 (let [conn (ds/create-conn {:schema
                             [{:db/ident :friend
                               :db/valueType :db.type/ref
                               :db/cardinality :db.cardinality/one}]}
                            )]

   (ds/schema @conn)))
prints
{:schema
 [{:db/ident :friend,
   :db/valueType :db.type/ref, 
   :db/cardinality :db.cardinality/one}]}
… I still have not resolved my original problem though.

Richie16:04:29

Oh.

(ds/create-conn {:friend
                             {:db/ident :friend
                              :db/valueType :db.type/ref
                              :db/cardinality :db.cardinality/one}})
Ok…

lilactown16:04:19

it's unfortunate that datascript does get unbearably slow at large amounts of data

lilactown16:04:48

it's one of the reasons why I wouldn't recommend it as a catch-all for UI app state

raspasov17:04:19

@lilactown It’s actually kinda slow even with very little data on React Native; I am not sure exactly why. Maybe I don’t fully understand how to use it. If you try to use the indices directly, it’s faster; but any use of (d/q ...) is often quite slow.

raspasov17:04:55

But otherwise the functionality is very nice, so I’m sticking with… but have to resort to some hacks to make the UI snappy.

raspasov17:04:49

One idea I’m about to experiment with is treating it as a remote database; basically assume it’s a remote database, and show loaders/spinners everywhere;

raspasov17:04:29

That obvi. won’t make it faster, but maybe it will “feel” faster 🙂

raspasov17:04:27

The “hacks” involve having an extra app-state the I render out of, rather than querying directly from Datascript; querying datascript multiple times per second is a big NO-GO

indy17:04:13

Are you using it with re-frame? Re-frame will run all the queries for every transact, which is what the posh library tried to solve. I was planning to use datascript too but if there is such a perf problem, I'll have to think of something else. Has anyone tried running reframe + datascript in a web worker?

raspasov17:04:51

@kslvsunil no, I use React directly; no extra libs

raspasov17:04:28

If I paint myself in the corner, at least I have a chance of understanding how I got there! haha 🙂

indy17:04:35

Oh dang, then it's going to much much slower with re-frame.

raspasov17:04:50

No idea, never used re-frame/reagent.

indy17:04:52

I wonder how roam research uses datascript

raspasov17:04:56

Only om.next back in the day.

raspasov17:04:32

And while I haven’t used them, it’s hard to say whether they are faster or slower; but it will most likely be more difficult for me to analyze the whole thing.

raspasov17:04:07

Web worker is something I’ve thought about;

raspasov17:04:38

It will make it “faster” in the sense that it won’t block your JS UI thread, but the results will still often take 100+ ms to arrive;

raspasov17:04:23

So you’re definitely in the realm of what I wrote just above: you’d have to treat DataScript as a fully remote database and never even pretend that it’s local (it will definitely be all async)

😔 3
raspasov17:04:36

Basically, anytime things take more than 16ms in JavaScript, you run into UI smoothness trouble (1000 / 60fps = 16 ms)

indy17:04:54

Yup agree completely to "It will make it 'faster' in the sense that it won’t block your JS UI thread, but the results will still often take 100+ ms to arrive;". There is also lilactown's https://github.com/lilactown/autonormal which I was recommended in the asami channel. Guess I'll start spiking out with autonormal. All I want is a well structured normalized app state, which I could probably tame by having some discipline. But the problem is every time I want a new state slice, I'll have to think about the structure of it, it's nesting level, yada yada, which is time consuming and burns a lot of cognitive energy unnecessarily

rickmoynihan10:04:22

FYI there is also this: https://github.com/wotbrew/idx/ Not sure how they compare. cc @lilactown

👍 3
lilactown17:04:40

My 15 min review: on the surface, idx implements custom data structures which implicitly handle indexing and have a bunch of additional helper functions for doing a bunch of additional operations on those data structures, like sorting, matching, and others. autonormal uses clojure.core datastructures (maps/vectors/sets) to store your data, so the normalization and indexing is transparent to the developer, and does not give you any additional tools for sorting or matching this. autonormal also has an EQL engine for querying maps (including following references created by autonormal). idx does not provide a way of declaratively, recursively querying arbitrary EDN

👍 3
lilactown18:04:40

autonormal won't give you the full power of datalog, but on the other hand because it's not tuple-based it should be much faster

lilactown18:04:20

I benchmarked it early on against datascript and mapgraph and it was orders of magnitude faster than DS. I've added some features after that that may have slowed it down a bit. I should redo those benchmarks and publish them

lilactown18:04:05

React Native could probably get away with treating datascript as a "remote" database since JS runs on a separate thread from the actual UI, but the browser cannot; if transacting takes 250ms then that's 250ms that the UI thread isn't doing anything else (like responding to clicks or key presses)

lilactown18:04:39

oh I see you mentioned webworker @raspasov. yes. then it's measuring the cost of serialization

lilactown18:04:26

I think that the idea of shoving all app state into a single store is naive anyway. DS in a webworker for domain state might be good enough and open up other novel things like sharing that state across browser windows

3
Saikyun11:04:50

thanks @tomisme for the issue btw. I tried a bit of posh and datascript yesterday, and it seems like posh has trouble with certain queries. it seems like the athens work would solve that (by just using datascript queries).