Fork me on GitHub
#untangled
<
2016-05-04
>
currentoor21:05:16

Is there a way to customize the merge function untangled passes to om.next?

tony.kay21:05:46

not really...what is missing?

tony.kay21:05:40

you're not supposed to need to fool around at the db format layer...so I'd like to know if we're missing necessary generality.

currentoor21:05:58

I've got large blobs of data (for charts and tables) it's coming in as nested maps. The merge function takes a long time to merge these blobs into app-state, probably because it's recursive and looking for idents and what not.

currentoor21:05:48

But these blobs are not something I query inside. A shallow merge would work just fine.

currentoor21:05:10

So it would be great if there was a way to mark something to be shallow merged.

currentoor21:05:51

Changing to same data to nested vectors instead of nested maps resulted in 2-3x performance gain in the merge function.

adambrosio21:05:46

@currentoor: what’d you use to get that performance data?

tony.kay21:05:50

ok, so this is a harder problem than you might recognize...I agree that your solution of being able to mark things as shallow would help; but you cannot pass metadata over the wire.

tony.kay21:05:25

The query could be used to help...e.g. shallow merge anything that is at a raw (non-join) prop

tony.kay21:05:47

Also, some of those bits (like the prewalk) could be changed to use a compiled specter transform to make it faster (for tempid replacement)

tony.kay21:05:33

Anything you might do to optimize this would be useful to any general user of Untangled...so I'd say it'd be better to optimize what we have

adambrosio21:05:49

+1 for compiled specter transform

tony.kay21:05:33

Also, why is swap! so slow?

tony.kay21:05:01

and set-global-loading, which is also a swap!

tony.kay21:05:27

I guess your state is large enough that the update with structural sharing is costly

tony.kay21:05:41

cause only about 40% of the time is merge

tony.kay21:05:56

actually 28%

tony.kay21:05:27

meaning that your performance gain only gets you about an overall 20% improvement.

tony.kay21:05:05

So, another question to ask: Why not place an animated GIF loading symbol on the screen so that when the work happens, the user feels better. Sometimes large things just take time.

tony.kay21:05:23

or all of the above

currentoor21:05:27

Shallow merge wherever queries end + better walker seem like good ideas. I can also try to send less data.

currentoor21:05:44

GIF is something we'll probably do regardless.

tony.kay21:05:32

Less data is a nice option if you can incrementally load things

tony.kay21:05:08

The more advanced merge is a really hard thing to write....ask @adambros about the mark/sweep stuff in pre/post merge.

therabidbanana22:05:11

@tony.kay: worth pointing out that this is not actually that large of a data set, we'd definitely see larger in the real world. Is there a way we could maybe store it outside the global app state where it's less expensive? (These are essentially blobs of immutable data for rendering tables - the reports never change)

tony.kay22:05:12

You could use the same basic algorithm to hang metadata on the data structure, then use a simple deep-merge that switches to shallow when the items are marked...not really an online algorithm, though, so that might not speed it up as much as just doing the merge based on the query...but then you're needing a parser

tony.kay22:05:18

@therabidbanana: hm. So, at the moment the only way to do that would be to hack into the parser...and you'd lose app state tracking for that data.

tony.kay22:05:40

I think optimizing the central merge is the better first-draft

tony.kay22:05:53

rather than making the whole model more complex

tony.kay22:05:22

We've not done anything optimized in merge, and know it needs it...it is a planned thing.

tony.kay22:05:37

if you're bumping up against it, then there is a place you can contribute simple_smile

tony.kay22:05:15

If it turns out what we want is a way to point to external data, then we can add that as a feature as well.

therabidbanana22:05:42

On the animated gif point - this is stop the world work by the browser, dropping framerate to seconds per frame - so not likely to help

tony.kay22:05:14

Yeah, I understand...

tony.kay22:05:33

those two optimizations may make it super-fast, though

therabidbanana22:05:03

Definitely worth exploring then, from the sounds of it.

tony.kay22:05:18

for that matter, if you pre-process the response to know what paths are shallow, you might even be able to use that to make tempid replacement even faster

tony.kay22:05:37

though I think the compiled specter thing is likely to be quite good

therabidbanana22:05:52

We were going to try the nested vectors approach to limit how much the merge functions has to inspect, but seems like that still relies on the merge function being kind of fast

tony.kay22:05:37

As I said: we know it needs optimizing. It is a first draft that is known correct, but not fast

tony.kay22:05:14

so rather than hacking, let's work the central problem (optimization), then consider alternatives when/if that hits a wall

therabidbanana22:05:14

Sounds reasonable - specter sounds like an interesting approach, I've only briefly looked at it before.

tony.kay22:05:29

it's pretty cool

currentoor22:05:25

I could hack around this next week. @tony.kay I'll raise an issue describing the issue and proposed solutions.

tony.kay22:05:33

works for me, thanks!

currentoor22:05:52

here's the issue

currentoor22:05:21

@adambros: maybe you have some insights to add (from you pre/post merge work)

adambrosio22:05:48

i did mark and sweep missing, but i dont think that’s what your profiler said was taking a long time

adambrosio22:05:14

if you can find what functions are slow I can try to help

kenbier23:05:54

does untangled.datomic have support for altering attributes in a migration?

adambrosio23:05:26

can you clarify what you mean by altering attributes?

kenbier23:05:49

altering schema attributes

kenbier23:05:09

i.e. changing the cardinality of an existing attr

adambrosio23:05:53

im not 100% sure but that sounds like you apply another migration that changes the cardinality

adambrosio23:05:26

ie a transaction

tony.kay23:05:40

@adambros: on mark/sweep: It uses the query, which has some symmetry with the merge optimization

tony.kay23:05:10

e.g. instead of "marking" missing stuff, it could "mark shallow merge points"

currentoor23:05:13

@kenbier: I don't think it's possible to change the cardinality without a re-import

tony.kay23:05:39

@kenbier: Right...if you can do it in datomic, you can do it, since a migration is nothing more than transactional data (since schema is just transactional data) that gets executed and recorded as "done" by our library...really the tracking is all we added. The schema functions come from Datomic-schema

tony.kay23:05:44

(Yuppiechef)

kenbier23:05:51

@tony.kay: thanks, makes sense. ill check out there dsl

currentoor23:05:53

@tony.kay: I just forked the untangled-client, any suggestions for how to develop? Is there a dev mode to execute it? Test runner perhaps?

ethangracer23:05:25

@currentoor: there is only a test build for untangled-client. if you want to see how it is performing, I’d make a cookbook recipe and make changes to untangled-client in checkouts

ethangracer23:05:47

that way you can see how an app is using it while in active development. i’ve done it several times and it’s great