@petterik: Very cool. I was looking at the btset code wondering how hard it would be to compare them. Would you mind sharing the code to calculate the changes?

@misha @petterik So, I took a stab at generating the :db/add :db/retract diff using @petterik’s code. Not sure I covered all of the cases, but it seems promising. Let me know if you see anything wrong?

@seantempesta awesome! I like where this is going

@seantempesta why would you generate tx? what is the exact use case?

I’m planning to forward it to the datomic database, and then stream the changes back to the client.

Why don't you let datascript generate those for you?

I suppose you could do optimistic updates too. Apply the changes locally, call the remote update, and on confirm retract the locally applied changes.

Creating a new db from the return of datoms-diff is equiv to datomic/since ?

I still don't get, why or when would you need to diff 2 dbs "manually"

As opposed to “listening”?

yes

@misha I use something like this in my om.next reads

cache the db used during last read

update the value with the changes

I am generating diffs, but for hash-map cache, to send it to datascript in another thread. so I don't have a DS to do it for me. but in your case it looks like you have all the control of what's happening with conn, so why don't just listen for tx-data, collect it in some pocket, and once per whatever send that to datomic?

@misha: So how would it work if the user makes quite a few transactions to the database? What about temp id resolution across transactions?

if you use ds/tempid - it gives different one each time (assuming you are within same process, not across devices)

you can accumulate tempids resolution map w/o overwriting any, while, again, you are in the same process

well shit. I should should try this.

if, on the other hand, you are using -1, -2 - then ¯\(ツ)/¯

Well, as you said I can control that.

what I am working on now, is generating tx-s for a hash-maps diffs, to maintain fast-reads cache in a main thread (with optimistic updates and stuff). will share soon

it works great for simple basic stuff, like: get-by-id, get-children, create/update/remove etc. will need to put some more thought into subscribable queries, but if datascript would run in a separate worker/thread – it'll have all the time in the world to just rerun queries

Yeah, it’d be great to have it on a separate thread. I was doing that for a while with my re-frame-workers library. But the underlying react-native-workers library wasn’t actively maintained so I gave up on it.

in a RN context, I just wrapped it in native module with a so-so api: transit in, transit out

yeah, you’ve mentioned that before. I’ll consider doing that too if I hit performance problems with the datascript db. Now I’m doing server side queries for anything complex and the local datascript db is just doing basic key lookups.

Hmm. I think so? But without the flexibility of arbitrarily specifying the transaction/time. Damn, and here I thought I made something useful.

I am questioning usefulness of that cache 5 times a day

since you'd pretty much give up nested pull patterns and queries (would need to either rerun those, or pay communication overhead)

Unless you've already written functions that walk nested pull patterns returning adds and retracts using (d/since (d/history db) <tx>) for datomic 😇

Making the diffing lazy and queryable with [e a v tx added?] would enable to me use what I'm already using on the backend

I'm very exited about this

@petterik when do you do nested pp walking?

for server/client sync, I just take all client related "datoms since last synced tx" and send them to client.

(I'm constantly paranoid about missing some essential use case, which will make me start over)

@misha We do pp walking for (almost) every server read. Each page requires their own set of data, and we return only what's changed for that dataset. If you're familiar with om.next reads, we basically keep a tx for every client for each read. What would "datoms since last synced tx" mean for http://etsy.com ? Read everything anyone does on the site?

(I'm also paranoid about missing some essential use case btw)

(I don't work at etsy)

digesting

ok, I see, in om.next you are not pulling everything on entity, and walking pp makes sense in that context, true.

on the other hand, as soon, as you deliver that diff to client, the need to cherry-picking what to send to server is gone, no?

I mean, on server you need to cherry-pick what to send to client. On client, you need to send only new txs, all of them, right?

On the client we do "mutations", send actions to execute on the server, along with reads to acquire the changes

"all of them" part is pretty trivial with ds/listen!.

We don't send datascript/datomic txs from client to server

ah, ok.

do you walk pps on client?

Not yet, but I hope to do it soon.

or do you have DS on server too?

Datomic on the server, datascript client

why/when would you need to walk pp on client? to issue should-component-update=true?

The UI is made up of reads, these reads are executed on each render (without taking to account om optimizations). Right now I'm doing (d/pull-many ...) for each call to the read. I want to be able to incrementally update the value from the last time the read was called

Using something like what we talked about today, but it has to be lazy to be efficient (I think)

I think I understand: you want to pull everything once, and then just pull updated attributes as patches

Yes!

And I've been doing that part successfully for a while now

on the server

I'm interested to see if this can be applied on the client as well

* call datascript pull from om view?

are your pp deep? is combination of equality-check-you-wrote with patch-pull – faster than just pulling entity again?

They are deep sometimes. However, we also need retracts which pull doesn't give us

do you keep txs arrived from server on a client?

"txs arrived"?

you call server, it send you txs to be applied to DS, right?

Yes

if you keep them, did you consider to scan those for a "did something change" info?

I did not

It's certainly something to consider though

keep, obviously, until nobody would need those anymore

right

that might require each view to keep track of the latest tx-id seen

Hi, I use clojurescript javascript interop, stuck in something with object. I use echarts library and (doto chart (.setOption (clj->js SOMEMAP))) then the error occurred in the setOption function's (OPTION_INNER_KEY in option). Here is the setOption function:

setOption: function (option, optionPreprocessorFuncs, onlyGraphic) {
	            zrUtil.assert(
	                !(OPTION_INNER_KEY in option),
	                'please use chart.getOption()'
	            );

	            this._optionManager.setOption(option, optionPreprocessorFuncs);

	            this.resetOption(null, onlyGraphic);
	        },

Uncaught TypeError: Cannot use 'in' operator to search for '_ec_inner' in null
    at ExtendedClass.setOption ()
    at ECharts.echartsProto.setOption ()
    at 
    at Object.ReactErrorUtils.invokeGuardedCallback ()
    at executeDispatch ()
    at Object.executeDispatchesInOrder ()
    at executeDispatchesAndRelease ()
    at executeDispatchesAndReleaseTopLevel ()
    at Array.forEach (native)
    at forEachAccumulated ()
setOption @ echarts.js:2304
echartsProto.setOption @ echarts.js:400
(anonymous) @ views.cljs?rel=1495377145023:147
ReactErrorUtils.invokeGuardedCallback @ react-dom.inc.js:8972
executeDispatch @ react-dom.inc.js:2979
executeDispatchesInOrder @ react-dom.inc.js:3002
executeDispatchesAndRelease @ react-dom.inc.js:2431
executeDispatchesAndReleaseTopLevel @ react-dom.inc.js:2442
forEachAccumulated @ react-dom.inc.js:15401
processEventQueue @ react-dom.inc.js:2618
runEventQueueInBatch @ react-dom.inc.js:8996
handleTopLevel @ react-dom.inc.js:9007
handleTopLevelImpl @ react-dom.inc.js:9084
perform @ react-dom.inc.js:14702
batchedUpdates @ react-dom.inc.js:8760
batchedUpdates @ react-dom.inc.js:12815
dispatchEvent @ react-dom.inc.js:9159

@cmal (doto chart (.setOption (clj->js SOMEMAP))) passes 2 args to .setOption here

What should I write if I want to do chart.setOption(obj)?

I am curious why the (clj->js SOMEMAP) is null..

Ahh.. I think I should ask this question in re-frame channel. Thanks!

That probably because of the dispatch-sync I used in on-click event handler

your call does not add up with fn signature: setOption: function (option, optionPreprocessorFuncs, onlyGraphic)