Clojurians
# om

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

tony.kay 00:34:57

With regard to remotes and parsing: I'm working on a nested set of components where remotes are involved. As I walk the query in read, I'd like to call the parser recursively. This works fine with locals, since I can return {:value (parser ...)}; however, the return value of the parser on a target is a query, so I cannot exactly return { :remote (parser ... :remote) } because that is supposed to be an AST.

I can use expr->ast, but then I end up with a vector of vectors (because the outer parser wants to wrap the elements of the parsed selector in a vector to make it a query). I can dance around the data structures and make it work, but all this seems kinda messy, and I'm wondering if the story is still evolving, and I should just not be trying this particular thing yet. Or perhaps I've missed a simple trick.

dnolen 00:39:47

@tony.kay: ok so the problem here is that ast parsing doesn’t compose simply like parse does

tony.kay 00:39:55

correct

jdubie 00:39:56

i’ve also ran into this @tony.kay. i ended up using gather-sends. which i’m not sure is public

dnolen 00:40:15

@tony.kay: it seems like there are two parts to this

dnolen 00:40:25

1) thread :target via env

dnolen 00:40:31

so you don’t have to do this at every level

dnolen 00:40:43

2) defer ast->expr till the very end

tony.kay 00:41:16

er. expr->ast?

dnolen 00:41:29

no ast->expr

dnolen 00:41:47

I don’t think you want to work on the raw expr

tony.kay 00:42:10

OH...so don't RUN the parser in remote mode at all?

dnolen 00:42:23

it’s too gnarly for stuff like parameterized joins

dnolen 00:42:37

not suggesting that

dnolen 00:42:46

just trying to see if 1) & 2) give you want you want

dnolen 00:43:03

but I think I’m missing something here

tony.kay 00:43:09

yeah, the AST is expected

tony.kay 00:43:11

in the output

tony.kay 00:43:15

(of read)

dnolen 00:43:29

so why can't you do stuff like this?

dnolen 00:44:41

(let [ast (exp->ast selector)] {:remote (update-in ast [:sel] #(parse env % :remote))})

tony.kay 00:46:48

that might work...I see the general idea. Yeah, I was trying to post-convert the returned query instead of that.

dnolen 00:47:32

right I think this should work and it supports recursion pretty much as well as the value oriented stuff

tony.kay 00:47:45

I'll give it a shot and let you know if there is a problem

tony.kay 00:48:16

@dnolen: thanks

dnolen 00:48:48

@tony.kay: no problem, happy to know if how it works isn’t satisfactory for some reason

dnolen 00:48:55

but I think it should be sufficiently flexible

dnolen 00:49:04

for people who want to use recursive calls to parse and for those people who do not

tony.kay 00:52:03

I'm not sure how you'd do an app of any size without recursive parse...if you have any nesting where the top-level components are all local, and then you have nested components that need to hit a server, well, you have to compose the remote stuff to root...how would you do it without recursion (well, other than walking the sub-query manually...but isn't that a recursive parse you've just written manually???)

dnolen 00:54:55

@tony.kay: use Datomic

dnolen 00:54:59

or DataScript

dnolen 00:55:01

you bottom out fast

tony.kay 00:55:35

I am, and you do, but there are still a couple of level to go through, and recursion seems the natural choice to get to the part that is "ready to send"

tony.kay 00:57:02

but yeah, I don't have any suggestions for making it simpler...I just wasn't seeing the algorithm, and was making sure you didn't have further dev you were going to do in this area that would make is easier somehow.

tony.kay 00:57:19

(about to write a significant amount of code :simple_smile: )

dnolen 01:05:06

another thing to consider is just how much nesting is going on

tony.kay 01:05:06

I think it wasn't very evident to me (and evidently others like @jdubie) because I was thinking of the AST as an opaque thing...an implementation detail to be passed around, but not messed with, so I was trying to convert it after the fact using a known function

dnolen 01:05:14

while I think recursive queries are cool

dnolen 01:05:30

I’m not so convinced that queries should be ever be more than 4 or 5 levels deep

tony.kay 01:05:52

I agree with both statements

dnolen 01:06:26

and I’m also not yet convinced that helper functions don’t help with the cases where you have to get through the first couple of levels

dnolen 01:06:40

@tony.kay: still I’m too heads down to know what people are trying

dnolen 01:07:22

so I could very well be wrong about this and people should show simple examples based on real world things that reveal things require lots of effort

dnolen 01:07:42

so far from what I’ve seen a lot of effort in Om usually means 5 lines of code

dnolen 01:07:47

but I dunno, maybe that’s a lot!

tony.kay 01:08:07

in clojure: it depends on the 5 lines :simple_smile:

dnolen 01:08:17

@tony.kay: exactly!

noonian 01:10:00

I ran into the need for recursive queries immediately. Basically anytime a component wants to render a subcomponent that wants to query for root keys (doesn’t know its a sub-component).

tony.kay 01:10:38

so far, I'm seeing a recursive use of simple read/parser calls as the simplest way to walk down a nested query that contains a mix of local/remote stuff...but things like specter also seem like possible contenders...nothing prevents me from transforming the nested query at the "top"

noonian 01:10:59

but I’ve since been trying to think of the queries more as requesting what data you need and using extra parse functions to do the plumbing and not care as much what the real underlying data structure looks like in my components

dnolen 01:11:19

@tony.kay: yes don’t let let my examples lead you astray

tony.kay 01:11:32

I still lean towards recursion, because most people get recursive parsers pretty readily.

dnolen 01:11:46

recursive parse should work, but it may very well be that other methods are better suited for different kinds of apps

noonian 01:12:01

I ran into issues with normalization when using recursion in the parser

dnolen 01:12:33

@noonian: there shouldn’t be any issues with that far as I know

tony.kay 01:12:44

right, I've not seen normalization issues

dnolen 01:13:33

@noonian the root values problem is a different thing from queries. This is why I fixed up shared.

dnolen 01:13:47

if you need to push user info, internationalization etc. around don’t do that with queries

noonian 01:14:31

how does shared fix the root values problem when the values are dynamic?

dnolen 01:15:50

@noonian: so that’s a “it just depends"

dnolen 01:16:06

for global user state etc. shared is fine, just re-render the root

dnolen 01:17:32

more concrete examples of what you’re trying to do would be useful here

noonian 01:18:35

For instance, if I have value :msg in my state that shows notifications to the user from different parts of the app, so many components want access to it to render it.

noonian 01:19:11

not a great example since it would probably be the top-level components responsibility to render the msg but thats the type of thing I’ve ran into

dnolen 01:19:35

right easier to understand if you give a real example

dnolen 01:19:49

I’m having a hard time imagining replicating global dynamic information into each component

noonian 01:23:17

My usecase for recursive queries was that I was trying to make a “layout” component that renders a different sub-component based on the app state (using query mutations to change its query dynamically). This is a much more complex example but in this case I desired that all of the sub-components be unaware that they were sub-components and that the “layout” component not require special knowledge about the subcomponents. So a different motivation for wanting recursion but I think the implementation issue is the same as with the :msg example.

noonian 01:24:16

I’ve kind of tabled that experiment for now though and am just using a giant union of the queries at the top level. I saw some discussion in here about maybe adding delayed queries or something in the future and I think that is trying to solve the same problem as I was.

noonian 01:25:42

anyway I’m just mentioning this because it sounded like you wanted to hear more use cases

dnolen 01:26:01

@noonian that’s an interesting sounding example but hard for me to understand the issues without knowing more details about you actually tried and how you went about it

dnolen 01:26:21

at this stage it still difficult to know if people are getting stuck on how to reformulate the problem - or a limitation of what’s there

noonian 01:29:56

I also found myself using params in place of state for the component since I had to mutate the query at the same time as change the state but also have access to the (initial) state in my static query impl.

dnolen 01:30:43

yeah this is a bit too abstract for me.

dnolen 01:30:52

much easier to look at some small code example that shows what you tried

chris-andrews 01:32:37

Is it allowed in om.next for a component to return multiple idents?

chris-andrews 01:33:14

I think it probably doesn’t make sense, but I have been thinking about how to have multiple pieces of normalized data

dnolen 01:33:22

@chris-andrews: not allowed

dnolen 01:33:56

you have unions if you need something to represent multiple types of things

chris-andrews 01:36:32

Ok, follow-up, is it a bad idea to use two subcomponents that each have an ident, and then have their parent component utilize their data? I think I can actually work around this, but some things like WebGL can be a little tricky to break down into components, but the ident is really useful for data normalization

chris-andrews 01:37:25

So essentially, each child component’s query gets used, but they just return nil in their render function, and the parent component actually uses their data because it has access to the WebGL context

dnolen 01:39:43

@chris-andrews: this also doesn’t make sense to me

dnolen 01:39:49

why do you need these child components at all then?

chris-andrews 01:41:00

Simply because their query enables data normalization via Ident. The whole scene needs certain data, but a parent component can’t have multiple Idents, so that seemed like an option to me for working around that

chris-andrews 01:42:59

I think it’s probably not a well thought out question, and it’s not a big deal. Mainly was just wondering if the mapping definitely needed to be 1 component = 1 Ident

dnolen 01:43:09

just not following the logic here and I don’t see how WebGL is involved

dnolen 01:43:32

instantiated components can only have 1 ident

dnolen 01:43:38

however

dnolen 01:43:45

N instantiated components can have the same ident

chris-andrews 01:45:42

Ok, thanks for clearing that up a little. The only reason WebGL is involved is because in the past it has been a little tricky for me to render children to a scene, because you need the parent context, but that’s really not very difficult to work around

dnolen 01:49:27

landed tempid support, just needs a bit more testing

dnolen 01:50:18

pretty excited about it, you should be able to build up graphs of objects that don’t have remote permanence yet (and their corresponding UI elements) and batch a transaction

dnolen 01:50:48

if there are tempids in the remote transaction result then the temporary ids and UI bits will automatically be migrated to the new information

dnolen 01:51:39

removes a giant complexity hairball from the kinds of things you see people do in more sophisticated client apps

dnolen 01:51:55

where users get to manipulate even moderately complex temporary relationships

dnolen 01:52:35

the only thing left to do is cache eviction and I have a pretty good idea how to keep it dead simple for most applications (nothing for users to do)

dnolen 01:52:51

this means we’re getting really, really close to beta

dnolen 01:53:12

if there’s anything people would like to see changed, enhanced or tweaked now is a good time to speak up.

dnolen 01:53:49

Also simple complete examples of conceptual difficulties as we’ve already discussed today are welcome - they will help determine if there are any real remaining gaps

jannis 01:56:57

Cool. I already have an app that creates new temporary objects and sends them to the remote to create for real. I'll try to test the tempid stuff tomorrow.

dnolen 01:58:10

@jannis cool please do, gonna give it a spin in my om-next-demo

jannis 02:01:19

The only enhancement I currently wish for is what we briefly discussed last night (something like txbind - possibly in a completely different form - either baked into transact! or as a separate function to resolve transaction reads to component subqueries before they are passed to transact!). Everything else just works so far. I'm really liking it. :simple_smile:

noonian 02:03:04

@dnolen: here’s the example I mentioned of a recursive parser and using params as state. I definitely don’t know if this is a good direction to go in but I hope it at least demonstrates the idea a little bit https://www.refheap.com/111552

jannis 02:04:37

@dnolen: Perfect, sorry for not filing it myself :simple_smile:

dnolen 02:08:33

@noonian: yeah there several ways to solve that I don’t know about using params so heavily for this

dnolen 02:09:15

I’ll play around with your example

noonian 02:10:05

thanks

dnolen 02:10:30

but in all likelihood it’ll just be a little bit simpler and leverage subquery

dnolen 02:10:42

while colocated queries are neat

dnolen 02:11:00

I think it’s important to keep data concerns and view concerns somewhat distinct for cases like this

noonian 02:12:07

Yeah I’d love to see an example of how to use subquery. It wasn’t obvious to me from the docs.

noonian 02:15:39

abp: thanks. I hadn’t seen it but that is exactly what I am doing now.

noonian 02:16:45

which is to stick everything into the root query statically, but then you have the motivation for delayed queries

abp 02:17:29

oh yea

abp 02:18:06

well I'm just lurking posting occasionally when I think I see something :simple_smile:

noonian 02:18:35

yep, thats spot on with what I was trying to do. I was using a router also but didn’t include that in the refheap snippet

tony.kay 02:30:35

So, after playing with it some more, it occurs to me that I'm going to end up with a remote query that is structured from root, even for the remote stuff. This seems necessary for it to know where the result of the remote query's result is going, E.g. if I have [{:panel [:status {:list-of-server-things [:a :b :c]}]}] I'm going to end up seeing [{:panel [{:list-of-server-things [:a :b :c]}]}], right?

tony.kay 02:32:13

which means I have to write send to "strip out" the UI bits at the top if I want a reusable thing on the server (cause the mobile might not have the nested panel)

tony.kay 02:33:17

e.g. I'm going to send the query fragment [{:list-of-server-things [:a :b :c]}] over the wire, then plug the result back into the UI-based structure before passing it to the callback

tony.kay 02:35:46

If I'm understanding that correctly, then I'm wondering where we ended up on the path optimization story. I don't see read-ident on alpha19. If we do the path optimization, then can we make it so components with an Ident always use the alternate read, and therefore won't need this extra processing?

dnolen 02:56:41

@tony.kay: need to rewind a bit here, way too many things going on

dnolen 02:57:34

I do not understand what you mean by the second sentence

dnolen 02:57:45

“This seems necessary for it to know where the result of the …"

dnolen 02:57:49

can you explain what you mean?

tony.kay 03:00:11

sorry, yes

tony.kay 03:01:01

I think I've confused myself. First question:
1. Does a parse with a target need to return a result that structures to root?

tony.kay 03:02:23

E.g. in your remote sync tutorial, the dynamic query includes all of the components all the way to the root

tony.kay 03:02:47

I am assuming this is required, else it does not know how to associate the query/result with the UI components

tony.kay 03:04:45

@tony.kay uploaded a file: Untitled and commented: from tutorial

tony.kay 03:07:36

@dnolen: so remotes have the same requirements on query structure as local

tony.kay 03:15:59

I was walking through the steps to deal with a remote when the query ends up having UI-specific bits "above" the real server query. Seems like they are:
1. Write the read function to leave only those bits that have to do with the remote.
2. If (1) has to be structured to root, the resulting query will have UI-specific elements wrapped around the server-interesting bits. Strip that bit off before sending it over the wire. parse -> [{:ui-bit [{:server-bit [:a]}]}] -> xfer over wire [{:server-bit [:a]}] -> server
3. On response, plug the server value back into a structure that includes those UI-specific bits of structure and give that to the callback

dnolen 03:21:31

@tony.kay: there’s no point in removing the UI specific bits

dnolen 03:21:37

this how web servers are written

dnolen 03:21:44

you may have some app specific top level routes

dnolen 03:21:47

this isn’t a problem

dnolen 03:22:32

and those routes are not in anyway conflated w/ remotes

tony.kay 03:22:42

@dnolen: What? If I have a mobile UI that has one kind of top-level structure, and a desktop app that has a different one, and neither of them have persisted data at the root, then I don't want the UI specific query going over the wire

dnolen 03:23:27

@tony.kay: this is just missing the point

tony.kay 03:23:34

I guess so :disappointed:

dnolen 03:23:36

you already need keys that many client won’t need anyway

dnolen 03:23:46

your keys are already app specific

dnolen 03:24:05

and key without reification in the DB

dnolen 03:24:18

is ...

dnolen 03:24:20

app specific

tony.kay 03:24:41

I know that, but if I have some container that uses only local client state, but composes in something that needs something from the server, then the query will have that client-specific route in the overall query, won't it?

tony.kay 03:24:59

client-specific...e.g. in RAM of client, not on server

tony.kay 03:27:02

when I say "UI specific", I mean that I've put some data in app local state that is for controlling things like a modal pop up...what if that modal needs to compose in some query for a list of surveys, for example. The "list of surveys" is a query rooted at the user of the application.

tony.kay 03:27:31

the modal is nested in the UI at some deep level unrelated to the "list of surveys"

dnolen 03:28:08

you need to be more clear

dnolen 03:28:13

“app specific” is meaningless here

dnolen 03:28:17

you just mean local only keys

tony.kay 03:28:26

sorry, yes, local only PATHS

tony.kay 03:28:40

from root...at some point I end up "bottoming out", but I have all these local-only paths in the way

tony.kay 03:29:07

so I strip those off before talking to the server

dnolen 03:29:28

@tony.kay: right so I do not think that’s a good idea

tony.kay 03:29:50

so my server has to understand that I'm showing a modal?

dnolen 03:30:45

write some routing logic that knows to just to get the subparts

dnolen 03:30:59

i.e. it doesn’t care about the keys actually it just knows the real stuff is underneath

dnolen 03:31:21

screwing around with the structure of the thing breaks everything

dnolen 03:31:28

like merging etc.

dnolen 03:32:05

the analogy here is redirects

tony.kay 03:34:22

so, I'm not seeing how to write this routing logic you speak of. I assume it is client-side. But, Om runs the parser from root on :remote. The result of that is a query with the UI structure, right?

dnolen 03:35:38

there a ton of ways to do this

dnolen 03:35:57

parse doesn’t have to be a multimethod

dnolen 03:36:09

it can be function and then defer to a multi method

dnolen 03:36:28

you’ll get some top level keys you don’t care about but you know the next level you do

dnolen 03:36:37

just write some generic recursive code

dnolen 03:37:31

basically what you’re suggesting is exactly something we do not want to do

dnolen 03:37:39

which is be in the business of reshaping

dnolen 03:38:55

Falcor does not do this

dnolen 03:38:58

Relay does not do this

dnolen 03:39:01

we’re not going to do it

tony.kay 03:39:12

I got all the concepts. We're crossing wires again :wink:

dnolen 03:39:39

then re-frame the question

dnolen 03:39:46

how would you do this in Falcor or Relay?

tony.kay 03:40:02

I've been trying :simple_smile:

In Relay, you can root a component

tony.kay 03:40:31

In falcor, you hook a model to a data source, and end up with something you can query directly

tony.kay 03:41:02

Relay: plop a profile (rooted at user 32) anywhere in my UI

dnolen 03:42:10

right but this isn’t quite the same

dnolen 03:42:24

you have to root on some ID

dnolen 03:42:50

and Relay there’s no such thing as local stuff

tony.kay 03:43:23

I see that. And with Om, we end up "rooting" at the Root of the UI, right?

dnolen 03:43:29

in Falcor you have to be OK with an async model

tony.kay 03:44:12

my misunderstanding could be the answer to this question: Does the result of Om's call to Parse on a remote have to follow the UI structure?

dnolen 03:44:46

@tony.kay: I’m just trying to get you to understand how Relay & Falcor actually work

dnolen 03:44:50

what they mean

dnolen 03:44:56

and how Om works at the moment.

dnolen 03:45:15

and then maybe we can have an actual idea :simple_smile:

dnolen 03:45:29

so the way Om works gets you something

dnolen 03:45:38

purely synchronous reasoning model

dnolen 03:45:50

seamless merging model

dnolen 03:45:58

including local state

dnolen 03:46:14

what maybe missing

dnolen 03:46:23

is whether remoting should happen on subfragments

dnolen 03:46:29

instead of always at the root

tony.kay 03:46:38

that is my point :simple_smile:

tony.kay 03:46:47

ok, we're on the same page now

dnolen 03:47:28

@tony.kay: well now what we know what we’re talking about at all :simple_smile:

tony.kay 03:49:48

So, I was saying that my send function gets this "remote from root" query, and since I don't want to write a bunch of different routes on the server for each possible UI bit, I'd want my send to do the "remove local path" part...but you're saying you'd rather see the server just kind of elide the bits as if going through redirect logic

dnolen 03:50:21

right I have a tendency to want to think through on how we could solve without changing anything at all

dnolen 03:50:37

and then think about alternatives

tony.kay 03:51:15

So, that was why I mentioned path optimization earlier. With the read-ident idea, we could use that as a way to handle many of the cases...assuming the query for such a component always tried read-ident first.

dnolen 03:51:20

the alternative would require parsing entry point to be quite different on the server

dnolen 03:51:27

path optimization is related but not enough

dnolen 03:51:55

because it doesn’t tell you how put the original thing back together for the client

dnolen 03:52:15

so the client may have to do that

tony.kay 03:52:17

because you get a response that is "disconnected"

dnolen 03:52:24

as you suggested

dnolen 03:53:06

so yeah you could pass roots to the server and that makes parsing entry point uniform

dnolen 03:53:21

[:list/of-stuff {[:foo 0] …} …]

tony.kay 03:53:34

but the query fragments are different all over the place for other components with the same ident

dnolen 03:53:35

but the client would need to track where those need to go

dnolen 03:53:43

[:foo 0] is trivial case

dnolen 03:53:51

but not these random keys

tony.kay 03:55:24

Yeah, I see the problem...could we pass some marker (e.g. component path) into send so it could return that with the response?

tony.kay 03:56:07

or even close over it with the callback that send is given

dnolen 04:02:48

@tony.kay: ok so I agree with you … it ’s a problem

dnolen 04:03:00

but will need to think about it for a bit

dnolen 04:04:24

@tony.kay: I think we can contain it so that servers and client code is unaffected

dnolen 04:04:34

and specifying roots can be an optional thing

dnolen 04:04:55

so whatever people are doing right now will still work

dnolen 04:05:09

and people who want fine grained control of remoting can get the behavior they want

dnolen 04:36:59

@tony.kay: ok current idea, you can specify a query is a root during parse

dnolen 04:37:27

in send we can as you suggest construct the actual roots in to a single query to the server

dnolen 04:37:39

and upon response reconstruct the original shape

dnolen 04:38:02

specifying a query is a root would just be marking the query ast

tony.kay 05:07:02

@dnolen: I like the idea. So, the assumption (a fair one) is that no nested component will specify “I’m a root query” if a parent had already…we’re assuming the “bottoming out” happens once. So, sibling-like things could all root themselves.

dnolen 05:07:29

@tony.kay: I don’t want components to be involved in this stuff

dnolen 05:07:41

only parsing makes this decision

tony.kay 05:07:46

sorry, I just said component because the query is colocated there

dnolen 05:08:04

@tony.kay: no such assumption

dnolen 05:08:14

saying a query is a root will be a fully dynamic thing

dnolen 05:08:26

it happens during parse so you will have full control

tony.kay 05:09:16

I was trying to think about the case where a query AST says “I’m root”, but some sub-portion of that query wants to “change root”. Can the marker be nested?

dnolen 05:09:35

ah no I think parent query takes precedence

dnolen 05:09:41

since I don’t see the value of repeating the query

dnolen 05:09:52

but it won’t be disallowed or anything

dnolen 05:10:07

but it means you can write this logic without worrying about things going bad

tony.kay 05:10:21

I just am not familiar enough with the AST to know if this marker could be recursive

tony.kay 05:10:56

Yeah, I like the general idea…was just exploring composition

dnolen 05:11:20

yeah I can’t think of a case where a subroot would be useful to actually act on

dnolen 05:11:40

that is ignoring roots under a root seems like the right idea to me at the moment

dnolen 05:12:25

@tony.kay: but your idea is a good one I admit wholeheartedly. It gets us a nice Falcor feature w/o giving up on the synchronous model.

tony.kay 05:13:25

thanks, I very much appreciate the amount of sync stuff you’ve managed to pull off. Only one place in the entire app needs to ever see an async call.

dnolen 05:13:39

it also makes the local / remote split more reasonable

tony.kay 05:14:33

yeah, I’m starting to feel better about that part as well. I really didn’t like the idea of having separate declarative queries, but I was also disliking the local complected with the remote query. This is making it more palatable.

dnolen 05:15:07

@tony.kay: also this is a totally optional knob

jannis 05:15:10

Hmm. I have refs in my app state (normalized and all) but if a query for e.g. [{:person/list [:db/id :person/name]}] comes back from the remote and I have local props set on them (e.g. {:person/list {1 {:db/id 1 :person/name "John" :app/expanded true}}}), the local props are dropped. Is this a bug or am I supposed to overwrite one of the merge functions for this to work?

What comes back has the shape {:person/list {1 {:db/id 1 :person/name "John"}}}.

dnolen 05:15:17

and one that’s easy to turn on without breaking your whole app

dnolen 05:16:09

@jannis the default merge could probably be tweaked a bit

dnolen 05:16:37

@jannis look at the reconciler to see how it works

dnolen 05:16:43

default-merge-tree I think.

dnolen 05:16:47

it’s pretty naive

jannis 05:17:23

Yep, I tried overloading merge-tree but it didn't work. I'll revisit.

jannis 05:19:22

Something that resembles (merge-with #(merge-with merge %1 %2) {:person/list ...local...} {:person/list ...remote...}) should, at least in this particular case, do the job of preserving local props.

dnolen 05:36:59

so I’m pretty ok with how recursive remote query construction works, it’s really more or less like parsing but on the AST instead of the actual query

dnolen 05:37:19

but the real thing to look at here is the marking some part of the query as a root

jannis 05:40:44

Overloading merge-tree worked.

dnolen 05:44:42

@jannis cool!

jannis 05:45:55

@jannis uploaded a file: Untitled and commented: It's pretty ugly and special-cased. :wink:

jannis 05:47:04

Would a complete recursive walk, merging maps at all levels, make sense or is that undesirable?

dnolen 13:09:42

@jannis not as the default but I could see that making sense for some apps

jannis 13:20:01

One problem with it is that properties that disappear on the server side (e.g. :person/subscribed? or something) would never be deleted if all maps returned by sends are merged. A different idea could be to mark parts of the ast with hints like "preserve [:app/expanded ...] here" and have merges honour those hints.

dnolen 13:21:25

@jannis yes this is why we can’t supply a sensible merge

jannis 13:22:14

Then again... a prop disappearing would easily break the query contract set by components.

dnolen 13:22:32

that too

dnolen 13:23:01

to me the best thing to do is … gasp … return nil for deleted properties

jannis 13:27:13

Mhm. Doesn't break contracts and is different from empty values ("", [], {}...). I'm fine with that.

thomasdeutsch 15:26:14

Have to ask, to make sure. If i have a component that needs to get data from two root branches, my options are: 1. data from one branch goes to the root and is a :shared value 2. data from one branch goes to the root and is passed down via om/computed 3. a custom parser

dnolen 15:28:04

only 1 and 3 seem appropriate to me

thomasdeutsch 15:39:08

i would say that 1 would be good for something like shared resources (i18n, ...) or other data that would normally be placed at the root.

dnolen 16:04:01
  1. I would use mostly for event handlers
dnolen 16:04:39
  1. for the cases where I really need to get something dynamic from elsewhere
dnolen 16:05:14

along with the regular props

dnolen 16:19:18

@thomasdeutsch: also if I was using DataScript I would might just add a relationship in this case

dnolen 16:19:28

or if I was using the default Om DB, a link

dnolen 16:19:42

then I don’t have to bother with writing a custom parse

dnolen 16:20:13

nothing is stopping you from adding custom relationships at the client

thomasdeutsch 16:25:23

yes, right. But if i have a dynamic list? then i am back at those two options i think.

dnolen 16:31:12

@thomasdeutsch: I don’t follow

dnolen 16:31:54

my previous statements don’t preclude dynamic lists as far as I can tell

thomasdeutsch 16:33:03

then, what do you mean by adding a "relationship"

thomasdeutsch 16:37:50

i thought that you would suggest me to add refs to my entities ( but that would exclude dynamic data )

dnolen 16:39:37

@thomasdeutsch: why can’t refs point to dynamic data?

dnolen 16:39:50

DataScript certainly has no such limitation far as I remember

thomasdeutsch 16:47:48

right. but i nave no entity that contains a list of items - i have to find the items.

dnolen 16:49:46

@thomasdeutsch: ah mean you need to run a query for this thing

dnolen 16:50:09

@thomasdeutsch: wasn’t clear to me that was a requirement

dnolen 16:50:46

DataScript lets you use regular values and thought that might work in this case

dnolen 16:50:53

but not sure why you can’t use pull here

dnolen 16:51:01

unless again you just need the generic power of query

denik 16:56:04

when running a local state transition with om/update-state! in om next componentDidUpdate is not being triggered, though (.forceUpdate component) is called internally. Why is that?

denik 16:58:31

when (.forceUpdate component) is called manually it does get triggered

dnolen 17:00:11

@denik: might be something weird about the base React class no idea

dnolen 17:01:12

@denik: would probably require some experimentation and some cross verification with some simple plain React 0.14 code

denik 17:04:37

@dnolen turns out it’s not being called b/c the component is found in the reconciler: (if-let [r (get-reconciler component)] (do (p/queue! r [component]) (schedule-render! r)) (.forceUpdate component))

denik 17:04:50

first case applies

dnolen 17:05:10

@denik you need to be careful when reading Om Next code :wink:

denik 17:05:11

the components has no queries and no props

dnolen 17:05:37

if it gets scheduled .forceUpdate will get called later

denik 17:05:39

@dnolen working on understanding it better, sorry

denik 17:06:32

okay I’ll try to dig in and report if I find something useful

dnolen 17:07:23

@denik so that might be a bug, look for the other forceUpdate call

dnolen 17:07:48

but it’s not clear to me that it is - you’ll see we check if the component should update

dnolen 17:07:52

if state changed we will

denik 17:09:05

@dnolen (p/schedule-render! reconciler) is not truthy

dnolen 17:09:45

@denik re-render only gets scheduled one time

dnolen 17:10:08

you must be getting to .forceUpdate

dnolen 17:10:21

you haven’t said that your thing doesn’t re-render just that the life-cycle bit doesn’t get called

dnolen 17:11:09

if you are getting rendered then you have to do what I said earlier

dnolen 17:11:17

verify that this isn’t a React bug somehow

thomasdeutsch 17:12:15

@dnolen: in conclusion, if i have a component that needs to query some data, i need a custom parser?

dnolen 17:13:30

@thomasdeutsch: that’s always the case :simple_smile:

dnolen 17:14:07

but I still fail to see why you can do this with a join

thomasdeutsch 17:14:12

yes.. ok.. i need a parser that is not only a (pull selector)

dnolen 17:14:24

no I didn’t say that

dnolen 17:14:28

you can do this with pull

dnolen 17:14:34

unless you need something more fancy in your query

dnolen 17:15:14

if you can fix in your DB do that over custom parsing

dnolen 17:15:23

I’ve been seen saying fix it in the DB

denik 17:28:08

@dnolen I think I found the issue: in reconcile! get-computed expects a component but is passed props https://github.com/omcljs/om/blob/a539a49979c878c47e2784202b11075093a74301/src/main/om/next.cljs#L1270

dnolen 17:29:37

@denik that’s also not true

dnolen 17:29:53

get-computed can take props or component

dnolen 17:30:14

@denik I would spend a lot more time on this before coming back with ideas

bhauman 17:39:09

@dnolen: so the only way to change an IQueryParams is through a side effect? Just verifying.

bhauman 17:39:47

I'm sure you are answering the same questions over and over again

dnolen 17:43:06

@bhauman: changing a query is a side effect

bhauman 17:43:18

cool just checking

dnolen 17:43:19

however unlike component local state it’s recorded into the atom

dnolen 17:43:28

so it doesn’t break time travel or anything like that at all

bhauman 17:43:54

I was thinking that was the case, gets rid of a lot of messiness

dnolen 17:44:17

will need to accommodate DataScript here but we’ll get around to it

bhauman 17:44:18

just wanted to make sure

denik 17:50:29

@dnolen component? throws because get-computed passes it (props c) which is nil. It tries to read om$isComponent from nil.

jannis 18:01:01

@denik: Quick fix for you: don't pass nil to the component as props.

dnolen 18:01:51

if it’s a defui instance it cannot take nil for props

dnolen 18:02:41

hrm actually this should only be true for IQuery defui instances

denik 18:04:00

that fixes it indeed. before (component) now (component {}

denik 18:05:08

however changing (get-computed (props c)) in reconciler! to (get-computed c) also resolves it for this particular case

dnolen 18:05:08

@denik try with master

dnolen 18:05:19

the bug is definitely component?

dnolen 18:05:36

@denik thanks for digging in

denik 18:05:37

thanks @dnolen!

denik 18:05:43

:simple_smile:

tony.kay 19:22:26

@dnolen: Just walked through the query root stuff with the team. It actually works out that the composition falls out without needing direct sub-roots support. Say you have a list of items that comes from the server: (rest-style URI /items). The new query root lets you remote that over. Then later a user clicks on an item (rest-style URI /items/42) and you want to show that as a modal over the list (as a UI sub-component).

You already have the items list in memory (app-state caching), so the parser won't even make a remote query for the items again, and the parser can just mark the specific item as the query root for that interaction, even though it is nested in the UI tree

So, something that you would analyze as a static UI tree (UI path rest-style: /dashboard/items/edit-dialog/42) looks like it might want multiple roots, but the trick is you don't need them at the same time!

tony.kay 19:24:33

I have not found a use-case (yet) that doesn't fall into this category, because if you think about "sub-roots" they always need the context of an upper root, which must have been resolved (due to sync model) at an earlier time.

dnolen 19:25:34

Yes this was my point last night about query root marking being a runtime decision

tony.kay 19:26:11

I didn't grok the "over time" aspect of that until I walked through some examples

dnolen 19:26:44

Anyways awesome feature glad you pushed for it!!

tony.kay 19:27:15

thanks. I think it is going to make our lives a lot easier.

dnolen 19:30:29

@tony.kay: so I already landed what’s needed as far as I can tell

dnolen 19:30:51

I’m loathe to change the signature of send in order to fully automate this stuff

dnolen 19:31:08

so instead there is a helper now called process-roots

dnolen 19:31:36

it returns a {:selector new-selector :rewrite (fn [res] …)}

dnolen 19:31:56

:selector is what you actually use, :rewrite you use to recover the response expected by your UI

tony.kay 19:34:27

I'll try it out. I have a talk to give today, so may take me a bit :simple_smile:

dnolen 19:59:15

so now that things are shaping up I think it’s time to really fixup some naming issues

dnolen 19:59:31

this means there will be some churn but it should be search & replace issue for everyone

dnolen 19:59:51

:selector is meh, I’m going to make this :query consistently everywhere

dnolen 20:00:17

the AST is now obviously part of the public api and needs sensible naming conventions

dnolen 20:00:25

:dkey is yuck for public stuff

dnolen 20:01:01

so AST will get documetation and :dkey will become :dispatch-key, stuff that humans can read and not want to break their computer

dnolen 20:01:33

if there are other naming conventions that people do not like, speak now (or soon) or forever hold your peace

josephjnk 20:05:44

can anyone tell me how to use time-travel with Om next? I can figure out how to get past state with (om/from-history reconciler #uuid "20cd52ee-7ea3-4178-b44a-aba6c4b69a9a"), but I'm not sure how to overwrite the current state with what this returns

jannis 20:48:47

@dnolen: I am about to try temp ids. What's the format for returning tempids -> real ids mappings though? And how would a server-side parser return the mapping?

dnolen 20:49:05

you return it in :value

dnolen 20:49:19

the structure of :value is now {:keys […] :tempids {…}}

dnolen 20:49:24

for transactions

dnolen 20:49:39

:tempids should a map of ident -> ident

jannis 20:49:42

Ah, yes. That makes sense.

dnolen 20:50:03

:tempids {[:todo/item #om.id[“…”]] [:todo/item 0]}

jannis 20:50:47

Yep, I was guessing as much from the tests and the reconciler sources. Just didn't know where :tempids would come from.

dnolen 20:50:53

actually that’s not right, fixing the above

dnolen 20:51:17

the server needs to be able to know what the tempids are

dnolen 20:51:24

so there’s a custom type now TempId with read/write handlers

jannis 20:51:27

But of course. res in merge is what is returned through :value.

jannis 20:51:53

Yep, saw that.

dnolen 20:51:58

OK cool

dnolen 22:17:36

landed a major refactor

dnolen 22:17:45

:selector is history, now :query

dnolen 22:18:02

fn params and source docs should reflect this as well

dnolen 22:18:54

yes fixed in the AST too

dnolen 22:19:10

the parser ns has grammar docs and AST description forthcoming

dnolen 22:19:13

working on that now

noonian 22:33:35

so in read implementations for a join expression the env will contain a :query key with the QueryRoot that is the value of the join? Instead of the old :selector?

dnolen 22:34:22

every where :selector was before you now have :query

jannis 22:38:20

Quick question about tempids: what's the best way to send them over to the server-side parser? Seems like transit doesn't handle them out of the box. So I I'm thinking add read/write handlers for them to my transit code?

dnolen 22:38:39

@jannis Om Next already provides them :simple_smile:

dnolen 22:38:49

see the bottom of om.next

dnolen 22:39:07

just finished adding docstrings

jannis 22:39:40

Oh :simple_smile:

dnolen 22:43:39

reader and writers for both Clojure and ClojureScript exist

dnolen 22:43:46

should fix om.next.server

jannis 22:49:47

Ok, client side works: (cljs-ajax.core/POST {:params tx :writer (om.transit/writer) :reader (om.transit/reader) ...}) - done :simple_smile:

dnolen 22:58:41

ok cool

dnolen 22:58:48

I haven’t actually tested this yet so you will beat me to it :simple_smile:

jannis 22:59:27

On the server side, (wrap-transit-json-params :encoder (om.next.server/reader)) (similar for response) should work. I'll test in a few minutes.