Fork me on GitHub
#untangled
<
2016-07-07
>
curtosis00:07:14

so I think I may finally be sort of grokking this... 1) my (server) call gets data into app-state at :sources 2) the reconciler normalizes and stuffs the real data into :sources/by-id and replaces the entries in :sources with ids - this works because I've defined Source as a component that has initial state and query 3) I run a :post-mutation to move the :sources list to [:sources-tab 1 :sources] 4) Profit?

tony.kay00:07:00

Yep, that's the general idea

curtosis00:07:35

slowly, emerging from the haze

tony.kay00:07:48

@curtosis: In plain Om, the general idea is that you merge it in and use a parser to morph it to a different form on the fly. In Untangled, we've found it easier to deal with it this way

tony.kay00:07:30

everybody understands data transforms and moves. It denormalizes your database a bit, but that is tractable to manage.

tony.kay00:07:46

well, it CAN denormalize it...

tony.kay01:07:40

for performance of your UI, I kinda felt like you were either going to be doing memoization (and managing the caching of that) in the parser or denormalizing the database. Given that cache invalidation is one of the "hard problems" of CS...a little denormazliation seems pretty tolerable.

curtosis01:07:40

now the tricky bit is that the Source component itself doesn't actually render ... it's a placeholder for the query, ident, etc. It actually gets rendered by a table component, so I need to deref it (`db->tree`, I think) to get access to the actual data for the cells.

curtosis01:07:53

yeah, that sounds eminently reasonable.

tony.kay01:07:15

you mean the thing you did the query with wasn't part of your UI?

tony.kay01:07:47

You should not need to make the tree. Untangled already has db->tree internally to make the tree. You need to make a subgraph in your db that matches your UI query.

curtosis01:07:10

so it kind of is in the UI...

tony.kay01:07:45

Right. Typically you have some sub-portion of the UI that represents real persisted data. And then there is all the local crap that makes up the other bits of UI.

curtosis01:07:53

I have a Sources component (is a tab) that has a :sources prop that gets its query from a Source component.

tony.kay01:07:32

Right, and that query needs to compose up the tree to root. If you use InitialAppState to build state up the tree

curtosis01:07:36

Sources renders a table, but because of the underlying component (fixed-data-table) I don't render it a Source at a time.

curtosis01:07:45

Yup, that bit's working great.

tony.kay01:07:22

oh, so you're saying your getting a subgraph from your query, because your query doesn't walk (query for) the entire tree of data?

curtosis01:07:05

Sources has a list :sources [[:sources/by-id 1] [:sources/by-id 2] ... ] ... but I have to provide a cell function that essentially maps (id, column-key) -> renderable Cell

tony.kay01:07:26

Each of those cells should be a component that queries for data

curtosis01:07:39

ahaaaaaaaa!

tony.kay01:07:00

just keep composing it until you've got components that mirror the data all the way down

curtosis01:07:39

hmm... would you define a different Cell component per column type? e.g. NameCell, FilenameCell, etc?

tony.kay01:07:03

Do the different cells need different queries?

tony.kay01:07:40

If so, that is a union query, just like in tabs, but where the cell type varies over the collection.

tony.kay01:07:04

So you'd have a "union switcher" component, and yes, a component for each cell type

curtosis01:07:21

I think yes... NameCell just needs :name; FilenameCell just needs :filename

tony.kay01:07:22

and your data would need a way to discern which it was in the ident function

tony.kay01:07:58

It's just like the tabs example, but instead of a single ident you have a vector of them (for each row)

tony.kay01:07:12

(Row is probably a component too)

curtosis01:07:26

it is, but I don't get to create the row

tony.kay01:07:14

That may be part of your data transform: make it look the way you need it to look for the UI

tony.kay01:07:38

Detect that "if it has :name, I'll add in a :type :namecell", etc

tony.kay01:07:39

The "magic" has to happen somewhere. In Untangled: it usually happens in a mutation (after a load or event)

tony.kay01:07:52

Try to limit the magic in the UI layer

curtosis01:07:36

yeah, that part makes sense. I'm just trying to adapt it to how this 3rd-party component expects things.

tony.kay01:07:48

oh...you're trying to use a React table component?

curtosis01:07:04

it made more sense than fighting jquery at the time.... 😉

tony.kay01:07:32

Ah, then what you want to do is this: Don't write query stuff for the parts that go to that component at all. Just drop a blob of JS data there

tony.kay01:07:45

query for the blob of js data, and pass that through

curtosis01:07:20

no value in keeping it in the database for other UI bits to operate on?

tony.kay01:07:29

a "property" in Om/Untangled can have any value: js Date, js map, Datascript database, whatever

tony.kay01:07:42

it still goes in the app database

curtosis01:07:07

but not normalized/by-id-ified?

curtosis01:07:37

heh.. yeah, that was my conclusion too. 😛

tony.kay01:07:38

If your goal is to display static data from a server: who cares

tony.kay01:07:04

If your goal is to interact dynamically with the data, then that is more complicated

curtosis01:07:11

in my case, the table display is just a summary. the real point is to edit the data.

tony.kay01:07:40

In that case you're probably going to find a parity mismatch with the design of the pre-built component and Om/Untangled.

curtosis01:07:03

(which I fear leads me down the nested tabs route, but that's a problem for tomorrow...)

tony.kay01:07:12

If your editing is outside of the table widget, then it isn't too bad

curtosis01:07:19

yes, that's my plan.

tony.kay01:07:25

one approach:

tony.kay01:07:38

1. Query and normalize the data 2. Write the UI and mutations that modify that data 3. Write an "update-table" mutation that transforms (via db->tree and clj->js, etc) into a form usable by the display widget 4. Trigger that mutation with modifications, and include a follow-on read for re-rendering the table

tony.kay01:07:08

So, you'll basically keep updating a "display version" of the information separate from the database table objects.

tony.kay01:07:13

A "view", so to speak

curtosis01:07:21

makes perfect sense

tony.kay01:07:34

In fact, that's how I imagine it...in SQL terms. There are the real tables, and there are views.

curtosis01:07:47

so my guess about db->tree wasn't completely wrong?

tony.kay01:07:53

in what way?

tony.kay01:07:06

You can certainly use db->tree as the transform tool

tony.kay01:07:18

you don't have the entire DB in a UI component, so you cannot do it there

tony.kay01:07:57

ok..gotta head out. good luck

curtosis01:07:27

thanks! Got some more ideas to mull over.... and should sleep on it as well. 🙂

curtosis15:07:14

I can't figure out where/why my data/props are getting rewritten... I have a :post-mutation to put loaded data into the right place for the tab to grab it. Three scenarios: 1) Straightforward untangled style: (assoc-in state [:sources-tab 1 :sources] idents) - works fine, Sources tab component gets the idents. 2) Manually stash the real data: (assoc-in state [:sources-tab 1 :sources] (values sources-by-id)) - data is there in app-state; Sources component gets [{} {} {} {} {} {} {} {} {} {}] in the sources prop 3) Manually stash the real data at app-state root: (assoc-in state [:sources-raw] (values sources-by-id)) - data is there in app-state; Sources component gets [nil nil nil nil nil nil nil nil] in the sources-raw prop (which has a corresponding link '[:sources-raw _] in its query.

curtosis15:07:18

I could sort of expect #2 if the query is filtering what comes from the app-state, but #3 should be coming through intact, it seems.

therabidbanana15:07:40

I think this might be something I've seen before trying to fight post-mutations - but I'm probably getting my explanation of what I was fighting way wrong: basically it seems like something that has an ident on the component you write the query for must be normalized to be queryable.

therabidbanana15:07:35

I was trying to do something like scenario #3 you describe, and seeing the same thing

curtosis15:07:24

hmm... but that doesn't make sense to me. I mean, in #3 it's just data I've stashed and could be anything. Why is the query machinery mucking about in it?

therabidbanana15:07:05

I think I just had an array of dumb objects and not even a component with an ident, and it hated that too - just having an array of objects was enough to cause issues, and the fix was to give it the full om treatment (in my case, a list of users, had to build a User component with ident and use that query)

therabidbanana15:07:21

As to the why, I'm not entirely sure, didn't really make sense to me either. 😄

curtosis15:07:03

I've got another set of random data getting stuffed at app-state root that works fine

curtosis15:07:18

In a component that also has queries.

therabidbanana15:07:40

Is it a *vector? I was seeing the effect not happen on *maps. (I also had a current-user I was pulling via a link to root query that worked wonderfully)

curtosis15:07:51

(and accesses that data via a link)

curtosis15:07:15

mine is an :app-info map

curtosis15:07:22

and that works like a champ.

curtosis15:07:31

so you think it's because it's a vector?

therabidbanana15:07:59

Yeah, that was the thing I was fighting for sure - for some reason it didn't act the same because it was a vector

therabidbanana15:07:33

Like some underlying part of the system wanted to force me to normalize anything in a vector, was the sense I got.

therabidbanana15:07:07

Maybe it has to do with the default read untangled provides, or maybe it's underlying how om queries work when they encounter *vectors, not sure

curtosis15:07:08

just confirmed: a map at app-state root comes through intact via the link

curtosis15:07:32

yeah, that's really not expected behavior to me - at least not yet 😉

therabidbanana16:07:17

Definitely not for me either - I spent a good half day at least banging my head against my desk with this. 😄

curtosis16:07:58

can't stuff arbitrary data into the component's map, though (#2) which is, more or less, expected; that part of app-state is "owned" by the component and the query machinery.

therabidbanana16:07:50

I think that's the same root cause - if you put idents there instead, you might have better luck

ethangracer16:07:48

maps at top level keywords accessed by links will work

ethangracer16:07:58

maps nested in vectors anywhere in the app-state will not work

ethangracer16:07:12

it’s an om issue with db->tree

ethangracer16:07:20

it assumes that vectors contain idents when the query has a join

curtosis16:07:40

yeah, the idents worked fine (that's #1). My problem is I need to get at the underlying data, and I feel like om/ref->any is the "wrong way"

ethangracer16:07:02

I was talking more about #2

ethangracer16:07:02

I’ve never tried or run into #3

therabidbanana16:07:32

It's basically the same thing because you join - just to a root area instead of further in the tree

therabidbanana16:07:27

This definitely seems like a rough edge worth pointing out in the tutorial or something (maybe it was and I didn't pay attention? Lots of good info there)

ethangracer16:07:41

I wouldn’t be surprised if it were missing

ethangracer16:07:03

it’s one of those things that’s more of a function of untangled being an alpha framework… operating on top of another alpha framework

ethangracer16:07:19

we could definitely benefit from a known issues section

therabidbanana16:07:51

Yeah, for sure - but surprisingly this was probably the first thing that acted completely unexpected. Once I got the hang of how queries work, for the most part everything has made a ton of sense.

ethangracer16:07:03

that’s awesome to hear

ethangracer16:07:20

the changes tony has mode most recently in 0.5.3 really helped to simplify things too

ethangracer16:07:35

we’re really happy with how much simpler it’s becoming

ethangracer16:07:22

@curtosis: did any of that help at all or are you still stuck?

curtosis16:07:05

I think it helps. 🙂

curtosis16:07:14

What's confusing me now is where the line is...

curtosis16:07:47

as in, at app-state root :sources-data [{:key val}{:key val}] will not work

curtosis16:07:10

but this does: :sources-data {:stuff [{:key val}{:key val}]}

curtosis16:07:48

is that because :stuff doesn't show up in any component or Root and so om just doesn't care?

therabidbanana16:07:52

I think it's more because you're not joining on :stuff. As long as you don't try to write a join query that looks into the :stuff key you'd probably be okay

therabidbanana16:07:24

So you can treat :stuff as a raw value and that'd probably be okay

therabidbanana16:07:05

But if you tried to go {:stuff [:key]}, then you'd need to normalize the things in that array

therabidbanana16:07:19

It generally makes sense I suppose if you think of any join syntax as its equivalent in SQL - you'd need anything you join to be in normalized into a separate table, so if you do a join in your query, then that's expected behavior. Hopefully that'll help me think about it in case I run into this again.

ethangracer16:07:32

^^ yes that is my understanding as well

ethangracer16:07:03

@curtosis: your first example doesn’t work because om assumes that the vector is a list of idents, and tries to denormalize data that is already denormalized

ethangracer16:07:22

the second one works because :sources-data is a map, not a vector — om doesn’t make assumptions about its structure

curtosis16:07:37

@ethangracer: right, but the gotcha is that :stuff is a vector, and the magic is that by not invoking any query machinery on :stuff it gets ignored.

ethangracer16:07:28

yup… definitely an esoteric detail

ethangracer16:07:57

there is one way it kind of makes sense — if you don’t include a nested query, om just dumps what it finds

ethangracer16:07:15

if you DO include a nested query, it goes through denormalization

ethangracer16:07:31

if the data isn’t normalized, there’s no need to add a nested query

curtosis16:07:25

I keep reminding myself that this is so much simpler once I get through the mind warp. 😛

curtosis16:07:39

(and, admittedly, my current problem is trying to deal with a 3rd-party react component that doesn't map nicely to my data component hierarchy)

curtosis16:07:35

the details might still be scroll-up-able, but the short version is: in a normal Sources table you'd iterate over Source rows, no problem. But fixed-data-table doesn't do that; it iterates over cells... so I could either decompose everything down to individual Cell components or just side-load everything in a format that is easier to work with.

curtosis16:07:20

or punt to om/ref->any to do the lookup.

w1ng17:07:28

is there a way to set params for the post-mutation of untangled/load transaction?

therabidbanana18:07:20

I don't think post-mutations can take params @w1ng

ethangracer18:07:08

@w1ng nope, no post-mutation parameters. what parameters apart from the app-state do you need?

w1ng19:07:08

im writing an autocomplete component and want to pass the path where to merge the loaded data

jasonjckn19:07:49

in untangled-todomvc, send-support-request mutation returns the ID in the mutation,

(defmethod apimutate 'support-viewer/send-support-request [e k p]
  {:action
   (fn []
     (let [_ (swap! last-id inc)
           id @last-id]
       (timbre/info "New support request " id)
       (swap! requests assoc id p)
       id))})
how do I get access to that ID in the client?

jasonjckn19:07:22

I want the client to know the request ID, not need to look at the server logs

w1ng19:07:57

as a workaround i made a mutation which sets the params i want in the post-mutation in the app state and the post-mutation reads the params from there, but i dont think thats a good solution

ethangracer19:07:02

@w1ng so you have an autocomplete component that might merge data to different places in the app state depending on the data. currently your workaround is to store that location in app-state so the post-mutation knows where to put the data. am I getting that right?

ethangracer19:07:51

is there no way to tell where the data belongs based on what’s returned from the server?

ethangracer19:07:05

just trying to get a sense of how to suggest alternative approaches

ethangracer19:07:12

@jasonjckn: you should be able to access it from app-state

jasonjckn19:07:29

@ethangracer: where is it loaded?

ethangracer19:07:39

the return values of action thunks are dropped

jasonjckn19:07:50

hm ok, i'll take a look

ethangracer19:07:54

oh, you’re not swapping on the app state

ethangracer19:07:02

I haven’t played much with the support viewer

jasonjckn19:07:16

i think there's no way to get the return value of a remote mutation

jasonjckn19:07:27

i need to use untangled-load

ethangracer19:07:22

yeah not too sure about the support-viewer

ethangracer19:07:42

I know that you have to return a map from the mutation, keyed either by :tempids or :value

ethangracer19:07:15

so the last line of your code above would have to be:

{:value id}

jasonjckn19:07:10

(defmethod mutate 'permalink/create [{:keys [ast state] :as env} k p]
  {:remote (df/remote-load env)
   :action (fn []
             (df/load-data-action state '[:permalink]
                                  :marker false
                                  :params {:permalink {}}))})
gives client-side error
[  1.214s] [om.next] transacted '[(permalink/create)], #uuid "80e58bd9-b41c-4ee8-aa38-844e749449a0"
user.cljs:96 APP-STATE DIFF:  {[:ui/loading-data] [:+ true :- false]}
core.cljs:3081 Uncaught Error: Doesn't support namespace: 

jasonjckn19:07:31

if I change

:permalink {}
to
:permalink nil
it works

jasonjckn19:07:37

same problem with

#(transact! [(untangled/load {:query [:permalink]
                                                                :params {:permalink {}}})])

w1ng19:07:56

@ethangracer: yes exactly. the location im storing in the app state has two keys :from and :to . the idea was to have a generic autocomplete component which can handle different types of data (clients, products,...) based on the data passed to it and the transactions passed to onChange

jasonjckn19:07:11

nevermind the issue is on my end, not untangled

w1ng20:07:42

@ethangracer: heres my code:

(defmethod m/mutate 'app/set-load-targets [{:keys [state] :as env} k {:keys [from to]}]
  {:action (fn []
             (swap! state assoc-in [:load-data-targets :to] to)
             (swap! state assoc-in [:load-data-targets :from] from)
             )})

(defmethod m/mutate 'app/merge-load-targets [{:keys [state] :as env} k _]
  (let [current-tab (get-in @state [:current-tab 0])
        from (:from (:load-data-targets @state))
        to (:to (:load-data-targets @state))]
    {:action (fn []
               (swap! state assoc-in [current-tab :tab to] (from (:load-data @state)))
               )}))

(autocomplete/ui
 {:floatingLabelText "Clients"
  :onSelect #(m/set-value! this :ui/selectclients-val %)
  :onUpdateInput (fn [search-text data-source]
                   (m/set-value! this :ui/selectclients-text search-text)
                   (om/transact!
                    this
                    `[(app/set-load-targets {:from :clients
                                             :to :ui/selectclients})
                      (untangled/load {:query
                                       ({:load-data
                                         [(:clients
                                           {:fuzzysearch
                                            ~search-text})]})
                                       :refresh [:current-tab]
                                       :post-mutation app/merge-load-targets})
                      :current-tab ]))
  :data (map client-select-item selectclients)
  :searchText selectclients-text})

w1ng20:07:24

the autocomplete/ui call is in a component

ethangracer20:07:10

@jasonjckn: yes, I’ve been seeing this ^^ too

ethangracer20:07:30

@w1ng: so what you’re saying is you’d like to have some way of getting the :from and :to keys into the mutation. interesting. let me think about that for a bit

ethangracer20:07:18

your workaround is how I’d approach it as well, need to spin some brain cycles on whether that’s a desirable pattern or not

currentoor21:07:40

In our app we've got some attributes in client side queries that are not actually retrievable with the pul API. For example the query [:dashboard/title :dashboard/created-at] title works with datomic.api/pull but created-at is derived with a query over the :db/txInstant.

currentoor21:07:11

Ideally we'd like to have an extensible pull wrapper, call it pluck, where we could specify custom ways to query keys like :dashboard/created-at but the results get merged back in such that it looks like a regular pull API call.

currentoor21:07:22

@ethangracer @tony.kay: maybe you've run into this usecase? Any suggestions for how to address it?

currentoor21:07:26

I suppose I should also ask in the datomic channel.

currentoor21:07:23

I'm thinking an extensible API like so would be useful.

ethangracer21:07:11

@currentoor: definitely have run into the use case, we are post-processing data on the server to rename the keys as the client expects them to appear

ethangracer21:07:40

I like the idea of that kind of api

currentoor21:07:47

@ethangracer: if you change the queries on the client will the server omit them or send back extra data?

ethangracer22:07:13

I exclude them from the server query using :without on data fetch

ethangracer22:07:46

then when the server returns the data it normalizes it using the full UI query

ethangracer22:07:11

oh, actually i’m using a client-side post process as well

currentoor22:07:22

right we have some issues like that also

currentoor22:07:02

like querying :dashboard/by-id will always return created-at updated-at whether the client asks for them or not

currentoor22:07:47

i could go into the server side read function and put some conditional logic for these two keys but that seems way to special-casey

currentoor22:07:04

a pull api wrapper would be much preferable

ethangracer22:07:23

let’s think about that then, how would it work

ethangracer22:07:32

the UI query would have to remain intact for normalization

ethangracer22:07:16

so either before the query is sent to the server, or after the query hits the server, we would have to intelligently replace keys that correspond to the database schema

ethangracer22:07:43

then once the data is returned, we’d replace the data returned from the database with the keys required by the client query

ethangracer22:07:03

missing anything?

currentoor22:07:09

yeah that sounds about right

currentoor22:07:27

i was thinking maybe a ring middleware type approach would be appropriate

currentoor22:07:34

since we need to do pre and post processing

currentoor22:07:56

with multimethods for the individual keys

ethangracer22:07:59

which would have access to that pluck api you mentioned?

ethangracer22:07:06

yeah I think i’m with you

ethangracer22:07:46

why is this more preferable than manual modification by query?

currentoor22:07:54

but the tricky bit is traversing the tree in this ring-middleware thing and then putting stuff back in the right spot

currentoor22:07:27

oh because it might not be just a simple query attribute re-naming

currentoor22:07:55

like :dashboard/created-at there is not created-at attribute in the dashboard model

currentoor22:07:05

it's derived from the transaction time

ethangracer22:07:56

yeah i’m still not totally clear on what you mean by that

ethangracer22:07:31

it sounds like you have some datomic schema for the namespace dashboard

ethangracer22:07:37

but created-at is not one of them

ethangracer22:07:01

so you want to write a custom query to fill that field

currentoor22:07:02

created at is retrived like so

(d/q '[:find
         ?created-at .
         :in $ ?d
         :where
         [?d :dashboard/organization _ ?org-tx _]
         [?org-tx :db/txInstant ?created-at]]
       db
       eid)

currentoor22:07:23

but on the client created-at is a regular field like anything else on the dashboard

ethangracer22:07:08

yeah we haven’t run into this exact issue — we have some reporting stats that we calculate using aggregates, so we have to invent keys to return to the client, since there is somewhat significant server-side processing involved in formatting that aggregated data properly

ethangracer22:07:34

so similar idea, I guess I’m wondering if this api you’re suggesting could be easily extended to support that use case as well

ethangracer22:07:10

I definitely see how it could be useful, my only remaining question would be where it belongs. untangled-datomic? untangled-server? would this apply to other DB models?

currentoor22:07:27

my guess would be untangled-datomic unless we can make it super general

currentoor22:07:09

i'm going to try and pair with @therabidbanana on this

currentoor22:07:22

hopefully we can come up with something that covers both our usecases

ethangracer22:07:48

sounds good, let me know if I can do anything to help

ethangracer22:07:36

@currentoor: snuck in a bit of time from iCTO

ethangracer22:07:15

2 thoughts. one is that a created-at date derived from a database transaction is unreliable, since any subsequent modification of that attribute would change the transaction time

ethangracer22:07:51

so a true query to get the initial created-date would be somewhat involved, its easier to store a date as a separate attribute on the entity

ethangracer22:07:26

second thought: we don’t want to introduce more middleware that is only going to be used for certain kinds of queries, because it’ll just slow server response times for the general use case queries that don’t have this concern

currentoor22:07:10

Yeah created-at in general should use the history db and grab the smallest timestamp, but for our usecase the organization is only set once and we have validations preventing it from being updated.

currentoor23:07:37

Also I meant ring middleware style decorator pattern, not actual server middleware.

currentoor23:07:05

Lol that would be ridiculous! But I can see how what I said was confusing.