so I think I may finally be sort of grokking this... 1) my (server) call gets data into app-state at :sources 2) the reconciler normalizes and stuffs the real data into :sources/by-id and replaces the entries in :sources with ids - this works because I've defined Source as a component that has initial state and query 3) I run a :post-mutation to move the :sources list to [:sources-tab 1 :sources] 4) Profit?

Yep, that's the general idea

slowly, emerging from the haze

@curtosis: In plain Om, the general idea is that you merge it in and use a parser to morph it to a different form on the fly. In Untangled, we've found it easier to deal with it this way

everybody understands data transforms and moves. It denormalizes your database a bit, but that is tractable to manage.

well, it CAN denormalize it...

heh

for performance of your UI, I kinda felt like you were either going to be doing memoization (and managing the caching of that) in the parser or denormalizing the database. Given that cache invalidation is one of the "hard problems" of CS...a little denormazliation seems pretty tolerable.

now the tricky bit is that the Source component itself doesn't actually render ... it's a placeholder for the query, ident, etc. It actually gets rendered by a table component, so I need to deref it (`db->tree`, I think) to get access to the actual data for the cells.

yeah, that sounds eminently reasonable.

you mean the thing you did the query with wasn't part of your UI?

You should not need to make the tree. Untangled already has db->tree internally to make the tree. You need to make a subgraph in your db that matches your UI query.

so it kind of is in the UI...

Right. Typically you have some sub-portion of the UI that represents real persisted data. And then there is all the local crap that makes up the other bits of UI.

I have a Sources component (is a tab) that has a :sources prop that gets its query from a Source component.

Right, and that query needs to compose up the tree to root. If you use InitialAppState to build state up the tree

Sources renders a table, but because of the underlying component (fixed-data-table) I don't render it a Source at a time.

Yup, that bit's working great.

oh, so you're saying your getting a subgraph from your query, because your query doesn't walk (query for) the entire tree of data?

Sources has a list :sources [[:sources/by-id 1] [:sources/by-id 2] ... ] ... but I have to provide a cell function that essentially maps (id, column-key) -> renderable Cell

Each of those cells should be a component that queries for data

ahaaaaaaaa!

just keep composing it until you've got components that mirror the data all the way down

hmm... would you define a different Cell component per column type? e.g. NameCell, FilenameCell, etc?

Do the different cells need different queries?

If so, that is a union query, just like in tabs, but where the cell type varies over the collection.

So you'd have a "union switcher" component, and yes, a component for each cell type

I think yes... NameCell just needs :name; FilenameCell just needs :filename

and your data would need a way to discern which it was in the ident function

It's just like the tabs example, but instead of a single ident you have a vector of them (for each row)

(Row is probably a component too)

it is, but I don't get to create the row

That may be part of your data transform: make it look the way you need it to look for the UI

Detect that "if it has :name, I'll add in a :type :namecell", etc

The "magic" has to happen somewhere. In Untangled: it usually happens in a mutation (after a load or event)

Try to limit the magic in the UI layer

yeah, that part makes sense. I'm just trying to adapt it to how this 3rd-party component expects things.

oh...you're trying to use a React table component?

yep

it made more sense than fighting jquery at the time.... 😉

Ah, then what you want to do is this: Don't write query stuff for the parts that go to that component at all. Just drop a blob of JS data there

query for the blob of js data, and pass that through

no value in keeping it in the database for other UI bits to operate on?

a "property" in Om/Untangled can have any value: js Date, js map, Datascript database, whatever

it still goes in the app database

but not normalized/by-id-ified?

depends

heh.. yeah, that was my conclusion too. 😛

If your goal is to display static data from a server: who cares

If your goal is to interact dynamically with the data, then that is more complicated

in my case, the table display is just a summary. the real point is to edit the data.

In that case you're probably going to find a parity mismatch with the design of the pre-built component and Om/Untangled.

(which I fear leads me down the nested tabs route, but that's a problem for tomorrow...)

If your editing is outside of the table widget, then it isn't too bad

yes, that's my plan.

ok

one approach:

1. Query and normalize the data 2. Write the UI and mutations that modify that data 3. Write an "update-table" mutation that transforms (via db->tree and clj->js, etc) into a form usable by the display widget 4. Trigger that mutation with modifications, and include a follow-on read for re-rendering the table

So, you'll basically keep updating a "display version" of the information separate from the database table objects.

A "view", so to speak

makes perfect sense

In fact, that's how I imagine it...in SQL terms. There are the real tables, and there are views.

so my guess about db->tree wasn't completely wrong?

in what way?

You can certainly use db->tree as the transform tool

you don't have the entire DB in a UI component, so you cannot do it there

ah ok

ok..gotta head out. good luck

thanks! Got some more ideas to mull over.... and should sleep on it as well. 🙂

I can't figure out where/why my data/props are getting rewritten... I have a :post-mutation to put loaded data into the right place for the tab to grab it. Three scenarios: 1) Straightforward untangled style: (assoc-in state [:sources-tab 1 :sources] idents) - works fine, Sources tab component gets the idents. 2) Manually stash the real data: (assoc-in state [:sources-tab 1 :sources] (values sources-by-id)) - data is there in app-state; Sources component gets [{} {} {} {} {} {} {} {} {} {}] in the sources prop 3) Manually stash the real data at app-state root: (assoc-in state [:sources-raw] (values sources-by-id)) - data is there in app-state; Sources component gets [nil nil nil nil nil nil nil nil] in the sources-raw prop (which has a corresponding link '[:sources-raw _] in its query.

I could sort of expect #2 if the query is filtering what comes from the app-state, but #3 should be coming through intact, it seems.

I think this might be something I've seen before trying to fight post-mutations - but I'm probably getting my explanation of what I was fighting way wrong: basically it seems like something that has an ident on the component you write the query for must be normalized to be queryable.

I was trying to do something like scenario #3 you describe, and seeing the same thing

hmm... but that doesn't make sense to me. I mean, in #3 it's just data I've stashed and could be anything. Why is the query machinery mucking about in it?

I think I just had an array of dumb objects and not even a component with an ident, and it hated that too - just having an array of objects was enough to cause issues, and the fix was to give it the full om treatment (in my case, a list of users, had to build a User component with ident and use that query)

As to the why, I'm not entirely sure, didn't really make sense to me either. 😄

I've got another set of random data getting stuffed at app-state root that works fine

In a component that also has queries.

Is it a *vector? I was seeing the effect not happen on *maps. (I also had a current-user I was pulling via a link to root query that worked wonderfully)

(and accesses that data via a link)

mine is an :app-info map

and that works like a champ.

so you think it's because it's a vector?

Yeah, that was the thing I was fighting for sure - for some reason it didn't act the same because it was a vector

Like some underlying part of the system wanted to force me to normalize anything in a vector, was the sense I got.

Maybe it has to do with the default read untangled provides, or maybe it's underlying how om queries work when they encounter *vectors, not sure

just confirmed: a map at app-state root comes through intact via the link

yeah, that's really not expected behavior to me - at least not yet 😉

Definitely not for me either - I spent a good half day at least banging my head against my desk with this. 😄

can't stuff arbitrary data into the component's map, though (#2) which is, more or less, expected; that part of app-state is "owned" by the component and the query machinery.

I think that's the same root cause - if you put idents there instead, you might have better luck

maps at top level keywords accessed by links will work

maps nested in vectors anywhere in the app-state will not work

it’s an om issue with db->tree

it assumes that vectors contain idents when the query has a join

yeah, the idents worked fine (that's #1). My problem is I need to get at the underlying data, and I feel like om/ref->any is the "wrong way"

aaaaaaah

I was talking more about #2

I think

I’ve never tried or run into #3

It's basically the same thing because you join - just to a root area instead of further in the tree

oh, ok

I imagine

This definitely seems like a rough edge worth pointing out in the tutorial or something (maybe it was and I didn't pay attention? Lots of good info there)

I wouldn’t be surprised if it were missing

it’s one of those things that’s more of a function of untangled being an alpha framework… operating on top of another alpha framework

we could definitely benefit from a known issues section

Yeah, for sure - but surprisingly this was probably the first thing that acted completely unexpected. Once I got the hang of how queries work, for the most part everything has made a ton of sense.

that’s awesome to hear

the changes tony has mode most recently in 0.5.3 really helped to simplify things too

we’re really happy with how much simpler it’s becoming

@curtosis: did any of that help at all or are you still stuck?

I think it helps. 🙂

What's confusing me now is where the line is...

as in, at app-state root :sources-data [{:key val}{:key val}] will not work

but this does: :sources-data {:stuff [{:key val}{:key val}]}

is that because :stuff doesn't show up in any component or Root and so om just doesn't care?

I think it's more because you're not joining on :stuff. As long as you don't try to write a join query that looks into the :stuff key you'd probably be okay

So you can treat :stuff as a raw value and that'd probably be okay

But if you tried to go {:stuff [:key]}, then you'd need to normalize the things in that array

It generally makes sense I suppose if you think of any join syntax as its equivalent in SQL - you'd need anything you join to be in normalized into a separate table, so if you do a join in your query, then that's expected behavior. Hopefully that'll help me think about it in case I run into this again.

^^ yes that is my understanding as well

@curtosis: your first example doesn’t work because om assumes that the vector is a list of idents, and tries to denormalize data that is already denormalized

the second one works because :sources-data is a map, not a vector — om doesn’t make assumptions about its structure

@ethangracer: right, but the gotcha is that :stuff is a vector, and the magic is that by not invoking any query machinery on :stuff it gets ignored.

yup… definitely an esoteric detail

there is one way it kind of makes sense — if you don’t include a nested query, om just dumps what it finds

if you DO include a nested query, it goes through denormalization

if the data isn’t normalized, there’s no need to add a nested query

I keep reminding myself that this is so much simpler once I get through the mind warp. 😛

(and, admittedly, my current problem is trying to deal with a 3rd-party react component that doesn't map nicely to my data component hierarchy)

the details might still be scroll-up-able, but the short version is: in a normal Sources table you'd iterate over Source rows, no problem. But fixed-data-table doesn't do that; it iterates over cells... so I could either decompose everything down to individual Cell components or just side-load everything in a format that is easier to work with.

or punt to om/ref->any to do the lookup.

is there a way to set params for the post-mutation of untangled/load transaction?

I don't think post-mutations can take params @w1ng

@w1ng nope, no post-mutation parameters. what parameters apart from the app-state do you need?

im writing an autocomplete component and want to pass the path where to merge the loaded data

in untangled-todomvc, send-support-request mutation returns the ID in the mutation,

(defmethod apimutate 'support-viewer/send-support-request [e k p]
  {:action
   (fn []
     (let [_ (swap! last-id inc)
           id @last-id]
       (timbre/info "New support request " id)
       (swap! requests assoc id p)
       id))})

I want the client to know the request ID, not need to look at the server logs

as a workaround i made a mutation which sets the params i want in the post-mutation in the app state and the post-mutation reads the params from there, but i dont think thats a good solution

@w1ng so you have an autocomplete component that might merge data to different places in the app state depending on the data. currently your workaround is to store that location in app-state so the post-mutation knows where to put the data. am I getting that right?

is there no way to tell where the data belongs based on what’s returned from the server?

just trying to get a sense of how to suggest alternative approaches

@jasonjckn: you should be able to access it from app-state

@ethangracer: where is it loaded?

the return values of action thunks are dropped

hm ok, i'll take a look

oh, you’re not swapping on the app state

I haven’t played much with the support viewer

i think there's no way to get the return value of a remote mutation

i need to use untangled-load

ah

yeah not too sure about the support-viewer

I know that you have to return a map from the mutation, keyed either by :tempids or :value

so the last line of your code above would have to be:

{:value id}

(defmethod mutate 'permalink/create [{:keys [ast state] :as env} k p]
  {:remote (df/remote-load env)
   :action (fn []
             (df/load-data-action state '[:permalink]
                                  :marker false
                                  :params {:permalink {}}))})

[  1.214s] [om.next] transacted '[(permalink/create)], #uuid "80e58bd9-b41c-4ee8-aa38-844e749449a0"
user.cljs:96 APP-STATE DIFF:  {[:ui/loading-data] [:+ true :- false]}
core.cljs:3081 Uncaught Error: Doesn't support namespace: 

if I change

:permalink {}

:permalink nil

same problem with

#(transact! [(untangled/load {:query [:permalink]
                                                                :params {:permalink {}}})])

@ethangracer: yes exactly. the location im storing in the app state has two keys :from and :to . the idea was to have a generic autocomplete component which can handle different types of data (clients, products,...) based on the data passed to it and the transactions passed to onChange

nevermind the issue is on my end, not untangled

@ethangracer: heres my code:

(defmethod m/mutate 'app/set-load-targets [{:keys [state] :as env} k {:keys [from to]}]
  {:action (fn []
             (swap! state assoc-in [:load-data-targets :to] to)
             (swap! state assoc-in [:load-data-targets :from] from)
             )})

(defmethod m/mutate 'app/merge-load-targets [{:keys [state] :as env} k _]
  (let [current-tab (get-in @state [:current-tab 0])
        from (:from (:load-data-targets @state))
        to (:to (:load-data-targets @state))]
    {:action (fn []
               (swap! state assoc-in [current-tab :tab to] (from (:load-data @state)))
               )}))

(autocomplete/ui
 {:floatingLabelText "Clients"
  :onSelect #(m/set-value! this :ui/selectclients-val %)
  :onUpdateInput (fn [search-text data-source]
                   (m/set-value! this :ui/selectclients-text search-text)
                   (om/transact!
                    this
                    `[(app/set-load-targets {:from :clients
                                             :to :ui/selectclients})
                      (untangled/load {:query
                                       ({:load-data
                                         [(:clients
                                           {:fuzzysearch
                                            ~search-text})]})
                                       :refresh [:current-tab]
                                       :post-mutation app/merge-load-targets})
                      :current-tab ]))
  :data (map client-select-item selectclients)
  :searchText selectclients-text})

the autocomplete/ui call is in a component

https://github.com/untangled-web/untangled-client/issues/22

@jasonjckn: yes, I’ve been seeing this ^^ too

@w1ng: so what you’re saying is you’d like to have some way of getting the :from and :to keys into the mutation. interesting. let me think about that for a bit

your workaround is how I’d approach it as well, need to spin some brain cycles on whether that’s a desirable pattern or not

In our app we've got some attributes in client side queries that are not actually retrievable with the pul API. For example the query [:dashboard/title :dashboard/created-at] title works with datomic.api/pull but created-at is derived with a query over the :db/txInstant.

Ideally we'd like to have an extensible pull wrapper, call it pluck, where we could specify custom ways to query keys like :dashboard/created-at but the results get merged back in such that it looks like a regular pull API call.

@ethangracer @tony.kay: maybe you've run into this usecase? Any suggestions for how to address it?

I suppose I should also ask in the datomic channel.

I'm thinking an extensible API like so would be useful.

@currentoor: definitely have run into the use case, we are post-processing data on the server to rename the keys as the client expects them to appear

I like the idea of that kind of api

@ethangracer: if you change the queries on the client will the server omit them or send back extra data?

I exclude them from the server query using :without on data fetch

then when the server returns the data it normalizes it using the full UI query

oh, actually i’m using a client-side post process as well

right we have some issues like that also

like querying :dashboard/by-id will always return created-at updated-at whether the client asks for them or not

i could go into the server side read function and put some conditional logic for these two keys but that seems way to special-casey

a pull api wrapper would be much preferable

let’s think about that then, how would it work

the UI query would have to remain intact for normalization

so either before the query is sent to the server, or after the query hits the server, we would have to intelligently replace keys that correspond to the database schema

then once the data is returned, we’d replace the data returned from the database with the keys required by the client query

missing anything?

yeah that sounds about right

i was thinking maybe a ring middleware type approach would be appropriate

since we need to do pre and post processing

with multimethods for the individual keys

which would have access to that pluck api you mentioned?

yeah I think i’m with you

yeah

interesting

why is this more preferable than manual modification by query?

but the tricky bit is traversing the tree in this ring-middleware thing and then putting stuff back in the right spot

right

oh because it might not be just a simple query attribute re-naming

like :dashboard/created-at there is not created-at attribute in the dashboard model

it's derived from the transaction time

yeah i’m still not totally clear on what you mean by that

it sounds like you have some datomic schema for the namespace dashboard

but created-at is not one of them

correct?

yes

ok

so you want to write a custom query to fill that field

created at is retrived like so

(d/q '[:find
         ?created-at .
         :in $ ?d
         :where
         [?d :dashboard/organization _ ?org-tx _]
         [?org-tx :db/txInstant ?created-at]]
       db
       eid)

but on the client created-at is a regular field like anything else on the dashboard

right

yeah we haven’t run into this exact issue — we have some reporting stats that we calculate using aggregates, so we have to invent keys to return to the client, since there is somewhat significant server-side processing involved in formatting that aggregated data properly

so similar idea, I guess I’m wondering if this api you’re suggesting could be easily extended to support that use case as well

I definitely see how it could be useful, my only remaining question would be where it belongs. untangled-datomic? untangled-server? would this apply to other DB models?

my guess would be untangled-datomic unless we can make it super general

i'm going to try and pair with @therabidbanana on this

hopefully we can come up with something that covers both our usecases

sounds good, let me know if I can do anything to help

@currentoor: snuck in a bit of time from iCTO

2 thoughts. one is that a created-at date derived from a database transaction is unreliable, since any subsequent modification of that attribute would change the transaction time

so a true query to get the initial created-date would be somewhat involved, its easier to store a date as a separate attribute on the entity

second thought: we don’t want to introduce more middleware that is only going to be used for certain kinds of queries, because it’ll just slow server response times for the general use case queries that don’t have this concern

Yeah created-at in general should use the history db and grab the smallest timestamp, but for our usecase the organization is only set once and we have validations preventing it from being updated.

Also I meant ring middleware style decorator pattern, not actual server middleware.

Lol that would be ridiculous! But I can see how what I said was confusing.