* you might have different middleware/api (we have castra and compojure for some other stuff) * with on-error you have the ability to actually also catch the function call with the passed arguments, not sure how you could do that with a extra middleware

when reporting/logging exceptions you wanna give as much context as possible (i also log all the headers and user info - who/where had error)

Love the Panoply project!

hey, @micha is with-dom supposed to do something when an element is added back to the dom thanks to *-tpl?

@thedavidmeister no, it just does something the first time it sees the element in the dom

then it's done

any way to respond to something being added/removed?

you could add lifecycles, i guess

i haven't seen a case where that is needed though

just a third party lib

the auth0 lock widget has show/hide methods

just don't take it out of the dom

toggle visibility instead

it’s navigation though

e.g. logged in vs. logged out

i don't follow

why can't it stay in the dom all the time

and just hide/show

instead of taking it in and out

of the dom

i don’t know, auth0 seems to pull it out of the dom when you log in

still mid-debug

shouldn't it manage its own state, if it's the one pulling itself out of the dom

that shouldn't have anything to do with the -tpl business

but when i log in, then nav around a bit, then logout then navigate back to the start, it’s missing

well my top level nav/routes is handled with bidi and a case-tpl

yes it probably should manage its own state

i’m not blaming hoplon 😛

just wondering what my options are

i would lift it out of the case-tpl probably

your application will only have one auth badge or whatever

you could modify with-dom to never stop polling

it could miss some events though

so that's maybe not a good solution

mmm, polling seems inefficient

with-dom is pretty tight, like 20ms

it doesn't do much though

takes it maybe a millisecond or two to do its work

well actually i currently have 2 different pages with auth0 widget elements on it

the home page and the “403” (not a real 403)

i wrapped it up hoplon style so it looks like

(h/defelem login
  [_ _]
  (let [container (h/div
                   :id auth0.config/login-form-id)
        lock  (js/Auth0Lock. auth0.config.client-id auth0.config.domain)]

   (h/with-dom container
    (j/cell= (.show
              lock
              (auth0.config/lock-options)
              (fn [e p t]
               (when-not e (auth0.api/login! p t))))))

   container))

i think i would do away with the case-tpl there

and use :toggle on the clauses

hmmm

but then i’d have my entire site sitting in the dom

seems wrong

the case-tpl is like

(let [r (j/cell= (:handler route.state/location))]
             (h/div
               :id "index"
               (h/case-tpl r
                           :landing (el.landing-page.dom/landing-page)

                           :projects (el.projects.dom/projects)
                           :project (el.project.dom/project)
                           :project-insights (el.project-insights.dom/project-insights)

                           :privacy (el.privacy-policy.dom/privacy-policy)

                           :access-denied (el.access-denied.dom/access-denied)

                           :pattern-lib (el.pattern-lib.dom/pattern-lib)))))))

that list is just going to get longer

usually though the layout is hierarchical

with a small number of layouts

possibly many different screens inside each top level layout

but not usually a lot of different layouts in the application

like where headers and footers and whatnot are concerned

that's the thing i'd lift out of the case-tpl

i still don’t think it fixes the problem?

then you're never taking the auth thingy out of the dom

nono

it takes itself out of the dom when you log in

i think i wasn’t clear about that

where does case-tpl come into it?

that’s just how i did navigation

so i had a vague idea about “hooking in” to it putting the landing page or whatever back in the dom

to tell auth0 to get itself back

i think that (.show ) function should do the trick

with-dom works the first time, all good

but then say, you log in, do some stuff in the app, then log out, then navigate back to the landing page

auth0 removed itself from being visible on that landing page

whether its a toggle or a case tpl, or however you arrange the heirarchy

it doesn’t change the fact that the landing page is being re-used, not regenerated like it would be with a full page reload

so yeah.. there’s just a missing chunk in the page where the login form should be

yeah i would do it such that the auth thingy is not nested in a case-tpl

i don’t think that helps though

then it's just like any other html

unless this library isn't compatible with single page apps

which is possible

it’s supposed to be

i mean they explicitly advertise SPAs

i actually always refresh the page when the user logs out

it's too tricky to ensure that all sensitive state is correctly expunged

like imagine someone leaves their workstation unlocked, expecting that logging out was enough

things like that

also it gets even more tricky when the user can log out and then logs in as a different user

oh, but you can make a function that returns a cell that clears itself when you aren’t logged in

sure but we found that to be a source of bugs

how so?

our QA people found the bugs, not us

because we never tested the weird cases they did

but what type of bugs?

like weird issues with caching

whenever you have data cached it's tricky to invalidate that based on login info

you can add callbacks to zero out cells and stuff

but that was unwieldy

(let [c (j/cell nil)
      nuke! (reset! c nil)]
  (j/cell= (when-not logged-in? (nuke!)))
  c)

something like that

yeah

well caches are tough generally

you can do that, but you run into issues if the use refreshes

for example

like they're logged in, but you can't know that until you make a request to the backend

why?

oh, no i have a JWT in local storage

i know they’re logged in straight away

when they load the page it establishes a websocket and fresh state that is relevant comes from the server

based on the params from bidi

interesting

everything server-y is just datoms from datascript that i store as-is

or can be recalculated with little overhead

anyway, refreshing the page on logout is not the worst idea

it might be the most straightforward way to deal with auth0 at least

do you have the server-y part abstracted to the point where you could make a demo of it?

sounds interesting

uuum, it’s pretty simple, i honestly don’t know how far i’ll get before needing a refactor 😛

but i can show a few snippets

hang on

ok cool, i need to get ready for work but i'll be back in a bit

(defn ratom->datom
  "Convert a datom from rethink (not a real datom) to a real datom"
  [r]
  (d/datom  (:e r)
            (keyword (:a r))
            (reader/read-string (:v r))))

(defn datom->ratom
  "Convert a single datom to a hash that is rethink/sente friendly"
  [d]
  { :e (:e d)
    :a (:a d)
    :v (prn-str (:v d))
    :tx (:tx d)
    :added (:added d)})

(defn tx-report:wire!
  "Push datoms from a tx-report down the wire as ratoms"
  ([event tx-report] (tx-report:wire! event tx-report {}))
  ([event tx-report data]
   (sente.wire/send!  {:event event
                       :data (merge {:tx-data (->> tx-report :tx-data (map datom->ratom))}
                                    data)
                       :spinny :saving
                       :success (fn [r] r)})))

i just scrape datoms being added removed straight from tx reports on datascript

(defn tx-data->datoms:persist!
  "Persist datoms from transaction data into rethinkdb"
  [conn owner table data]
  {:pre [(:tx-data data) (:project-id data) (:stamp data)]}
  (->
    (r/branch (-> (r/table :projects)
                  (r/get (:project-id data))
                  (r/get-field :owner)
                  (r/eq owner))
              (when-let [datoms (tx-data->datoms (:tx-data data))]
                (-> datoms
                    (r/for-each
                      (r/fn
                        [datom]
                        ; Use the time that data hits the web server, not the database, as
                        ; they may be on opposite sides of the world.
                        (r/insert (r/table table) (r/merge  { :stamp (:stamp data)
                                                              :project-id (:project-id data)}
                                                            datom))))))
              nil)
    (r/run conn)))

i check the validity of owner from the jwt before that point so it should not be forgeable

there’s an index on the stamp and tx id, so i just send back the freshest datoms that are relevant for the datascript db that is needed

long term i’m thinking i might move to a model where you have read only db/state snapshots (which i could get from datascript with a prn-str) and a read/write realtime collaboration thing that doesn’t hit the server at all and peers just send datoms to each other

¯\(ツ)/¯

@micha reload on logout idea works great, thanks for the help 🙂

a nice pragmatic one-liner

i can probably delete some other code too now

yup, just deleted 20 LOC and have more peace of mind too 🙂

So, I have this pattern I’ve started using in my code, but I’m wondering if I’m missing some inbuilt capability or otherwise doing something wrong. What I have is several places in my application where I have to render a group of controls against elements in a collection. That is, I’ve got a collection of compound things, and each one gets its own set of sliders, text boxes, etc. I started out just creating a cell which would react to the entire collection. Something like this:

(formula-of [coll]
  (for [item coll] (div …)))

The obvious problem here is that any time anything at all changes about the collection (say, I update :foo of the third item) it re-renders the UI for the entire collection. While that’s kind of okay a lot of the time, it definitely isn’t always okay.

I tried using for-tpl, as this seems to be what it’s for, but I ran into trouble. Unfortunately it’s far enough in the past now that I can’t quite remember what the issue was. I believe it was something along the lines of problems with the “reuse DOM elements” aspect. Like, I would delete an item, add a new one in, and the new one would not look like a fresh item, but would look like the old, deleted one.

Forgot to mention that adding items to and deleting items from the collection is a key operation.

So what I wound up doing is something more like this:

(let [indexes (-> coll count range cell=)]
  (formula-of [indexes]
     (for [index indexes
             :let item-c (cell= (get @coll index)]
        (div …)))))

So, basically, a cell that reacts to the size of the collection, and inside it, UI that reacts to changes in the items themselves.

But that deref there looks weird, and the fact I’m rolling this myself makes me think I’m totally missing something.

so there is a “gotcha” with tpl that you may have run into

like, for-tpl should be fine

but you have to set your data that items are based off to the empty state before deleting it

Ew.

not sure if that is relevant

Well, it moves policy about what new items should look like into the delete code, doesn’t it?

well actually, that’s an oversimplification

it’s more...

because cell value propagation is based on value equality

if the old and new cells are the same value

you can get weird behaviour

Right, I could see that. My above approach has similar issues.

I.e. imagine you do an atomic operation to swap the order of two items.

Actually, that might still work.

so in my case the simple thing to do (this was for form inputs) was just set an empty string before delete

because the empty string won’t accidentally match anything someone typed

Yeah, I have a bunch of potentially complex nested structures as the items of my collection. Some of them are e.g. booleans.

But I guess a sentinel like ::none could always be used.

Still, gross.

yeah, well the thing is that items can be complex too

so you can use like a map or something

Yes, now imagine that I want to introduce something like spec. Boom!

the equality thing is definitely a gotcha though

But I’m not doing that, so that’s an invented problem.

like, entities in datascript really got me thinking

because two entities are the same if their entity id is the same

regardless of the values

so nothing would ever update 😛

Huh? Datascript mandates that?

Ick.

Maybe I misunderstood, though.

i’m not sure what your problem with for-tpl was and it’s tough if you can’t remember

but pretty much every time i’ve ever had issues with “stale” items

it was because i didn’t realise an equality check was doing something unexpected somewhere

OK, thanks for that perspective. Alternative formulation: what’s wrong with what I’m doing, if anything?

i think i missed why you have to deref coll?

other than that, if you’re adding and removing items from the dom in a way that means all the removed items just pile up in memory somewhere, that’s not good

(cell= (for [x xs] (div x))) and anything that works like that will have this problem

The deref was a red herring - sorry.

i mean, deref is fine, there’s nothing bad about it

it’s just unusual to need to do it in cell=

i’d be more worried about potential gc issues if you are trying to re-invent for-tpl

some of the other guys in this chat can give more fine grained details about the specifics of that

Yeah, in my case, the places I’m using it are not going to result in a large amount of garbage, since additions/deletions are user-driven and infrequent, but it’s a fair point.

oh well i wouldn’t stress then

I should go back and try for-tpl is my main takeaway.

at the end of the day, just profile it with a realistic scenario

and see if it’s a big deal

if there’s an actual problem with for-tpl it would be really good to see an issue in github 🙂

Actually, my app is already complex enough that I’m having a hard time understanding the profile info. Partly because I’m not a browser guy.

And partly because it does a boatload of math.

http://weathergen.s3-website-us-east-1.amazonaws.com/v026/

do you use the flame chart profiler?

I have seen it. Have not yet spent the time to understand what it’s telling me.

ah, well it’s like, as functions are getting called

it goes further down

Yes, the stack is…a bit deep. 🙂

If you have a second I’d love your analysis. Just fire up that page and hit “step forward” one time.

The math takes place on four web workers, so the main thread is all UI.

oh um

yeah i did that

it’s all minified

Oh, right.

If you’re willing, I can post a non-minified version.

well the shape will be about the same

see, b has lots of sub things

but it’s probably low level

this anonymous function looks a bit suss though

as do these three down here

this is pretty much how to use the chart 🙂

Standard profiler stuff in other words. I’ve already been doing that with the other views, which are more similar to other profilers I’ve used.

Nothing obvious is jumping out at me.

@jumblerg I'm having a dumb moment - trying out hoplon/ui with the Panoply project and for-tpl complains:

(elem :sh (r 1 1) :bt 6 :a :mid
      (for-tpl [event events]
        (div event)))))

@thedavidmeister Thanks for the nudge on that, though: now that I’m looking at it again the chart is making way more sense. And I like the visualization.

@candera could you split things out a bit? like, do the hardcore math and updating the UI in separate steps

a little timeout here and there can really help the browser feel more responsive

also if this is deterministic, memoize?

@jumblerg sorry for the noise - event wasn't a Node 😞 dumb moment resolved

@dm3 i do that exact thing all the time 😛

@dm3, @thedavidmeister: no worries. same.

also, we should really put a disclaimer on that thing too, panoply is very much a wip, and not an example of best practices... yet.

@thedavidmeister Possibly could break up the rendering into chunks, although the degree of asynchrony is fairly high already - web workers handle the computation. If I split the UI updates up, it means “tearing”.

I have thought about memoization. There’s less opportunity for that in the front end.

The flame chart has pointed out that str is a surprising portion of the rendering time, though. Still down in the 5-10% range, though.

yeah, i mean, that update grid function just looks slow @candera

It’s modifying something like 10,000 SVG elements.

@dm3: incidentally, if your goal is to render a list of things with some space between them, you might look at the gutter :g attribute instead.

Maybe twice that.

hmmm

@jumblerg a gutter on what?

would it be faster to make a single svg?

And, for that reason, doesn’t use cells. That’s all JS interop against the DOM.

Sorry, I meant 10,000 elements of an SVG. Like <path>s and whatnot.

well that’s definitely what is slow here, based on what you posted above

@dm3: maybe

(elem :sh (r 1 1) :p 6 :gv 6
  (for-tpl [{:keys [name]} events]
    (elem name)))))

Yes. 🙂

i do wonder if there’s a more performant way to handle the svg stuff

@candera are you doing your math in web workers?

like fewer, more complex paths or something

i learned recently that web workers have special support for matrix math

you should also avoid the lower-level hoplon element ctors, they will break things because the parent, which is responsible for all positioning, cannot manage them.

you can pass a typed array back and forth without copying

@jumblerg thanks! what if I want to lay elements out vertically?

layout is specified through a) size :s and b) position :p :g :a. the elem is responsible for its own size, but knows nothing of its position. the parent alone determines the positioning of children.

@alandipert Yes. Math is in web workers.

oh

:sh (r 1 1) on the child elem

(elem :sh 400 :p 6 :gv 6
  (for-tpl [{:keys [name]} events]
    (elem :sh (r 1 1) name)))))

yep!

@alandipert Yeah, I’m aware of the direct copy stuff. I need to take advantage of that. The transit encode/decode isn’t too bad, but zero would be better.

the current version of ui, btw, still has vertical positioning issues. i have this solved, but the solution still lives on my local machine and needs some serious refactoring and merging before we can integrate it with master.

sounds like its not your current bottleneck anyway

curious tho, is the math parallelizable? they have special things for web workers pools also

Well, I think the program is sophisticated enough at this point there’s no single bottleneck. Decoding is 5%, different bits of rendering are 10% each, etc. etc.

one demo i saw was convulution kernel w/ 4 web workers in parallel

The real win would be run a) the rendering, and b) the math on the GPU via WebGL.

oh yeah

The math here is trivially parallelizable due to the algorithmic choices I made. However, the wins stop around 4-6 workers on my machine.

But in theory I could use 3600 web workers, one for each grid cell. 🙂

WebGL would be a total rewrite.

And a move away from Clojure for much of it.

@candera any chance of using fewer paths that are each more complex?

i imagine there would be some overhead of each path element

@thedavidmeister Yes, sorry - saw that suggestion go by. It’s interesting. So the thing here is those little hockey-stick looking things are wind vectors. There are only about five of them. I reuse them via SVG <use>, and just change the rotation vector.

But I suppose I could render a path that would encompass several of them. I suspect it would not be a win, and really suspect it would not be a big win, but I can’t prove that.

On a fast machine, I should add, the responsiveness is reasonable. And high frame rates aren’t really a requirement.

well you could pretty easily whip up some arbitrary svgs

Rendering is about 180ms on my machine. Slow but not super painful.

one using your current technique and one using a big path string

and see if it helps, without actually refactoring your main app

String assembly turns out to be slow, BTW.

Performance! A pain the butt since the dawn of computing! 🙂

yeah

ruins all the fun things

Of course, the psychotic among us (I count myself) also find it amusing sometimes. 🙂

Believe me, I have seriously contemplated writing the WebGL code, since it would absolutely destroy any monkeying around in ClojureScript I could do.

But man it would suck to move away from Clojure, as I have other things in mind for the code where it would be nice to run it on the JVM, e.g.

@jumblerg I loved reading about hoplon/ui, but writing it is pure joy! No freaking CSS!

the CSS indoctrination for apps must end!

ui is incredible

css, you're banished from the land!

BEGONE

glad to hear you're having a good experience with it!

i'm pretty psyched about the possibilities after we let the last few layout quirks resolved on master.

it just does what you think it will, which is never the case with CSS

unless you know all of it

yeah... the trick was drawing out that box model - coming up with a general thing that works in all cases, instead of a specialized layout and positioning scheme for every corner.

the new version will eliminate most of the extra elements though, and only use them when they're necessary. it has a new box model, basically, but one that makes heavy use of calc fns and transformation matrices.

it's great to hear feedback like that. sometimes you lose perspective when you're working on a project for a long time: it becomes hard to tell if it is something that works only for you because you know it well or whether it works for other people too.

one thing that WILL throw people off from the beginning are the short names for all the options

like :s, :p, etc

would be good to have the long versions too

you're not the first person to suggest this.

i'm wary of having hoplon ui codebases where some ppl use the long form keys and others the short, but maybe i need to revisit this.

maybe you can only have one or the other? maybe you have 2 versions of the lib

there's something about the concise syntax that i think makes the layout easier to digest.

i guess that defeats libraries

we can support both trivially

it's great to have

but it's like a power tool

kinda vimlike, sure.

e.g. unix shell utils

has all the long versions and short ones

hm, yeah, you have a point here i guess.

to play devil's advocate, there are only few things to remember once you get the pattern.

Have you considered using namespaced keywords for the long form, perhaps?

@candera: yeah.

It might be interesting to make those normative and then allow some sort of aliasing.

Perhaps with canonical aliases.

i actually had something like this implemented a while ago, then removed it based on a "do things well one way" heuristic.

micha has actually worked something into hoplon core for this

that makes the "keys" real functions

Yeah, there’s a lot to be said for that, especially when something is going to be pervasive. It’s one thing to have :g somewhere in your code, used exactly once. UI looks like it’s going to be something where you have to buy in to make use of it, so the conventions are a one-time cost amortized over much usage.

yeah, exactly

on the other hand, if we replaced the keywords with real functions

we could have (ns (:require [hoplon.ui.size :refer [s sh sv]])) (elem sh (r 11) ...) etc.

if the attributes were split up into granular namespaces, the namespaces might provide more context.

and people could use the qualified namespaces if they wanted

(elem hoplon.ui.size/sh (r 1 1) ..) i dunno.

don't think that approach is necessarily ideal, but lots of possible permutations here.

that's a pretty great api imo

at least, i see no immediate downsides. esp. pretty because people can use the var-renaming machinery in ns to write things however they want

yeah

micha's idea there.

and i think the context is another benefit of using functions with actual namespaces instead of keywords.

keywords, an epicycle?

offtopic: what's the best way to pprint a js object and show it verbatim?

i seem to find a new reason to hate them about once a year

better increment the badge lol.

@dm3 i do (.log js/console <obj>) and chrome prints it all special

yeah, I want to show it to the user

(.log js/console ...)?

ah

hm. what sort of object is user-facing?

https://google.github.io/closure-library/api/goog.format.JsonPrettyPrinter.html

i don't know how it handles non-json

there's JSON/stringify.

hm, it should be just data so I might get out with pr-str and Fipp

what's the catch with scrollable elements?

how do I make the whole window scrollable?

json.stringify does pretty printing

JSON.stringify(thingy, null, 2)

for example

dies with circular references etc of course

@jumblerg first thing I ran into that isn't obvious - how do I get the "default" browser behaviour of having the whole page scrollable? I have the same top-level structure as the Panoply project

(window (list (header) (body)))

:scroll true

scrolling is one of those areas that gets messy on the current master, due to a browser thing called stacking contexts. it is special cased on window.

basically, if you set overflow on an element, regardless of whether it is overflow x or overflow y, it changes the stacking context of that element.

so changes to x affect y and vice versa. 😕

also, you only get the elastic scroll in chrome on body, but not the other elements, which is a bit annoying.

I see

ok, now I have this

(elem 
      (let [caused (cell= (:event/caused-by event))]
       (when-tpl (cell= (boolean (seq caused)))
         (elem
           (for-tpl [{:keys [data timestamp]} caused]
             (elem
               (cell= (as-timestamp timestamp))
               (edn data))))))))

is there some interplay with when-tpl that I'm missing?

when I remove when-tpl and do a :toggle with the same condition - everything is fine

ah no, :toggle just displays it all the time

the *-tpls should behave the same way they do in hoplon.

an elem is just a fancy supernode; hoplon.core can't tell the difference.

it sees it the same way as a div.

figured that 🙂 yeah, will figure out why it doesn't work

i don't see anything obvious, but try eliminating the first child/cell in the innermost elem.

there is one bug on one of the many branches, maybe the current master you're using, that doesn't properly handle cells/*-tpls as siblings.

i'm going to straighten all this this stuff out over the holidays.

now it works, how strange...

@micha I always forget - what was the solution to resetting a cell from inside a cell=?

My solution has been: don’t use cell= 🙂

(defmacro formula-of
  "Emits a form that will produce a cell using the formula over the
  specified input cells. Avoids some of the code-walking problems of
  the Hoplon macros. Cells can be either a vector, in which case the
  cells will be re-bound to their values under the same names within
  `body`, or a map whose keys are binding forms and whose values are
  the cells to bind.

  E.g.
  (formula-of [x y z] (+ x y z))

  (formula-of
    {x-val x-cell
     {:keys [a b]} y-cell}
    (+ x-val a b))"
  [cells & body]
  (if (map? cells)
    `((javelin.core/formula
       (fn ~(-> cells keys vec)
         [email protected]))
      [email protected](vals cells))
    `((javelin.core/formula
       (fn ~cells
         [email protected]))
      [email protected])))

Then:

(formula-of {foo-val foo-cell} (reset! foo-cell (f foo-val)))

That’s a stupid usage example, but shows the form @dm3.

Anyway, Micha probably has a better answer, but that’s what one Hoplon n00b (me) has done.

(cell= (when foo (~(partial swap! bar) inc)))

yeah, right. Thx!

I did a do-watch

if anyone has time over the holidays, it would be great we could investigate the impact of using the async/defer attributes on the script tag. https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script

micha mentioned that he tried them a long time ago, and that there were browser-implementation issues associated with using them at the time. it would be worth investigating again.

since most of the perceived latency associated with using a hoplon app is a consequence of parsing the initial javascript, if we could start the process of parsing and even evaluating code while index.html.js is still streaming from the server, we might see a significant improvement in performance.

Y’all, I’ve been really digging following all the discussions here over the past few months. I’m not actively developing in Hoplon/CLJS currently, but if plans go well, I will be in the not too distant future. Lots of great developments, not the least of which is UI.

Hi @jamieorc. Long time, no see. 🙂

hey @candera!

Yeah, I’ve been lurking on here for months

and I really enjoyed your interviews with @micha and @alandipert, @candera

Yeah, those were fun ones. Of course, I have to say that now that I’m their work-slave.