Fork me on GitHub
#hoplon
<
2015-09-27
>
onetom03:09:48

@mynomoto: i dont have a strong opinion on IRC. i never liked it because of the hassle with the history and bouncers and logger bots, but one undebatable upside is that the protocol is free and the freenode network is independent from political or commercial influence... (at least thats how i imagine it)

mynomoto04:09:47

@onetom: I think it's more of a practical matter than a political one. There are more people and conversations here than there and having one "official" place is better than be scattered over many.

mynomoto04:09:55

I don't want people to leave irc, only not sending new people there if there is more talk going on here.

mynomoto04:09:49

I like gitter better but I know a lost battle 😉

onetom05:09:36

btw, i actually have gitter running too; primarily to follow the https://github.com/red/red project...

onetom05:09:05

I think u just found the critical word here: official

onetom05:09:34

Maybe we should just say something along those lines, like: Most of the community is on [Slack], so that's the most active place for discussions and the primary place to get Hoplon support. There are alternative/fallback channels too: [Gitter] & [IRC].

colin.yates09:09:35

hi all - is this or Discourse the right place to ask questions?

colin.yates09:09:29

I have a bunch of newbie/clarity questions which I posted to http://hoplon.discoursehosting.net/t/newbie-clarity-questions-around-cells/552 - in general is it sufficient to post to discourse?

voytech09:09:53

Hi. I was recently trying to test some castra calls using ring mock. It took me a little time to get it working, and this is because castra frontend pass some castra specific header like transport method and csrf. I am now wondering if there should be support for mock castra requests for testing on backend ? So that it is not necessery e.g. for newbie to look into castra code to be able to construct request. Or maybe it should be sufficient to only do clojurescript tests ?

micha13:09:55

hi colin.yates and voytech!

colin.yates13:09:05

@micha - the man himself simple_smile

micha13:09:14

how's it going?

colin.yates13:09:02

I am still googling around different use-cases - I posted a few newbie Qs at discourse which would be great if you could take a look at (you know, your time being mine to spend :-)).

micha13:09:14

haha sure

colin.yates13:09:27

but yeah, I think javelin is exactly the right model for how I view the model creation (as a sequence of data transformations)

micha13:09:39

i've been inconsistent with reading discourse

micha13:09:49

job deadlines and whatnot

micha13:09:58

one sec let me get caught up

micha13:09:25

ok so for your first question

micha13:09:54

i can just sketch out the answers here and then post to discourse later, is that cool?

colin.yates13:09:03

that would be great

micha13:09:25

so the way javelin works is each cell keeps references to its "sinks" and "sources"

micha13:09:40

sinks are other cells that reference it (formula cells or lenses)

micha13:09:54

and sources are cells it references (it's a formula cell or lens)

micha13:09:24

changes happen as a result of swap! or reset! on an input cell

micha13:09:37

or possibly a transaction involving multiple input cells

micha13:09:10

each input cell (or lens) has a watcher on it that calls the "propagator"

micha13:09:27

the propagator is a function that given a cell where the change originates, computes the dependency graph from the sources and sinks of each cell, walks the graph doing a sort of topological sorting of the cells in the graph

micha13:09:50

such that no cell comes before the cells it depends on in the sequence of cells

micha13:09:15

then the propagator propagates the change to each cell in turn

micha13:09:52

the change is computed by calling a thunk with the sources as arguments, dereferencing any sources that are cells

micha13:09:45

if the result of recomputing the thunk is the same as the existing value in the formula, then propagation is skipped for all of the cells that depend on this cell

micha13:09:01

(unless they depend on another cell that has changed)

micha13:09:47

the dependency ordering info is part of the cell, computed when the cell is created

micha13:09:32

basically cell rank is a monotonically increasing sequence

micha13:09:52

when you make a new cell it will have a rank that is higher than any previously created cell's rank

micha13:09:18

and when it's time to propagate we walk the sinks and build a priority queue with the rank as the priority

micha13:09:01

so to answer your question: dereferencing a cell is free

micha13:09:10

it's an eager system with caching

colin.yates13:09:11

(and great explanation!)

micha13:09:23

swapping, however is not free

micha13:09:36

because you may be updating things you don't care about

micha13:09:50

although that has never been a problem for any of my applications so far

micha13:09:26

we use cells as the plumbing really

micha13:09:40

cell formulas are always small, one liners

micha13:09:01

if the formula starts to get complex it's usually more productive to move the logic into a function

micha13:09:08

and call the function from the formula

micha13:09:33

because usually complex formulas indicates some other absrtaction you need, which you can't really do very well in formulas

micha13:09:36

so the "debugging cell=" question

micha13:09:44

consider:

micha13:09:56

this will evaluate the formula's thunk every time clicks changes

micha13:09:31

because the pruning happens on the sources, not the value of the formula itself

micha13:09:48

it can't know what the value of the formula will be without evaluating the thunk, right?

micha13:09:41

the value of the formula could involve non-pure functions, too, like (cell= (or (even? clicks) (rand-int clicks)))

micha13:09:07

so one weird trick you can do

micha13:09:28

disregard that

micha13:09:24

ok there we go

colin.yates13:09:39

got it - I see

micha13:09:24

so the REPL should work, i don't see why not

micha13:09:54

but remember that the REPL and clojure itself have a lot of hacks built in around reference types

micha13:09:04

like how to dynamically redefine them etc

micha13:09:41

and cljs doesn't have real vars yet, which are the foundation of all reference types, the most primitive one (because of the relationship between vars and evaluation)

micha13:09:57

so javelin cells keep their own sources and sinks references

micha13:09:12

instead of redefining them you probably want to mutate them in place

micha13:09:20

so the references are kept intact

micha13:09:55

javelin.core has functions like set-formula!, destroy-cell! etc

micha13:09:15

you can use these from the REPL to modify cells without disturbing the cell graph

colin.yates13:09:10

so if I change the defn that a cell= delegates to I can just use (set-formular! ..) on that cell?

colin.yates13:09:08

can I ask a few more Qs then about cells?

micha13:09:41

oh sorry, it's set-cell!= that you want

micha13:09:43

the macro

micha13:09:57

set-formula! is a lower-level function that does the same thing

micha13:09:05

but the macro provides some sugaring

micha13:09:23

sure fire away simple_smile

colin.yates13:09:12

thanks simple_smile. So in terms of garbage collection if I have a (defelem top-panel-page-1) and (defelem top-panel-page-2..) and in those I create various (cell= (transform some-top-level-cell)) inside each elem...

colin.yates13:09:28

.. if I show page-1 then those cells and formula get inserted into the graph. If I then show page-2 (and I am (condp = page :page 1 (top-panel-page-1) :page 2 (top-level-page-2)) then I assume all those cells created in page-1 are removed as the page-1 dom tree is removed?

micha13:09:45

actually no

colin.yates13:09:51

If I then go back to page-1 is assume those page-1 cells are recreated?

micha14:09:04

that's the react model more or less

micha14:09:12

we tend to leverage static allocation

micha14:09:20

and allocation pools

micha14:09:42

generally in an application you have a relatively small number of "screens"

micha14:09:53

or rather i should say screen templates

micha14:09:58

layouts kind of

micha14:09:13

basically places where varying data will be displayed to the user

micha14:09:40

we statically allocate those wherever possible and just hide them when they're not in use

micha14:09:55

for sequential things, like table rows for example

micha14:09:05

we allocate them into a pool

micha14:09:23

when a row is removed from the dom we don't try to deallocate it

micha14:09:35

instead we move it back into the pool

micha14:09:47

when we need one later we get it from there instead of creating a new one

micha14:09:03

this approach allows us to greatly simplify the whole model

micha14:09:17

because you only need to write the constructor

micha14:09:26

you don't need any lifecycle protocols

micha14:09:53

without weakmaps in js i don't know if there is an automatic way to handle garbage collection

micha14:09:16

the existing weakmap polyfills etc are not sufficient

micha14:09:27

really we need weak references

micha14:09:44

there are some ideas we've been thinking about

colin.yates14:09:57

right - I don’t see what happens to cells that were created in the deallocated template rows though?

micha14:09:05

nothing gets deallocated

colin.yates14:09:18

sorry, I mean the ‘inactive pool'

micha14:09:26

oh it's like the um

micha14:09:33

the hotel guest example

micha14:09:41

where you have a motel with 100 rooms say

micha14:09:49

and maids are assigned to rooms

micha14:09:58

different guests move into and out of the rooms

micha14:09:05

but the maids remain the same

micha14:09:18

the cells are the maids

micha14:09:21

so for example

micha14:09:42

loop-tpl is the pool allocator

micha14:09:59

the body of the loop-tpl is the constructor

micha14:09:20

initially that will be called 3 times, and you'll have an unordered list with 3 things in it

micha14:09:48

and the pool will be empty

micha14:09:04

now suppose you do (swap! data conj 4)

micha14:09:34

since the pool is empty it must allocate another list item

micha14:09:36

so it does that

micha14:09:43

now the list has 4 items in it

micha14:09:59

now suppose you do (swap! data pop)

micha14:09:31

now the data cell will contain [1 2 3] again

micha14:09:47

and the 4th list item is removed from the dom and stored in the pool

micha14:09:02

but what happens to the d reference in the 4th list item?

micha14:09:26

well it's bound to (cell= (guard (nth data 3))) under the hood

micha14:09:47

the guard is there because nth will throw on invalid indexes

micha14:09:59

so d will be nil for the 4th list item

micha14:09:13

note that d, the maid, has not changed rooms

micha14:09:24

it's still looking at the 4th item of data

micha14:09:40

but now the 4th item of data is nil, because data has no 4th items

micha14:09:53

so now it's an empty li element

micha14:09:06

now suppose you do (swap! data conj 42)

micha14:09:25

the loop-tpl sees it has things in the pool, so it doesn't create any new thing

micha14:09:35

it just puts the 4th item back in the dom

micha14:09:59

and the 4th item is now a li with "42" as its text content

micha14:09:10

becuase d also automatically updated itself

micha14:09:16

does that make sense?

colin.yates14:09:12

I think so. Can I sanity check an example closer to my intention?:

colin.yates14:09:17

(defc people {1 “Bob” 2 “Jo”})
(defc locations {1 “Home” 2 “Work”})

(defn denormalise-people [i] (cell= (get people i)))
(defn denormalise-locations [i] (cell= (get locations i)))


(defc records [{:person 1 :location 1} {:person 2: location 1}])

(defn denormalise-record [{:keys [person location] :as record]
  (tr 
    (td (denormalise-people person))
    (td (denormalise-locations location))))

(defn render-table [records]
  (table
    (tbody
      (loop-tcl :bindings [record records]
        (denormalise-record record))))

colin.yates14:09:10

where any of the defc’s might change (and will themselves be formula cells on a stream of incoming domain events)

colin.yates14:09:25

(again it is made up, but representative of my intentions)

micha14:09:33

this looks legit to me

micha14:09:58

this will allocate as many things as the maximum number of items in records

micha14:09:08

and it will never deallocate them

micha14:09:34

so you'd paginate records or something

micha14:09:55

basically records represents the window of things the user is seeing at one time

micha14:09:05

which is constrained by the weak human brain

colin.yates14:09:35

right, we allow the user to chose the number (10, 20, 50, 100, etc.)

micha14:09:52

we will usually make a state machine for this

colin.yates14:09:02

but because it is ordinally indexed, going from 100 to 10 is fine because the 90 ‘orphans’ in the pool will be nil

micha14:09:06

so we can hook the state transition functions for the machine to event handlers

colin.yates14:09:50

changing the ‘window’ will fire an event over a web-socket to the server to return more results which will then (reset! records …) in this example

micha14:09:58

the cells will still exist, but the formula values will be nil

micha14:09:30

for the ones that are removed from the dom when you decrease the number of items in records, that is

micha14:09:47

for those guys the value of the record cell will be nil

colin.yates14:09:09

This is just great - thanks Micha.

colin.yates14:09:57

I need to run unfortunately but my last assumption is that because of the mechanics of the cells, changing something like the name of a ‘person’ will result in a trivial amount of recalculations and then hoplon will simply update a ‘td’ for example

micha14:09:03

actually there is a slight issue with your code

colin.yates14:09:33

there usually is simple_smile

micha14:09:41

so denormalize-record is a function that returns a dom template more or less

micha14:09:53

you pass it a cell containing the values that will change over time

micha14:09:07

but the cell itself is like an atom or a var

micha14:09:17

you can't dereference it directly

micha14:09:27

we provide cell-let to do that

micha14:09:47

that will destructure the record cell to give you cells for the location and person

micha14:09:11

you can then make formula cells for those which you can bind to the text content of the td elements

micha14:09:32

notice here that cells are only allocated once per item

micha14:09:47

the body of the loop-tpl only runs once per item

micha14:09:58

which is good

colin.yates14:09:37

why does line 13 and 14 need to wrap the result of (denormalise…) in a cell=? as the denormalise- fns return a cell=?

micha14:09:48

oh my bad

micha14:09:57

good catch simple_smile

onetom14:09:19

micha, u have just written up a great "how hoplon & javelin works" tutorial. can we just chuck it into the hoplon wiki until we have a more distilled version?

micha14:09:41

that would be awesome simple_smile

colin.yates14:09:55

just to be very explicit, denormalise-person may actually be really expensive which is why I want to ensure it is only run once - I think we are on the same page and I am looking forward to this.

onetom14:09:14

colin.yates: do u mind if i leave ur name in there too (just so i dont have to edit it too much)?

colin.yates14:09:22

not at all - go for it

micha14:09:23

i would recommend going easy on the functions that return cells

micha14:09:54

i find it tends to work better to let functions return values and build cells around them anonymously in place

micha14:09:08

because cells compose in a different way than functions do

micha14:09:31

basically i try to keep the javelin layer at the very top of my ladder of abstractions

micha14:09:54

that layer can handle the caching for you automatically

colin.yates14:09:38

in my use-case though I would want to access the result of the very expensive calculation that (denormalise-person) does everywhere … and make sure that two calls to denormalise-person for the same person get cached…

colin.yates14:09:50

I don’t think I understood what you are proposing?

micha14:09:08

ah i see now

colin.yates14:09:14

or are you saying that denormalise-person can be very expensive and consume a cell, but where-ever I call it I should wrap the call in a (cell= (denormalise-person 1)) relying on multiple (cell= (denormalise-person 1))s?

micha14:09:18

yes it makes sense what you did there

colin.yates14:09:37

ah OK - phew simple_smile

colin.yates14:09:52

I really have to run - thanks a bunch micha - I will be back later and may ping you then if you are on-line. My manic-job-of-the-afternoon is to re-write my current re-frame app which uses immutant and web-sockets and prototype with hoplon, so many questions will undoubtedly arise simple_smile. May as well throw boot in instead of lein for fun.

micha14:09:39

awesome i'm interested to hear your impression/criticisms

onetom14:09:16

micha: i was wondering about loop-tpl today... is there a reason why is it expecting just 1 body expression: https://github.com/hoplon/hoplon/blob/master/src/hoplon/core.clj#L158

onetom14:09:51

similarly im not sure why would the bindings should be constrained to just 1 pair of destructuring form and sequence?

micha14:09:01

the body i think should be wrapped in an implicit do

micha14:09:11

but the bindings get more tricky

micha14:09:23

i did have a branch that worked like doseq/for

onetom14:09:29

i was thinking more like wrapping it into (spliced ...)

micha14:09:34

that allowed multiple bindings

micha14:09:47

well you can always do that by just wrapping with []

micha14:09:56

but you might want to perform side effects

micha14:09:15

so an implicit do seems like the right way to go, no?

onetom14:09:52

in my experience to my colleagues it was more intuitive to expect the ability to throw in multiple dom elements into the body and have them repeated...

onetom14:09:19

i can remember a case when we only wanted side effects

onetom14:09:22

if we wanted some debugging, we just wrap the (dom-elem ...) as (let [_ (print ...)] (dom-elem ...))

onetom14:09:17

which leads back to the multiple binding question... if it would be possible to have multiple bindings then side-effects could be just thrown in there

micha14:09:41

yeah it can be done, i did it at one point

micha14:09:55

but there were some inconsistencies in the way you use it that i didn't like

onetom14:09:57

u have a substantial code base, so maybe u can do a quick grep to see how often r u doing side effecting from withing a loop-tpl

micha14:09:07

i never do

micha14:09:25

i do wrap things in vectors to splice them in quite a few places though

micha14:09:45

but it would be non-idiomatic i think in terms of general clojure macro style

micha14:09:18

not that we need to slavishly enforce clojure idiom

onetom14:09:24

making splicing automatic would decrease indentation level though...

micha14:09:50

ok yeah i'm for it

onetom14:09:18

loop-tpl already expresses a level of grouping anyway...

onetom14:09:30

okay, i tell u why these questions came up

onetom14:09:58

i was trying to create a generic menu and sub-menu builder and it became a monster quite quickly

onetom14:09:27

it doesnt fully work at the moment but it looks something like this:

onetom14:09:41

so as u can see i had to throw in an extra let which could have just gone under :bindings

onetom14:09:50

and some earlier versions had extra stuff next to the menu-item too

onetom15:09:17

the aggressive indentation in the sublime lispindent plugin prompted me to think about how to be a bit less verbose and that's why i looked into the loop-tpl implementation

onetom15:09:49

as u can see i even broke the :bindings attributes into a separate line to save a bit on the indentation

onetom15:09:40

so my 3rd question/idea is: do we really need this :bindings attribute and such an eccentric name as loop-tpl?

onetom15:09:30

Since im not expecting to have such an invasive change, I was thinking how can I have such a custom (defmacro x ... loop-tpl* ...) automatically compiled into every .hl file? I haven't found any extension hook in boot-hoplon at 1st glance....

onetom15:09:56

btw, i came up w the name x because it symbolizes multiplication... loop is already taken stamp is quite short but suggest a very imperative approach foreach? every? ... something-let or let-something?

onetom15:09:10

i know the (loop-tpl :bindings ...) was invented to blend in to the rest of the dom element constructors but it's so long an unorthodox, it's just not very practical...

onetom15:09:07

now that hoplon v6 is about to come out, it think it makes sense to add such practical polish to it before it gets announced to a broader audience...

onetom15:09:41

just we just get rid of the :on-*s and :do-*s...

micha15:09:55

we were thinking to rename loop-tpl to just splice

micha15:09:06

(splice :for [x xs] ...)

micha15:09:45

that would work well with the implicit splice, too

micha15:09:15

i'm totally fine with extending the loop-tpl bindings

micha15:09:40

i think it needs a little design work though to be intuituve and regular to use

micha15:09:14

the loop-tpl is very much like the clojure unquote-splicing

onetom15:09:10

(i had to step away to attend my crying daughter)

onetom15:09:39

so (splice :for ... is already nicer, though as a non-native speaker i still don't understand really what splice mean, what does it have to do with ropes and still confuse it with slice

onetom15:09:32

imean my 1st exposure to splice was the javascript function: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/splice and it simply makes no fckin sense even today to me...

micha15:09:11

yeah splicing means like interweaving or patching one thing as a section of another thing

micha15:09:15

like for ropes

micha15:09:30

you can splice a new piece to replace a worn section of rope

micha15:09:19

the idea it expresses is that the collection is being altered by adding or removing a contiguous section from the middle somewhere

onetom15:09:55

just to demonstrate my personal (or rather european's) confusion, if i translate back the meanings of splice from hungarian to english, i get the following: twist, interweave, twine, furl, cockle, kink

onetom15:09:20

(besides splice of course)

onetom16:09:36

(insert :every [thing some-list]
        (elem thing))
or alternatively inlay instead of insert, though insert doesn't really collide with anything...

onetom16:09:48

and as i was reading your explanations even pool crossed my mind. of course my own 1st reaction was "it's too low-level and implementation detail revealing/bound" but actually, if i assume i don't know that there is a pooling mechanism behind it, it still works quite well:

(pool :of [thing some-list]
      (elem thing))

onetom16:09:54

very concise and expressive. it doesn't suggest some kind of nesting level imho. doesnt collide with anything built in

onetom16:09:28

also pool reads as loop just backwards ;D

onetom16:09:35

ah, playing w letters and moving along this splicing of ropes/wires, there is the very famous loom https://en.wikipedia.org/wiki/Loom_(video_game) https://upload.wikimedia.org/wikipedia/en/3/3b/Loom.png

onetom16:09:42

(loom :in [thing some-list]
      (elem thing))

onetom16:09:03

but honestly this attribute thing is really unnecessary. it won't return just 1 dom element but a vector, so i don't see a need for making it look like dom elements... it's actually more confusing than explanatory. just the other week i made some element returning a loop-tpl and i tried to call it as a function to extend it with further kids...

(defelem thingies [attrs kids]
    (loop-tpl ...))

(let [default-thingies (thingies :class "blah")]
   (default-thingies :click #(...)))
something like this...

onetom16:09:01

hmmm now the only problem is loom has nothing to do with weaving when it's a verb (as i thought in all my life simple_smile

onetom16:09:52

these would work for me too:

(*** [thing some-list]
     (elem thing))

(... [thing some-list]
     (elem thing))

micha16:09:37

yeah we could do that

micha16:09:58

we didn't do it initially and have resisted it so far because we wanted to be able to support html syntax

micha16:09:17

but i'm not sure if that's something we want to continue supporting forever?

micha16:09:42

i wouldn't want to break it without getting the okay from the community

micha16:09:39

<loop-tpl bindings="{{ [ x xs ] }}">...</loop-tpl>

onetom16:09:16

oh, r u still using hoplon thru html?

micha16:09:27

i'm not but i don't know for sure that nobody is

onetom16:09:41

okay, so i wouldn't bore everyone with this naming if there would be a way to hook in some customization into the hoplon compiler, so some project-wide macros and functions could be required in automatically into every .hl file. is there a way to do that or it's actually better not to do that?

onetom17:09:42

ah, i think i found it. (hoplon :refers #{'my.ns})

colin.yates17:09:56

May I ask a Q about castra?

onetom17:09:20

colin.yates: of course

colin.yates17:09:46

At the moment I use websockets which push a ‘new-state:N’ to each client, the client then immediately asks the server (via a websocket) to get the new state since last seen. The server then pushes all that new state with a :new-state N checksum.

micha17:09:15

@onetom: sorry for the late reply, but you're right there

micha17:09:29

that's the (undocumented) hook you can use

colin.yates17:09:49

I think I can do the same in castra by the following - have the state returned by castra always include the latest checksum and have the client do an rpc call every second and when it pushes anything to the client check the :new-state flag (does that make sense)?

onetom17:09:03

micha: not sure though how to hook macros in, but i think i can figure it out on my own

colin.yates17:09:19

effectively, the state that is synchronised between the server and the front-end is the batch of state last-seen since that client last checked.

micha17:09:27

@onetom: it'll refer the macros too, automatically

onetom17:09:58

thats how i understood the source too but it didnt work on 1st try. but no worries, im on track

onetom17:09:08

java.lang.AssertionError: Assert failed: Can't find macro_test.cljs in classpath

micha17:09:16

i think you may need to have at least one cljs definition

onetom17:09:22

that .cljs is the problem

micha17:09:23

inaddition to the macros

micha17:09:45

it's undocumented because there are some gotchas in there

micha17:09:07

also make sure you don't define anything that conflicts with a previous definition

micha17:09:26

it doesn't do any checking of which things it might cause a name clash with

flyboarder17:09:23

how could I go about creating a new element? not using defelem but something that will print <new-elem> as html?

colin.yates17:09:32

if I can put it more simply - how do I wire up castra to only show the client the most recent stream of events since they last connected

micha17:09:41

@flyboarder: look in hoplon.core namespace

micha17:09:54

there is a function in there like make-elem-ctor or something like that

micha17:09:10

you can use that

flyboarder17:09:21

those are private are they not?

micha17:09:29

basically you can do (.createElement js/document "new-elem")

micha17:09:51

any browser will handle that as a block element

micha17:09:07

you can use css to have it display as an inline element or table cell or whatever you want

micha17:09:20

this is something we might consider for defelem also

micha17:09:25

@colin.yates: you already have an event driven backend?

micha17:09:56

one thing i've wanted to try is to have a sort of hybrid approach

micha17:09:14

the thing i want to avoid is the complexity of the event driven backend

onetom17:09:16

doesnt sound like it fits the castra model which was made for unidirectional data-flow, no?

micha17:09:36

i wanted to have the stateless request-response model in the server

micha17:09:44

but with server-push also via websockets

micha17:09:57

the thing i was thinking of trying is to do polling on the server

micha17:09:06

it would go like this:

micha17:09:15

client connects to websocket

micha17:09:32

websocket is allocated and managed by ring middleware wrapping castra

micha17:09:55

the ws middleware would then begin polling castra

donmullen17:09:08

@colin.yates would be interesting to gather your insights in comparing re-frame+lein w/ hoplon+boot — would make a for great blog post (see that you are working on your blog site with jekyll — you should check out https://github.com/hashobject/perun 😉.

micha17:09:08

sending data to the client whenever the response changes

micha17:09:21

this has a number of interesting properties

micha17:09:31

1. if we use datomic the polling is nearly free

colin.yates17:09:42

@donmullen: yeah - I have so many great things to put on there, but time 😞

micha17:09:59

2. since we're using websockets we can keep track of the last thing we sent to the client, so we can compute diffs and patch on the client

micha17:09:12

eg. we don't need to send the entire response

micha17:09:37

3. our business logic and api is still request-response stateless functions

micha17:09:55

the websocket layer and event stuff is boilerplate in a library

micha17:09:10

does that sound like a relatively sane approach to you?

colin.yates17:09:34

it does, but I think I am almost there already except I don’t want the fully-realised model on the server.

micha17:09:17

so i think with castra if you want to get a response that is "as of" the last time they asked

colin.yates17:09:21

I want the equivalent of /events?from=10 which would return something like {:latest-event 51 :events [10, 11,12 ….51]}

micha17:09:30

you can simply add that as a parameter to your RPC function, no?

micha17:09:37

and have the client keep track of their own state

micha17:09:44

they just provide it with each request

colin.yates17:09:23

so the client passes {:last-seen 10} to the rpc, the server constructs a view which is the latest events which castra syncs to the server - I see.

micha17:09:57

the state can be stored in localstorage on the client perhaps

micha17:09:00

or whatever you want

micha17:09:13

so you don't need sticky sessions or anythign like that on the backend

colin.yates17:09:28

exactly, I want the backend to be stateless really

colin.yates17:09:43

another similar use-case would be tailing a server-side log

onetom17:09:44

micha: normally im quite skeptical about polling, but what u described with those 3 points add up into a pretty strong argument

colin.yates17:09:03

+1 - it really does. It also handles lost clients very well as well

micha17:09:07

the key there is datomic simple_smile

colin.yates17:09:35

you don’t need datomic for this though. My back-end is pure DDD and event-sourcing and it is a perfect fit for this simple_smile

flyboarder17:09:37

@onetom: I agree, generally i try to avoid polling for performance reasons, and less over-the-wire

onetom17:09:42

"several people are typing" simple_smile thats what slacks says below my input field

micha17:09:45

can you describe your stack?

colin.yates17:09:07

All domain transitions are captured in discrete events. To get the latest model you simply (reduce ) the events for that aggregate root

colin.yates17:09:23

the event-store happens to be in MS SQL server but it could be anywhere.

micha17:09:35

oh wow, interesting

colin.yates17:09:55

So all I need to do is stream any events that have happened since the client last talked to the server

micha17:09:57

you compact events from time to time i guess?

onetom17:09:08

colin.yates: DDD as in Domain Driven Design?

micha17:09:11

to stay within bounded db table size?

colin.yates17:09:25

this isn’t a particularly high traffic site (internal LAN)

micha17:09:33

this sounds promising

micha17:09:53

i imagine something like aurora on aws would be suitable for something like this

micha17:09:00

for a more highly loaded application

colin.yates17:09:21

ah, unfortunately for me this is all closed-source and commercial (think medical data)

colin.yates17:09:34

it does mean I get a pretty nicely controlled environment to play with though

micha17:09:59

do you have a caching layer on top of the SQL engine?

colin.yates17:09:01

I am also going slightly off-track by doing all the event denormalisation on the client side

colin.yates17:09:25

which means travelling back in history is a trivial thing - simply discard any events since a particular timestamp.

colin.yates17:09:59

I haven’t needed one yet - MS SQL is pretty quick and the working set tends to be small (e.g. everybody is interested in the last 10K events for example)

colin.yates17:09:20

for searching etc. I denormalise some stuff but those are themselves projections of the event stream

micha17:09:32

man this sounds awesome

colin.yates17:09:44

DDD + CQRS + event streaming makes your brain hurt but yeah, if you can manage to get it working it is brilliant.

colin.yates17:09:06

my current architecture is web-socket (and re-frame) based so clients don’t poll, they just get sent the events in real-time

micha17:09:13

it makes my brain feel better after all the pain of the normal SQL pattern, really

onetom17:09:50

colin.yates: what are you challenges with re-frame?

colin.yates17:09:50

the really nice thing is that your entire domain model can be persistence agnostic - ‘loading’ an entity is really nothing more, and can be nothing more than reducing the events for that aggregate root

colin.yates17:09:05

the main one is the fact that subscriptions aren’t free to dereference.

colin.yates17:09:20

it rocks, it really does, but I made an incorrect assumption about it.

micha17:09:41

caching can make dereferencing a point in time free

colin.yates17:09:50

So you can imagine an event stream of 20K events going into a data-transformation which trims by time going into one which hydrates each entity etc.

colin.yates17:09:00

I had many subscriptions which did that the calculations

colin.yates17:09:32

to be blunt, I tried to implement the spreadsheet model using subscriptions and it worked really nicely, except I noticed many things were being recalculated

colin.yates17:09:50

re-frame is still feasible, I could simple use the app-db as a denormalised cache of sorts

micha17:09:57

i experimented with a spreadsheet implementation inside of the datomic transactor

colin.yates17:09:59

I also ran into some issues, almost certainly mine where things wouldn’t always update when they should - things like that. I am absolutely not claiming they are bugs, more my lack of knowledge

micha17:09:00

i think it could work, but it would require some serious design hammocking

micha17:09:56

the problem, as i see it, is that the client has very little memory and no access to the usual ways we alleviate memory pressure

colin.yates17:09:58

to be honest, I could achieve what I want with a trivial end-point which I poll using cljs-ajax passing in the parameters I am interested in which in general are simply the checksum of the last seen id

colin.yates17:09:20

yes, this is definitely not a general solution for unbounded lists etc.

micha17:09:27

yeah the cljs-ajax approach is how we do it currently, using castra just to handle all the boilerplate and whatnot

colin.yates17:09:36

ah OK. I see.

micha17:09:52

castra also implements the CQRS data flow

micha17:09:05

like calling an RPC function int he client is the command

micha17:09:14

it returns a promise you can use to handle exceptions

micha17:09:20

but it doesn't return state

micha17:09:26

state goes into a javelin cell

micha17:09:41

formulas that reference the javelin cell are the query in the CQRS parlance

micha17:09:20

this "handle errors at the caller, but handle state globally" model has worked very well for us

micha17:09:05

i lied, the promise can be used to handle state too, but we hardly ever do that

colin.yates17:09:21

so I think I can still get this to work with castra by calling the rpc with the ‘last-seen’ sequence-id. The server then loads the (select event_blob from t_events where sequence_id>last_seen) into the cell

micha17:09:32

you totally can, yes

colin.yates17:09:33

thanks again all.

colin.yates17:09:53

(and yes, I really must blog/produce a template project when I get all the dots working)

micha17:09:05

yeah that would be awesome

micha17:09:21

we're looking for something like this at adzerk

colin.yates17:09:59

if I can give you one bit of advice it is to know your domain. By far the hardest thing was getting the aggregate roots defined when doing DDD.

micha17:09:28

aggregate roots?

colin.yates17:09:37

The notion of state changes as discrete events is orthogonal and to be honest gives me 80% of the benefit without the DDD modelling. Were I to do it again I might skip the more formal DDD aspects

colin.yates17:09:11

(aggregate roots are part of DDD - they encapsulate a graph of data which should be atomically changed and represents a constraint boundary)

colin.yates17:09:38

so, a Customer and their address might be an aggregate root - you wouldn’t change the address separately from the customer and the customer might enforce some constraints on the address

colin.yates17:09:59

I tell you, if you haven’t jumped into DDD it is a whole new world whose pain and skill has little to do with technical or tooling skills

colin.yates18:09:26

Eric Evans’ ‘Blue Book’ is the authoritive reference but this is also really good: (finds link…)

colin.yates18:09:16

gotta go put the kids to bed

micha18:09:52

thanks for all the info!

micha18:09:47

martin fowler is unstoppable

onetom18:09:40

it's really like an entity with components in datomic

colin.yates18:09:25

the ‘promise’ is that you can design your aggregate roots to be whatever they need to be to support your business process/constraints. Those aggregate roots service commands, the effect of that command is then captured as discrete events (CustomerAddedAdvert, CustomerChangedAdvert etc.). Your RO model(s) can then subscribe to those events and denormalise the heck out of things for super speedy.

colin.yates18:09:58

No more single model which (badly) serves both the read and the write layers.

micha18:09:54

yes this is key

colin.yates18:09:00

the beauty comes when you realise that the event-stream is the audit log and all the various SQL/no document databases are simply projections of those events.

colin.yates18:09:12

Change the view - no problem, blow it away and then replay all the events

micha18:09:13

+9823982374234

colin.yates18:09:31

ha - I thought ‘is that your phone number?’ and then got it. Man, I need to stop working simple_smile

micha18:09:47

haha the infinite improbability drive kicked in

micha18:09:01

it's the phone number of that hot chick i lost when i was drunk 5 years ago

micha18:09:41

in my current project we have an interesting CQRS kind of thing going on

micha18:09:00

i'm building a new UI to replace this crazy C# MVC thing we currently have

micha18:09:24

i have a castra backend that builds views (compiled on the server as queries) for reads

micha18:09:34

those do lots of joins and whatnot to denormalize

micha18:09:41

and we can hand optimize them

micha18:09:11

for writes castra simply calls out to our existing REST API

micha18:09:24

and then performs the query to send to the client

micha18:09:54

it's been a really powerful tool for dealing with legacy code

micha18:09:11

it forms a nice clean seam on which we can pry in a number of ways without much risk

micha18:09:24

my clojure backend doesn't even have permission to write to any databases

micha18:09:43

so there's no way it can introduce any new operational bugs

colin.yates18:09:43

are you using materialised views - I was really excited about them when I first heard of them but then looked at the list of restrictions (in MS SQL) and found out they really weren’t that flexible

micha18:09:56

oh yeah we're not using those

micha18:09:02

too scary for migrations

colin.yates18:09:11

but now, nothing beats a simple event-stream which can be consumed to denormalise anything - no need for views or anything

micha18:09:12

basically my code can't interfere with migrations in any way

micha18:09:25

so we compile the query in the castra end

micha18:09:37

it amounts to a view, but it's all done in clojure

micha18:09:03

that means we don't have any state in the database for the views, wich is key for my sanity

colin.yates18:09:09

I see. Are you using https://github.com/jkk/honeysql? That rocked my world as well.

micha18:09:16

people are working on the legacy code as i work on the new thing

micha18:09:40

no, none of those things are really useful for serious work in my experience

micha18:09:55

i basically have code that emits sql

micha18:09:29

templates, more or less

micha18:09:46

we made some pretty fancy string interpolation macros and we just use that

micha18:09:10

i've always found sql dsls to fall over when things get real

colin.yates18:09:48

ah OK, I think you might be misunderstanding honeysql? It turns a Clojure map into SQL, that’s all. It is brilliant for composing SQL queries (as it is just data).

micha18:09:49

especially when you need to hand optimize the joins

colin.yates18:09:17

I used it to good effect for our internal reporting library, The idea of building up the strings by hand ran into a brick wall pretty quickly.

micha18:09:17

yeah but you can't really optimize joins well with those

onetom18:09:21

colin.yates: btw, i heard about this approach u describe from this article for the 1st time: http://martinfowler.com/articles/lmax.html how about you?

micha18:09:34

we don't concatenate strings, we form views

micha18:09:40

that are parameterized

micha18:09:49

we can parameterize at compile time too, though

colin.yates18:09:23

@onetime I read about that when I was back in the Spring world, a long time ago. Very exciting at the time and it made me wish I had a use-case where that was the bottleneck simple_smile

micha18:09:25

we don't have composable sql views or anything

colin.yates18:09:33

@micha oh I see - got you.

micha18:09:38

each castra query corresponds to a view

micha18:09:54

and we parameterize it so castra can supply those as arguments to the view fn

colin.yates18:09:08

I toyed with yesql for that but didn’t really settle on anything

micha18:09:19

yeah yesql is the closest analogy

micha18:09:26

but it's hopelessly tied to files

micha18:09:34

which we don't find to be useful at scale

colin.yates18:09:54

and breaks Cursive’s engine behind all those macros (or at least it did)

onetom18:09:04

colin.yates: are u following the advances of kafka + samza? any opinion on them? mentioning the materialized views reminded me this talk: https://www.youtube.com/watch?v=yO3SBU6vVKA

colin.yates18:09:57

I’m not following them closely but I do remember watching that video, about 35 minutes in (if I recall) and thinking, yeah, that is CQRS + event sourcing simple_smile

colin.yates18:09:18

to be honest, I haven’t yet needed to do anything that needed more than one machine to scale

colin.yates18:09:42

and I find a lot of ‘scaling’ problems can be designed away

colin.yates18:09:04

I see - yeah, honeysql is not going to help there.

flyboarder18:09:10

man the conversations that come out of this thread awesome

onetom18:09:36

thats pretty much my experience too... if it was running on multiple machines then i can usually re-architect so 1 machine was sufficient for it... simple_smile

micha18:09:18

those defview macros define functions that are paginated, sortable, filterable views

colin.yates18:09:23

did you not find the isnull(x, 1) came out way more expensive than (and x is nil or x = 1) in the query optimiser (been years since I looked at this though)

micha18:09:58

oh that could very well be the case, i need to spend some time with optimization

micha18:09:08

that query is fast enough for now though

onetom18:09:28

simple_smile sounds like a trick from the "celko" sql book... i was doing the proof reading of the hungarian translation and translated some of it too http://www.amazon.com/Joe-Celkos-Smarties-Fourth-Edition/dp/0123820227

colin.yates18:09:29

I also managed to get rid of those is nils by a neat trick by equality and the fact that nil is always (for all intents and purposes) nil. So (and c.isdeleted <> 1) is true when c.isdeleted is nil or 0

colin.yates18:09:47

(I think - been a while, but it made a huge difference to the query plan)

micha18:09:51

aha, that's great

micha18:09:11

the idea is that castra can call that function like this:

colin.yates18:09:09

It’s really old now, but http://shop.oreilly.com/product/9780596008949.do is a great SQL book

micha18:09:13

(campaigns {:filters {:networkid {:type :range :from 1234 :to 1234}} :sorting {:by :id :order :desc} :paging {:page 45 :page-size 50}})

micha18:09:37

that pagination state etc is stored in the client

micha18:09:46

and provided with eaech castra request

micha18:09:02

castra passes it through to the view after validation etc

colin.yates18:09:27

since I have your attention simple_smile

colin.yates18:09:43

do you guys emit from the dom or render with :display none for tabs and things like that?

micha18:09:03

almost always via :toggle

colin.yates18:09:03

each tab might be a non-trivial page with a 1000 row table and a summary chart

micha18:09:23

we don't usually make tables with that many rows

micha18:09:31

the user can only really look at maybe 100 rows

micha18:09:46

we use typeahead search and pagination extensively

micha18:09:12

we have tables that are basically unbounded in size

micha18:09:35

because we haev a powerful api so people can make hundreds of thousands of anything pretty much

colin.yates18:09:17

now jquery is more accessible I might re-think pagination...

micha18:09:20

if you don't deallocate the dom elements, does it matter if they're actually in the dom or not?

colin.yates18:09:44

no, I guess not. I remember reading something about reflowing being affected but I can’t remember the details

micha18:09:47

i was thinking no

onetom18:09:57

it mostly matters during debugging because they are in the way so u cant see the tree from the forest simple_smile

micha18:09:23

yeah it would be nice if dev tools could shade the hiden things

colin.yates18:09:24

either way I am hoping to have cells backed by those thousands of entities and pagination will be local

micha18:09:42

it's trivial to implement pagination on top of cells

micha18:09:17

the only thing you lose there is command-f searching

micha18:09:27

which sucks but it's a tradeoff

onetom18:09:31

colin.yates: im preparing a repo to showcase these kind of things: https://github.com/exicon/webapp-skeleton

micha18:09:05

command-f is only a partial solution, so i'm willing to forgo it

onetom18:09:12

we also hit some performance issues because we had ~400 row list of mobile apps with their icon and ratings in the DOM 3 times...

micha18:09:17

and work on better typeahead widgets

micha18:09:10

@onetom: that is awesome

colin.yates18:09:11

this is strictly desktop (actually Chrome)

micha18:09:36

a chrome extension would give you huge power to make a really computed thing

micha18:09:54

because you can have encrypted database files and stuff locally

micha18:09:39

you'd be able to really leverage the event sourcing append-only log style

micha18:09:13

since you can cache and rollup in the client whenever you want and save it to disk where oyu have unlimited space

micha18:09:40

a real peer to peer architecture

micha18:09:03

i think something like that is maybe the holy grail

micha19:09:01

i use it to bring in cljs-console

micha19:09:18

so i am not tempted to use console.log() anywhere

micha19:09:24

and it ends up in production of course

onetom19:09:33

whats that? sounds something what i would like to have too simple_smile

micha19:09:53

like oh i'll just add a debug log here real quick and remember to remove it in 5 minutes

micha19:09:03

and i never remember until someone comments about it in QA

onetom19:09:11

u r clearly not pairing then

micha19:09:33

yeah mostly not

micha19:09:49

we only have 7 programmers, and people need to move around a lot

onetom19:09:49

we always go thru the diffs before we commit, so console logs practically never slip into the code

onetom19:09:37

we are also 7 programmers including myself... which is 3.5 pairs...

onetom19:09:06

someone "pairs alone" as one of my colleagues said the other day simple_smile

mynomoto19:09:09

I don't like magical refers, make things hard to grep 😕

colin.yates19:09:51

gotta go - back tomorrow - it’s been great all - learnt loads simple_smile

onetom19:09:52

mynomoto: i would like to define a loop-tpl replacement. u wouldnt want to require that explicitely, would u? 😉

mynomoto19:09:34

I would want to do (:require [hoplon.core :as h]) at all places everytime.

micha19:09:58

@mynomoto: we can make it optional

micha19:09:25

we just need to factor the boot task a little more

micha19:09:36

currently the boot task does only a few things

micha19:09:06

one thing it does is generate the boilerplate for the html file and the .cljs.edn file based on a single .cljs.hl page file

mynomoto19:09:08

@micha: no problem, if if find the time I may look at it myself.

micha19:09:21

the other thing it does is add the refers to the generated cljs namespace

micha19:09:37

but otherwise the generated cljs namespace is unaltered

onetom19:09:23

boot-hoplon is quite clear now (or i just read it enough times 😉 at least im satisfied with the speed i could discover this :refers thing.

mynomoto19:09:23

@micha: decoupling those will make it possible to use hoplon as an api to create dom things. I think this is already possible, only no examples around.

micha19:09:39

yes it's already possible

micha19:09:53

we do need to do some work on the polyfill i think though

micha19:09:08

i'd like to make the polyfill lazy

micha19:09:21

i.e. alter dom element instances instead of the element prototype

micha19:09:31

and only when we need to

micha19:09:41

which would be when you add a cell child to an element

micha19:09:47

like if you use loop-tpl etc

onetom19:09:55

btw, just the other day we used an svg dom element generated by a vanilla js lib as a function: https://github.com/exicon/hoplon-chart-examples/blob/master/src/index.cljs.hl#L149-L155

micha19:09:00

if you don't use loop-tpl there is no need to apply the polyfill

onetom19:09:21

i felt super powerful that moment 😉

micha19:09:30

which would ensure that if you "mount" hoplon in another app it won't be causing any ill effects outside the mountpoint

mynomoto19:09:56

@micha: what ill effects?

micha19:09:50

if there is a lot of churn in the dom it might cause performance issues

micha19:09:21

i dunno, it just seems like something we'd want to avoid

mynomoto19:09:00

Yeah, make sense.

onetom19:09:49

awww, shit, the conversation about javelin cells and whatnot has scrolled out of my slack buffer ;(

micha19:09:28

that chart thing is super cool

micha19:09:44

the tooltip

onetom19:09:18

it's a nice clash of the hoplon and the jquery worlds, no?

onetom19:09:15

@mynomoto: phew simple_smile thx! i saw this site once but havent bookmarked it so never found it again. how do u find it usually? and who is operating it? doesnt look official (as-in slack related) is it opensource? can i run it for our company slack account?

mynomoto19:09:16

No idea, I just remembered seem it and googled it to find the link 😉

onetom19:09:09

indeed 1st hit... last time i google i couldnt find... 😕

onetom19:09:19

micha: i was bitten again by loop-tpl binding behaviour. i haven't wrapped my seq into a formula cell... i guess it's a negligible overhead from performance perspective, so i would make it the default behaviour. can't really think of a case where i wouldn't want to do that.

onetom19:09:26

(loop-tpl :bindings [[sub-label sub-path] (cell= (partition 2 items))]
                 (menu-item ... ))
vs
               (loop-tpl :bindings [[sub-label sub-path] (partition 2 items)]
                 (menu-item ... ))

micha19:09:55

yeah i can see that

micha19:09:08

and it would also be fine if items was not a cell

onetom19:09:09

if i dont want reactivity i would just use map probably...

micha19:09:22

yeah that's true

micha20:09:10

the only issue i can see is like

micha20:09:44

(loop-tpl :bindings [thing (create-things-cell items)]
  ...)

micha20:09:03

where create-things-cell is a function that returns a cell

micha20:09:22

probably not a big deal

onetom20:09:50

and items?

micha20:09:01

items could be a cell

onetom20:09:46

but u were just discouraging functions returning cells a bit earlier, right?

micha20:09:11

yeah i mean sometimes you need to do it simple_smile

onetom20:09:17

so sometimes (which is the rarer case) u would just

(let [things (create-things-cell items)]
   (loop-tpl :bindings [thing things]
      ...))
right?

micha20:09:34

yep that would work

micha20:09:07

it's just a little harder to understand what you did wrong if you forget to do that

onetom20:09:51

harder than: "Uncaught Error: [object Object] is not ISeqable" ?

onetom20:09:15

because that's all i get when i forgot the (cell= ) from around that (partition ...) call

onetom20:09:43

i think a tiny bit of better error reporting here could have a huge impact on adoption

onetom20:09:38

btw, i also get a completely blank page in this case

micha20:09:25

the [object Object] part is a hard thing to fix

micha20:09:32

because it's being generated by javascript

micha20:09:42

i mean by cljs core anyway

micha20:09:18

it should be showing you the cljs representation

micha20:09:45

Uncaught Error: #<Cell [1, 2, 3]> is not ISeqable

micha20:09:56

that would give you all the information you need

micha20:09:56

[object Object] doesn't tell you anything about cljs things

onetom20:09:30

yeah, that would be quite a useful error message

onetom20:09:27

but gnite for now (it's almost morning here) thanks a lot for the tons of smartness u poured on us today!

micha20:09:58

see you later, i hope it wasn't too bad simple_smile

onetom20:09:22

what do u mean by too bad?! it was awesome!

onetom20:09:09

(pour :some [thing things]
   (elem thing))
😉 gnite!

micha20:09:35

ah pour, i like it