@mynomoto: i dont have a strong opinion on IRC. i never liked it because of the hassle with the history and bouncers and logger bots, but one undebatable upside is that the protocol is free and the freenode network is independent from political or commercial influence... (at least thats how i imagine it)

@onetom: I think it's more of a practical matter than a political one. There are more people and conversations here than there and having one "official" place is better than be scattered over many.

I don't want people to leave irc, only not sending new people there if there is more talk going on here.

I like gitter better but I know a lost battle 😉

btw, i actually have gitter running too; primarily to follow the https://github.com/red/red project...

I think u just found the critical word here: official

Maybe we should just say something along those lines, like: Most of the community is on [Slack], so that's the most active place for discussions and the primary place to get Hoplon support. There are alternative/fallback channels too: [Gitter] & [IRC].

hi all - is this or Discourse the right place to ask questions?

I have a bunch of newbie/clarity questions which I posted to http://hoplon.discoursehosting.net/t/newbie-clarity-questions-around-cells/552 - in general is it sufficient to post to discourse?

Hi. I was recently trying to test some castra calls using ring mock. It took me a little time to get it working, and this is because castra frontend pass some castra specific header like transport method and csrf. I am now wondering if there should be support for mock castra requests for testing on backend ? So that it is not necessery e.g. for newbie to look into castra code to be able to construct request. Or maybe it should be sufficient to only do clojurescript tests ?

hi colin.yates and voytech!

@micha - the man himself

how's it going?

I am still googling around different use-cases - I posted a few newbie Qs at discourse which would be great if you could take a look at (you know, your time being mine to spend :-)).

haha sure

but yeah, I think javelin is exactly the right model for how I view the model creation (as a sequence of data transformations)

i've been inconsistent with reading discourse

job deadlines and whatnot

one sec let me get caught up

thanks .

ok so for your first question

i can just sketch out the answers here and then post to discourse later, is that cool?

that would be great

ok great

so the way javelin works is each cell keeps references to its "sinks" and "sources"

sinks are other cells that reference it (formula cells or lenses)

and sources are cells it references (it's a formula cell or lens)

changes happen as a result of swap! or reset! on an input cell

or possibly a transaction involving multiple input cells

each input cell (or lens) has a watcher on it that calls the "propagator"

the propagator is a function that given a cell where the change originates, computes the dependency graph from the sources and sinks of each cell, walks the graph doing a sort of topological sorting of the cells in the graph

such that no cell comes before the cells it depends on in the sequence of cells

then the propagator propagates the change to each cell in turn

right

the change is computed by calling a thunk with the sources as arguments, dereferencing any sources that are cells

if the result of recomputing the thunk is the same as the existing value in the formula, then propagation is skipped for all of the cells that depend on this cell

(unless they depend on another cell that has changed)

the dependency ordering info is part of the cell, computed when the cell is created

basically cell rank is a monotonically increasing sequence

when you make a new cell it will have a rank that is higher than any previously created cell's rank

and when it's time to propagate we walk the sinks and build a priority queue with the rank as the priority

so to answer your question: dereferencing a cell is free

perfect

it's an eager system with caching

(and great explanation!)

swapping, however is not free

because you may be updating things you don't care about

although that has never been a problem for any of my applications so far

we use cells as the plumbing really

cell formulas are always small, one liners

if the formula starts to get complex it's usually more productive to move the logic into a function

and call the function from the formula

because usually complex formulas indicates some other absrtaction you need, which you can't really do very well in formulas

so the "debugging cell=" question

consider:

this will evaluate the formula's thunk every time clicks changes

because the pruning happens on the sources, not the value of the formula itself

it can't know what the value of the formula will be without evaluating the thunk, right?

the value of the formula could involve non-pure functions, too, like (cell= (or (even? clicks) (rand-int clicks)))

right

so one weird trick you can do

whoops

disregard that

ok there we go

got it - I see

so the REPL should work, i don't see why not

but remember that the REPL and clojure itself have a lot of hacks built in around reference types

like how to dynamically redefine them etc

and cljs doesn't have real vars yet, which are the foundation of all reference types, the most primitive one (because of the relationship between vars and evaluation)

so javelin cells keep their own sources and sinks references

instead of redefining them you probably want to mutate them in place

so the references are kept intact

javelin.core has functions like set-formula!, destroy-cell! etc

you can use these from the REPL to modify cells without disturbing the cell graph

so if I change the defn that a cell= delegates to I can just use (set-formular! ..) on that cell?

yep

can I ask a few more Qs then about cells?

oh sorry, it's set-cell!= that you want

the macro

set-formula! is a lower-level function that does the same thing

but the macro provides some sugaring

sure fire away

thanks . So in terms of garbage collection if I have a (defelem top-panel-page-1) and (defelem top-panel-page-2..) and in those I create various (cell= (transform some-top-level-cell)) inside each elem...

.. if I show page-1 then those cells and formula get inserted into the graph. If I then show page-2 (and I am (condp = page :page 1 (top-panel-page-1) :page 2 (top-level-page-2)) then I assume all those cells created in page-1 are removed as the page-1 dom tree is removed?

And

actually no

If I then go back to page-1 is assume those page-1 cells are recreated?

that's the react model more or less

we tend to leverage static allocation

and allocation pools

generally in an application you have a relatively small number of "screens"

or rather i should say screen templates

layouts kind of

basically places where varying data will be displayed to the user

we statically allocate those wherever possible and just hide them when they're not in use

for sequential things, like table rows for example

we allocate them into a pool

when a row is removed from the dom we don't try to deallocate it

instead we move it back into the pool

when we need one later we get it from there instead of creating a new one

I see

this approach allows us to greatly simplify the whole model

because you only need to write the constructor

you don't need any lifecycle protocols

without weakmaps in js i don't know if there is an automatic way to handle garbage collection

the existing weakmap polyfills etc are not sufficient

really we need weak references

there are some ideas we've been thinking about

right - I don’t see what happens to cells that were created in the deallocated template rows though?

nothing gets deallocated

sorry, I mean the ‘inactive pool'

oh it's like the um

the hotel guest example

where you have a motel with 100 rooms say

and maids are assigned to rooms

different guests move into and out of the rooms

but the maids remain the same

the cells are the maids

so for example

loop-tpl is the pool allocator

the body of the loop-tpl is the constructor

initially that will be called 3 times, and you'll have an unordered list with 3 things in it

and the pool will be empty

now suppose you do (swap! data conj 4)

since the pool is empty it must allocate another list item

so it does that

now the list has 4 items in it

now suppose you do (swap! data pop)

now the data cell will contain [1 2 3] again

and the 4th list item is removed from the dom and stored in the pool

but what happens to the d reference in the 4th list item?

well it's bound to (cell= (guard (nth data 3))) under the hood

the guard is there because nth will throw on invalid indexes

so d will be nil for the 4th list item

note that d, the maid, has not changed rooms

it's still looking at the 4th item of data

but now the 4th item of data is nil, because data has no 4th items

so now it's an empty li element

now suppose you do (swap! data conj 42)

the loop-tpl sees it has things in the pool, so it doesn't create any new thing

it just puts the 4th item back in the dom

and the 4th item is now a li with "42" as its text content

becuase d also automatically updated itself

does that make sense?

(parsing)

I think so. Can I sanity check an example closer to my intention?:

sure

(defc people {1 “Bob” 2 “Jo”})
(defc locations {1 “Home” 2 “Work”})

(defn denormalise-people [i] (cell= (get people i)))
(defn denormalise-locations [i] (cell= (get locations i)))


(defc records [{:person 1 :location 1} {:person 2: location 1}])

(defn denormalise-record [{:keys [person location] :as record]
  (tr 
    (td (denormalise-people person))
    (td (denormalise-locations location))))

(defn render-table [records]
  (table
    (tbody
      (loop-tcl :bindings [record records]
        (denormalise-record record))))

where any of the defc’s might change (and will themselves be formula cells on a stream of incoming domain events)

(again it is made up, but representative of my intentions)

this looks legit to me

this will allocate as many things as the maximum number of items in records

and it will never deallocate them

so you'd paginate records or something

basically records represents the window of things the user is seeing at one time

which is constrained by the weak human brain

right, we allow the user to chose the number (10, 20, 50, 100, etc.)

totally

we will usually make a state machine for this

but because it is ordinally indexed, going from 100 to 10 is fine because the 90 ‘orphans’ in the pool will be nil

so we can hook the state transition functions for the machine to event handlers

exactly

changing the ‘window’ will fire an event over a web-socket to the server to return more results which will then (reset! records …) in this example

the cells will still exist, but the formula values will be nil

for the ones that are removed from the dom when you decrease the number of items in records, that is

right.

for those guys the value of the record cell will be nil

This is just great - thanks Micha.

I need to run unfortunately but my last assumption is that because of the mechanics of the cells, changing something like the name of a ‘person’ will result in a trivial amount of recalculations and then hoplon will simply update a ‘td’ for example

actually there is a slight issue with your code

there usually is

so denormalize-record is a function that returns a dom template more or less

you pass it a cell containing the values that will change over time

but the cell itself is like an atom or a var

you can't dereference it directly

we provide cell-let to do that

that will destructure the record cell to give you cells for the location and person

you can then make formula cells for those which you can bind to the text content of the td elements

notice here that cells are only allocated once per item

the body of the loop-tpl only runs once per item

which is good

why does line 13 and 14 need to wrap the result of (denormalise…) in a cell=? as the denormalise- fns return a cell=?

oh my bad

good catch

micha, u have just written up a great "how hoplon & javelin works" tutorial. can we just chuck it into the hoplon wiki until we have a more distilled version?

that would be awesome

just to be very explicit, denormalise-person may actually be really expensive which is why I want to ensure it is only run once - I think we are on the same page and I am looking forward to this.

colin.yates: do u mind if i leave ur name in there too (just so i dont have to edit it too much)?

not at all - go for it

i would recommend going easy on the functions that return cells

i find it tends to work better to let functions return values and build cells around them anonymously in place

because cells compose in a different way than functions do

basically i try to keep the javelin layer at the very top of my ladder of abstractions

that layer can handle the caching for you automatically

in my use-case though I would want to access the result of the very expensive calculation that (denormalise-person) does everywhere … and make sure that two calls to denormalise-person for the same person get cached…

I don’t think I understood what you are proposing?

ah i see now

or are you saying that denormalise-person can be very expensive and consume a cell, but where-ever I call it I should wrap the call in a (cell= (denormalise-person 1)) relying on multiple (cell= (denormalise-person 1))s?

yes it makes sense what you did there

ah OK - phew

I really have to run - thanks a bunch micha - I will be back later and may ping you then if you are on-line. My manic-job-of-the-afternoon is to re-write my current re-frame app which uses immutant and web-sockets and prototype with hoplon, so many questions will undoubtedly arise . May as well throw boot in instead of lein for fun.

awesome i'm interested to hear your impression/criticisms

micha: i was wondering about loop-tpl today... is there a reason why is it expecting just 1 body expression: https://github.com/hoplon/hoplon/blob/master/src/hoplon/core.clj#L158

similarly im not sure why would the bindings should be constrained to just 1 pair of destructuring form and sequence?

the body i think should be wrapped in an implicit do

but the bindings get more tricky

i did have a branch that worked like doseq/for

i was thinking more like wrapping it into (spliced ...)

that allowed multiple bindings

well you can always do that by just wrapping with []

but you might want to perform side effects

so an implicit do seems like the right way to go, no?

in my experience to my colleagues it was more intuitive to expect the ability to throw in multiple dom elements into the body and have them repeated...

i can remember a case when we only wanted side effects

if we wanted some debugging, we just wrap the (dom-elem ...) as (let [_ (print ...)] (dom-elem ...))

which leads back to the multiple binding question... if it would be possible to have multiple bindings then side-effects could be just thrown in there

yeah it can be done, i did it at one point

but there were some inconsistencies in the way you use it that i didn't like

u have a substantial code base, so maybe u can do a quick grep to see how often r u doing side effecting from withing a loop-tpl

i never do

i do wrap things in vectors to splice them in quite a few places though

but it would be non-idiomatic i think in terms of general clojure macro style

not that we need to slavishly enforce clojure idiom

lol

making splicing automatic would decrease indentation level though...

ok yeah i'm for it

loop-tpl already expresses a level of grouping anyway...

okay, i tell u why these questions came up

i was trying to create a generic menu and sub-menu builder and it became a monster quite quickly

it doesnt fully work at the moment but it looks something like this:

so as u can see i had to throw in an extra let which could have just gone under :bindings

and some earlier versions had extra stuff next to the menu-item too

the aggressive indentation in the sublime lispindent plugin prompted me to think about how to be a bit less verbose and that's why i looked into the loop-tpl implementation

as u can see i even broke the :bindings attributes into a separate line to save a bit on the indentation

so my 3rd question/idea is: do we really need this :bindings attribute and such an eccentric name as loop-tpl?

Since im not expecting to have such an invasive change, I was thinking how can I have such a custom (defmacro x ... loop-tpl* ...) automatically compiled into every .hl file? I haven't found any extension hook in boot-hoplon at 1st glance....

btw, i came up w the name x because it symbolizes multiplication... loop is already taken stamp is quite short but suggest a very imperative approach foreach? every? ... something-let or let-something?

i know the (loop-tpl :bindings ...) was invented to blend in to the rest of the dom element constructors but it's so long an unorthodox, it's just not very practical...

now that hoplon v6 is about to come out, it think it makes sense to add such practical polish to it before it gets announced to a broader audience...

just we just get rid of the :on-*s and :do-*s...

we were thinking to rename loop-tpl to just splice

(splice :for [x xs] ...)

that would work well with the implicit splice, too

i'm totally fine with extending the loop-tpl bindings

i think it needs a little design work though to be intuituve and regular to use

the loop-tpl is very much like the clojure unquote-splicing

(i had to step away to attend my crying daughter)

so (splice :for ... is already nicer, though as a non-native speaker i still don't understand really what splice mean, what does it have to do with ropes and still confuse it with slice

imean my 1st exposure to splice was the javascript function: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/splice and it simply makes no fckin sense even today to me...

haha

yeah splicing means like interweaving or patching one thing as a section of another thing

like for ropes

you can splice a new piece to replace a worn section of rope

the idea it expresses is that the collection is being altered by adding or removing a contiguous section from the middle somewhere

brb

just to demonstrate my personal (or rather european's) confusion, if i translate back the meanings of splice from hungarian to english, i get the following: twist, interweave, twine, furl, cockle, kink

(besides splice of course)

(insert :every [thing some-list]
        (elem thing))

and as i was reading your explanations even pool crossed my mind. of course my own 1st reaction was "it's too low-level and implementation detail revealing/bound" but actually, if i assume i don't know that there is a pooling mechanism behind it, it still works quite well:

(pool :of [thing some-list]
      (elem thing))

very concise and expressive. it doesn't suggest some kind of nesting level imho. doesnt collide with anything built in

also pool reads as loop just backwards ;D

ah, playing w letters and moving along this splicing of ropes/wires, there is the very famous loom https://en.wikipedia.org/wiki/Loom_(video_game) https://upload.wikimedia.org/wikipedia/en/3/3b/Loom.png

(loom :in [thing some-list]
      (elem thing))

but honestly this attribute thing is really unnecessary. it won't return just 1 dom element but a vector, so i don't see a need for making it look like dom elements... it's actually more confusing than explanatory. just the other week i made some element returning a loop-tpl and i tried to call it as a function to extend it with further kids...

(defelem thingies [attrs kids]
    (loop-tpl ...))

(let [default-thingies (thingies :class "blah")]
   (default-thingies :click #(...)))

hmmm now the only problem is loom has nothing to do with weaving when it's a verb (as i thought in all my life

these would work for me too:

(*** [thing some-list]
     (elem thing))

(... [thing some-list]
     (elem thing))

yeah we could do that

we didn't do it initially and have resisted it so far because we wanted to be able to support html syntax

but i'm not sure if that's something we want to continue supporting forever?

i wouldn't want to break it without getting the okay from the community

<loop-tpl bindings="{{ [ x xs ] }}">...</loop-tpl>

oh, r u still using hoplon thru html?

i'm not but i don't know for sure that nobody is

okay, so i wouldn't bore everyone with this naming if there would be a way to hook in some customization into the hoplon compiler, so some project-wide macros and functions could be required in automatically into every .hl file. is there a way to do that or it's actually better not to do that?

ah, i think i found it. (hoplon :refers #{'my.ns})

May I ask a Q about castra?

colin.yates: of course

At the moment I use websockets which push a ‘new-state:N’ to each client, the client then immediately asks the server (via a websocket) to get the new state since last seen. The server then pushes all that new state with a :new-state N checksum.

@onetom: sorry for the late reply, but you're right there

that's the (undocumented) hook you can use

I think I can do the same in castra by the following - have the state returned by castra always include the latest checksum and have the client do an rpc call every second and when it pushes anything to the client check the :new-state flag (does that make sense)?

micha: not sure though how to hook macros in, but i think i can figure it out on my own

effectively, the state that is synchronised between the server and the front-end is the batch of state last-seen since that client last checked.

@onetom: it'll refer the macros too, automatically

thats how i understood the source too but it didnt work on 1st try. but no worries, im on track

java.lang.AssertionError: Assert failed: Can't find macro_test.cljs in classpath

i think you may need to have at least one cljs definition

that .cljs is the problem

inaddition to the macros

ah

it's undocumented because there are some gotchas in there

also make sure you don't define anything that conflicts with a previous definition

it doesn't do any checking of which things it might cause a name clash with

@colin.yates: parsing...

how could I go about creating a new element? not using defelem but something that will print <new-elem> as html?

if I can put it more simply - how do I wire up castra to only show the client the most recent stream of events since they last connected

@flyboarder: look in hoplon.core namespace

there is a function in there like make-elem-ctor or something like that

you can use that

those are private are they not?

basically you can do (.createElement js/document "new-elem")

any browser will handle that as a block element

you can use css to have it display as an inline element or table cell or whatever you want

this is something we might consider for defelem also

@colin.yates: you already have an event driven backend?

I do

one thing i've wanted to try is to have a sort of hybrid approach

the thing i want to avoid is the complexity of the event driven backend

doesnt sound like it fits the castra model which was made for unidirectional data-flow, no?

i wanted to have the stateless request-response model in the server

but with server-push also via websockets

the thing i was thinking of trying is to do polling on the server

it would go like this:

client connects to websocket

websocket is allocated and managed by ring middleware wrapping castra

the ws middleware would then begin polling castra

@colin.yates would be interesting to gather your insights in comparing re-frame+lein w/ hoplon+boot — would make a for great blog post (see that you are working on your blog site with jekyll — you should check out https://github.com/hashobject/perun 😉.

sending data to the client whenever the response changes

this has a number of interesting properties

1. if we use datomic the polling is nearly free

@donmullen: yeah - I have so many great things to put on there, but time 😞

2. since we're using websockets we can keep track of the last thing we sent to the client, so we can compute diffs and patch on the client

eg. we don't need to send the entire response

3. our business logic and api is still request-response stateless functions

the websocket layer and event stuff is boilerplate in a library

does that sound like a relatively sane approach to you?

it does, but I think I am almost there already except I don’t want the fully-realised model on the server.

so i think with castra if you want to get a response that is "as of" the last time they asked

I want the equivalent of /events?from=10 which would return something like {:latest-event 51 :events [10, 11,12 ….51]}

you can simply add that as a parameter to your RPC function, no?

and have the client keep track of their own state

they just provide it with each request

so the client passes {:last-seen 10} to the rpc, the server constructs a view which is the latest events which castra syncs to the server - I see.

the state can be stored in localstorage on the client perhaps

or whatever you want

so you don't need sticky sessions or anythign like that on the backend

exactly, I want the backend to be stateless really

another similar use-case would be tailing a server-side log

micha: normally im quite skeptical about polling, but what u described with those 3 points add up into a pretty strong argument

+1 - it really does. It also handles lost clients very well as well

the key there is datomic

you don’t need datomic for this though. My back-end is pure DDD and event-sourcing and it is a perfect fit for this

@onetom: I agree, generally i try to avoid polling for performance reasons, and less over-the-wire

"several people are typing" thats what slacks says below my input field

can you describe your stack?

All domain transitions are captured in discrete events. To get the latest model you simply (reduce ) the events for that aggregate root

the event-store happens to be in MS SQL server but it could be anywhere.

oh wow, interesting

So all I need to do is stream any events that have happened since the client last talked to the server

you compact events from time to time i guess?

colin.yates: DDD as in Domain Driven Design?

yep

to stay within bounded db table size?

this isn’t a particularly high traffic site (internal LAN)

this sounds promising

i imagine something like aurora on aws would be suitable for something like this

for a more highly loaded application

ah, unfortunately for me this is all closed-source and commercial (think medical data)

it does mean I get a pretty nicely controlled environment to play with though

do you have a caching layer on top of the SQL engine?

I am also going slightly off-track by doing all the event denormalisation on the client side

which means travelling back in history is a trivial thing - simply discard any events since a particular timestamp.

I haven’t needed one yet - MS SQL is pretty quick and the working set tends to be small (e.g. everybody is interested in the last 10K events for example)

for searching etc. I denormalise some stuff but those are themselves projections of the event stream

man this sounds awesome

DDD + CQRS + event streaming makes your brain hurt but yeah, if you can manage to get it working it is brilliant.

my current architecture is web-socket (and re-frame) based so clients don’t poll, they just get sent the events in real-time

it makes my brain feel better after all the pain of the normal SQL pattern, really

colin.yates: what are you challenges with re-frame?

the really nice thing is that your entire domain model can be persistence agnostic - ‘loading’ an entity is really nothing more, and can be nothing more than reducing the events for that aggregate root

the main one is the fact that subscriptions aren’t free to dereference.

it rocks, it really does, but I made an incorrect assumption about it.

caching can make dereferencing a point in time free

So you can imagine an event stream of 20K events going into a data-transformation which trims by time going into one which hydrates each entity etc.

I had many subscriptions which did that the calculations

to be blunt, I tried to implement the spreadsheet model using subscriptions and it worked really nicely, except I noticed many things were being recalculated

yeah

re-frame is still feasible, I could simple use the app-db as a denormalised cache of sorts

i experimented with a spreadsheet implementation inside of the datomic transactor

I also ran into some issues, almost certainly mine where things wouldn’t always update when they should - things like that. I am absolutely not claiming they are bugs, more my lack of knowledge

i think it could work, but it would require some serious design hammocking

the problem, as i see it, is that the client has very little memory and no access to the usual ways we alleviate memory pressure

to be honest, I could achieve what I want with a trivial end-point which I poll using cljs-ajax passing in the parameters I am interested in which in general are simply the checksum of the last seen id

yes, this is definitely not a general solution for unbounded lists etc.

yeah the cljs-ajax approach is how we do it currently, using castra just to handle all the boilerplate and whatnot

ah OK. I see.

castra also implements the CQRS data flow

like calling an RPC function int he client is the command

it returns a promise you can use to handle exceptions

but it doesn't return state

state goes into a javelin cell

formulas that reference the javelin cell are the query in the CQRS parlance

this "handle errors at the caller, but handle state globally" model has worked very well for us

i lied, the promise can be used to handle state too, but we hardly ever do that

so I think I can still get this to work with castra by calling the rpc with the ‘last-seen’ sequence-id. The server then loads the (select event_blob from t_events where sequence_id>last_seen) into the cell

you totally can, yes

thanks again all.

(and yes, I really must blog/produce a template project when I get all the dots working)

yeah that would be awesome

we're looking for something like this at adzerk

if I can give you one bit of advice it is to know your domain. By far the hardest thing was getting the aggregate roots defined when doing DDD.

aggregate roots?

The notion of state changes as discrete events is orthogonal and to be honest gives me 80% of the benefit without the DDD modelling. Were I to do it again I might skip the more formal DDD aspects

(aggregate roots are part of DDD - they encapsulate a graph of data which should be atomically changed and represents a constraint boundary)

so, a Customer and their address might be an aggregate root - you wouldn’t change the address separately from the customer and the customer might enforce some constraints on the address

I tell you, if you haven’t jumped into DDD it is a whole new world whose pain and skill has little to do with technical or tooling skills

Eric Evans’ ‘Blue Book’ is the authoritive reference but this is also really good: (finds link…)

http://www.amazon.co.uk/Domain-driven-Design-Tackling-Complexity-Software/dp/0321125215 is the blue book

https://vaughnvernon.co/?page_id=168 is a book for mere humans

gotta go put the kids to bed

thanks for all the info!

micha: also http://martinfowler.com/bliki/DDD_Aggregate.html

martin fowler is unstoppable

it's really like an entity with components in datomic

the ‘promise’ is that you can design your aggregate roots to be whatever they need to be to support your business process/constraints. Those aggregate roots service commands, the effect of that command is then captured as discrete events (CustomerAddedAdvert, CustomerChangedAdvert etc.). Your RO model(s) can then subscribe to those events and denormalise the heck out of things for super speedy.

No more single model which (badly) serves both the read and the write layers.

yes this is key

the beauty comes when you realise that the event-stream is the audit log and all the various SQL/no document databases are simply projections of those events.

Change the view - no problem, blow it away and then replay all the events

+9823982374234

ha - I thought ‘is that your phone number?’ and then got it. Man, I need to stop working

haha the infinite improbability drive kicked in

it's the phone number of that hot chick i lost when i was drunk 5 years ago

in my current project we have an interesting CQRS kind of thing going on

i'm building a new UI to replace this crazy C# MVC thing we currently have

i have a castra backend that builds views (compiled on the server as queries) for reads

those do lots of joins and whatnot to denormalize

and we can hand optimize them

for writes castra simply calls out to our existing REST API

and then performs the query to send to the client

it's been a really powerful tool for dealing with legacy code

it forms a nice clean seam on which we can pry in a number of ways without much risk

my clojure backend doesn't even have permission to write to any databases

so there's no way it can introduce any new operational bugs

are you using materialised views - I was really excited about them when I first heard of them but then looked at the list of restrictions (in MS SQL) and found out they really weren’t that flexible

oh yeah we're not using those

too scary for migrations

but now, nothing beats a simple event-stream which can be consumed to denormalise anything - no need for views or anything

basically my code can't interfere with migrations in any way

so we compile the query in the castra end

it amounts to a view, but it's all done in clojure

that means we don't have any state in the database for the views, wich is key for my sanity

I see. Are you using https://github.com/jkk/honeysql? That rocked my world as well.

people are working on the legacy code as i work on the new thing

no, none of those things are really useful for serious work in my experience

i basically have code that emits sql

templates, more or less

we made some pretty fancy string interpolation macros and we just use that

i've always found sql dsls to fall over when things get real

ah OK, I think you might be misunderstanding honeysql? It turns a Clojure map into SQL, that’s all. It is brilliant for composing SQL queries (as it is just data).

especially when you need to hand optimize the joins

I used it to good effect for our internal reporting library, The idea of building up the strings by hand ran into a brick wall pretty quickly.

yeah but you can't really optimize joins well with those

colin.yates: btw, i heard about this approach u describe from this article for the 1st time: http://martinfowler.com/articles/lmax.html how about you?

we don't concatenate strings, we form views

that are parameterized

we can parameterize at compile time too, though

@onetime I read about that when I was back in the Spring world, a long time ago. Very exciting at the time and it made me wish I had a use-case where that was the bottleneck

we don't have composable sql views or anything

@micha oh I see - got you.

each castra query corresponds to a view

and we parameterize it so castra can supply those as arguments to the view fn

I toyed with yesql for that but didn’t really settle on anything

yeah yesql is the closest analogy

but it's hopelessly tied to files

yes

which we don't find to be useful at scale

and breaks Cursive’s engine behind all those macros (or at least it did)

colin.yates: are u following the advances of kafka + samza? any opinion on them? mentioning the materialized views reminded me this talk: https://www.youtube.com/watch?v=yO3SBU6vVKA

I’m not following them closely but I do remember watching that video, about 35 minutes in (if I recall) and thinking, yeah, that is CQRS + event sourcing

to be honest, I haven’t yet needed to do anything that needed more than one machine to scale

@colin.yates: for example: https://gist.github.com/9ce484f7ea7320a66d3f

and I find a lot of ‘scaling’ problems can be designed away

I see - yeah, honeysql is not going to help there.

man the conversations that come out of this thread awesome

thats pretty much my experience too... if it was running on multiple machines then i can usually re-architect so 1 machine was sufficient for it...

those defview macros define functions that are paginated, sortable, filterable views

did you not find the isnull(x, 1) came out way more expensive than (and x is nil or x = 1) in the query optimiser (been years since I looked at this though)

oh that could very well be the case, i need to spend some time with optimization

that query is fast enough for now though

sounds like a trick from the "celko" sql book... i was doing the proof reading of the hungarian translation and translated some of it too http://www.amazon.com/Joe-Celkos-Smarties-Fourth-Edition/dp/0123820227

I also managed to get rid of those is nils by a neat trick by equality and the fact that nil is always (for all intents and purposes) nil. So (and c.isdeleted <> 1) is true when c.isdeleted is nil or 0

(I think - been a while, but it made a huge difference to the query plan)

aha, that's great

the idea is that castra can call that function like this:

It’s really old now, but http://shop.oreilly.com/product/9780596008949.do is a great SQL book

(campaigns {:filters {:networkid {:type :range :from 1234 :to 1234}} :sorting {:by :id :order :desc} :paging {:page 45 :page-size 50}})

nice

that pagination state etc is stored in the client

and provided with eaech castra request

castra passes it through to the view after validation etc

since I have your attention

do you guys emit from the dom or render with :display none for tabs and things like that?

almost always via :toggle

each tab might be a non-trivial page with a 1000 row table and a summary chart

we don't usually make tables with that many rows

the user can only really look at maybe 100 rows

we use typeahead search and pagination extensively

we have tables that are basically unbounded in size

because we haev a powerful api so people can make hundreds of thousands of anything pretty much

now jquery is more accessible I might re-think pagination...

if you don't deallocate the dom elements, does it matter if they're actually in the dom or not?

no, I guess not. I remember reading something about reflowing being affected but I can’t remember the details

i was thinking no

it mostly matters during debugging because they are in the way so u cant see the tree from the forest

yeah it would be nice if dev tools could shade the hiden things

either way I am hoping to have cells backed by those thousands of entities and pagination will be local

exactly

it's trivial to implement pagination on top of cells

the only thing you lose there is command-f searching

which sucks but it's a tradeoff

colin.yates: im preparing a repo to showcase these kind of things: https://github.com/exicon/webapp-skeleton

command-f is only a partial solution, so i'm willing to forgo it

we also hit some performance issues because we had ~400 row list of mobile apps with their icon and ratings in the DOM 3 times...

and work on better typeahead widgets

@onetom: nice - thanks.

@onetom: that is awesome

this is strictly desktop (actually Chrome)

brb

a chrome extension would give you huge power to make a really computed thing

because you can have encrypted database files and stuff locally

you'd be able to really leverage the event sourcing append-only log style

since you can cache and rollup in the client whenever you want and save it to disk where oyu have unlimited space

a real peer to peer architecture

i think something like that is maybe the holy grail

micha: thanks confirming the :refers feature. it worked indeed: https://github.com/exicon/webapp-skeleton/commit/99917510e84709efce7a96b72b0217c6317534e5

i use it to bring in cljs-console

so i am not tempted to use console.log() anywhere

and it ends up in production of course

lol

whats that? sounds something what i would like to have too

like oh i'll just add a debug log here real quick and remember to remove it in 5 minutes

and i never remember until someone comments about it in QA

u r clearly not pairing then

https://github.com/adzerk-oss/cljs-console

yeah mostly not

we only have 7 programmers, and people need to move around a lot

we always go thru the diffs before we commit, so console logs practically never slip into the code

we are also 7 programmers including myself... which is 3.5 pairs...

someone "pairs alone" as one of my colleagues said the other day

I don't like magical refers, make things hard to grep 😕

gotta go - back tomorrow - it’s been great all - learnt loads

mynomoto: i would like to define a loop-tpl replacement. u wouldnt want to require that explicitely, would u? 😉

my pleasure @colin.yates

I would want to do (:require [hoplon.core :as h]) at all places everytime.

@mynomoto: we can make it optional

we just need to factor the boot task a little more

currently the boot task does only a few things

one thing it does is generate the boilerplate for the html file and the .cljs.edn file based on a single .cljs.hl page file

@micha: no problem, if if find the time I may look at it myself.

the other thing it does is add the refers to the generated cljs namespace

but otherwise the generated cljs namespace is unaltered

boot-hoplon is quite clear now (or i just read it enough times 😉 at least im satisfied with the speed i could discover this :refers thing.

@micha: decoupling those will make it possible to use hoplon as an api to create dom things. I think this is already possible, only no examples around.

yes it's already possible

we do need to do some work on the polyfill i think though

i'd like to make the polyfill lazy

i.e. alter dom element instances instead of the element prototype

and only when we need to

which would be when you add a cell child to an element

like if you use loop-tpl etc

btw, just the other day we used an svg dom element generated by a vanilla js lib as a function: https://github.com/exicon/hoplon-chart-examples/blob/master/src/index.cljs.hl#L149-L155

if you don't use loop-tpl there is no need to apply the polyfill

i felt super powerful that moment 😉

which would ensure that if you "mount" hoplon in another app it won't be causing any ill effects outside the mountpoint

@micha: what ill effects?

if there is a lot of churn in the dom it might cause performance issues

i dunno, it just seems like something we'd want to avoid

Yeah, make sense.

awww, shit, the conversation about javelin cells and whatnot has scrolled out of my slack buffer ;(

that chart thing is super cool

the tooltip

it's a nice clash of the hoplon and the jquery worlds, no?

@onetom: http://clojurians-log.mantike.pro/hoplon/2015-09-27.html

we took the example from https://gionkunz.github.io/chartist-js/examples.html#behavior-with-jquery and hoplonified it a bit

@mynomoto: phew thx! i saw this site once but havent bookmarked it so never found it again. how do u find it usually? and who is operating it? doesnt look official (as-in slack related) is it opensource? can i run it for our company slack account?

No idea, I just remembered seem it and googled it to find the link 😉

indeed 1st hit... last time i google i couldnt find... 😕

micha: i was bitten again by loop-tpl binding behaviour. i haven't wrapped my seq into a formula cell... i guess it's a negligible overhead from performance perspective, so i would make it the default behaviour. can't really think of a case where i wouldn't want to do that.

(loop-tpl :bindings [[sub-label sub-path] (cell= (partition 2 items))]
                 (menu-item ... ))
vs
               (loop-tpl :bindings [[sub-label sub-path] (partition 2 items)]
                 (menu-item ... ))

yeah i can see that

and it would also be fine if items was not a cell

if i dont want reactivity i would just use map probably...

yeah that's true

the only issue i can see is like

(loop-tpl :bindings [thing (create-things-cell items)]
  ...)

where create-things-cell is a function that returns a cell

probably not a big deal

and items?

items could be a cell

but u were just discouraging functions returning cells a bit earlier, right?

yeah i mean sometimes you need to do it

so sometimes (which is the rarer case) u would just

(let [things (create-things-cell items)]
   (loop-tpl :bindings [thing things]
      ...))

yep that would work

it's just a little harder to understand what you did wrong if you forget to do that

harder than: "Uncaught Error: [object Object] is not ISeqable" ?

because that's all i get when i forgot the (cell= ) from around that (partition ...) call

i think a tiny bit of better error reporting here could have a huge impact on adoption

btw, i also get a completely blank page in this case

the [object Object] part is a hard thing to fix

because it's being generated by javascript

i mean by cljs core anyway

it should be showing you the cljs representation

Uncaught Error: #<Cell [1, 2, 3]> is not ISeqable

that would give you all the information you need

[object Object] doesn't tell you anything about cljs things

yeah, that would be quite a useful error message

but gnite for now (it's almost morning here) thanks a lot for the tons of smartness u poured on us today!

see you later, i hope it wasn't too bad

what do u mean by too bad?! it was awesome!

(pour :some [thing things]
   (elem thing))

ah pour, i like it

later!