Fork me on GitHub
#hoplon
<
2016-01-18
>
micha00:01:03

@alandipert: isn't that what cycle + rest/ first does?

alandipert00:01:23

Yeah but the explicit rotate would make super clean I think

micha00:01:11

(defc things ["one" "two" "three"])
(defc= slides (cycle things))
(defc= slide (first slides))
(with-interval 1000 (swap! slides rest))

alandipert00:01:46

My rebuttal is quite elegant but can't fit in the margin of the iPad

alandipert00:01:54

I leave it as an exercise

meow00:01:38

oh my, not enough room in this channel for my ego

laforge4901:01:58

All that is left now is to update the api doc and then do the releases.

laforge4903:01:32

[ANN] notify 0.1.1 uses a decay timer to reduce polling by inactive clients.

levitanong07:01:48

@alandipert: pardon my intrusion—while the rotate would make it super clean, afaik there’s no rotate function in clojure. Wouldn’t you have to code it in manually?

thedavidmeister10:01:29

hey, if i'm not planning to use cell=, is there an advantage to cell over atom (or vice versa)?

thedavidmeister10:01:59

is there a way to respond to a Hoplon element being rendered?

thedavidmeister10:01:23

i got a 3rd party library sticking something in the DOM, which gets blown away by Hoplon

thedavidmeister10:01:48

would be good to have it do it's thing after Hoplon builds the page

thedavidmeister11:01:23

ah, so the actual issue is that anything i pull in as a dependency with cljsjs, that does something to the page as it is pulled in

thedavidmeister11:01:31

will get blown away when Hoplon does its thing

genRaiy12:01:54

quick question on Hoplon and Server Sent Events (SSE) … I know that the architecture supports one way data updates / CQRS and SSE is a very natural way to achieve this from a READ perspective … do you have any thoughts on how the RPC mechanism could be hooked into SSE data rather than a web socket?

genRaiy12:01:39

perhaps I’m missing something on how state changes from client X are distributed to client Y and Z without Y and Z needing to poll for the state changes

laforge4913:01:14

The fact that hoplon/notify uses polling (and not even long poll) to receive notifications (events) from the server is an embarrassing testimony to my ignorance. But I'm a back-end java dev with minimal web-server experience and a clojure newbie. @raymcdermott: I'd love to move notify to more appropriate tech!

laforge4914:01:54

Now you may have noticed that in notify we use registered functions to process notifications. This is because notifications can be changes which require updates to client state rather than being updated client state. The problem with tying notifications directly to hoplon/javelin input cells is that multiple identical notifications (add 1 to the page counter) would be deduped, resulting in the loss of significant notifications. Fortunately it is very easy in hoplon to write a function which adds 1 to an input cell using swap!

laforge4914:01:29

Really, I am thinking that the ideal solution here would be to have alternatives to notify which use the same API but different transports.

genRaiy14:01:03

I agree - let’s hear what the giants have to say when they awake from their slumber 😉

meow15:01:37

I added Hoplon here - would love to see a presentation at Clojure/west 2016 https://github.com/clojurewest/clojurewest2016/wiki/Suggested-Topics

laforge4915:01:54

As a non-web dev, I find hoplon very hot. But the biggest problem is being limited to just castra. Easy enough to integrate other transports into hoplon, except it hits on my personal weakness as a non-web dev. Clojurescript is a boon too, as I'm weakest when it comes to JS. But I especially love hoplon's lightweight approach to reactive web pages. (Though the demos seem especially weak in this regard!)

laforge4916:01:55

[ANN] notify 0.2.0-- Notify now includes the timestamp of when a notification was created on the server. Functions registered to process notifications on the client now take that timestamp as a second argument. The castra-notify-random and castra-notify-chat demos have both been updated. No further changes are planned for notify at this time.

micha16:01:47

@thedavidmeister: can you describe what you're wanting to do a little more?

micha16:01:48

hoplon can coexist very well with other js libraries, but there may be things you need to do, depending on the assumptions those libraries make

meow16:01:22

I just know that @micha and @alandipert are brilliant developers

micha16:01:22

@laforge49: to do full event-driven backend i think we'd need to go a completely different direction

micha16:01:49

i don't usually work on facebook type applications

micha16:01:00

i work more on AWS console type applications

micha16:01:08

or gmail type applications

micha16:01:17

you know, business frontends

micha16:01:29

that means you don't have billions of users all talking to each other

micha16:01:49

and expecting real-time updates from any random user to another

micha16:01:09

for business type applications castra works well for me

micha16:01:00

i would like to make some middleware that sits between the castra middleware and the castra client, that uses SSE or websockets

micha16:01:24

it would still be request-response, i.e. the client makes a request and gets a response

micha16:01:40

but the middleware would keep track of the client's state

micha16:01:52

then it could pass only patches across the wire

laforge4916:01:49

For now, I've got what I need to start working on some aatree demos. I really should let someone else run with the ball re transport. simple_smile

micha16:01:24

yes i want to see some aatree demos!

micha16:01:40

will you add them to the hoplon demos?

laforge4916:01:23

Nothing fancy to start with, though I still very much want to work towards db change notification subscriptions for clients.

laforge4916:01:14

Another thing I keep thinking about is reworking aatree, immature as it is, to run in clojurescript as well. My todo list then is way too big, which is why I wanted to stabilize notify for now and start using it.

micha16:01:49

i wonder if it would make sense to implement it in clojurescript first, so you can make demos to see how it works in applications

micha16:01:04

and then when you have it all pretty much nailed down port to clj

laforge4916:01:11

I see aatree as having 3 layers. The first layer is done for clojure. Layer 2 is very much like datomic. Layers 2 and 3 are implemented in Java, where layer 3 contains things like access control.

micha16:01:00

is layer 1 there the "outermost" layer?

micha16:01:03

like the API?

micha16:01:41

like backwards compared to the OSI layer scheme?

laforge4916:01:44

no, the lowest layer. It gets hidden by the higher layers. But it is also usable independent of them, at least the clojure version is.

micha16:01:59

yes i'm tracking

laforge4916:01:13

With layer two I do time, entities and entity relationships. And time is complets, whereas datomic only covers the smallest part of time.

laforge4916:01:15

So you can see changes to an entity or relationship over time, and navigate freely between those views and datomic-style time-travel.

laforge4916:01:44

--Datomic only has 1 time index, whereas I use 3.

laforge4916:01:56

But I really want to get layer 1 finished. It has use cases which are not supported by the other layers.

laforge4916:01:58

What is currently lacking in layer 1 of aatree is journaling. Every update needs to first create a journal entry which can be archived and then processed. This facilitates fast recoveries, hot backups and data mining.

laforge4916:01:36

But before going any further, I want to add a back end to the chat demo . simple_smile

micha16:01:51

sounds pretty awesome

laforge4916:01:11

Been working on this for over a decade.

laforge4916:01:42

But I'm pretty burnt on working alone. So whatever I do needs to be accessible.

laforge4916:01:10

Success only comes from the cross-pollination of different perspectives and different skill sets. 😄

laforge4916:01:20

I am also quite excited about a deep integration between a database and reactive web clients.

micha16:01:17

yeah cross-pollination is key

genRaiy18:01:24

@micha: in my experience business front-ends can also benefit from shared real-time updates. Stock level or booking system are the pretty common examples. Dashboards and other visualisations greatly benefit from push style events.

micha18:01:50

sure, but that's very different from facebook say

micha18:01:59

where users are talking to each other in real time

micha18:01:06

it's a much simpler problem

micha18:01:28

polling works really well there, since the individual updates are essentially stateless

micha18:01:46

yeah i mean that's a brutal solution

micha18:01:03

because you then need to couple all of your code to the db schema

micha18:01:16

you lose all encapsulation and layered architecture stuff

micha18:01:28

with say a stock ticker, for example

micha18:01:36

that's very well suited to sse

micha18:01:26

and with castra for instance, you can wrap it in middleware that keeps track of the client's state, either by keeping a websocket reference or by request sequence numbers (related to tx ids in datomic)

micha18:01:48

the middleware is an optimization to reduce the traffic sent over the wire for updates

micha18:01:05

the middleware can compute the diff of new state / old state

micha18:01:08

and just send that

micha18:01:16

and middleware in the client can apply the patch

micha18:01:37

that still allows your business logic and your rpc endpoints to be stateless

micha18:01:56

the state is essentially boilerplate then, implemented in the middleware

micha18:01:30

so for a stock ticker the most naive thing would be to just poll the backend

micha18:01:44

that will provide real time updates

micha18:01:20

to optimize you can add middleware in the server that polls the castra endpoint, so the client doesn't need to send requests over the wire

micha18:01:52

with that optimization the client still polls, but the responses are not returned to the client until the state changes on the server

genRaiy18:01:58

ok, so a little like conditional GET in HTTP?

micha18:01:02

so client calls endpoint the first time. the middleware calls the castra endpoint and collects the result. it then compares to the last result it has seen for that client/endpoint pair. since there are no previous requests that will be nil. it then computes the diff of previous and current, and if there is any difference sends the diff across the wire to the client

micha18:01:37

now the client is polling, so it makes a second request, with a sequence number that corresponds to its previous state

micha18:01:02

the middleware on the server does the same thing it did before, calling the castra endpoint

micha18:01:24

but suppose nothing changed on the backend, i.e. the castra rpc function returns the same thing it returned last time

micha18:01:38

in this case the middleware goes into a loop

micha18:01:06

periodically calling the castra endpoint and computing a diff of the result with the stored previous state of the client

micha18:01:32

it doesn't return any result yet

micha18:01:36

(this is the sse part)

micha18:01:45

long polling basically

micha18:01:13

it continues to loop, calling the castra rpc function until something changes

micha18:01:22

when that happens it sends the diff as a response

micha18:01:39

and middleware in the client applies the patch to its own state

micha18:01:20

this kind of serverside polling should be pretty scalable for databases like datomic where reads are cheap and local to the peer

micha18:01:44

and where state can be described by an immutable database value or tx id

micha19:01:14

basically the client can send a request to a castra endpoint with some immutable value that describes its current state

micha19:01:46

and the server delays responding until the endpoint returns something other than the client's current state

micha19:01:25

i dunno if that makes sense at all

genRaiy19:01:04

we need a whiteboard 😉

micha19:01:16

haha yeah

micha19:01:29

i have one actually

genRaiy19:01:32

so you’re saying do polling on the server-side

micha19:01:36

exactly yes

genRaiy19:01:11

using the tx-report-queue might obviate that for Datomic

micha19:01:18

this allows you to preserve the vastly simpler stateless architecture for your backend application

micha19:01:44

yeah but the tx queue is too low level

micha19:01:14

it's a huge simplification to organize your backend as a namespace of functions that the client can call

genRaiy19:01:16

it does indicate change but needs a lot of parsing

micha19:01:27

well it's the meaning that's the problem

micha19:01:42

like there is no a priori relationship between functions and datoms

genRaiy19:01:06

agreed, it is too general

micha19:01:10

if you have to declare that relationship, which you would, then you've defeated most of the benefits of the lisp abstractions

micha19:01:40

like imagine if you make programs like that

micha19:01:52

like externally mapping vars to functions

genRaiy19:01:58

I think I need some schooling there ...

micha19:01:31

i mean a normal clojure program has functions that compute things, referring to various vars and whatnot to perform their computations

micha19:01:46

i think the tx queue is similar to adding watches to vars

micha19:01:58

and trying to figure out which functions will need to be called when they change

micha19:01:11

your program would become spaghetti very fast

micha19:01:18

essentially GOTO all over the place

genRaiy19:01:54

via core.async 😛 but yes, your point is well made

micha19:01:15

with the serverside polling you'd probably want to use core.async

micha19:01:21

or some kind of fibers

micha19:01:38

so you wouldn't need to have a dedicated thread for it

micha19:01:44

i mean for each client

genRaiy19:01:12

sure I just meant that we could use flashy libs and end up with GOTO equivalents

micha19:01:27

haha yeah it's easy to end up there

genRaiy19:01:08

ok I like the idea of Datomic clients sending the database value as that is trivial to compare with the current value so making queries against databases that are different will usually have the chance to obtain novelty

micha19:01:59

yeah the state of the client could be associated with a datomic tx id

genRaiy19:01:06

would that just be some meta data on the state

micha19:01:07

and the client could send that with the request

micha19:01:28

the server then performs its query, and compares to a query against the client's tx id

micha19:01:33

and computes a diff

genRaiy19:01:20

yes, it leaks out that we’re using Datomic but I don’t think that’s a huge issue … hiding this stuff has wasted 20 years of enterprise computing

micha19:01:35

the tx id is just a uuid

micha19:01:43

it doesn't need to be datomic specific

micha19:01:48

it could be anything

micha19:01:07

it could be a key corresponding to something in redis, who knows

micha19:01:24

like my current project is using sql server

micha19:01:39

we could maybe store client state in memcached

micha19:01:40

or something

genRaiy19:01:44

well we know, but yes, just put a different generic label on it

micha19:01:45

and still be able to compare

micha19:01:18

datomic makes it all much cheaper

micha19:01:22

and easier

genRaiy19:01:34

what would be a good entry point in the Castra middleware for that …. or it just another defrpc?

micha19:01:48

i would make middleware that wraps castra

micha19:01:59

i think they're separate concerns really

micha19:01:24

the middleware could do the polling by repeatedly calling the next handler

micha19:01:36

which would be castra, presumably

micha19:01:55

it would just keep track of the client's state

micha19:01:18

and transform the result in the :body to be a patch instead of the whole state

micha19:01:37

and in the client you'd need some middleware between the castra client and the ajax call

micha19:01:47

that would apply the patch on the client side

micha19:01:02

and pass the patched state up to the castra client code

micha19:01:31

the client middleware would handle the tx metadata, too

micha19:01:26

i don't think anything would need to change in your serverside or clientside code really

micha19:01:31

other than adding the middleware

genRaiy19:01:55

I guess there isn’t anything like that on "the market” so let me have a think and a play … great ideas and as you say, solves for most of the problems without binding the client into the low level schema

genRaiy19:01:56

BTW @micha: if you draw it up on the whiteboard, post a photo on the channel!

micha19:01:50

an interesting thing we have been thinking about is the "s3c", the ServerSide Stem Cell

micha19:01:31

like if you have a function that only makes a datomic query, there must be a way to make that like a formula cell

micha19:01:35

on the server

micha19:01:47

by watching the tx queue etc

micha19:01:22

that would still allow encapsulation etc, because it would be a general solution involving analysis of the datalog query perhaps

micha19:01:41

then you could create formula cells for the client in the server

micha19:01:53

and directly map them via websocket or sse

genRaiy19:01:53

now that would be something

genRaiy19:01:19

that really would be something

micha19:01:25

yeah that's like a pie in the sky dream that won't die

genRaiy19:01:05

the concepts are circling the same need

genRaiy19:01:34

I have this view, tell me when in changes

micha19:01:58

(defquery [[?q ...]] conn state-cell)

genRaiy19:01:05

I have this view, is it still OK?

micha19:01:28

that would hook up to the tx queue and reset! the state-cell whenever the result of that query changes

genRaiy19:01:46

I was hearing about rethinkdb and they have something like this

genRaiy19:01:07

some query that is automatically recalculated on the server and patches pushed out to clients

micha19:01:38

yeah this is what we want

micha19:01:51

we need more than that though

micha19:01:08

we need formula cells because it might not be as simple as just the query

micha19:01:18

like we want composable things

micha19:01:36

but the underlying primitive is the "reactive query" thing

genRaiy19:01:13

yeah, so that changes in the reactive query can fire off other updates in the view

micha19:01:04

well you'd have a formula cell in the server that is synced with a cell in the client

micha19:01:16

when it changes on the sever the client is updated too

micha19:01:44

castra provides this, but you have to poll to get changes

micha19:01:08

the "formula cell" in the server is a function

genRaiy19:01:34

yes, that’s our conversation 😛

micha19:01:50

i wonder if watching the tx queue is actually more expensive than polling

micha19:01:05

because you'd need to do more work, potentially

micha19:01:13

or you'd need to process the queue in batches

micha19:01:22

poll the queue essentially

genRaiy19:01:51

the queue is a LinkedBlockingQueue

genRaiy19:01:05

so it’s ‘cheap’ to poll

micha19:01:13

what i mean is in terms of waking up threads

micha19:01:18

and doing work

micha19:01:35

like with simple serverside polling you wake up every now and then and make a datomic query

micha19:01:53

is that more expensive than waking up to observe the tx queue?

micha19:01:02

and deciding whether to make the query or not?

micha19:01:58

i suppose if you did the query analysis the reactive query could register itself with a global tx queue processor

micha19:01:13

and then go to sleep until it's called

genRaiy19:01:46

checking if the queue has anything new should be low cost

genRaiy19:01:40

global tx queue processor … maybe that is starting to sound quite stateful?

micha19:01:59

you'd also need to know if any of the new things are interesting

micha19:01:08

which could be less low cost

genRaiy19:01:33

yes, and as we agreed earlier, hard to generalise

genRaiy19:01:11

just taking the query and running it is cheap on the Peer or at least its cost is understandable

micha19:01:46

yeah you can tune your memcached or whatever

micha19:01:54

to make repeated queries cheaper

genRaiy19:01:11

the nice thing is that the performance cost is in the hands of the function composer

laforge4919:01:27

I see being able to qualify what constitutes an interesting change, and then filtering the transactions for changes of interest to all the clients. Only when there is an interesting change do you do the underlying query to see if it is really a change of interest and what the change is from the client's perspective.

genRaiy19:01:39

there isn’t any hidden weirdness

micha19:01:42

yeah putting the tradeoffs in the function composer is very LCDI

micha19:01:56

which we like

genRaiy19:01:36

@laforge49: I think we are struggling with working out how could that be achieved

laforge4919:01:30

And what I'm talking about is only a further optimization. simple_smile

laforge4919:01:24

Also, I tend to look at things from the perspective of a db internals dev, which is not at all useful in many cases as the overall architecture should generally be db agnostic. 😄

laforge4919:01:44

But yeah, we all share the same excitement. A closer integration between reactive web apps and databases via a new framework.

genRaiy19:01:56

the graphQL folks are also circling this problem of course

laforge4919:01:11

I just think there are advantages to using the db journal entry stream to trigger activities, like a kind of data mining.

genRaiy19:01:26

the difference here is the cell model

genRaiy20:01:26

yes, datomic tx-queue, mongo oplog, rethinkdb queries are all pushing on the same problem

laforge4920:01:30

And having things like distributed datomic peers helps keep you from dying of query poll overhead.

genRaiy20:01:21

yes, the fact that txn infrastructure is first class matters

micha20:01:31

it's very hard to beat a stateless server in terms of spaghetti elimination

laforge4920:01:34

though when you say tx-queue, you really are talking about the same thing as I am.

genRaiy20:01:20

yes, I know - I used to be a DB engineer too so we probably share many of the same perspectives

genRaiy20:01:54

Ingres, back in my day 😉

laforge4920:01:05

gotta run an errand. You'all enjoy now. simple_smile

micha20:01:29

the main assumption of the castra model is that interesting changes occur as a result of some action by the user, not spontaneously

micha20:01:47

which is mostly true for business applications

genRaiy20:01:19

multi-user though

micha20:01:22

and it's flexible enough to be useful with whatever storage/database layer you have

micha20:01:46

i think AWS console is a good example of a scalable multi-user application

micha20:01:08

it's not a toy, like facebook. people do real work on there

genRaiy20:01:11

but talking it through, I’m convinced that polling - at some layer - is going to get us out of jail

micha20:01:33

yeah you can see some interesting things in the way amazon does it in their console UI

micha20:01:51

they have a mix of polling, explicit "refresh" buttons, etc

micha20:01:04

i think at the end of the day that's what you end up having to do

genRaiy20:01:38

the AWS console really sucks 455 though as a design

micha20:01:01

i mean it has really difficult scaling problems to solve

micha20:01:18

you can't have say a combobox with all your instances in it

micha20:01:23

cause maybe you have lots of them

genRaiy20:01:25

I mean the way that you have so many different navigation choices and looks all mashed together

micha20:01:48

yeah i'm talking about the engineering i guess

micha20:01:00

like imagine a world in which everything must be paginated

micha20:01:12

like even the list of pages in a pagination widget!

micha20:01:27

you might have hundreds of pages of something, so you can't even do that the simple way

micha20:01:52

i think this is where the rubber meets the road for real

micha20:01:39

in this distributed system kind of world i think polling might actually be the best option

genRaiy20:01:41

I suppose that’s why many bespoke clients exist for specific AWS use cases

genRaiy20:01:16

once you have to visualise 1000s of things pagination makes no sense anymore. You hit a google search problem, nobody goes past the first page

micha20:01:46

typeahead etc so key

genRaiy20:01:50

everything is tuned to getting the result in the first hit (hence query predictions)

genRaiy20:01:04

we needed that just then simple_smile

genRaiy20:01:36

ok, so how does that all fit into s3c?

micha20:01:00

well i guess s3c isn't as awesome as it might seem, maybe

micha20:01:08

i go back and forth with it constantly

micha20:01:15

so it needs to be built, just to see

genRaiy20:01:01

as long as you can pass it a query to resolve, it could be fine - like you say it has the job to calculate whether the client needs updating

genRaiy20:01:11

might actually be quite easy

micha20:01:15

i'm pretty convinced though that we need to have a menagerie of state machines in the client

genRaiy20:01:19

as least doing it that way

micha20:01:23

like a state machine zoo

micha20:01:39

we started developing some at adzerk for our application

micha20:01:55

like we have a general purpose form-machine

micha20:01:05

this is a stateful thing that implements a state machine

micha20:01:29

exposing the knobs as functions that operate on it and cells by which it exposes its current state

genRaiy20:01:37

ok yes, so some all reacting inside the client cage and some being allowed to interact with external entities like the server

micha20:01:45

so you can call the state transition functions

micha20:01:57

the machine knows how to handle errors, for one thing

micha20:01:00

which is huge

micha20:01:12

because in a distributed system ther are many different ways you need to do this

micha20:01:21

like can you retry?

micha20:01:27

what about out of order results?

micha20:01:30

things like that

micha20:01:41

having a pre-made zoo is key

genRaiy20:01:03

but like you say you are mainly just focussed on the thin pipe between the client server and not the general problem of scheduling distributed computing

genRaiy20:01:32

so that zoo has a chance

micha20:01:49

the goal here is to make it comprehensible and simple to use

micha20:01:01

like a few key abstractions that can be composed

micha20:01:00

we didn't do any optimization of the deeper things, we just did the thin pipeline state machine stuff

micha20:01:08

and our app is super responsive and fast to use

genRaiy20:01:54

that’s what we need - most of the deep optimisations are chimeras

micha20:01:22

this is a screen in our app

micha20:01:39

it has a datatable with pagination, search, etc

micha20:01:52

and you can update any of the things with a modal form

micha20:01:13

upsert-machine is a subspecies of the general form-machine

micha20:01:39

and all of our hoplon form components can work with any form machine

micha20:01:50

does validation, etc

micha20:01:55

shows errors in the right place

micha20:01:10

modal goes away when something succeeded

micha20:01:37

also fancy things like clicking outside the modal makes the modal go away, but only if you don't have unsaved changes in the form

micha20:01:52

this is all hooked into the form machine in a pretty straightforward way

micha20:01:40

we use an interesting pattern of macros + dynamic bindings

micha20:01:54

like screen/with-screen there, is a macro that sets dynamic bindings

micha20:01:08

same with form/with-form

micha20:01:42

various hoplon elements are implemented to get default attribute values from dynamic vars if they're not explicitly provided

genRaiy20:01:52

that’s a bit of a blizzard to be honest!

micha20:01:04

so the (form/validation-message) thing in there doesn't have any configuration at all

micha20:01:20

because it gets its config from the dynamic vars bound by with-form

micha20:01:35

this means that you can put that guy anywhere, like nested inside 50 divs, whatever

micha20:01:54

and you don't need to pass through the varibales it needs, or explicitly set attributes

micha20:01:14

like the with-form macro binds the form machine

micha20:01:47

and various elements in the body (possibly deeply nested) will be able to access the form machine to do their work

micha20:01:12

this is why we don't need to explicitly configure the text inputs with where their values go

genRaiy20:01:15

how do you link fields together (so for example password must be the same in both fields)?

micha20:01:28

that's a validation concern

micha20:01:39

the default behavior is to do validation on the backend

micha20:01:43

since you need that anyway

micha20:01:59

castra has a *validate-only* mode

micha20:01:16

the form machine handles the details of this

genRaiy20:01:41

ah OK that’s maybe why FRP UI is not so visible in the demos?

micha20:01:51

the form machine will not enable as-you-type validation until you've submitted the form once

micha20:01:06

but if you submit and there is a problem

micha20:01:15

then every keypress triggers validation

micha20:01:25

(there is some throttling, but it's transparent)

micha20:01:41

and the errors in the form update as you type

micha20:01:12

well in the example i pasted we've made some very high level abstractions

micha20:01:21

so you hardly see any of the plumbing at all

micha20:01:38

that page there is completely declarative, in that way

micha20:01:46

but it's not like a brittle template

genRaiy20:01:05

maybe this is a philosophy thing but I think clients should be smart enough to help the user immediately rather than deferring to the server. It’s wise to distrust the client but that doesn’t mean that the code should mistreat them 😉

micha20:01:19

yes, that's also supported

micha20:01:32

the form machine can be configured with clientside validation fns too

micha20:01:51

those are only used to tell the user that something is not valid

micha20:01:59

which is a key aspect of a sane ui

micha20:01:09

only the server can tell the user that something is okay

genRaiy20:01:09

is that work all proprietary to adzerk?

micha20:01:30

that way you don't have the ui telling the user that everything is fine, and the server rejecting the request

micha20:01:13

we're going to open source it

genRaiy20:01:57

yes, the balance between the client smarts and the server truth is a balancing act

genRaiy20:01:14

that’s great to here (re open sourcing)

micha20:01:15

but we've found that there isn't much to be gained from clientside validation in our app really

micha20:01:44

the server is plenty fast enough, the user can't tell

micha20:01:54

the diference

genRaiy20:01:58

classic example I had (and is the canonical reactive form) is the user registration form

micha20:01:09

yeah that always needs serverside validation anyway

micha20:01:24

because you can't ship a list of all used email addresses to the client simple_smile

genRaiy20:01:38

is a username available … exactly

micha20:01:09

i mean every server endpoint needs to have solid validation on it

micha20:01:17

so i think it makes sense to start there

micha20:01:26

and add clientside validation if there is some need to do so

micha20:01:35

we just haven't found the need at all

genRaiy20:01:37

definitely, especially with the security fears these days coming to the fore

micha20:01:00

if you do like

micha20:01:23

(binding [castra.core/*validate-only* true]
  (my-rpc-fn ...))

micha20:01:26

from the client

micha20:01:42

that will send the rpc request to the server with some metadata in the headers

micha20:01:58

the server will see that metadata and run the rpc endpoint in validation mode

micha20:01:10

that means all the preconditions like :rpc/pre will run

genRaiy20:01:14

ok so a kind of rehearsal mode

micha20:01:21

but the body of the rpc function will not be evaluated

micha20:01:56

this way if validation fails (validation is implemented as :rpc/pre expressions) you get an exception in the client

micha20:01:04

the same as if you had submitted the form

genRaiy20:01:04

is that possible now?

micha20:01:08

yes, it's in there

micha20:01:59

serverside validation is pretty much free with this model

genRaiy20:01:03

@micha: I could talk all night about this stuff but I have to go off and do a few family things

micha20:01:24

later dude!

genRaiy20:01:45

I will look into this stuff and I have learned a lot through this conversation

micha20:01:06

me too, thanks!

micha21:01:06

:incomplete is the initial state

micha21:01:35

yeah this thing lacks any kind of real state machine abstraction

micha21:01:21

did you find the DSM abstraction you used there to be helpful?

micha21:01:46

i mean over the naive just code a thing that happens to be a DSM

meow21:01:18

yes, I think the DSM is one of the really good things in QML

meow21:01:36

learned a lot working it out using a canonical calculator example

micha21:01:45

can you compose them?

micha21:01:17

like my form machine thing seems like it could be factored into a few smaller machines that are composed to form the whole

meow21:01:24

hrm, well, they are hierarchical

micha21:01:10

the formalization lets you be confident that you've covered all the possible states, is one benefit?

meow21:01:17

well, it goes back to good design in the end - most of the examples online and/or in the HSM standards are awful

meow21:01:33

which I learned by making a calculator that actually works

meow21:01:52

Donald Knuth said something to the effect that it took him 6 tries to get the state machine for the elevator in his building defined correctly and even then he basically gave up for fear that he would never get it finished

micha21:01:14

the traffic light example is also a good one

micha21:01:25

a real mental rabbithole of uncertainty and doubt

meow21:01:50

I refactored that calculator many, many times

micha21:01:00

do you know of a good state machine library for cljs?

micha21:01:15

i looked around and didn't find anything that appealed to me

meow21:01:15

pretty happy with it, but then decided QML was a dead-end and switched to Clojure

meow21:01:40

nope, haven't seen anything compelling in Clojure

meow21:01:54

ztellman might have something

micha21:01:38

yeah i've tried to use that

micha21:01:49

but it's not clear to me how that relates to my thing there

micha21:01:04

like i can see how you'd make a parser or something with it

micha21:01:45

have you seen this?

meow21:01:34

read it - not crazy about it

micha21:01:47

what was not good?

meow21:01:08

read pretty much everything I could get my hands on about HFSMs

meow21:01:30

the theory is easy, putting it into practice less so

micha21:01:48

lisp though

micha21:01:58

the theory is the practice in lisp

meow21:01:04

yep - maybe I'll give it another shot some day

meow21:01:20

the DSM in QML is awesome

micha21:01:46

do you have a good intro to state machines reference you'd recommend?

micha21:01:10

like for a person who knows nothing about it

micha21:01:14

asking for a friend simple_smile

meow21:01:27

hahahahahahaha

meow21:01:36

hang on, just peed in my pjs

meow21:01:09

so what you should tell your "friend" is that there are no good references

micha21:01:35

they'll be disappointed

meow21:01:43

that book isn't too bad

meow21:01:55

friends are always disappointed

laforge4921:01:03

I find documentation in general is essential. And the more the better. Be it state machines or anything else. It lets you develop a different perspective on the code and that is when the bugs start popping out.

meow21:01:02

if you have the kind of problem that a state machine is good for then a state machine is awesome - better than all the unit tests in the world

laforge4921:01:16

But I find in general that it is difficult to abstract reasonably simple state machines. And trying to be complete is just too overwhelming!

micha21:01:23

yeah unit tests are especially useless for stateful things like that

micha21:01:45

tests have just as many bugs as the code!

meow21:01:06

take a look at my calculator code example and you will see, imnsho, a good example of a fully functioning trivial/non-trivial system

micha21:01:19

yeah i'm trying to work through it

meow21:01:42

the gui is just a dumb layer on top

meow22:01:07

that enables/disables keys that are valid/invalid depending on the current state

meow22:01:34

all the logic is in the machine

micha22:01:58

a nice thing is that the machine is completely decoupled from actual calculating

micha22:01:07

like i don't see any + implementation there

meow22:01:26

its in there somewhere

laforge4922:01:33

What I dislike about statemachine ish software is that it generally composes badly--the whole becomes quite fragile. This is how I characterize actor systems. As soon as you use state to filter message processing, you've got intimately coupled state machines.

micha22:01:41

i mean in StateMachine.qml

meow22:01:19

@laforge49: I agree and felt like developing the state machine was a very fragile process

micha22:01:23

@laforge49: the form-machine thing has been a huge simplification in our application

micha22:01:39

it's like an order of magnitude difference

laforge4922:01:56

cool. Hey, use what works where it works. simple_smile

micha22:01:59

in terms of being able to understand complex stateful stuff, like any real world application will have

micha22:01:28

like we code all the things that interact with the server to delegate to a state machine

micha22:01:44

that manages the state of the session, more or less

laforge4922:01:54

That's where I prefer synthesis to analysis--keep it simple so you can understand it

meow22:01:01

DSM.SignalTransition {
                signal: keyManager.addSubPressed
                targetState: operatorState
                onTriggered: processor.calculateAll(false);
            }

micha22:01:19

maybe the quest for a formal representation is flawed

micha22:01:32

maybe ad-hoc is actually better

laforge4922:01:10

A good semantic model wins hands down. This is why naming is key in software development.

laforge4922:01:58

I find I keep changing the meaning of things as the code matures.

laforge4922:01:28

--refining the semantic model

meow22:01:06

function calculateAll(updateExpression) {
        var num = operandBuffer.number;
        var operator;
        if (operators.length === 2) {
            operand = operands.pop();
            operator = operators.pop();
            switch (operator) {
                case "*":
                    operand *= num;
                    break;
                case "/":
                    operand /= num;
                    break;
            }
            num = operand;
            if (equalKeyRepeatsLastOperation) {
                operandBuffer.update(operand);
            }
        }
        if (operators.length === 1) {
            operand = operands.pop();
            operator = operators.pop();
            switch (operator) {
                case "+":
                    operand += num;
                    break;
                case "-":
                    operand -= num;
                    break;
                case "*":
                    operand *= num;
                    break;
                case "/":
                    operand /= num;
                    break;
            }
        } else {
            operands.pop();
            operand = operandBuffer.number;
        }
        operands.push(operand);
        calculationResult = operand;
        if (updateExpression) {
            expressionBuilder.push("=");
            expressionBuilder.push(calculationResult);
        }
        operators = operators;  // To trigger any bindings to this list.
    }

meow22:01:27

so StateMachine are just the states and Processor has the logic

laforge4922:01:26

state machines do result in pretty clean code... when you only have one. But I think it is because the semantic model is so clear.

micha22:01:46

ok maybe you could tell me what you think of the form-machine i made, in relation to the formal DSM approach

micha22:01:17

i can describe the overall concept or organization

meow22:01:28

gopherit

micha22:01:33

ok cool, thanks

micha22:01:47

so the idea is there is a cljs record type, the FormMachine

micha22:01:13

it implements IFormMachine, which are methods that can be called on the form machine instance, these are the "signals"

meow22:01:18

need a cigarette break but I'll catch up so spill yer guts

micha22:01:36

the form machine's record fields are actually javelin cells

micha22:01:44

so the state of the machine is kept in these cells

micha22:01:08

in the specific case of the FormMachine there are three signals: reset, validate, and submit

micha22:01:18

and when you construct a new form machine with the form-machine function, you provide an action (a function to call that will do the ajax, following a certain convention for reporting error or success)

micha22:01:48

the convention there is that the action when called will return a js promise

micha22:01:18

you also provide a schema which is describing which things could have errors related to them (eg. form fields or whatever)

micha22:01:54

and a cell that represents the current value of the form and a callback to call when form has been successfully submitted

micha22:01:21

the machine has its own fields that are cells, as i said before

micha22:01:43

like (:error the-machine) returns a map of error cells

micha22:01:24

so for instance if my schema is like {:foo nil} then there will be a cell you can get via (get-in the-machine [:error :foo])

micha22:01:36

which will contain the error message for the :foo field

micha22:01:00

so from a very high level:

micha22:01:38

you create the form machine with some configuration that includes functions to call when different states are entered or exited

micha22:01:03

and with some configuration about what kind of state to keep track of internally

micha22:01:12

(this is the schema for instance)

micha22:01:36

then you can call the reset, submit, or validate methods on the machine, and things will happen

micha22:01:58

if things happen it will be reflected in cells that change their values

micha22:01:17

for example the (get-in my-machine [:error :foo])

micha22:01:36

the form machine also can trigger state changes when it's own internal formula cells change

micha22:01:10

derp line 100 and 101 there

micha22:01:18

the idea is that maybe i have a form with username and password fields

micha22:01:38

i can express the form data as {:username (cell ...) :password (cell ...)}

micha22:01:05

and my login! castra rpc function can take that map as its argument, say

micha22:01:44

and if there is an error it could attach {:username "must not be blank"} to the exception it throws

micha22:01:56

in the case that the username was submitted blank

micha22:01:25

so to express this relationship the form machine has the :data and :error fields

micha22:01:37

which are both maps

micha22:01:01

i'm in the weeds, disregard all this

meow22:01:47

welcome to state machine hell - can I get you a latte or toasted bagel?

meow22:01:38

one thing that became clear to me working on the calculator is that the import part is the behavior

meow22:01:57

something is going to happen, like the user hits the = key

meow22:01:28

what does that mean when the = key is pressed, what behavior intent does that represent?

meow22:01:36

depends on the state of the machine

meow22:01:56

focus on variations in expected behavior

meow22:01:11

create states and transitions to reflect that

meow22:01:23

look at my code

meow22:01:06

I will not be humble about it - the code might suck and it sure isn't clojure but I modeled a calculator better than any examples I found online and I think I saw them all

meow22:01:25

took a lot of work to get it

meow22:01:55

then you just group things together and put them in a hierarchy that works

meow22:01:04

easy peasy

micha22:01:57

yeah the heirarchical part is interesting

meow22:01:24

can also be a quagmire

meow22:01:37

I think I did that more than once

micha22:01:51

like class hierarchies

meow22:01:04

if you aren't careful, yes

micha22:01:09

much tuning, and then a new thing presents it self and whoa

micha22:01:28

wheels aren't part of a car now, they're a separate thing, uh oh

meow22:01:40

like I said, eventually it all comes back to good design

laforge4922:01:52

@meow: What you said is key, about focusing on the meaning and intent. It is like deriving proofs in physics rather than memorizing them. Always go back to first principles: What am I doing? simple_smile

meow22:01:03

look in that doc for G.4 Calculator Example for an example of a really poorly thought out example - or a good example of how not to do a real state machine

meow22:01:35

such examples do a real injustice to a good idea like state machines, imo

meow22:01:22

@laforge49: yes, or like letting go of your OO instincts and adopting a functional approach

meow22:01:40

that was hard for me when I started doing Clojure 9 months ago

laforge4922:01:27

You've been doing it longer than me!

meow22:01:48

but from the look of your github you are smarter than me

meow22:01:54

so its a tie

laforge4922:01:15

But I learned oo in C++, where using functions instead of methods was encouraged--keep the objects simple and reusable!

meow22:01:27

I'm just persistent and stubborn and a pain in the ass

laforge4922:01:29

no, you've got that wrong. I'm just full time playing with clojure. I'm very very slow on the uptake--ask micha! I'm just tenacious.

meow22:01:26

two of us can drive the geniuses crazy

laforge4922:01:30

@meow: sent you an invitation. I just created the aatree organization to replace the laforge49/aatree project. I want to include more clojurescript among other things.

laforge4923:01:53

@micha sent you an invite too. simple_smile

laforge4923:01:09

Yeah, I tend to ask the dumb questions and then write up the answer I get. I figure there are a lot of other dumb people out there that will appreciate it.

laforge4923:01:52

Developers usually have a serious issue when it comes to targeting an audience!

meow23:01:55

sweet, thanks

laforge4923:01:19

I'm in process of linking aatree to the slack clojurians aatree channel.

laforge4923:01:46

So it had to be a community instead of the existing project.

micha23:01:03

this scxml doc is interesting

micha23:01:15

looking at the microwave example now