Fork me on GitHub
#mount
<
2016-02-02
>
tolitius01:02:34

@wkf: let me know which solution worked for you, so I can keep a note of it, in case others need help with the start order, thx!

wkf01:02:24

took a break for dinner, but I’ll let you know when I try it simple_smile

wkf14:02:15

@tolitius: I got it working for now, just by putting the elasticmq namespace before the sqs namespace

wkf14:02:29

I say “for now” because it feels brittle

wkf14:02:43

fwiw, I really miss having a way to formally express dependencies

wkf14:02:39

it would be nice to be able to pass dependencies like component, maybe something like (mount/using {#’app.sqs/sqs [#’app.elasticmq/elasticmq]})

wkf14:02:05

so you could add constraints to the inferred dependency graph

wkf14:02:27

also, despite my criticism, mount feels better than component when developing at the repl

wkf14:02:02

also, I’d like to help, if you’re open to this kind of feature

tolitius14:02:12

@wkf: I agree. the only case where it proves useful so far is in case of "one off" states that need a specific start order. I did not want to make it a requirement, but I agree, this is a very valid use case. it's actually not a bad idea.. to be able to specify some desired order before (mount/start) I actually welcome criticism, since it helps me think harder simple_smile

wkf14:02:23

excellent… I have some other thoughts as well, if you’re interested

tolitius14:02:33

sure, keep them coming simple_smile

tolitius14:02:18

I am trying to be very careful not to complicate the API, hence I am still thinking on depends-on feature.. for about a month now

wkf14:02:52

It seems like it would be quite useful to be able to wrap or transform states. for instance, I’ve got app.config/config, and in development, I’d like to add/change a few values

wkf14:02:58

I could add a new state, and swap it out

wkf14:02:09

but really, I just want to modify it slightly

tolitius14:02:12

i.e. it might make sense to add :depends-on on the actual state..

wkf14:02:57

would be nice to have something like (mount/hook #’app.config/config (fn [c] (assoc c :whatever 123)))

wkf14:02:43

but, hey, let’s put a pin in that, and talk about :depends-on

wkf14:02:10

ha, sorry for not searching first simple_smile

tolitius14:02:34

all good, it's only an idea at this point

wkf14:02:37

I think a central place to specify dependencies can add clarity

wkf14:02:29

it’s nice to be able to look at the graph, as data

wkf14:02:48

I’ve used component a bunch, and I’ve never really liked component/using

wkf14:02:58

I almost exclusively use component/system-using

wkf14:02:12

and at a glance, I can see how the pieces of my system fit together

tolitius14:02:20

I hear you.. thinking :)

wkf14:02:09

ah, interesting

tolitius14:02:11

what I didn't want to do is specify all the dependencies, since it will have to be kept in sync with a natural order.

wkf14:02:28

I think of it more like you’re enriching the natural dependencies

wkf14:02:45

to handle indirect dependencies (ie, I depend on a rest api provided by another state, not its namespace)

tolitius14:02:57

but I agree. explicit dependencies to help out structure the start order are very good to have

tolitius14:02:11

one difficulty that I see with enriching is validation.. but it might be easier than I think simple_smile

tolitius14:02:01

i.e. in case a state a is "enriched" with a dependency on state b, how do we know that by starting b a bit earlier to satisfy this dependency we not hurting it (`b`) by now not yet starting states that b naturally depends on

wkf14:02:15

but if we were to have access to the natural dependency graph, we could add new constraints

wkf14:02:25

and derive a total order

wkf15:02:48

I see the problem though

tolitius15:02:54

access is there, "graph" is not so much simple_smile

wkf15:02:55

we’re left with just the linear order

wkf15:02:59

from clojure

tolitius15:02:27

I "feel" there is a solution without breaking the idea of relying on the compiler, but I have not worked hard on validation yet..

tolitius15:02:02

(the :depends-on idea)

tolitius15:02:16

whether it is on a state level or central..

wkf15:02:46

yah, where to put it is just an api question, and component does both

wkf15:02:50

still have to solve the problem...

tolitius15:02:54

if it is on a state level, but a final explicit order is still available via (graph/states-with-deps), I feel specifying :on-depends on the individual state still has value of only needing to (mount/start) without remembering (mount/something-else args) beforehand

wkf15:02:25

so, here’s something interesting...

wkf15:02:31

in my original example

wkf15:02:43

the sqs component has an indirect dependency on elasticmq

wkf15:02:55

in other words, I couldn’t place it on the component directly

wkf15:02:01

err, state

wkf15:02:05

because it doesn’t always have that dependency

tolitius15:02:38

("component" / "state".. same thing), yes, I am not dismissing the central meta for :depends-on, I see it being useful

wkf15:02:59

gotcha, gotcha

wkf15:02:42

just thinking out loud here… I’m not sure I would care if, as a first pass, the mount/using api required I (the user) specify all dependencies

tolitius15:02:42

maybe some kind of :after / :before can work here.. then elasticmq can be:

(defstate elasticmq :start foo 
                    :stop bar
                    :before #{#'app/sqs})

wkf15:02:58

that’s interesting

tolitius15:02:21

usually (from my experience) introducing a public API is rarely a temp solution simple_smile

wkf15:02:06

maybe an even looser constraint would work… if you use mount/using, and you specify dependencies for a state, you must specify all its dependencies

wkf15:02:17

ie, once you take it out of the natural order, you need to take full responsibility

tolitius15:02:41

huh.. that's not a bad idea

wkf15:02:41

meh, I’m wrong, that still wouldn’t work

wkf15:02:01

actually, yah, it would

wkf15:02:09

just working it through in real time

wkf15:02:11

sorry for the spam 😉

tolitius15:02:27

I am not a fan if using since it feels too generic in this context

tolitius15:02:34

(as a word)

wkf15:02:40

oh, sure

wkf15:02:49

mount/start-with-deps or something

wkf15:02:52

might be more clear

tolitius15:02:59

yea, the name will come naturally simple_smile

wkf15:02:30

oh, that reminds me… any thoughts on separating the “configuration” from the “starting”?

wkf15:02:12

seems like it might feel nice to express the ways in which you’re modifying load order/dependencies once

wkf15:02:17

outside of the lifecycle

tolitius15:02:44

you mean something like a conf driven by boot / lein / -D ?

wkf15:02:11

nothing so glamorous, something like (mount/with-states {#’blah #’bloop}) (mount/with-deps {#’hello #’sir}) (mount/start)

wkf15:02:42

“persistent” changes to the way mount/start and mount/stop work

wkf15:02:03

seems useful, especially when I’m using more than one start-with-* function

wkf15:02:41

(an alternative would be a start function that takes different kinds of modifications… state swaps, deps, hooks, etc)

tolitius15:02:31

ah.. yea, that's definitely on the road map

tolitius15:02:39

I might sacrifice the idea of (mount/start-with) taking swaps, and have it to just take a map. this is not exactly "persistent" (i.e. before a start is called), since I like it to stay atomic, but it should achieve the flexibility:

(mount/start-with {:swap {} 
                   :only #{}
                   :except #{}
                   :with-deps {}})

tolitius15:02:04

something like that..

tolitius15:02:20

is there any specific reason you want them persistent?

wkf15:02:08

if there were separate functions for setting up deps, swaps, hook

wkf15:02:15

I’d like to set them

wkf15:02:20

before starting the states

wkf15:02:23

but if I can do it all at once

wkf15:02:29

that’s probably preferable

tolitius15:02:29

another way I was thinking some time ago is:

(-> (with {#'app.nyse/db test-db
           #'app.nyse/publisher test-publisher}) 
    (without #{#'app/nrepl feed-listener})
    (with-args args)
    start)

tolitius15:02:08

but start-with taking a map looks less confusing and "easier" to visualize / share / understand

wkf15:02:27

could easily have both, since those functions above could just create the map

wkf15:02:51

though, data > dsl

tolitius15:02:02

well, I'll definitely have all those functions to process the map internally anyway simple_smile

tolitius15:02:32

yea, something like a map can find usages / be provided from many places: * REPL * boot task / lein profile * config * dev.clj * etc..

tolitius15:02:42

it's a bit shaky to break (mount/start-with) though since people's tests depend on it.. but I don't think / hope it is not too late

wkf15:02:26

yah, luckily it’s still 0.x

wkf15:02:38

not too late to make changes, especially since it’ll lead to a better tool

tolitius15:02:53

you're right, the version would help here

arnout16:02:38

Hi @wkf and @tolitius. I saw you mention me (and mount-lite) on GitHub.

arnout16:02:54

Mount-lite actually has a 'parallel' branch now: https://github.com/aroemers/mount-lite/tree/parallel

arnout16:02:43

In it are both a automatically deduced state dependency graph, and a way to influence that graph.

arnout16:02:10

Maybe this can be some inspiration to Mount and above discussion?

wkf16:02:51

oh, awesome

wkf16:02:00

seems super useful

arnout16:02:04

(just like I hope to inspire a composable API for Mount, which you also discussed I see 😄)

tolitius16:02:30

@arnout: the compatibility is definitely a great idea :) I saw the parallel work, it may not work for ClojureScript though

wkf16:02:56

@tolitius: I wrote a parallel dependency loader in javascript

wkf16:02:19

would have to think a bit how to do it clojurescript

wkf16:02:15

“parallel”

tolitius16:02:37

the idea of building a graph of off vars don't really project well to cljs since, unless bootstrap is used, have no support for Clojure ns API in :advanced mode

tolitius16:02:00

simply following @wkf idea of providing the full graph in :with-deps might solve it in use cases that need it

arnout16:02:50

Heh, removing suspend and resume? Cool, I can then remove "I don't need suspending" from mount-lite's README 😉

arnout16:02:26

Hmm, I think Mount is great, because of the implicit dependency "graph" it creates (whether a proper graph or sequential does not matter)

arnout16:02:59

"It just works" - except for when it doesn't, in @wkf's use case

wkf16:02:12

it’s interesting that you say that

tolitius16:02:22

yea, I stumble upon these (suspend/resume) often enough to question their need. You saw it first :)

wkf16:02:36

to me, the real value proposition comes form the way mount affects working at the repl

wkf16:02:39

vs component

wkf16:02:51

I never found it particularly tedious to express the dependency graph with data

tolitius16:02:06

yes, development was the essential motive

wkf16:02:08

not being able to easily “play” at the repl was the real downer

arnout16:02:36

But do you want to globally store such a data structure, like set-dependencies? Or do you want to specify it each time you do one of the many REPL actions (`start-with`, start-except, etc.)?

tolitius16:02:40

@arnout: agree with the value of implicit graph, but am option to override it does seem like a valid feature

tolitius16:02:04

(the example there)

tolitius16:02:31

typing from a phone.. :)

arnout17:02:09

Sure, that's a very valid way to go

tolitius17:02:26

yea, I think your real graph in lite is quite powerful

arnout17:02:54

Boasting myself here, but that graph also makes for quite a nice :up-to start/stop option. It only starts (or stops) the necessary states to start/stop the given "up-to" state.

arnout17:02:05

And that :up-to option is actually also used for the "cascading stop" on a redefinition; meaning never a running state that suddenly lost its dependency. Which can still be overridden of course by the :only option.

arnout17:02:24

./boasting-mode off

tolitius17:02:50

:) can you share a use case for :up-to?

arnout17:02:00

Whenever you want to test a certain state for example. Just (start (up-to #'state/under-test)) and your good to go, as all its dependencies will have started as well, without needing any knowledge of what states that would be.

arnout17:02:38

Or if you only want to stop part of your system, but some base states can keep running.

tolitius17:02:14

do you see it used instead of start-without / stop-except ?

arnout17:02:15

And, with the automatic "cascading stop", you can safely redefine a state. It will only stop the necessary part of the app, and after a (start) everything is back up.

arnout17:02:51

Yes, because those two require knowledge about the implicit state dependency sequence/graph.

arnout17:02:31

Those functions are enough of course, but up-to is eh.. more convenient in that case?

tolitius17:02:00

I see. I tend to minimize the number of states.. so I don't have that many in a single app usually. Since I tend to create states for external resources, and usually there are not a lot of those. But I guess when apps have many, yea, it could be more convenient

wkf18:02:52

thinking more about :with-deps, and the proposed changes to starts-with, and I think it’s more complicated than it needs to be

wkf18:02:56

I’m thinking of a data structure like this:

(mount/start-with
  {:states
   [#'app.config/config
    #'app.server/server]
   :dependencies
   {#'app.server/server [#'app.config/config]}})

wkf18:02:28

which is expanded to something like this...

wkf18:02:57

{:states
  {#’app.config/config #’app.config/config
   #’app.server/server #’app.server/server}
…}

wkf18:02:12

(I left out the dependencies, but they are still there)

wkf18:02:27

“swaps” could be handled in the :states map

wkf18:02:54

but basically… mount/start-with takes a whole “system”

wkf18:02:10

and under the hood

wkf18:02:23

perhaps we could generate that “system” data structure

wkf18:02:33

seems like we could describe :only, :except, :with-deps

wkf18:02:38

as modifications to that data structure

wkf18:02:26

we could also come up with some well defined semantics for how the “natural system” could be merged with the provided system

wkf18:02:30

in other words...

wkf18:02:57

(mount/start-with-merge
  {:states 
    {#’app.config/config #’app.repl/config}})

wkf18:02:13

would merge the states from the “natural system” with the above data

wkf18:02:22

and have the effect of a swap

wkf18:02:20

under the hood, there could be some refactoring to implement the start* functions in terms of this system data structure

wkf18:02:46

with a 0-arity mount/start call using the “natural system”

wkf18:02:03

mostly just thinking out loud

tolitius19:02:09

hm.. if I understood you correctly start-with-merge would mutate the running system?

tolitius19:02:35

what's the benefit of hiding intentions (i.e. :only, :except, ...) behind a vector / map?

wkf19:02:01

sorry, there was no mutation in that case, let me try to explain more clearly

wkf19:02:35

I’m thinking of a data structure that would contain an exhaustive list of states and dependencies

wkf19:02:49

and then using that data structure to start/stop the states

wkf19:02:12

this is mostly a question of implementation

wkf19:02:50

right now, there’s only an implicit set of states and dependencies

wkf19:02:06

let’s call that the system

wkf19:02:05

a definition of the states in the app, and their dependencies

wkf19:02:11

if we had that data structure, it would be pretty straightforward to implement the stop/start behavior as it exists now

wkf19:02:43

in fact, I think start and stop could be functions of a system-spec

wkf19:02:04

and then :only, :except, :with-deps, etc

wkf19:02:10

would be transformations of the spec

wkf19:02:19

that would yield a valid system spec

wkf19:02:33

but with fewer states, or different deps, etc

tolitius19:02:47

looks like you are talking about a https://github.com/tolitius/yurt simple_smile

tolitius19:02:06

i.e.

dev=> (yurt/blueprint)
{:components
 {"neo.conf/config" {:status :not-started},
  "neo.db/db" {:status :not-started},
  "neo.www/neo-app" {:status :not-started},
  "neo.app/nrepl" {:status :not-started}},
 :blueprint
 {"neo.conf/config" {:order 1},
  "neo.db/db" {:order 2},
  "neo.www/neo-app" {:order 3},
  "neo.app/nrepl" {:order 4}}}

tolitius19:02:54

a detached system which internally (by it's type), knows how to be started / started-with / stopped

wkf19:02:45

I’ll have to take a look

wkf19:02:52

looks like it’s pretty close to what I was talking about

tolitius20:02:43

right.. it still does not do lifecycle fns composition, since.. well, it is just 4 days old

tolitius20:02:15

it was really written to mostly show that mount design does not limit developers by a single(ton) system, and many more than one system can be created, bound locally and used in the same runtime

tolitius20:02:37

but it can very well be used beyond that, of course

tolitius20:02:43

as to mount, I would like to keep it system transparent, and mostly worry about the meat of the app (that has nothing to do with mount). I see "composable lifecycle" as something that is mostly helpful in testing. I do however agree that "indirect dependencies" should be a core feature that is a lot more important to have

arnout21:02:16

@wkf: In mount-lite, there are these start* and stop* functions you talk about. They take a single spec: {#'statevar {:start ... :stop ...}, ...}. This spec is being created by supplying one or more maps containing :only, :except and :substitute (and more mount-lite specific) keys to start/`stop`. Whenever you supply this to start (or stop) it is munged into the simple spec that start* expects, by starting with a complete spec, and processing the :only, :except etc options on it. This design has suited me well, by allowing me to add more options quite easily, without altering the current API. Is that what you meant?

wkf21:02:57

yah, something similar to that

wkf21:02:24

@tolitius: I’m not as concerned about supporting multiple systems, I’m just thinking that the “system” abstraction internally might be a decent way to handle the indirect dependencies

wkf21:02:53

I cloned down the repo, I’ll take a stab at it with code, you can see what it looks like