Fork me on GitHub
#mount
<
2017-03-17
>
tolitius01:03:22

> the db function should be wrapped in a defstate also @lucian303 this would not be a good solution

tolitius01:03:31

calls to db are just functions

tolitius01:03:14

defstate should be only used for stateful things (I/O, thread pools, etc.)

lucian30303:03:25

@tolitius in my actual code i'm replacing a def w/ a defstate. it's stateful because it depends on the db as well as an external file. so (def ^:dynamic *nwords* (load-dict)) becomes (defstate ^:dynamic *nwords* :start (load-dict)) where load-dict makes the actual call into the db namespace and also loads the external file. does that seem like a proper use case? i tried to pare all that down for the example on github so it's not apparent there.

yenda11:03:47

is it possible with mount to change a state in a function ? for instance I have a logger that logs to the console and later during startup I want to add logging to a message broker if a connection becomes available

yenda12:03:29

I just saw in the hubble repo that there is an on-change function I will look into that

tolitius12:03:53

@lucian303 yes, depending on the use case this is correct. for example if your application has configuration you are reading in, depending on what it is in dict, it may or may not be loaded as a part of a :start function of a config state. for example, in case you use: https://github.com/tolitius/cprop, it could be a part of:

(defstate config :start (-> (load-config) (merge (load-dict))))

tolitius12:03:08

@yenda how / where do you check > if a connection becomes available ?

yenda12:03:38

@tolitius it eventually becomes available in the config at some point during the startup of the application

tolitius12:03:19

> I want to add logging to a message broker and stop logging to console?

yenda12:03:33

not necessarily does it matter ?

tolitius12:03:00

depending on the logging lib you use, this could be simply done with something like logback appenders. For example here is from the SocketAppender:

Consequently, if the remote server is reachable, then log events will eventually arrive there. Otherwise, if the remote server is down or unreachable, the logging events will simply be dropped. If and when the server comes back up, then event transmission will be resumed transparently. This transparent reconnection is performed by a connector thread which periodically attempts to connect to the server.

tolitius12:03:30

i.e. I am not sure why you would need a state for a logger, as this is usually baked into the logger lib

yenda12:03:01

I use my own logging library

tolitius12:03:44

ah.. ok, that's interesting, why not any of libs that exist in JVM: log4j2, logback, etc. ?

yenda12:03:40

because I want to log structured logs and I made macros that capture variables in scope among other things such as project name and version, thread and hostname...

tolitius12:03:02

right, so this is a function, not a state

tolitius12:03:42

but what do you usually use to log?

tolitius12:03:58

i.e. clojure.tools.logging / timbre / ?

yenda12:03:08

my macros

tolitius12:03:50

macros is evaluated at compile time: i.e. there is no way to pass it a connection / open file

tolitius12:03:10

do you have a dependency on clojure.tool.logging?

tolitius12:03:26

in boot.biuld, project.clj?

yenda12:03:52

no my macros then expand to a call to my log function

tolitius12:03:29

ok, and what does that function do, does it use something like clojure.tool.logging?

yenda12:03:59

no it writes to stdout and publish the message if the broker connection is available

tolitius12:03:47

interesting, how do you control the log level: i.e. info vs. trace vs. error, etc.. ?

tolitius12:03:51

or you don't need to?

yenda12:03:52

I am currently experimenting by doing a rewrite with mount

tolitius12:03:11

yea, sure, I'd just like to understand the problem better

yenda12:03:18

before log would call an append multimethod dispatching to the list of appenders

tolitius12:03:51

I see, so you have a list of appenders and those are states because they need access to external resources?

yenda12:03:49

actually this is not really my problem the logger was one expemple, in general I would like to have a mean to redefine states as the program startsup because I have lots of chicken and egg problems

tolitius12:03:02

then I would have that "broker appender" as a state that would always log, but would drop or buffer log messages until the connection is available rather then mutate the logger state

yenda12:03:49

why is it preferable ?

yenda12:03:52

is it bad to mutate state in mount ? instead I could just keep the way things were with the multimethod and have the list of available appenders to dispatch to in the config state

yenda12:03:03

and only this one will mutate and the logger will not be a state at all

tolitius12:03:11

it's not "bad", but it is usually better solved on the "function" level. the hubble watcher watches changes from consul: i.e. there is an envoy: https://github.com/tolitius/envoy watcher that triggers the change. In your case, if I understood it correctly, it seems like you'd like to simply delay logging messages to a socket until the connection is there, which could simply be done with message buffer, in which case you won't lose any messages

tolitius12:03:13

since you have a list of appenders that you always want to keep around (until the app is stopped) I would not solve it by changing the state, but rather by creating apenders once at the app start

yenda12:03:58

well actually in some cases I would not want all appenders it can be configured

tolitius12:03:56

right, but you would read that config at the start time, right?

yenda12:03:42

that is the chicken egg problem I was talking about earlier the config can be either a file or fetched from a configuration service that is accessed via rabbitmq

tolitius12:03:25

right, so you are reading a config and create you appender state(s) based on it: appender(s) state depends on a config state:

(defstate config :start (load-config))
(defstate appenders :start (do-with config))

yenda12:03:26

to get the config it reads the config string which can be a file url or rabbitmq connection, if it is a rabbitmq connection it fetches the config from there else it looks for a rabbitmq config in the file

tolitius12:03:57

I think I see where you are coming from. hubble example does a similar thing: i.e. it has a config in a file + the one in consul

tolitius12:03:14

so it merges the one in consul on top of the one in a file

tolitius12:03:48

i.e. this way once the config is loaded:

(defstate appenders :start (do-with config))
knows which one to use

tolitius12:03:07

and appenders will be started after the config since it depends on it

yenda12:03:41

yes but the trick is that the logging should start first with a console appender and update the list of appenders once new ones are added

yenda13:03:06

the problem is that it needs to be really dynamic to accomodate with different use cases

yenda13:03:22

I have a working version that doesn't use any library: it makes things available as soon as possible using delays and bootstrap namespaces to avoid circular dependencies

tolitius13:03:02

I see, so you want to be able to plug appenders at runtime

tolitius13:03:19

not necessarily delaying them on start

yenda13:03:46

yes I want to make things more dynamic and also add mount as the system management library

yenda13:03:04

basically the project is a library for config and logging and messaging for microservices

yenda13:03:45

so a microservice code would define a few defstate and mount/start in the main

yenda13:03:35

and be able to mount/stop during dev

yenda13:03:23

right now you start with (config/load) which loads the config and starts the broker-connection if one is needed but you can't restart cleanly in the repl during development

tolitius13:03:44

than it would probably be better to have an appenders state which would be a vector behind an atom: i.e. you can start / stop it, and also swap! conj it whenever you need to add to it. the problem with simply restarting that sate would be other components that are "currently" accessing it at runtime

tolitius13:03:05

> you can't restart cleanly in the repl during development why?

yenda13:03:16

for instance tif you close the rabbitmq connection you also close open channels

yenda13:03:58

restarting it won't make the rabbitmq appender work again

tolitius13:03:23

> you also close open channels what do you mean?

yenda13:03:31

it's just a rabbitmq thing to connect to rabbitmq you need a tcp connection and then a channel that is some lightweight not thread-safe connection on top of it

tolitius13:03:59

if rabbitmq connection is a state, you don't have to restart it if you don't want to:

(mount/stop-except #'app/rabbit-conn)
(mount/start)

yenda13:03:37

sorry I wasn't clear I was talking about the current working solution that doesn't use mount

yenda13:03:23

I am trying to make a better version that is less complex and has a clearer and cleaner bootstraping and can be started/stopped during dev time

tolitius13:03:13

ah, ok. yes, the restart in REPL should not be a problem if you use mount. if you need to restart both a connection and channels, you rabbit state would be {:conn conn :channels channels}, then your stop function would stop both

tolitius13:03:08

or (mount/stop-except #'app/rabbit-conn) in cases you don't need to stop / start it

tolitius13:03:00

or ^{:on-reload :noop} ( https://github.com/tolitius/mount#on-reload ) in case you don't need it to get restarted as the namespace recompiles

tolitius13:03:32

but usually you would just do (mount/start) / (mount/start) and it would successfully restart this connection along with the channels

yenda13:03:35

there is no problem on that part before the heavy refactoring I tested it with mount and the old bootstraping model and it was perfect

yenda13:03:58

currently the problem could be put that way: states are defined in multiple steps - appenders: 1. starts with [:console] 2. update appenders when available in config - config: 1. starts reading from param 2. gets file to read or rabbitmq connection 3. fetch config from rabbit when connection is available - rabbit-connection 1. starts when available in config

yenda13:03:51

I think I need to play a bit more with mount first to find a better design

tolitius14:03:52

so far the way I see it is something like:

(defstate config :start (-> (load-config)
                            (merge (read-rabbit-config))))
(read-rabbit-config) would read config from rabbit only when needed, otherwise {}
(defstate rabbit-connection :start (r/connect (config :rabbit))
                            :stop (r/diconnect rabbit-connection))

tolitius14:03:36

+ a possible logger / appenders state that would be a vector behind an atom

tolitius14:03:47

@yenda: just curious, why do you need to implement your own logging solution vs. using clojure.tools.logging / timbre / slj4j, etc..

yenda14:03:23

Convenience and structured logs, it captures variables from the scope as well as informations that other loggers don't

tolitius14:03:47

but since this is done by your macro, can it just wrap an existing logging solution?

tolitius14:03:03

to add "variables from the scope as well as informations that other loggers don't"

yenda14:03:02

It wouldn't really add any benefit

tolitius14:03:26

for one you would not have to deal with appenders

yenda14:03:02

As far as I remember you still need to configure the appenders in timbre

yenda14:03:13

And you can't do it at runtime

tolitius14:03:08

in both, log4j and logback you can

yenda15:03:46

I'll think about it. Thanks for all the help !

tolitius17:03:34

sure, good luck. let me know where you end up

yenda19:03:15

@tolitius for now I didn't work on the error handling part but I went again with the bootstrap namespaces method to avoid the circular dependencies

yenda19:03:48

and it work quite nicely it starts/stops and everything including logging seems to work nicely