Fork me on GitHub
#clojure
<
2018-07-27
>
emccue02:07:28

Can I get some criticisms of that code?

emccue02:07:45

I feel like I am over designing freaking config too early

mg03:07:24

Until you start adding more implementations of Config, config-val can just be a function that takes a Properties argument. merge-configs can just do a regular merge and return a new merged Properties

mg03:07:53

although personally I would go map first rather than Properties first, unless you have some interop thing that needs it to be Properties

mg03:07:05

Even then it might make more sense to convert the map to a Properties right before interop, rather than using Properties everywhere

emccue04:07:17

yeah i can probably add an (into {} ..)

emccue04:07:02

only reason I didnt convert it was because i only ever read, I didn't need to work with it

emccue04:07:58

Ill get rid of the protocol for now but keep the config-val function

emccue04:07:31

I think I'll need to load secrets from AWS and some less important values from a config value

emccue04:07:53

but I think you are right

otwieracz07:07:35

Hi!

đź‘‹ 4
otwieracz07:07:54

How do you store and access configuration in your Clojure apps?

otwieracz07:07:07

I've got such case:

otwieracz07:07:52

I need some convenient way to store and access information like file-path or other properties which are used by variety of functions in my system.

otwieracz07:07:23

keeping thing „purely functional” I end up passing dozen of parameters for each function.

Shantanu Kumar14:07:31

I tend to use http://bract.github.io and http://GitHub.com/kumarshantanu/dime to get the effect you mentioned. Sorry about the unintentional plug.

otwieracz07:07:47

Or creating some one map (like system) with all the configuration stuff.

otwieracz07:07:14

But then I end up passing this config map to each function… is this the correct way to go?

otwieracz07:07:53

Or I should not think about it and just pass all necessary arguments to function?

otwieracz07:07:38

And care about it later, in place where it will be executed (especially that this function is still „low level” and does not necessarily have to know anything specific about how configuration will be stored, etc.

val_waeselynck07:07:15

@slawek098 there are really 2 concerns to address here - one is storage and encoding of the configuration; the other is turning that configuration into system components that your code can access

val_waeselynck08:07:44

For the second concern, there are 2 approaches - represent the system as a map that gets passed to each function (as you said - this is the approach taken by the Component https://github.com/stuartsierra/component and Integrant https://github.com/weavejester/integrant libraries), OR represent it via global Vars (that's the approach taken by the Mount library - https://github.com/tolitius/mount). I'm personally very much in favour of the first approach for most cases, but there is controversy about that in the Clojure community.

val_waeselynck08:07:51

About storage and encoding, the typical approach is to store it as EDN - potentially in conjunction with a lib like Aero https://github.com/juxt/aero.

val_waeselynck08:07:18

The "evaluation" of that config will depend on the library you use - Integrant offers it's own EDN format, while for Component you may want to use something like Schematic (https://github.com/walmartlabs/schematic)

val_waeselynck08:07:28

You can also roll out your own homemade alternative to Component / Integrant, but it can be tricky to get right once you aim at making it REPL-friendly.

otwieracz08:07:53

Well, usually I've crafted something like „system workflow reloaded” but simpler just to fulfill my needs.

otwieracz08:07:34

Maybe I have to dig into Component and Integrant finally…

đź‘Ť 4
jeroenvandijk08:07:36

@val_waeselynck Do you happen to know a good example application where integrant is being used to orchestrate the system?

val_waeselynck09:07:02

I'm no authority on "good", but there's https://github.com/juxt/edge

jeroenvandijk10:07:56

@val_waeselynck Thank you! This is useful. I'm curious for typical usage patterns of Integrant. We've created a wrapper around Component that does a lot of the hard work like dependency resolution, configuration boilerplate etc. I'm wondering if I'm missing out, but it seems Integrant still requires a lot of manual work regarding dependency resolution. And I also notice that there are many ways in how you can use integrant (both strength and weakness IMO)

val_waeselynck10:07:26

> but it seems Integrant still requires a lot of manual work regarding dependency resolution. What do you mean by that? I don't seem to have encountered this problem.

reborg09:07:25

Never liked the idea of having functions to be force-fed a system map or other components. But I do like the core idea of components, to have a single entry point to put the system in the right state. For this reason, if I had to pick a mainstream lib, I'd pick mount over ss.components (for instance). In practice though, I rolled my own and never looked back (if you are curious, it's not a library, it's a convention on top of tools.namespace https://github.com/reborg/scccw/blob/master/COMPONENTS.md).

val_waeselynck09:07:37

controversy in action 🙂 I actually very much like the idea of the "system as an argument", because it reifies the environment of your code into a value, making it more programmable and explicit. I find this useful for testing among other things, and also for operations like "forking" etc.

reborg09:07:43

Yeah, no worries, at the edge of personal taste.

val_waeselynck09:07:24

Component vs Mount is the new Vim vs Emacs

mpenet09:07:46

You could argue component (or integrant) do not force anything, you're free to add a few fns to make the system accessible globally

val_waeselynck09:07:47

the local-vs-global mismatch is gonna hurt though

reborg09:07:12

Oh yes, in theory @mpenet. But people I work with tend to do what the manual and the examples show.

mpenet09:07:33

I personally don't like the mount approach. but yeah, vim/emacs etc

mpenet09:07:11

We use component at work, but if I had to start over I'd choose integrant.

val_waeselynck09:07:13

@mpenet I've been disappointed in Integrant actually - I find it's choice of "inheritance as the mechanism for reuse" very impractical, making the configuration quite hard to read, especially when some type of component appears in several places in the system, or when components are polymorphic.

mpenet09:07:28

Ah, interesting

mpenet09:07:06

There are a few things I don't like with component as well, first of all it being quite closed to contribs and managed in a rather odd way (pr's a never merged, patches are usually rewriten, even for typos)

âž• 4
reborg09:07:35

And that's also why I prefer to roll my own (and try to convince people to do the same 🙂.) You need <50 LOC to have a fully working component system in your project, and that's completely under your control.

reborg09:07:08

Yeah, then I have the colleague who is not used to it and objects saying "this is not standard". Blah, the best framework is no framework.

manuel09:07:04

I've only used mount so far, but now you got me interested in rolling my own.

đź‘Ť 4
jeroenvandijk11:07:14

Did some of you work with a composition tool like https://github.com/RedBrainLabs/system-graph that wraps component and resolves dependencies? I haven't seen much development in this area

manuel11:07:51

I didn't, sorry.

jeroenvandijk12:07:12

Not sure if it's what I mean. I see that you have to manually declare dependencies. From the README

:app {:sc/create-fn 'user/map->App
         :sc/refs {:server :webserver}
         :sc/merge [{:to [:api-keys] :from [:api :keys]}]}
I might miss some nuance of course

jeroenvandijk12:07:43

We have created something that allows to declare component constructors in an edn file and configuration and dependencies will be sorted out based on the components definition. This is really nice in a big system. I haven't seen something like it yet

val_waeselynck12:07:04

I think I don't see the difference between "manually declaring the dependencies" and what you're saying - if there's a dependency, it has to be declared, right?

jeroenvandijk12:07:47

True, but the difference is whether you declare it on a local level, next to the component. [Or in our case in the definition of the component (we have a wrapper).] Or on a global level. The global level is much more fragile and very tedious in my experience. Especially if you want to create different versions of systems like test/dev/production etc

jeroenvandijk12:07:39

E.g. for dev we can swap out a key-value store with an in-memory version which has different dependencies, but with a local approach you don't notice this

jeroenvandijk12:07:08

Another thing is that the dependency is declared on a local level anyway (as you are using it in the definition). So the global declaration is just extra and unnecessary work i think

jeroenvandijk12:07:06

Makes sense or too vague?

dominicm12:07:37

we're doing this via aero also

dominicm12:07:51

We don't reuse components much, so we've been looking at moving to integrant

jeroenvandijk12:07:49

Does aero do the dependencies for you?

dominicm12:07:14

We have a :dependencies key in the components.

jeroenvandijk12:07:53

Ah interesting

jeroenvandijk12:07:28

Do you know a public example of that approach?

dominicm12:07:53

Unfortunately not

val_waeselynck12:07:19

For dependency resolution, my favorite approach is actually the one taken by integrant - having a special #ref EDN tag / data structure naming dependencies, and building the DAG from that. This way you can have both inline or globally resolved deps in an arbitrary structure

jeroenvandijk13:07:54

Ok cool. I'll have to try that

otwieracz11:07:15

(= [{:state :changed,
     :template "procmailrc",
     :config [["start" []] ["end" []]]}
    {:state :deleted,
     :template "procmailrc",
     :config [["start" []] ["antivirus" []] ["end" []]]}]
   [{:template "procmailrc",
     :config [["start" []] ["antivirus" []] ["end" []]],
     :state :deleted}
    {:template "procmailrc",
     :config [["start" []] ["end" []]],
     :state :changed}])

otwieracz11:07:28

Why it returns false?

otwieracz11:07:40

ah, I know, sorry!

otwieracz11:07:11

I have to use set here.

otwieracz11:07:17

Because order does not matter.

otwieracz11:07:40

Is spec clever? If I've got function with :pre check checking one argument and then this function calls another function verifying the same entity against the same spec, is spec going to waste time checking it again?

otwieracz12:07:09

I want to execute function on every item from list. However, this is actually changing the filesystem and ends up as a lazy expression never really evaluated… Is doall the proper way or maybe there's some different construct than currently used map which is not lazy?

sundarj13:07:06

there are run! (like map) and doseq (like for)

martinklepsch13:07:40

Is there some idiom for (first (filter ,,,))?

martinklepsch13:07:05

You could use some but then the predicate has to return the original value

martinklepsch13:07:25

Same context: is there a helper that like (fn [pred x] (when (pred x) x))?

noisesmith16:07:01

no, but I've defined it as "when-pred" in multiple projects

Robert17:07:05

@alexmiller are you excepting GitHub pull requests for the clojure windows installer branch ? I have some fixes to contribute.

Alex Miller (Clojure team)17:07:51

No, but feel free to comment on the ticket

hyankov17:07:35

Hi all, I looked for channel dedicate to json parsing but I only found #tmp-json-parsing which doesn’t appear to be active so I’ll ask my question here: Is there a benchmark comparing json->edn decoding with and without converting keys to keywords? I tried googling for a while but got nothing. data.json, cheshire, jsonista doesn’t really matter, I guess results would be relatively similar in all

noisesmith17:07:24

the main concern with keywordizing is if you will be hard-coding the keys in your logic handling the incoming data, and whether the keys actually make sense as keywords

noisesmith17:07:15

for example "foo bar baz" and "0" are valid json keys, but are various degrees of bad as keywords (though the keyword function isn't picky)

hyankov17:07:36

@noisesmith thanks for pointing these out. However, I’m still interested in the performance overhead keyword introduces when parsing json

hyankov17:07:21

I can run the benchmark myself but was curious whether someone already did it

noisesmith17:07:40

it's negligible compared to creating the data structures

noisesmith17:07:23

I would be surprised to ever see a situation where keywordizing was the bottle neck that was making json parsing too slow (outside of pathological keys...)

hyankov17:07:41

:thumbsup: Cool! Thanks, that’s what I was looking for:)

hiredman17:07:50

there was a whole jira thing about it

🙇 8
hyankov17:07:12

Interesting issue, thanks for sharing!

rustam.gilaztdinov18:07:18

guys, can you help with refactror-nrepl? i have this in .lein/profiles.clj

{:user {:plugins [[refactor-nrepl "2.3.1"]
                  [cider/cider-nrepl "0.17.0-SNAPSHOT" :exclusions [org.clojure/tools.nrepl]]]}}
when run lein repl I always get this:
Error loading refactor-nrepl.middleware: java.io.FileNotFoundException: Could not locate cider/nrepl/middleware/util/misc__init.class or cider/nrepl/middleware/util/misc.clj on classpath., compiling:(refactor_nrepl/middleware.clj:1:1)
Exception in thread "main" java.lang.RuntimeException: Unable to resolve var: refactor-nrepl.middleware/wrap-refactor in this context, compiling:(/private/var/folders/dd/yjt452yn0zs1hs4_sd077rsm0000gn/T/form-init8768462867645520700.clj:1:8336)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6875)
	at clojure.lang.Compiler.analyze(Compiler.java:6669)
	at clojure.lang.Compiler.analyze(Compiler.java:6625)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3834)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6870)
	at clojure.lang.Compiler.analyze(Compiler.java:6669)
        ...........
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.RuntimeException: Unable to resolve var: refactor-nrepl.middleware/wrap-refactor in this context
	at clojure.lang.Util.runtimeException(Util.java:221)
	at clojure.lang.Compiler$TheVarExpr$Parser.parse(Compiler.java:710)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6868)
	...

arrdem18:07:30

@rustam.gilaztdinov plugins are dependencies which get injected to leiningen, refactor-nrepl and cider-nrepl are dependencies which need to be present in your project in order to function.

arrdem18:07:59

it may be correct to have refactor-nrepl as a plugin, but cider-nrepl almost certainly needs to be a dependency not a plugin.

rustam.gilaztdinov18:07:49

no, first -- added those plugins for local project doesn't solve this, and second -- I want to have this plugins global, for all my projects

rustam.gilaztdinov18:07:13

I cleaned all from ~/.m2

noisesmith18:07:17

cleaning all from m2 is rarely useful, do you understand what he means about using a dependency instead of a plugin? this is code needed in your running process, not the leiningen process that creates it

rustam.gilaztdinov18:07:44

yes, I understand him) cider and refactor-nrepl needs for development, right? so, why I should add this to dependencies?

rustam.gilaztdinov18:07:44

I think -- this should work from ~/.lein/profiles.clj globally

noisesmith18:07:59

plugins are for things that need to load in the leiningen process, dependencies are for things needed in the clojure process. The cider and refactor-nrepl stuff needs to be present in your repl, not the leiningen that starts it

noisesmith18:07:45

so it's still in ~/.lein/profiles.clj but under the :dependencies key, not the :plugins key

noisesmith18:07:30

each one should document whether it should be in plugins vs. dependencies checking the docs, they should work as plugins (the source of confusion here being that they are plugins that exist to change the deps in your process, plus set up your repl middleware) one thing to double check is the exclusion you are specifying - if you exclude a dep and nothing else provides it you'll just break anything that tries to use it

rustam.gilaztdinov19:07:50

i check docs, and all says me to use it in .lein/profiles.clj

noisesmith20:07:46

right, nobody has questioned thhat

dominicm06:07:44

They both should be plugins, the bug here is that refactor-nrepl doesn't play nice with the new cider. Try 2.4.0-SNAPSHOT for refactor-nrepl.

đź‘Ť 4