Fork me on GitHub
#clojure
<
2017-03-12
>
bradford00:03:19

Hey fam, I'm using Pedestal to manage 100k HTTP connections to various proxies. I want to route data to a particular proxy. Would you recommend storing an atom of {ip : core-async-channel} open every connection open that I can then access at will, or is there a more 'functional' way of doing it? I know this is a very broad question 😛

nicmcphee04:03:31

Does anyone know of an easy way to output valid EDN format from Clojure. Basic printing tools generally work fine, but they print things (e.g., rationals) that aren’t actually in the EDN spec and won’t be properly read by other EDN parsers like edn-java. I can hack around this, but it seems weird (esp. since EDN comes from the Clojure world) that, e.g., clojure.edn doesn’t have something that will spit out valid EDN. Thanks.

bcbradley07:03:55

@nicmcphee there isn't a function to automatically convert rationals into floats while converting a structure into a string because there are rationals that do not have an exact finite-length representation as floats. I find it strange that the edn spec doesn't allow for rationals.

Ethan Miller09:03:02

When I try to use ring’s jetty adapter, I’m get an error that I can’t get around. The project has [ring “1.5.1”] as a dependency in the project. If I then run the repl and do (use ‘ring.adapter.jetty), I get this error: > CompilerException java.lang.NoClassDefFoundError: org/eclipse/jetty/http/HttpParser$ProxyHandler, compiling:(ring/adapter/jetty.clj:27:11) If I create a fresh test project,and add just [ring “1.5.1”] as a dependency, run the repl, and then try the same use command, it loads up just fine. I haven’t any good ideas about why this might be he happening. Anyone here have a clue?

mccraigmccraig09:03:29

@ezmiller77 have you checked out the deps that are actually being used ( lein deps :tree ) ? it's possible one of your other deps has a transitive dep that conflicts with one of ring's

Ethan Miller09:03:32

@mccraigmccraig I had not. Just looked at the output tof that command. Not entirely sure what it all means. At the top of the output there are a lot of suggestions regarding including “exclusions” with a lib called commons-codec.

Ethan Miller10:03:13

Just tried adding it for ring in my dependencies. Then deleted ring in .m2 dir and reinstalled. But still getting the error. That was just a shot in the dark though.

Ethan Miller10:03:26

What’s a “transitive dependency”?

mccraigmccraig10:03:22

@ezmiller77 so your project.clj lists a bunch of dependencies - those deps themselves have deps etc - it's possible that there are conflicting versions in the transitive set of dependencies

mccraigmccraig10:03:23

e.g. say your project has deps [A,B] and project A has deps [Cv1] while project B has deps [Cv2] ... your project now has transitive deps [A,B,Cv1,Cv2] and lein will have to choose which version of C to include

mccraigmccraig10:03:52

sometimes that works out, and sometimes it doesn't

mccraigmccraig10:03:14

when it doesn't you have to help lein out by specifying manual :exclusions in your project.clj

mccraigmccraig10:03:15

however, if the only suggestions you are getting from lein are relating to commons-codec that doesn't sound like your problem, since you are missing a jetty class

mccraigmccraig10:03:40

try this @ezmiller77 - specify a top-level :exclusions [commons-codec] in your project.clj, then in your :dependencies manually specify the version of commons-codec that your desired version of ring/jetty etc depends on

mccraigmccraig10:03:06

the top-level :exclusions causes commons-codec to be removed from any transitive deps included, leaving only the version you manually specify

Ethan Miller10:03:32

Will try that. Hard to know, though, what version jetty might be relying on. The whole ring tree toward the end doesn’t seem to mention commons-codec at all:

Ethan Miller10:03:54

[ring "1.5.1"]
   [ring/ring-core "1.5.1"]
     [clj-time "0.11.0"]
       [joda-time "2.8.2"]
     [commons-fileupload "1.3.1"]
     [commons-io "2.5"]
     [crypto-equality "1.0.0"]
     [crypto-random "1.2.0"]
   [ring/ring-devel "1.5.1"]
     [clj-stacktrace "0.2.8"]
     [hiccup "1.0.5"]
     [ns-tracker "0.3.0"]
       [org.clojure/java.classpath "0.2.2"]
       [org.clojure/tools.namespace "0.2.10"]
   [ring/ring-jetty-adapter "1.5.1"]
     [org.eclipse.jetty/jetty-server "9.2.10.v20150310"]
       [javax.servlet/javax.servlet-api "3.1.0"]
   [ring/ring-servlet "1.5.1"]

Ethan Miller10:03:19

Unless, it’s related to those libs prefixed with commons-...

mccraigmccraig10:03:30

look in your ring-only project - lein deps :tree there should show any commons-codec dep ring has

mccraigmccraig10:03:10

(`lein deps :tree` will only show the chosen transitive deps, not all of them)

Ethan Miller10:03:21

Chosen in the project.clj?

Ethan Miller10:03:36

Maybe that’s related to why the tree in the test project for ring looks quite different?

Ethan Miller10:03:09

[ring "1.5.1"]
   [ring/ring-core "1.5.1"]
     [clj-time "0.11.0"]
       [joda-time "2.8.2"]
     [commons-fileupload "1.3.1"]
     [commons-io "2.5"]
     [crypto-equality "1.0.0"]
     [crypto-random "1.2.0"]
     [org.clojure/tools.reader "0.9.1"]
     [ring/ring-codec "1.0.1"]
       [commons-codec "1.6"]
   [ring/ring-devel "1.5.1"]
     [clj-stacktrace "0.2.8"]
     [hiccup "1.0.5"]
     [ns-tracker "0.3.0"]
       [org.clojure/java.classpath "0.2.2"]
       [org.clojure/tools.namespace "0.2.10"]
   [ring/ring-jetty-adapter "1.5.1"]
     [org.eclipse.jetty/jetty-server "9.2.10.v20150310"]
       [javax.servlet/javax.servlet-api "3.1.0"]
       [org.eclipse.jetty/jetty-http "9.2.10.v20150310"]
         [org.eclipse.jetty/jetty-util "9.2.10.v20150310"]
       [org.eclipse.jetty/jetty-io "9.2.10.v20150310"]
   [ring/ring-servlet "1.5.1"]

Ethan Miller10:03:23

But you are right, there commons-codec shows up as a trans dep.

Ethan Miller10:03:15

But that did it!!

Ethan Miller10:03:00

I add the top-level exclusion and then [commons-codec “1.6”] as shown in the test project ring dep tree.

Ethan Miller10:03:00

Thanks @mccraigmccraig ! Had wasted several hours on that.

mccraigmccraig10:03:16

yw @ezmiller77 - i usually put :pedantic? :abort in the top-level of project.clj which forces you to resolve all transitive dep ambiguities (or it errors)

mccraigmccraig10:03:09

they can be very tiresome on some projects. i have been known to hanker after npm

mccraigmccraig10:03:05

but afaics npm style deps are not possible with java or clojure because of the global namespace / lack of first-class packages

dominicm11:03:35

@mccraigmccraig you can do npm style via Mr Anderson!

mccraigmccraig11:03:44

ah, i had forgotten about mr-anderson @dominicm - how well does it work out in practice?

dominicm11:03:50

@mccraigmccraig it's got a few small edge cases, but mostly things I'd consider bad code anyway. Most people use it for CIDER without realising

dominicm11:03:36

@mccraigmccraig the only real limitation is the fact you can no longer pass objects between libraries. But we operate on data right? :D

mccraigmccraig11:03:00

what does 'objects' mean in this context @dominicm ?

dominicm11:03:54

@mccraigmccraig java objects and maybe records too. Can't have two versions of the same library which create the object, as they'll be different.

dominicm11:03:07

So bidi1s vhostmodel is different to bidi2s. If library A gives the model to bidi2, and you created it via bidi1, it won't work.

mccraigmccraig11:03:24

gotcha - that's not too bad - i'll give mr.a a spin next time i'm scratching my head over dep conflicts

pwrflx12:03:06

hi! when using CIDER's "go to definition" function, is there a way to skip pressing enter? (eg before actually jumping to the definition is asks you again what symbol you are interested in)

dergutemoritz12:03:25

@pwrflx Yes, set cider-prompt-for-symbol to nil

plexus13:03:54

I'm trying to understand this behavior of :exclusions in leiningen dependencies. buddy-auth and transit-clj both have transitive dependencies on com.fasterxml.jackson.dataformat/jackson-core, with differing versions

plexus13:03:03

:dependencies [,,,
               [buddy/buddy-auth "1.4.1"]
               [com.cognitect/transit-clj "0.8.297"]
               ,,,]

plexus13:03:12

[com.cognitect/transit-clj "0.8.297"] -> [com.cognitect/transit-java "0.8.319"] -> [com.fasterxml.jackson.core/jackson-core "2.3.2"]
 overrides
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]

Consider using these exclusions:
[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]

plexus13:03:09

Leiningen suggests excluding the buddy-auth version, but I actually want to exclude the transit-clj version, but that doesn't work. If I do this

plexus13:03:12

:dependencies [,,,
               [buddy/buddy-auth "1.4.1"]
               [com.cognitect/transit-clj "0.8.297" :exclusions [com.fasterxml.jackson.core/jackson-core]]
               ,,,]

plexus13:03:28

then com.fasterxml.jackson.core/jackson-core simply doesn't get loaded

plexus13:03:59

it also doesn't show up as a cheshire dependency when doing lein deps :tree, even though the above warning shows that it is

plexus13:03:23

$ lein deps :tree
 ...
 [buddy/buddy-auth "1.4.1"]
   [buddy/buddy-sign "1.4.0"]
     [cheshire "5.7.0"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"]
 ...
 [com.cognitect/transit-clj "0.8.297"]
   [com.cognitect/transit-java "0.8.319"]
     [com.fasterxml.jackson.core/jackson-core "2.3.2"]

plexus13:03:32

anyone seen this before? Is this a leiningen bug?

plexus13:03:08

I've "fixed" it by explicitly using loading the right version, but I'd like to understand why :exclusions doesn't seem to work in this case

rauh13:03:23

@plexus Can you show your entire project.clj? That exclusion should def work

plexus14:03:48

note the commented out final dependency, i.e. quick fix

plexus14:03:40

it's also strange that the cheshire -> jackson-core dependency doesn't show up in lein deps :tree, right?

rauh14:03:10

No, that's expected, the final tree will show exactly what will be loaaded

rauh14:03:35

So if some other dep actually pulls it in, then it won't show up.

plexus14:03:42

ok, now it's working with :exclusions... not sure what I did wrong before. Thanks for the rubberducking! 🩆

keymone14:03:38

what is idiomatic core.async way of mapping a function over messages in a channel?

plexus14:03:15

you can pass a transducer when creating a channel

plexus14:03:48

(def ch (chan (map my-fn)))

keymone14:03:54

that is preferred over (map fn ch) ?

plexus14:03:06

not sure if that works, a channels isn't a collection (a seq).

plexus14:03:27

the alternative would be to read from the channel yourself in a loop and apply the function to each value

keymone14:03:04

ok, i’ll try transducer

plexus14:03:25

not an expert on core.async so don't quote me on that, but the transducer version is definitely efficient and idiomatic, it's one of the use cases for which transducers were introduced

keymone14:03:06

it does seem nice, though it also means i have to know what i want to do with a channel when i’m creating it

qqq15:03:13

(defn repeat-str [s n]
  (apply concat (repeat n s)))
Is there a better way of doing this?

paulocuneo16:03:37

how can i get a host-independent AST from tools.analyzer? i.e. independent of cljs or cl# or cljvm? Im currently experimenting with clojure.tools.analyzer.jvm

tbaldridge17:03:17

@paulocuneo that's not easy since many macros, forms and builtin sexprs are platform dependent

tbaldridge17:03:36

and analysis happens after macro expansion,

paulocuneo17:03:18

oh ok, will play with clojure.tools.analyzer.jvm for know. Or maybe switch macroexpand-all. Just wanted a macro-expanded/standartrised AST.

mobileink18:03:48

i need to call a fn for each complete path in a map. e.g. for

{:foo {:bar {:baz "hello"}}}
call a fn, passing "hello" as arg. what's the best way to do this?

mobileink18:03:06

i could have a dispatch fn for each level, or i could write a vectorize fn that turns each path into a vector and dispatch on that. but i'm guessing there's a better way.

mobileink18:03:00

or a dispatch map, where the leaves are the fns to apply to the leaves of the data map?

paulocuneo18:03:27

what does complete path means? you mean the leafs of the tree?

mobileink18:03:01

i don't need to worry about the non-leaf nodes, although that would be a more complete solution.

mobileink18:03:38

i guess this boils down to kinda minimal css selector (or xslt) logic.

paulocuneo18:03:59

knowing the path would be just: (f (get-in tree [:foo :bar :baz]))

mobileink18:03:59

right, but i would have to construct the selection vector from the data. that's where a vectorize fn would come in handy.

mobileink18:03:05

i'm kinda liking the idea of a dispatch map, e.g.

{:foo {:bar {:baz (fn [x] ...)}}}

mobileink18:03:48

somebody must have done this aort of thing before.

yonatanel19:03:58

@mobileink Is the shape of the data known? Is it a single function for all leafs? What's the required output?

paulocuneo19:03:43

(Not analizing performance )this would flatten the tree.

(flatten
                 (clojure.walk/prewalk (fn [a] (if (map? a)
                                                (seq a)
                                                a)) {:foo {:bar {:baz "hello"}}})

mobileink19:03:47

@yonatanel the data map represents html meta elements, so each leaf will generate a <meta. .. > element. a single fn would work, but would have to be oarameterized by the path.

mobileink19:03:38

@paulocuneo hmm. i don't think walking would work, since it does not afaik remember the path. there may be multiple optional keys at each level.

paulocuneo19:03:06

how about something like(didnt test it yet)

(defn merge-apply [dispatch tree]
  (merge-with (fn [dispatch tree]
                (cond
                  (map? tree)     (merge-with merge-apply dispatch tree)
                  (map? dispatch) tree ;; handled missing dispatch as identity
                  :else           (dispatch tree)))
              dispatch 
              tree))

yonatanel19:03:33

@mobileink Can you design a flat map in the first place or is it not feasible?

mobileink19:03:20

@yonatanel for example, instead of msapplication-window,

{:ms {:application {:window ... }}}
and similar for :apple etc. it's mostly about aesthetics, and clojurization.

yonatanel19:03:20

@mobileink This is not necessarily beautiful clojure, unless other parts of your code enjoy the nested map

mobileink19:03:41

@paulocuneo i'll have to stare at that a while. dunno about the second map? expr.

paulocuneo19:03:27

dispatch map is the map with the fns

mobileink19:03:47

might not be pretty clojure, but it beats the snot out of HTML. 😉

paulocuneo19:03:49

tree is the map to witch you want to apply the dispatch map

paulocuneo19:03:28

it may have bugs, since its just a sketch

paulocuneo19:03:47

merge-with is baised to the left map

paulocuneo19:03:14

(if i remember correctly)

yonatanel19:03:06

@mobileink How do you represent width and height?

mobileink19:03:19

the data map is written by the user, for a webpage. the idea is that html meta data should be just like clojure meta data - you express it with a map.

mobileink19:03:04

:width, :height. or maybe :w and :h.

yonatanel19:03:07

But then you have to remember that :width and :height, though being separate leafs, must go into the same meta tag as a single string

mobileink19:03:44

for example:

<meta name='msapplication-window' content='width=800;height=600'>
becomes sth like
{:ms {:application {:window {:w 800 :h 600}}}}

mobileink19:03:46

yes, good point, it's not enough to just deal with leaves. thanks.

yonatanel19:03:29

@mobileink With how clojure.spec works and the headache of traversing a nested map, it might be more elegant to do something similar to {:ms.application.window/size {...}

mobileink19:03:45

i hadn'thought of that. but i'm already namespacing my keywords so i'm not sure that would work. i guess i could put / where you have .

mobileink19:03:07

e g {::html/ms/application.. etc.}

mobileink19:03:45

how i love namespaces!

mobileink19:03:00

but that would complicate things with spec, i think. just starting to use it. i don't know if it's possible to validate kws with spec.

yonatanel19:03:58

@mobileink What do you mean validating kws?

paulocuneo19:03:18

@mobileink the code i share you may handle intermediate nodes with a little tweaking

mobileink19:03:20

so that e.g. ::html/ms/aplicattion would be caught.

mobileink19:03:17

@paulocuneo thank you, i will play around with it.

yonatanel19:03:36

@mobileink spec isn't meant to check keys that aren't supposed to be there since it might not only be typos but also keys you will support in the future. Adding your own code to check typos is easier than traversing maps. BTW separate namespace words with a dot and put a slash before the keyword.

mobileink19:03:33

what about required? can i say in spec that ::html/foo is optional, but if you have it ::html/foo/bar is required?

mobileink19:03:50

the problem is that kw names are not really structured, are they? isn't "/" just another char, with no special significance?

tbaldridge19:03:52

Spec is based on predicates, you can do almost anything

tbaldridge19:03:09

yeah, and ::html/foo/bar isn't a keyword 😉

tbaldridge19:03:17

it's :html.foo/bar

tbaldridge19:03:31

or something like ::hf/bar

mobileink19:03:18

::hf/bar/baz is not a kw? sorry, not at my machine so can't test, but i'm pretty sure i've used / in the name part of kws.

mobileink20:03:56

@tbaldridge i'm assumming the elements in s/keys :req [.. ] must be kws. can they be general predicates (fns)?

tbaldridge20:03:30

not in s/keys but you can put them in s/and and under a s/def

mobileink20:03:12

ok, thanks, will experiment.

nicmcphee20:03:59

@bcbradley Thanks. I realize that you can’t swap between rationals and floats and preserve exact semantics, but I would have hoped that Clojure would have tools for generating valid EDN (or expand EDN to include rationals).

bcbradley20:03:06

@nicmcphee try converting the rationals to strings and storing them that way

bcbradley20:03:55

if I were you and I wanted a general solution, I'd walk through everything and when i found something that satisfies (ratio? ...) I'd replace it with the data structure [::Ratio X], where X is a string version of the ratio that can be converted back later

bcbradley21:03:27

you want ::Ratio there because it will allow you to find the the strings that are supposed to be ratios if you walk again

bcbradley21:03:48

{::Ratio X} would also work

dominicm21:03:49

Sorry, I'm confused, EDN does support rationals?