Fork me on GitHub
#clojure
<
2016-07-13
>
nunb00:07:20

Is there already a project that combines Clojure (ring/compojure) and mitm proxies like Betwixt or Openresty?

nunb00:07:19

I guess littleproxy and this is what I need https://github.com/ganskef/LittleProxy-mitm

stunami00:07:49

newbie question.. But Im looking for a nice way to cope with dos2unix in a file i am slurp-ing.

stunami00:07:40

Have a file thats windows based and has a carriage ruturn and a new line. \r\n . Am running on a mac, and when I slurp and then split the file, the final string has the “0\r"

stunami00:07:55

(clojure.string/replace "_queue-1\\r" #"\r" "") => "_queue-1\\r"

stunami01:07:58

(its basically ^M im tring to cope with)

akiva01:07:55

Don’t you mean \r\n?

stunami01:07:17

Hmm, correct, but Im using split to parse through the file

stunami01:07:28

(clojure.string/split my-string #"\n")

stunami01:07:08

and therefore its not included. Logically now Im trying to change that to #”\r\n” but having some interesting

akiva01:07:29

This could be a time where you might want to reach for and line-seq. http://clojuredocs.org/clojure.core/line-seq

akiva01:07:35

Check out the first example.

stunami01:07:13

Thanks @akiva I’ll give it a go

stunami01:07:18

looks like the wayto go

akiva01:07:12

Excellent!

krchia03:07:17

i think i’m missing something here

krchia03:07:25

sorry, would this question have been better in beginners?

gfredericks03:07:55

krchia: you did the same thing twice and got a different result?

krchia03:07:42

i wanted something like {:left 1 :left 2}

krchia03:07:50

then i just realized now what a dumb thing that is

krchia03:07:32

i’m just trying to associate a bunch of values with the direction left

krchia03:07:15

actually, i could’ve just done {:left i-already-had-this-list} 😞

krchia03:07:22

sorry to waste your time

martinklepsch09:07:04

Can anyone recommend some reads/intros on protocol based design, I rarely use them and I feel like I'm missing out 😄

lmergen09:07:54

it's mostly useful when designing libraries and allowing your users to extend your library. so if you're not into that a lot, you're probably not missing out a lot either 🙂

rickmoynihan09:07:46

martinklepsch: not sure I could recommend anything in particular... What are you struggling with? As lmergen says it's really about where you want extensibility to other things... I use protocols quite a lot though... but not only in libraries for end users to extend. Places to look to apply them are basically anywhere where you're doing a conditional on type, where the condition is doing the same kind of operation... i.e. anywhere you want polymorphism on the first argument. If you're doing java interop they can also be handy

rickmoynihan09:07:44

For example here: https://github.com/Swirrl/grafter/blob/master/src/rdf-common/grafter/rdf/io.clj#L207 I use a protocol to define two way type coercions between types in a library and in clojure/java

martinklepsch09:07:56

There's nothing in particular I'm struggling with I guess, I just feel like I don't have the intuition on when protocols might be a better solution. Also I remember that protocols can make testing easier which is something that sounds generally good 🙂

rickmoynihan09:07:57

the benefit isn't just that end users can extend it, it also expresses the fact that it's polymorphic

rickmoynihan10:07:29

I think polymorphism is the main thing you want to think about... they can to a lesser extent model domain entities/interface expectations... but I find that very much plays second fiddle...

rickmoynihan10:07:10

though if you do start using protocols you'll probably end up with records modelling some domain entities e.g. User etc...

mpenet10:07:03

cyanite is a good example

mpenet10:07:29

it has multiple implementations of its building blocks for different contexts/loads

varunshankar5510:07:12

Hi, Is there a way to write clojure .clj file to a .edn file (I would like to convert functions , maps,etc)

varunshankar5510:07:08

I found com.theoryinpractise to generate edn file automatically when a maven build is done.

varunshankar5510:07:55

The pom.xml has the following plugin: <plugin> <groupId>com.theoryinpractise</groupId> <artifactId>clojure-maven-plugin</artifactId> </plugin>

dm310:07:03

@martinklepsch: I find protocols are best when you have more than one polymorphic function being a part of a whole thing. If you have just one it's OK to use multimethods. However, if the functionality is part of a library I find that protocols are always better as they are more "open".

dm310:07:33

multimethods are also more powerful as you can dispatch on an arbitrary function, but I find that useful in very rare cases

dm310:07:04

performance-wise protocols are always better

rickmoynihan13:07:38

dm3: Interesting, I have a slightly different "style rule" on choosing between protocols/multimethods... I'll use a protocol even for a single function on the whole... And would always prefer protocols over multimethods for all cases where we're dispatching on class. One difficulty with multimethods is that their dispatch rules can be arbitrarily complex, so by saying it's a protocol you immediately communicate precisely how dispatch works.

akiva13:07:19

I actually use defmulti quite a bit. The moment a bunch of if statements become cond statements and complexity ensues, I find defmulti really steps in, helps with extensibility, etc. I do keep the dispatch-fn very simple, though.

rickmoynihan13:07:43

akiva: multimethods are super cool - and the extensibility (as with protocols) is great - I'm just saying that if you can do it with a protocol, use a protocol - otherwise use a multimethod.

akiva13:07:35

For me protocols are specifically for types. Sometimes I want to dispatch based on a key in a map.

rickmoynihan13:07:56

e.g. anytime you see (defmulti foo class) you should replace with a protocol

akiva13:07:19

That I agree with.

rickmoynihan13:07:27

akiva: agreed - and that's what I'm saying

rickmoynihan13:07:52

you can't use a protocol to dispatch on a map key... so you need a multimethod

rickmoynihan14:07:43

likewise if you want to dispatch on (comp :some-keyword meta) or a set/subset etc...

akiva14:07:13

Yep. But I rarely use records. I tend to stick with maps unless there’s a good reason to enforce formality.

rickmoynihan14:07:35

akiva: records are maps - you can assoc new keys on them if you want... so the reason I'd use a record over a map is if I ever wanted to create a map that had a :type key - that I would want to dispatch on.

rickmoynihan14:07:22

and if I wanted type based polymorphic dispatch on a map - I'd prefer to make those maps records/protocols than use a multi method

akiva14:07:46

That’s one of the good things about Clojure and Lisps in general: there are multiple ways of getting things done and not one of those ways is necessarily the right way.

rickmoynihan14:07:22

sure... I'm not saying my preference is the right way... I'm just justifying my reasoning

akiva14:07:47

Same here.

rickmoynihan14:07:15

this discussion is really about crude heuristics for when to prefer one to another... there are tradeoffs behind every decision and specific circumstances lead you down different paths.

akiva14:07:44

Indeed. I guess my guiding principle is to shy away from formal structures such as records unless you absolutely need it. I’ve messed around with it both ways (loose maps vs. ’strict' records) and, as you point out, there are pros and cons to both.

dominicm15:07:47

I'm trying to build some core.async consumers/subscribers, and I'm a little confused on how to approach the problem. I want to make sure I get this right first time. Is there any information on how to set up something like this? My only conclusion is some kind of async/thread (avoiding go?) loop in a component. Is there any other approaches I could/should take?

Alex Miller (Clojure team)15:07:33

I don’t think there’s enough there to answer that question. why avoid go? (btw, there is #C05423W6H too)

Alex Miller (Clojure team)15:07:03

@rickmoynihan: I capture my thoughts on this subject in Clojure Applied if you’re curious. too much to repeat here.

Alex Miller (Clojure team)15:07:51

@dominicm: I also cover what you’re asking about at a broad level in Clojure Applied (using async, and async + components), but again too much to repeat here

dominicm15:07:22

@alexmiller: Have a copy at work, I'll make sure to check it out. I thought go was a bad idea due to the limited thread-pool and that go was more appropriate for async pushing onto channels instead of processing them. I'm not too well versed in async understanding.

Alex Miller (Clojure team)15:07:48

it’s something to be aware of, but using go loops is fine for processing async data. it’s really a question of whether you need a dedicated thread or are able to multiple over a pool.

Alex Miller (Clojure team)15:07:04

also, as of the last release, you can configure the size of that pool

Alex Miller (Clojure team)15:07:44

from a design point of view, I think it’s more important that the parts you expose are the channel ends

rickmoynihan15:07:16

thanks alexmiller I'll take a look

daveliepmann15:07:00

Question: I'm able to call Clojure functions starting from strings by using an approach like (ns-resolve *ns* (symbol "+")). Trying the same approach with Java instance methods fails (`(ns-resolve ns (symbol "System/getenv")) => nil`), I assume because the namespace does not resolve the .instanceMember macro or the dot operator. Is there another approach that would allow me to call Java methods starting from a string describing the method?

Alex Miller (Clojure team)15:07:31

@dominicm: and it’s a bit more flexible if a component does not have to create the channels but is instead given them - that allows a higher-level thing to create the channels, with appropriate buffer semantics, and fit those pipes into the components

dominicm15:07:34

@alexmiller: That's the exact design I was leaning towards. The only things that's internal is a "kill-chan" which serves to let the thread know when to shut down.

Alex Miller (Clojure team)15:07:53

@daveliepmann: starting from strings, you are basically in the realm of Java reflection

Alex Miller (Clojure team)15:07:04

you can find a class with (Class/forName “java.lang.System”)and then use (.getMethods class …) etc to find a method, which can then be invoked with an array of args.

Alex Miller (Clojure team)15:07:32

you could also read-string and eval

rickmoynihan15:07:05

@alexmiller: TBH I'm more interested in when to choose between transducers and lazy sequences - than records/maps which I think I have a good grasp on... Regarding transducers/lazy-sequences which method should be the default for sequence-like manipulation... how to consider the tradeoffs etc... I understand the costs of laziness - but for where the performance doesn't matter so much what is your default option these days?

rickmoynihan15:07:47

I still reach for lazy-seqs - but it's probably because I'm a lot more familiar with them than transducers

Alex Miller (Clojure team)15:07:18

@daveliepmann: (eval (read-string "(System/getenv)”))

rickmoynihan16:07:04

I hadn't seen it no... thanks... the http://clojure.org guide looks good too... I think I understand many of the pros/cons of each.... I think I'm more looking for an "if in doubt - use XXXX` kinda rule... I suspect lazy seqs are still the default default... but I definitely need to write more transducers

Alex Miller (Clojure team)16:07:33

if your collection is small it doesn’t matter

Alex Miller (Clojure team)16:07:48

if your number of transformations is 1 or 2, it probably doesn’t matter

Alex Miller (Clojure team)16:07:01

so use whatever’s comfortable there (prob most situations)

Alex Miller (Clojure team)16:07:28

most people have found that into is their favorite place to use transducers

Alex Miller (Clojure team)16:07:42

as it allows you to transform into exactly the output collection you want

rickmoynihan16:07:50

yeah... re into that's definitely the impression I'm getting

Alex Miller (Clojure team)16:07:03

think harder about transducers when dealing with: lots of data, lots of composed transforms, places where you want to reuse transforms in multiple contexts (async + collections), places where your source is an external resource and you want to know when you’re done (eagerly)

dg16:07:14

There are lots of times where transducers are simply easy quality of life improvements, like mapping over a channel

rickmoynihan16:07:18

My feeling is transducers require more thinking but provide more opportunities for efficiency etc... as is frequently the case when decomplecting

rickmoynihan16:07:51

dg: I think if I used core.async - I'd reach for them more often... but I've not needed to do much event/stream stuff over the past few years.... I have a number of times when I've needed to reached for juc.BlockingQueue's etc to marshal things across threads... but it's usually been pretty simple stuff... I don't think core.async would've helped massively (and I suspect it won't be quite as fast).

Alex Miller (Clojure team)16:07:08

one benefit of async go over BlockingQueues is you don’t have to tie up a thread on the consumer side - if nothing is in the queue, the block is parked and awakes only when there is work to do

Alex Miller (Clojure team)16:07:25

really a benefit over all the Java queues

dg16:07:51

I didn't mean to say core.async is the solution, it's just an example of a place where transducers fit so well that they may be easier than other solutions

dg16:07:30

They don't always need to be a heavily considered "is this the right time?" option. If the solution is expressed very simply with the built-in transducers, I'd default to just using that.

rickmoynihan16:07:13

yeah that is definitely a benefit... but we're probably only running half a dozen of these extra threads in the worst cases.

rickmoynihan16:07:20

dg: agree on just using it - if its not that important... that's why I've been reluctant to change the BlockingQueue's for async

rickmoynihan16:07:43

I think the other thing that's yet to become clear to me are all the tradeoffs around dealing with I/O... I'm currently wrapping an API that uses a thread-based I/O model - and this code basically does it the old-school clojure way of hiding the I/O behind a lazy-seq abstraction.... Obviously it can and will leak resources in some circumstances; but, with some careful coding in a few places in 3+ years of production usage its not been a big source of problems. Regardless I do consider the potential for resource leaks a problem; and would like to move to a better alternative... but I now basically have 3 alternatives to look at... 1) Use core.async channels 2) Use reducers 3) Use transducers There are some additional issues in moving away from the lazy-seq model in that some libraries such as clojure.data.csv also manage I/O in this way. Curious if I'm missing something and what peoples thoughts are on this.

rickmoynihan16:07:26

btw alexmiller thanks for the tips it's useful

mccraigmccraig16:07:03

@rickmoynihan: adding to your pile, i've been using https://github.com/ztellman/manifold for async i/o for a while - the Deferred and Stream impls can convert to/from other abstractions (e.g. core.async chan to Stream) which has made integration with existing libs a snap

rickmoynihan16:07:22

mccraigmccraig: thanks...

asolovyov17:07:51

Anybody knows a good way to turn Schema's errors into a user-facing error messages? 🙂

josh.freckleton17:07:29

Is there any existing macro for defining my own syntax, eg a new form that uses banana brackets: (| ... |)

josh.freckleton17:07:46

or do i have to roll that myself?

semperos17:07:09

I believe what you’re asking for is reader macros, which are not open for extension in Clojure

josh.freckleton17:07:35

@semperos: ah right, reader macros, haven't used them much. So I can't extend my own reader macros? So my use case is more or less not possible?

semperos17:07:41

@josh.freckleton: correct, as a design decision, Clojure does not allow end-users to add further reader macros; what use-case were you envisioning for your new syntax?

josh.freckleton17:07:29

@semperos: I'm playing with some ideas, and trying to learn what's possible/not. Right now, I'm wondering how I could go about creating custom collection types, defined by syntax. For example, this is similar to maps, but I'm thinking a type like data RoseTree a = RoseTree [RoseTree a], or my choice of clojure forms: (| <node value> <branch 1> <branch 2> <etc.> |)

josh.freckleton17:07:53

then I could define utility functions like tree?

semperos17:07:02

I’d put out two rules of thumb on that front

semperos17:07:44

first as a general rule of thumb, you want to stick with using Clojure core collections and its sequence api as much as possible

semperos17:07:32

second, before moving onto syntax, I’d play with defining custom deftypes/defrecords if you need to experiment with how your tree needs to behave

joshmiller17:07:04

I’m having some trouble understanding how keywords work with respect to clojure.spec… Why is it that (s/def :spec-test-ns/name string?) works, but (s/def (keyword “spec-test-ns” “name”) string?) fails with CompilerException java.lang.AssertionError: Assert failed: k must be namespaced keyword or resolvable symbol? The type of both is clojure.lang.Keyword.

semperos17:07:31

might get more focused discussion @joshmiller if you ask in #C1B1BB2Q3

semperos17:07:16

@josh.freckleton: often one finds that a combination of well-designed and well-named functions using regular Clojure maps, for example

semperos17:07:36

gets you where you want to be + the added benefit of being a data structure with lots of built-in support and idioms

josh.freckleton17:07:05

@semperos: > first ... sequence api ... agreed, so I was thinking of extending that. > second ... good advice. I get carreid away sometimes > well-desigined ... functions ... I think you're right. It'd still be cool to have reader macro-access!

semperos17:07:57

I’ve done personal projects on that front @josh.freckleton where I play with the LispReader and Compiler classes; I think you can learn a lot about how Clojure on the JVM is put together by those experiments, but yeah, in Clojure-as-released you don’t have the flexibility

semperos17:07:05

EDN, however, does have a story around data literals which is related

semperos17:07:22

and probably worth learning about if you haven’t already

josh.freckleton17:07:00

> EDN has a story around data literals what do you mean, I thought I was familiar with EDN but are you saying it's extensible?

semperos17:07:03

it’s not a syntactic free-for-all, but it is extensible

josh.freckleton17:07:05

ah, that's what the "E" stands for! haha

josh.freckleton17:07:36

oh that's right! I vaguely recall elarning this, so with the # symbol, I can define arbitrary read-time macros? (i'm still reading throughyour link, forgive if this is obvious)...

semperos17:07:49

it provides you a hook into the reader, so that what follows your #-prefixed tag is passed to a function you write

semperos18:07:27

you’ll need to read into how one tells the reader where to find those functions (differs for core Clojure and something like tools.reader) and restrictions, but that link is a starting point for EDN’s functionality there

semperos18:07:20

that and the next section

semperos18:07:07

but again, I stress the need to focus on the API and its behaviors/boundaries before getting too caught up on the syntactic front

josh.freckleton18:07:42

> before getting too caught up on the syntactic front yes, I agree, thanks for pointing me to these references!

josh.freckleton18:07:19

I'll put them away to mull over at a future time when/if they'll help after I've focused on behaviors/boundaries 🙂

ghadi18:07:40

@sekao: Most exciting mailing list announcement ever. 👏

peeja19:07:55

Is there an easy way to turn a tree of nested maps into a "flattened" map of the leaf nodes, keyed by vector paths to each leaf?

peeja19:07:48

I'm working through walking the tree correctly, but it seems like the kind of thing that might turn out to be two core functions plugged together if you know what you're doing. 🙂

dg20:07:36

From there you could map the result across a get-in call if performance isn't super important

lwhorton20:07:49

is there an idiomatic way to force a (conj nil foo #{})? (force conjs’ struct type as opposed to automatically being a list?)

lwhorton20:07:46

i could do some (if (nil? thing) #{foo} (conj thing foo)) but that feels .. verbose

donaldball20:07:15

Are you looking for (fnil conj #{})?

lwhorton20:07:57

that’s pretty sweet

madstap20:07:58

@peeja: I did that exact thing the other day, using the same stackoverflow answer and this was my solution, using keys-in from that SO answer. It's terribly inefficient I think, though.

(defn key-paths [m]
  (into {}
        (map (fn [path] [path (get-in m path)]))
        (keys-in m)))

peeja20:07:46

I'm currently running with

(defn leaf-seq
  [branch? children root]
  (let [walk (fn walk [node]
               (lazy-seq
                (if (branch? node)
                  (mapcat walk (children node))
                  [node])))]
    (walk root)))

(defn flatten-with-paths [tree]
 (leaf-seq
  (fn [[path node]]
    (associative? node))
  (fn [[path node]]
    (reduce-kv #(conj %1 [(conj path %2) %3]) [] node))
  [[] tree]))

peeja20:07:05

where leaf-seq is a modified tree-seq that doesn't include branches

tom21:07:46

Is there a way to run a function after all tests have been run with lein test?

tom21:07:54

I saw that, seems to be per namespace. I'm trying to figure out the best way to run something after all have been run.

seancorfield21:07:23

Add a shutdown hook in your code, perhaps?

seancorfield21:07:41

(the JVM lets you register a function to run as the JVM is shutting down)

tom21:07:17

I'll take a look at it, thanks!

danburton22:07:57

So, wrt clojure.tools.namespace.repl/refresh. I have the following in my ~/.lein/profiles. :injections [(require '[clojure.tools.namespace.repl :as repl])] However, it seems that whenever I call (repl/refresh), it gets rid of the injection, and the next time I want to call (repl/refresh) I have spell out the whole module name. Any tips?

danburton22:07:26

Another random question. Is there a difference between some? and boolean?

danburton22:07:21

Oh I see. (some? false) #=> true, but (boolean false) #=> false. It would appear this is the only difference.

dg23:07:41

(some? x) is literally (not (nil? x)), and (boolean x) is a coercion to a type

dg23:07:02

Even if they're usually the same result, I'd differentiate based on which kind of thing I'm trying to do