Fork me on GitHub
#clojure
<
2017-03-20
>
sova-soars-the-sora00:03:28

Could someone kindly point me to a looping primer for Clojure? I want to iterate over a vector. I have segments that I want to upsert. So like [ 0 1 2 3 4 5 6 7] becomes [ j j j j 4 5 6 7] to represent the first 4 slots taken by item j. I have learned that I can use (assoc vec index replacement) ... what if I want to replace over a span of indices? How can I go about this =]?

noisesmith00:03:20

the straightforward way would be a reduce, you could also use loop though

noisesmith00:03:04

+user=> (reduce (fn [v [idx e]] (assoc v idx e)) [:a :b :c :d :e] [[0 'j] [1 'j] [2 'j]])
[j j j :d :e]

qqq00:03:16

in clojure, is there a way to say "default to everything is private; only export the following functions" ?

mars0i00:03:33

@sova I don't know what the larger context is, but in Clojure you can often avoid explicit looping, or even using reduce as noisesmith suggests. map is very useful, and there are many handy functions that often end up doing what you want very simply without anything loop-ish at all. concat, partition, partition-by might be useful for what you described, for example. (My apologies if I'm only telling you things you know.)

noisesmith00:03:16

marsOi - there's no way to do that replacement with map, concat, partition, partition-by etc. - it's an associative operation, not a sequential one, as presented

noisesmith00:03:06

for a series of similar associative ops, reduce is usually the best pick (unless something more specialized is available - but replacing a list of indexes isn't a built in)

noisesmith00:03:21

but absolutely, we can often avoid loop and reduce, and should when we can

yonatanel00:03:42

Really just for fun: (apply assoc [1 2 3 4] (interleave (range 1 3) (repeat :j)))

noisesmith00:03:00

or really (apply assoc [0 1 2 3 4 5 6 7] [0 'j 1 'j 2 'j]) meets the original problem statement pretty directly

noisesmith00:03:15

(if I read it right)

yonatanel00:03:42

Where is a good place to start learning about performance and profiling in clojure including memory consumption, cpu cycles and cache misses?

paulocuneo02:03:17

Im no profiling expert, but locking at cache misses/cpucycles seems sooo low level. First thing I suggest to try(as a didactic exersice), is start a java program in debug mode, set a breakpoint, y launch the java visualvm, start sampler, and then release the breakpoint. The sampler will show the % of cpu utilization an memory utilization. If your program does anything interesting like a for loop. the sampling will give a measure of usage. Once I have worked in an app that did linear lookup upon 30.000 registers. (at first i didnt know it was doing linear search) then i refactor the search into a tree-search, the lookup time was reduce to %30 of the original. The final solution was to not do the search at all(well kinda). O(n) -> O(log n) -> O(1) Often "developers" change code to "improving performance" but they just pulled out of their a$$. Always do at least a "debugging profile" so you can know where the program spend most of its time. Its better if you have benchmarks that simulate "real program input". the jvm does a lot of optimizations under the hood(google jit and hotswap, inlining) just write nice readable code and the jvm will take care of memory and optimizations. Also write "good algorithms" when you have voluminous inputs.

yonatanel08:03:28

Thanks. I'll check out visualvm. The low level stuff is interesting though.

paulocuneo01:03:49

oh, that reminded me ... a question i have, whats the clojure convention when generating lambda names, i need to know how to lookup the fn in the java visualvm profiling tool

noisesmith01:03:24

@paulocuneo the only right way to do it is to give them names

noisesmith01:03:07

it still has to have randomly generated aspects, but at least you can look it up in a profiler

noisesmith01:03:27

othwerwise, it will have the name of it's parent in it (if any) as a nested class name

noisesmith01:03:19

this makes comp, partial, #(), etc. much less useful than they should be, sadly

paulocuneo01:03:23

it would be nice if clojure compiler embedded the line+column in the name

noisesmith01:03:31

that would be nice, yes

noisesmith01:03:13

similar problem with defmulti / defmethod - the stack traces are just garbage

noisesmith01:03:40

on the opposite side of things, stack traces where you have protocols implemented by reify or records are more readable than regular functions, actually

noisesmith01:03:19

(and it's more straightforward to map from the thing in a profiler to its definition and back, of course, in your case)

noisesmith01:03:06

@paulocuneo - in fact, I've gone so far as to replace hash-maps with records that don't implement any interfaces, just so that in my profiler I would know exactly who created them

paulocuneo01:03:52

guess i have to live with that for now.

paulocuneo01:03:50

@noisesmith may be some one could come up with a lein plugin that embeds line+col in nameless fns

noisesmith01:03:23

it would need to be a compiler patch

noisesmith01:03:29

that is, clojure.core

noisesmith01:03:43

lein is a dep manager and build tool

paulocuneo01:03:52

or maybe justa a nasty source rewrite

paulocuneo01:03:58

before compiling

noisesmith01:03:46

I guess you could replace fn, comp, partial, and #() with macros with that behavior - for one's own code at least

sova-soars-the-sora02:03:48

Whoa, thanks very much everyone! Was not expecting such an excellent response. Well, may I ask what interleave does ? my (interleave (range 1 7)) gives me nothing different from just (range 1 7)

noisesmith02:03:43

@sova it interleaves collections - doesn't do much with one collection

sova-soars-the-sora02:03:36

Ah okay, I see, it takes 2 to tango/ interleave.

wei05:03:24

is there a good way to get the size (memory footprint) of an atom?

wei05:03:19

trying to get a sense of how our 8GB heap is allocated. also open to other tools/suggestions

noisesmith05:03:53

a profiler can help you look at allocated data and what owns it

noisesmith05:03:16

for example visualvm (free, sometimes comes with the jdk), or yourkit

noisesmith05:03:01

any tool that helps analyze java heap data (there are many) can work with clojure (though it becomes helpful to understand some clojure implementation to interpret what you get...)

hit02306:03:57

doubt from the book- Clojure for the Brave and True: (defn whiney-str [rejects] {:pre [(set? rejects)]} (fn [x] (if (rejects x) (str "I don't like " x) (str x)))) (def whiney-strinc (comp (whiney-str #{2}) inc)) (whiney-strinc 1) ; => "I don't like 2" What does {:pre ...} do here and what does # { 2 } mean? http://www.braveclojure.com/appendix-b/#Anchor

fellshard06:03:04

That block - {:pre [(set? rejects)]} - is a special metadata map. It's setting a precondition on the function, throwing an exception if rejects is not a set. #{2} is a set literal. Much like you can create a vector like ["this"] and a map {:like "this"}, you can create a set #{"like" "this"}.

skadinyo08:03:09

Anyone know how to merge excel cell using docjure?

skadinyo09:03:35

Oh another alternative. Thanks

nickik11:03:35

@skadinyo I don't know how active or good this is. Simple the only thing I know about Spreadsheets.

skadinyo12:03:47

Turns out I still using docjure but interact directly to apache poi. Thanks anyway 🙂

placeboza12:03:42

Anybody doing Codingame with Clojure?

foobar12:03:42

How can I replace the first matching item in a collection? (in the case where subsequent items may also match)

manutter5112:03:06

hmm, this works:

manutter5112:03:34

looks a bit awkward, tho, there must be a better way.

foobar12:03:08

I think passing multiple collections into map might wokr

foobar12:03:34

How do I supply a value for my short map and then cycle nil though

bronsa12:03:54

@manutter51 what are you trying to do?

foobar12:03:54

e.g. second collection has one element and then returns nil

foobar12:03:02

Not sure if that works with map or not

foobar12:03:15

I want to do something to the first matching item only

manutter5112:03:26

@bronsa I was taking a stab at answering @foobar ‘s question 🙂

bronsa12:03:53

user=> (second (reduce (fn [[rep acc] val] [(dissoc rep val) (conj acc (get rep val val))]) [{1 :a} []] (range 10)))
[0 :a 2 3 4 5 6 7 8 9]

bronsa12:03:57

somethign like this would work too

manutter5112:03:55

I came up with a version using loop/recur:

manutter5112:03:45

bleh, no, my loop terminator fails, it infinite-loops

foobar13:03:51

This seems to work: (map #(if (not (nil? %2)) (* %1 %2) %1) (range 1 5) (conj (cycle [nil]) 2))

foobar13:03:20

Hmmm, except that doesn't work 😞

schmee13:03:50

foobar with Specter:

dev=> (setval [(filterer #(= % 2)) FIRST] :a [1 2 3 4 5 2 2])
[1 :a 3 4 5 2 2]

foobar13:03:10

I went with loop recur

foobar13:03:39

Thanks for the suggestions

borkdude14:03:58

I just learned that Clojure has a subs function. I always thought it wasn’t present in clojure.string because people were supposed to just use interop… 😄

rafaelzlisboa14:03:11

is there any significant performance difference between doseqing in a vector vs in a set?

ericnormand14:03:54

@rafaelzlisboa probably depends on the size

ericnormand14:03:03

it would make a great blog post 🙂

ericnormand14:03:34

you can do some experiments using criterium https://github.com/hugoduncan/criterium

ccann14:03:33

is there anything like cljfmt that checks for idioms like kebab-case instead of camelCase variable names?

shaun-mahood14:03:00

Has anyone ever come across an app or website geared towards reading code rather than writing it? I find that there are plenty of times I would like to load up a library and read through things on a tablet or phone, but doing that on Github isn't particularly nice.

lvh15:03:11

If I have a specter path I use in a bunch of places but I need to tweak it in one place. Do I def it as a vector and conj it, or is there a good reason to use e..g providepath? How do I use providepath?

lvh15:03:45

Context: I want to use [::statement sr/ALL ::action] — most of the time, and then [::statement sr/ALL ::action sr/ALL] in one place, where sr is of course the specter namespace.

rauh15:03:01

@lvh You could (def statement-actions (sr/comp-paths ::statement sr/ALL ::action)) and then use it (sr/select [statement-action sr/ALL] ...)

lvh15:03:22

Ah! didn’t know about comp-paths, thanks

negaduck16:03:52

hello. I use cider in emacs. Is there a way to eval something using elisp, like (cider-eval “(use ‘figwheel-sidecar.repl-api)(cljs-repl)”)?

dpsutton16:03:20

(cider-interactive-eval "(println \"hi\")")

dpsutton16:03:09

@negaduck are you configuring a cljs repl by any chance?

dpsutton16:03:19

checkout cider-cljs-lein-repl and cider-cljs-boot-repl defcustoms, which are automatically invoked when you M-x jack-in-clojurescript https://github.com/clojure-emacs/cider/blob/master/cider.el#L453

dpsutton16:03:35

if that's what you are doing, of course

negaduck16:03:40

@dpsutton, yes, I usually do M-x cider-connect, then evaluate (use ‘figwheel-sidecar.repl-api)(cljs-repl). I have a project with lein-figwheel that starts a nrepl, so I’m not sure if I need to run cider-jack-in-clojurescript.

dpsutton16:03:00

yeah, exactly. and those defcustom's are made to do exactly what you are thinking

dpsutton16:03:19

you can use dir-locals to override them in that particular directory if you like or set them globally

dpsutton16:03:33

there's a #cider channel if you want further help, we're probably monopolizing this channel a bit now

Pablo Fernandez18:03:04

Is it possible to specify the credentials directly in project.clj when using https://github.com/s3-wagon-private/s3-wagon-private ?

ccann18:03:47

like this?

:repositories {"private" {:url "s3://..."
                            :username :env/aws_access_key_id
                            :passphrase :env/aws_secret_access_key}}

Pablo Fernandez18:03:16

@ccann: I was hopping like this: :username "fooo

ccann18:03:30

what’s stopping you from doing that?

Pablo Fernandez18:03:45

I don't know if that works. Do you?

ccann18:03:28

@pupeno I just copy-pasted my access key and secret key as strings in to :username and :passphrase and re-built my uberjar and it worked

Pablo Fernandez18:03:40

Awesome! Thanks 🙂

Pablo Fernandez18:03:14

How do you deploy the library to s3? Does lein do it natively?

ccann18:03:29

lein deploy private if your repo is named private as above

mars0i18:03:04

@shaun-mahood what don't you like about reading on github? would it help to make a pdf from each page and read in a good pdf reader? I like reading github, actually, but you can't add comments easily, as in e.g. iAnnotate.

shaun-mahood18:03:32

@mars0i: That's kind of the direction I'm looking, and I've done it the pdf way before. I guess it's mainly trying to see if there's some tooling out there that I'm missing out on before I go overboard and write one specifically for this (or more likely, think about writing it, realize I should be doing something else, and stick it on my unending list of future projects) 🙂

roberto18:03:36

does anyone have a preferred clojure library that abstracts over the Java Sound API?

m19:03:21

hi! is there a way to circumvent clojure's hygienic macro mechanism? It looks like you can only 'let' if using the x# notation to create randomized symbol names. Check this snippet to see, what I'm trying to do with it: http://pastebin.com/CbAUm2dV

bja19:03:07

(defmacro anaphora-demo [x & body] `(let [~'anaphoric ~x] (do ~@body)))

bja19:03:49

(anaphora-demo "blue" (prn "my favorite color is: " anaphoric))

moxaj19:03:18

@m you might also want to look at gensym - as for the macro, you probably want to iterate those names at compile time

moxaj19:03:43

(defn (symbol "whatever") ...) will not work

moxaj19:03:33

maybe something like this:

(defmacro foobar [xs]
  `(do ~@(for [x xs]
           `(defn ~(symbol x) []
              (prn ~x)))))

mars0i19:03:22

@shaun-mahood have you looked at one of these tools that puts source into a "notebook" with intergrated text, graphics, etc.? I don't remember the name of the one I looked at. someone else here would know. of course that means cloning the repo first.

shaun-mahood19:03:09

@mars0i: No, that sounds cool though!

Alex Miller (Clojure team)19:03:38

gorilla or jupyter are the two mostly likely “notebook” things you’re thinking of

Alex Miller (Clojure team)19:03:57

but I don’t think they are doing what you’re asking for

shaun-mahood19:03:54

They both look cool but a little more built for writing and sharing than reading (at least from what I can tell).

mars0i19:03:16

Gorilla was the one I looked at. Yes, it's not primarily for reading, but I wonder whether there's a subset of the features that would get you closer to what you want. Just a thought.

borkdude19:03:22

What is the preferred way in Clojure for keyword args, I forgot. Pass them as a map or as separate args?

bja19:03:30

This Jupyter kernel has a nice docker image: https://github.com/roryk/clojupyter

borkdude19:03:32

so [{:keys …}] or [& {:keys …}] as function args

rauh19:03:48

@borkdude A map, hands down.

bja19:03:08

@borkdude I greatly prefer a map. Otherwise composing the arguments leads to needless ceremony.

seancorfield19:03:10

@borkdude Pass as a single map. The separate args way doesn’t compose nicely.

borkdude19:03:14

I prefer it too. Thanks!

Alex Miller (Clojure team)19:03:45

separate is imo preferred only at api boundaries meant for human repl use

Alex Miller (Clojure team)19:03:02

for example, the spec api

borkdude19:03:36

It isn’t always obvious where the human repl boundary is… I use the REPL for all my code 😉

Alex Miller (Clojure team)19:03:57

for sure :) there is a degree of “taste” involved

fingertoe20:03:35

Is it proper to call a function with a massive map that you already have in memory, rather than parsing out the one or two details your function is really going to need? Let the receiving functions sort it out?

dpsutton20:03:18

do you intend to test the function?

dpsutton20:03:32

would be annoying to test this thing if you had to make a hugely nested map

dpsutton20:03:55

unless by massive map you mean shallow (ie, many keys)

m20:03:00

@moxaj @bja I'm seeing your macros just now. great!

shader20:03:56

I noticed the warning in ring-middleware-format to "be careful about keywordizing user input!"; what are the possible risks of doing so?

m20:03:19

@moxaj The macro you posted works. I'm just wondering why it needs to be 'for'... With doseq it does not do...

moxaj20:03:49

@m if you used doseq, it would iterate the input sequence, construct a bunch of (defn ..) expressions, throw them away, and return nil

moxaj20:03:25

then, that nil would be unquote-spliced into the do form, resulting in (do)

seancorfield20:03:02

@shader Because you can craft a DoS attack by causing the app to intern arbitrary strings as keywords.

shader20:03:23

so, it sticks around in memory more

shader20:03:38

that does sound like it could be a problem...

seancorfield20:03:17

Yup, a script could cause you to run out of memory if you call keyword on arbitrary input.

noisesmith20:03:48

also you can end up with garbage, nonsense keywords that are misleading when printed out and invalid as data literals (much smaller problem of course)

noisesmith20:03:02

for example, json keys with spaces in them are valid

fingertoe20:03:22

@dpsutton my context would be an amazon Alexa app - Amazon hands me a pile of Json. I dispatch on a couple of items within that input, the remaining input may or may not be relevant to the downstream functions. But if I name the json data as Context, for example I can just pass it to any Any function rather than having to decompose it at the front.

mpenet21:03:47

isn't kw interning using weakrefs now? didn't check that stuff in a long time, but if kw are indeed never gc'ed stuff like ring wrap-keyword-params would also be problematic

noisesmith21:03:56

the risk would be if someone could predict the hash state (and thus generate an arbitrary number of collisions), but I find that doubtful, even if a client knew they were the first to connect to your app after a restart

noisesmith21:03:28

I’m strongly of the opinion that keywords are for literals in source files and not user input though (I’m aware most people disagree and prefer to turn user input strings into keywords if they are used as keys in a hash map)

tanzoniteblack21:03:58

I used to always like having json convert to keyword encoded maps, just because I liked in the repl and in the code using keywords to grab things. I ended up moving away from this after I realized how much of a performance hit my code was taking in production by doing the string to keyword conversions. It got a lot better with either clojure 1.7 or 1.8 (I don't remember exactly what changed, but string->keyword got a lot faster for json conversion, maybe it was the transducer release?), but still just not worth it

mpenet21:03:03

@noisesmith it' usually a bad idea yes

mpenet21:03:36

we could have an equivalent of erlang list_to_existing_atom, but meh

dergutemoritz22:03:41

FYI: There's find-keyword

a.sammich21:03:45

Is there a more appropriate channel for clojureclr?

noisesmith21:03:29

there’s #clr but it seems not to be very active

mpenet21:03:46

it's a common source of memory leak for beginners in erlang yes, but it's not a problem in practice once you know about it

a.sammich21:03:15

Thanks @noisesmith. I missed it.

mars0i21:03:42

@fingertoe I thought about this, a little bit, for the project I'm working on. I pass the same map around from function to function. I assume that it doesn't get copied, since it's immutable--just the pointer actually goes from function to function. (Someone correct me if this is wrong.) I like the idea of only passing the relevant arguments, because that makes it clear which ones are being used. It's documentation that's not there if you pass the big map. On the other hand, I found that I'd change something and use another one of the values in the map, and if I'd passed only the relevant parts of it, that would then mean that I would have to change every function that called this function. To me it seems like a judgement call which is better. Depends. Others will have more experience with this question than I have, though.

a.sammich22:03:07

@mars0i with destructuring you get the best of both worlds. I tend to just pass the whole map.

mars0i22:03:46

Good point. let with destructuring is always the first line of the function.

a.sammich22:03:13

Just destructure in the function argument vector. (defn foo [{:keys [a b c] :as m}])

a.sammich22:03:47

Or, if you want the local variables renamed (defn foo [{arg-a :a arg-b b :keys [c d] :as m}])

mars0i22:03:33

yes, but for some reason I don't like destructuring in the fn args. It's messy. I'd rather see one map arg and then the other args. and then destructure separately.

mars0i22:03:02

taste, I guess.

mike70657422:03:13

Does anyone know how I might create a spec with a working built-in generator for a value that is a permutation of a specific collection?

mike70657422:03:59

(s/def ::number (set (range 1 101))
(s/def ::permutation-of-numbers (s/every ::number :count 100 :distinct true))

mike70657422:03:44

That works sometimes, but the generator will fail to find 100 unique values fairly often

mike70657422:03:53

Never mind, I think I figured it out:

mike70657422:03:12

(def numbers (range 1 101))
(s/def ::permutation-of-numbers
  (s/with-gen
    (s/coll-of (set numbers) :distinct true :count 100)
    #(gen/fmap
      shuffle
      (gen/return numbers))))

hiredman22:03:17

that is going to behave badly if you are using it for generative tests

hiredman22:03:33

(using clojure.core/shuffle)

mike70657422:03:51

In what way?

hiredman22:03:38

gfredericks would know better than me, but I am pretty sure shrinking depends on deterministic generation

mike70657422:03:45

That makes sense, I saw gen/shuffle earlier, I'll use that instead

devn22:03:12

I want to make a simple spec which only uses multi-byte UTF-8 strings. Ideas?

devn22:03:34

Something more specific than (s/def ::utf8 string?)

hiredman22:03:26

can you write a predicate for what you want?

hiredman22:03:03

string? is out because java.lang.String is utf-16

hiredman22:03:11

(by definition)

devn22:03:15

i could, just curious if anyone had suggestions on how to write it

mike70657422:03:13

Are there any strong reasons for wanting to avoid a runtime dependency on test.check?

mike70657422:03:42

The spec guide seems to imply that there are

mike70657422:03:47

Aside from the idea of minimizing the number of dependencies in general

m22:03:55

anyone an idea how I can make the following macro returning qualified symbols?

m22:03:09

(defmacro my [] '(defn f [& args] (prn args)))

gfredericks23:03:07

@m I don't think it's clear what "returning qualified symbols" means here; maybe describe what you want the macro to do more generally?

m23:03:32

you are right, that was not clear enough. The function that the macro defines will print the list oft the args passed to it. However, how would one change it that the args to f are passed to prn directly. One would be tempted to use (prn ~@args) , but this does not work due to the normal tick ' that is used. However having a ` backtick instead causes troubles with the [& args] part.

m23:03:49

So I'm looking for something like this: `(defmacro my [] `(defn f [& args] (prn ~@args)))`

m23:03:16

[arr, sry. this is confusing because of the outer backticks which actually should mark the whole as a codeblock.]

jeff.terrell23:03:54

@m - You can use lines with triple backticks to start and end the code block containing backticks.

noisesmith23:03:02

your next problem is that args are not visible to the macro, so ~@args# will be an error

noisesmith23:03:15

because the macro exists before args are supplied

m23:03:58

exactly! this is the problem...

noisesmith23:03:12

@m a macro needs to return the form that clojure should compile

noisesmith23:03:34

it can’t access data that isn’t in the literal text of the macro call in the source code

m23:03:21

right. I see now, because the length of args can change at runtime...

noisesmith23:03:49

not only that - because all your macro can see is the symbol “args” - it can’t access the actual value

noisesmith23:03:58

because the defn sees the args, not the macro

noisesmith23:03:49

you could generate code inside the defn that does something with args - but at that point you’re better off writing a function (instead of your macro, or as a helper the macro can generate a call to)

m23:03:51

so the only thing I can do is to use apply in the fn (which I initially wanted to avoid because of the overhead, but makes sense now)

noisesmith23:03:28

yeah, generating a call to apply works, and don’t worry about the overhead - if it’s not a protocol method the .applyTo method is always getting called regardless

m23:03:40

true, it does not need a macro anymore then..

noisesmith23:03:44

just look at a stack trace to see 😄

darwin23:03:07

what would be wrong with this? (defn f [& args] (apply prn args)), or just (def f prn) I think you don’t need a macro at all

noisesmith23:03:31

I think m just came to that decision actually

mike70657423:03:53

Are you trying to create a macro that defines a function that will always print its args before executing?

noisesmith23:03:05

I assumed prn was a simple placeholder

noisesmith23:03:14

(replacing “magic goes here”)

mike70657423:03:37

That might make more sense

m23:03:34

Yes, prn was a placeholder 😉

m23:03:00

@noisesmith are you sure that apply does not have a huge overhead? compare: (time (doall (dotimes [_ 1e7] (+ 1 2)))) and: (let [x [1 2]] (time (doall (dotimes [_ 1e7] (apply + x)))))

m23:03:46

On my machine its 100 times slower...

noisesmith23:03:02

doall around dotimes is silly

noisesmith23:03:22

the difference there is not because of apply though, it’s because of inlining

m23:03:28

what do you mean by inlining?

noisesmith23:03:02

the clojure compiler knows when it can generate a direct call to add two longs, instead of two instances of Number

m23:03:44

aha. Interesting. When using string concatenation as an example the time difference is much lower (only the double!)

noisesmith23:03:53

yeah - so maybe there is another cost I’m forgetting in there to double the time

noisesmith23:03:17

but the apply method usually is in there, you can see in stack traces for regular function calls

noisesmith23:03:42

oh - wait, now I’m remembering, apply does have an overhead, which is why clojure.core functions have so many arity overloadings instead of jumping straight to apply for the 3 arg case

noisesmith23:03:59

it’s not huge, but clearly also not free