yes... a record already implements java.util.Map

if you want to do something custom, that’s what deftype is for

also potemkin's def-map-type and def-derived-map may be of use to you if you're doing that https://github.com/ztellman/potemkin

(doseq [i (range 100000)] (curse!))

(defn foo [x] ...) // foo takes only a arg
(defn bar [ ... foo  ...] ... (foo 20)) // foo here shadows global foo
(bar ... foo=hash-map)

so now, I don't get an error because hash-maps can serve as functions

and for the life of me, I couldn't figure out why function foo wasn't being called

but what about the times when shadowing is what you want? neither eastwood or cursive warn about shadowing. maybe check on kibit and joker i guess but I doubt they do either

Warning on shadowing is definitely something I’ve considered, but it is what you want freqently.

(i.e. a local called map or list or something)

It would have to be a fairly unobtrusive warning, but it’s bitten me from time to time too.

cursive may have caught this if you didn't use foo anywhere else, ie the (defn foo would be grayed out because not used anywhere

it's used elsewhere

what threw me off was the combination of: foo only took one argument, hashmaps can be treated as functions of 1 arg

so instead of getting a runtime error / exception at where (foo x) happens, I just silently get the wrong answer

@qqq what do you use as an editor?

I use emacs / cider

eastwood will warn about shadowing iirc

doesn't exactly fix your issues, but I use https://github.com/Fanael/rainbow-identifiers to make it so these kind of things jump out at me more easily

ah you're right it does

new as of 0.1.5 https://github.com/jonase/eastwood#local-shadows-var

limiting it to "and is called as a function" is smart, and would probably drop the number of false positives a lot

heh, what do you know

(defn foo [x]
  x)

(defn bar []
  (let [foo {:cat "cat"}]
    (foo :cat)))

local: foo invoked as function shadows var: #'yummly.metadata.ads/foo

@bfabry @tanzoniteblack: thanks for the suggestions

Hi, i've got a plumatic.plumbing question: is there any namespace qualified keywords support for fnk and defnk?

what's the standard way of organizing tests to various files? eg if I have a file component.clj, should I place by deftests into that? or create a separate component_test.clj? If yes, where should that file go to? into the same dir as the component.clj or to a parallel directory structure?

@pwrflx most of the time I use separate file with "_test" suffix located in "test" directory under project root. "test" directory pretty much mirror the "src" dir.

@pwrflx: If you're using leiningen, running lein new <proj> will give you some hints about that

that's good pint too - you should end up with hierarchy that I've just described.

@jumar @jstew will look at it thx

Looking for some inspiration here. I've inherited a stack overflow that happened at least twice, but it's not a regular occurrence and it hasn't happened since before I arrived.

java.lang.StackOverflowError at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$filter$fn__4580.invoke(core.clj:2679) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at 

It just repeats the same trace until it overflows.

I'm trying to interpret this, obviously lazy seqs and a filter(?) There are only a handful of places in the code using 'filter', and a few more using 'remove', but nothing yet that seems out of the ordinary and would cause this.

Unfortunately I cannot reproduce this, any hints at what I might also look at?

remove works via filter, so that's one thing to keep in mind

if you look at the entire stack trace you should see the point where it enters the loop, and that's the best place to start

Yeah, I figured and confirmed that in the 'remove' source, so I've been tracking that as well.

Yes to the entire trace, I fear it was lost and this is the start of what was saved 😛

are you sure you aren't just applying filter to a seq a few million times?

(doall (nth (iterate (partial filter number?) s) 100000)) will result in a similar looking stacktrace

(where s is some seq)

@hiredman that's certainly possible, I'll re-examine with this in mind. Unfortunately nothing jumped out before - looks like single applications of the filter to the collections in question.

Thanks for the help @tbaldridge and @hiredman

concat doesn't show up in your stacktrace, but it is a common culprit in stack overflows when used repeatedly

@the2bears do you have any non-clojure* stacktraces?

it's possible that some lazy sequence is not using the lazy-seq macro correctly...

I looked at concat as well, it's used in the code once or twice but I figured since it's not in the trace it's not the issue.

you don't have the whole trace

@ghadi, unfortunately this sample is what I have (... and lots of it repeating).

Agreed, I'll have to wait to see if this happens again and make sure the whole trace is grabbed at the time.

I think once I have that the investigation will go smoother 🙂

Throwable->map in Clojure 1.8 is really useful

Not familiar with that, I'll look into it, thanks!

can anyone help me understand this behavior with lazy-seq (Clojure 1.8)?

user> (def s (lazy-seq
               (println "resolving seq")
               (cons 1 (lazy-seq
                         (println "resolving seq2")
                         (cons 2
                               ((fn [x]
                                  (lazy-seq
                                    (println "boom" x)
                                    (/ 1 0))) 
                                :x))))))
#'user/s
user> (first s)
resolving seq
1
user> (second s)
resolving seq2
2
user> (nth s 2)
boom :x
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
user> (nth s 2)
boom nil
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)

One of the later lazy seqs throws an error, but when I try to realize that seq again, the body somehow loses the values in its lexical closure

More pubsub questions. Can a single input-channel handle having more than one publication with different topic-fn? My reading of the docs leads me to believe this is true. However as soon as I create a second pub on the input-ch (with different topic-fn) the notifications stop working on the first pub. Any thoughts?

^:once fns

the "thunk" inside a lazy seq is marked with ^:once metadata, making it a "once-fn" -- practically this means that the closed-over values disappear and are cleared after the only legit invocation of the function @aengelberg

oh. wow.

but then why is it trying to call the same function again if it errors, if it's designed to only be called once?

because of the error

lazy-seqs cache the result value, but don't cache the error

why not

there’s a ticket about this I think

but really the general policy is that we don’t make any guarantees about seqs that blow up

https://dev.clojure.org/jira/browse/CLJ-1332

I guess I'll make sure my seq doesn't blow up then

that’s not the particular ticket I was thinking of (although it’s the same thing), was thinking of: https://dev.clojure.org/jira/browse/CLJ-2069

i wonder if there is a perf impact to that ^

this is a good example of a ticket that I have a hard time predicting how Rich will react to it - could as easily be declined as accepted

based on things I’ve heard him say in the past

The reason I encountered this is I had to turn a lazy seq into an iterator that was then passed to a Java class which I guess decided to call .next despite getting an error...?

Oh wait, it looks like the standard call pattern .hasNext followed by .next breaks down when there's an error in the underlying seq.

user> (def i (.iterator ...)) ; same seq as above
#'user/i
user> (.hasNext i) (.next i)
resolving seq
true
1
user> (.hasNext i) (.next i)
resolving seq2
true
2
user> (.hasNext i) (.next i)
boom :x
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
boom nil
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)

Seems like this is a standard use case that shouldn't break?

I'll vote for the jira ticket 🙂

some workarounds in your current situation - convert your lazy seq usage to an eduction with transducer (if that’s possible)

or call (clojure.lang.SeqIterator. the-seq)

which is kind of some now unused inside baseball

is SeqIterator a different impl than the standard .iterator call?

it is, although it’s hard for me to say if either of these will actually get past your issue without trying them

I'll try it out, thanks

Is the ^:once metadata on the generated fn necessary to prevent some sort of memory leak? Or is it just for a perf improvement? I'm curious if I can get around this by manually calling (clojure.lang.LazySeq. (fn [] ...)) instead of (lazy-seq ...)

IIRC, the thunk function will be set to null and collected after LazySeq has invoked it. The ^:once metadata allows the closed-over values to be collected after they are used and before the thunk function has even finished executing.

I don't think the thunk function is set to null, since as you said, it doesn't cache errors, and I'm getting errors again when I call the seq multiple times

so it must have held onto the thunk

it's not set to null if it throws

oh, nvm

https://github.com/clojure/clojure/blob/43540be97668ed453cad74e18f434c3e8bd5d638/src/jvm/clojure/lang/LazySeq.java#L40-L41

I assume that if I don't keep any references to the head of the lazy seq itself, I shouldn't need to care about how LazySeq manages its pointers internally?

right

👍

Thanks for the help, Ghadi and Alex

and @hiredman

hi, does anyone know if it is possible to cancel a promesa promise on the JVM or otherwise workaround the absence of p/timeout on the JVM?

what client are you using?

just raw Java Interop

(a/go
      (loop []
        (when-not @poison
          (let [records (.poll ^KafkaConsumer consumer (:poll-timeout opts))]
            (doseq [r records]
              (a/put! ch (deserialize-fn (.value ^ConsumerRecord r)))))
          (recur)))
...

actually, I'd say that heap graph doesn't actually show anything

periodic garbage collection and noise

Do Ring sessions guarantee any kind of uniqueness?

(it shows a properly working GC, imo)

just wasn't sure if there was something I could do on my end to make the consumer not create so much garbage to collect 🙂

I wouldn't worry about it

It does seem odd to me though that there appears to be about 400MB of garbage created every minute. Most likely due to something else in the system, but that's rather high for a mostly idle app.

ha, garbage generation 🙂 something that's tricky to observe and get insights about

400MB of garbage says little about whether it's 400 x 1MB or 40K x 100KB (out of hat values. workload specific)

@dadair classic use case for a memory profiler.

In Yourkit you can capture all allocations over a few minutes and then the profiler will tell you where the allocations came from.

fwiw, I don't spot any huge garbage generating Clojure construct in the code dadair pasted

Agreed, it's probably in some sort of logging library.

and ehm - sorry for sidetracking - is the topic "garbage generating Clojure constructs" interesting to some ? I have dusty personal code in a drawer that was written with the aim of exploring this topic.

Trying to run VisualVM memory profiling but it causes the app to hang and eventually errors out saying it can't establish a profiling connection -- sampling works, just not profiling (but I'll move this discussion to #beginners)

try yourkit first imo, VisualVM is 1) more trouble 2) less actionable results (or maybe its just harder for me)

dadair : w.r.t. reducing dev bloat, you should not be worried about garbage generation

its mostly going to impact GC metrics (how often, how long) which are hardly noticeable unless you're doing 60Hz stuff

set smaller java max heap sizes, let the JVM explode whenever it grows outside of what you're comfy giving it

I wouldn't tune my stack to my development environment too hard - it's already enough work to tune it for production environments

I have two maps where some of the map values are byte arrays.How would I go about comparing the maps for equality as (= (byte-array [1]) (byte-array [1])) returns false? i.e. is there something I could use in clojure or do I need to write something that walks the maps and runs Arrays/equals or equivalent manually?

@mbjarland one option would be replacing each byte-array with a (vector-of :byte)

which could be done with a map / into or a tree-walk, depending on the complexity of your data structure

@noisesmith…looks promising. Thanks for the pointer

Hello all, is there an easy way to convert some key value of a map instead coping all key-values one by one? [{:a 1234 :b "some" ...} {:a 1223 :b "some" ...}] to {[:a "new_one" :b "some" ...} [:a "new_2" :b "some" ...}] in this case I´ll convert :a value to new :a value

@fabrao (map #(update % :a f) ...) ? I don’t know how you come up with your new values…

but if you have a function f that takes 1234 and returns “new_one” and takes 1223 and returns “new_2" that will work

well, I´ll get the :a value transforme it and update it into array of map

humm, I´ll try it now

@noisesmith that was too easy, thank you, it worked. I see I´m clojure newbie

@fabrao you’ll find that most things that involve data transforms are quite simple in clojure, though sometimes you’ll need to reconsider specific approaches (eg. using associative data structures instead of linear scans, updating a cache of derived data instead of doing in place updates by index, using keyed accumulators in reduce instead of mutating an item as you iterate a collection…)

@noisesmith so updating map is what kind of approaches?

it’s one of the simple things I didn’t list

I only gave examples of the ones that tend to be a little trickier for people

ok, sometimes the imperative programming is in my memories 🙂

sometimes what helps is thinking of how you would do things in SQL

we have group-by etc.

(update m :a inc) is a lot like update table some_row set a = (a + 1)

ugh - that’s not good SQL but hopefully you get the idea

@fabrao think of it as function composition. Clojure has two basic functions to manipulate maps, assoc and update. You would use the former when you want to set the key of a map to a value. It takes a map, a key and a value, and returns a new map with your given value under your given key

If a value is already present in the map for the key you want to assoc, it will override it, e.g.

Now, if you wanted to transform the value under :a in the map by some function f (for example, increment it with inc), you could do this:

But that’s pretty verbose. Clojure core includes the function update, which takes a map, a key and a function, applies the function to the value under that key, and returned an updated map (as @noisesmith demonstrated)

In your case, you had a vector of maps that you wanted to update. Clojure core has the function map which applies a function to every entry in a vector, and returns an updated vector

What you wanted to do was to update a key for each map in your vector, so it’s quite natural to use the function map to then call update on each entry

This probably doesn’t always apply, but when manipulating data like this try to think of what you would do to update the “most nested piece” (here, maps), and then try to think how you would apply this operation if the piece you want to update is nested in a vector, or another map, etc

You might also want to check out https://github.com/nathanmarz/specter. It’s by no mean necessary to manipulate simple datastructures, but it might give you some insights on how to deal with immutable data

@fabrao try to use

->

map

In particular, I think the concept of “navigation” is quite helpful, even when thinking about mapping over a vector of maps, like you did

NB for “navigation” you can check out zippers, which literally let you represent your traversal and updates as an arbitrary series of navigations of the data

zippers add a lot of overhead, whereas specter navigators generally run with near-optimal performance