I wish there was something like for but with indices

having to use map-indexed is so much less ergonomic for certain problems

I can help you out with that

one second while i steal other peoples code and hack it at the repl

does that do what you want?

there's always this idiom: (for [[i x] (map-indexed vector l)] ...)

:thinking_face: that doesn’t iterate twice?

my official answer: don't think about it, its fine

it's a lazy-seq, it's realized on demand

so it only needs to iterate once (but it does build an intermediate collection, which means some garbage that wouldn't get created in the lazy-seq, reduce, or loop version of the code)

but any of those would be lower level

or even ignoring the lazy semantics

iterating twice and doing one thing each time is the same as iterating one and doing 2 things

in terms of actual machine behavior that is not actually true

chasing pointers is more expensive than math

but most of the time you shouldn't have to care

also, directly relevant to this case, on the jvm short lived objects (eg. our two element vectors) that only exist in a limited scope are actually cheap, there's a special case for it in the gc

they become more expensive when they stick around and get accessed again

nerd

👍

I've probably spent too much time learning trivia about the jvm and functional programming, but the main point here is that intuitions about performance and resources that come from place oriented programming get trickier in the language and environment we use

how do I supress the output from a certain module at the repl

im trying out monger and the database logs are sortof getting in the way

@emccue With emacs/cider the monger noise gets shuffled up and it doesn't interrupt whatever I'm typing at the repl prompt. Would be nice to be able to turn it off and on though. Wouldn't want to suppress it altogether, it's sometimes useful feedback.

it doesnt interrupt my typing, but it definitely interrupts my reading

because anything evaluated will scroll away to heaven as a flurry of cluster notifications flow on by

yeah, it's a pain. Have to catch it by scrolling up a bit so the stream of notifications doesn't move it up any further. I'm really hoping someone has a better solution to this

Newbie here, sorry if this question is stupid - Why is *command-line-args* becoming nil when running my program from a compiled jar file? it shows actual input values though when being called from lein run ...

Hmm. I'm not sure *command-line-args* is expected to work with AOT compilation.

If you have a -main function, it's called with the command-line arguments, though.

is there a way to assign a name to existing anonymous function?

i have a feeling it’s somewhere in metadata but my googling skills fail me

purpose is to have better error message when that function fails

@keymone You can do (fn my-function [...] ...) and my_function will show up in the stacktrace. (Also useful for writing recursive functions, because you can refer to my-function in the function body.)

yeah, that bit i know, but i need a way to assign name to existing anonymous function that i didn’t create

You can’t do that

😞

just to clarify - there’s no way to do anything to existing fn object so that it shows up in (str x) or the error message generated involving it?

could you not wrap it (fn SPOT-THIS [& args] (apply not-my-anon-function args)) or something like that?

if you just want something that is easier to see in a stacktrace

yeah, that’s my last-resort option

is it possible to treat namespaces as a value?

no -- not like first-class modules in ocaml

how’d you know exactly what I wanted? 😭

I've been around a while 🙂

that said, if you really want to, there is a decently complete api for manipulating/extending/creating namespaces..

you can use protocols and reify to achieve some similar functionality

it's just not very convenient to use if you're trying to emulate functors

mainly I appreciate the aesthetics of file-as-a-unit in OCaml; the idea that you can kind of “pass around files” and transform them is quite nice

yeah well, that just wouldn't play well with clojure's evaluation semantics where the unit is an expression, not a file or a namespace

yeah. my original thinking was to have an API where you could give it a namespace that had a few expected vars for certain functionality, and the API would simply call them.

sounds like it’s not going to be very nice to do

https://github.com/Bronsa/neurotic

namespaces are not modules, even conceptually, if you want an abstraction/encapsulation mechanism, like @ghadi says, protocols/reify/defrecord are what you want

@dpsutton eugh don't use that :P

haha 🙂

agree with @bronsa who agrees with me: you want a protocol and various impls of it.

Is there a specific channel for java/JVM issues?

we’re seeing unexpected behaviour in or Clojure app with > 1TB of off-heap memory usage; which is incredibly hard to diagnose. As far as we know there aren’t any native calls to malloc, so the suspects are bytebuffers (either from the XML parsing or JDBC) or something completely else. But all the usual tools for JVM inspection fail here …

@joelkuiper JVM observability is really quite good, especially in this area. Do you use a particular metrics library? I suggest micrometer if not -- it has built in exporting of off-heap / on-heap buffers. Same with prometheus and its associated client libraries. The source of that data is JMX

If your hypothesis is correct, you should see a large size of off-heap buffers

hmm I’ll take a look at micrometer, jvisualvm wasn’t of any use

it’s being created at 1.5GB/s and is outside the heap and perm gen stuff (also the GC does /nothing/ :p)

GC doesn't apply to off-heap

I know 🙂

have any mmaped files?

not in this particular bit of code

it’s really just a JDBC stream (from Postgresql) to an XML lib to RabbitMQ (to produce messages for upstream processing)

Spidey sense says to make sure the JDBC stream is lazy -- I forget the details, but large resultsets could be buffered in mem if you don't ask for a lazy realization

set fetchSize

yeah, that was also I hunch I had … but I /think/ that was being set. Let me double check 😛

would jdbc stream be on the heap?

we have set fetch-size, so that’s probably not it. it’s been the same value for years now.

(deftype SqlReduce [conn query params query-timeout fetch-size ptf]
  clojure.lang.IReduce
  (reduce [this f]
    (.reduce ^clojure.lang.IReduceInit this f (f)))
  clojure.lang.IReduceInit
  (reduce [this f init]
    (trace "DB: reducing sql query" query params)
    (jdbc/with-db-transaction [tx conn]
      (let [rfn (^:once fn* [coll]
                 (let [coll (if ptf (pmap ptf coll) coll)]
                   (reduce f init coll)))
            opts {:fetch-size fetch-size
                  :timeout query-timeout
                  :result-type :forward-only
                  :concurrency :read-only}
            prepared (jdbc/prepare-statement
                      (:connection tx) query opts)
            sql-params (cons prepared params)]
        (jdbc/query tx sql-params {:result-set-fn rfn})))))

it’s set to fetch-size=1000 so that’s potentially not it

pmap? why?

sorry, it doesn't matter, don't let my wtf side track this

the ptf can be somewhat expensive, so we were hoping that some parallelism might speed it up (essentially doing some record pre-processing)

this bit of code hasn’t changed though, and it used to work 😛

check the versions of jdbc and the db drivers? try downgrading them to see?

narrowed it down to the creation of pg-objects when doing a database insert

it seems these objects are never cleared and exist off-heap

@joelkuiper what psql, just normal jdbc?

normal jdbc

we’re now seeing what object creation is the problem, could be cheshire as well (for some jsonb field)

it’s caused by a function like this one, but maybe not exactly this one

(defn clj->strarray
  "Returns a SQL string array object from a given collection.
  Returns nil if v is nil."
  [v]
  (when v
    (let [vals (map unkeyword v)
          vals (map #(str \" (str/replace % "\"" "\\\"") \") vals)]
      (pg-object "text[]" (str "{" (str/join ", " vals) "}")))))

looks like it’s a memory leak in cheshire

(defn clj->json
  "Returns a SQL json object from a given value.
  Returns nil if v is nil."
  [v]
  (when v
(pg-object "json" (cheshire/generate-string v))))

(or the underlying jackson or something)

hmm… Cheshire is very widely used, you’d think a memory leak in the generate-string method would have been reported by now :thinking_face:

Jackson even more so

we’re going to try swapping it for something else. Could also be the pg-object “json” thing

but it’s definitely that function

and agree, it seems to be far fetched … but so is generating 1,5GB/s of off-heap allocations for no reason 😛

where is the pg-object function from?

any chance you are interning a lot of keywords with cheshire?

yes that is definitely the case

(defn pg-object
  "Returns a new PGobject with type and value set."
  [type value]
  (doto (PGobject.) (.setType type) (.setValue value)))

from (:import org.postgresql.util.PGobject)

okay not cheshire. we were leaking a gzip inflater instance somewhere. kudos to the the excellent detective work of @wagjo!

Anyone getting a 503 from clojars currently?

Hi @joshua.d.horwitz! At what url? On a deploy or get?

Toby! Long time no talk, a get

Trying to pull down an artifact, or on the http://clojars.org site? The repo is served by a CDN, so is different machinery

pull down artifact, checked site and got a 503 as well, might just be on my end

does http://clojars.org/ 503, or just a particular URL on the site?

no, http://clojars.org is fine, just http://repo.clojars.org

ah, http://repo.clojars.org is the CDN. It's working for me, is the 503 consistent for you, or intermittent?

consistent, must be on my end, I’ll restart and see what happens

Thanks @tcrawley

My pleasure! Holler if you have other issues.

when folding over a map - I keep getting this error ClassCastException clojure.lang.PersistentHashMap$3 cannot be cast to clojure.lang.IFn clojure.core.reducers/fjinvoke

It seems to error out after the first call to the combinef

what does your fold look like?

(r/fold sosy-merge sosy-reducer paths-to)

where paths-to is a map

` sosy-merge (fn ([] {}) ([m1 m2] (merge-with set/union m1 m2)))`

`sosy-reducer (fn ([] {}) ([xs k v] (let [depth (inc curr-depth) paths (if (empty? path-prefix) (map #(vector (get (:rel-types db) (key %))) v) (reduce (fn [pths prefix] (reduce #(conj %1 (apply conj prefix (vector (get (:rel-types db) (key %2))))) pths v)) '() path-prefix)) cncpt (nth (:concepts db) k) tui (get-in db (conj (into [] cncpt) :tui))] (if (some #(contains? #{"T184" "T033" "T037" "T046"} %) tui) (reduce #(assoc %1 %2 (conj (get %1 %2 #{}) cncpt)) xs paths) (if (= depth 2) xs (merge-with set/union xs (get-symptoms-and-path db cncpt depth paths)))))))`

why does the reducer function take 3 args?

apparently for map - it takes 3 args - basically it splits into key val

It was not working before with just 2 args - and I saw an example in the doc pointing to this issue

no, a reducing function will always get two args, but you can destructure a map entry into a [k v] pair

umm?

https://clojuredocs.org/clojure.core.reducers/fold#example-5abe9360e4b045c27b7fac2b

oh, never mind, that's weird

it seems to always error out after the first call to combinef

although I checked that it is doing what it is supposed to and that the args passed in are correct

I am doing one thing though - that I am calling fold recursively - so basically the get-symptoms-and-path call will fold and return a intermediate map

so get-symptoms-and-path ends up calling the code shown?

yes

well - seems to be some issue with folding over a map - when I change it to vector - I don't get the error

may be I will try with a simpler example and see if it reproduces

maybe it would simplify things to call seq or vec on the arg and use the two-arg reduce

Multiple people on my team are reporting Clojars issues as well. Status page says everything is up, but downloading jars now fails with 503s.

Has anyone tried to use Clojure to control IoT devices? I'm mostly interested in Z-Wave.

There is openHab in Java, but, curiously, it does not allow scripting in Java (hence, no Clojure either).

wdym no scripting?

No custom logic. openHab has a scripting language "when this event happens, do this", but it's can't load arbitrary Java code to react to events.

openHab doesn't need to know, it'll be our secret

And even if it could, I dislike it so much I'd better use something else. It can't even survive disconnection and reconnection of Z-Wave gateway (USB stick) and has to be restarted, while the way to add a new Z-Wave device to the network is to take the gateway, bring it to the device, push buttons on gateway and device and plug the gateway back.

@ghadi Have you managed to control openHab from Clojure?

nope. was thinking you can install a global event handler that dispatched to a clojure function, which would allow you to hot-reload/dispatch

I'm reading the docs again and I think I see a better way: there is a REST interface with an events subscription endpoint. I'll try to use it rather than trying to load something into openHab itself.

@eigenhombre can you give example jars that are failing? Is it all jars for you?

@tcrawley sent it to you out-of-band.

@eigenhombre @tcrawley we were having issues pulling lein-git-deps but it looks like it's ok now

@dbernal One team member who was reporting issues says it’s working for him now as well, thanks.