Fork me on GitHub
#clojure
<
2022-05-12
>
pinkfrog01:05:45

Possible to make this code more concise? Overall, I’d like to maintain the (when-some) overall structure.

(when-some [fname (some-> (some-func-that-returns-a-fname)
                       (as-> fname (format "../base/path/%s" fname))
                       (as-> fname (when (.exists (io/file fname)) fname)))]
    (perform-some-operations fname))

hiredman02:05:25

The fact that you need to call exists and then conditional on that return the name kills your ability to thread

hiredman02:05:32

If you think of code as a graph where names are edges in the graph (a data flow graph) what makes for good threading is code where that graph is just a straight line with a single edge between each node

pinkfrog02:05:49

Yes. When sometimes the arg position have to be changed due to the different invocation functions, I would go to (as->) . But I am not sure how to make the code more clean.

hiredman02:05:06

If you switch to some->> you don't need as-> for format

hiredman02:05:36

Because the format call follows the linear data flow

hiredman02:05:08

You also don't need the outer when-some

pinkfrog02:05:53

How to handle this part? two fnames

(.exists (io/file fname)) fname)

hiredman02:05:42

as-> is the way to handle that if you want to keep threading

hiredman02:05:57

Or just write a function that does that and call it

hiredman02:05:50

I have had former coworkers say they can tell which code I wrote because of all the threading I used. You don't need to thread everything

pinkfrog02:05:33

some->> and as-> does not play well with the other. If I use some->>, I guess I have to write a function to handle (.exists (io/file fname)) fname)

pinkfrog02:05:05

> You also don’t need the outer when-some the final form is a syntax-quoted macro form, and seems have conflict with threading form

jcf07:05:31

The main thing that jumps out at me is the mixture of pure operations with IO-based. I'd break this into two steps. The first that produces a java.io.File (regardless of whether anything exists; just pure in-memory operations), and then I'd check if the file exists and conditionally perform the operation. The benefits are you can test the pure stuff more easily, you can reuse the named file generation logic elsewhere, and the IO (potentially) becomes more isolated.

(defn filename+base-path
  ^File [s]
  (io/file "../base/path" s)

(let [file (filename+base-path)]
  (when (.exists file)
    (perform-io file)))
(Coding on my phone; apologies!)

jcf07:05:54

@UGC0NEP4Y hope that helps. :man-bowing:

thanks3 1
jcf07:05:15

I wouldn't be upset if asked to come up with better names during code review. 😅 🙈

hiredman02:05:30

The vector makes the condition always truthy

pinkfrog02:05:08

updated the coee

dominic08:05:13

Good morning! (UK time anyway) Context: I’m doing some file watching to trigger a reload operation, but I want to wait until things have stabilised to avoid doing potentially dozens/hundreds of calls. I’ve got something working using futures which I cancel and replace with a new one each the function is called:

(def reload-op (atom nil))

(defn trigger-reload []
  (when-let [op @reload-op]
    (future-cancel op))
  (reset! reload-op
          (future
            (Thread/sleep 20)
            (println "reloading...")
            (reload))))
I’m just wondering if there’s a better / more idiomatic way of doing something like this? Are there potential problems/risks with this approach? Using Thread/sleep always makes me think there must be a better way!

dominic08:05:11

Also is this kind of question better suited to #beginners?

p-himik08:05:19

You need something that's often called a debounced function call. A few implementations that I saw used Executors/newSingleThreadScheduledExecutor.

dominic08:05:14

Thanks will look into that. Most debounce mechanisms ignore subsequent calls within a time window though, right? I’m looking to ignore previous calls within a specific window because I want to be sure I’m using the latest state.

hiredman08:05:04

This is a race condition

hiredman08:05:31

(if you are calling this from multiple threads)

Ben Sless08:05:54

It might be easier with channels, where one process handles reloads, another just debounces, and a last to handle submission

jcf08:05:14

If you want the most recent call to win, a sliding buffer with a core.async timeout would do the trick. You'll need to make sure you have a way to cleanup if you want this code to be reloadable too. I often see a sentinel value put onto the channel to handle shutting things down. https://stackoverflow.com/questions/35663415/throttle-functions-with-core-async

👀 1
dominic09:05:37

@U0NCTKEV8 race condition in terms of access to the atom?

p-himik09:05:37

> Most debounce mechanisms ignore subsequent calls within a time window though, right? I’m looking to ignore previous calls within a specific window because I want to be sure I’m using the latest state. I don't really remember looking at any debounce implementation that would ignore subsequent calls. In my memory, it's always "the latest call wins".

p-himik09:05:51

And yes, it's a race condition because you deref and reset the atom in two separate instructions. You can end up with 2+ calls to reload being scheduled.

p-himik09:05:46

That's why swap! exists. But at the same time, you can't have any side-effects in its argument function, so using an atom is not a good fit here.

dominic09:05:59

Yeah I think you’re right about debounce, think I got mixed up with throttling. In my particular hacky scenario the race condition isn’t a huge problem, but I’d prefer to do it right. I’ll have a proper look at core.async - thanks for all the feedback 👍

p-himik11:05:57

I think this is an alright implementation for a no-arg function:

(defn debounce [^Runnable f ^long delay-ms]
  (let [scheduler (Executors/newSingleThreadScheduledExecutor)
        task (atom nil)]
    (fn []
      (let [[old _] (reset-vals! task (.schedule scheduler f delay-ms TimeUnit/MILLISECONDS))]
        (when old
          (.cancel ^Future old true))))))

dominic11:05:28

ah nice, feels more similar to my original approach than going down the core.async route, except reset-vals! avoids the race-condition, right? (and obviously doing proper scheduling rather than making the thread sleep)

p-himik11:05:17

It should avoid the race condition, yes.

flowthing09:05:36

This threw me off:

λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}}}' -M -r
Clojure 1.10.3
user=> ((fn [{:keys [a]}] a) '({:a 1}))
Execution error (IllegalArgumentException) at user/eval1$fn (REPL:1).
No value supplied for key: {:a 1}

λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> ((fn [{:keys [a]}] a) '({:a 1}))
1
This must be a result of keyword argument functions accepting maps in 1.11.1, but I hadn't realized that this also means that you can now pass a list (but not a vector) of maps to a function that expects (and destructures) a single map and kinda have it work. I guess this is intentional, though?

jcf10:05:43

To my eyes, that is not intuitive at all. I'd very much expect the list to be the thing being destructured in both cases. If I want to apply a list of arguments to a function, I use apply. :thinking_face: Does that imply the code below behaves differently as of 1.11?.

(defn f [a b] [a (:k b)])
(defn g [a {:keys [k]] [a k])
I need to have a play with this when I'm back at my computer.

flowthing10:05:04

It depends on what you pass to g, but yes.

flowthing10:05:44

λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}}}' -M -r
Clojure 1.10.3
user=> (defn g [a {:keys [k]}] [a k])
#'user/g
user=> (g 1 {:k 2})
[1 2]
user=> (g 1 '({:k 2}))
Execution error (IllegalArgumentException) at user/g (REPL:1).
No value supplied for key: {:k 2}

flowthing10:05:56

λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> (defn g [a {:keys [k]}] [a k])
#'user/g
user=> (g 1 {:k 2})
[1 2]
user=> (g 1 '({:k 2}))
[1 2]
user=> (g 1 [{:k 2}])
[1 nil]
user=>

jcf11:05:43

Thanks for sharing those examples, @U4ZDX466T! 🙇

Alex Miller (Clojure team)11:05:16

It is intentional, yes. The trailing data is added as if by conj.

flowthing11:05:42

OK, thanks. The release notes make it clear that that is the case for functions that take keyword arguments, but I guess I was just surprised to find that it is true for functions that don't take keyword arguments, too.

flowthing07:05:31

I'd like to press on this a bit, if I may... this just seems quite counterintuitive to me:

λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> (defn handle-pokemon
  [{:keys [name move]}]
  (printf "Do something with %s and %s\n" name move))
#'user/handle-pokemon
user=> (def pikachu {:name "Pikachu" :type :electric :move "Thunder Shock"})
#'user/pikachu
user=> (handle-pokemon pikachu)
Do something with Pikachu and Thunder Shock
nil
user=> (handle-pokemon [pikachu])
Do something with null and null
nil
user=> (handle-pokemon (map #(select-keys % [:name :move]) [pikachu]))
Do something with Pikachu and Thunder Shock
nil
user=> (def charmander {:name "Charmander" :type :fire :move "Ember"})
#'user/charmander
user=> (handle-pokemon (map #(select-keys % [:name :move]) [pikachu charmander]))
Do something with null and null
nil
I guess I'm wondering whether I should now consider passing a seq of maps to a fn that expects and destructures one map undefined behavior? Or whether it's actually defined in some (undocumented?) way?

💯 1
Geekingfrog10:05:18

How do you profile things in clojure/jvm land? What tools and technique do you use? I'm especially interested in profiling local applications. I have something giving me OOM and I'm struggling a bit to figure that out.

jcf11:05:52

I’ve used VisualVM and YourKit in the past. • https://visualvm.github.io/https://www.yourkit.com/ YourKit worked better than VisualVM when I was troubleshooting a Datomic Peer years ago with help from one of the Datomic team.

☝️ 1
jcf11:05:34

There are some posts floating around related to GC tuning that might be worth a look too. Are you configuring the garbage collector at all?

Geekingfrog11:05:36

Not at all. Although I'm 95% sure it's simply a very inefficient way of doing things, since on smaller input it works fine.

p-himik11:05:49

This is a good starting point: http://clojure-goes-fast.com/

👀 1
4
dominic11:05:31

I’ve used https://github.com/ptaoussanis/tufte for profiling specific functions, not sure how relevant here

👀 1
jumar12:05:01

clj-async-profiler tends to be very good for cpu profiling and it can profile allocations too. But the very first thing would be: • Read the exception trace - what does it tell you? • How much memory do you really have? Is it enough for the normal mode of operation? • (if it's a problem) Is heap or non-heap memory a problem?

Ben Sless13:05:22

Start with visualvm Recognize key areas which cause issues in your application Profile them with clj-async-profiler

Ivar Refsdal13:05:53

If you are experiencing an OOM, I would: 1. Dump the heap to a file by adding something like this to your java arguments:

-XX:+ExitOnOutOfMemoryError
      -XX:+HeapDumpOnOutOfMemoryError
      -XX:HeapDumpPath=...
2. Open the heap dump in https://www.eclipse.org/mat/. It can show you heap used by thread, which again will give a stacktrace and possibly the culprit of the OOM if you are lucky 🙂 Edit: at least that strategy worked for me when we had an OOM.

💯 1
Geekingfrog17:05:45

clj-async-profiler is working nicely, I'm not going blind anymore thanks. Although adding the debug symbols to the jdk was a pain, but that's another story.

Nom Nom Mousse14:05:15

I want to load user-defined files in my Clojure application. The attached code works in the REPL, but if I try it in my app, I get the error:

Can't change/establish root binding of: *ns* with set
😕

Nom Nom Mousse14:05:52

Is there a way to evaluate files that are not in the class path during runtime? Or is there some way to add stuff to the class path safely during runtime? I could (slurp "/some/path/test.clj") and do (spit "somewhere on the class path") but how would that work when I make my program into a JAR? Can I spit into the classpath then?

dpsutton14:05:02

(load-file "any-arbitrary-path") should work

Nom Nom Mousse14:05:49

Jeez, I feel silly. At least I learnt much trying to implement it.

pinkfrog14:05:18

I see tools often use the -A:deps. But where does that from? I do not define the {:alias {:deps …}} personally.

clojure -A:deps -Tantq help/doc

Alex Miller (Clojure team)14:05:29

that one alias is built-in

Alex Miller (Clojure team)14:05:48

it's in the root deps.edn, which is a resource inside tools.deps lib

pinkfrog14:05:06

What’s the value?

Raimon Grau14:05:00

clojure -Sdescribe will tell you the config-files paths

thanks3 1
Alex Miller (Clojure team)14:05:50

what's the value of what?

Alex Miller (Clojure team)14:05:41

-Sdescribe actually will lie to you about the root deps.edn - it does list a root deps.edn but that is actually no longer used, it's there as legacy support for tools that might have previously used it

Alex Miller (Clojure team)14:05:35

the main intent behind the :deps alias is to add tools.deps to your classpath

Raimon Grau14:05:44

oh... thanks for explaining the -Sdescribe behavior! good to know

Nom Nom Mousse14:05:22

I want to read files with load-file. These files are valid Clojure with one addition, a function/macro called defrule:

;; some-path/rules.clj
(ns rules
  (:require [clojure.set :as set]))

(def whatevz set/difference)

(defrule bwa-map
  "Map DNA sequences against a reference genome with BWA."
  {:wildcards [:sample :genome]
   :externals  [:genome :fastq]
   :output    "bwa-map.bam"
   :resources {:cpu 1}
   :params    {:rg "@RG\\tID:{{wildcards.sample}}\\tSM:{{wildcards.sample}}"}
   :shell     "bwa mem -R '{{params.rg}}' -t {{resources.cpu}} {{externals.genome}} {{externals.fastq}} | samtools view -Sb - > {{output.0}}"})
load-file isn't going to be able to read that file (ERROR: Unable to resolve symbol: defrule in this context). But what is the simplest way of making it work? I could read some-path/rules.clj, add a (defn defrule [form] ...) as the second form in the file, spit it out to a temp-file and then use load-file on the modified file, but that feels hacky. I could also just require users to add (:require [blah :refer defrule]) in the ns declaration.

Alex Miller (Clojure team)14:05:02

you could slurp the file, manipulate it, and then load-string if you wanted, just depends on the world you want to build for people

👍 1
Alex Miller (Clojure team)14:05:48

having it be "almost Clojure" has the significant downside that it breaks all Clojure tooling that some users might be able to use

dpsutton14:05:05

another thought is that the ns form might be hurting more than helping. You could control the namespace (and thus the aliases in that namespace) from the calling location of load file perhaps?

🧠 1
Nom Nom Mousse14:05:55

I will consider both. But come to think of it from a different perspective, I want the files to be pure Clojure as that might be a stepping-stone into Clojure for the people in my research group (and other potential users).

Alex Miller (Clojure team)14:05:23

seems worth it to give people a little more boilerplate then

duckie 1
Nom Nom Mousse15:05:44

I'm trying to add metadata to a def in a macro:

(defmacro t [n] `(def ~n ^:rule ~{}))
(t hihi)
hihi
;; {}
(meta hihi)
;; nil
However:
(def hiyahiya ^:rule {})
{:rule true}
How can I add metadata in a macro?

Joshua Suskalo15:05:53

Metadata added with ^ is done at read-time, and when your macro is read in that ~{} gets read as (unquote {}) which then has the metadata on it, which means that the metadata never makes it into the resulting code. If you want to add metadata in resulting code, look at the function with-meta

👍 1
Joshua Suskalo15:05:01

for example:

(defmacro t
  [n]
  `(def ~n (with-meta ~{} {:rule true})))

🙏 1
Nom Nom Mousse15:05:25

How can I find all variables in a namespace with specific metadata?

(ns-publics 'rules)
=>
{samtools-sort #'rules/samtools-sort,
 samtools-index #'rules/samtools-index,
 whatevz #'rules/whatevz,
 bwa-map #'rules/bwa-map,
 idnt #'rules/idnt}
I know that three of these have the metadata :rule true, but how do I collect them?

p-himik15:05:28

Just filter by the values of meta?

Joshua Suskalo15:05:45

You can just grab vals on the map to get all the vars and then do a filter call, and since you have put the metadata on the value the var holds instead of on the var itself you'll have to throw a deref into your filter function.

Nom Nom Mousse15:05:51

(meta (deref #'rules/samtools-sort))

Joshua Suskalo15:05:18

yeah. Although if this is the purpose of the meta I'd recommend putting it onto the var instead of onto the value.

Nom Nom Mousse15:05:56

(for [kv (ns-publics 'rules)
      :let [v (deref (val kv))]
      :when (:rule (meta v))]
     v)

Nom Nom Mousse15:05:26

> putting it onto the var Wonderful. Had no idea I could do that 😄

Nom Nom Mousse15:05:26

I can't use with-meta on the variable name in my macro it seems. Remember this: https://clojurians.slack.com/archives/C03S1KBA2/p1652368484896899

Joshua Suskalo15:05:28

you can also just call vals to get all the values without this whole list comprehension

Joshua Suskalo15:05:56

I mean I guess using for this way is fine, but I'll admit it strikes me as a little odd.

Joshua Suskalo15:05:20

Yeah, so you don't use with-meta on the variable name, you use it on the symbol you pass in

🧠 1
Joshua Suskalo15:05:40

(defmacro t [n] `(def ~(with-meta n {:rule true}) ~{}))

Joshua Suskalo15:05:55

notice the with-meta call is inside the unquote.

Nom Nom Mousse15:05:36

I appreciate your help but it doesn't work for me :thinking_face:

(defmacro t [n] `(def ~(with-meta n {:rule true}) ~{}))
(t a)
(meta a)
;; nil

Joshua Suskalo15:05:49

yeah the meta isn't on a

Nom Nom Mousse15:05:53

On the pointer, not on what is pointed at, I guess.