This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-12
Channels
- # announcements (4)
- # asami (3)
- # babashka (18)
- # babashka-sci-dev (2)
- # beginners (55)
- # calva (18)
- # clj-commons (23)
- # clj-kondo (2)
- # cljfx (7)
- # cljs-dev (15)
- # clojure (96)
- # clojure-europe (43)
- # clojure-losangeles (3)
- # clojure-nl (2)
- # clojure-uk (11)
- # clojurescript (1)
- # datalevin (2)
- # datomic (10)
- # events (1)
- # google-cloud (4)
- # gratitude (16)
- # helix (3)
- # hyperfiddle (63)
- # inf-clojure (13)
- # introduce-yourself (4)
- # ipfs (2)
- # jobs (2)
- # jobs-discuss (12)
- # leiningen (7)
- # lsp (15)
- # off-topic (38)
- # polylith (24)
- # portal (27)
- # remote-jobs (8)
- # sci (27)
- # spacemacs (5)
- # specter (1)
- # sql (5)
- # xtdb (41)
Possible to make this code more concise? Overall, I’d like to maintain the (when-some) overall structure.
(when-some [fname (some-> (some-func-that-returns-a-fname)
(as-> fname (format "../base/path/%s" fname))
(as-> fname (when (.exists (io/file fname)) fname)))]
(perform-some-operations fname))
The fact that you need to call exists and then conditional on that return the name kills your ability to thread
If you think of code as a graph where names are edges in the graph (a data flow graph) what makes for good threading is code where that graph is just a straight line with a single edge between each node
Yes. When sometimes the arg position have to be changed due to the different invocation functions, I would go to (as->) . But I am not sure how to make the code more clean.
I have had former coworkers say they can tell which code I wrote because of all the threading I used. You don't need to thread everything
some->> and as-> does not play well with the other.
If I use some->>, I guess I have to write a function to handle (.exists (io/file fname)) fname)
> You also don’t need the outer when-some the final form is a syntax-quoted macro form, and seems have conflict with threading form
The main thing that jumps out at me is the mixture of pure operations with IO-based.
I'd break this into two steps. The first that produces a java.io.File
(regardless of whether anything exists; just pure in-memory operations), and then I'd check if the file exists and conditionally perform the operation.
The benefits are you can test the pure stuff more easily, you can reuse the named file generation logic elsewhere, and the IO (potentially) becomes more isolated.
(defn filename+base-path
^File [s]
(io/file "../base/path" s)
(let [file (filename+base-path)]
(when (.exists file)
(perform-io file)))
(Coding on my phone; apologies!)Good morning! (UK time anyway) Context: I’m doing some file watching to trigger a reload operation, but I want to wait until things have stabilised to avoid doing potentially dozens/hundreds of calls. I’ve got something working using futures which I cancel and replace with a new one each the function is called:
(def reload-op (atom nil))
(defn trigger-reload []
(when-let [op @reload-op]
(future-cancel op))
(reset! reload-op
(future
(Thread/sleep 20)
(println "reloading...")
(reload))))
I’m just wondering if there’s a better / more idiomatic way of doing something like this? Are there potential problems/risks with this approach? Using Thread/sleep
always makes me think there must be a better way!You need something that's often called a debounced function call.
A few implementations that I saw used Executors/newSingleThreadScheduledExecutor
.
Thanks will look into that. Most debounce mechanisms ignore subsequent calls within a time window though, right? I’m looking to ignore previous calls within a specific window because I want to be sure I’m using the latest state.
It might be easier with channels, where one process handles reloads, another just debounces, and a last to handle submission
If you want the most recent call to win, a sliding buffer with a core.async timeout would do the trick. You'll need to make sure you have a way to cleanup if you want this code to be reloadable too. I often see a sentinel value put onto the channel to handle shutting things down. https://stackoverflow.com/questions/35663415/throttle-functions-with-core-async
@U0NCTKEV8 race condition in terms of access to the atom?
> Most debounce mechanisms ignore subsequent calls within a time window though, right? I’m looking to ignore previous calls within a specific window because I want to be sure I’m using the latest state. I don't really remember looking at any debounce implementation that would ignore subsequent calls. In my memory, it's always "the latest call wins".
And yes, it's a race condition because you deref and reset the atom in two separate instructions. You can end up with 2+ calls to reload
being scheduled.
That's why swap!
exists. But at the same time, you can't have any side-effects in its argument function, so using an atom is not a good fit here.
Yeah I think you’re right about debounce, think I got mixed up with throttling. In my particular hacky scenario the race condition isn’t a huge problem, but I’d prefer to do it right. I’ll have a proper look at core.async - thanks for all the feedback 👍
I think this is an alright implementation for a no-arg function:
(defn debounce [^Runnable f ^long delay-ms]
(let [scheduler (Executors/newSingleThreadScheduledExecutor)
task (atom nil)]
(fn []
(let [[old _] (reset-vals! task (.schedule scheduler f delay-ms TimeUnit/MILLISECONDS))]
(when old
(.cancel ^Future old true))))))
ah nice, feels more similar to my original approach than going down the core.async route, except reset-vals!
avoids the race-condition, right? (and obviously doing proper scheduling rather than making the thread sleep)
This threw me off:
λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}}}' -M -r
Clojure 1.10.3
user=> ((fn [{:keys [a]}] a) '({:a 1}))
Execution error (IllegalArgumentException) at user/eval1$fn (REPL:1).
No value supplied for key: {:a 1}
λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> ((fn [{:keys [a]}] a) '({:a 1}))
1
This must be a result of keyword argument functions accepting maps in 1.11.1, but I hadn't realized that this also means that you can now pass a list (but not a vector) of maps to a function that expects (and destructures) a single map and kinda have it work. I guess this is intentional, though?To my eyes, that is not intuitive at all. I'd very much expect the list to be the thing being destructured in both cases. If I want to apply a list of arguments to a function, I use apply
. :thinking_face:
Does that imply the code below behaves differently as of 1.11?.
(defn f [a b] [a (:k b)])
(defn g [a {:keys [k]] [a k])
I need to have a play with this when I'm back at my computer.λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}}}' -M -r
Clojure 1.10.3
user=> (defn g [a {:keys [k]}] [a k])
#'user/g
user=> (g 1 {:k 2})
[1 2]
user=> (g 1 '({:k 2}))
Execution error (IllegalArgumentException) at user/g (REPL:1).
No value supplied for key: {:k 2}
λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> (defn g [a {:keys [k]}] [a k])
#'user/g
user=> (g 1 {:k 2})
[1 2]
user=> (g 1 '({:k 2}))
[1 2]
user=> (g 1 [{:k 2}])
[1 nil]
user=>
Thanks for sharing those examples, @U4ZDX466T! 🙇
It is intentional, yes. The trailing data is added as if by conj.
OK, thanks. The release notes make it clear that that is the case for functions that take keyword arguments, but I guess I was just surprised to find that it is true for functions that don't take keyword arguments, too.
I'd like to press on this a bit, if I may... this just seems quite counterintuitive to me:
λ clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.1"}}}' -M -r
Clojure 1.11.1
user=> (defn handle-pokemon
[{:keys [name move]}]
(printf "Do something with %s and %s\n" name move))
#'user/handle-pokemon
user=> (def pikachu {:name "Pikachu" :type :electric :move "Thunder Shock"})
#'user/pikachu
user=> (handle-pokemon pikachu)
Do something with Pikachu and Thunder Shock
nil
user=> (handle-pokemon [pikachu])
Do something with null and null
nil
user=> (handle-pokemon (map #(select-keys % [:name :move]) [pikachu]))
Do something with Pikachu and Thunder Shock
nil
user=> (def charmander {:name "Charmander" :type :fire :move "Ember"})
#'user/charmander
user=> (handle-pokemon (map #(select-keys % [:name :move]) [pikachu charmander]))
Do something with null and null
nil
I guess I'm wondering whether I should now consider passing a seq of maps to a fn that expects and destructures one map undefined behavior? Or whether it's actually defined in some (undocumented?) way?How do you profile things in clojure/jvm land? What tools and technique do you use? I'm especially interested in profiling local applications. I have something giving me OOM and I'm struggling a bit to figure that out.
I’ve used VisualVM and YourKit in the past. • https://visualvm.github.io/ • https://www.yourkit.com/ YourKit worked better than VisualVM when I was troubleshooting a Datomic Peer years ago with help from one of the Datomic team.
There are some posts floating around related to GC tuning that might be worth a look too. Are you configuring the garbage collector at all?
Not at all. Although I'm 95% sure it's simply a very inefficient way of doing things, since on smaller input it works fine.
I’ve used https://github.com/ptaoussanis/tufte for profiling specific functions, not sure how relevant here
clj-async-profiler tends to be very good for cpu profiling and it can profile allocations too. But the very first thing would be: • Read the exception trace - what does it tell you? • How much memory do you really have? Is it enough for the normal mode of operation? • (if it's a problem) Is heap or non-heap memory a problem?
Start with visualvm Recognize key areas which cause issues in your application Profile them with clj-async-profiler
If you are experiencing an OOM, I would:
1. Dump the heap to a file by adding something like this to your java
arguments:
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=...
2. Open the heap dump in https://www.eclipse.org/mat/.
It can show you heap used by thread, which again will give a stacktrace and possibly the culprit of the OOM if you are lucky 🙂
Edit: at least that strategy worked for me when we had an OOM.clj-async-profiler is working nicely, I'm not going blind anymore thanks. Although adding the debug symbols to the jdk was a pain, but that's another story.
I want to load user-defined files in my Clojure application. The attached code works in the REPL, but if I try it in my app, I get the error:
Can't change/establish root binding of: *ns* with set
😕Is there a way to evaluate files that are not in the class path during runtime?
Or is there some way to add stuff to the class path safely during runtime? I could (slurp "/some/path/test.clj")
and do (spit "somewhere on the class path")
but how would that work when I make my program into a JAR? Can I spit into the classpath then?
Jeez, I feel silly. At least I learnt much trying to implement it.
I see tools often use the -A:deps
. But where does that from? I do not define the {:alias {:deps …}} personally.
clojure -A:deps -Tantq help/doc
that one alias is built-in
it's in the root deps.edn, which is a resource inside tools.deps lib
what's the value of what?
-Sdescribe actually will lie to you about the root deps.edn - it does list a root deps.edn but that is actually no longer used, it's there as legacy support for tools that might have previously used it
the main intent behind the :deps alias is to add tools.deps to your classpath
oh... thanks for explaining the -Sdescribe behavior! good to know
I want to read files with load-file. These files are valid Clojure with one addition, a function/macro called defrule
:
;; some-path/rules.clj
(ns rules
(:require [clojure.set :as set]))
(def whatevz set/difference)
(defrule bwa-map
"Map DNA sequences against a reference genome with BWA."
{:wildcards [:sample :genome]
:externals [:genome :fastq]
:output "bwa-map.bam"
:resources {:cpu 1}
:params {:rg "@RG\\tID:{{wildcards.sample}}\\tSM:{{wildcards.sample}}"}
:shell "bwa mem -R '{{params.rg}}' -t {{resources.cpu}} {{externals.genome}} {{externals.fastq}} | samtools view -Sb - > {{output.0}}"})
load-file isn't going to be able to read that file (ERROR: Unable to resolve symbol: defrule in this context).
But what is the simplest way of making it work?
I could read some-path/rules.clj
, add a (defn defrule [form] ...)
as the second form in the file, spit it out to a temp-file and then use load-file on the modified file, but that feels hacky.
I could also just require users to add (:require [blah :refer defrule])
in the ns
declaration.you could slurp the file, manipulate it, and then load-string if you wanted, just depends on the world you want to build for people
having it be "almost Clojure" has the significant downside that it breaks all Clojure tooling that some users might be able to use
another thought is that the ns
form might be hurting more than helping. You could control the namespace (and thus the aliases in that namespace) from the calling location of load file perhaps?
I will consider both. But come to think of it from a different perspective, I want the files to be pure Clojure as that might be a stepping-stone into Clojure for the people in my research group (and other potential users).
I'm trying to add metadata to a def in a macro:
(defmacro t [n] `(def ~n ^:rule ~{}))
(t hihi)
hihi
;; {}
(meta hihi)
;; nil
However:
(def hiyahiya ^:rule {})
{:rule true}
How can I add metadata in a macro?Metadata added with ^
is done at read-time, and when your macro is read in that ~{}
gets read as (unquote {})
which then has the metadata on it, which means that the metadata never makes it into the resulting code.
If you want to add metadata in resulting code, look at the function with-meta
How can I find all variables in a namespace with specific metadata?
(ns-publics 'rules)
=>
{samtools-sort #'rules/samtools-sort,
samtools-index #'rules/samtools-index,
whatevz #'rules/whatevz,
bwa-map #'rules/bwa-map,
idnt #'rules/idnt}
I know that three of these have the metadata :rule true
, but how do I collect them?You can just grab vals
on the map to get all the vars and then do a filter
call, and since you have put the metadata on the value the var holds instead of on the var itself you'll have to throw a deref
into your filter function.
(meta (deref #'rules/samtools-sort))
yeah. Although if this is the purpose of the meta I'd recommend putting it onto the var instead of onto the value.
(for [kv (ns-publics 'rules)
:let [v (deref (val kv))]
:when (:rule (meta v))]
v)
> putting it onto the var Wonderful. Had no idea I could do that 😄
I can't use with-meta on the variable name in my macro it seems. Remember this: https://clojurians.slack.com/archives/C03S1KBA2/p1652368484896899
you can also just call vals
to get all the values without this whole list comprehension
I mean I guess using for
this way is fine, but I'll admit it strikes me as a little odd.
Yeah, so you don't use with-meta on the variable name, you use it on the symbol you pass in
(defmacro t [n] `(def ~(with-meta n {:rule true}) ~{}))
notice the with-meta
call is inside the unquote.
I appreciate your help but it doesn't work for me :thinking_face:
(defmacro t [n] `(def ~(with-meta n {:rule true}) ~{}))
(t a)
(meta a)
;; nil
yeah the meta isn't on a
it's on #'a
On the pointer, not on what is pointed at, I guess.
@U0232JK38BZ Have you seen https://github.com/agentbellnorm/dativity library?
Nope 😄