This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-07-19
Channels
- # announcements (1)
- # babashka (26)
- # beginners (42)
- # calva (3)
- # cider (5)
- # cljs-dev (7)
- # cljsjs (1)
- # cljsrn (13)
- # clojure (95)
- # clojure-europe (12)
- # clojure-germany (3)
- # clojure-italy (12)
- # clojure-nl (1)
- # clojure-spec (16)
- # clojure-uk (25)
- # clojurescript (13)
- # clojureverse-ops (5)
- # community-development (1)
- # conjure (8)
- # cryogen (3)
- # datomic (5)
- # deps-new (7)
- # exercism (1)
- # fulcro (34)
- # honeysql (9)
- # hyperfiddle (3)
- # instaparse (2)
- # introduce-yourself (2)
- # jobs (6)
- # kaocha (1)
- # lambdaisland (1)
- # lsp (47)
- # malli (3)
- # membrane (38)
- # off-topic (9)
- # polylith (36)
- # reitit (7)
- # releases (1)
- # remote-jobs (2)
- # shadow-cljs (11)
- # spacemacs (7)
- # sql (7)
- # tools-deps (19)
Hi everyone, could I get a code review on my first stateful transducer, please? I have a situation where I’m processing a reducible series of elements, and if none are processed from the main series I’d like to have the processor process the elements from the provided backup elements:
(defn with-backup [other]
(fn [rf]
(let [seen? (volatile! false)]
(fn
([] (rf))
([result]
(if @seen?
(rf result)
(rf (reduce rf result other))))
([result input]
(vreset! seen? true)
(rf result input))))))
In particular, I’m not sure if I should be doing any special handling of reduced
here, or if the (rf (reduce rf result other))
is funky in some way I’m not aware of. It seems to work correctly from my initial testing.Here are some of my initial tests:
(into [] (with-backup [3 4 5]) [1 2 3])
=> [1 2 3]
(into [] (with-backup [3 4 5]) [])
=> [3 4 5]
(into [] (with-backup [3 4 5]) nil)
=> [3 4 5]
(into []
(comp (with-backup [3 4 5])
(map inc))
[1 2 3])
=> [2 3 4]
(into []
(comp (with-backup [3 4 5])
(map inc))
[])
=> [4 5 6]
(into []
(comp
(filter (constantly false))
(with-backup [3 4 5])
(map inc))
[1 2 3])
=> [4 5 6]
the completion arity short circuits calling the remaining completion arities and instead restarts the reducing arity.
try your with-backup
wrapping a reducing function with a completion arity that you need to be called and you'll see this bad behavior
oh i see you are doing that. but i think you are making an assumption that rf
needs to return a collection and that's not true in general but could be in your subset of uses
I would also reference some other transducers that "flush" on completion like partition-all
:
(defn partition-all
"Returns a lazy sequence of lists like partition, but may include
partitions with fewer than n items at the end. Returns a stateful
transducer when no collection is provided."
{:added "1.2"
:static true}
([^long n]
(fn [rf]
(let [a (java.util.ArrayList. n)]
(fn
([] (rf))
([result]
(let [result (if (.isEmpty a)
result
(let [v (vec (.toArray a))]
;;clear first!
(.clear a)
(unreduced (rf result v))))]
(rf result)))
([result input]
(.add a input)
(if (= n (.size a))
(let [v (vec (.toArray a))]
(.clear a)
(rf result v))
result))))))
notice how the remaining item is flushed and the completion arity is called edit: oops. nvmd. looks like you are calling the completion arity too
it also "feels" wrong to be feeding values from a transducers. is there a reason the process can't be fed the backups directly. eg. something like:
(into []
(if-let [xs (seq [1 2 3])]
xs
[3 4 5]))
(into []
(if-let [xs (seq [])]
xs
[3 4 5]))
Yes, I think I should be calling the completion arity correctly according to the docs, but I’m not 100% sure on that.
> i think you are making an assumption that rf needs to return a collection and that’s not true in general but could be in your subset of uses I’m not sure what you mean by that, could you elaborate?
> is there a reason the process can’t be fed the backups directly Only that that requires the initial collection to be realised as a seq, and the test to be in all the places I would use this transducer (unless I’m misunderstanding what you’re saying here)
doesn't the initial have to be realised either way?
I don’t think so - I could pass an eduction as both the original series and the backup series, so they’d both only be used “on demand”. In this case, I’m migrating some old code that built up a series of collections and then concat’ed them, and I’m ending up with a bunch of layered eductions using cat
to replace them. With the transducer version I can just return the whole lot and when the final processing is done is when the eduction would be invoked.
I was confused about this point recently. But the rf doesn’t necessarily return something. It could be shoving things in a database or a channel or whatever. The contract doesn’t require a tangible collection as the return
Right, in my case I’m feeding elements to a stateful processor which shows the completions in Cursive, so I’m actually not building up a collection, my rf is feeding elements to that instead.
But I think that the form above should support that, unless I’m misunderstanding something.
looks good to me
Calling the conpletion arity on the non completion arity is where you mess up right?
it seems to match how partition-all
handles flushing. The only improvement I see is maybe calling unreduced
before calling the completion arity
oh, nvmd, reduce
will take care of that for you anyway
> Calling the conpletion arity on the non completion arity is where you mess up right?
I don’t think I do that, do I? The only place I (rf <something>)
is called is within each of the two branches of the if
within the completion arity of my fn.
In which case, I think I’m confused… does this look ok, or do we think there’s a problem for the non-collection case?
I think it's right
I think cgrand documented a lot of the considerations for custom xforms in https://www.youtube.com/watch?v=XiCwN-fv7os from his experience building xforms library.
Anyone tried https://openjdk.java.net/jeps/350 to speed up clojure start time?
I have. It works, but is also very cumbersome to use.
or rather, I've tried the older version, not the newer dynamic one, which may make it less cumbersome
yeah I tried it too. very cumbersome and very very slow to build. so you'll trade one really slow startup for a few slightly faster startups. in the end not worth it IMHO
you also have to re-do it every time the classpath changes I think (at least you had to when I tested)
Ya, it's so unergonomic to use. The last JEP mention making another JEP in the future to just automate it so it re-caches automatically when needed. But nothing was done yet.
I think the potential benefit of the new dynamic stuff is that you could create a static base archive for just Clojure itself, and then make everything else dynamic which would improve part of the startup time
It seems with JDK 12 they do that for the JDK itself. But I don't think this is the kind of thing you can bundle in a Jar or anything if you wanted to do the same for Clojure.
while debugging I often want make assertions about computed values which aren't started in variables. Is there an idiom for doing this. E.g., I'd like to assert that reduce
doesn't return nil
. Somewhere in my code a nil
is getting inserted in a computation and I havent yet figured out where.
[id (map->State
{:index id
:initial (= 0 id)
:pattern (reduce (:combine-labels dfa) (map :pattern (partitions-map id)))
:accepting (member id new-fids)
:transitions new-transitions})]
Ordinarily I'd wrap the reduce in a let
binding the return value of let to a variable xyzzy
and make assertions about that variable, than have the let just return the value of xyzzy
. But perhaps there is a better, less invasive idiom?can I wrap some sort of spec incantation around an expression without effecting control flow nor return value?
(defmacro expect-not-nil [form] `(let [v# form] (if v# v# (throw (Exception. (str "unexpected nil found when evaluating:" (quote form))))))
https://clojure.org/guides/spec#_instrumentation_and_testing I think you can instrument functions with their specs
@kristof, yes that's basically what I'm doing without the macro. Perhaps I need to write my own macro (not difficult), just wondering if I'm reinventing the wheel. In general something like expect which takes a value to return and a predicate to assert is true.
@U010VP3UY9X There's one in spec https://clojuredocs.org/clojure.spec.alpha/assert
Ben, are you suggesting that I instrument the reduce
function? that sounds dangerous. I certainly don't want to interfere with an internal use of reduce which really needs to return nil.
If you don't want to pull in that whole dep, it's much better to write a little macro. And remember that macros are not functions so you'd have to wrap it with a #(...) function literal if you're going to pass that into reduce
If you're calling reduce in your function it's more complicated, but if you pass its result as an argument to a function you wrote, you can certainly spec that argument to require it cannot be nil
personally I think it's a mistake that assert returns nil when it passes. there should have something like (assert expr predicate-fn optional-error-msg) which returns the result if predicate-fn is not ni.
Sounds more like you want to instrument a function that calls reduce or maybe the function you pass into reduce.
returning nil is just an example. the most general case has nothing to do with nil. I just want to pinpoint here in my program some condition is becoming true or is failing where it shouldn't
Maybe something with data readers?
(require '[clojure.spec.alpha :as s])
(s/def ::numero number?)
(s/valid? ::numero 3)
(defn speco
[spec form]
`(let [ret# (do ~@form)]
(assert (s/valid? spec ret#))
ret#))
(binding [*data-readers* {'spec (fn [form] (let [s (:spec (meta form))] (speco spec form)))}]
(+ 1 #spec ^{:spec ::numero} (+ 2 3)))
ben, I think that's far too complex for what I need. I've written a simple debugging function assert-debug
which takes a value and a side-effecting-function.
(defn assert-debug "call the assertion function as side effect, and return the given value."
[expr assertion]
(assertion expr)
expr)
the side effecting assertion
function is responsible for making assertions which appropriate error messages.the assertion
function can use spec
if it needs to, or just call assert
related discussion on HN (about CL nil
): https://news.ycombinator.com/item?id=27872280
My take on nil
is that it is pragmatic but also sometimes annoying like in your case because it is overloaded and lacks context in of itself. Some Clojure code return a keyword instead to signify meaning like :clojure.spec.alpha/invalid
@U010VP3UY9X https://gist.github.com/krstoff/7ccdf387bc50caeb071e9d1b18f03024#file-check-macro-clj
(check (not true)) (check (+ 1 2) (fn [i] (= i 3))) (check (+ 1 2) (partial = 3) (partial odd?) (partial even?))
yes in my case, there wasn't supposed to be a nil. It was coming from somewhere and I was just trying to find its origin. In the end it came from calling :pattern
on a map which didn't contain such a pattern. I finally found it.
@kristof, I decided to allow the caller to make the assertion, thus the caller can decide whether to provide a 2nd argument to assert if he wants to.
Does anyone else have the problem that clojure.test
obscures lines numbers in stack traces?
I have code like the following:
(defn all-states-have-patterns [dfa]
(doseq [q (states-as-seq dfa)]
(assert (not (nil? (:pattern q)))
(cl-format false "dfa=~A contains state ~A with nil pattern"
dfa q))))
(deftest t-test-1
(testing "particular case 1 which was failing"
(let [dfa-1 (assert-debug (rte-to-dfa '(:+ (:cat String (:? Long)))
1) all-states-have-patterns)
dfa-2 (assert-debug (rte-to-dfa '(:cat (:* String) Long)
2) all-states-have-patterns)
dfa-sxp (assert-debug (synchronized-product dfa-1 dfa-2
(fn [a b]
(and a b))
(fn [q1 _q2]
((:exit-map dfa-1) (:index q1)))) all-states-have-patterns)
....
But I get a stack trace like this: as if coming from the REPL.
1. Unhandled java.lang.AssertionError
Assert failed: dfa=#<Dfa 7 states> contains state #<State 4> with
nil pattern (not (nil? (:pattern q)))
xymbolyco_test.clj: 236 clojure-rte.xymbolyco-test/all-states-have-patterns
xymbolyco_test.clj: 234 clojure-rte.xymbolyco-test/all-states-have-patterns
REPL: 552 clojure-rte.util/assert-debug
REPL: 550 clojure-rte.util/assert-debug
REPL: 246 clojure-rte.xymbolyco-test/fn/fn
REPL: 241 clojure-rte.xymbolyco-test/fn
REPL: 240 clojure-rte.xymbolyco-test/fn
test.clj: 203 cider.nrepl.middleware.test/test-var/fn
test.clj: 203 cider.nrepl.middleware.test/test-var
test.clj: 195 cider.nrepl.middleware.test/test-var
test.clj: 218 cider.nrepl.middleware.test/test-vars/fn/fn
test.clj: 687 clojure.test/default-fixture
test.clj: 683 clojure.test/default-fixture
test.clj: 218 cider.nrepl.middleware.test/test-vars/fn
test.clj: 687 clojure.test/default-fixture
Is this just an artifact of using cider?Oh, I think I see the problem. In Cider cnt-c c
unfortunately does not record the file name and line number with the function being defined.
not a problem of clojure.test
at all, rather a problem with cider.
You'd have the same problem if you typed the defn
in at the repl - there is no source file for it to point to. I tend to use cider-load-buffer
(which is bound to C-c C-k
by default, but I have it running on a save hook) which will eval the whole buffer including the line numbers ...
I'm considering using [chime](https://github.com/jarohen/chime) for scheduling jobs, but I also need to read from cron-like syntax to generate a periodic-seq
. Any ideas on how to approach this? It'd be great if there was a lib for that
there are surely job schedulers that use cron syntax already?
any such libraries need some scaffolding to turn into servers (seems like they presume being in-process)
right, most of the time if you want periodic job execution, you want state that persists across restarts / crashes of the vm, and that's where the real complexity would come in
in that case why not (future (loop [] (Thread/sleep N) (f) (recur))
- hard to get lighter weight than that in clojure world
http://clojurequartz.info is worth considering and it also can be setup with cron expressions
@smith.adriane a good lib and also pretty much the opposite of light weight
Is that a requirement?
it's something the OP mentioned at the start
I don’t see it? https://clojurians.slack.com/archives/C03S1KBA2/p1626718776210700
I didn't mention it, yes, but it's desirable to be lightweight
The requirements rn are quite loose to say the least, and I'm supposed to be doing an MVP whatever that means
we have a "cron" service at nubank that services triggered crons by either calling an HTTP endpoint or dropping a message on a Kafka topic
would be nice to have something "cloud native" that is lighter weight than quartz but more full-featured than CloudWatch Events
like for repeatable tasks that you can monitor externally (cloudwatch alarms if you dont get logs) and restart trivially (reboot a machine) thats good nuff
so I wouldn't knock just
(defmacro restart-on-failure
[& body]
"Performs the given task, but retries if an exception is raised"
`(let [task# (fn []
(try
{:result (do ~@body) :ok true}
(catch Exception e#
(log/info (keyword (str *ns*) "restarting-forever-loop") true
(keyword (str *ns*) "error") e#)
{:ok false})))]
(loop []
(let [res# (task#)]
(if (:ok res#)
(:result res#)
(recur))))))
Someone here lately told me about java TimerTask and it works nicely without any additional dependencies (as long as cron exprs are not needed)
There is also https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledExecutorService.html that provides a nicer API because you can schedule execution of Clojure functions (because they are Runnables).