Fork me on GitHub

Hi everyone, could I get a code review on my first stateful transducer, please? I have a situation where I’m processing a reducible series of elements, and if none are processed from the main series I’d like to have the processor process the elements from the provided backup elements:

(defn with-backup [other]
  (fn [rf]
    (let [seen? (volatile! false)]
        ([] (rf))
         (if @seen?
           (rf result)
           (rf (reduce rf result other))))
        ([result input]
         (vreset! seen? true)
         (rf result input))))))
In particular, I’m not sure if I should be doing any special handling of reduced here, or if the (rf (reduce rf result other)) is funky in some way I’m not aware of. It seems to work correctly from my initial testing.


Here are some of my initial tests:

(into [] (with-backup [3 4 5]) [1 2 3])
=> [1 2 3]
(into [] (with-backup [3 4 5]) [])
=> [3 4 5]
(into [] (with-backup [3 4 5]) nil)
=> [3 4 5]
(into []
      (comp (with-backup [3 4 5])
            (map inc))
      [1 2 3])
=> [2 3 4]
(into []
      (comp (with-backup [3 4 5])
            (map inc))
=> [4 5 6]
(into []
        (filter (constantly false))
        (with-backup [3 4 5])
        (map inc))
      [1 2 3])
=> [4 5 6]


mixing arities like that will make you very angry in the future


the completion arity short circuits calling the remaining completion arities and instead restarts the reducing arity.


try your with-backup wrapping a reducing function with a completion arity that you need to be called and you'll see this bad behavior


oh i see you are doing that. but i think you are making an assumption that rf needs to return a collection and that's not true in general but could be in your subset of uses


I would also reference some other transducers that "flush" on completion like partition-all:

(defn partition-all
  "Returns a lazy sequence of lists like partition, but may include
  partitions with fewer than n items at the end.  Returns a stateful
  transducer when no collection is provided."
  {:added "1.2"
   :static true}
  ([^long n]
   (fn [rf]
     (let [a (java.util.ArrayList. n)]
         ([] (rf))
            (let [result (if (.isEmpty a)
                           (let [v (vec (.toArray a))]
                             ;;clear first!
                             (.clear a)
                             (unreduced (rf result v))))]
              (rf result)))
         ([result input]
            (.add a input)
            (if (= n (.size a))
              (let [v (vec (.toArray a))]
                (.clear a)
                (rf result v))


notice how the remaining item is flushed and the completion arity is called edit: oops. nvmd. looks like you are calling the completion arity too


it also "feels" wrong to be feeding values from a transducers. is there a reason the process can't be fed the backups directly. eg. something like:

(into []
      (if-let [xs (seq [1 2 3])]
        [3 4 5]))

(into []
      (if-let [xs (seq [])]
        [3 4 5]))


Yes, I think I should be calling the completion arity correctly according to the docs, but I’m not 100% sure on that.


> i think you are making an assumption that rf needs to return a collection and that’s not true in general but could be in your subset of uses I’m not sure what you mean by that, could you elaborate?


> is there a reason the process can’t be fed the backups directly Only that that requires the initial collection to be realised as a seq, and the test to be in all the places I would use this transducer (unless I’m misunderstanding what you’re saying here)


doesn't the initial have to be realised either way?


I don’t think so - I could pass an eduction as both the original series and the backup series, so they’d both only be used “on demand”. In this case, I’m migrating some old code that built up a series of collections and then concat’ed them, and I’m ending up with a bunch of layered eductions using cat to replace them. With the transducer version I can just return the whole lot and when the final processing is done is when the eduction would be invoked.

👍 4

I was confused about this point recently. But the rf doesn’t necessarily return something. It could be shoving things in a database or a channel or whatever. The contract doesn’t require a tangible collection as the return


Hired man was the one who helped me understand that one


Right, in my case I’m feeding elements to a stateful processor which shows the completions in Cursive, so I’m actually not building up a collection, my rf is feeding elements to that instead.


But I think that the form above should support that, unless I’m misunderstanding something.


looks good to me


Calling the conpletion arity on the non completion arity is where you mess up right?


Or I might be totally misleading you and sorry if so


it seems to match how partition-all handles flushing. The only improvement I see is maybe calling unreduced before calling the completion arity


oh, nvmd, reduce will take care of that for you anyway


> Calling the conpletion arity on the non completion arity is where you mess up right? I don’t think I do that, do I? The only place I (rf <something>) is called is within each of the two branches of the if within the completion arity of my fn.


You’re right. And that’s exactly how transduce does it.


In which case, I think I’m confused… does this look ok, or do we think there’s a problem for the non-collection case?


I think it's right


I think cgrand documented a lot of the considerations for custom xforms in from his experience building xforms library.


Thanks for the help, everyone!


Anyone tried to speed up clojure start time?

Alex Miller (Clojure team)05:07:14

I have. It works, but is also very cumbersome to use.

Alex Miller (Clojure team)05:07:39

or rather, I've tried the older version, not the newer dynamic one, which may make it less cumbersome


yeah I tried it too. very cumbersome and very very slow to build. so you'll trade one really slow startup for a few slightly faster startups. in the end not worth it IMHO


you also have to re-do it every time the classpath changes I think (at least you had to when I tested)


Ya, it's so unergonomic to use. The last JEP mention making another JEP in the future to just automate it so it re-caches automatically when needed. But nothing was done yet.

Alex Miller (Clojure team)05:07:41

I think the potential benefit of the new dynamic stuff is that you could create a static base archive for just Clojure itself, and then make everything else dynamic which would improve part of the startup time


It seems with JDK 12 they do that for the JDK itself. But I don't think this is the kind of thing you can bundle in a Jar or anything if you wanted to do the same for Clojure.

Jim Newton08:07:05

while debugging I often want make assertions about computed values which aren't started in variables. Is there an idiom for doing this. E.g., I'd like to assert that reduce doesn't return nil. Somewhere in my code a nil is getting inserted in a computation and I havent yet figured out where.

[id (map->State
      {:index id
       :initial (= 0 id)
       :pattern (reduce (:combine-labels dfa) (map :pattern (partitions-map id)))
       :accepting (member id new-fids)
       :transitions new-transitions})]
Ordinarily I'd wrap the reduce in a let binding the return value of let to a variable xyzzy and make assertions about that variable, than have the let just return the value of xyzzy. But perhaps there is a better, less invasive idiom?

Ben Sless08:07:36

Sounds like spec can do it, no?

Jim Newton08:07:52

can I wrap some sort of spec incantation around an expression without effecting control flow nor return value?


(defmacro expect-not-nil [form] `(let [v# form] (if v# v# (throw (Exception. (str "unexpected nil found when evaluating:" (quote form))))))

Ben Sless08:07:41 I think you can instrument functions with their specs

Jim Newton08:07:49

@kristof, yes that's basically what I'm doing without the macro. Perhaps I need to write my own macro (not difficult), just wondering if I'm reinventing the wheel. In general something like expect which takes a value to return and a predicate to assert is true.

Jim Newton08:07:15

Ben, are you suggesting that I instrument the reduce function? that sounds dangerous. I certainly don't want to interfere with an internal use of reduce which really needs to return nil.


If you don't want to pull in that whole dep, it's much better to write a little macro. And remember that macros are not functions so you'd have to wrap it with a #(...) function literal if you're going to pass that into reduce

Ben Sless08:07:15

If you're calling reduce in your function it's more complicated, but if you pass its result as an argument to a function you wrote, you can certainly spec that argument to require it cannot be nil


personally I think it's a mistake that assert returns nil when it passes. there should have something like (assert expr predicate-fn optional-error-msg) which returns the result if predicate-fn is not ni.


Sounds more like you want to instrument a function that calls reduce or maybe the function you pass into reduce.


instrumenting is a little overkill for this kind of sanity-check

Jim Newton08:07:57

returning nil is just an example. the most general case has nothing to do with nil. I just want to pinpoint here in my program some condition is becoming true or is failing where it shouldn't


i've written the same thing a million times 😞

Ben Sless08:07:45

Maybe something with data readers?

(require '[clojure.spec.alpha :as s])

(s/def ::numero number?)

(s/valid? ::numero 3)

(defn speco
  [spec form]
  `(let [ret# (do [email protected])]
     (assert (s/valid? spec ret#))

(binding [*data-readers* {'spec (fn [form] (let [s (:spec (meta form))] (speco spec form)))}]
  (+ 1 #spec ^{:spec ::numero} (+ 2 3)))

Jim Newton08:07:40

ben, I think that's far too complex for what I need. I've written a simple debugging function assert-debug which takes a value and a side-effecting-function.

(defn assert-debug "call the assertion function as side effect, and return the given value."
  [expr assertion]
  (assertion expr)
the side effecting assertion function is responsible for making assertions which appropriate error messages.

Jim Newton08:07:40

the assertion function can use spec if it needs to, or just call assert


related discussion on HN (about CL nil): My take on nil is that it is pragmatic but also sometimes annoying like in your case because it is overloaded and lacks context in of itself. Some Clojure code return a keyword instead to signify meaning like :clojure.spec.alpha/invalid


(check (not true)) (check (+ 1 2) (fn [i] (= i 3))) (check (+ 1 2) (partial = 3) (partial odd?) (partial even?))


For extra robustness, you need to wrap both expansions in a try-catch, however

Jim Newton11:07:11

yes in my case, there wasn't supposed to be a nil. It was coming from somewhere and I was just trying to find its origin. In the end it came from calling :pattern on a map which didn't contain such a pattern. I finally found it.

Jim Newton11:07:30

@kristof, I decided to allow the caller to make the assertion, thus the caller can decide whether to provide a 2nd argument to assert if he wants to.


I think you want (doto assert) It plays nicely with -> chains

Jim Newton09:07:55

Does anyone else have the problem that clojure.test obscures lines numbers in stack traces? I have code like the following:

(defn all-states-have-patterns [dfa]
  (doseq [q (states-as-seq dfa)]
    (assert (not (nil? (:pattern q)))
            (cl-format false "dfa=~A contains state ~A with nil pattern"
                       dfa q))))

(deftest t-test-1
  (testing "particular case 1 which was failing"
    (let [dfa-1 (assert-debug (rte-to-dfa '(:+ (:cat String (:? Long)))
                                          1) all-states-have-patterns)
          dfa-2 (assert-debug (rte-to-dfa  '(:cat (:* String) Long)
                                           2) all-states-have-patterns)
          dfa-sxp (assert-debug (synchronized-product dfa-1 dfa-2 
                                                      (fn [a b]
                                                        (and a b))
                                                      (fn [q1 _q2]
                                                        ((:exit-map dfa-1) (:index q1)))) all-states-have-patterns)
But I get a stack trace like this: as if coming from the REPL.
1. Unhandled java.lang.AssertionError
   Assert failed: dfa=#<Dfa 7 states> contains state #<State 4> with
   nil pattern (not (nil? (:pattern q)))

        xymbolyco_test.clj:  236  clojure-rte.xymbolyco-test/all-states-have-patterns
        xymbolyco_test.clj:  234  clojure-rte.xymbolyco-test/all-states-have-patterns
                      REPL:  552  clojure-rte.util/assert-debug
                      REPL:  550  clojure-rte.util/assert-debug
                      REPL:  246  clojure-rte.xymbolyco-test/fn/fn
                      REPL:  241  clojure-rte.xymbolyco-test/fn
                      REPL:  240  clojure-rte.xymbolyco-test/fn
                  test.clj:  203  cider.nrepl.middleware.test/test-var/fn
                  test.clj:  203  cider.nrepl.middleware.test/test-var
                  test.clj:  195  cider.nrepl.middleware.test/test-var
                  test.clj:  218  cider.nrepl.middleware.test/test-vars/fn/fn
                  test.clj:  687  clojure.test/default-fixture
                  test.clj:  683  clojure.test/default-fixture
                  test.clj:  218  cider.nrepl.middleware.test/test-vars/fn
                  test.clj:  687  clojure.test/default-fixture
Is this just an artifact of using cider?

Jim Newton09:07:47

Oh, I think I see the problem. In Cider cnt-c c unfortunately does not record the file name and line number with the function being defined.

Jim Newton09:07:10

not a problem of clojure.test at all, rather a problem with cider.


You'd have the same problem if you typed the defn in at the repl - there is no source file for it to point to. I tend to use cider-load-buffer (which is bound to C-c C-k by default, but I have it running on a save hook) which will eval the whole buffer including the line numbers ...

Tiago Dall'Oca18:07:36

I'm considering using [chime]( for scheduling jobs, but I also need to read from cron-like syntax to generate a periodic-seq . Any ideas on how to approach this? It'd be great if there was a lib for that


there are surely job schedulers that use cron syntax already?


Cron4j was the most lightweight I could find.


any such libraries need some scaffolding to turn into servers (seems like they presume being in-process)


not sure what the use-case is


right, most of the time if you want periodic job execution, you want state that persists across restarts / crashes of the vm, and that's where the real complexity would come in


whatever man, start it on your laptop and turn off sleep, it’ll be fine


YAGNI amirite?




in that case why not (future (loop [] (Thread/sleep N) (f) (recur)) - hard to get lighter weight than that in clojure world

phronmophobic19:07:19 is worth considering and it also can be setup with cron expressions


@smith.adriane a good lib and also pretty much the opposite of light weight


Is that a requirement?


it's something the OP mentioned at the start

Tiago Dall'Oca19:07:34

I didn't mention it, yes, but it's desirable to be lightweight

Tiago Dall'Oca19:07:47

The requirements rn are quite loose to say the least, and I'm supposed to be doing an MVP whatever that means

👍 2

we have a "cron" service at nubank that services triggered crons by either calling an HTTP endpoint or dropping a message on a Kafka topic


would be nice to have something "cloud native" that is lighter weight than quartz but more full-featured than CloudWatch Events


in JS code, if the jobs don't super matter what i've seen is just


setInterval(task, 10000);


with some hard coded delay


like for repeatable tasks that you can monitor externally (cloudwatch alarms if you dont get logs) and restart trivially (reboot a machine) thats good nuff


so I wouldn't knock just

(defmacro restart-on-failure
  [& body]
  "Performs the given task, but retries if an exception is raised"
  `(let [task# (fn []
                   {:result (do [email protected]) :ok true}
                   (catch Exception e#
                     (log/info (keyword (str *ns*) "restarting-forever-loop") true
                               (keyword (str *ns*) "error") e#)
                     {:ok false})))]
     (loop []
       (let [res# (task#)]
         (if (:ok res#)
           (:result res#)


  (while true
    (Thread/sleep 10000)


(I have that macro from ~2 years ago from a random side thing)


Someone here lately told me about java TimerTask and it works nicely without any additional dependencies (as long as cron exprs are not needed)


There is also that provides a nicer API because you can schedule execution of Clojure functions (because they are Runnables).

☝️ 4
👏 2