A proper documentation for try: https://clojure.org/reference/special_forms#try

And you're correct, you can return something only from try or catch. yAnd you're correct,

Your conversion-* functions seem to also be multimethods. Is the same naming approach used on them, like (defmethod conversion-D1 'and method-conversion-D1-and ...)?

good catch. indeed some but not all my conversion-* functions are methods. most are defined with defn.

anyway, I'll apply the naming to those that are methods and see what happens.

yup, that's an improvement

BTW I discovered that I can indeed profile allocation. Just use (prof/start {:event :alloc}) instead of (prof/start {}). Some other tricks http://clojure-goes-fast.com/blog/clj-async-profiler-tips/.

Perhaps take a look at clojure.math.combinatorics/combinations or perhaps other functions from that library.

Not sure if you can just generate all the combinations and then shuffle those - depends on the parameters to your problem.

Perhaps something like the following could work:

1. Pick a random combination of cards (that you still have in reserve). 2. Return the hand, do whatever book keeping is necessary: decrease counts of cards in the chosen hand etc. 3. Loop if more hands need to be chosen.

thanks! I have an iterative solution that's similar to what you suggested but it doesn't always result in a solution if you end up with one bucket full and the others empty. my hack is to retry until it runs. wondering if there's a more elegant way to solve it, perhaps framing it as a graph traversal problem?

Perhaps create empty packs first, then deal all your aces to random packs that still need cards, then all the twos etc... when you are dealing, you ensure you don't deal two aces to the same pack.

oh, that's a great idea! reversal of what I've been doing

just to follow up, here's what I ended up with https://gist.github.com/yayitswei/d44ae8cba31544532285bf49295b5950

(defn deal-hands [n decks]
  (loop [left (mapv shuffle decks)
         results []]
    (if (empty? left) results
        (recur (->> (map pop left) (remove empty?))
               (concat results (->> (map peek left) shuffle (partition n)))))))

Depends on what "awkward" means. Maybe #(not (every? nil? %)) is better.

I guess to me the double “some some” is a bit awkward and doesn’t communicate its meaning strongly.

I guess #(not-every? nil? %) is more clear in its intent.

To communicate the meaning strongly, just use a name:

(defn has-some-content? [coll]
  (some some? coll))

Oh, right, there's also not-every?.

yeah, I keep forgetting about the not- versions of most of the core functions too, but your reply reminded me of it 🙂

I don’t think that would be a great approach. Typically, services managed by Component are stateful things you need to start and stop like DB connection pools, and any function that uses one of them is not pure. That pain you feel passing them around is good. It should encourage you to separate your business logic out into pure functions and to keep the ones that touch stateful components simple and shallow.

(let [xs '(1 2 3)]
  (for [x xs
        y xs
        :when (< x y)]
    [x y]))

doesn't work for

(let [xs '(3 2 1)]
  (for [x xs
        y xs
        :when (< x y)]
    [x y]))

also doesn't work if xs is infinite.

because the first loop will never finish, and y will never take on the 2nd value.

my feeling is it is not possible without consuming a huge amount of memory. maybe I'm wrong

math.combinatorics lib may have stuff like this

it does work in both cases.

(let [xs '(3 2 1)]
  (for [x xs
        y xs
        :when (< x y)]
    [x y])) ;; => ([2 3] [1 3] [1 2])

(first
  (let [xs (range)]
    (for [x xs
          y xs
          :when (< x y)]
      [x y]))) ;; => [0 1]

if the input is '(a b c d e f g) I want to generate (for example) all the pairs not containing d before generating any which contain d.

but won't all the elements of your second suggestion be [n 1] for some n? when does y advance to the 2nd value?

OK, I should try it before commenting.

the all will be [0 n]

clojure-rte.rte-core> (take 10 (let [xs (range)] (for [x xs y xs :when (< x y)] [x y]))) ([0 1] [0 2] [0 3] [0 4] [0 5] [0 6] [0 7] [0 8] [0 9] [0 10]) clojure-rte.rte-core>

indeed so the sequence doesn't contain all the pairs 😞

right, but because input is infinite it can’t output finite result without extra conditions

perhaps it is obvious what my goal is, but I'm trying to find the first pair (a b) which matches a given predicate, but realising as little as possible from the input sequence.

yes but it can generate an infinite sequence of ALL the pairs. your suggestion fails to do that. for example it could generate [0 1] [0 2] [0 3] [1 2] [0 4] [1 3] [0 5] [1 4] [2 3] [0 6] [1 5] [2 4] .... i.e., first all the pairs which sum to 1, then all the pairs which sum to two, then all the pairs which sum to 3 ....

that would contain all the pairs, not just the ones starting with 0

maybe that's my solution......

however, it would be great if [1 2] were generated before [0 3] as that would mean in some cases we would never have to realize the 3rd element of the input sequence.

strictly speaking it will generate ALL the pairs. But to materialize ALL of them you will need to wait infinite time

no, you're algorithm will never generate [1 2]

I would like an algorithm that will generate any pair after some finite time.

@U064X3EF3 with regard to math.combinatorics and similar libraries. I don't understand how to find the documentation? it is expected that perspective users read the source code to find all the functions? or am I missing something obvious? I see the examples. but not the list of functions and their explanations

math.combinatorics has combinations, but it doesn't work with infinite inputs.

https://clojure.github.io/math.combinatorics/

@U010VP3UY9X https://clojure.github.io/math.combinatorics/

if I'm not mistaken, the algorithm you're looking for sounds a lot like "zigzag ordering" (not sure what's the official name), which traverses an array like this: https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/JPEG_ZigZag.svg/200px-JPEG_ZigZag.svg.png. the example is a finite array, but i'd guess you could generate an infinite sequence of indecis

Similar, but not exactly like this. Zigzag won't mach all pairs.

Something like this? @U010VP3UY9X

(defn lazy-combinations
  ([coll]
   (lazy-combinations [(first coll)] (next coll)))
  ([seen coll]
   (when (seq coll)
     (lazy-seq
       (let [x (first coll)]
         (concat (map (fn [s]
                        [s x])
                      seen)
                 (lazy-combinations (conj seen x) (next coll))))))))

(take 10 (lazy-combinations (range)))
=> ([0 1] [0 2] [1 2] [0 3] [1 3] [2 3] [0 4] [1 4] [2 4] [3 4])
(some #(= [100 200] %) (lazy-combinations (range)))
=> true
(vec (lazy-combinations (range 5)))
=> [[0 1] [0 2] [1 2] [0 3] [1 3] [2 3] [0 4] [1 4] [2 4] [3 4]]

First time ever I had to use lazy-seq, heh.

Here is my attempt which won't work because (into [] seq) will never finish.

(defn lazy-pairs-2
  [seq]
  (cond (empty? seq)
        ()

        :else
        (let [vec (into [] seq)]
          (for [i (range (count vec))
                j (range i)]
            [(vec j) (vec i)]))))

Is there something wrong with my version or are you simply trying to come up with your own algorithm?

no, I think you're algorithm produces the same as mine. I think I'll use it. I was wondering whether it could be done without recursion.

it can't be done with loop/recur because the recursion is not tail-recursion.

Mine doesn't use recursion, at least not in a way that consumes the stack.

@U0178V2SLAY yes that's the idea except that it too greedily realizes the input sequence. For example, it realizes [0 3] before [1 2].

@U2FRKM4TW is that true? I didn't realize that subtlety .

As a demonstration:

(first (drop 100000000 (lazy-combinations (range))))
=> [8989 14142]

lazy-seq creates a closure. See the top example here: https://clojuredocs.org/clojure.core/lazy-seq

I guess concat , lazy-seq, and lazy-combinations really do return before the otherwise recursive call to lazy-combinations happens, right?

Great. I think I'll use your code if you don't mind. and I'll put a comment with you as the author.

Right. Well, except for the only truly recursive call due to the additional arity. Sure, go ahead! :) Happy to help.

BTW you use a vector as seen. why? why not a list?

cons onto list is faster than conj onto vector, right?

sorry if that's a stupid question;.

And since it's my first usage of lazy-seq, perhaps it could be improved - may be worth asking a separate question in the main channel. Not stupid at all! Perhaps a list with cons is better. But you'd have to measure really carefully. And it's O(1) either way. Besides, using a list with cons will change the order of the pairs. No idea if that matters in your case, but I find the vector-based result more intuitive.

cons onto list is faster than conj onto vector

But we also iterate over it after each cons/`conj`. A list would still be faster?

I believe so, yes (but iterating is going to be the same time complexity either way)

with vectors, conj is not really O(1). if there's room in the tail (best/average case) then you add it to the tail. but in the worst case, the tail is full and you're (potentially) making 1 or more new tree nodes

Thanks! Makes sense.

use macroexpand on your form and see what pops out

I think macroexpand fails.

why guess when you can check?

Clojure 1.10.3
user=> (defmacro foo
  "Test whether there exists an element of a sequence which matches a condition."
  [[var seq] & body]
  `(some (fn exists-some [~var]
           [email protected]) ~seq))
#'user/foo
user=> (macroexpand '(foo [x '(1 2 3)] (even? x)))
(clojure.core/some (clojure.core/fn user/exists-some [x] (even? x)) (quote (1 2 3)))
user=>

so that fn form needs to be fixed

yes I really don't understand clojure symbol resolution well enough to understand why I need ~' before the exists-some symbol in the macro definition. But that fixed it. Several people suggested the same thing.

Replace it with ~'exists-some.

foo above is not anaphoric. clojure.core/for is anaphoric, but doesn't use [~x] in that context, as far as I can tell.

Oh, wait, foo actually is anaphoric, my bad.

how is it anaphoric. the caller provides a variable name, and the macro expects that the caller references that variable in the given body.

maybe I misunderstand anaphoric

It captures the name and allows you to use that name later in the body. At least, that's my understanding of the concept.

From Wiki: > An anaphoric macro is a type of https://en.wikipedia.org/wiki/Macro_(computer_science) that deliberately captures some form supplied to the macro which may be referred to by an anaphor (an expression referring to another)

ahhh. hmmm. indeed. that's not the intent. the intent is simply to allow the profiler to use the name fn-exists-some rather than fn- followed by some cryptic number.

but yes you are right, if the user knows the name he can use it. 😞

Nah, it's about the name of the var, not the name of the function.

That [~var] makes it possible to both define and use some particular name in that [email protected].

Similar to how you can use :let in for.

the example: (foo [x '(1 2 3)] (even? x))

the user provides x and he provides a body which references x ... is that anaphoric?

I thought anaphoric meant that the macro decided var

Looking at the actual examples - seems like you're indeed correct and my initial understanding was wrong.

I find that sometimes a form like

(exists [x some-input] (or (even? x) (prime? x)))

but obviously it is syntax-mappable to a call to some with a (fn ..) which uses the given variable name.

yeah you are right i think. you control the binding, rather than the macro introducing some symbol it thinks is appropriate

What I don’t get is why you are collecting the body as a variadic parameter. What would happen if I put in more predicates?

collecting the body?

You can replace & body with body and [email protected] with ~body - this will allow only one expression. With the & and @ there, you allow multiple expressions, and the results of all but the last one will be ignored - just like in a regular fn body.

Or you can combine such multiple expressions within e.g. (and ...). A matter of preference, I suppose. HoneySQL does that, at least v1.

what about (exists [x some-seq] (prn [:x x]) (even? x)) ?

I'm not sure if leaving room for such statements is a good thing. You can always wrap something in a do to help with debugging. But making an explicit place for it is just asking for potential errors.

and of course (exists [x some-seq] (and (even? x) (prime? (inc x))))

^ that will work in either approaches.

what kind of potential errors are you referring to?

(exists [x some-seq] (even? x) (prime? (inc x))) - where you forget and . Or not you but someone else could think that there's an implicit and . There will be no error message, just a silent ignore of (even? x).

ahh. but I would never do that, so I don't consider it as potential. interesting that someone might think that is a reasonable thing to do.

> I would never do that Famous last words. ;) As I mentioned - where in HoneySQL does just that. (hh/where [:= :a 1] [:= :b 2]) will end up in ("a" = 1) AND ("b" = 2).

well these are macros I've used for 20 years, so it is hard for me to think about seeing them the first time.

if you convert the namespace to a class resource, and that resource can be loaded, then you're using compiled code

there is not a good public function to give you that resource name unfortunately. but it's: • replace - with _ • replace . with / • append "init.class"

I tried https://clojuredocs.org/clojure.core/load but had no success with it. It tells me

Could not locate clojure/test_tool/ui__init.class, clojure/test_tool/ui.clj or clojure/test_tool/ui.cljc on classpath.

(ns test-tool.ui
  (:gen-class))
(load "test_tool/ui")

Use clojure.java/resource to check whether the test_tool/ui__init.class resource exists

Thank you! Will try!

I should have explained the context more: the variable describes whether output should be sent to stdout or stderr. The valid values it can have in the outputting library is :err or :out.

Perhaps process-stream or somethIng is better. Having the possible values (out/err) in the name isn't good practice.

I agree

I don’t like process-stream though, because a stream would be something else typically

but it’s definitely better than err-or-out

I would call it write-to or write-target or something along those lines maybe.

But I’m not a naming expert, just what I personally like.

can you be a bit more specific regarding your use case and performance budget? Also, this is a good starting point https://github.com/joinr/performancepaper

in general, most things the JVM is trying to optimize are also things that benefit Clojure pretty directly, and Clojure has seen benefits over time from ongoing JVM optimizations

(one notable place where Clojure can have different GC characteristics than a typical Java program is if you keep around a very long-lived stateful persistent collection - from an object perspective, you can have an object chain of "mutations" that is long back to a root object that is very old whereas in Java, you'd probably just reuse the same existing collection object. that said, newer collectors like Shenandoah work great with Clojure so not sure it matters.)

Thanks for the link. That has some useful info. No specific use case, we are developing increasingly large streaming ML applications in Clojure. The metaphor is natural, functional transformation of data streams, but performance ups and downs can be a mystery when those transformations are complex. So I'm trying to pull together some guidelines when to resort to Java interop. Math operations are one place that Java interop seems to be a win and not too ugly. This week a question arose whether performance is better if one accesses Mongo we use as an intermediate store sometimes via Java interop or Monger. The same has come up with Kafka/jackdaw. Clearly profiling is the best way to determine for a particular case. But generally it would be nice to have some best practices to start from when performance rather than Clojure purity matters most.

in general, I am hesitant to give up on Clojure abstractions and make direct interop calls unless it is in the very hot path and really makes a difference. In which case, heck yeah. :) Math ops seem like a reasonable place to focus.

You have some options in Clojure land, as well: fast primitive math: https://github.com/generateme/fastmath ML is pretty matrix heavy, you might want to look at https://github.com/uncomplicate/neanderthal and https://github.com/cnuernber/dtype-next

But profiling first is good

I managed to reach throughputs deserializing JSON (in production environment, not just an experiment) where turning keys to keywords had a performance impact, but it's important to know if you're even there

I can write a lot about it (and have already) but knowing where the problem is is the first step

use criterium, clj-async-profiler, to get an initial picture JMH to get accurate microbenchmarks if you need to VisualVM and JFR for a holistic picture And JITwatch if you must, but I doubt you will unless you're trying to do HFT, which you aren't

@U064X3EF3 You've hit on a key issue. In our cases we do have some very long-lived data objects that originate from clojure, but not kept as large collections of objects in clojure. In those cases we don't use any type of clojure collection but effectively just flatten them and store them in a Mongo collection. In other situations, we have very large unbounded sequences of data, which commonly originate from as many as thousands of Kafka streams. We consume, transform,, and produce 10s, 100s, or 1000s of Kafka streams. So from this perspective, and this is where JSON ser-des is relevant that @UK0810AQ2 mentions, Kafka streams are really the large immutable collections we are manipulating. This makes most of the Clojure hot in a sense because every single operation is repeated large numbers of times. The profiling tools @UK0810AQ2 mentions could be quite helpful for us going forward.

I sometimes embed clj-async-profiler in an application and trigger it externally, gives you a very good profile of CPU activity. There's a lot here depending on your level of abstraction. If it's all streams and KTables then you may not be holding large objects for long. Not enough information to give a more useful response. Profile first

but always remember to solve the performance problems related to the platform, too, not just the language. Kafka can have performance pitfalls, so can kafka streams, make sure those are solved, too

for example, if you're using compression, use zstd and not gzip

@UK0810AQ2 @U064X3EF3 Thanks for all the suggestions. I'm going to pass all of this on the team

> This week a question arose whether performance is better if one accesses Mongo we use as an intermediate store sometimes via Java interop or Monger. The same has come up with Kafka/jackdaw. with "wrappers" there will never be a generic answer (about quality, performance, etc.) - some are simply better implemented than others, and all of them fail miserably when you leave the use cases that the wrapper author was considering for example, I worked on jackdaw when I was employed at Funding Circle, and I would be suspicious of how well it handles your use case of very large numbers of streams - it might just happen to work, but that also wasn't a usage we were designing around. Of course it might be the case that our abstractions are minimal enough that the issue never comes up (jackdaw is lower level than most wrappers) but on the other hand the kinds of abstractions we chose might make your use case more complicated than the interop version would be