Clojurians Log v2

Thanks, I'll try that on my machine!

This is indeed much much faster than all the other solutions I tried. I tried a couple from Github or the Advent of Code subreddit and all of them where so much slower. I'll try to find if it's an issue on my end. Thank you!

This was my other attempt going full java and it was terribly slow as well.

It also seems like cider hangs when evaluating code at times, which makes my previous testing kinda moot :/

did you use *warn-on-reflection* to figure out where the type hints were needed?

I didn't know it was a thing... 🫠

I'm going to start over from what @didibus wrote and understand what's going on. If you or others have have good recommendations for a profiler, I'd be grateful.

None of these do what you want: (^int constantly 1) -- that attaches the type hint to the function constantly and then evaluates the fn call anyway.

Yep I've realized afterwards, I don't think it's a good way to find the solution tho. I'm messing around and bothering you guys 😬

It's #beginners -- we've all signed up to help 🙂

It's slow because you are using the multi-arity indices of aset. With working with arrays, stick to aset and aget (don't use aset-int and such). And never use the multi-arity indices. Instead do (aset (aget arr i) j val) And you also need to make sure you type hint properly to avoid reflection and boxing. For some reason, the specialized versions like aset-int are slower than the generic aset properly type hinted. And the multi-indice one will use reflection internally which is slow.

ooof good to know

And one thing that's tricky is that, warn-on-reflection only warns if there is reflection in your own code namespace, not if reflection is happening inside a function you are calling. So in this case it's very difficult to know that aset with multi indices will do reflection.

https://ask.clojure.org/index.php/729/aset-aget-perform-poorly-multi-dimensional-arrays-even-hints

Here's a good article with tips for working with arrays: https://cuddly-octo-palm-tree.com/posts/2022-03-06-opt-clj-8/

I hadn't noticed it at first, but your original code was also using the multi-arity indices of aset and that's probably why it was slow. I think the learning here is... Don't assume Clojure's immutable vectors or sequences are slow at first. Go for the most straightforward implementation. As we see in mine, using vectors, it completes under a second. Then you can try to optimize the "simple" way, use transients, maybe be smarter around your use of sequences, use transducers, avoid flatten if mapcat will do, etc. Finally, if that's not fast enough, make sure it's not the BigO of your algorithm, and if that's not the case I'd say now try to interop with Java. Even before arrays though, I'd start and try mutable Java collections like ArrayList and so on. Make sure you avoid reflection. Finally, drop down to arrays, but be careful with some of the unintuitive APIs around them that are really slow because they do reflection internally.

Thanks, I think it's a good approach indeed. I think I got frustrated in my original solution (not using Java arrays), because I didn't have the right intuition for how to manipulate a vector of vectors for the solution.

It takes some time to build that intuition, the functional immutable approach is very different, it's like you are building the structure back up each time and working of the last modification to the next. It's one of the challenging mental shift. I think those 4clojure exercises are pretty good to learn that gradually, it's mostly focused on it.

Thanks for all the good advice!

+1 for 4clojure, it scratches the same itch as doing puzzles for me, and it's a great way to learn. also check other people's answers and follow people with interesting answers. some of them are weird code golfing, but eventually even the code golfing tricks, though bad for production code, are good insight to how the language works on a deeper level.

Also, I think you asked for a profiler, this is the one I like to use: https://github.com/clojure-goes-fast/clj-async-profiler

oh wait the old 4clojure seems to be gone, and the new one seems not to have login? that's too bad

I hesitate to point you towards all those haha, because optimization is kind of a rabbit whole when learning the language, you should not worry too much about that. But all the libs from clojure-goes-fast are great to optimize. This one lets you see what Clojure code compiles too (as java code): https://github.com/clojure-goes-fast/clj-java-decompiler This one let's you know how much memory Clojure code uses: https://github.com/clojure-goes-fast/clj-memory-meter And the profiler is good for profiling. I normally have them all as :dev alias dependencies in my projects so they are available at the REPL if I need them.

@noisesmith Ya, the original is gone, and the new one runs all in-browser (uses scittle I think). But the solutions of the old are available, you just can't publish your own.

I see lists of solutions, but no user names attached and no functionality to "follow" a user to check their solutions to other problems

This is the dump from the old: https://drive.google.com/file/d/1hHrygxAs5Do8FpHC9kphYnmyTwZvISnb/view?usp=sharing Not sure if it includes that.

bingo!

{ "_id" : 1, "description" : "This is a clojure form.  Enter a value which will make the form evaluate to true.  Don't over think it!  If you are confused, see the <a href='/directions'>getting started</a> page.  Hint: true is equal to true.", "difficulty" : "Elementary", "tags" : [], "tests" : [ "(= __ true)" ], "times-solved" : 1432, "title" : "Nothing but the Truth", "user" : "dbyrne" }

oh wait - that's not solution data though, it just shows the user who presumably added the test

the version of reverse that makes me nostalgic for old lisps

(partial reduce conj ())

@noisesmith I was thinking about doing this at some point indeed! My initial solution was using Python and setting up a grid was pretty trivial there but I'll think of a better solution which is closer to idiomatic clojure. A flat set of positions feels better to work with as well.

I've tried some new things as well as a solution from here https://github.com/rxedu/adventofcode-2015-clojure/blob/master/src/adventofcode/day_06.clj But it's still very slow, I get can easily get this under 2 sec in Python but Clojure takes about 50sec 😕

You're facing the reality that Clojure is made to help with messy code-base that are difficult to understand, evolve, extend, or fix bugs in and don't let you deliver working features quickly. It's primary design considerations is the above. That's why the defaults are immutable, favor correctness, and so on. Now the trick if you want to implement fast numerics in Clojure, or just in general for performance: 1. Type hints (to avoid reflection when doing Java interop) 2. Avoiding boxing overhead (working with primitive numeric types) 3. Using mutation 4. Dropping down to specialized tools

I think the Advent of Code is maybe not the thing to get started with, there are a lot of problems with mutability like the one above. I just didn't expect a x20 slow down :/ I'll try something else to play around with it.

You can get it fast, but I'd say it's not "beginner friendly". Because you have to learn to type hint, work in primitives, interop to use Java arrays or mutable collections, etc.

you are still using flatten, flatten is never a good choice if performance matters at all

and with shallow nesting like yours, flatten is very easy to replace

oh - wait, looks like flatten is in a defn but no longer in a solution code path

But yes, I agree, I don't think leetcode or adventofcode style are good places to start to learn Clojure. I mean, they all really end up requiring a solution that is basically the same in any language, just with slightly altered superficial syntax. I think you are better off doing: https://4clojure.oxal.org/ These are exercises to learn functional approaches to data manipulation. And then just try to implement some real applications.

also, as a minor thing: you use clojure.string without requiring the namespace. this is not guaranteed to work, and best practice is to always use :require in your ns form for all namespaces you use

@didibus I feel like bookmarking this thread for next time someone asks how clojure performance compares to java

(when newcomers show code that is too slow, we get one answer about clojure perf, when newcomers ask how performant clojure is we get another much different answer :D)

@noisesmith I'm ok with digging in the docs but I couldn't find too much info about escape hatches to Java, if I've trying a couple solutions not shown here and they're all invariably much slower than my baseline Python solution. Thanks @didibus I'll try that!

I think it's technically the same answer, but the context is different. Clojure lets you build apps that are responsive, high throughput, and scale to many concurrent users. In that sense you could call it a performant language. It should beat Python, Ruby, and most dynamic languages, and hold its own against Java. You mostly get that by default, without needing to optimize much. That’s the typical out of the box experience. Clojure also gives solid performance for data transformation. That means low latency when moving, restructuring, or converting data. Strings, regexes, type conversions. Maybe not quite as automatic as the above, since there are some gotchas with sequences, but transducers help with most of that, even if they aren’t the default interface. Where things get debated more is here: Clojure can do numeric computation well, but not out of the box. You'll hit boxing, persistent collections aren't ideal, and sequences aren't great for iteration. You'll need to optimize and maybe reach for specific libraries. Generally you can achieve close to Java here as well, and if you drop down to C/C++ and MKL interop and such, GPU acceleration, and all that, you can get pretty close to metal fast, but it takes more work and care. Clojure doesn’t start up fast unless you use workarounds like native compilation, which limits your libraries and how you write code, or go with Clojure-like tools like babashka.

I took a crack at it. This is my first pass. On my machine it took: "Elapsed time: 384.930875 msecs"

Without using transients, I get "Elapsed time: 875.022458 msecs". So plain vectors seem plenty fast here. Though I can't compare to Python on my machine, but I can't imagine a slower computer would go from less than 1 sec on my machine, to 50 sec.

For future reference - please avoid posting multiple messages back to back in the channel. It's much better to either edit the original post to include more info or to start a thread and put additional comments there. > it didn't make sense to me to create a new 2d vector at each iteration That definitely shouldn't be your first thought, especially when you're a beginner. Also, vectors in Clojure aren't like contiguous arrays, they're more like trees - replacing a fragment of a vector won't create a copy of each and every element and will instead reuse most of the vector (well, that's true for sufficiently large vectors, and 1000 is a sufficiently large size). > For some reason it's terribly slow. You haven't provided all the code so hard to tell.

for one thing flatten is very slow on large inputs, nested reduce would perform better

flatten on grid is creating a new 1000x1000 element lazy-seq that nobody needs

oh, it's only 1000 elements, but that still is a problem

you could try (apply + (map (partial apply +) grid)) instead on line 11

or probably better:

(ins)user=> (def grid [[1 2 3] [4 5 6] [7 8 9]])
(ins)user=> (reduce (partial apply +) 0 grid)
45

for idiomatic purposes if nothing else:

(def actions-p1
    {"on" (constantly 1)
     "off" (constantly 0)
     "toggle" #(- 1 %)})

There's also an #adventofcode channel for folks to discuss and critique solutions to AoC (although it's mostly active when the new AoC drops each year, I suspect you can find folks there who've done the 2015 one in Clojure).

oh, now I remember, the problem isn't just (or even primarily) that flatten creates a lazy seq - there are efficient lazy-seqs. it's that it ends up doing deep nested concats (via tree-seq -> mapcat) that blow up perf especially for large inputs

or it did last I checked, might be better in newer clj versions

at least once upon a time, community wisdom was "never use flatten"

In general, if you go with arrays, you should stick with them. You can't iterate over an array using iterators or other such abstractions, so you should just loop element wise over it and sum it up.

Thanks everyone for the answers.

outside the box: what about a set of checked indexes rather than an array that has thousands of untouched zeros?

ins)user=> (defn grid-set [g x y] (conj g [x y]))
#'user/grid-set
(ins)user=> (defn grid-unset [g x y] (disj g [x y]))
#'user/grid-unset
(ins)user=> (-> #{} (grid-set 0 0) (grid-set 1 0) (grid-unset 0 0) count)
1

Also mapcat when you need to flatten by one level is better (in general).