this library is so friggin cool

I have no memory of this ticket :) but interested in seeing a proposal!

Oh, this came from someone else, that's why I don't remember it, just forwarded it along

@U064X3EF3 Well, it was fun speculating about how you might have come about it anyway 🙂

I do have a vaguely related clojure.test ticket that I did file though

https://clojure.atlassian.net/browse/CLJ-2639

vaguely related in that it's about getting better feedback on exceptions from tests

@U064X3EF3 attached a proposal to https://clojure.atlassian.net/browse/CLJ-2525

How does this compare to the data structures in https://github.com/lacuna/bifurcan?

They are definitely comparable - it would be interesting to profile them against each other. Aside from performance (which if Bifurcan has any major advantages we should usurp them) I would say the ham-fisted datastructures implement all the Clojure interfaces - Indexed, IPersistentVector, IEditableCollection, etc. You can see this with a quick scan of the java source code or the api implementation. So a mutable ham-fisted map is a bitmap trie but it implements IEditableCollection so you can use the transient functions, assoc! and friends and persistent! works. ham-fisted aims to be drop-in compatible with Clojure's existing pathways. Furthermore there is a lot in ham-fisted aside from datastructures - as I note in the reddit post and the README there are a lot of algorithmic primitives that either perform better, such as group-by, or that plain don't exist and/or perform much better such as update-values and group-by-reduce. Finally there are Clojure-friendly wrappers for all the java primitive array types so you can pass a primitive array backed data to the rest of your Clojure program and nth, reduce and friends all work correctly and perform much better. There are type-specific implementations of sort and sort-indirect provided for ints, longs, and doubles that perform much better than the generic java sort pathways. So I would say overall two things aside from performance which both libraries focus on. First - it is drop-in replacement for many core Clojure datastructures and functions and second there are new primitives to the core clojure functions that I think are interesting/worth researching further.

Well, and another thing there is a lazy-noncaching namespace which provides drop-in replacements of map, filter, and concat which perform better than eduction in some pathways and don't require you to rewire your code from pure clojure.core to transducer form and which in map's case also preserve the random-access property of the input if all the inputs are themselves random-access.

@UDRJMEFSN > there is a lazy-noncaching namespace which provides drop-in replacements of map, filter, and concat which perform better than eduction in some pathways and don't require you to rewire your code from pure clojure.core to transducer form That's really interesting! @U4YGF4NGM and I have been doing a similar thing in #clavascript which is a CLJS syntax to JS compiler which implements some CLJS-esque things directly in JS (but most of the things work directly with mutable data structures, so no CLJS data structures). The implementation there is based on generators. Properties of lazy values returned from map, filter, range, etc are that they are not cached and not chunked, so consuming the value twice will make calculations happen twice, but otherwise they should behave pretty much identical to CLJ(S). You can play around with that here: https://clavascript.github.io/clavascript/ ham-fisted seems like a really interesting lib: I guess the idea is that you can use many things as drop-in replacements with clojure.core and it should all mix and match without rewriting your code, but get better performance... right?

You are exactly right about lazy-noncached. I think that is a more error prone programming model overall but the performance benefits can be substantial as long as you respect the sharp edges. Drop-in - well that is the theory:-). - I am working on this on far less than part time so ymmv. You will get better performance if you move further towards the library and use some of the primitives in it - update-values, union, etc are much faster if you are using datastructures from the library and if you know you are dealing with doubles sort is much faster if you pass in a double array - also true for int and longs. In some respects it is also a far simpler bridge into dtype-next land and something I intend to integrate into dtype next as a substrate. I think overall I want a community platform where we can try out different datastructures and algorithmic primitives in a testable way to move the language forward. I think there is room to improve both clojure.core and the persistent datastructures but I don't think the core language is the place to do it until we are damn sure we are ahead and not causing regressions.

Speaking of JS - there is probably room for this type of experimentation in CLJS meaning I think a very careful examination of the datastructures and algorithms that underly the CLJS system would yield substantial benefits but I have no proof of this -- it is just a suspicion.

Yes and since they are built on protocols, it should be fairly similar to what you did I guess

Yeah, I'd be interested in helping out with a cljs port of this

Rather than a port, I would love a CLJC, even if the CLJS part just contains scaffolding to begin with, subcontracting to cljs.core (where semantically equivalent, of course). This would provide an additional layer of generality as well as a baseline to test against.

My thought would be that you would need to write very directly to the js substrate and you would lose performance with cljs or cljc. This may not be true but that is my experience from clj/java - some pieces really do need to be written in Java.

So you would have a js library of datastructures and a cljs wrapper. Again, there is an assumption here but if you are going to absolute top performance you need to make sure the cljs->js translation is perfect for that use case and I don't know that it is.

I'm sure, but would that change the surface verbs: group-by-reduce, etc.?

Oh no it wouldn't. I see what you mean now.

The key would be to go CLJC at the level where there's semantic equivalency. At that level, test suite should be reusable for both even.

Yes- speaking of test suite - I could use help there.

Rewriting the core data structures and core functions in JS would have tree-shaking benefits with #cherry too - currently we're trying to figure out how to make the Google Closure output tree-shakable but it's very hard

Maybe it's possible to plain steal some parts from the Clojure test suite where applicable. https://github.com/clojure/clojure/blob/master/test/clojure/test_clojure/vectors.clj

The idea is to have a once-optimized compiled version of CLJS core which we can then re-use to build upon, so you can continue tree-shaking with esbuild

The vectors test suite is a great start. I thought the google clojure compiler did the tree shaking.

I'd like to see a persistent data structure lib that can read and write to wasm memory, such that one day when the JVM runs on WASM, we can synchronously bang on the same structure from both CLJ and CLJS

@UDRJMEFSN google closure does whole-program optimization, but the output isn't compatible with es6 treeshakers, so you can't re-use that beyond the context of an application

got it.

Don't know if you're aware but recently I found this approach to immutability in JS: https://github.com/immerjs/immer

Their objects are 100% compatible with JS interop, so you don't have to do conversions which are often expensive.

More on their performance: https://immerjs.github.io/immer/performance/

They "freeze" their objects in development, to protect you from accidentally mutating, but they elide that in production

They use a sort of transactional model with copy-on-write within the transaction. Once copied within a transaction there is no need to copy further. That is a sort of interesting twist similar to transients.