Fork me on GitHub
#clojure
<
2022-01-21
>
didibus00:01:38

If I have 3 lists of ids, and I want to return a table of them where when they are equal they are on the same row. For example: Given: `[1 2 3 4 5 6 7], [2 4 7 9], [1 2 5 10 11 12]` I want:

``````[[1 nil 1]
[2 2 2]
[3 nil nil]
[4 4 nil]
[5 nil 5]
[6 nil nil]
[7 7 nil]
[nil 9 nil]
[nil nil 10]
[nil nil 11]
[nil nil 12]]``````
How would I do that? I can only think of mutable solutions, anyone has a clean Clojurish way for this?

jeff.terrell00:01:52

`concat`enate them, get the `distinct` ids, `sort` those, then iterate through the sorted ids to generate the rows of your table.

👍 1
didibus00:01:47

``````(let [coll1 [2 1 3 4 5 6 7]
coll2 [2 7 4 9]
coll3 [1 2 5 12 10 11]]
(-> (concat coll1 coll2 coll3)
(distinct)
(sort)
(->>
(reduce
(fn [acc e]
(let [in1? (.contains coll1 e)
in2? (.contains coll2 e)
in3? (.contains coll3 e)]
(conj acc [(if in1? e nil) (if in2? e nil) (if in3? e nil)])))
[]))))
;; => [[1 nil 1]
;;     [2 2 2]
;;     [3 nil nil]
;;     [4 4 nil]
;;     [5 nil 5]
;;     [6 nil nil]
;;     [7 7 nil]
;;     [nil 9 nil]
;;     [nil nil 10]
;;     [nil nil 11]
;;     [nil nil 12]]``````

hiredman00:01:03

``````((fn f [& inputs]
(lazy-seq
(when (some seq inputs)
(let [elem (apply min (keep first inputs))]
(cons
(for [i inputs
:let [e (first i)]]
(when (= e elem) e))
(apply f (map (fn [m] (if (= (first m) elem) (rest m) m)) inputs)))))))
[1 2 3 4 5 6 7],
[2 4 7 9],
[1 2 5 10 11 12])``````

didibus00:01:21

I think that works but only if they are sorted already?

hiredman00:01:48

sure, merge sorts usually do

didibus00:01:07

I have to study your solution a little more haha.

didibus00:01:12

But thanks as well!

noisesmith00:01:05

simplification of the reduce version

``````(let [coll1 (set [2 1 3 4 5 6 7])
coll2 (set [2 7 4 9])
coll3 (set [1 2 5 12 10 11])]
(->> (concat coll1 coll2 coll3)
(sort)
(distinct)
(reduce
(fn [acc e]
(conj acc ((juxt coll1 coll2 coll3) e)))
[])))``````

hiredman00:01:28

oh man, I have an idea

noisesmith00:01:50

actually that reduce should just be map

noisesmith00:01:53

``````(let [coll1 (set [2 1 3 4 5 6 7])
coll2 (set [2 7 4 9])
coll3 (set [1 2 5 12 10 11])]
(->> (concat coll1 coll2 coll3)
(sort)
(distinct)
(map (juxt coll1 coll2 coll3))))``````

4
noisesmith00:01:10

I think as simple as you can make it with clojure.core without getting into obfuscation

jeff.terrell00:01:24

Beautiful. One nit: I'd do distinct before sort, for (probably slightly) better performance.

noisesmith00:01:24

probably changing distinct into dedupe is the best perf fix

jeff.terrell00:01:30

That could be better for smaller inputs, but fundamentally `distinct` is O(n) (as is `dedupe`) whereas `sort` is O(n*lg(n)) so you'd want to minimize the number of elements you're sorting.

noisesmith00:01:39

also, sorting the collections before making sets of them would spare cycles, but sacrifices some simplicity in terms of the number of locals being used

hiredman01:01:36

``````(iteration
(fn [[elem inputs]]
(let [i (map (fn [m] (if (= (first m) elem) (rest m) m)) inputs)]
[(apply min Long/MAX_VALUE (keep first i)) i]))
:vf (fn [[elem inputs]]
(for [i inputs]
(when (= (first i) elem)
elem)))
:some? (fn [[_ inputs]]
(some seq inputs))
:initk [Long/MIN_VALUE [[1 2 3 4 5 6 7], [2 4 7 9], [1 2 5 10 11 12]]])``````

hiredman01:01:13

(I just had the realization that the reason I can never get a merge sort to work right as a reduce is that it is an unfold not a fold)

wevrem05:01:15

@U0K064KQV works with arbitrary number of inputs:

``````(defn merge-sort [& cs]
(let [v (map first cs)]
(when (apply not= nil v)
(let [low (reduce min (keep first cs))
v (map #{low} v)
rcs (map (fn [v? c] (if v? (rest c) c)) v cs)]
(cons v (lazy-seq (apply merge-sort rcs)))))))``````

hiredman00:01:12

its a merge sort

stopa04:01:43

Hey team curious question: Is there a tutorial out there, which implements something like Clojure’s persistant map from scratch?

lsenjov04:01:34

Possibly try looking at Immutable.js? Afaik they copied Rich's implementations

👍 1
Nom Nom Mousse09:01:26

I have an atom that keeps track of the resources used by the currently running jobs. I also have a function that uses that atom to see what jobs I could possibly start based on the resources left. So basically I want to do something like:

``````(let [[runnable-jobs resources-left] (find-runnable-jobs-and-resources-left-afterwards @resources-left-atom jobs)]
(reset! @resources-left-aftom resources-left))``````
Here you see that there is a lag between using the atom and updating it. Is there a way to lock access to the atom for these operations? Normally swap! would do this, but I cannot use it since I return two things.

Nom Nom Mousse09:01:35

I know agents might solve this but I'd like to just get what I have working without too many changes.

Nom Nom Mousse09:01:37

Something like

``````(with resources-left ;; locking
(do x)
(do y))``````

p-himik09:01:01

Do everything inside `swap!` - should be fine as long as you don't side-effect in that "everything".

Nom Nom Mousse09:01:39

As you see my code returns two things, only the latter of which should go in the atom.

``(swap! resources-left-atom ->runnable-jobs ready-jobs)``
The above code would set the `resources-left-atom` to `[resources runnable-jobs]` right? I only want it set to `resources`.

p-himik09:01:42

> As you see my code returns two things I understand what you mean now, but to be precise - your code above returns the value of `(reset! ...)`, which is `resources-left`. Two solutions that I see: • Store `runnable-jobs` in the atom • Use `swap-vals!` and run `find-runnable-jobs` twice - once within `swap-vals!` to find the new `resources` and reset the atom and the second time outside of `swap-vals!` but using its results to actually get the `runnable-jobs` value

Nom Nom Mousse09:01:12

Thanks for your suggestions. Are you able to write a two-three line snippet of pseudocode for the last suggestion?

p-himik09:01:39

``````(defn find-runnable-jobs-and-resources-left-afterwards [resources-left possible-jobs]
"Must be a pure function."
[(dec resources-left)
[(first possible-jobs)]])

(def resources-left (atom 10))
(def possible-jobs (vec (range 10)))

(let [[old-resources-left new-resources-left1]
(swap-vals! resources-left (fn [rl]
(let [[rl _rj] (find-runnable-jobs-and-resources-left-afterwards rl possible-jobs)]
rl)))

[new-resources-left2 runnable-jobs]
;; This call is exactly the same as the one inside `swap-vals!`.
(find-runnable-jobs-and-resources-left-afterwards old-resources-left possible-jobs)]
(assert (= new-resources-left1 new-resources-left2))
(run-jobs! runnable-jobs))``````

Nom Nom Mousse09:01:59

Thanks ever so much for your help 😄

👍 1
mpenet09:01:50

you could use swap-vals

Nom Nom Mousse09:01:58

I'll play around with it to see what it does 😄 Thanks for the suggestion

mpenet09:01:08

I guess it depends what kind of guarantees you need/want

Nom Nom Mousse09:01:22

Another option is using a local atom which I update from within `find-runnable-jobs`. It feels a bit ugh to me.

``````(let [jobs-pointer (atom nil)]
(swap! find-runnable-jobs-and-resources-left-afterwards resources-left-atom jobs-pointer)
;; jobs-pointer is now updated with runnable-jobs``````

p-himik09:01:13

It's wrong because the function that you pass into `swap!` is now side-effecting. Which is explicitly prohibited for `swap!` in its docstring.

Nom Nom Mousse09:01:51

Good point 🙂

Nom Nom Mousse09:01:31

Since the side-effect is resetting the unwatched, local atom I do not see a great danger but still. I guess it is one of the reasons it felt ugh

p-himik10:01:38

There is danger - the inner `reset!` might be run multiple times, and between those calls, there might be another `reset!` from another thread that uses the same functionality. You can end up with resources left being set to 5, then to 4, then back to 5, whereas in reality the actual resources left will be 4.

🧠 1
Nom Nom Mousse09:01:27

I might be overthinking this but I guess I should try to avoid all sources of potential inconsistency since they might bite me later in unexpected, hard-to-debug ways.

Nom Nom Mousse10:01:29

I am glad I asked this question because I learned a lot, but I think I realized that I have three separate atoms that are so tightly intertwined that they really should be one: 1. `jobid->process` (to store the currently running processes) 2. `jobid->job-state` (whether they are `:not-ready`, `:ready`, `:in-progress`, `:failed`, or `:done` 3. `resources-left` (what resources are available given the already running jobs) Things that are related should be changed together. I was trying too hard to decomplect.

👍 1
orestis11:01:25

I have a simple but annoying refactoring I want to make. I have a bunch of `defn`s spread throughout the project that have to be changed into a macro invocation (say `defmoo`), while also requiring the namespace like `(:require [bar.moo :refer [defmoo]])`. The vars are already in a map so I don't need to grep to find them. But I would like to automate the refactoring. I guess rewrite-clj is the way to go for the actual change. Is there a higher-level API that deals with files on the disk though? Or is that not even needed really, and I can just slurp -> parse -> rewrite -> str -> spit?

magnars12:01:49

slurping and spitting is fine, since rewrite-clj will maintain any whitespace and comments

orestis13:01:06

Indeed it works perfectly

clj-kondo data analysis can sometimes be a handy way to quickly find what you are looking for, then rewrite-clj can be used to rewrite. Drop by #rewrite-clj if you wanna chat more.

orestis17:01:05

In my particular case I literally had all the vars as vals in a map so I could just use that and meta to get the file name. Now I'm addicted to this kind of thing though, so I will check out clj-kondo analysis next time.

Awesome! Feedback on rewrite-clj and its docs are always welcome, so please share if you found anything good/bad/confusing in #rewrite-clj

jerger_at_dda12:01:46

While sitting in a activity-pub workshop ... does anybody know a clojure based activity-pub implementation?

Franklin12:01:41

What does this error message mean? `src/milia/api/io.clj:87:27: reflection: call to java.lang.String ctor can't be resolved.`

Franklin12:01:08

here's the code on line 87 `(parse-json-response (String. body "UTF-8"))`

Ben Sless12:01:03

The second argument needs to be a charset instance, not a string

Ben Sless12:01:11

What library are you using for parsing?

Franklin12:01:52

sweet! thanks

Franklin13:01:19

I'm using cheshire

Franklin13:01:44

although I have to say, that error message is super unclear; I don't think I'd have noticed the issue by myself

Ben Sless13:01:55

This error message is from clojure's reflector, not cheshire

Ben Sless13:01:23

But I'm asking because cheshire can deserialize bytes or stream directly, no need to turn it to a string at all

Ben Sless13:01:51

this is just adding extra work for nothing :man-shrugging:

Franklin13:01:20

there's a try catch that looks like this

``````(try+
(json/parse-string body true)
(catch ClassCastException _
(parse-json-response (String. body (Charset/forName "UTF-8"))))``````

Franklin13:01:37

not sure why the person who wrote it originally added it

Franklin13:01:51

the whole mtd looks like this

``````(defn parse-json-response
"Parse a body as JSON catching formatting exceptions."
[^String body]
(try+
(json/parse-string body true)
(catch ClassCastException _
(parse-json-response (String. body (Charset/forName "UTF-8"))))
(catch JsonParseException _
(str "Improperly formatted API response: " body))))``````

Franklin13:01:55

it's recursieve

Franklin13:01:21

actually, it seems the error still isn't resolved after adding `(Charset/forName "UTF-8")`

p-himik13:01:26

The code above seems to be trying to handle different types of `body` while also specifying that it can be only `String`. So either `^String` should be removed or the `(catch ClassCastException ...)`.

p-himik13:01:07

Which one - depends on how the `parse-json-response` function is used in the rest of the code. If its argument can be nothing but a string, then `^String` should be left in place.

Franklin13:01:24

cool, thanks

Franklin13:01:17

Why do you think https://github.com/jonase/kibit might give this error for a `.cljc` file?

``````Check failed -- skipping rest of file (src/milia/api/http.cljc:2:15)

Joshua Suskalo15:01:56

Has there been much consideration put towards making `clojure.lang.Numbers/ops` extensible at all? So that potentially users could define new numeric types, which could enable things like automatic differentiation and things like complex numbers to work with existing library code?

Joshua Suskalo15:01:58

Thanks. I was just curious 'cause I hadn't seen anything in http://ask.clojure.org about it, and it seems like it would be very useful in certain domains.

Alex Miller (Clojure team)15:01:18

there have been informal conversations about it over the years

Alex Miller (Clojure team)15:01:31

is there an example other than complex numbers?

Joshua Suskalo15:01:59

In my personal work I'd like to be able to make vectors use the clojure core ops for their operations, but that's because I do gamedev and that's niche.

Joshua Suskalo15:01:15

Automatic differentiation though is very useful in the ML space and is enabled by custom numeric types.

Joshua Suskalo15:01:55

It's how Julia does AD, and as a result Julia has some really good support for custom ML models with low effort.

Alex Miller (Clojure team)15:01:42

I'm not sure that there is really a big bar here that couldn't be solved with making your own functions

Alex Miller (Clojure team)15:01:33

and if there are perf concerns, you're probably ultimately going to want to be working outside the clojure framework for numeric ops

Joshua Suskalo15:01:51

The key usecase for AD would be to be able to use it to get derivative information on library code that you don't have source-level control over, so while yes "more functions" could do it, that increases maintenance burden and introduces more code.

Joshua Suskalo15:01:02

But yeah, for most people it probably wouldn't affect much.

Joshua Suskalo15:01:26

I was just curious if there had been thought put into it, and I'm happy to hear that there has been.

Sam Ritchie04:01:10

#sicmutils has all of this (AD, complex, quaternions, tons more) and does it with its own set of operators that shadow the core ones

Sam Ritchie04:01:01

it’s not that big of a deal to shadow +, -, etc; one nice trick is that you can overload all operations to handle symbolic arguments, and then “compile” numerical functions that need to be fast by • passing symbolic inputs • simplifying the resulting expression, extracting common subexpressions etc • wrap up in a `fn` and call `eval` on the result basically what Julia is doing, as a library here

Joshua Suskalo18:01:43

Right, and that's pretty nice, but the issue is that a math-based library can't use this unless you do like a `with-redefs` or they already use it

Sam Ritchie19:01:07

Yup, and with-redefs won't actually work because the math operators get inlined

Sam Ritchie19:01:35

I had forgotten that :) THAT is the big win, being able to differentiate through other people’s code etc

nonrecursive15:01:27

hey y’all, I’ve put together a component library that’s an alternative to integrant/mount/component/clip and was hoping some kind and curious souls might be interested in checking it out before I officially announce it? the repo is https://github.com/donut-power/system. I tried to write a good README to explain it. I created the channel #donut for anyone interested 🙂 any feedback would be very welcome

Ben Sless16:01:36

Asking the obvious question, why a 5th solution?

Joshua Suskalo16:01:18

Obviously not OP, but looking at it, it's the fact that there's no reliance on either global state or protocol extension. "it's just data"

Joshua Suskalo16:01:35

Along with user-defined lifecycle steps.

Ben Sless16:01:42

That's an interesting perspective from what appeals to you as a user

Ben Sless16:01:19

Have you ever found you need any other life cycle step other than start and stop?

Joshua Suskalo16:01:24

It enables architectures where the system isn't the "root" of the program, and when designing a program like that I think yes, other steps would be useful.

Ben Sless16:01:09

Could you elaborate on this perspective?

Joshua Suskalo16:01:09

Sure. For example you could design a socket server that has one component that's the socket, and then send a clocked signal to accept connections, adding all the connections as components into the system. Then on clock pulses the connections can register themselves as complete and get removed, and when you want to halt the program you can send a halt signal.

Joshua Suskalo16:01:13

Just off the top of my head.

Joshua Suskalo16:01:10

Also the fact that you could use a system map as a component instance itself enables you to use data-driven signal sending and handling between components, rather than needing a protocol with a fixed small set of acceptable signals.

Ben Sless16:01:22

Nothing stopping you from defining your own protocol just like your own signals, no?

Joshua Suskalo16:01:09

there is something stopping you from `assoc`ing in a new signal handler if you use protocols though

Ben Sless16:01:29

And with topological sorting I don't see the utility in nesting systems

Ben Sless16:01:43

You mean dynamically registering a new handler?

Joshua Suskalo16:01:56

Yeah, Which protocols can't really do

Joshua Suskalo16:01:23

besides just making a "signal" method on the protocol and implementing what donut.system is already doing yourself

Joshua Suskalo16:01:23

Basically as I look at this, this is a functional representation of Alan Kay's message-passing style OOP that predates what the term is now used for. Which I think there's some use for when composing systems.

nonrecursive16:01:14

I also wanted custom signals for e.g. `:heartbeat` signals or other signals that could be used for polling system health. in the past, with integrant, I also tried to create a custom step to do verbose validation before actually starting a system, and found that to be pretty awkward

nonrecursive16:01:54

This is helping me think of how I could come up with clearer motivating examples for donut.system's capabilities. All of them address real-world problems I've run into

nonrecursive16:01:14

it might not be obvious why component groups are nice, but I've found them very useful for a couple cases: • creating multiple instances of a component, where that component depends on a little sub-graph of components • creating a foundation for component libraries. I can create a set of related components that are able to use local references, publish that, and then you can just stick that in your own system map and it will work

nonrecursive16:01:21

really appreciate this feedback

Joshua Suskalo16:01:44

Yeah, being able to use it in libraries is something I'm kinda two ways about. On the one hand it's nice to not be limited and it's flexible, but on the other hand these sorts of systems are usually useful in applications, less in libraries, and if libraries start providing things in terms of components that forces you to choose this dependency injection library.

Joshua Suskalo16:01:19

it would be very useful in larger companies and with things like polylith though.

nonrecursive16:01:15

that's a really good point. I've tried to design it so that libraries can easily provide a donut.system component group, but they don't have to in order to work correctly. libraries wouldn't even need to require the donut.system library, they could just provide some component group map

Joshua Suskalo16:01:10

how would they do a cross reference though?

nonrecursive17:01:14

within a component group you can use `(ds/ref :component-name)` and that will resolve to the component named `:component-name` within the same group, so if your library has some set of related components they can use those local refs to refer to each other -does that answer your question? Or, does this example help? https://github.com/donut-power/system#groups-and-local-refs

Joshua Suskalo17:01:38

Right, my point was more that if you don't depend on donut.system, you can't do cross references.

nonrecursive17:01:37

oh right, shoot. I see what you mean. good point

nonrecursive17:01:05

maybe I can redesign that to use a map or vector for refs instead of a `donut.system/Ref`

1
Joshua Suskalo17:01:46

Definitely plausible, that could allow dependency-free components.

Joshua Suskalo17:01:55

It'd be important to document though.

nonrecursive17:01:59

this is great, thank you!

nonrecursive01:01:34

I’ve changed the ref implementation to use `[:donut.system/ref ref-key]` or `[:donut.system/group-ref group-name]` instead of a record 🙂

Max15:01:50

Is there a quick summary anywhere on the differences between donut and clip?

nonrecursive15:01:40

There isn't - iirc some of the differences include • donut has component groups • donut doesn't try to do any symbol resolution (I vaguely recall clip having some set of rules for this?) • donut has built in support for subsystems • donut doesn't yet support alternative execution, like async exec, like clip does. It should be fairly easy to add that though

🙏 1
Ben Sless20:01:03

bonus point: donuts are tasty

🍩 2
nonrecursive21:01:17

that’s how I get ya!

Joshua Suskalo15:01:33

I quite like the extensible signal system and the fact that it doesn't rely on any global state like multimethod definitions and similar.

Joshua Suskalo16:01:00

Something I recommend is that the special keys on components should be namespaced, so `::ds/depends-on`, `::ds/conf` . Then `:start` and `:stop` would be non-namespaced because they're signal handlers.

Joshua Suskalo16:01:16

That way you don't limit what signals the user could decide to send.

nonrecursive16:01:01

thanks for taking a look! and glad to hear you like it 🙂 integrant's been my favorite but the multimethod approach made it awkward to provide alternative implementations, like in tests. thanks for the suggestion - that makes sense

Joshua Suskalo16:01:56

Yeah, I agree, that's a problem I've seen to, that I normally handle with hierarchies, but I like this solution too. I want to take it out and kick the tires and see which one I like better.

Joshua Suskalo16:01:02

This looks really good though!

🎉 1
orestis17:01:08

If you want to write a defn-like macro, is there a standard way to support metadata, optional docstrings, pre/post conditions etc? Surely many people have tried that before. I guess I could use spec conform for that?

amithgeorge22:01:52

You might also like the approach here https://github.com/galdre/morphe

Joshua Suskalo17:01:30

I normally use spec conform for these, yes

escherize17:01:08

I extended a library to make it easier to handle defn forms: https://github.com/escherize/defrag It gives you access to the arg vector, arg values, the name of the function and the body.

👍 1
Joshua Suskalo18:01:22

I see in your example in the readme, is there a reason you're doing `(println (pr-str ...))` instead of just `(prn ...)`?

Alex Miller (Clojure team)18:01:42

usually to prevent interleaving of multithreaded output

1
vemv21:01:07

Does that mean that `prn` is racy?

Alex Miller (Clojure team)22:01:48

the output stream is a shared resource and each thread can interleave while writing

Alex Miller (Clojure team)22:01:15

pr-str makes a string, and println writes each "part" as a chunk, so it's a matter of whether you write N chunks or 1

vemv22:01:34

What is a "part" here?

Alex Miller (Clojure team)22:01:07

println is basically calling pr on each arg passed to println

vemv22:01:12

Got it yes, I was confused because the discussed readme has only 1 argument Personally I moved to prn after a few years of pr-str+println because I never use println with more than 1 argument (which comes naturally for me because I'm a heavy `->` er... normally I'll add `(doto prn)` in the middle of a chain) I still use pr-str in other places like logging though.

Joshua Suskalo16:01:28

ah, that makes sense, if you're passing more than one argument.