You can definitely do this, because you can write spark jobs from clojure

I do get what you’re asking! Storm was clojure to start and may still be clojure

https://storm.apache.org/

yes it makes me sad, that now became java alot, i read that they re-wrote Clojure code in Java

and i was wondering if this is for perfomance reasons, or just because more people know Java

Whenever I’ve seen these rewrites happen the latter is definitely a big factor, and the folks doing the rewrite always talk about the former. In theory the performance can be the same…

But writing in Java definitely makes it easier to consume from scala, clojure etc imo

clojure is really fast! It’s not like python where it’s terribly slow unless you call out to C, Fortran etc (that’s my understanding, could be wrong)

thank you for the information, i hope sometime, we have a Clojure project that uses all the benefits of Clojure to make it more popular. But ok all functional languages for now are not popular.

The vast majority of people don’t know anything about any language, most of the pie is still up for grabs!

I thought they mostly rewrote Spark/Storm/whatever in Java to allow for a larger community of contributors, since there are so many more Java developers out there than Clojure developers?

Dev „availability“ is the misconception of those rewrites. In my experience, performance is no issue (btw: Never was with sparkling, the Clojure/Spark lib). Quite the opposite: I could squeeze more performance out oder the system built on sparkling, because I was faster achieving the goal and then use time to find the performance bottlenecks (which usually were rooted in me not leveraging data characteristics right from the start).

In my experience Clojure programs can get the job done faster than Java programs (given that you will put a limited amount of energy into programming) because you, the programmer, "work smarter, not harder"... the program spends no time making "safe copies" of structures... can use zippers instead of xslt... and you spend less time debugging and more time at a higher level.

But random anecdotes should be paired with a citation to Rich Hickey's comment that attaining "on-the-metal" speed was a guiding principle.

See this: https://clojure.atlassian.net/browse/CLJ-2647

I don't see anything suspicious about it.

The message is in stderr, shouldn't it be in stdout since there's no actual error?

stderr is not about errors per se, it's more about diagnostic output.

Alright, so this is something git does and clojure.java.shell doesn't change anything on top of it?

Yeah.

Looks like everything is working as intended

I'm no expert here, but I suspect that if the secondary output stream was referred to as 'altout' instead of 'stderr' then a great many people would be less confused by its purpose.

Yeah, I think this is all that's said about stderr in the POSIX standard: > At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output).

What diagnostic means is an exercise left to the reader

Yeah, exactly Thank you all, TIL

reduce is preferred. the implementation of apply just calls reduce

reduce therefore is the same semantically and provides better performance

this is wrong

oh? I'm curious about that.

Rich has mentioned that the reduce arity without an 'init' argument is regretful

Yeah, in this case it will provide equivalent behavior because + will never be called with 1 argument in the reduce case, but with 0 elements it will be called with 0 arguments and produce 0, the same as apply, and with 2 or more elements it will correctly sum them.

(apply + ints) or (reduce + 0 ints) would be my preference, in that order

what's the reason to prefer apply when the high-arity version of + just calls out to reduce?

not because of performance, but because it's closer to what your intent is

one op, many args

I suppose?

(reduce + 0 ints) is describing a particular strategy for adding (sequential)

apply doesn't appear to call out to reduce but perhaps I'm misunderstanding https://github.com/clojure/clojure/blob/clojure-1.10.1/src/clj/clojure/core.clj#L660

+ with varargs calls a reduce internally

apply itself does not, but the high-arity version of + which is what apply will call does

(but doesn't have to!)

oh I see

also the non-init case of reduce is just fine, the problem is that it's not documented it requires that the values contained in the list and the function must together produce a monoid. It might make more sense for it to exist under its own name, like mfold or something, but it has every reason to exist.

or rather the documentation expresses that constraint, just not in a way that makes it clear that's what it requires.

no it's not 'just fine'

> Who knows what the semantics of reduce are when you call it with a collection and no initial value? No one, right. No one knows. It's a ridiculous, complex rule. It's one of the worst things I ever copied from Common List was definitely the semantics of reduce. It's very complex. If there's nothing, it does one thing. If there's one thing, it does a different thing. If there's more than one thing, it does another thing. It's much more straightforward to have it be monoidal and just use f to create the initial value. That's what transduce does, so transduce says, "If you don't supply me any information, f with no arguments better give me an initial value." from https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/InsideTransducers.md

(I recommend we move this part of the convo to a different thread)

So yeah, back to original question: (apply + lst) matches the semantic the best (reduce + lst) happens to match the semantic because numbers and + form a monoid and it will be more performant (on the current implementation which is technically subject to change but likely not to in the near future) (reduce + 0 lst) matches the semantic, requires no obscure maths knowledge for it to make sense, and will likely be the most performant of the three (with the same caveat)

Thanks for that quote-with-link, Ghadi. TIL.

So I agree that the way the documentation for reduce is worded it's confusing, and I've seen what Rich says about it before, and on some level I agree that coming at it from a strict values perspective it is weird, but it is just a monoid. It's requiring that the list has monoidal values in it, and it has join on the 2-arity of the function and zero on the zero-arity, which results in a pure monoid.

It's a sensible semantic, but confusing compared to the general version of reduce and therefore it would perhaps be a better idea to have it be under a different name if it could be changed.

This is why next.jdbc only implements IReduceInit for plan: https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc.clj#L225 -- there is no sane default init value for most result set reducing functions.

Yes, precisely because it doesn't construct a monoid with sensible reducing functions, and that's why reduce is a poor name for the monoidal fold operation.

And I applaud that decision to only implement ireduceinit

I think this cake is baked