Fork me on GitHub
#clojure
<
2020-06-27
>
emccue02:06:00

Or even just java parallel streams

codonnell12:06:29

I'm trying to use depot, but it only reports outdated dependencies in my main :deps; it doesn't seem to check dependencies in aliases even when I add those aliases at the CLI. Is this just me? If not, have others found a way around this?

p-himik12:06:55

> even when I add those aliases at the CLI How exactly do you add those aliases?

codonnell12:06:15

For example: clojure -A:outdated:backend

codonnell13:06:24

Here's a minimal deps edn for the above:

{:deps {}
 :aliases {:backend {:extra-deps {org.postgresql/postgresql {:mvn/version "42.2.12"}}}
           :outdated  {:extra-deps {olical/depot {:mvn/version "1.8.4"}}
                       :main-opts  ["-m" "depot.outdated.main"]}}}

p-himik13:06:29

With depot, you have to use -a for the aliases that your project uses. Run clj -A:outdated --help for more details.

pmonks16:06:57

You may already be aware of this, but it bit me in coming over from lein-ancient: depot only checks your explicitly declared dependencies, but does not check transitive dependencies.

seancorfield18:06:19

This is why deps.edn-based tooling should use t.d.a. to handle the dependencies instead of trying to read the EDN files directly. You get confusing semantics.

seancorfield18:06:52

We gave up on Depot at 2.0 when they changed the semantics (from t.d.a. to reading EDN files) and we wrote a script based on clojure -Stree to do arbitrary depth outdated deps analysis.

seancorfield18:06:15

It's ugly but it has the "correct" semantics and it can check any path through your classpath building alias combinations.

p-himik19:06:49

@seancorfield I couldn't find anything quickly - why did they decide to read EDN files instead of using t.d.a?

seancorfield19:06:55

The original Depot (up to 1.8.4) used t.d.a. for the analysis but the update/rewrite code had to read the EDN file anyway and it had different semantics (first bad idea) then in 2.0 the analysis was rewritten to use the same EDN processing as the update/rewrite code (second bad idea).

pmonks04:06:45

Sorry for coming back in so late, but by any chance is your script available somewhere public, @seancorfield?

seancorfield04:06:46

Haha... no, I'm somewhat ashamed of it -- it's a pretty serious hack! I may try to turn it into a releasable form at some point but it's low on my priority list.

๐Ÿ’ฏ 1
seancorfield04:06:26

It's about 30 lines of fgrep and sed ๐Ÿ™‚

seancorfield04:06:13

The TL;DR is that it runs clojure -Stree (with whatever aliases you want) and then pulls all the top-level dependencies from that output. Then it creates a new temp project and builds a deps.edn file with all those dependencies with versions replaced with {:mvn/version "RELEASE"}, and then it runs clojure -Stree on that project and then sorts and diffs the output of the two clojure commands, and formats that into a nice table.

๐Ÿ‘ 1
seancorfield19:06:30

Question about clojure.core.reducers: that namespace has several ForkJoin helpers declared, most of which are private https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj#L20-L34 but strangely both pool and fttask are public, although they are not part of the API documentation https://clojure.github.io/clojure/clojure.core-api.html#clojure.core.reducers -- If a library wants to implement CollFold, it's likely to need the equivalent of these helpers, and it seems like it should also use the same pool so I'm wondering how folks are going about this?

seancorfield19:06:18

I'm working on next.jdbc and making plan support CollFold. The result of execute! is already foldable because it's "just" a vector and c.c.r already has a fork-join implementation for vectors (`foldvec`). Since you need to use plan if you want to stream result sets from a database and process them on-the-fly, I want that to be foldable so you can reduce in parallel to improve the performance of processing very large streamed result sets. I have a working implementation but I found myself copying in some of those private helpers as they clean up the code substantially and I am relying on clojure.core.reducers/pool so that my CollFold implementation shares compute resources rather than competes with the reducers library.

seancorfield19:06:14

How many folks are using the reducers stuff and, in particular, the parallel fold stuff? Has anyone implemented CollFold?

dpsutton20:06:48

http://grep.app is really nice for searching public github repos for usages

๐Ÿคฏ 2
seancorfield21:06:11

This doesn't belong in this channel. #community-development is for discussions about Slack alternatives etc. @U0W13P4LX

seancorfield21:06:56

Feel free to post it in that channel if you want. I've deleted it from here.

borkdude20:06:35

@seancorfield Ghadi recently placed this comment on Reddit: > Besides the idea of fold, I would not use reducers anymore. Transducers are a much better construct/design, fully decoupled from collection. https://www.reddit.com/r/Clojure/comments/hf772r/reducers_in_clojure/fvzqjye?utm_source=share&amp;utm_medium=web2x

seancorfield20:06:51

Yeah, I had a feeling that I'd seen something around that @borkdude Thanks for the link. @dpsutton https://grep.app/search?q=%23%27r/fj shows that quite a few libraries out there are relying on the internals of clojure.core.reducers at this point ๐Ÿ˜

seancorfield21:06:05

Interesting that @ghadi calls out fold as being still worthwhile. I wonder if we'll see an official parallel fold implemented over transducers...

alexmiller21:06:29

We talked about it when transducers went in

alexmiller21:06:05

That tesseract library has an impl of this I think?

alexmiller21:06:23

But maybe I misremember

seancorfield21:06:33

"Tesser folds do not preserve the order of inputs" ๐Ÿ˜ž

seancorfield21:06:46

Tesser is way more complex than I'd want to use, when there's clojure.core.reducers available today. If I wanted to easily managed Hadoop jobs etc, yup, Tesser looks great.

datran23:06:46

what web servers support websockets? I've been using immutant for a long time, but I've been hearing that its basically un-maintained nowadays

phronmophobic23:06:06

I've been using http-kit, https://github.com/http-kit/http-kit. I think it's also looking for maintainers, but I haven't had any issues with it and I'm not sure what new features I would be interested in

datran23:06:36

Yeah, I'm still pretty happy with Immutant, so I doubt I'll be changing code that isn't broken. But I am curious what I should be using for new projects. I'll look at http-kit, thanks.

seancorfield23:06:30

Jetty supports WebSockets according to its docs.

seancorfield23:06:17

At work we're using Netty and a Java-based http://Socket.IO library via interop but that's a fair bit of heavy lifting.

seancorfield23:06:12

If you're using a http://Socket.IO client, I don't know how easy it is to use anything except that server-side library so it'll depend on what you need WebSockets for @U0LE4C7CK

hiredman00:06:03

https://gist.github.com/hiredman/9a086f7b34f8d4fc3f95a251a616bafd is a toy nio server that speaks just enough http to upgrade to websockets (and doesn't support all of the websockets rfc)

hlship01:06:14

The subscription support in Lacinia is based on Pedestal (Jetty) and web sockets. So Pedestal does a good job, AFAIK.

hlship01:06:27

I only wrote it, I donโ€™t get to use it (subscriptions in Lacinia).

datran01:06:49

Ok, I looked at Clojure Web Servers blog post by http://purelyfunctional.tv[0] and it looks like there aren't a ton of choices if you want websockets. I'm thinking the ring-jetty9-adapter[1] looks promising, I don't know if I'm ready to go into pedestal-land. [0]https://purelyfunctional.tv/mini-guide/clojure-web-servers/

datran01:06:16

I am looking forward to see how https://github.com/metosin/pohjavirta progresses, too. I like ikitommi's focus on performance. That and those metosin guys keep coming up with brilliant stuff

โž• 3
borkdude08:06:48

Aleph also supports it. I'm using it from yada.

Linus Ericsson22:06:31

Jetty supports websockets decently. Jetty is my goto-choice because it has like all the features. Of course there are alternatives, but HTTP is sort of complicated these days.