Fork me on GitHub
#clojure
<
2022-02-04
>
Joshua Suskalo00:02:39

What's the state of the art in http server libraries atm? I'm using reitit for routing, I'll probably be using sieppari for interceptors, and I'm curious what the modern recommendation is for servers. Ring is the old stand-by, but it focuses mostly on middleware and while that can be adapted for interceptors that's still stuff I'd have to do for it and it wouldn't likely connect well with the existing ring libraries for auth, sessions, and other stuff. There's http-kit which is pretty cool but the site and docs are unmaintained for the most part (despite the actual lib still seeing updates) and has some interesting features about which I know very little, like websocket promotion.

hiredman00:02:08

ring isn't an http server, it is an interface that defines http request responses in terms of clojure function calls. Often when people speak about ring in terms of http servers they are refering to ring-jetty-adapter, but there are are adapters for a number of http servers

Joshua Suskalo00:02:36

right, I suppose I haven't dived into that ecosystem enough to remember that off the top of my head. Thanks for the clarification!

hiredman00:02:54

jetty can also be used via the java servlet api, or directly via its api

hiredman00:02:10

and supports things not readily exposed via ring (like websockets)

Joshua Suskalo00:02:50

right, and so does http-kit, with a similar api to ring

hiredman00:02:38

as to state of the art, interceptors are maybe still that, but they have been around a while and haven't really unseated ring as the go to

hiredman00:02:05

at work we are still mostly ring, compojure, and jetty

hiredman00:02:21

we have two services that use bidi for routing (which predates reitit, but provides some features I really like over compojure like reverse routing), and one service that uses a netty based http server

hiredman00:02:07

we've experimented with using http-kit over jetty, but didn't get as good newlic metrics reporting

Joshua Suskalo00:02:13

I have seen reverse routing as a feature, but I'll admit I have yet to get when it would be useful.

hiredman00:02:08

avoids having to hard code paths for links, form actions, etc

hiredman00:02:33

so if I change the path for a handler, all the links to it still work

hiredman00:02:09

which I think is great, but yeah, a lot of people are not blown away by

Darin Douglass00:02:43

reverse-routing++ ❤️

hiredman00:02:21

the bidi stuff does bite us from time to time because everything else is ring/compojure so sometimes things need to be worked around for one reason or another

Darin Douglass00:02:28

(reitit has the same thing and will build the url for you from params)

ghadi00:02:34

we use interceptors at nubank for not only web servers, but Kafka message consumption and production

ghadi00:02:13

I find an assembly line / bulletin board style of work separation easier to deal with than function composition (ring)

Ben Sless06:02:05

Interceptors are clearly an easier mental model The downsides are performance, if you care about maximizing latency throughput or minimizing response times. They're still an excellent fit 99% of time, although they can fall into the trap mentioned in out of the tarpit of passing a bit ol' context map which contains too much

jumar08:02:44

“Maximizing latency”? You meant throughput?

Joshua Suskalo00:02:33

that seems pretty cool

ghadi00:02:18

interceptors handle request coercion, serialization, validation, auth, signing/validation, all sorts of things

JohnJ00:02:45

via pedestal?

ghadi00:02:01

yeah, the interceptor machinery is separated from the web stuff

JohnJ00:02:01

ah cool, I guess nubank goes full async everything with pedestal?

JohnJ01:02:32

almost 😉 - skimming through pedestal docs looks like the interceptor model is just better when you need async

phill10:02:51

Here are some non-async things I like about Pedestal's interceptors. Interceptor chains assemble as data, not code. Interceptors' error handling is more purposeful. The context map, distinct from the Request (where the same stuff winds up in Ring), lets my program complain about the Request without including the sausage-making ingredients.

JohnJ15:02:05

is non-async awkward in pedestal?

ghadi15:02:48

not at all

ghadi15:02:03

what gives you that impression?

JohnJ16:02:19

ah just curious, wanted to clarify that point before diving deep in since most seems to doing async stuff with it, thx

ghadi16:02:57

I have the opposite impression

ghadi16:02:26

I think people generally stay sync except for very very small pockets

flefik00:02:32

I’m struggling to get the -P switch to work (to cache dependencies in CI) My makefile contains:

deps:
	clojure -P -J-Xmx1024m -X:jar :jar target/app.jar
jar:
	clojure -J-Xmx1024m -X:jar :jar target/app.jar
And my dockerfile:
FROM openjdk:12-jdk-alpine AS build
RUN apk add bash curl build-base
RUN curl -O 
RUN chmod +x linux-install-1.10.1.478.sh
RUN ./linux-install-1.10.1.478.sh
RUN apk add bash \
    gcompat \
    libsass
RUN adduser -S app app

WORKDIR /app
COPY Makefile deps.edn ./
RUN chown -R app /app
USER app
RUN make deps
USER root
COPY entrypoint.sh /app/
COPY resources /app/resources/
COPY dev /app/dev/
COPY src /app/src/
RUN chown -R app /app
RUN chmod +x /app/entrypoint.sh
RUN make jar
USER app

EXPOSE 3000
CMD ["/app/entrypoint.sh"]
As you can see make deps is ran, and a couple of lines down I also run make jar. Yet, RUN make jar will still re-download all the dependencies. Why isn’t -P respected?

hiredman00:02:57

different users have different ~/.m2 directories

flefik00:02:59

oh dear you’re right, i’m running one as root and the other as app that’s not it, even if I put them right next to each other, make jar will re-download the deps

flefik00:02:52

USER app
RUN make deps #bump
RUN make jar

EXPOSE 3000
CMD ["/app/entrypoint.sh"]
^ has the same issue

hiredman00:02:04

I would replace the calls to make file with the direct calls to the clojure cli for debugging purposes

hiredman01:02:16

if it is actually downloading the same stuff again that is very odd, I would start using ls to see if after make deps runs app has an ~/.m2 and it has things in it, is there a .cpcache directory, etc

pinkfrog11:02:11

HI. I wonder why we have a puts queue for async channel? https://github.com/clojure/core.async/blob/6ac8ed2a26c67bbc3cc43e4e650f97c984666b0c/src/main/clojure/clojure/core/async/impl/channels.clj#L158. Why don’t we directly rely on the channel buffer?

Ben Sless11:02:35

What if you have an unbuffered channel? Have you watched the video which goes a bit into the implementation of channels?

pinkfrog12:02:00

> What if you have an unbuffered channel? Every queue is buffered, why not make the things explicit?

pinkfrog12:02:47

Having a put! only makes things complicated as clojure will from time to time throw exceptions when the puts queue reaches its max size.

pinkfrog12:02:22

I will watch the video first.

Ben Sless12:02:30

You're right that the implementation is leaky

Ben Sless12:02:38

But if channels actually register callbacks behind the scenes you need a queue for that

Ben Sless12:02:55

Since you have putters and takers, why not have a queue for each?

Timur Latypoff12:02:36

I also remember something about interaction of transducers and async chans. Since transducers may produce multiple results per step, there was a semantic mismatch between limited-buffer channels and transducer logic — so maybe this has something to do with handling this mismatch gracefully?

Ben Sless13:02:08

iirc the way it's handled is with Channels' internal buffers being actually unbounded, and while external "puts" are rejected, they can "overflow" from transducers producing multiple values.

👍 1
Fahd El Mazouni11:02:08

Hello ! is there a "better" way to check if a namespaced keyword is from a specific namespace other than (= (namespace said-keyword) "namespace-name") ? Thanks in advance !

p-himik13:02:37

Pretty sure that's the way to do it.

walterl14:02:34

@U9W44J4RW I'm curious; what do you find lacking in that approach, that you think could be done better?

Fahd El Mazouni14:02:53

I guess I expected there to be a (namespaced? ns-symbol key)

Fahd El Mazouni14:02:55

thanks guys, I suppose my eyesight isn't that bad

roklenarcic15:02:33

I am using a library that returns values in clj but in cljs it returns a chan that will eventually have that value. This presents a problem that processing of that value must now have split logic for channel vs non-channel value. Is there some sort of small utility library that would smooth over that?

p-himik15:02:04

Wouldn't always putting the value in the CLJ case into a channel and then work with a channel in both CLJ and CLJS solve the problem? I doubt adding a whole new library to your project just because of this small thing would be worth it.

p-himik15:02:31

I meant that you can create a new one-off channel on your own, to mimic what's going on in CLJS.

p-himik15:02:53

And if the problem arises from calling table->rows directly then it seems like you could just call kv-konserve/table->rows instead on the CLJ side of things.

roklenarcic16:02:43

I’ll manage something I think, thanks

roklenarcic15:02:51

I want to put into spec that something is a core.async channel, but I cannot find a predicate for that… am I missing something?

Joshua Suskalo15:02:21

spec does not have a function for that, and indeed core.async itself doesn't have a predicate for channels

Joshua Suskalo15:02:36

the best you can get from what I've seen is satisfies? with ReadPort and WritePort

kingcode16:02:32

For some reason, the following compiles and runs fine:

(let [{:keys [a b c] 
       :or {a 1 b 2 
            c (for [i (range a)
                    j (range b)]
                [i j])}} nil]
   [a b c])
;;=> [1 2 ([0 0] [0 1])]
But the following doesn’t:
(defn transform2 
  [{:keys [algo opts] frame :hydrated fill :compressed}]
  (let [{:keys [rcs rsiz csiz] 
         :or {rsiz (count frame)
              csiz (-> frame first count)
              rcs (for [i (range rsiz)
                        j (range csiz)] [i j])}} opts]))
;;=> ... Unable to resolve symbol: csiz in this context

What am I missing? Thanks for any comment..

thumbnail17:02:56

Your first example isn't considered desired/stable behavior either iirc (using restructured keys in a default that is)

kingcode17:02:34

OK Thanks 🙂

Alex Miller (Clojure team)17:02:52

maps are unordered, :or exprs are run in arbitrary order, and we make no promises about when those evaled in combination with let bound keys

kingcode17:02:22

Makes sense - indeed. Thank you!

Alex Miller (Clojure team)17:02:34

:or is best for default values, not computations

👍 1
afleck20:02:23

can i rely on the observation that (into [] (apply sorted-set (reverse (range 10000))) produces a sorted vector?

afleck20:02:31

in what cases could it be not reliable?

afleck20:02:31

you’re saying it doesn’t hold generally for (into [] (sorted-set-by … ))?

Jan K21:02:48

into [] will keep the order of the source collection

☝️ 1
afleck21:02:09

ok, thank you

clj.max21:02:59

Generic Q as somebody who doesn't always follow the Clojure world closely: have there been any announcements/roadmaps about where Clojure is headed now w/ NuBank in play? How has the strategy/roadmap changed? That kind of thing. Thanks.

Alex Miller (Clojure team)21:02:38

This statement from Ed when Cognitect joined Nubank is a great overview of intent and (18 months later) continues to be pretty close match https://building.nubank.com.br/welcoming-cognitect-nubank/

thanks3 1
Alex Miller (Clojure team)22:02:18

we don't really do roadmaps per se, but when we talk about problems we want to solve they are not substantially different now than they were pre-Nubank. they are influenced and informed by being in colocation with the largest Clojure/Datomic user base) but not sure that's really changed the trajectory too much.

Alex Miller (Clojure team)22:02:06

(except all the Nubank numbers there seem small now :)