Clojurians Log v2

thanks @borkdude, I'll have a look at that as well

thanks @rahul080327 for the context, it's really helpfull - I haven't played with that but I will give it a try. I found https://ask.clojure.org/index.php/8990/how-to-do-a-http-request-over-a-unix-socket-in-clojure

Perhaps the answer has changed with Java 17

I found an http server over unix sockets in rust and I think I can use that to test the connection and make a piece of code work - or a reproducer

why not talk to docker directly?

I don't know the API and I want to have something simple

that I can also println debug

makes sense

as far as i understand java 16+ has made it possible to connect to a socket using the SocketChannel class and we need to do the write of the whole http payload onto the output stream it gives. not sure if theres any higher level abstractions than that in java stdlib. maybe if we have a thing to convert url and opt map to the http payload string, we can maybe attempt a self contained client.

> why not talk to docker directly? did you mean shelling out to it too? @borkdude

no, I meant instead of the http server example

it seems contaijners needs an abstraction layer that leaves the option to do the communication in different ways

or it needs a simplified java 17 fork without external deps

I forgot to mention @eugen.stan, @rahul080327 has also made this thing: https://github.com/lispyclouds/pod-lispyclouds-docker ;)

> it seems contaijners needs an abstraction layer yeah thats what i was thinking of, pass a :transport key in the client opt map and pass :http-client or :curl

A pod in go might also work, since all the docker stuff is already in go and it would make it pretty light weight

yeah the pod "works" but has some streaming caveats. my hope is to make contajners src compatible with bb

yeah that would be optimal

you could however implement streaming using sockets in a pod as well

with some client side wrapping code

but it seems for most scripting cases you wouldn't need streaming, am I wrong?

problem with the go approach is that the go lib changes all the time and quite the coupled api with the server, quite a task to maintain. with the clj approach its simply adding the openapi yaml

the go lib too is essentially a glorified http client 😄

> but it seems for most scripting cases you wouldn't need streaming, am I wrong? yeah most of the cases should be covered

:)

ok, to summarize all the options: 1. make contaijners source compatible with bb (either using babashka.curl or JDK 17 which needs a bb upgrade) 2. use nbb + dockerode or something 3. use the docker pod and extend with features if necessary. streaming can be implemented with some more work 4. Just use the JVM for these tasks 5. Use another language for these tasks

a good bunch of options

using nbb would mean to migrate the code I have to cljc / cljs

yeah this would be a different direction and having docker stuff for bb would be good

I'll first look at doing something in Java to have it in bb

so no tooling is changed

bb is not yet based on java 17 but this is possible

some heads up maybe, if somehow we need to extend some classes or something to make java's http client talk unix sockets, we may need some :gen-class stuff and not sure if that would be bb compatible. thats what we had to do with the OkHttp client

an example: https://github.com/into-docker/unixsocket-http/blob/main/src/unixsocket_http/impl/FixedPathTcpSocket.clj

why do you need an http client at all if you communicate over domain sockets

docker is doing http over sockets

socket is binary stream, http is a protocol on top of that

does the JDK 11 client perhaps now support this in JDK 17?

afaik, no

it seems like babashka.curl isn't a bad idea then

for bb

i could not find a way to tell it to use a unix socket not a tcp one

did you ask nicely ? (joking 😄 )

or the pod

#spire is also making a pod which supports streaming

i wanna side on the src compat, i would know most if not all things should work in a simpler way

I think we should also raise an issue to make the default http client work over unix sockets in jave

and adding bb.curl should not the too hard

@rahul080327 isn't the contaijners stuff async by default?

no, as the underlying OkHttp client has quite a different callback based async behaviour, never gotten around to implement that. however for streaming its easier to throw it in a loop in a future and read events

ok this will make things much easier then for bb compat

example: https://github.com/lispyclouds/contajners/blob/main/doc/002-general-guidelines.md#streaming-logs

when and if project loom lands, no need for async 😛

I found this Java implementation https://github.com/gesellix/unix-socket-factory that might work to convert to clojure See https://github.com/kohlschutter/junixsocket/issues/79 and https://github.com/docker-client/docker-filesocket/blob/a03c2238fa40f20be86fac72e6edf668a314b8c4/src/main/java/de/gesellix/docker/client/filesocket/UnixSocket.java#L36

yeah so this is where we would need the :gen-class to extend classes and supply to the java client

I think you could also use reify instead

But this is making things more complex anyways. I would say try the bb.curl approach and make reader conditionals

checked the imports and they use some dependencies - so adapting might require some work

isolate the http client namespace into one place, make it .cljc and use bb.curl in the :bb condition

yeah its the client and the ability to read pem files for which https://github.com/into-docker/pem-reader is used, which i think should be bb compatible? needs data.codec as the only dep. curl can handle pem files on its own and does a better job 😄

I haven't used data.codec with bb before, so that assumption needs testing :)

yeah if we are doing bb.curl this dep can be dropped too

that reminds me, mutual TLS is a bit tricky with the java client too, docker/podman uses it for authenticating to remote container engines

http client won't work unless it's fixed. It has :

static void checkURI(URI uri) {
        String scheme = uri.getScheme();
        if (scheme == null)
            throw newIAE("URI with undefined scheme");
        scheme = scheme.toLowerCase();
        if (!(scheme.equals("https") || scheme.equals("http"))) {
            throw newIAE("invalid URI scheme %s", scheme);
        }
        if (uri.getHost() == null) {
            throw newIAE("unsupported URI %s", uri);
        }
    }

Major facepalm

client was built when there where no unix sockets in JDK so this was not on the roadmap

ah interesting, i assumed i needed to inherit something like OkHttp but the issue seems to be much more basic, TIL

would it still be possible to pass a http://blah/<rest of the docker path> to it? this is essentially how curl works

that is the first blocker and I submitted a JDK bug and twitted about it - hopefully someone sees it 🙂

https://twitter.com/ieugen222/status/1451877192766574592

I did not try it further

nice, i think if we go ahead with the bb.curl approach as a starter and if and when we have native support we can deprecate it

perhaps as the the client might take a long time to get fixed if the fix is more complicated

if the fix is as simple as fixing the checkURI then it might get backported to 16

you can try to override this class and compile it locally with a change ;)