Fork me on GitHub
#babashka-sci-dev
<
2021-10-23
>
Eugen10:10:52

thanks @borkdude, I'll have a look at that as well

Eugen10:10:46

thanks @rahul080327 for the context, it's really helpfull - I haven't played with that but I will give it a try. I found https://ask.clojure.org/index.php/8990/how-to-do-a-http-request-over-a-unix-socket-in-clojure

borkdude10:10:26

Perhaps the answer has changed with Java 17

Eugen10:10:28

I found an http server over unix sockets in rust and I think I can use that to test the connection and make a piece of code work - or a reproducer

borkdude10:10:02

why not talk to docker directly?

Eugen10:10:57

I don't know the API and I want to have something simple

Eugen10:10:14

that I can also println debug

borkdude10:10:53

makes sense

lispyclouds10:10:59

as far as i understand java 16+ has made it possible to connect to a socket using the SocketChannel class and we need to do the write of the whole http payload onto the output stream it gives. not sure if theres any higher level abstractions than that in java stdlib. maybe if we have a thing to convert url and opt map to the http payload string, we can maybe attempt a self contained client.

lispyclouds10:10:37

> why not talk to docker directly? did you mean shelling out to it too? @borkdude

borkdude10:10:24

no, I meant instead of the http server example

borkdude10:10:13

it seems contaijners needs an abstraction layer that leaves the option to do the communication in different ways

borkdude10:10:26

or it needs a simplified java 17 fork without external deps

borkdude10:10:24

A pod in go might also work, since all the docker stuff is already in go and it would make it pretty light weight

lispyclouds10:10:27

yeah the pod "works" but has some streaming caveats. my hope is to make contajners src compatible with bb

borkdude10:10:38

yeah that would be optimal

borkdude10:10:02

you could however implement streaming using sockets in a pod as well

borkdude10:10:12

with some client side wrapping code

borkdude10:10:47

but it seems for most scripting cases you wouldn't need streaming, am I wrong?

lispyclouds10:10:01

problem with the go approach is that the go lib changes all the time and quite the coupled api with the server, quite a task to maintain. with the clj approach its simply adding the openapi yaml

lispyclouds10:10:36

the go lib too is essentially a glorified http client 😄

lispyclouds10:10:10

> but it seems for most scripting cases you wouldn't need streaming, am I wrong? yeah most of the cases should be covered

borkdude10:10:08

ok, to summarize all the options: 1. make contaijners source compatible with bb (either using babashka.curl or JDK 17 which needs a bb upgrade) 2. use nbb + dockerode or something 3. use the docker pod and extend with features if necessary. streaming can be implemented with some more work 4. Just use the JVM for these tasks 5. Use another language for these tasks

Eugen10:10:36

a good bunch of options

Eugen10:10:53

using nbb would mean to migrate the code I have to cljc / cljs

borkdude10:10:14

yeah this would be a different direction and having docker stuff for bb would be good

Eugen10:10:18

I'll first look at doing something in Java to have it in bb

Eugen10:10:29

so no tooling is changed

borkdude10:10:57

bb is not yet based on java 17 but this is possible

lispyclouds10:10:49

some heads up maybe, if somehow we need to extend some classes or something to make java's http client talk unix sockets, we may need some :gen-class stuff and not sure if that would be bb compatible. thats what we had to do with the OkHttp client

borkdude10:10:48

why do you need an http client at all if you communicate over domain sockets

lispyclouds10:10:08

docker is doing http over sockets

Eugen10:10:19

socket is binary stream, http is a protocol on top of that

borkdude10:10:39

does the JDK 11 client perhaps now support this in JDK 17?

borkdude10:10:12

it seems like babashka.curl isn't a bad idea then

lispyclouds10:10:21

i could not find a way to tell it to use a unix socket not a tcp one

Eugen10:10:25

did you ask nicely ? (joking 😄 )

lispyclouds10:10:20

if only there was sudo in java

borkdude10:10:50

#spire is also making a pod which supports streaming

lispyclouds10:10:05

i wanna side on the src compat, i would know most if not all things should work in a simpler way

Eugen10:10:18

I think we should also raise an issue to make the default http client work over unix sockets in jave

lispyclouds10:10:20

and adding bb.curl should not the too hard

borkdude10:10:36

@rahul080327 isn't the contaijners stuff async by default?

lispyclouds10:10:15

no, as the underlying OkHttp client has quite a different callback based async behaviour, never gotten around to implement that. however for streaming its easier to throw it in a loop in a future and read events

borkdude10:10:51

ok this will make things much easier then for bb compat

lispyclouds10:10:47

when and if project loom lands, no need for async 😛

lispyclouds10:10:02

yeah so this is where we would need the :gen-class to extend classes and supply to the java client

borkdude10:10:37

I think you could also use reify instead

borkdude10:10:06

But this is making things more complex anyways. I would say try the bb.curl approach and make reader conditionals

Eugen10:10:06

checked the imports and they use some dependencies - so adapting might require some work

borkdude10:10:44

isolate the http client namespace into one place, make it .cljc and use bb.curl in the :bb condition

lispyclouds10:10:25

yeah its the client and the ability to read pem files for which https://github.com/into-docker/pem-reader is used, which i think should be bb compatible? needs data.codec as the only dep. curl can handle pem files on its own and does a better job 😄

borkdude11:10:08

I haven't used data.codec with bb before, so that assumption needs testing :)

lispyclouds11:10:36

yeah if we are doing bb.curl this dep can be dropped too

lispyclouds11:10:22

that reminds me, mutual TLS is a bit tricky with the java client too, docker/podman uses it for authenticating to remote container engines

Eugen11:10:55

http client won't work unless it's fixed. It has :

static void checkURI(URI uri) {
        String scheme = uri.getScheme();
        if (scheme == null)
            throw newIAE("URI with undefined scheme");
        scheme = scheme.toLowerCase();
        if (!(scheme.equals("https") || scheme.equals("http"))) {
            throw newIAE("invalid URI scheme %s", scheme);
        }
        if (uri.getHost() == null) {
            throw newIAE("unsupported URI %s", uri);
        }
    }

borkdude11:10:51

Major facepalm

Eugen11:10:26

client was built when there where no unix sockets in JDK so this was not on the roadmap

lispyclouds11:10:45

ah interesting, i assumed i needed to inherit something like OkHttp but the issue seems to be much more basic, TIL

lispyclouds11:10:21

would it still be possible to pass a http://blah/<rest of the docker path> to it? this is essentially how curl works

Eugen11:10:38

that is the first blocker and I submitted a JDK bug and twitted about it - hopefully someone sees it 🙂

Eugen11:10:08

I did not try it further

lispyclouds11:10:17

nice, i think if we go ahead with the bb.curl approach as a starter and if and when we have native support we can deprecate it

Eugen11:10:17

perhaps as the the client might take a long time to get fixed if the fix is more complicated

Eugen11:10:47

if the fix is as simple as fixing the checkURI then it might get backported to 16

borkdude12:10:54

you can try to override this class and compile it locally with a change ;)