Fork me on GitHub
#aleph
<
2022-03-28
>
skynet03:03:59

anyone know if it's possible to use Aleph TCP in a graal native image? it seems to pull in all too much of netty when requiring the aleph.tcp namespace

Matthew Davidson (kingmob)14:03:13

Aleph is built on Netty, so I would expect it to pull in a lot of it. In particular, it requires

(def netty-modules
  '[transport
    transport-native-epoll
    codec
    codec-http
    handler
    handler-proxy
    resolver
    resolver-dns])
which results in the following dep tree:
[io.netty/netty-codec-http "4.1.65.Final"]
 [io.netty/netty-codec "4.1.65.Final"]
 [io.netty/netty-handler-proxy "4.1.65.Final"]
   [io.netty/netty-codec-socks "4.1.65.Final"]
 [io.netty/netty-handler "4.1.65.Final"]
 [io.netty/netty-resolver-dns "4.1.65.Final"]
   [io.netty/netty-codec-dns "4.1.65.Final"]
 [io.netty/netty-resolver "4.1.65.Final"]
 [io.netty/netty-transport-native-epoll "4.1.65.Final"]
   [io.netty/netty-transport-native-unix-common "4.1.65.Final"]
 [io.netty/netty-transport "4.1.65.Final"]
   [io.netty/netty-buffer "4.1.65.Final"]
   [io.netty/netty-common "4.1.65.Final"]

Matthew Davidson (kingmob)14:03:05

You may have some issues building Aleph with Graal. A few people have asked over the last couple years, but Aleph was never built with Graal’s requirements in mind. There’s issues with top-level single namespaces in deps, statically-initialized threads, and probably some others. If you want to try it and catalog the issues you find, we could use the assistance.

👍 1
skynet14:03:09

@U10EC98F5 thanks for the reply, I guess I'm not sure why the aleph.tcp namespace causes netty classes like io.netty.channel.epoll.Native to be initialized, while aleph.http does not and can be built fine here https://github.com/clj-easy/graalvm-clojure/tree/master/aleph I'm starting to dig into the aleph.netty namespace and I think it might be due to some macro in there that tcp uses

Matthew Davidson (kingmob)14:03:03

I’m not familiar with clj-easy, although I know @UM4NVAD62 has been working to get Aleph working with it. It’s another layer on top of Graal, and has its own issues (namely, it can’t handle single-segment namespaces, which some of Zach’s libs use since they were written at the dawn of time). At a low level, the use of epoll shouldn’t have anything to do with your protocol of interest (e.g., http vs tcp). It’s supposed to be an optimization for Linux. That being said, I haven’t looked into that stuff at all since becoming maintainer. WHy do you want/need to not load io.netty.channel.epoll.Native?

skynet15:03:19

@U10EC98F5 so that specific clj-easy repo graalvm-clojure has just small example projects showing basic graalvm native image compatibility for various clojure libs. you can see that it needs to tell graal to initialize various netty classes at runtime in order to work https://github.com/clj-easy/graalvm-clojure/blob/master/aleph/project.clj#L32 I believe you're thinking of this repo https://github.com/clj-easy/graal-build-time#graal-build-time which does indeed have the single-segment ns issue you mentioned I see people claim that they have success building at least parts of netty with graal native https://github.com/netty/netty/issues/10616#issuecomment-997282655 so that makes me think it might be possible to get a netty tcp client in a native image. but something in either Aleph or the specific netty classes that it uses is causing these classes to be initialized at build time, which breaks the native-image build for me

skynet20:03:27

making some progress, got an image to build, and am now running into reflection issues in gloss. I see someone else did some work on that https://clojurians.slack.com/archives/C0G922PCH/p1641253399033500 might try starting there and adding more type hints that I need

skynet00:03:22

@U10EC98F5 fixed my issue with Gloss, needed to replace the match-loop fn with a version that doesn't use eval https://github.com/skynet-gh/gloss/commit/d3db3b9d7f5a39d1ae504b0c2adae1d43f18b1f0

Matthew Davidson (kingmob)01:03:25

The original gloss is archived, but a few people have had problems, and it's available in clj-commons now, which we can edit. Maybe make an issue for it?

skynet01:03:50

perfect, I'll do that. thanks for managing these old libraries.

skynet01:03:02

what are your plans for the libraries with single segment namespaces like byte-streams ?

Matthew Davidson (kingmob)03:03:23

@UM4NVAD62 was doing some work on it, but I haven’t heard back from him in a while. Without him, I’ll get around to it when I get to it. But for both byte-streams and primitive-math, the plan is to copy the code to a multi-segment ns, and leave the old code in place. For byte-streams, I’ll update manifold/aleph to use the multi-segment ns, which should alleviate those clj-easy issues.

👍 1
skynet03:03:48

that makes sense, thanks! if I get impatient enough I might send a PR to add the multi-segment version of byte-streams, since it causes a pain in graal

Matthew Davidson (kingmob)04:03:30

@U0DSU64Q1 Great! Here’s what I sent Piotr about his changes (in https://github.com/piotr-yuxuan/byte-streams) for primitive-math and byte-streams. The tl;dr is we want to copy code to a ns like clj-commons.byte-streams, and deprecate the single-segment ns for byte-streams (but not primitive-math, I think that would cause chaos, it’s too popular). ------------------- I’m catching up on the primitive-math issue for clj-easy. Your fork of the master branch looks good, want to make a PR for it against the clj-commons repo? My only request is to remove ^:deprecated and @Deprecated() from the namespace/packages. The deprecation comments should remain, though, and maybe add a note at the bottom of the README explaining that the nested ns is available if needed. The reason for this is, with over 4 million downloads, we don’t want lots of people upgrading, and immediately thinking they need to change their namespaces to fix something, or asking questions, when the change is only necessary for a handful of people. I can see this causing a lot of unnecessary work by accident. We can do the same with byte-streams when primitive-math is ready, but we can leave the deprecation metadata in, because unlike primitive-math, byte-streams will probably get updates.

👍 1
skynet15:03:06

cool ty, I'll take a look when I get a chance, probably this coming weekend. I think I understand what you're saying, that primitive-math should get the cloned namespaces first, and merged and released, and then byte-streams can since it depends on it

skynet23:03:34

@U10EC98F5 found some time to look at primitive-math https://github.com/clj-commons/primitive-math/pull/14 let me know if that PR is in the right direction, happy to make other changes if you want