Fork me on GitHub
#graalvm
<
2020-10-01
>
ghadi17:10:47

the cognitect aws-api dynamically loads namespaces based on the target service's protocol

ghadi17:10:51

s3's protocol is rest-xml

ghadi17:10:02

sqs's protocol is query

borkdude18:10:58

Side note: this is a way to call AWS at native (startup) speeds using babashka: https://github.com/tzzh/pod-tzzh-aws It's a pod leveraging the Go API but the Go code is generated using babashka itself (could also be done using Clojure, detail)

borkdude18:10:12

I guess it's a similar approach as cognitect AWS API except the code generation as all done beforehand, no reflection stuff

borkdude18:10:17

(he also published this today: https://github.com/tzzh/pod-tzzh-mail - similar approach leveraging a Go mail library)

borkdude18:10:24

Babashka pods can also be called from the JVM btw

ghadi19:10:43

@borkdude aws-api does not do reflection

ghadi19:10:11

it reads edn descriptor files that describe the service endpoints

borkdude19:10:28

Ah, thanks. There may be other issues why it's heavy on GraalVM native-image. One issue at a glance: $ ls src/cognitect/aws, I spot that it has a dynaload namespace. If that's similar to what spec does, it already explains something. I made a variant of dynaload with specific GraalVM settings: https://github.com/borkdude/dynaload

ghadi19:10:09

for AOT, you can preload all the dynaloaded things, like the repo does above

ghadi19:10:39

the set of dynaloaded namespaces is bounded

borkdude20:10:19

yes, but even then, having code around with find-var, require on the non-top-level (function bodies) in it can still make the image more bloated than necessary. It will work, but it will also be more bloated than necessary. See e.g. https://clojure.atlassian.net/browse/CLJ-2582

borkdude20:10:25

What I would probably do is fork that lib, get rid of these references and optimize

borkdude20:10:04

If it doesn't do reflection, it is weird why such a huge reflection config is necessary in the AOT-ed lib to make it work

ghadi20:10:50

I don't know, I haven't analyzed it. Speaking with my maintainer hat on, aws-api does not explicitly do anything "reflective". It just delays the loading of a few namespaces.

ghadi20:10:02

needs core.async, xml, json libraries, + jetty

borkdude20:10:40

Maybe it would already help if this part was avoided:

src/cognitect/aws/dynaload.clj
14:    (or (resolve s)

ghadi20:10:44

reflect-config.json seems to include cheshire, which we don't use

sergey20:10:07

that might have been me experimenting with some stuff

borkdude20:10:14

yeah, I saw that, seems weird

ghadi20:10:11

the only truly dynamic thing we plan on doing is having a pluggable http client

ghadi20:10:46

right now it's hardcoded to jetty

borkdude20:10:04

is it?

(defn resolve-http-client
  [http-client-or-sym]
  (let [c (or (when (symbol? http-client-or-sym)
                (let [ctor @(dynaload/load-var http-client-or-sym)]
                  (ctor)))
              http-client-or-sym
              (let [ctor @(dynaload/load-var (configured-client))]
                (ctor)))]
    (when-not (client? c)
      (throw (ex-info "not an http client" {:provided http-client-or-sym
                                            :resolved c})))
    c))

ghadi20:10:46

it is -- we don't expose this functionality yet

borkdude20:10:08

anyway, these are the kinds of spots that need attention when dealing with GraalVM native-image probably. With pprint there was only one or two lines that needed changing and boom, 20mb less binary size

borkdude20:10:36

alter-var-root can help patching these things without even touching the code

ghadi20:10:42

I am more interested in figuring out how to structure truly dynamic code to be graal-sympathetic

ghadi20:10:03

like how could you compile aws-api with the jetty client vs with the something-else-http client

ghadi20:10:02

not dynamic in the produced image, just a higher level front-end API to image generation

ghadi20:10:55

e.g. here is my program, it includes some multimethods, I want you to preload these namespaces which extend the multimethods, this is my entrypoint

ghadi20:10:17

like what is the data that is in @sergey923’s repo?

ghadi20:10:16

{:preload-nses [cognitect.aws.protocols.rest-xml 
               cognitect.aws.protocols.query 
               cognitect.aws.protocols.rest-json]
 :entrypoint latacora.foo/main}

ghadi20:10:08

same stuff with jlink in the jvm

ghadi20:10:39

you tell it the set of modules, but also any dynamic Services that need to be present

borkdude20:10:38

@ghadi I haven't looked into it myself but Quarkus is a JVM framework which has loads of modules/extensions that work with GraalVM. It might have some clues as to how to approach what you're interested in

borkdude20:10:27

@ghadi fwiw, my dynaload variant has a setting for GraalVM: https://github.com/borkdude/dynaload it behaves fully dynamic on the JVM, but less dynamic in CLJS and static in GraalVM native-image (where you're supposed to require the namespaces in a certain order, sure that could be configured using some .edn file)