Fork me on GitHub
#graalvm
<
2019-06-25
>
sogaiu03:06:02

i've replicated the problem with using the chocolatey-based windows-sdk-7.1 -- attempts to build a native image result in the error mentioned here: https://github.com/oracle/graal/issues/1258#issue-442767728 -- this was in a windows 10 vm. will try a manual set up later.

sogaiu03:06:46

looks like uninstalling the chocolatey windows-sdk-7.1 package and then manually installing the windows sdk makes things work out -- at least the build is proceeding further now.

sogaiu04:06:50

@borkdude by modifying PATH on a machine where clj-kondo.exe did not work, i had success in getting it to work

sogaiu05:06:55

i compared some dependency walker info for a clj-kondo.exe working in one vm with one that did not work on a host machine. some differences i noticed include: 1) the working side used dlls from openjdk for some things while the not-working side used dlls from powershell core 2) it didn't seem to be a case of the working side was finding more dlls than the not-working side -- if anything it looks like at least for the set of things satisfied by openjdk vs powershell core, the working side found less i tweaked PATH (so on windows, i guess this also affects how dlls are located...) for the not-working side so that it would pick up some dlls via openjdk instead of powershell core, and the result was that clj-kondo.exe started working.

borkdude07:06:27

@sogaiu that's promising. do you know which dll it is?

sogaiu07:06:49

my current impression is that it's not clear whether the problem is a single dll -- a different set of dlls is being picked up so it may be more than one

sogaiu07:06:50

i will try to see if dependency walker can export more info about the files -- i only saw paths when i did the comparison

borkdude07:06:08

so maybe this problem can be solved with scoop or chocolatey by setting a path to the right dlls or having a dependency on openjdk? (cc @rahul080327)

sogaiu07:06:19

possibly -- don't know if you looked at the stackoverflow info on /mt vs /md, but my impression is that the choice the graal folks have made at the moment by not supporting static is one of the contributing factors to problems like this. i don't know if it's practical for them to support --static any time soon, but that seems like a better way to address the issue -- and get more folks being willing to test native-image.

sogaiu07:06:28

iiuc, on windows, PATH affects which dlls get used by a program -- i think this explains why i kept seeing such varied behavior. installing other software in / uninstalling software from your environment can destabilize execution of existing software because that can lead to changes in PATH, which unfortunately can affect which dlls a program sees / uses.

sogaiu07:06:14

if scoop or chocolatey can arrange to have a startup exe or script that sets PATH appropriately right before launching clj-kondo, may be that can address things?

borkdude08:06:14

maybe yes. I think Rahul mentioned something about an installer, that bundles dlls, but I don't know how that works

borkdude08:06:37

maybe it's best to wait for the GraalVM to bring more clarity around this

sogaiu08:06:40

possibly - i think it might be good to give the graal folks some feedback though. two points i see as problematic for people to test out native-image on windows are: 1) the environment set up should be automatable (e.g. doable via appveyor and the like), 2) --static should be supported to eliminate dll-related issues, but also for deployment reasons.

lispyclouds09:06:43

sorry for the late chime in, the thing that I was mentioning to @borkdude was https://sourceforge.net/projects/nsis/

lispyclouds09:06:57

this could be a (last ditch) way to package the generated exe along with the DLLs from whatever build env it was produced in

lispyclouds09:06:29

AFAIK, the exe should pick up the DLLs if they are in the PWD too

lispyclouds09:06:30

so if we can install the exe and DLLs in to say C:\Program Files\clj-kondo it should work i think

lispyclouds09:06:59

and NSIS is fully scriptable. can be packed onto appveyor

sogaiu09:06:23

i remember using nsis -- i think there's also innosetup -- is the former preferred these days?

lispyclouds09:06:59

my exp is most with NSIS as it produced the most compatible setup.exe

lispyclouds09:06:46

and the fact that the scripting is quite scriptable

lispyclouds09:06:48

personally i dont see a fool proof way of getting this to work till we have a control of the kind of flags used by native-image when calling the linker i guess

lispyclouds09:06:48

apart from using a bundling installer. the good old wizard based windows setups 🙏:skin-tone-3:

lispyclouds09:06:48

also i may be woefully wrong as my windows muscles are getting rusty 😕

sogaiu09:06:37

well, your windows muscules are probably less rusty than mine 🙂

sogaiu09:06:00

btw, i came across this: https://docs.microsoft.com/en-us/windows/desktop/Dlls/dynamic-link-library-search-order may be there is some relevant info -- or there might be a better doc about dll search order

borkdude09:06:07

if the dlls are picked up from the same dir as the .exe, then it could also work with scoop which I think allows downloading dlls along with the main zip and places them in the same dir... not sure

lispyclouds09:06:03

so from @sogaiu’s link:

The search order is as follows:
- The directory from which the application loaded.
- The current directory.

lispyclouds09:06:13

so it may just work

lispyclouds09:06:23

most of my windows dev revolved around Qt, which had its fair weight of DLLs and it did work from the PWD of the exe

borkdude09:06:28

so if we can get clarity of which dlls those are, then I could try to add them to the scoop bucket for testing

lispyclouds09:06:29

so i guess one way could be: - install the 7.1 SDK - dump the dependency list - copy over the exe and the DLLs into a folder - pack it up

borkdude09:06:08

oh yes, just the entire thing as a .zip

lispyclouds09:06:09

yeah, an MVP could be as simple as a zip of the stuff

lispyclouds09:06:02

and the trust on the fact that we arent supplying malicious DLLs. (this is my old paranoia typing)

lispyclouds09:06:11

> oh yes, just the entire thing as a .zip Essentially thats what NSIS is with a more human, interacting and democratic way of installing

lispyclouds09:06:25

a lot of them show the path to the DLL too. can try zipping them?

borkdude09:06:10

I tried dependencywalker, but it was a bit overwhelming

lispyclouds09:06:22

yeah theres a lot of noise

lispyclouds09:06:14

i have a feeling we are gonna solve https://github.com/oracle/graal/issues/1407 before they do

borkdude09:06:06

LOL. I don't have a Windows virtual machine right now because my work laptop had to go in for repair, and the current one has a limited disk. Isn't there something non-GUI that's available on appveyor maybe? I could give the GUI approach a stab, but not soonish

lispyclouds09:06:02

I can try that too but in a similar situation of limited disk space 😕

lispyclouds09:06:23

pretty active project! TIL

lispyclouds09:06:40

looks wayyy less noisy too

borkdude09:06:23

cool, we can mess with it on appveyor 🙂

lispyclouds09:06:20

yeah last time it just took 19 failed builds. have about 5000 more to go!

borkdude09:06:37

failed builds are cheap right?

lispyclouds09:06:54

yeah its time/build thats billed AFAIK

borkdude09:06:11

I was assuming this is free 😉

lispyclouds09:06:26

billed in the sense there is a limit

lispyclouds09:06:04

like CircleCI's 1000 build minutes per month

borkdude09:06:56

ah I didn't even know about that limit

lispyclouds09:06:55

@borkdude I'd suggest using the Appveyor UI to mess around as its way less roundtrips and time waste as making new commits

borkdude09:06:32

ok, I might try it, but currently I'm focusing on a different clj-kondo issue, namely inlining some deps and this breaks the tests 😕

borkdude09:06:23

apart from that, it's bloody hot here

lispyclouds12:06:42

Berlin's approaching 40 here. 🤯

lispyclouds12:06:18

That too when its the most vegan place i know 😛

borkdude09:06:55

(at least for what we're used to in NL ;))

borkdude10:06:21

@taylor I have inlined some deps (rewrite-clj) using mranderson. now I see that those files are also compiled by clj-native-image. somehow the protocol defined in one of those inlined files is unbound now:

Caused by: java.lang.IllegalArgumentException: Unbound: #'clj-kondo.impl.rewrite-clj.v0v6v1.rewrite-clj.node.protocols/Node is not a protocol
any ideas? the full output is here: https://circleci.com/gh/borkdude/clj-kondo/4074

taylor10:06:28

Hmm not familiar with the inlining thing but maybe something is being compiled out of order?

taylor10:06:08

Is there a branch to play with?

borkdude10:06:04

it seems compiling out of order is the problem. I've got it working with an uberjar now it seems, so I might revert to that, least friction

👍 4
taylor11:06:19

I’ll see if I can figure something out before I have to go write Scala at work haha 🥺

borkdude11:06:57

no worries, I can go on with my work using the uberjar. just fyi

taylor11:06:50

I think (in-ns 'clj-kondo.impl.rewrite-clj.v0v6v1.rewrite-clj.parser.core) in that file might be tripping it up because I’m using clojure.tools.namespace.find but not sure yet

borkdude11:06:23

ah, you're doing static inspection?

taylor11:06:20

by way of clojure.tools.namespace, yes

taylor11:06:02

find-namespaces-in-dir returns different results if I comment that out, though it still doesn’t resolve the issue

taylor11:06:00

probably a red herring — even if I remove that ns altogether it still fails

taylor12:06:15

got past that particular error by just not using defprotocol+ but then I reach a different one about cognitect.transit$reader being unreachable, so I’m assuming there’s something pretty fundamentally broken about how this is getting AOT’d in clj.native-image :thinking_face:

sogaiu11:06:11

regarding search order, i also found this: https://github.com/numpy/numpy/wiki/windows-dll-notes -- it has some reflections / observations regarding the previous link, plus other info.

borkdude11:06:42

cool, it seems putting them in the same dir via a .zip makes sense

sogaiu12:06:48

i am not sure yet - trying to digest the list of exceptions.

Adrian Smith18:06:02

Has any one ever tried compiling a binary for alpine linux? using graalvm and native image?

Adrian Smith18:06:44

I've just seen an option for --static is that how to do it?