Fork me on GitHub
#clj-kondo
<
2019-11-01
>
borkdude20:11:04

I would appreciate it if someone could go through the steps in this issue to verify if it's clear to reproduce: https://github.com/oracle/graal/issues/1804

sogaiu20:11:47

CLJ_KONDO_VERSION probably needs to be filled in with some value, right?

borkdude20:11:21

thanks, edited

sogaiu20:11:21

we also need a reflection file i think

sogaiu20:11:06

cd project -> cd /tmp/project ?

borkdude20:11:58

made the wrong edit, corrected

sogaiu20:11:28

lol, i'm not sure i've ever waited this long for dependencies to finish downloading πŸ™‚

sogaiu20:11:42

my lein classpath execution fails at:

Could not transfer artifact binaryage:env-config:pom:0.2.0 from/to clojars (): Read timed out
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.

sogaiu20:11:04

a subsequent invocation prints a classpath

sogaiu20:11:37

extra double quote at end of:

./clj-kondo --lint "$(lein classpath)" > /tmp/log.txt"
?

sogaiu20:11:43

i did not observe an exception for:

./clj-kondo --lint "$(lein classpath)" > /tmp/log.txt

sogaiu20:11:14

fwiw:

$ native-image --version
GraalVM Version 19.2.1 CE

lee20:11:53

I can take a pass as well, if that helps.

borkdude20:11:54

maybe I should try the upgrade to 2.1

lee20:11:32

not that it is terribly important but https://github.com/oracle/graal/files/3799384/clj-kondo-2019.10.26-standalone.jar.zip contains __MACOSX dir

> unzip -l ~/Downloads/clj-kondo-2019.10.26-standalone.jar.zip
Archive:  /Users/lee/Downloads/clj-kondo-2019.10.26-standalone.jar.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
  8704607  11-01-2019 15:35   clj-kondo-2019.10.26-standalone.jar
        0  11-01-2019 15:45   __MACOSX/
      985  11-01-2019 15:35   __MACOSX/._clj-kondo-2019.10.26-standalone.jar
---------                     -------
  8705592                     3 files

borkdude20:11:48

I'm still experiencing the problem with the upgrade to graalvm 19.1.2

borkdude20:11:25

yeah, that's an artifact of zipping the jar on my machine, can ignore that

lee20:11:27

Also kind of obvious, but you might want to replace "Build the following .jar" to "Unzip then build the following .jar"

lee20:11:19

I don't see the version of Graal you are testing against in the issue.

borkdude20:11:24

also added MacOS 10.14.6

borkdude20:11:31

which I am using and also the original reporter

lee20:11:32

as someone walking through the issue it would be helpful to have the reflection.json come before the command because the native-image command requires it.

borkdude20:11:49

it would be interesting to discover if linux users didn't have this problem

lee20:11:57

Is $NATIVE_IMAGE common? if not just replace with native-image

borkdude20:11:15

good point, done

sogaiu20:11:48

was using linux, and also used plain native-image

lee20:11:51

you no longer need "where $NATIVE_IMAGE..."

borkdude20:11:29

@sogaiu could you try this also with the binary from github, and/or on a macos machine?

borkdude20:11:41

if you don't trust the binary from github, then ignore my request πŸ™‚

lee21:11:04

am using macos

sogaiu21:11:06

lol, just don't have access to macos for a few weeks

borkdude21:11:27

cool lread, I'm awaiting your findings

sogaiu21:11:13

btw which binary do you mean?

borkdude21:11:13

scroll to 'assets'

sogaiu21:11:59

will try when i get back

lee21:11:58

in your code snippet, comment # this directory will be used to create a RandomAccessFile in might not be finished or has extra word in?

borkdude21:11:29

it will be used to create a file in

borkdude21:11:50

"to put it in"

lee21:11:29

oh how about "# a RandomAccesFile will be created in this directory"

lee21:11:04

(original confused me, might confuse reviewer)

borkdude21:11:20

excellent, done

lee21:11:37

delete trailing double quote in code snippet?: ./clj-kondo --lint "$(lein classpath)" > /tmp/log.txt"

lee21:11:38

successfully reproduced FileNotFoundException

sogaiu21:11:34

did your lein classpath succeed without problems?

borkdude21:11:04

thank you lread. That is also on MacOS right? which version?

borkdude21:11:29

And could you also verify that the problem does not occur when linting with the jar?

lee21:11:00

am on that last step now... should mention that java -jar ... command should executed from /tmp/project as it needs project.clj

lee21:11:36

and maybe.. redirect output from java -jar to different log file as to not overwrite previous results.

lee21:11:31

also mention that an exit code of 3 is expected for java -jar ...

borkdude21:11:45

the results are not important and probably the log file from the first step is completely empty

borkdude21:11:31

the only important thing is the exception. the error code is probably on me for not catching the exception from the main function, so not for them to sort out

borkdude21:11:36

it's a clj-kondo thing

lee21:11:05

maybe route to /dev/null then.

lee21:11:23

to be super clear it is not a variable.

lee21:11:37

just humble suggestions of course

borkdude21:11:50

oh I get what you mean with the exit code: they should ignore it even it it's not zero

borkdude21:11:04

good suggestions

lee21:11:00

I am running macOS 10.15 (19A602)

lee21:11:31

(that thing in brackets seems new? apparently is the build number)

borkdude21:11:11

added your version to the issue as well

borkdude21:11:18

isn't 19a602 a machine model?

lee21:11:22

so that's it full walkthrough of issue done. looks good.

borkdude21:11:48

ah it's a supplemental update probably

lee21:11:50

looks like it eh? apparently is build number: https://support.apple.com/en-is/HT201260

borkdude21:11:31

so no exception using the .jar?

lee21:11:42

no exception, exited with code 3

borkdude21:11:49

super helpful. so far it only seems to happen on MacOS

borkdude21:11:05

maybe it has to do with too many file handles or something, don't know, low level stuff!

lee21:11:56

yeah, just started playing with graal today, a big bucket of mystery to me!

lee21:11:20

your projects are good references for graalvm thanks!

borkdude21:11:15

jet might be the simplest one of the bunch to look at

lee21:11:35

hmmm just noticed macOS 10.15.1 is available, I doubt that would make a difference for your issue...

borkdude21:11:47

but they are compiled more or less the same, using lein uberjar + the script/compile script

lee21:11:42

ya, super helpful. I'm going the tools deps route, and figured out what I need for that today.

borkdude21:11:58

note that I went that route before, but had reasons to go to lein

borkdude21:11:11

but feel free to explore πŸ™‚

lee21:11:37

hmm... interesting! uberjaring is certainly more straightforward with lein.

borkdude21:11:50

basically because lein had better capabilities for AOT compilation

borkdude21:11:57

boot might do it good as well

borkdude21:11:18

but I found nothing that came close to this using deps.edn. sogaiu mentioned an interesting tool recently which might do it

lee21:11:44

so many competing packagers for tools deps. it is confusing.

borkdude21:11:17

yeah well there's also clj.native-image, but I ran into limitations. basically all tools which wrap the native-image command will be limiting, in my experience an AOT-ed uberjar + shell script works better

borkdude21:11:32

in that wrappers are limiting by definition

borkdude21:11:46

or confusing, sometimes, at the least

lee21:11:35

I am using currently using depstar to build my jar and am following instructions for aot compiling from here: https://clojure.org/reference/compilation#_gen_class_examples

lee21:11:00

I have used maven to build my jars in the past.

borkdude21:11:11

one problem I ran into with clj.native-image was that it tried to compile my namespaces, but somehow missed a few or didn't do it in the right order

lee21:11:48

I'm sure I'll hit problems, my graalvm playground only has 2 namespaces!

borkdude21:11:48

I also ran into problems with it on Windows. Eventually I made a script/compile and script/compile.bat

lee21:11:43

did you consider compiling your tests with graal (along with your src) and running them that way? I think that's the approach typically taken for plain old cljs testing.

lee21:11:03

might not make sense for your command line tools... but for a pure library without a command line.. maybe?

borkdude22:11:40

@lee No, I run the tests from the JVM, but only change the way how to call the thing under test. When doing native I call the binary using clojure.shell/sh

borkdude22:11:06

Also for libraries, I just make a main that I test (see e.g. sci)

lee22:11:58

yes, understood, that makes sense for sci and jet

sogaiu22:11:15

no exception when using downloaded binary, fwiw

borkdude22:11:34

@sogaiu thanks, that re-inforces the hypothesis that it's specific to macOS

borkdude22:11:37

@lee Are you thinking about tests for rewrite-cljc in this case?

lee22:11:55

yeah, I should move this convo over to #rewrite-clj

lee22:11:08

or maybe #graalvm

borkdude22:11:12

I don't mind πŸ˜‰

lee22:11:25

you are a gentle moderator simple_smile

borkdude22:11:13

I haven't tried compiling clojure.test to GraalVM but if that works, then that's a fine approach I'd say.

lee22:11:14

I think it might make sense for rewrite-cljc - if it works!

borkdude22:11:39

I haven't got any GraalVM tests in edamame, but since I'm using that extensively in sci and babashka, I test it that way

lee22:11:01

yeah makes sense

lee22:11:31

btw, I just upgraded to macOS 10.15.1 (19B88), same result for your graal issue.

borkdude22:11:15

thanks, added the version to the issue

borkdude22:11:59

what's weird is that the size of the classpath makes a difference for getting a FileNotFoundException

borkdude22:11:08

and only on macOS πŸ€”

lee22:11:01

it is a puzzle

lee22:11:28

I suppose, if you were curious, you could do some println debugging prior to opening the file... check what returns you get for .exists and .canRead, maybe try .getAbsolutePath.

borkdude22:11:08

that's a good suggestions. this API should work even when the file doesn't exist yet, but it's good to add that debugging.

lee12:11:39

ah, ok, makes sense

lee12:11:23

@, I’m actually looking at the code https://github.com/borkdude/clj-kondo/blob/f6ab3624defa04708887bc9687805b4303f5dbb4/src/clj_kondo/impl/cache.clj#L58 don’t you need a .close somewhere for that RandomAccessFile?

lee12:11:06

yeah, just a guess but problem occurs when classpath is long, so maybe the misleading file not found exception due to a resource leak blowing some os limit?

borkdude12:11:18

I'll try this

borkdude12:11:24

it doesn't hurt to add it anyway

borkdude12:11:21

didn't help, but I'll keep it in

borkdude12:11:52

I'll merge it to master shortly

borkdude22:11:00

so .exists returning either true or false doesn't say that much, since it "should just work TM" in both cases

sogaiu22:11:27

for a bit of extra confidence, i went through the issue steps on an ubuntu vm (though with the binary downloaded from releases) -- no exception in the vm either

sogaiu22:11:47

certainly lots of having to restart lein classpath though

borkdude22:11:27

it's probably a GraalVM/macOS substitution problem for PosixUtils.java or something

sogaiu22:11:57

ah -- it's probably totally unrelated, but i keep hearing on #shadow-cljs about jvm issues on macos

borkdude22:11:44

what could be tried is not only retrying the locking of the file, but also the creation of the file:

(defmacro with-cache
  "Tries to lock cache in the scope of `body`. Retries `max-retries`
  times while sleeping 250ms in between. If not succeeded after
  retries, throws `Exception`."
  [cache-dir max-retries & body]
  `(let [lock-file# (io/file ~cache-dir "lock")
         _# (io/make-parents lock-file#)
         raf# (RandomAccessFile. lock-file# "rw")
         channel# (.getChannel raf#)]
     (loop [retry# 0]
       (if-let [lock#
                (try (.tryLock channel#)
                     (catch java.nio.channels.OverlappingFileLockException _#
                       nil))]
         (try
           [email protected]
           (finally (.release ^java.nio.channels.FileLock lock#)))
         (do
           (Thread/sleep 250)
           (if (= retry# ~max-retries)
             (throw (Exception.
                     (str "clj-kondo cache is locked by other process")))
             (recur (inc retry#))))))))

borkdude22:11:59

but that would only mask the underlying problem

borkdude22:11:09

which is unknown yet