Fork me on GitHub
#shadow-cljs
<
2020-01-19
>
p-himik08:01:02

It seems there's an issue in FileWatcher. If a new directory is created with a bunch of files and then removed shortly thereafter, there can be a java.nio.file.NoSuchFileException:

Exception in thread "async-thread-macro-1" java.nio.file.NoSuchFileException: /%python-subdir-within-project%/__pycache__/flow_control.cpython-37.pyc.140019814933808
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
	at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2760)
	at shadow.util.FileWatcher.registerAll(FileWatcher.java:41)
	at shadow.util.FileWatcher.pollForChanges(FileWatcher.java:114)
	at shadow.util.FileWatcher.pollForChanges(FileWatcher.java:76)
	at jdk.internal.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at shadow.cljs.devtools.server.fs_watch_jvm$poll_changes.invokeStatic(fs_watch_jvm.clj:17)
	at shadow.cljs.devtools.server.fs_watch_jvm$poll_changes.invoke(fs_watch_jvm.clj:16)
	at clojure.core$map$fn__5866.invoke(core.clj:2753)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.core$seq__5402.invokeStatic(core.clj:137)
	at clojure.core$apply.invokeStatic(core.clj:660)
	at clojure.core$mapcat.invokeStatic(core.clj:2783)
	at clojure.core$mapcat.doInvoke(core.clj:2783)
	at clojure.lang.RestFn.invoke(RestFn.java:423)
	at shadow.cljs.devtools.server.fs_watch_jvm$watch_loop.invokeStatic(fs_watch_jvm.clj:46)
	at shadow.cljs.devtools.server.fs_watch_jvm$watch_loop.invoke(fs_watch_jvm.clj:33)
	at shadow.cljs.devtools.server.fs_watch_jvm$start$fn__73779.invoke(fs_watch_jvm.clj:77)
	at clojure.core.async$thread_call$fn__34862.invoke(async.clj:484)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

p-himik08:01:28

Just to give some context - my project is a mix of Clojure, ClojureScript, and Python. I work with the Python code just by calling shell/sh.

p-himik08:01:35

So it seems like if a file is deleted when Files.walkFileTree is doing its job, there's a chance the whole walk will fail because of it. I guess somewhat of a fix would be to keep calling Files.walkFileTree until there's no NoSuchFileException.

thheller11:01:58

@p-himik why is it watching that directory in the first place?

p-himik11:01:06

Because that directory is on the classpath.

thheller12:01:21

I already have a setting to ignore certain files/paths on the classpath

thheller12:01:40

thats not currently configurable but I could add an option to let you exclude __pycache__?

thheller12:01:57

why is it on the classpath though? seems like tmp files should not go there 😛

p-himik12:01:51

The Python sources are not tmp though. But Python creates __pycache__ right next to the sources. I could of course move Python sources out of the classpath altogether. But having them there gives a nice ability to not have to deal with arbitrary paths when some files are shared between CLJ and Python code.

p-himik12:01:02

Not sure that code above would work though. E.g. I have src on the classpath. Somewhere in there are both CLJ and Python sources. __pycache__ will be created there as well - not at the top level.

p-himik12:01:49

Also, just excluding something doesn't solve the potential issue itself since you can reproduce it with arbitrary code. But one could argue that that's a problem with Files.walkFileTree. Not sure.

thheller12:01:50

yes but there first is a check if a directory should even be traversed

thheller12:01:01

so if it just stops at pycache__ the problem should be gone

thheller12:01:16

I'd advise keeping it off the classpath though ... I doubt python will be able to read them from there if you move to a uberjar

p-himik12:01:16

The check at shadow.build.classpath/get-classpath-entries? It would leave src in there. And then a FileWatcher would be created for src. And FileWatcher doesn't exclude anything.

p-himik12:01:26

If I move to an uberjar - how would I even use Python sources stored in there? I don't think I can use an uberjar at all at this point.

p-himik12:01:27

There's a plan to get rid of Python, which is nice. But the current plan is to replace it with Swift, which will make some things even worse. :)

karthik12:01:15

Hi, I’m new to Clojure world here. I’m using ClJS to create and deploy firebase functions. I’m using shadow-cljs to perform the compilation to the JS lib. I’m not sure where to ask this. But is it possible to jump to js library implementation during the development phase ? Say for eg: I am importing an express js library and want to jump to implementation.

p-himik15:01:45

This is an IDE issue, it's only tangentially related to build tools.

p-himik15:01:55

FWIW, Cursive cannot do that.

karthik18:01:33

Ah okay. Thanks!! I tried Calva and Cursive and no luck.

daniel.spaniel17:01:44

Is there a way to write a hook for shadowjs to "run a clojure function" if a certain file changes in my repo ?

thheller18:01:28

well you can write one but nothing out of the box

odinodin18:01:37

I’ve got a cljs-dependency which references a transitive dependency via js/ prefix (js/snabbdom), but even though I’ve added snabbdom to npm deps it fails

odinodin18:01:54

any idea about how to fix this?

odinodin18:01:31

I can reference snabbdom directly though, so I know it has been properly added to shadow-cljs

thheller20:01:53

@odinodin if the library is just using js/snabbdom without any require for it you can create it by exporting that global variable in a namespace that included before that lib

thheller20:01:19

similar to this

thheller20:01:33

just need to make sure it is included before the lib tries to access js/snabbdom

👌 4
steveb8n22:01:33

Q: I’ve started running nodejs tests using the :autorun feature. this works great. when tests fail, the line numbers are for the .js artifact i.e. source maps are not used (by default). Is it possible to configure source maps in this mode so the line numbers are for the cljs source?

mccraigmccraig09:01:42

ah, but you are looking at the autorunner... hmm - dunno there

steveb8n22:01:42

thanks. I’ll try that out. worst case I’ll stop using the autorunner and use node watch to run the tests instead