This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-06-26
Channels
- # announcements (6)
- # beginners (328)
- # boot (2)
- # cider (72)
- # clara (6)
- # cljdoc (4)
- # cljsrn (5)
- # clojure (78)
- # clojure-europe (3)
- # clojure-italy (22)
- # clojure-nl (4)
- # clojure-spec (3)
- # clojure-uk (114)
- # clojurescript (22)
- # clojurex (54)
- # copenhagen-clojurians (1)
- # core-async (20)
- # cursive (8)
- # data-science (1)
- # datomic (22)
- # duct (11)
- # emacs (32)
- # events (1)
- # figwheel (2)
- # fulcro (18)
- # graalvm (53)
- # graphql (39)
- # luminus (6)
- # nrepl (6)
- # off-topic (53)
- # om (1)
- # re-frame (8)
- # reagent (19)
- # reitit (3)
- # shadow-cljs (28)
- # spacemacs (10)
- # sql (37)
- # tools-deps (33)
- # vim (9)
- # xtdb (6)
@borkdude @rahul080327 i have processed more of the dll-loading-related info -- especially the "exceptions" portion (and the short intro before it) in the numpy windows dll doc. there is some terminology that i'll use below which comes from the numpy windows dll doc, so please take a look or ask what something means 🙂 i also have a list of dlls that got used via openjdk in a case where clj-kondo.exe worked -- this is a total of 30 files (https://pastebin.com/Btn66zQA) under 1 MB total in size. likely also necessary would be msvcr100.dll which is under 1 MB. this info was obtained by examining output from dependency walker. other dlls also showed up, so if you want to have a look, i still have dependency walker output from a clj-kondo.exe that worked and one that didn't. i can post them somewhere if needed. for reference, the top-level dll dependencies of clj-kondo.exe appear to be: advapi32.dll, WS2_32.dll, IPHLPAPI.dll, MSVCR100.dll, MSWSOCK.dll, and kernel32.dll -- of these, on the win 10 system here, advapi32.dll, WS2_32.dll, and kernel32.dll show up as "known dlls". iiuc, MSVCR100.dll is from the vs c++ 2010 redistributable. the remaining IPHLPAPI.dll and MSWSOCK.dll, i presume get resolved via c:\windows\system32 in the s3o or similar order. regarding dll-loading, it appears there are broadly 2 paths one might take for implicit loading (i think that's the current case): one either uses side-by-side assemblies, or one doesn't. if one doesn't, it looks like the application is subject to "already-loaded" dlls and "known dlls" without exception (see the numpy windows dll doc). so far it's not clear the "known dlls" would be a problem. the "already-loaded" dlls though, i think this may account for the variable behavior i was observing. haven't replicated this since reading about it though. on the surface it doesn't seem like the kind of set up one would want. if one chooses to use side-by-side assemblies, presumably the path of a private assembly would be taken. although one may be subject to shared assemblies being searched before searching one's own private assemblies, it seems possible that if clj-kondo defines its own pirvate assembly, there shouldn't be any hit in a search of a shared assemblies (assuming one doesn't place something that would hit it there) so finding inappropriate dlls seem unlikely. with what i understand so far, it seems side-by-side assemblies would provide a stabler execution environment. one thing i'm still quite unclear on is the transitive dependencies of dlls on other dlls and whether that's likely to be a problem in this case. i recommend checking out the wikipedia article on dll hell (https://en.wikipedia.org/wiki/DLL_Hell) -- it has a solutions section too.
btw, i tried putting those 30 + 1 dlls in a folder w/ clj-kondo.exe downloaded from releases in a w10 vm. launching worked 🙂
Awesome @sogaiu! I think we can go the packaging way!
to be able to be define assembly manifests within the graal executable we need some support from the graal compiler i guess
currently i dont know of compiler and/or linker options we can pass to it
yeah that looks like the safe way to go now
till the graal compiler has msvc like options i guess
for reference: https://github.com/numpy/numpy/wiki/windows-dll-notes#user-content-making-an-sxs-assembly
@borkdude do you wanna try the exe + DLLs scoop bundle?
we can try on appveyor. thats the actual build env after all
ah, possibly a couple more issues: 1) 32-bit situation might require some different things? 2) didn't test --lint and other options...don't know if that would bring in more dlls -- both powershell core and openjdk had more dlls that might be relevant
for 1, i think for simplicity and maintenance reasons lets support 64 bit only. for 2 if i understand correctly all the (required and possibly required) DLLs are loaded at launch time. so should be okay i guess
catering to 32bit windows is just an unnecessary headache IMO
but there could be an explicit load call to a DLL which may cause the delayed load, so worth testing
ok, let's only support 64bit. if the appveyor build works in a bare Windows 10 machine, I'm happy (since that's what most people are using as a dev env on Windows I assume). I don't have time to mess around with scoop right now, maybe next weekend or the weekend after
I can give this a shot maybe early next week.
as a first step, we could maybe document which dlls should be copied, and let people verify that this works, and then automate it (using installer or scoop). installer might be the most fancy way 🙂
but updating via an installer might be more painful? as clj-kondo updates quite frequently
maybe including the dlls in the zip on appveyor is a small step? don't know which dlls though
looks like its these ones: https://pastebin.com/Btn66zQA
for reference, the top-level dll dependencies of clj-kondo.exe appear to be: advapi32.dll, WS2_32.dll, IPHLPAPI.dll, MSVCR100.dll, MSWSOCK.dll, and kernel32.dll -- of these, on the win 10 system here, advapi32.dll, WS2_32.dll, and kernel32.dll show up as "known dlls". iiuc, MSVCR100.dll is from the vs c++ 2010 redistributable. the remaining IPHLPAPI.dll and MSWSOCK.dll, i presume get resolved via c:\windows\system32 in the s3o or similar order.
from @sogaiuso these are the only ones missing after installing vs c++ 2010 redist? > the remaining IPHLPAPI.dll and MSWSOCK.dll, i presume get resolved via c:\windows\system32 in the s3o or similar order.
they look like pretty core libs. hopefully they are present on the target systems
if everything's already present, it doesn't explain the problems that were manifesting?
@borkdude just to be clear, the ones via the pastebin link (there are 30) are not in vs c++ redist afaik
maybe this is worth following up in the issue of graal proper? https://github.com/oracle/graal/issues/1407
hmm...iiuc,the first reponse at the following seems to suggest distributing those 30 dlls may be not in compliance with some licensing... https://social.msdn.microsoft.com/Forums/en-US/a28331ae-19a3-4a34-b3ba-1e8fd4430375/missing-apimswincore-dlls -- will keep digging
some of them appear to be listed here: https://docs.microsoft.com/en-us/windows/desktop/apiindex/windows-81-api-sets
yeah but technically we are distributing them via Github releases
as packaged stuff
yeah then scoop should be the only way of usage then
another data point: https://support.microsoft.com/en-us/help/2999226/update-for-universal-c-runtime-in-windows -- in the "File information" section, there are files with names that start with: Api-ms-win-
ah, some explanation about what some of them are: http://www.nirsoft.net/articles/windows_7_kernel_architecture_changes.html
some observations: the nirsoft page mostly lists files with l1-1-0 in the names -- there is only one file, api-ms-win-service-management-l2-1-0.dll, that does not have l1-1-0 in its name. in contrast some of the pastebin links don't have l1-1-0 in their names. also ms-win-crt-* are not listed in the nirsoft article.