This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-10-17
Channels
- # announcements (3)
- # babashka (3)
- # beginners (53)
- # biff (2)
- # calva (16)
- # cider (1)
- # clj-commons (1)
- # clj-kondo (97)
- # clj-on-windows (137)
- # clojure (49)
- # clojure-europe (63)
- # clojure-gamedev (1)
- # clojure-nl (2)
- # clojure-norway (50)
- # clojure-uk (4)
- # clojurescript (36)
- # core-async (28)
- # datomic (32)
- # emacs (22)
- # events (1)
- # graalvm (8)
- # honeysql (6)
- # jobs (2)
- # lambdaisland (5)
- # malli (6)
- # nbb (31)
- # off-topic (171)
- # pathom (14)
- # rdf (2)
- # reitit (4)
- # releases (2)
- # scittle (19)
- # shadow-cljs (46)
- # sql (6)
- # squint (6)
Working from a Windows 10 VM.
I decided try the build-clojure-msi.ps1
from a powershell shell and got:
PS Z:\proj\oss\casselc\clj-msi> .\build-clojure-msi.ps1
Downloading deps.exe version 1.11.1.1165
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+ [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
+ FullyQualifiedErrorId : TypeNotFound
Downloading runtime version 1.11.1.1165
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+ [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
+ FullyQualifiedErrorId : TypeNotFound
Move-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:68 char:5
+ Move-Item -Path "$Destination\ClojureTools\*" -Destination $Desti ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Z:\proj\oss\cas...me\ClojureTools:String) [Move-Item], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.MoveItemCommand
Remove-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:69 char:5
+ Remove-Item -Path "$Destination\ClojureTools"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Z:\proj\oss\cas...me\ClojureTools:String) [Remove-Item], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.RemoveItemCommand
Downloading WiX binaries
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+ [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
+ FullyQualifiedErrorId : TypeNotFound
Creating new MSI at Z:\proj\oss\casselc\clj-msi\out\clojure-1.11.1.1165.msi
.\wix_bin\candle.exe : The term '.\wix_bin\candle.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:117 char:1
+ .\wix_bin\candle.exe .\installers\combined-permachine.wxs -o out\cloj ...
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.\wix_bin\candle.exe:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
.\wix_bin\light.exe : The term '.\wix_bin\light.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:118 char:1
+ .\wix_bin\light.exe -b files -b resources -ext WixUIExtension "-cult ...
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.\wix_bin\light.exe:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Done
Maybe I am using wrong version of PowerShell
PS C:\Users\lee> $PSVersionTable.PSVersion
Major Minor Build Revision
----- ----- ----- --------
5 1 19041 1682
Ok, I installed https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.2
PS C:\Users\lee> $PSVersionTable.PSVersion
Major Minor Patch PreReleaseLabel BuildLabel
----- ----- ----- --------------- ----------
7 2 6
Am still getting build issues:
.\build-clojure-msi.ps1
Downloading deps.exe version 1.11.1.1165
Downloading runtime version 1.11.1.1165
Downloading WiX binaries
MethodInvocationException: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31
Line |
31 | [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "ExtractToDirectory" with "3" argument(s): "Access to the path 'C:\Program
| Files\PowerShell\7\wix_bin' is denied."
Creating new MSI at Z:\proj\oss\casselc\clj-msi\out\clojure-1.11.1.1165.msi
.\wix_bin\candle.exe: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:117
Line |
117 | .\wix_bin\candle.exe .\installers\combined-permachine.wxs -o out\cloj …
| ~~~~~~~~~~~~~~~~~~~~
| The term '.\wix_bin\candle.exe' is not recognized as a name of a cmdlet, function, script file, or
| executable program. Check the spelling of the name, or if a path was included, verify that the path is
| correct and try again.
.\wix_bin\light.exe: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:118
Line |
118 | .\wix_bin\light.exe -b files -b resources -ext WixUIExtension "-cult …
| ~~~~~~~~~~~~~~~~~~~
| The term '.\wix_bin\light.exe' is not recognized as a name of a cmdlet, function, script file, or
| executable program. Check the spelling of the name, or if a path was included, verify that the path is
| correct and try again.
Done
It seems there are hard-coded paths in the script?
Move-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.
Trying out the current new and improved https://github.com/casselc/clj-msi/releases/tag/clojure-1.11.1.1165-with-ui
Oh right… I have an existing deps.exe
renamed to clojure.exe
on my path. I’ll delete that before proceeding.
From a cmd shell, install looks good:
C:\Users\lee>which clojure
/c/Users/lee/AppData/Local/Apps/clojure/launcher/1.11.1.1165/clojure
C:\Users\lee>which clj
/c/Users/lee/AppData/Local/Apps/clojure/launcher/1.11.1.1165/clj
I think you are right!:
C:\Users\lee>which which
/c/Users/lee/scoop/apps/git/current/usr/bin/which
thanks for trying it out. Just got home, so I can take a look at the build issues in a bit. I didn't really make any effort to keep it friendly for older PS versions, but I think win10 should have a minimum version of .net framework to include all the relevant System.IO.Compression
stuff.
Looking pretty nice @U015879P2F8! I’ll try to see what my .net framework version is.
were you by chance running powershell in an elevated/admin terminal? there are some cases where the shell and the .net runtime running in the shell process can get out of sync about what the current working directory is, which seems to be the source of that last error I see
possibly related to running from a drive that's not c: also, I can try that on another machine later
no, you shouldn't have to, just wanted to rule that out as a source of the inconsistency
yeah, my machine and github actions was good enough for me 🙂
Yah, good point. Would be nice to understand minimum requirements for ongoing dev/support of build script.
I mean could the build script technically be rewritten in something more awesome like babashka?
easily
it's just downloading a few things and running two commands
I learned powershell long before I learned clojure, so why powershell is not always an obvious question to me
but it definitely doesn't need to be
My experience 🧵
Like @UE21H2HHD I used Edge (Canary in my case) b/c it's my default browser and I clicked the .msi
file in the downloads from GH releases. I got the same two warnings about being potentially unsafe but expanded the warnings and chose Keep
then Keep anyway
. After a good long while (checking the .msi
file I think), the installer UI opened. I accepted the license, then just clicked Install
and it quickly did it thing.
I opened cmd
and typed clj
-- no such command. Tried clojure
-- no such command. Also tried with .exe
-- same results.
I opened PS and tried the same -- with the same results. So it didn't get added to my PATH I assume?
I can't remember accepting a license, but maybe I just didn't pay attention
Can you try where clojure
and make sure you close all existing cmd windows before doing so?
I don't run cmd or PS normally -- there were no windows open. However, I'd used MS Terminal to start cmd and PS and so I just tried starting cmd directly and got this instead:
C:\Users\seanc>clj
Exception in thread "main" java.lang.Exception: Couldn't find 'java'. Please set JAVA_HOME.
at borkdude.deps$_main.invokeStatic(deps.clj:444)
at borkdude.deps$_main.doInvoke(deps.clj:433)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at borkdude.deps.main(Unknown Source)
C:\Users\seanc>
so it did add it to the PATH (I guess MS Terminal caches that at startup?)But, yeah, very likely I don't have a JVM anywhere on the Windows side (only on WSL2).
I reopened Terminal, and now I get the "expected" failure (no Java) on both cmd and PS. So that was pretty slick (modulo the warnings about untrusted MSI provider).
I can also remove the license step, just using a pre-built UI sequence that includes one and easier with the tooling to drop in a license file than to wire up the sequence differently
and yeah, terminal caches the environment even for new tabs which is annoying
Seems like a good idea to present the license to me, but Clojure core team can decide those details.
I can also add some detection to show a popup if no java is detected
I wonder if the installer should attempt to check if Java can be found and offer a tip.
The clj
error Sean showed above is probably good enough.
So how do we get that MSI to pass the virus warnings? A package manager? And that will probably also take care of the Java warning since it will depend on Java or so?
If any of the non-Oracle JDKs support bundling could also build an installer that could install it as an optional component
I think I would just provide some links in the installer where to download those "official" Java installers
submit MSIs to MS for smartscreen specifically, or sign the MSI if the org has a code signing cert
> Do you have to smartscreen every new update? potentially, for the browser/defender screening I think there's also an element of if the MSIs were hosted somewhere that people regularly downloaded from the reputation of the site would eventually stop triggering the warnings
(I found this: https://dogschasingsquirrels.com/2020/01/27/signing-msi-installers-with-a-code-signing-certificate/comment-page-1/)
Tangent: why does command history just-work on Windows? Is that an aspect of Windows terminal apps?
would mainly just need access to the cert/private key from CI pipeline. WiX (the tool I'm using to compile the MSI) has some tooling to deal with signing
Gonna shutdown my Windows VM for now. Happy to do more experiments/trials in the future, lemme know.
If my memory isn't failing me, being able to scroll up through the current session's history goes back to DOS. There's no persistent history by default for cmd that I know of, but powershell does keep history
🙂 working on this over the weekend I realized in retrospect that the arc of my career was pretty strongly influenced by debugging MSIs ~20 years ago
which I subsequently rarely used again and repressed
my first tech job was a tier 1 call center agent for ExxonMobil, and with the shift I had I worked 6am-6pm every other Saturday/Sunday where I had no internet access, undiagnosed ADHD, and offline copies of the golden years of MSDN/Technet documentation
when they migrated from NT4 to XP, they decided to contract out the repackaging of all of their apps as MSIs and it went not great. I had read basically all the extant docs on MSI internals, so when users would call in with broken installation I'd go way overboard in troubleshooting but was generally able to manually repair installer registry entries etc that had gone haywire and track down the authoring problems in the original packages. At some point I went home and wrote a tool that used the public installer APIs to query the installer service about status of various installed products and compare that to what actually existed in the installer services registry database and the installed files/components and find discrepancies and fix them various ways. It slowly spread around to all the help desk agents and then one day someone from the engineering team wanted to talk to whoever was bringing in outside tools to the help desk
and luckily instead of getting fired I got moved to a tier 3 support role
OK, back again... So a quick
PS C:\Users\seanc> winget install Microsoft.OpenJDK.17
later (and restart the damn PS window again!) and now I have it working:
PS C:\Users\seanc> clj
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Clojure 1.11.1
user=> (System/getProperty "java.version")
"17.0.4.1"
user=>
I was hoping MS would have a build of 19 available:
PS C:\Users\seanc> winget search Microsoft.OpenJDK
Name Id Version Source
-----------------------------------------------------------------------------------
Microsoft Build of OpenJDK with Hotspot 17 Microsoft.OpenJDK.17 17.0.4.101 winget
Microsoft Build of OpenJDK with Hotspot 16 Microsoft.OpenJDK.16 16.0.2.7 winget
Microsoft Build of OpenJDK with Hotspot 11 Microsoft.OpenJDK.11 11.0.16.101 winget
PS C:\Users\seanc>
But, yeah, this is pretty slick overall.
And whether or not the Clojure core team adopts deps.exe
, not sure I mentioned this yet, but I think the innovation, initiative and effort is awesome @U04V15CAJ! ❤️
Summary for @alexmiller: the MSI installer which installed deps.clj's deps.exe to clj.exe and clojure.exe on that PATH worked great. It comes with some virus warnings you have to OK manually which can probably be fixed using a code certificate. Some opportunities for polish: • Warn when there is no java detected (this seems to go beyond what other installers for clj do) or link to some downloads where you can get java • Remind to restart Terminal after executables have been added to the PATH Next steps: • Make the experimental installer known on the wiki, so it can be tested for a while "outside of core"? • Code signing? • ???
We need to put some thought in up-front about what type of installation and upgrade scenarios we want to support/allow. There's a lot of subtlety around how/when you bump versions and IDs on various parts of the package depending on what you want to allow.
basically, as far as the installer service is concerned there can only ever be one copy of a product installed, but multiple products can share components, so you have to decide how you want to lie to it about what a product is and what the components in your products are
so nice to have your expertise here @U015879P2F8!
(which is basically the product, but if we can get away with making everything a component, why not)
yeah, the primary questions I see are whether it's desirable to be able to update the launcher and runtimes independently, and whether side-by-side installations of different versions should be allowed, or if installing a newer version should always upgrade the old one in place
the runtime is determined by your dependencies and Java version. all clj
does is construct a classpath for those dependencies and launch java
- basically
the tools jar is part of the launch process: calculating the classpath is what happens in there
I guess when I say runtime I'm referring primarily to the tools jar, so I probably need a different word for it. The scenario I'm picturing is basically for some reason we need to update deps.exe or whatever the final equivalent is, but don't need to update the tools.jar. If they're one package that requires a version number bump on the entire package to trigger the update, so the package version is disconnected from the Clojure version - which isn't a huge deal, just needs to be clear up front
You will mostly upgrade the tools jar (and other files that come from that zip file that you download from cognitect).
The deps.exe changes less frequently. The thing I update in there the most if the default DEPS_CLJ_TOOLS_VERSION
So I guess the question is: when a user wants to upgrade clj.exe, do they uninstall the current one and reinstall the new one, or "upgrade" the components using the new installer - right?
but I think I have a slightly wrong idea in my head about how it works, so that may be an unlikely scenario
So for many releases, this is the only line that changes: https://github.com/borkdude/deps.clj/blob/864d6c5d511182621ae198c74385dbdb1eafff17/src/borkdude/deps.clj#L14 (which still results in a different deps.exe, but you could also just set this with the env var)
maybe that's a good solution, the installer is already explicitly setting the tools directory via environment variable, easy to add another. Then new versions of the executable become much more infrequent. And for anyone wanting to test side by side with different versions they can extract the contents wherever they want and override the env vars by user or process or whatever.
my initial thought was to have a product for the deps.exe and another for the tools.jar+deps.exe that both use the same component ID for the deps.exe. Then you can do things like if the user has tools(vA)+deps(vB)
installed and then installs depsonly(vB+1)
, the only thing that should happen is the executable file gets updated in place. Then, later whenever they install tools(vA+x)+deps(vB+1)
it would only update the tools files, or if they installed tools(vA+x)+deps(vB+y)
it would update everything. There would only ever be a single instance of the files on disk and the two product installations are basically just refcounts on the component, i.e. in the scenario above uninstalling the depsonly package would still leave the executable on disk until the tools+deps package was also uninstalled.
but really it just boils down to process, there's no inherent reason why a bugfix to the deps.exe shouldn't be an update to the whole product
just for my own understanding of the universe of options, is it possible to build a native executable that incorporates the tools.jar+etc as either embedded resources or directly integrating the deps.clj+tools code?
I guess they have to be on the filesytem to hand them off to the installed jvm
Me trying out to build scoop manifest to install out of msi installer (alternative install experience) 🧵
No installation dialogs, no warnings. I could not resist to restructure directories inside the package. Scoop is versioned by default so doesn't make much sense to have deeply nested directories with multiple version based directories inside. environment variable is set on install. Works great.
Spent lots of paper route money on Atari VCS, Atari 400, Atari 800, Atari 130XE, and then… I stopped being a paperboy.
> No installation dialogs, no warnings. I could not resist to restructure directories inside the package. Scoop is versioned by default so doesn't make much sense to have deeply nested directories with multiple version based directories inside. environment variable is set on install. the file layout can definitely be simplified once some decisions are made on what kind of upgrades the msi needs to support. You might be able to cut your manifest down to bins and env variables when we're done
I have a stupid idea I want to try at some point based on all of this, but the gist of it is I think it might be possible to REPL into an active installer session
the question of course, is why would you?
sounds like the biggest of all RCE exploits
RCE from an MSI is not so hard, you can invoke whatever binaries or VBScript/JScript you want to embed in the MSI at multiple phases of the session
the only really interesting/novel bit would be embedding and invoking something that knows it's running in an installer and could handover repl control in between each phase
my original thought was just embedding a (possibly slimmed down windows-specific) babashka as a runtime for custom actions scripts the same way VBScript/Jscript work now, so you could use clojure to call out to the OS or whatever from a normal MSI