Clojurians Log v2

Working from a Windows 10 VM. I decided try the build-clojure-msi.ps1 from a powershell shell and got:

PS Z:\proj\oss\casselc\clj-msi> .\build-clojure-msi.ps1
Downloading deps.exe version 1.11.1.1165
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+     [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

Downloading runtime version 1.11.1.1165
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+     [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

Move-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:68 char:5
+     Move-Item -Path "$Destination\ClojureTools\*" -Destination $Desti ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Z:\proj\oss\cas...me\ClojureTools:String) [Move-Item], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.MoveItemCommand

Remove-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:69 char:5
+     Remove-Item -Path "$Destination\ClojureTools"
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Z:\proj\oss\cas...me\ClojureTools:String) [Remove-Item], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.RemoveItemCommand

Downloading WiX binaries
Unable to find type [System.IO.Compression.ZipFileExtensions].
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31 char:5
+     [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.IO.Compression.ZipFileExtensions:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

Creating new MSI at Z:\proj\oss\casselc\clj-msi\out\clojure-1.11.1.1165.msi
.\wix_bin\candle.exe : The term '.\wix_bin\candle.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:117 char:1
+ .\wix_bin\candle.exe .\installers\combined-permachine.wxs -o out\cloj ...
+ ~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (.\wix_bin\candle.exe:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

.\wix_bin\light.exe : The term '.\wix_bin\light.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
At Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:118 char:1
+ .\wix_bin\light.exe -b files -b resources  -ext WixUIExtension "-cult ...
+ ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (.\wix_bin\light.exe:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Done

Maybe I am using wrong version of PowerShell

PS C:\Users\lee> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      19041  1682

Maybe try the pre-built binary?

Ok, I installed https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.2

PS C:\Users\lee> $PSVersionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-----  -----  -----  --------------- ----------
7      2      6

Thought this might be useful too?

To work out kinks in build?

Am still getting build issues:

.\build-clojure-msi.ps1
Downloading deps.exe version 1.11.1.1165
Downloading runtime version 1.11.1.1165
Downloading WiX binaries
MethodInvocationException: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:31
Line |
  31 |      [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory(
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "ExtractToDirectory" with "3" argument(s): "Access to the path 'C:\Program
     | Files\PowerShell\7\wix_bin' is denied."

Creating new MSI at Z:\proj\oss\casselc\clj-msi\out\clojure-1.11.1.1165.msi
.\wix_bin\candle.exe: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:117
Line |
 117 |  .\wix_bin\candle.exe .\installers\combined-permachine.wxs -o out\cloj …
     |  ~~~~~~~~~~~~~~~~~~~~
     | The term '.\wix_bin\candle.exe' is not recognized as a name of a cmdlet, function, script file, or
     | executable program. Check the spelling of the name, or if a path was included, verify that the path is
     | correct and try again.

.\wix_bin\light.exe: Z:\proj\oss\casselc\clj-msi\build-clojure-msi.ps1:118
Line |
 118 |  .\wix_bin\light.exe -b files -b resources  -ext WixUIExtension "-cult …
     |  ~~~~~~~~~~~~~~~~~~~
     | The term '.\wix_bin\light.exe' is not recognized as a name of a cmdlet, function, script file, or
     | executable program. Check the spelling of the name, or if a path was included, verify that the path is
     | correct and try again.

Done

It seems there are hard-coded paths in the script?

Move-Item : Cannot find path 'Z:\proj\oss\casselc\clj-msi\files\runtime\ClojureTools' because it does not exist.

that Z: path is from my vm.

I feel I’m missing some zip support but not sure how to install it.

If only this script were written in bb. 😉

I’ll move onto the prebuilt binary for now.

Trying out the current new and improved https://github.com/casselc/clj-msi/releases/tag/clojure-1.11.1.1165-with-ui

Going for full Windows experience so downloading via Edge. It warns twice!

I chose “keep” and “keep anyway”

Double clicking on msi gives me another warning:

I only got that blue popup while downloading with chrome

I chose “Run anyway” and got an install wizard GUI.

Oh right… I have an existing deps.exe renamed to clojure.exe on my path. I’ll delete that before proceeding.

From a cmd shell, install looks good:

C:\Users\lee>which clojure
/c/Users/lee/AppData/Local/Apps/clojure/launcher/1.11.1.1165/clojure

C:\Users\lee>which clj
/c/Users/lee/AppData/Local/Apps/clojure/launcher/1.11.1.1165/clj

I'm surprised which works on Windows. It's called where there right?

I think you are right!:

C:\Users\lee>which which
/c/Users/lee/scoop/apps/git/current/usr/bin/which

Cool, I see “Clojure for Windows” under “Settings” -> “Apps & features”

And I can “Uninstall” it from here just fine. Neat!

thanks for trying it out. Just got home, so I can take a look at the build issues in a bit. I didn't really make any effort to keep it friendly for older PS versions, but I think win10 should have a minimum version of .net framework to include all the relevant System.IO.Compression stuff.

Looking pretty nice @chuck.cassel! I’ll try to see what my .net framework version is.

were you by chance running powershell in an elevated/admin terminal? there are some cases where the shell and the .net runtime running in the shell process can get out of sync about what the current working directory is, which seems to be the source of that last error I see

Ah. I did not run PowerShell as Administrator. I can retry with that.

possibly related to running from a drive that's not c: also, I can try that on another machine later

no, you shouldn't have to, just wanted to rule that out as a source of the inconsistency

At least it works on Github Actions :)

yeah, my machine and github actions was good enough for me 🙂

Yah, good point. Would be nice to understand minimum requirements for ongoing dev/support of build script.

fwiw, same result from C: drive

Why Powershell though, good question ;)

I mean could the build script technically be rewritten in something more awesome like babashka?

easily

it's just downloading a few things and running two commands

yeah makes sense

Ah cool.

I learned powershell long before I learned clojure, so why powershell is not always an obvious question to me

but it definitely doesn't need to be

So maybe not worth debugging a ps1 script if we might rewrite as bb script.

I’d even volunteer to do that, when the time comes, if it is interesting to folks.

Like @lee I used Edge (Canary in my case) b/c it's my default browser and I clicked the .msi file in the downloads from GH releases. I got the same two warnings about being potentially unsafe but expanded the warnings and chose Keep then Keep anyway. After a good long while (checking the .msi file I think), the installer UI opened. I accepted the license, then just clicked Install and it quickly did it thing.

I opened cmd and typed clj -- no such command. Tried clojure -- no such command. Also tried with .exe -- same results.

I opened PS and tried the same -- with the same results. So it didn't get added to my PATH I assume?

I can't remember accepting a license, but maybe I just didn't pay attention Can you try where clojure and make sure you close all existing cmd windows before doing so?

I accepted license in GUI install. Which is the 2nd msi release, I think?

I don't run cmd or PS normally -- there were no windows open. However, I'd used MS Terminal to start cmd and PS and so I just tried starting cmd directly and got this instead:

C:\Users\seanc>clj
Exception in thread "main" java.lang.Exception: Couldn't find 'java'. Please set JAVA_HOME.
        at borkdude.deps$_main.invokeStatic(deps.clj:444)
        at borkdude.deps$_main.doInvoke(deps.clj:433)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at borkdude.deps.main(Unknown Source)

C:\Users\seanc>

But, yeah, very likely I don't have a JVM anywhere on the Windows side (only on WSL2).

you can try clj --help

(License is first page in install wizard)

Oh yes, I accepted that blindly

I reopened Terminal, and now I get the "expected" failure (no Java) on both cmd and PS. So that was pretty slick (modulo the warnings about untrusted MSI provider).

I think I did not have a terminal open before install.

yeah, I think you have to restart the Terminal thing

At least you don’t have to reboot Windows, eh?

heheh...

I can also remove the license step, just using a pre-built UI sequence that includes one and easier with the tooling to drop in a license file than to wire up the sequence differently

and yeah, terminal caches the environment even for new tabs which is annoying

Seems like a good idea to present the license to me, but Clojure core team can decide those details.

I can also add some detection to show a popup if no java is detected

I wonder if the installer should attempt to check if Java can be found and offer a tip. The clj error Sean showed above is probably good enough.

jinx

So how do we get that MSI to pass the virus warnings? A package manager? And that will probably also take care of the Java warning since it will depend on Java or so?

If any of the non-Oracle JDKs support bundling could also build an installer that could install it as an optional component

I think installing your preferred version of Java is probably a separate concern?

I think I would just provide some links in the installer where to download those "official" Java installers

Or deps.clj could print them

submit MSIs to MS for smartscreen specifically, or sign the MSI if the org has a code signing cert

Do you have to smartscreen every new update?

Is it tricky to get a code signing cert? And/or tricky to sign?

> Do you have to smartscreen every new update? potentially, for the browser/defender screening I think there's also an element of if the MSIs were hosted somewhere that people regularly downloaded from the reputation of the site would eventually stop triggering the warnings

(I found this: https://dogschasingsquirrels.com/2020/01/27/signing-msi-installers-with-a-code-signing-certificate/comment-page-1/)

I know just the man for all convoluted signing jobs.

Tangent: why does command history just-work on Windows? Is that an aspect of Windows terminal apps?

would mainly just need access to the cert/private key from CI pipeline. WiX (the tool I'm using to compile the MSI) has some tooling to deal with signing

nice!

Gonna shutdown my Windows VM for now. Happy to do more experiments/trials in the future, lemme know.

If my memory isn't failing me, being able to scroll up through the current session's history goes back to DOS. There's no persistent history by default for cmd that I know of, but powershell does keep history

Thanks, I used to dev mainly on Windows, so maybe I should remember this. But I don’t.

Sometimes it is good to forget.

🙂 working on this over the weekend I realized in retrospect that the arc of my career was pretty strongly influenced by debugging MSIs ~20 years ago

which I subsequently rarely used again and repressed

Ha! We all thank you for your service here!

Yes, thanks a lot

my first tech job was a tier 1 call center agent for ExxonMobil, and with the shift I had I worked 6am-6pm every other Saturday/Sunday where I had no internet access, undiagnosed ADHD, and offline copies of the golden years of MSDN/Technet documentation

Ah, sweet sweat memories!

when they migrated from NT4 to XP, they decided to contract out the repackaging of all of their apps as MSIs and it went not great. I had read basically all the extant docs on MSI internals, so when users would call in with broken installation I'd go way overboard in troubleshooting but was generally able to manually repair installer registry entries etc that had gone haywire and track down the authoring problems in the original packages. At some point I went home and wrote a tool that used the public installer APIs to query the installer service about status of various installed products and compare that to what actually existed in the installer services registry database and the installed files/components and find discrepancies and fix them various ways. It slowly spread around to all the help desk agents and then one day someone from the engineering team wanted to talk to whoever was bringing in outside tools to the help desk

and luckily instead of getting fired I got moved to a tier 3 support role

OK, back again... So a quick

PS C:\Users\seanc> winget install Microsoft.OpenJDK.17

PS C:\Users\seanc> clj
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Clojure 1.11.1
user=> (System/getProperty "java.version")
"17.0.4.1"
user=>

I was hoping MS would have a build of 19 available:

PS C:\Users\seanc> winget search Microsoft.OpenJDK
Name                                       Id                   Version     Source
-----------------------------------------------------------------------------------
Microsoft Build of OpenJDK with Hotspot 17 Microsoft.OpenJDK.17 17.0.4.101  winget
Microsoft Build of OpenJDK with Hotspot 16 Microsoft.OpenJDK.16 16.0.2.7    winget
Microsoft Build of OpenJDK with Hotspot 11 Microsoft.OpenJDK.11 11.0.16.101 winget
PS C:\Users\seanc>

But, yeah, this is pretty slick overall.

Great work @chuck.cassel!!

And whether or not the Clojure core team adopts deps.exe, not sure I mentioned this yet, but I think the innovation, initiative and effort is awesome @borkdude! ❤️

We need to put some thought in up-front about what type of installation and upgrade scenarios we want to support/allow. There's a lot of subtlety around how/when you bump versions and IDs on various parts of the package depending on what you want to allow.

basically, as far as the installer service is concerned there can only ever be one copy of a product installed, but multiple products can share components, so you have to decide how you want to lie to it about what a product is and what the components in your products are

yeah, so it just has to overwrite the tools jar and deps.exe for every "update"

so nice to have your expertise here @chuck.cassel!

(which is basically the product, but if we can get away with making everything a component, why not)

yeah, the primary questions I see are whether it's desirable to be able to update the launcher and runtimes independently, and whether side-by-side installations of different versions should be allowed, or if installing a newer version should always upgrade the old one in place

launcher and runtime: what we have here is just the launcher

the runtime is determined by your dependencies and Java version. all clj does is construct a classpath for those dependencies and launch java - basically

the tools jar is part of the launch process: calculating the classpath is what happens in there

I guess when I say runtime I'm referring primarily to the tools jar, so I probably need a different word for it. The scenario I'm picturing is basically for some reason we need to update deps.exe or whatever the final equivalent is, but don't need to update the tools.jar. If they're one package that requires a version number bump on the entire package to trigger the update, so the package version is disconnected from the Clojure version - which isn't a huge deal, just needs to be clear up front

You will mostly upgrade the tools jar (and other files that come from that zip file that you download from cognitect). The deps.exe changes less frequently. The thing I update in there the most if the default DEPS_CLJ_TOOLS_VERSION So I guess the question is: when a user wants to upgrade clj.exe, do they uninstall the current one and reinstall the new one, or "upgrade" the components using the new installer - right?

but I think I have a slightly wrong idea in my head about how it works, so that may be an unlikely scenario

So for many releases, this is the only line that changes: https://github.com/borkdude/deps.clj/blob/864d6c5d511182621ae198c74385dbdb1eafff17/src/borkdude/deps.clj#L14 (which still results in a different deps.exe, but you could also just set this with the env var)

maybe that's a good solution, the installer is already explicitly setting the tools directory via environment variable, easy to add another. Then new versions of the executable become much more infrequent. And for anyone wanting to test side by side with different versions they can extract the contents wherever they want and override the env vars by user or process or whatever.

But there will be updates to the executable sometimes.

E.g. when new options are added to the clojure CLI and in case of bugfixes

my initial thought was to have a product for the deps.exe and another for the tools.jar+deps.exe that both use the same component ID for the deps.exe. Then you can do things like if the user has tools(vA)+deps(vB) installed and then installs depsonly(vB+1), the only thing that should happen is the executable file gets updated in place. Then, later whenever they install tools(vA+x)+deps(vB+1) it would only update the tools files, or if they installed tools(vA+x)+deps(vB+y)it would update everything. There would only ever be a single instance of the files on disk and the two product installations are basically just refcounts on the component, i.e. in the scenario above uninstalling the depsonly package would still leave the executable on disk until the tools+deps package was also uninstalled.

but really it just boils down to process, there's no inherent reason why a bugfix to the deps.exe shouldn't be an update to the whole product

just for my own understanding of the universe of options, is it possible to build a native executable that incorporates the tools.jar+etc as either embedded resources or directly integrating the deps.clj+tools code?

I guess they have to be on the filesytem to hand them off to the installed jvm

That's it

Nice!

Coolio!

No installation dialogs, no warnings. I could not resist to restructure directories inside the package. Scoop is versioned by default so doesn't make much sense to have deeply nested directories with multiple version based directories inside. environment variable is set on install. Works great.

wait… atari-emulators!?!

Yeah, I'm maintaining this collection

Cool. Brings back pre-emulator-days memories!

I still own Atari 65XE, technically it's Atari 800XE East European variant.

Spent lots of paper route money on Atari VCS, Atari 400, Atari 800, Atari 130XE, and then… I stopped being a paperboy.

> No installation dialogs, no warnings. I could not resist to restructure directories inside the package. Scoop is versioned by default so doesn't make much sense to have deeply nested directories with multiple version based directories inside. environment variable is set on install. the file layout can definitely be simplified once some decisions are made on what kind of upgrades the msi needs to support. You might be able to cut your manifest down to bins and env variables when we're done

the question of course, is why would you?

sounds like the biggest of all RCE exploits

RCE from an MSI is not so hard, you can invoke whatever binaries or VBScript/JScript you want to embed in the MSI at multiple phases of the session

the only really interesting/novel bit would be embedding and invoking something that knows it's running in an installer and could handover repl control in between each phase

my original thought was just embedding a (possibly slimmed down windows-specific) babashka as a runtime for custom actions scripts the same way VBScript/Jscript work now, so you could use clojure to call out to the OS or whatever from a normal MSI