Clojurians Log v2
clj-on-windows
Hi @seancorfield, we all agree the .exe file is a much better solution to go with, but still it does not address the quoting issue though, it can fail on nested quotes (as per below example). I have an idea which has the potential to elegantly solve the quoting issue irrespective of OS and executable type that I need to put to trial. Quoting should be seen as an inherent incidental complexity that we should be looking to address rather than getting defeated by. My aim here is to get first class support for Clojure on windows that I see as fundamental to the wider Clojure reach, especially for newcomers. Asking people to move to WSL is IMHO a step towards the wrong direction. Window users should not be forced to unix conventions. WSL comes with its own challenges (interop between win programs and WSL programs, need to install an X server on Win10, need to configure port forwarding when opening a port to the outside world etc). Thanks
Quoting failure example
on bash (success):
$ deps -Sdeps "{:aliases {:demo {:jvm-opts [\"-Dx=\\\"y z\\\"\"]}}}" -e "1"
WARNING: Implicit use of clojure.main with options is deprecated, use -M
1> deps -Sdeps "{:aliases {:demo {:jvm-opts [\"-Dx=\\\"y z\\\"\"]}}}" -e "1"
Error while parsing option "--config-data {:aliases {:demo {:jvm-opts [\"-Dx=\\y": java.lang.RuntimeException: Unsupported escape character: \y
Yeah, I was just going to say: deps.clj does not solve any quoting issues. Quoting issues are an interplay of shell-specific quoting rules and software requiring users to use quotes on the command line (in the case of clj: quoted EDN forms).
This is one of the reasons I started writing https://github.com/babashka/cli#clojure-cli to offer the ability to create CLIs with minimal effort
When going with deps.exe -> clj.exe some things have already been mentioned but something else hasn't:
• Mentioned: GraalVM native-image could go away. I personally don't see this happening as Oracle offers enterprise support for this and it's adopted quite widely by all kinds of Java frameworks like Spring etc. as well
• Not mentioned: GraalVM native-image supports the most popular CPU architectures but not all of them. I recently had a question from someone who ran on PowerPC. I told him he could still use the .jar instead of the .exe and the issue was solved. But this may be a good requirement to study when replacing all of the things with a single other thing. I don't see a problem with an additional solution like deps.exe. You can simply rename deps.exe to clj.exe today and then you have a working installation.
Thanks @borkdude. The other think to consider when going with deps.exe (or any other tooling) is installation (the original topic of this discussion). Windows users in corporate environments are most likely to be behind Web Application Firewall nowadays and have restricted installation rights on their machines. This rules out using package managers such as Scoop whose range of packages they support is immense and most likely to be barred. Perhaps it is better to consider an MSI solution in this case (I believe @danielmartincraig is looking into this), that sysadmin can reference in their setup scripts.
@chaos deps.exe doesn't have to be installed: you can just place it anywhere on your machine and it should work
Indeed, though there is a high chance that WAFs nowadays are most likely to screen and potentially prevent any .exe files from downloading from untrusted sources from the net ... :(
We've recently seen this with babashka and submitting it to a Microsoft link unflagged it rather quickly
But this is definitely one of the things to add to the consideration matrix
Yes, I've noticed that. The case I'm referring to is not due to a false positive alert by the antivirus, but rather to a WAF policy to prevent all .exes from downloading, unless they are explicitly whitelisted ... :(
I see
So maybe with all things considered the powershell solution is an optimum in some sense? Or are there some corporate environments that also block powershell stuff? I think I've heard that before
I think if there is an installer endorsed by MS, then it should be fine, e.g. an MSI package or a PS module as we have now. It is easier for sysadmins who create the corporate windows build to include it in their build configuration, rather than trying to put something ad hoc in place. It is also much easier to convince them to open a WAF policy to allow users to download it themselves, since the installation will be registered and easy for them to audit (as oppose to dropping a random .exe file somewhere). PS is ok for installation purposes, but definitely not developer friendly (it is unfortunate that the (un)official Clojure windows script is written in PS 😞, it causes much pain and is irritating slow to start from the command prompt, but it is what it is). I would personally rather prefer an MSI/.exe solution or maybe a standalone .bat that calls java (a la @ericdallo's https://github.com/ericdallo/deps-bin perhaps), though I am not sure how well a standalone`.bat` would interact with a WAF.
OK, well, as always, if anyone from the core team wants to borrow anything from deps.clj, feel free to do so without my permission and if you need my help, just ping me
One detail on the topic of quoting: a powershell module gets ultra tricky when you are shelling out from a program.
You have to invoke the module with powershell. The quoting complexity of this scenario was a wall I decided to stop banging my head against and when I switched to deps.exe.
Also when shelling to powershell, the naive (like me) can all too easily not realize there needs to be special handling to capture and return the exit code. For me, this meant some failing tests on Windows native were going unnoticed on CI.
Hi @lee, indeed quoting gets twice as complicated when you try to invoke PS from another shell/process, because one needs to quote the args according to the outer shell’s convention while also accounting for the PS quoting rules. On top of that, java applies its own complicated quoting rules on windows just before calling to the win32 CreateProcess API, so it all becomes too complicated for a single person to reason about it reliably. Wrt to the exit status, I have raised another question to ask clojure alongside this one, proposing a “simple” fix. There is a straightforward workaround as you have mentioned, but is well hidden from the average user’s field of view.
@borkdude I guess I misunderstood some of what I heard you say about deps.clj and quoting then -- I thought it made life easier in that area? Does it really require the same triple quoting that is shown here https://clojure.org/reference/deps_and_cli#quoting ?
@seancorfield Yes, it doesn't matter if you call an .exe or whatever, passing command line arguments doesn't change
I've never said that deps.clj made quoting easier
Thanks for the clarification. That makes PS as a whole suck even more from my p.o.v. 😐
It does make shelling out easier: shelling out to an .exe is easier than shelling out to a powershell module
> That makes PS as a whole suck even more from my p.o.v. Why is that?
Because of the quoting required. It's so much more complex than macOS/Linux.
(and, as mentioned above, the steady stream of complaints from beginners that following READMEs, books, and tutorials "doesn't work" on PS, even when they've managed to install the Clojure module -- I was hoping that weird quoting stuff was just an artifact of using a PS script rather than inherent to PS itself)
cmd.exe and powershell also have different quoting rules, but even in bash working with quotes gets cumbersome if you throw some $(inline-command ...) in the mix
I think I have a “good” idea about how to manage argument quoting uniformly across the different OSs and this can also support any shell extensions the user would like to apply locally, but I’d like to get closure of this discussion first before getting roasted again with yet another suggestion 😨
Well, I for one am curious about your idea :)
so, we follow the good example set forward by the lisp reader, and create a pseudo reader literal, eg #clj/'
we then take control of arguments escaping by disallowing any characters that are designated special by any shell quoting mechanisms
e.g. if you want to create an edn with embeded quotes, we will use something like “#clj/' {:aliases {’Ddemo’D …}}”
’D is hte mnemonic for ”
# means comment in bash...?
thus when this argument is parsed by the tool deps, it will be identified as a special reader argument (starting with #clj/‘) and all occurances of ’<letter> will be converted to the actual character
let me get the example i used earlier in the thread, should be straightforward to understand I think
so this one:
deps -Sdeps "{:aliases {:demo {:jvm-opts [\"-Dx=\\\"y z\\\"\"]}}}" -e "1"deps -Sdeps "#clj/' {:aliases {:demo {:jvm-opts ['Dx='S'Dy z'S'D'D]}}}" -e "1"so we avoid using any of the escape seaquences in the argument and we define our own reader literal to parse the argument
’D = Double Quote, ’S for slash
so ' is the escape character
does it make some sense?
this is akin athink to reversion of control, we take control of quoting
Probably just reading the args from a file would be a better solution? deps -Sdeps @foo.edn
i think there is an issue there that you can’t put these as a single command liner on a web page
we want to keep things as they stand, but handle quoting in such a way that doesn’t cause an issue
I've done a similar but different suggestion here: https://ask.clojure.org/index.php/11585/convention-bypassing-parsing-reduce-quotes-arguments-passed
but then I wrote babashka.cli which deals with this problem now
yes it is exactly on the same spirit. The above is more clojurian since using a “pseudo” reader literal, and specifies what the escaped characters are simply by using ‘<letter>’. This can be easily implemented as a clojure library with no extra dependencies required, and if ’required by say the tools deps, it just a matter of calling a function on each and every argument to convert them into the end arguments. Very simple spec and easy to implement (famous last words, I need to test this out, I have already a list of characters across the different shells that need escaping, such as the backtick from PS)
and I have also thought how to make dead simple for a user to convert from one format to another without really needing any extra options on the command line
Does it support the inline JSON example in the ask issue?
I'm not immediately convinced that this idea is dead simple and more clojurian, but I don't want to roast your idea ;)
yes this is the thing, until i have something to demonstrate, it’s all up in the air. So this is why we keep this a secret 🙂
sorry no it wont work with the above jason, since it need the command to be escaped before hand, but you can $(bb <escape> command) like “clj/' $(bb <escapecmdcallingtothequatinglibrary> file)” and this will do it
so bb willl use this new tiny library to quote the json file before inserting it in “clj/' …”
the new libraries job should be dead simple,:if the string starts with “#clj/‘“, get the rest of the string starting after “#clj/” and convert all occurrences of ’<letter> to that character designated by the <letter>, and return that string
And what if you are arguments have single quotes?
you escape them with ‘’
so any single quote character in the string is escapes as quote quote
could work
e.g. “‘quoted string’” -> “#clj/' ‘’quoted string’’”
another case is what about the $ charracter which is special in bash. Well we designate ’R as the escape characger for dollaR
if you want it as literal, use that, if you want to invoke the shell subcommand, then just use $(…)
you might as well base64 escape your edn :P
nope, this is really horrible
this is what powershell does
you don’t have any idea at all what you are running
try run cider on windows and you get a horrible powershell comand line of cider starting the nrepl server, you have no idea what cider has run
It was a joke. What does powershell really do?
just use base64 decoding on that line
so you can call PS command with a single basey64 encoded string, which is the command to run and the arguments
cider uses this to bypass all this really complex quoting mechanism
then if you get an error with cider nrepl starting, good luck figurring out what might have gone wrong,
I need to look into this in cider and somehow either decode the nrepl starting command on the message displayed, or add an additional message with the decoded string
so to recap, the idea to me looks dead simple, it handles arguments uniformaly across all archs, and uses the familiar reader conditional to denote something special is happening. I would even argue it makes the quoted string even easier for a human to understand, only because it uses mnemonic characters like ’S ’D ’R or whatever to designate the quoted chars. Where do you think btw, this. might not be as simple as it seems?
I’m just trying to find a counterexample, because it looks to me too good to be true
I think the idea could work, but the downside is that you will have to convert normal EDN to this notation either in your head or programmatically, when you want to copy paste stuff
(I have also though about how to help with inputing this expressions and debuging them, all without the calling application which is using this library having to add additional arguments such as --quote-expression or --debug-expression to their list of args)
so
here is the answer
a call like “#clj/‘’” will instruct the library to ask the user to input the unquoted expression. so the user will run say deps “#clj/‘’” and they get a prompt like, please enter the expression to encode:, the user inputs the expression, the tool displays the encoded line, the program runs as expected with the encoded line
so the user can immediately see what the expression they would llike to quote looks like quoted, copy paste to “#clj/' …” next time they run the command
I have to run now. Thanks for sharing those ideas
thanks for keeping an open mind!
have a great evening
you too!
(and being so approachable to everyone)
Hi @borkdude, I've created a draft library of my idea to make argument quoting cross-platform as discussed above which I believe can solve the quoting issue. Do you think you could have a look and provide some feedback about its potential usefulness? I've tested it out with deps.clj. Basically, it solves the quoting issue by disallowing the superset of symbols that are known to be special across all platforms to appear on the command line, and provides a simple two char syntax starting with % to escape those characters. It is an opt-in solution that can be mixed with standards arguments. It borrows the idea of edn tags argument syntax in any way imaginable, such as reading from files (there is an example in the code where an argq/file tag is trivially defined to read the argument from a file). As such it should be easy to create a new tag (e.g. argq/shell) to run a a command with babashka.process and pass the result as an argument value (should address the aforemention clojure question). This is the first draft, and I am hoping to get for some early feedback that could either reveal any fatal fallacies that I might have overlooked or steer the development towards the right direction, thanks
https://github.com/ikappaki/argq.alpha
I think it would be better to gather feedback from multiple people so you could try posting it either here or in a channel with more people.
Sure, thanks!
note that % also means something in Windows: it denotes environment variables, but you probably know this
Indeed, I don't think they have any special meaning inside quotes but tis is relevant nevertheless to the readme file where I document the special characters
My personal impression is that such an alien syntax would scare people away even further and we as CLI writers should strive to not impose any quoting (or bespoke syntax) on people
but if it will help someone in the status quo, why not :)
I think the quoting part has the potential to solve the issue with publishing command line invocationd on web pages, so it can have some niche market
I've also added some tooling to help with writing and decoding the syntax to address that concern (how helpful that turns out to be is another matter :))
I think reading command line arguments from a file would be an easier solution
That said, please gather feedback from other people as well
Thanks! you are right about the % characters btw used as an escape char on windows, this is one of the fallacies I haven't considered!
Hi @borkdude, I have introduced a https://github.com/ikappaki/argq.alpha, trying to address in good faith any concerns with the library as they come in starting with what you said above wrt to potentially introducing yet another alien syntax layer. Let me know what you think, if you happen have the time to look into it. I have also changed the tagged literals to make them more descriptive (e.g. #clj/esc -- for escape -- instead of #clj/*), switched the escape character from % to * , and introduced a study case for the ask clojure question on how to pass the json output of a command as an argument to clojure -X. In addition I asked for feedback from the channel participants as suggested. I understand you might be busy or uninterested in this, thus don't feel obliged to answer or look into it. My aim is to hook this library up with either the clojure tools scripts or clojure.tools.deps.alpha, if it is proven to solve the cross platform quoting issue beyond any concerns. Thanks
@chaos I’m sorry that you feel roasted, but I also know from personal experience that it can be hard on the ego to invest a bunch of time into some idea and then have it critiqued and/or rejected. But in the end, personally, I’m ultimately fine with it. I admire the time and care that the Clojure team take, and very much respect its benevolent dictatorship.
And remember that just because an idea is not adopted, doesn’t mean it is not a good idea.
@lee personally I’m fine with this either way, rejected or approved. I think these discussions are done in good faith by everyone for the greater Clojure good, and it brings everybody else to the same page with regards to the other people’s perspective and experiences. I could as well be technically totally wrong, that’s why i put these ideas forward for “consideration” rather than demanding to be implemented 🙂 It’s very good to get roasted, this means the ideas are put to a good test by highly respectable people like yourself 🙂 What I would have been disappointed with is not wanting to discuss ideas that someone genuinely believes are for the greater good.
Coolio, well glad you are here @chaos, and nice to see someone representing the Windows native cause who is actually using Windows native!
@chaos According to the last State of Clojure survey analysis, 90% of Clojure users are on macOS/Linux, 5% are on Windows (using cmd.exe or Powershell), and 5% are on Windows/WSL2. Just offering a data point. I'm in that latter 5% and very happy with WSL2 as the best option since all the books/tutorials out there "just work", whereas that other 5% is full of users who try to follow instructions from books/tutorials and project READMEs and run into errors because of the peculiar argument quoting issues on cmd.exe and (different) on Powershell. It doesn't matter how easy the PS module install is made, it isn't going to fix those quoting issues so those beginners are just going to be even more frustrated: they can "easily" get clojure installed on Powershell but then it "doesn't work" (because they haven't read the section on Windows-specific argument quoting on http://clojure.org) and all the project READMEs have "the wrong instructions". I'm already fielding a steady stream of "bug reports" on clj-new and deps-new from PS users and I have to point each of them to http://clojure.org piece about quoting arguments.
I agree with @lee that it was a good alpha experiment but the UX is still pretty horrible even if you can defeat the PS installer stuff 😐 I think having a simple, downloadable .exe file is the way to go on Windows because that also addresses the quoting issues (right @borkdude?).
p.s., if you've posted on http://ask.clojure.org, there's really no need to raise it here and beg the core team to consider it, since they'll definitely see it on Ask and respond there (Ask is permanent, Slack isn't -- despite us currently having a complimentary Pro account).