Fork me on GitHub
#babashka
<
2019-12-22
>
Cora (she/her)04:12:54

hrm, it appears you need java.nio.file.Paths in order to contruct a Path object

Cora (she/her)04:12:55

which is needed in order to call createSymbolicLink on java.nio.file.Files

Cora (she/her)04:12:07

or at least that seems to be the case

holyjak11:12:30

@corasaurus-hex I believe File has a toPath?

👍 4
Cora (she/her)12:12:00

ahhh so it does, I'll try that!

Cora (she/her)18:12:51

yuuuup it works

👍 8
borkdude19:12:30

@corasaurus-hex @holyjak I didn't add java.nio.filePaths earlier because I never could figure out how to use it from Clojure:

$ clj -e '(java.nio.file.Paths/get (into-array String "src")))'
Execution error (IllegalArgumentException) at java.lang.reflect.Array/set (Array.java:-2).

borkdude19:12:14

but this seems to work now:

$ bb '(java.nio.file.Paths/get "/home/justin" (into-array [".lein" "profiles.clj"]))'
#object[sun.nio.fs.UnixPath 0x566ebefb "/home/justin/.lein/profiles.clj"]

👍 8
holyjak12:12:46

hm, I think into-array takes a seq while your 1st arg as "src" - you likely wanted ["src"]

borkdude19:12:28

(found it on StackOverflow)

Cora (she/her)19:12:11

the file -> toPath thing works well enough for me 🙂

Cora (she/her)19:12:38

oooooh good release!

Cora (she/her)19:12:46

delay will be really nice

Cora (she/her)19:12:34

and the current file in exceptions? 😚👌

Cora (she/her)20:12:32

and Paths, of course 💜

Cora (she/her)20:12:43

so, maybe a crazy idea, but what if babashka could also include a deps.edn-style comment at the top that will pull in deps when running with a shebang?

Cora (she/her)20:12:18

you can manually do it with with something like this:

#!/bin/sh
#_() ;; -*- mode: clojure -*-
#_(

   #_DEPS is same format as deps.edn. Multiline is okay.
   DEPS='
   {:deps {me.raynes/fs {:mvn/version "1.4.6"}}}
   '

   #_You can put other options here
   OPTS='
   -J-Xms256m -J-Xmx256m -J-client
   '

exec clojure $OPTS -Sdeps "$DEPS" "$0" "[email protected]"
)

(prn *command-line-args*)

Cora (she/her)20:12:44

but having a way to pull in deps in a comment would be super cool

borkdude20:12:26

@corasaurus-hex Yeah, let's first see how this library approach pans out (e.g. the clj-http-lite one)

borkdude20:12:11

Note that you can set the babashka classpath in BABASHKA_CLASSPATH as well

borkdude20:12:22

and you can require things in BABASHKA_PRELOADS.

borkdude20:12:37

so that gives enough room to not include any non-standard deps.edn stuff I think

borkdude20:12:51

Adding convenience is something you can add later, but fundamentals are hard to change

Cora (she/her)20:12:34

so you'd need to pull in that dep some other way before doing

bb --classpath "$(clojure -Spath)" "(require '[clj-http.lite.client :as client]) (:status (client/get \"\"))"
?

borkdude20:12:55

clojure will pull it for you if you do that

Cora (she/her)20:12:08

ahhh if you have a deps.edn

Cora (she/her)20:12:12

this is definitely understandable because of compatability

Cora (she/her)20:12:15

I guess the impulse for me is to have the script as self-contained as possible

Cora (she/her)20:12:40

but I guess it still has to install the deps so it's not actually self-contained

Cora (she/her)20:12:56

but it depends on clojure also being available

borkdude20:12:43

@corasaurus-hex You're free to obtain the git dep in any other way and set the classpath to the src folder

borkdude20:12:19

but that only works if that deps doesn't have any other deps. and dependency management is not a problem I want to re-invent the wheel for

Cora (she/her)20:12:37

it's definitely out of scope

borkdude20:12:03

@laheadle Following the example of clj-http-lite here: https://github.com/borkdude/clj-http-lite/blob/master/Readme.md#babashka I guess you could publish your curl-wrapping code also as a babashka library

👍 4
borkdude20:12:38

This way we could have multiple competing approaches for the same problem. Over time we can see what makes sense to include in bb itself.

👍 4
borkdude22:12:39

@corasaurus-hex What about something like this?

#!/usr/bin/env bb

(def deps '{:deps
            {clj-http-lite
             {:git/url ""
              :sha "f44ebe45446f0f44f2b73761d102af3da6d0a13e"}}})

(require '[clojure.java.shell :as shell])

(def classpath (:out (shell/sh "clojure" "-Sdeps" (pr-str deps) "-Spath")))

(add-to-classpath! classpath)

(require '[clj-http.lite.client :as client])

(:status (client/get "")) ;; 200

Cora (she/her)22:12:29

that's cool but it also still depends on clojure itself being available

borkdude22:12:59

that's just a detail here

Cora (she/her)22:12:16

like if I wanted to ship bb and a script in a docker container I'd also need to include the jvm and clojure

Cora (she/her)22:12:42

which is actually the use case here

Cora (she/her)22:12:07

I'm trying to sell my coworkers on this but a lot of the scripting we do is in docker containers for setup

borkdude22:12:31

You could just ship the library in the docker container and do (add-to-classpath! "/some/dir/with/clj-http-lite/src"). Without needing clojure.

borkdude22:12:02

or use git clone, or whatever, that's not really the point here

borkdude22:12:46

But then again, you could just use --classpath in that case

Cora (she/her)22:12:00

it definitely solves the problem of including a deps.edn, which I very much appreciate

Cora (she/her)22:12:43

definitely separate issues

Cora (she/her)22:12:17

but my original intent was shipping a script and a bb binary in a container and just being able to run it standalone

Cora (she/her)22:12:41

both worthy of being solved

Cora (she/her)22:12:54

but definitely not at the expense of re-solving package management

borkdude22:12:33

shipping bb + a script in a container and running it standalone is already possible today

borkdude22:12:46

if you mean Docker container

Cora (she/her)22:12:19

but pulling in deps requires including clojure+jvm or pre-pulling those deps and copying them over?

borkdude22:12:21

btw, lol:

#!/usr/bin/env bb

(def deps '{:deps
            {clj-http-lite
             {:git/url ""
              :sha "f44ebe45446f0f44f2b73761d102af3da6d0a13e"}}})

(require '[clojure.java.shell :as shell])

(def classpath (:out (shell/sh "clojure" "-Sdeps" (pr-str deps) "-Spath")))

(def script '(do (require '[clj-http.lite.client :as client])
                 (:status (client/get ""))))

(print (:out (shell/sh "bb" "--classpath" classpath (str script)))) ;; prints 200

borkdude22:12:42

yeah, pre-pull them and copy them over, as part of the image building

Cora (she/her)22:12:27

part of my thing is convincing coworkers to use this over bash and so reducing the extra work is a priority for me

Cora (she/her)22:12:57

it's definitely doable, particularly if I script the deps stuff so it's copy/paste

borkdude22:12:20

The joker author told me he never wrote a multi-file script himself so far btw, so this is probably a bit speculative still

Cora (she/her)22:12:44

yeah, we only have a datum of one here

Cora (she/her)22:12:04

my desires might be way out there

borkdude22:12:21

it's good to aim high. we have at least one public babashka library now

borkdude22:12:28

there might be more coming

Cora (she/her)22:12:43

there's probably a place for a babashka package manager. installing clojure packages and getting an executable for the installed package is a bit of a pain. people rely on homebrew and other package managers currently. in a scripting context it might make sense for there to be a package manager focused on providing executable binaries like in npm/npm or rubygems or pip

Cora (she/her)22:12:19

I don't know that it should be part of babashka but having a package manager is definitely aiming high

borkdude22:12:18

it seems there is a package manager for bash here: https://www.bpkg.sh/

👀 8
Cora (she/her)22:12:21

the first library here is pointing towards a useful niche, at least

Cora (she/her)22:12:48

oh interesting

Cora (she/her)22:12:14

it would be extremely clojure to rely on another ecosystem's package manager 😉

borkdude22:12:03

maybe we can adopt guix, so I can learn something new 😛

borkdude22:12:41

but I guess there is no way around installing a package manager before using it

Cora (she/her)22:12:46

sprinkle some nix on it

borkdude22:12:56

guix isn't nix btw, it's a different but similar project

Cora (she/her)22:12:06

there's a way around it but it's an impractical amount of effort for one person

Cora (she/her)22:12:19

for sure, but there's a similar spirit imo

borkdude22:12:28

I mean you either use clojure / lein to pull deps, or require people to install npm, rubygems, bpkg, whatever. I guess you can just choose it for yourself and your own scripts what makes sense. If your company is heavy on JS, just put some scripts in npm?

Cora (she/her)22:12:45

I've actually wanted to make a clojure package manager focused on providing access to executable binaries, for some time now

Cora (she/her)22:12:38

if it could provide installers for clojure, lein, etc, it would be really nice

borkdude22:12:00

what is it?

Cora (she/her)22:12:20

I haven't made it :)

Cora (she/her)22:12:32

it's a bit of a passing ambition currently

borkdude22:12:37

these tools already have good installers

Cora (she/her)22:12:07

but many package managers in scripting langs provide for adding executable files on your path

Cora (she/her)22:12:21

and I feel like clojure lacking that is a limiting factor

sogaiu22:12:47

guix is great -- but iirc there is one aspect to it that one should be aware of.

sogaiu22:12:49

the last time i worked with it, i recall them being very oriented toward free software only. that can be a problem for some folks who want to also use other bits. nixos on the other hand, doesn't have that kind of policy -- but last i checked they have a different significant issue.

sogaiu22:12:05

last i checked, to use nixos requires learning a pretty not-so-nice language. guix on the other hand gives you guile.

borkdude22:12:24

we need gunix

sogaiu22:12:50

lol. i've run systems with both for some time -- but that was a bit back. it's possible things have evolved / changed a bit.

Cora (she/her)22:12:48

they do, but they have to lean on N other package managers for each tool

borkdude22:12:31

@corasaurus-hex Don't agree. Leiningen has a bash script to install itself and so has clojure.

Cora (she/her)22:12:11

if all the packages that need executable files could lean on one package manager then it would save work for them. we could consolidate effort around making that package manager installable on each platform

borkdude22:12:41

Which is currently mostly brew (crossplatform for linux and mac) and maybe scoop for Windows

sogaiu22:12:33

i'm interested in the reality of how many folks use brew for linux-ish systems -- and whether that is growing.

borkdude22:12:38

But maybe we're getting a bit off topic. Babashka isn't going to provide a binary packagement system, nor will it probably invent its own dependency management system.

Cora (she/her)22:12:00

yeah for sure, it's off topic

Cora (she/her)22:12:44

I just see all these scripting languages providing bin files on your path and how much it expands their ability to provide certain tools

Cora (she/her)22:12:19

if every ruby library had to navigate adding bin files via whatever package managers it would have seriously limited its growth

borkdude22:12:00

Can you give an example of a ruby lib that needs a specific binary on the path?

Cora (she/her)22:12:27

rails, bundler, spring, zeus, etc

Cora (she/her)22:12:46

a hugely successful ecosystem build around providing the convenience of an executable on your path

Cora (she/her)22:12:05

not a binary, per se, but something that's executable

Cora (she/her)22:12:15

and you just need to work out installing rubygems and you're set

borkdude22:12:16

ah like npm also does, right?

Cora (she/her)22:12:28

and I think this really limits clojure

Cora (she/her)22:12:42

there are so many useful tools it could provide. it has to contend with startup time, though

Cora (she/her)22:12:48

which is where babashka comes in

borkdude22:12:02

maybe this is because CLIs weren't really an option before GraalVM, joker, lumo etc

Cora (she/her)22:12:33

oh definitely, it's just not practical for every execution to take a minimum of 800ms to start

borkdude22:12:42

I guess we can lean on another package manager for now, which could maybe be npm

Cora (she/her)22:12:43

but there's so much possibility here

borkdude22:12:02

I don't know if the npm folks like having other stuff than JS in there

Cora (she/her)22:12:06

definitely could. I'm not sure what a good solution is

Cora (she/her)22:12:25

just that once it's out there it opens up a whole world of tools that weren't practical in clojure

Cora (she/her)22:12:43

no one wants to run "clojure -arg sdkfjdkfj -e foo/main" to run a simple thing

Cora (she/her)22:12:30

one of the things that rails taught people was that ergonomics matter. we've seen it proven in elixir taking off so much.

Cora (she/her)22:12:48

and now we have ergonomics within reach, someone just needs to reach out and grab it

borkdude22:12:13

brew is also kinda nice. I guess one could write a babashka script and put it in brew with a dependency on babashka

borkdude22:12:23

and you can brew install and run it

Cora (she/her)22:12:38

but to get it into other package managers as well is a pain

Cora (she/her)22:12:42

good luck getting it into debian

Cora (she/her)22:12:55

but rubygems is installable basically everywhere

Cora (she/her)22:12:03

and then every other library can lean on that

borkdude22:12:10

brew works on linux too. debian packages are a pain in general

Cora (she/her)22:12:24

it's focusing the energy of getting things into package managers into a single thing

borkdude22:12:29

sure, why not. I don't know the ruby ecosystem very well

Cora (she/her)22:12:04

well, like getting npm installed is easier than getting your random npm package added to debian

Cora (she/her)22:12:08

or other package managers

Cora (she/her)22:12:23

I get that there are solutions here

Cora (she/her)22:12:38

it's not about current stuff not working as much as the possibility of making things better

borkdude22:12:13

the question is: does one make things better by inventing yet another package manager

Cora (she/her)22:12:29

in this case I think yes

Cora (she/her)22:12:06

and really whatever comes of it should lean super hard on maven et al

Cora (she/her)22:12:12

there will still be a place for installing tools explicitly, like on debian how you can add explicit ruby or node libraries via their own package manager

Cora (she/her)22:12:18

but the smaller tools aren't left hanging

Cora (she/her)22:12:50

I can apt-get install ruby-rails or whatever but that shouldn't preclude all the smaller libraries

Cora (she/her)22:12:45

there's friction here, is what I mean. if I want to use certain libraries I have to make my own interface for it because it's too much work for them to add cli interfaces and add them to all the relevant package managers.

Cora (she/her)22:12:05

it would be like your OS not providing a package manager that added files to your path

sogaiu22:12:16

have you tried nixos or guix?

Cora (she/her)22:12:37

I've tried nix but gave up because it was a lot of work on stuff I didn't much care to work on

Cora (she/her)22:12:45

I really like the ideas there

sogaiu22:12:33

similar -- it was way too much work. really cool stuff, but i didn't have the time or computing resources to keep up.

borkdude22:12:48

Same here, tinkered with nix package manager, but nothing more than that

sogaiu22:12:06

my uneasy management of multiple versions of things is done via things like nvm and the unfortunate java switching system provided in arch linux. some day may be some of this will be more sane.

Cora (she/her)22:12:27

I like asdf for most things

Cora (she/her)22:12:42

but the java thing sucks so I use sdkman for it

sogaiu22:12:37

i've looked at that a tiny bit -- but since it didn't handle java (or not well?) and i don't use most of the rest of its bits, using asdf + sdkman didn't seem like a win for me. if i used some of the other provided bits, the story might be different.

Cora (she/her)22:12:00

sdkman by itself is brilliant

Cora (she/her)22:12:44

I have to manage a lot of python and ruby and node and elixir versions and asdf works great for those

borkdude23:12:02

I've been using jenv for switching java versions

sogaiu23:12:40

hadn't heard of jenv -- will look at that. may be i'll look at sdkman again.

Cora (she/her)23:12:01

does jenv install versions?

borkdude23:12:17

no, but you can register versions and it detects installed ones and switch more easily between them

borkdude23:12:24

I use AdoptOpenJDK for installing

Cora (she/her)23:12:52

I'm lazy af so installers is crucial. good to know there's another thing to provide it

borkdude23:12:35

e.g. I can also register my downloaded Graal JVM in ~/Downloads with jenv and switch to that, per shell session, directory, system, etc

borkdude23:12:10

and if I throw it away, I just type jenv remove ...

Cora (she/her)23:12:47

you can install that via sdkman too

Cora (she/her)23:12:03

and switch for just the shell or make it a default

Cora (she/her)23:12:22

I was having a time at it earlier in trying to get the babashka tests running

Cora (she/her)23:12:37

I couldn't actually get them to run

Cora (she/her)23:12:50

well they ran but the main tests froze

Cora (she/her)23:12:10

I walked away in case it was just slow but hours later it was still stuck

Cora (she/her)23:12:17

I was trying to add Paths

borkdude23:12:40

if it freezes, that's probably the wait-for test.

Cora (she/her)23:12:58

it's probably something about my env

borkdude23:12:01

did you maybe have a socket REPL running?

Cora (she/her)23:12:24

I'm not sure? let me get to a computer quick

borkdude23:12:29

and what's your python version when you type python?

borkdude23:12:57

We currently use python to start a webserver quickly during the tests, but it assumes v2 and not v3

borkdude23:12:20

This may change in the future, if someone has a better idea of doing this

sogaiu23:12:39

the python issue may be more likely on linux boxes

sogaiu23:12:08

apple went against the pep recommendation in their naming of python binaries afaict

borkdude23:12:44

fwiw these tests run on both linux and mac on circleci

Cora (she/her)23:12:58

maybe that's it

borkdude23:12:33

one idea that I had was to just skip the python-required tests if python is not installed or is not v2

Cora (she/her)23:12:08

I use asdf for python so it's easy to switch it out

sogaiu23:12:14

> We expect Unix-like software distributions (including systems like macOS and Cygwin) to install the python2 command into the default path whenever a version of the Python 2 interpreter is installed, and the same for python3 and the Python 3 interpreter. > When invoked, python2 should run some version of the Python 2 interpreter, and python3 should run some version of the Python 3 interpreter. so if macos has python2, according to the recommendations, it ought to have python2 as a name for it -- do you think that is a reasonable interpretation of what's written above?

Cora (she/her)23:12:58

heyyyy look at that

Cora (she/her)23:12:00

it's running

👍 4
Cora (she/her)23:12:10

they ran successfully

borkdude23:12:17

python2 works on my system

Cora (she/her)23:12:29

you're running an http server with it?

sogaiu23:12:56

so if the tests use python2, then using python2 for the name doesn't seem too bad?

Cora (she/her)23:12:57

there's a trick to this where you can switch on which version it is

Cora (she/her)23:12:21

python3 -m http.server

borkdude23:12:31

feel free to try python2 in the tests, and see if that works on circleci as well

Cora (she/her)23:12:41

or

python -m http.server 8000 --bind 127.0.0.1

Cora (she/her)23:12:20

oh this is interesting

Cora (she/her)23:12:22

python -m $(python -c 'import sys; print("http.server" if sys.version_info[:2] > (2,7) else "SimpleHTTPServer")')

sogaiu23:12:23

i modify my local test to use python2 (the name) pretty frequently 🙂 will have to see if it works on circleci

borkdude23:12:28

one test is using python2 http server for testing if a socket is closed properly. since that server is single-threaded, it was blocking when a socket wasn't closed before

borkdude23:12:53

I don't know if the server in python3 is multi-threaded

Cora (she/her)23:12:27

$ python -V
Python 3.7.2
0 ~/Code/babashka (master=)
$ python -m $(python -c 'import sys; print("http.server" if sys.version_info[:2] > (2,7) else "SimpleHTTPServer")')
Serving HTTP on 0.0.0.0 port 8000 () ...
^C
Keyboard interrupt received, exiting.

borkdude23:12:29

but if there's another way to test this, feel free to refactor out the python stuff

Cora (she/her)23:12:50

I think requiring a python version is fine

Cora (she/her)23:12:58

just might be worth making that explicit somewhere

Cora (she/her)23:12:21

I could make a change to the README?

borkdude23:12:47

but maybe first making the python2 change is good so we can add that to the readme?

Cora (she/her)23:12:48

like "NOTE: requires python 2"

Cora (she/her)23:12:04

making the python2 change?

borkdude23:12:16

yeah, change python to python2 in the tests

borkdude23:12:23

and then see if that works on circleci

borkdude23:12:55

so then we can document that the user should have a system where python2 works

Cora (she/her)23:12:25

it borks in asdf on mac

130 ~/Code/babashka (master=)
$ python2 -m SimpleHTTPServer
asdf: No version set for command python2
you might want to add one of the following in your .tool-versions file:

python 2.7.17

Cora (she/her)23:12:51

probably want to do something like test for python2 then fall back to python?

Cora (she/her)23:12:58

maybe run python -V

Cora (she/her)23:12:02

and check the version

Cora (she/her)23:12:09

all doable 🙂

borkdude23:12:18

PR welcome for that as well

Cora (she/her)23:12:23

I've been drinking and making cookies and am in no state to change that right now

Cora (she/her)23:12:30

but I'm setting a reminder for tomorrow 🙂

Cora (she/her)23:12:07

if we're switching on version we can probably just support python 2 vs 3

borkdude23:12:12

so: trying python2, if that doesn't work, try python -V and check version, if that isn't the right version, print a warning and skip the tests that require python 2?

Cora (she/her)23:12:37

I mean it seems simple to allow for both versions?

Cora (she/her)23:12:51

just python -V | grep 'Python 3'

borkdude23:12:09

that would be fine by me, but like I said earlier: > I don't know if the server in python3 is multi-threaded

sogaiu23:12:10

fwiw, some searches suggest python 3's default web server is not multithreaded -- but i'm not sure if that will remain true (if it is atm)

Cora (she/her)23:12:24

it's python, it's all single threaded

borkdude23:12:29

ok then that's good

Cora (she/her)23:12:50

there was an attempt to remove the GIL but it failed

borkdude23:12:13

does this mean python apps suck for web development? I mean, you have to be able to serve two concurrent users?

sogaiu23:12:23

aren't there some thread-like things?

sogaiu23:12:27

there's twisted ofc

sogaiu23:12:31

but that's diff

Cora (she/her)23:12:55

there are other threads, and they always serve more than one thread, but only one executes at a given time

Cora (she/her)23:12:07

I'm pretty sure that's true of both python 2 and 3

Cora (she/her)23:12:33

it does suck for it, they use multiple processes to get around it

Cora (she/her)23:12:38

ruby and node have the same problem

sogaiu23:12:45

the important characteristic for the test was: > since that server is single-threaded, it was blocking when a socket wasn't closed before

Cora (she/her)23:12:11

I mean if I changed it to use python 3 and the test worked, is that acceptable?

Cora (she/her)23:12:29

or does it disguise some behavior?

Cora (she/her)23:12:43

unladen swallow

borkdude23:12:46

the test only fails when the socket is not closed AND the server is single threaded

Cora (she/her)23:12:49

that was the project to remove the python gil

borkdude23:12:16

so you'd have to check by not closing the socket and running python3, that should result in a failing test

Cora (she/her)23:12:45

class http.server.ThreadingHTTPServer(server_address, RequestHandlerClass)

    This class is identical to HTTPServer but uses threads to handle requests by using the ThreadingMixIn. This is useful to handle web browsers pre-opening sockets, on which HTTPServer would wait indefinitely.

borkdude23:12:30

is HTTPServer not around anymore in python3?

Cora (she/her)23:12:54

it's renamed to http.server

Cora (she/her)23:12:00

instead of SimpleHTTPServer

Cora (she/her)23:12:18

and http.server includes more things

Cora (she/her)23:12:22

so it's not a simple rename

Cora (she/her)23:12:32

but you can basically switch on the version and run them two different ways

borkdude23:12:45

maybe we should just use something more internally to check this which simulates a single-threaded connection handler

Cora (she/her)23:12:08

I'd be down with that

Cora (she/her)23:12:14

is babashka capable of a socket server?

borkdude23:12:23

it has a socket repl

Cora (she/her)23:12:30

oh, that seems likely then 😉

Cora (she/her)23:12:48

I don't have experience making socket servers in java but have a ton of experience in ruby

Cora (she/her)23:12:51

how different could it be?

borkdude23:12:26

anyway, that's the idea of the test. I don't care how it's tested, python is just a detail here

borkdude23:12:32

gotta go now, getting late

👋 12