Fork me on GitHub
#babashka
<
2019-12-22
>
Nate Sutton04:12:54

hrm, it appears you need java.nio.file.Paths in order to contruct a Path object

Nate Sutton04:12:55

which is needed in order to call createSymbolicLink on java.nio.file.Files

Nate Sutton04:12:07

or at least that seems to be the case

holyjak11:12:30

@nate_clojurians I believe File has a toPath?

👍 1
Nate Sutton12:12:00

ahhh so it does, I'll try that!

Nate Sutton18:12:51

yuuuup it works

👍 2
borkdude19:12:30

@nate_clojurians @holyjak I didn't add java.nio.filePaths earlier because I never could figure out how to use it from Clojure:

$ clj -e '(java.nio.file.Paths/get (into-array String "src")))'
Execution error (IllegalArgumentException) at java.lang.reflect.Array/set (Array.java:-2).

borkdude19:12:14

but this seems to work now:

$ bb '(java.nio.file.Paths/get "/home/justin" (into-array [".lein" "profiles.clj"]))'
#object[sun.nio.fs.UnixPath 0x566ebefb "/home/justin/.lein/profiles.clj"]

👍 2
holyjak12:12:46

hm, I think into-array takes a seq while your 1st arg as "src" - you likely wanted ["src"]

borkdude19:12:28

(found it on StackOverflow)

Nate Sutton19:12:11

the file -> toPath thing works well enough for me 🙂

Nate Sutton19:12:38

oooooh good release!

Nate Sutton19:12:46

delay will be really nice

Nate Sutton19:12:34

and the current file in exceptions? 😚👌

Nate Sutton20:12:32

and Paths, of course 💜

Nate Sutton20:12:43

so, maybe a crazy idea, but what if babashka could also include a deps.edn-style comment at the top that will pull in deps when running with a shebang?

Nate Sutton20:12:18

you can manually do it with with something like this:

#!/bin/sh
#_() ;; -*- mode: clojure -*-
#_(

   #_DEPS is same format as deps.edn. Multiline is okay.
   DEPS='
   {:deps {me.raynes/fs {:mvn/version "1.4.6"}}}
   '

   #_You can put other options here
   OPTS='
   -J-Xms256m -J-Xmx256m -J-client
   '

exec clojure $OPTS -Sdeps "$DEPS" "$0" "[email protected]"
)

(prn *command-line-args*)

Nate Sutton20:12:44

but having a way to pull in deps in a comment would be super cool

borkdude20:12:26

@nate_clojurians Yeah, let's first see how this library approach pans out (e.g. the clj-http-lite one)

borkdude20:12:11

Note that you can set the babashka classpath in BABASHKA_CLASSPATH as well

borkdude20:12:22

and you can require things in BABASHKA_PRELOADS.

borkdude20:12:37

so that gives enough room to not include any non-standard deps.edn stuff I think

borkdude20:12:51

Adding convenience is something you can add later, but fundamentals are hard to change

Nate Sutton20:12:34

so you'd need to pull in that dep some other way before doing

bb --classpath "$(clojure -Spath)" "(require '[clj-http.lite.client :as client]) (:status (client/get \"\"))"
?

borkdude20:12:55

clojure will pull it for you if you do that

Nate Sutton20:12:08

ahhh if you have a deps.edn

Nate Sutton20:12:12

this is definitely understandable because of compatability

Nate Sutton20:12:15

I guess the impulse for me is to have the script as self-contained as possible

Nate Sutton20:12:40

but I guess it still has to install the deps so it's not actually self-contained

Nate Sutton20:12:56

but it depends on clojure also being available

borkdude20:12:43

@nate_clojurians You're free to obtain the git dep in any other way and set the classpath to the src folder

borkdude20:12:19

but that only works if that deps doesn't have any other deps. and dependency management is not a problem I want to re-invent the wheel for

Nate Sutton20:12:30

oh absolutely

Nate Sutton20:12:37

it's definitely out of scope

borkdude20:12:03

@laheadle Following the example of clj-http-lite here: https://github.com/borkdude/clj-http-lite/blob/master/Readme.md#babashka I guess you could publish your curl-wrapping code also as a babashka library

👍 1
borkdude20:12:38

This way we could have multiple competing approaches for the same problem. Over time we can see what makes sense to include in bb itself.

👍 1
borkdude22:12:39

@nate_clojurians What about something like this?

#!/usr/bin/env bb

(def deps '{:deps
            {clj-http-lite
             {:git/url ""
              :sha "f44ebe45446f0f44f2b73761d102af3da6d0a13e"}}})

(require '[clojure.java.shell :as shell])

(def classpath (:out (shell/sh "clojure" "-Sdeps" (pr-str deps) "-Spath")))

(add-to-classpath! classpath)

(require '[clj-http.lite.client :as client])

(:status (client/get "")) ;; 200

Nate Sutton22:12:29

that's cool but it also still depends on clojure itself being available

borkdude22:12:59

that's just a detail here

Nate Sutton22:12:16

like if I wanted to ship bb and a script in a docker container I'd also need to include the jvm and clojure

Nate Sutton22:12:42

which is actually the use case here

Nate Sutton22:12:07

I'm trying to sell my coworkers on this but a lot of the scripting we do is in docker containers for setup

borkdude22:12:31

You could just ship the library in the docker container and do (add-to-classpath! "/some/dir/with/clj-http-lite/src"). Without needing clojure.

borkdude22:12:02

or use git clone, or whatever, that's not really the point here

borkdude22:12:46

But then again, you could just use --classpath in that case

Nate Sutton22:12:00

it definitely solves the problem of including a deps.edn, which I very much appreciate

Nate Sutton22:12:43

definitely separate issues

Nate Sutton22:12:17

but my original intent was shipping a script and a bb binary in a container and just being able to run it standalone

Nate Sutton22:12:41

both worthy of being solved

Nate Sutton22:12:54

but definitely not at the expense of re-solving package management

borkdude22:12:33

shipping bb + a script in a container and running it standalone is already possible today

borkdude22:12:46

if you mean Docker container

Nate Sutton22:12:19

but pulling in deps requires including clojure+jvm or pre-pulling those deps and copying them over?

borkdude22:12:21

btw, lol:

#!/usr/bin/env bb

(def deps '{:deps
            {clj-http-lite
             {:git/url ""
              :sha "f44ebe45446f0f44f2b73761d102af3da6d0a13e"}}})

(require '[clojure.java.shell :as shell])

(def classpath (:out (shell/sh "clojure" "-Sdeps" (pr-str deps) "-Spath")))

(def script '(do (require '[clj-http.lite.client :as client])
                 (:status (client/get ""))))

(print (:out (shell/sh "bb" "--classpath" classpath (str script)))) ;; prints 200

borkdude22:12:42

yeah, pre-pull them and copy them over, as part of the image building

Nate Sutton22:12:27

part of my thing is convincing coworkers to use this over bash and so reducing the extra work is a priority for me

Nate Sutton22:12:57

it's definitely doable, particularly if I script the deps stuff so it's copy/paste

borkdude22:12:20

The joker author told me he never wrote a multi-file script himself so far btw, so this is probably a bit speculative still

Nate Sutton22:12:44

yeah, we only have a datum of one here

Nate Sutton22:12:04

my desires might be way out there

borkdude22:12:21

it's good to aim high. we have at least one public babashka library now

borkdude22:12:28

there might be more coming

Nate Sutton22:12:43

there's probably a place for a babashka package manager. installing clojure packages and getting an executable for the installed package is a bit of a pain. people rely on homebrew and other package managers currently. in a scripting context it might make sense for there to be a package manager focused on providing executable binaries like in npm/npm or rubygems or pip

Nate Sutton22:12:19

I don't know that it should be part of babashka but having a package manager is definitely aiming high

borkdude22:12:18

it seems there is a package manager for bash here: https://www.bpkg.sh/

👀 2
Nate Sutton22:12:21

the first library here is pointing towards a useful niche, at least

Nate Sutton22:12:48

oh interesting

Nate Sutton22:12:14

it would be extremely clojure to rely on another ecosystem's package manager 😉

borkdude22:12:03

maybe we can adopt guix, so I can learn something new 😛

borkdude22:12:41

but I guess there is no way around installing a package manager before using it

Nate Sutton22:12:46

sprinkle some nix on it

borkdude22:12:56

guix isn't nix btw, it's a different but similar project

Nate Sutton22:12:06

there's a way around it but it's an impractical amount of effort for one person

Nate Sutton22:12:19

for sure, but there's a similar spirit imo

borkdude22:12:28

I mean you either use clojure / lein to pull deps, or require people to install npm, rubygems, bpkg, whatever. I guess you can just choose it for yourself and your own scripts what makes sense. If your company is heavy on JS, just put some scripts in npm?

Nate Sutton22:12:45

I've actually wanted to make a clojure package manager focused on providing access to executable binaries, for some time now

Nate Sutton22:12:38

if it could provide installers for clojure, lein, etc, it would be really nice

borkdude22:12:00

what is it?

Nate Sutton22:12:20

I haven't made it :)

Nate Sutton22:12:32

it's a bit of a passing ambition currently

borkdude22:12:37

these tools already have good installers

Nate Sutton22:12:07

but many package managers in scripting langs provide for adding executable files on your path

Nate Sutton22:12:21

and I feel like clojure lacking that is a limiting factor

sogaiu22:12:47

guix is great -- but iirc there is one aspect to it that one should be aware of.

sogaiu22:12:49

the last time i worked with it, i recall them being very oriented toward free software only. that can be a problem for some folks who want to also use other bits. nixos on the other hand, doesn't have that kind of policy -- but last i checked they have a different significant issue.

sogaiu22:12:05

last i checked, to use nixos requires learning a pretty not-so-nice language. guix on the other hand gives you guile.

borkdude22:12:24

we need gunix

sogaiu22:12:50

lol. i've run systems with both for some time -- but that was a bit back. it's possible things have evolved / changed a bit.

Nate Sutton22:12:48

they do, but they have to lean on N other package managers for each tool

borkdude22:12:31

@nate_clojurians Don't agree. Leiningen has a bash script to install itself and so has clojure.

Nate Sutton22:12:11

if all the packages that need executable files could lean on one package manager then it would save work for them. we could consolidate effort around making that package manager installable on each platform

borkdude22:12:41

Which is currently mostly brew (crossplatform for linux and mac) and maybe scoop for Windows

sogaiu22:12:33

i'm interested in the reality of how many folks use brew for linux-ish systems -- and whether that is growing.

borkdude22:12:38

But maybe we're getting a bit off topic. Babashka isn't going to provide a binary packagement system, nor will it probably invent its own dependency management system.

Nate Sutton22:12:00

yeah for sure, it's off topic

Nate Sutton22:12:44

I just see all these scripting languages providing bin files on your path and how much it expands their ability to provide certain tools

Nate Sutton22:12:19

if every ruby library had to navigate adding bin files via whatever package managers it would have seriously limited its growth

borkdude22:12:00

Can you give an example of a ruby lib that needs a specific binary on the path?

Nate Sutton22:12:27

rails, bundler, spring, zeus, etc

Nate Sutton22:12:46

a hugely successful ecosystem build around providing the convenience of an executable on your path

Nate Sutton22:12:05

not a binary, per se, but something that's executable

Nate Sutton22:12:15

and you just need to work out installing rubygems and you're set

borkdude22:12:16

ah like npm also does, right?

Nate Sutton22:12:28

and I think this really limits clojure

Nate Sutton22:12:42

there are so many useful tools it could provide. it has to contend with startup time, though

Nate Sutton22:12:48

which is where babashka comes in

borkdude22:12:02

maybe this is because CLIs weren't really an option before GraalVM, joker, lumo etc

Nate Sutton22:12:33

oh definitely, it's just not practical for every execution to take a minimum of 800ms to start

borkdude22:12:42

I guess we can lean on another package manager for now, which could maybe be npm

Nate Sutton22:12:43

but there's so much possibility here

borkdude22:12:02

I don't know if the npm folks like having other stuff than JS in there

Nate Sutton22:12:06

definitely could. I'm not sure what a good solution is

Nate Sutton22:12:25

just that once it's out there it opens up a whole world of tools that weren't practical in clojure

Nate Sutton22:12:43

no one wants to run "clojure -arg sdkfjdkfj -e foo/main" to run a simple thing

Nate Sutton22:12:30

one of the things that rails taught people was that ergonomics matter. we've seen it proven in elixir taking off so much.

Nate Sutton22:12:48

and now we have ergonomics within reach, someone just needs to reach out and grab it

borkdude22:12:13

brew is also kinda nice. I guess one could write a babashka script and put it in brew with a dependency on babashka

borkdude22:12:23

and you can brew install and run it

Nate Sutton22:12:38

but to get it into other package managers as well is a pain

Nate Sutton22:12:42

good luck getting it into debian

Nate Sutton22:12:55

but rubygems is installable basically everywhere

Nate Sutton22:12:03

and then every other library can lean on that

borkdude22:12:10

brew works on linux too. debian packages are a pain in general

Nate Sutton22:12:24

it's focusing the energy of getting things into package managers into a single thing

borkdude22:12:29

sure, why not. I don't know the ruby ecosystem very well

Nate Sutton22:12:04

well, like getting npm installed is easier than getting your random npm package added to debian

Nate Sutton22:12:08

or other package managers

Nate Sutton22:12:23

I get that there are solutions here

Nate Sutton22:12:38

it's not about current stuff not working as much as the possibility of making things better

borkdude22:12:13

the question is: does one make things better by inventing yet another package manager

Nate Sutton22:12:29

in this case I think yes

Nate Sutton22:12:06

and really whatever comes of it should lean super hard on maven et al

Nate Sutton22:12:12

there will still be a place for installing tools explicitly, like on debian how you can add explicit ruby or node libraries via their own package manager

Nate Sutton22:12:18

but the smaller tools aren't left hanging

Nate Sutton22:12:50

I can apt-get install ruby-rails or whatever but that shouldn't preclude all the smaller libraries

Nate Sutton22:12:45

there's friction here, is what I mean. if I want to use certain libraries I have to make my own interface for it because it's too much work for them to add cli interfaces and add them to all the relevant package managers.

Nate Sutton22:12:57

it's limiting

Nate Sutton22:12:05

it would be like your OS not providing a package manager that added files to your path

sogaiu22:12:16

have you tried nixos or guix?

Nate Sutton22:12:37

I've tried nix but gave up because it was a lot of work on stuff I didn't much care to work on

Nate Sutton22:12:45

I really like the ideas there

sogaiu22:12:33

similar -- it was way too much work. really cool stuff, but i didn't have the time or computing resources to keep up.

borkdude22:12:48

Same here, tinkered with nix package manager, but nothing more than that

sogaiu22:12:06

my uneasy management of multiple versions of things is done via things like nvm and the unfortunate java switching system provided in arch linux. some day may be some of this will be more sane.

Nate Sutton22:12:27

I like asdf for most things

Nate Sutton22:12:42

but the java thing sucks so I use sdkman for it

sogaiu22:12:37

i've looked at that a tiny bit -- but since it didn't handle java (or not well?) and i don't use most of the rest of its bits, using asdf + sdkman didn't seem like a win for me. if i used some of the other provided bits, the story might be different.

Nate Sutton22:12:00

sdkman by itself is brilliant

Nate Sutton22:12:44

I have to manage a lot of python and ruby and node and elixir versions and asdf works great for those

borkdude23:12:02

I've been using jenv for switching java versions

sogaiu23:12:40

hadn't heard of jenv -- will look at that. may be i'll look at sdkman again.

Nate Sutton23:12:01

does jenv install versions?

borkdude23:12:17

no, but you can register versions and it detects installed ones and switch more easily between them

borkdude23:12:24

I use AdoptOpenJDK for installing

Nate Sutton23:12:52

I'm lazy af so installers is crucial. good to know there's another thing to provide it

borkdude23:12:35

e.g. I can also register my downloaded Graal JVM in ~/Downloads with jenv and switch to that, per shell session, directory, system, etc

borkdude23:12:10

and if I throw it away, I just type jenv remove ...

Nate Sutton23:12:47

you can install that via sdkman too

Nate Sutton23:12:03

and switch for just the shell or make it a default

Nate Sutton23:12:22

I was having a time at it earlier in trying to get the babashka tests running

Nate Sutton23:12:37

I couldn't actually get them to run

Nate Sutton23:12:50

well they ran but the main tests froze

Nate Sutton23:12:10

I walked away in case it was just slow but hours later it was still stuck

Nate Sutton23:12:17

I was trying to add Paths

borkdude23:12:40

if it freezes, that's probably the wait-for test.

Nate Sutton23:12:58

it's probably something about my env

borkdude23:12:01

did you maybe have a socket REPL running?

Nate Sutton23:12:24

I'm not sure? let me get to a computer quick

borkdude23:12:29

and what's your python version when you type python?

borkdude23:12:57

We currently use python to start a webserver quickly during the tests, but it assumes v2 and not v3

borkdude23:12:20

This may change in the future, if someone has a better idea of doing this

sogaiu23:12:39

the python issue may be more likely on linux boxes

sogaiu23:12:08

apple went against the pep recommendation in their naming of python binaries afaict

borkdude23:12:44

fwiw these tests run on both linux and mac on circleci

Nate Sutton23:12:58

maybe that's it

borkdude23:12:33

one idea that I had was to just skip the python-required tests if python is not installed or is not v2

Nate Sutton23:12:08

I use asdf for python so it's easy to switch it out

sogaiu23:12:14

> We expect Unix-like software distributions (including systems like macOS and Cygwin) to install the python2 command into the default path whenever a version of the Python 2 interpreter is installed, and the same for python3 and the Python 3 interpreter. > When invoked, python2 should run some version of the Python 2 interpreter, and python3 should run some version of the Python 3 interpreter. so if macos has python2, according to the recommendations, it ought to have python2 as a name for it -- do you think that is a reasonable interpretation of what's written above?

Nate Sutton23:12:58

heyyyy look at that

Nate Sutton23:12:00

it's running

👍 1
Nate Sutton23:12:10

they ran successfully

borkdude23:12:17

python2 works on my system

Nate Sutton23:12:29

you're running an http server with it?

sogaiu23:12:56

so if the tests use python2, then using python2 for the name doesn't seem too bad?

Nate Sutton23:12:57

there's a trick to this where you can switch on which version it is

Nate Sutton23:12:21

python3 -m http.server

borkdude23:12:31

feel free to try python2 in the tests, and see if that works on circleci as well

Nate Sutton23:12:41

or

python -m http.server 8000 --bind 127.0.0.1

Nate Sutton23:12:20

oh this is interesting

Nate Sutton23:12:22

python -m $(python -c 'import sys; print("http.server" if sys.version_info[:2] > (2,7) else "SimpleHTTPServer")')

sogaiu23:12:23

i modify my local test to use python2 (the name) pretty frequently 🙂 will have to see if it works on circleci

borkdude23:12:28

one test is using python2 http server for testing if a socket is closed properly. since that server is single-threaded, it was blocking when a socket wasn't closed before

borkdude23:12:53

I don't know if the server in python3 is multi-threaded

Nate Sutton23:12:27

$ python -V
Python 3.7.2
0 ~/Code/babashka (master=)
$ python -m $(python -c 'import sys; print("http.server" if sys.version_info[:2] > (2,7) else "SimpleHTTPServer")')
Serving HTTP on 0.0.0.0 port 8000 () ...
^C
Keyboard interrupt received, exiting.

borkdude23:12:29

but if there's another way to test this, feel free to refactor out the python stuff

Nate Sutton23:12:50

I think requiring a python version is fine

Nate Sutton23:12:58

just might be worth making that explicit somewhere

Nate Sutton23:12:21

I could make a change to the README?

borkdude23:12:47

but maybe first making the python2 change is good so we can add that to the readme?

Nate Sutton23:12:48

like "NOTE: requires python 2"

Nate Sutton23:12:04

making the python2 change?

borkdude23:12:16

yeah, change python to python2 in the tests

borkdude23:12:23

and then see if that works on circleci

borkdude23:12:55

so then we can document that the user should have a system where python2 works

Nate Sutton23:12:25

it borks in asdf on mac

130 ~/Code/babashka (master=)
$ python2 -m SimpleHTTPServer
asdf: No version set for command python2
you might want to add one of the following in your .tool-versions file:

python 2.7.17

Nate Sutton23:12:51

probably want to do something like test for python2 then fall back to python?

Nate Sutton23:12:58

maybe run python -V

Nate Sutton23:12:02

and check the version

Nate Sutton23:12:09

all doable 🙂

borkdude23:12:18

PR welcome for that as well

Nate Sutton23:12:23

I've been drinking and making cookies and am in no state to change that right now

Nate Sutton23:12:30

but I'm setting a reminder for tomorrow 🙂

Nate Sutton23:12:07

if we're switching on version we can probably just support python 2 vs 3

borkdude23:12:12

so: trying python2, if that doesn't work, try python -V and check version, if that isn't the right version, print a warning and skip the tests that require python 2?

Nate Sutton23:12:37

I mean it seems simple to allow for both versions?

Nate Sutton23:12:51

just python -V | grep 'Python 3'

borkdude23:12:09

that would be fine by me, but like I said earlier: > I don't know if the server in python3 is multi-threaded

sogaiu23:12:10

fwiw, some searches suggest python 3's default web server is not multithreaded -- but i'm not sure if that will remain true (if it is atm)

Nate Sutton23:12:24

it's python, it's all single threaded

Nate Sutton23:12:29

there's a GIL

borkdude23:12:29

ok then that's good

Nate Sutton23:12:50

there was an attempt to remove the GIL but it failed

borkdude23:12:13

does this mean python apps suck for web development? I mean, you have to be able to serve two concurrent users?

sogaiu23:12:23

aren't there some thread-like things?

sogaiu23:12:27

there's twisted ofc

sogaiu23:12:31

but that's diff

Nate Sutton23:12:55

there are other threads, and they always serve more than one thread, but only one executes at a given time

Nate Sutton23:12:07

I'm pretty sure that's true of both python 2 and 3

Nate Sutton23:12:33

it does suck for it, they use multiple processes to get around it

Nate Sutton23:12:38

ruby and node have the same problem

sogaiu23:12:45

the important characteristic for the test was: > since that server is single-threaded, it was blocking when a socket wasn't closed before

Nate Sutton23:12:11

I mean if I changed it to use python 3 and the test worked, is that acceptable?

Nate Sutton23:12:29

or does it disguise some behavior?

Nate Sutton23:12:43

unladen swallow

borkdude23:12:46

the test only fails when the socket is not closed AND the server is single threaded

Nate Sutton23:12:49

that was the project to remove the python gil

borkdude23:12:16

so you'd have to check by not closing the socket and running python3, that should result in a failing test

Nate Sutton23:12:45

class http.server.ThreadingHTTPServer(server_address, RequestHandlerClass)

    This class is identical to HTTPServer but uses threads to handle requests by using the ThreadingMixIn. This is useful to handle web browsers pre-opening sockets, on which HTTPServer would wait indefinitely.

borkdude23:12:30

is HTTPServer not around anymore in python3?

Nate Sutton23:12:54

it's renamed to http.server

Nate Sutton23:12:00

instead of SimpleHTTPServer

Nate Sutton23:12:18

and http.server includes more things

Nate Sutton23:12:22

so it's not a simple rename

Nate Sutton23:12:32

but you can basically switch on the version and run them two different ways

borkdude23:12:45

maybe we should just use something more internally to check this which simulates a single-threaded connection handler

Nate Sutton23:12:08

I'd be down with that

Nate Sutton23:12:14

is babashka capable of a socket server?

borkdude23:12:23

it has a socket repl

Nate Sutton23:12:30

oh, that seems likely then 😉

Nate Sutton23:12:48

I don't have experience making socket servers in java but have a ton of experience in ruby

Nate Sutton23:12:51

how different could it be?

borkdude23:12:26

anyway, that's the idea of the test. I don't care how it's tested, python is just a detail here

borkdude23:12:32

gotta go now, getting late

👋 3