Fork me on GitHub
#babashka
<
2021-03-03
>
flowthing09:03:03

I see Babashka implements the clojure.main/with-bindings macro, but Babashka can't seem to resolve it:

user=> (require '[clojure.main :as main])
nil
user=> main/demunge
#object[babashka.impl.clojure.main$demunge 0x78db0375 "babashka.impl.clojure.main$demunge@78db0375"]
user=> main/repl
#object[babashka.main$fn__28956$fn__28957 0x4f9ed1e2 "babashka.main$fn__28956$fn__28957@4f9ed1e2"]
user=> main/with-bindings
Could not resolve symbol: main/with-bindings [at <repl>:4:1]
Do I need to do something special to require a macro? Or is it that they can't be required at all?

borkdude09:03:54

Babashka doesn't expose this macro to scripts right now I think

borkdude09:03:33

We can improve this if you need this

borkdude09:03:40

Post an issue in that case

flowthing09:03:00

All right, thanks! Will do.

flowthing09:03:24

Same with dynamic vars like *flush-on-newline* and *source-path*, I guess?

borkdude10:03:30

What's the difference between *source-path* and *file* ?

borkdude10:03:47

*source-path* has no docstring

flowthing10:03:07

Weirdly enough, *source-path* is just the file name of the source file. *file* is the full path.

borkdude10:03:33

maybe *source-path* is the relative path?

borkdude10:03:50

I've never really used or needed this

flowthing10:03:56

Well, it affects something in exception stack traces, I think, but I forget what exactly.

flowthing10:03:09

Either way, it's not very important, I think.

flowthing10:03:00

Ah, yeah, got it now. If you evaluate something without *source-path* bound, :trace in the exception map has e.g. [user$eval75800 invokeStatic "NO_SOURCE_FILE" 32]. However, if you bind it to the file name of the source file you're evaluating from, it has e.g. [user$eval76227 invokeStatic "user.clj" 32].

flowthing10:03:27

(`NO_SOURCE_FILE` instead of NO_SOURCE_PATH because of https://clojure.atlassian.net/browse/CLJ-1167)

borkdude10:03:28

The babashka traces only use *file*

Jakub Holý (HolyJak)11:03:02

Show time: Babashka-based better alternative to python -m http.server for serving files from a local directory: https://gist.github.com/holyjak/36c6284c047ffb7573e8a34399de27d8

👍 24
😎 3
borkdude11:03:27

Awesome :) I think if you will add #_" -*- mode: clojure; -*-" towards the top of the file, Github will understand the syntax (emacs as well)

3
borkdude11:03:07

Can you explain what is the "better than Python" part? :)

borkdude11:03:48

It seems navigating into directories doesn't work well yet.

borkdude11:03:08

This is in babashka/logo

borkdude11:03:45

Oh it does work, but there is no .. to navigate back :)

Jakub Holý (HolyJak)14:03:33

> Can you explain what is the "better than Python" part? 🙂 I can provide -d <dir> , am not limited to pwd

Jakub Holý (HolyJak)14:03:19

.. could be added, fo course 🙂 I am old fashioned and use ⬅️ in the browser 😉

Jakub Holý (HolyJak)14:03:14

(The ././ in your example could be fixed too but I feel it is good enough for the time being :))

Jakub Holý (HolyJak)14:03:58

BTW awesome work with bb, httpkit integration and babashka/fs!

borkdude14:03:10

Yes, it's coming together isn't it :)

💯 3
grazfather14:03:41

Hm, doesn’t work for me on osx

grazfather14:03:06

CallHistoryTransactions failed on newest bb

borkdude14:03:07

It does for me on macOS

grazfather14:03:32

Naw, i use python3 -m http.server all the time, iTerm is already in there. Unless you’re saying i’d have to grant it to bb specifically

borkdude14:03:28

That wouldn't surprise me

Jakub Holý (HolyJak)14:03:07

I am on OSX and have just upgraded to the latest bb, no issue. Big Sur 11.2.1.

grazfather14:03:15

I’ll try later on my non-work laptop 😉

Jakub Holý (HolyJak)19:03:55

@U04V15CAJ fs/glob seems to have problems with performance in same cases according to https://gist.github.com/holyjak/36c6284c047ffb7573e8a34399de27d8#gistcomment-3652518 Is that a known issue?

borkdude19:03:44

Repro welcome.

borkdude19:03:50

Not sure what goes on there

borkdude20:03:33

@U0522TWDA You can try: https://babashka.org/fs/babashka.fs.html#var-list-dir instead. But if there is a bug in glob, I'd like to know to get it fixed.

grazfather20:03:54

Hm, tried on my other laptop and it also doesn’t work.

grazfather20:03:47

Looks like it has trouble walking some directories

borkdude20:03:01

Might be related to the same issue reported under that gist

borkdude20:03:40

@U01KUQRDMLG Can you just try the same (babashka.fs/glob "." "*") expression from that dir?

borkdude20:03:00

bb -e '(babashka.fs/glob "." "*")'

grazfather21:03:38

Yeah, that hangs

grazfather21:03:04

or takes very long. I have a lot in my home dir

grazfather21:03:25

1: (babashka.fs/glob "." "*")
   ^--- Visiting /Users/graziano/./Library/Saved Application State/com.bitrock.appinstaller.savedState failed

Tomas Brejla22:03:11

I can confirm, it doesn't really freeze, it just seems to traverse all the nested directories/files, as strace confirms (see https://gist.github.com/holyjak/36c6284c047ffb7573e8a34399de27d8)

Tomas Brejla22:03:58

fs/list-dir works fine and is very quick

borkdude22:03:01

Weird, I did have an optimization for this when the pattern didn't contain any slashes. https://github.com/babashka/fs/blob/5b8cf66c4cc06bfc24615043c8f8c31f14321f2a/src/babashka/fs.cljc#L248

borkdude22:03:06

But there may be a bug

borkdude22:03:33

I recommend using list-dir for now then, if all you want is to list a dir :)

borkdude22:03:45

Found the issue with glob

borkdude22:03:40

^ @U01KUQRDMLG I think changing fs/glob in fs/list-dir should also fix it for you

borkdude22:03:48

^ @U0522TWDA Please adapt the gist before more people use it as is :)

borkdude22:03:04

Oh you already did, thanks!

Jakub Holý (HolyJak)22:03:41

Thank you for the quick fix to glob! 🙂

truestory 3
👏 3
grazfather03:03:36

Just tested, works well.

Tomas Brejla08:03:41

btw it also has fairly nice performance, compared to python's http.server. Here're some numbers, serving 7.5K index.html on my dell xps 13 with Intel(R) Core(TM) i7-8550U CPU @ 1.80GH

Running 10s test @ 
  8 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   206.34us  301.38us   9.88ms   98.08%
    Req/Sec     5.71k   428.99     6.81k    73.76%
  458684 requests in 10.10s, 3.29GB read
Requests/sec:  45414.91
Transfer/sec:    333.24MB
Here's the same with python3 -m http.server:
Running 10s test @ 
  8 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.88ms  428.80us   9.55ms   81.75%
    Req/Sec   507.72     99.10   808.00     90.30%
  33680 requests in 10.07s, 250.02MB read
Requests/sec:   3345.18
Transfer/sec:     24.83MB
Seems that this default python's http.server doesn't use threads (or at least not efficiently). Of course, bb version in way more memory hungry than the python one, but for simple native dev-server using generic compiled version of bb, it's still quite good. I also learned yesterday that you can actually pass -Xmxand similar flags to graalvm-natively compiled apps (therefore including bb). Didn't know that this was possible and yes, it works :thumbsup: (more info here https://www.graalvm.org/reference-manual/native-image/MemoryManagement/#memory-management-options)

zane19:03:41

Anyone in here editing Babashka scripts with clojure-lsp enabled? I’m curious what people are doing to inform clojure-lsp of e.g. Babashka’s built-ins.

ericdallo19:03:13

right now clojure-lsp does nothing as there is no deps.edn/project.clj file, but that's something we could have

ericdallo19:03:28

but we'd need to know somehow that's a bb script

ericdallo19:03:56

ATM, clojure-lsp check for deps.edn, project.clj, etc to get the classpath

ericdallo19:03:14

if we have some bb --classpath just like the other comands, that could work

ericdallo19:03:58

I think I discussed that with @U04V15CAJ in some thread in #lsp

zane19:03:51

> right now clojure-lsp does nothing as there is no deps.edn/project.clj file Got it. I have several deps.edn projets that also include Babashka scripts.

ericdallo19:03:47

Yeah, we need some reliable way to get the classpath, just like if user has a deps.edn file, we call clj -Spath to get the classpath

ericdallo19:03:03

but that's not always true for bb

zane19:03:36

Makes sense. I don’t see a clear way for clojure-lsp to get access to Babashka’s classpath (such as it is) given that Babashka is a native image. Perhaps a feature for writing out the classpath could be added to Babashka? Hmm.

ericdallo19:03:42

Yes, that was my idea, something like bb --classpath but we'd need from clojure-lsp when to run that

borkdude19:03:01

Or maybe clojure-lsp can have a pre-configured classpath that you can choose from a menu?

borkdude19:03:15

So people can add a deps.edn there with libs that occur in babashka

borkdude19:03:01

There are some ideas here. Needs hammock time. https://github.com/babashka/babashka/issues/733

ericdallo19:03:19

Interations with menus would need change on LSP clients, so I think it'd be better to only the server know somehow to get the classpath

ericdallo19:03:34

oh that's the issue, thanks

borkdude19:03:12

Oh yes, I sometimes forget there are other editors besides emacs

😆 3
borkdude19:03:45

So how would lsp know from only a script.clj that it should retrieve a classpath from babashka?

ericdallo19:03:35

yeah, that's the problem

ericdallo19:03:47

there is an idea

ericdallo19:03:32

user has a .lsp/config.edn file on the project root with some config maybe specifying the bb script files

ericdallo19:03:57

it'd still need a lot of changes on clojure-lsp specific to that

ericdallo19:03:14

a way to get the classpath following all the other build tools it'd be perfect

borkdude19:03:01

What if a user would have something like ;; *clojure-lsp-preset: babashka* in the script, e.g. at the end?

ericdallo19:03:53

still need a lot of changes 😕 as we scan the classpath on the LSP initialize request, that has only the project-root

ericdallo19:03:10

then clojure-lsp do its magic to check what kind of project it is

borkdude19:03:12

or #_{:clojure-lsp/classpath-cmd ["bb" ...]

zane19:03:14

> So how would lsp know from only a script.clj that it should retrieve a classpath from babashka? Many of my scripts have #!/usr/bin/env bb at the top, but I know that won’t always be the case. I imagine some people also use a .bb file extension.

ericdallo19:03:32

only after the initialize, that client send the didOpen for a specific file

ericdallo19:03:41

any ideas for this @U0BUV7XSA?

ericdallo19:03:09

Also, the initialize method is sent only when the server start, one time, the didOpen is sent for every file opened, we would need to track that somehow

ericdallo19:03:40

@U04V15CAJ maybe user could have a .bb-deps , then we could have a project spec on clojure-lsp of:

{:project-path ".bb-deps"
 :classpath-cmd ["bb" "--classpath"]}

ericdallo19:03:16

I know is not the ideal as scripts are kind of independent of other files

borkdude20:03:12

there is an issue about a bb.edn file, but most people just want a script without a bb.edn

borkdude20:03:31

anyway, if you add a deps.edn right now in your script dir, it should already work

borkdude20:03:37

maybe that's just the best solution

snoe20:03:24

is babashka clojure? maybe it is as simple as encouraging a different extension like cljs cljr and joker

borkdude20:03:57

yes, babashka scripts are compatible with clojure

borkdude20:03:32

and encouraging another extension: I don't personally like this, as syntax highlighting etc. don't work as nicely. just .clj

snoe20:03:02

ok, then it probably needs to be a client responsibility, our initializationOptions could take a classpath string and we just use that instead of running a project command if present.

ericdallo20:03:03

yes, good suggestion, we would still need to change emacs lsp-mode or vim client etc, but I can help with that from emacs side

ericdallo20:03:38

but the client would still need to know if the file is a bb script and need a way to get the classpath string, right @U0BUV7XSA?

snoe20:03:02

yeah, or they hardcode it to the babashka classpath they use most often and we use that when there's no project found

snoe20:03:10

something like that tho

ericdallo20:03:31

The built-in deps of bb can change I think, that's why a bb --classpath makes sense to me, WDYT @U04V15CAJ?

zane20:03:12

Getting clojure-lsp the right classpath is only half the battle, right? There’s also Babashka’s pre-defined built-in aliases.

ericdallo20:03:01

I don't know what is bb pre-defined aliases

borkdude20:03:12

@U050CT4HR I don't encourage to use those in scripts, that's mostly for one-liners

zane20:03:19

For instance, you can use the alias shell to refer to clojure.java.shell without explicitly requiring the alias.

borkdude20:03:25

Just add a require in scripts, even if you don't need one

snoe20:03:18

@UKFSJSM38 I don't think there's anyway to tell the difference between a babashka script and a clojure file so clojure-lsp calling the bb --classpath I don't think we can use

zane20:03:19

That’s what I’ve been doing, but clojure-lsp flagging usage of those aliases is probably not the expected behavior.

zane20:03:28

Maybe that’s fine. Just pointing it out.

borkdude20:03:39

It's actually clj-kondo flagging those aliases

zane20:03:51

Ah, right.

borkdude20:03:51

unless I misunderstand

zane20:03:14

No, you’re probably right.

zane20:03:10

But it’s still related to clojure-lsp in that go-to-definition, etc, won’t work on usages of such aliases.

ericdallo20:03:20

@U0BUV7XSA with the ;; *clojure-lsp-preset: babashka* suggestion from @U04V15CAJ we could know

ericdallo20:03:51

otherwise, how the client would know that need to send the bb --classpath ?

borkdude20:03:20

Are there other types of clojure files that this problem also holds for?

borkdude20:03:34

Maybe it can be solved in a more general way

ericdallo20:03:18

I don't know any other issue related to that, all build tools usually have a file for the deps and a command to get the classpath

snoe20:03:16

1) clojure-lsp is built around a single classpath for a project-root. since bb and the project would have separate cps I think it needs to be done carefully. So far we've gotten away with cljs and clj classpaths not colliding, but this is probably pointing us to a place where the classpaths get determined by source-paths. 2) The order of operations here gets messed up. We init and read classpaths and then we read the file, we don't want to read the file then read a classpath based on that. 3) for projects, I imagine a scripts/ dir could be created with some bb project file within it to indicate scripts are in a different project-root and they have a different classpath. 4) for standalone scripts, there's probably an lsp-command that could be made to re-crawl a given classpath. and like it'll be up to bb users to either write scripts to detect that it's babashka or run manually. even then, users are probably not mixing clj and bb files and the bb project file could be put in your ~/scripts/ dir or wherever you're writing.

borkdude20:03:09

sometimes people just put a script in the root, but I often put it in script if it's in a JVM clojure project

borkdude20:03:41

I don't think we can prescribe where people should put scripts

borkdude20:03:10

Just adding a deps.edn in the root of the script dir would solve it for now

borkdude20:03:29

This is a manual operation, but a sane workaround for now

👍 3
ericdallo20:03:16

Make sense, thanks for testing it @U04V15CAJ I think user still need to add the script as another project? to get a different project-root

borkdude21:03:36

I haven't actually tested it, but I think it should work ;)

👍 3
borkdude20:03:35

Query params middleware :)

borkdude22:03:19

I have a native compiled version of tools.deps.alpha here: https://github.com/borkdude/tools-deps-native-experiment I think I can make a babashka pod out of it. What would a tools.deps.alpha pod bring to babashka scripting? Think of it as being able to use the tools.deps.alpha library as you're used to on the JVM. You can do all kinds of crazy dependency-related stuff now in milliseconds.

awesome 9
borkdude23:03:04

But maybe the more exciting thing would be native deps downloading for babashka itself. Bye bye JVM startup time.

👍 6
nate23:03:03

that would be amazing! I've started using babashka.deps to include libraries and removing the jvm penalty for first runs would be great

nate23:03:27

feels like it could be a useful thing to have built-in to babashka

nate23:03:46

otherwise, we are replacing the jvm startup time with a pod download time

nate23:03:56

it would be less often, but still

borkdude23:03:17

I don't see this as going to be part of babashka, it would double the binary size

borkdude23:03:25

tools-deps-native is around 80mb

nate23:03:06

as a pod it would be downloaded once, so it's a much smaller wait than every time the deps change

nate23:03:26

awesome, great work!