Fork me on GitHub
#babashka
<
2020-04-24
>
rovanion08:04:50

Does anyone happen to know why I can't seem to (import '[clojure.lang.Agent]) and resolve clojure.lang.Agent/pooledExecutor?

borkdude08:04:55

@rovanion Because that isn't mapped in babashka yet. Can you explain why you would need this?

rovanion08:04:48

So the reason why I want to do this is to implement future but using a pooled executor instead of the standard cached executor. The normal cached executor will spawn 8000 threads if 8000 futures are created at the same time. The pooled executor gives me the ability to limit the number of threads spawned to not choke the kernel. Now I used this technique once in normal Clojure when I built a filesystem benchmark, I wanted to benchmark the filesystem and not the kernels ability to juggle threads, but I can totally live with spawning a ton of threads for the application I'm writing now. So no need to map it in just for me.

borkdude08:04:15

@rovanion Could you write out the entire program? Because it would not only involve getting access to clojure.lang.Agent but also interop with executors

rovanion08:04:52

This is the code I used back then.

borkdude09:04:13

could you maybe post this as a file attachment - it bloats the channel a bit πŸ™‚

rovanion09:04:27

But again, no need to do anything about it. I've realised that using the http://clojure.java.io API is probably the wrong API for my current program. I probably want to use java.nio.2

stijn09:04:01

@borkdude I have another suggestion for an addition; would like to use clojure.data/diff. I'm happy to create a PR, but not sure about what you would suggest for the naming

borkdude09:04:14

Why not the clojure.core one?

user=> (require '[clojure.data :as data])
nil
user=> (data/diff [1 2 3] [1 2 4])
[[nil nil 3] [nil nil 4] [1 2]]

stijn09:04:29

is that a babashka repl?

borkdude09:04:36

no, a normal Clojure REPL

stijn09:04:33

ok, so, my question was if adding it to babashka would be a good idea

borkdude09:04:48

yeah, I think adding clojure core thing to babashka is almost a no-brainer

borkdude09:04:03

I just haven't used this much, so I haven't added it

stijn09:04:07

ok, and don't create a special name, just clojure.data

borkdude09:04:32

take a look at the other namespaces for an example

borkdude09:04:39

e.g. babashka.impl/yaml, babashka.impl/xml etc

stijn09:04:56

ok, thanks

ahmed1hsn13:04:19

How can we use -IO in babashka script instead of command line? which public api functions are used for this?

b -IO '(filter #(< 1 (:foo %) 4) *input*)' < test.ednl

borkdude13:04:51

that would be something like

(ns foo
  (:require [clojure.edn :as edn]))

(doseq [x (filter #(< 1 (:foo 4) 4) (edn/read *in*))]
 (prn x))

borkdude13:04:56

@ahmed1hsn It was slightly more complicated:

(ns foo
  (:require [clojure.edn :as edn]))

(defn read-edn []
  (take-while #(not (identical? ::EOF %))
              (repeatedly #(edn/read {:eof ::EOF} *in*))))

(doseq [x (filter #(< 1 (:foo %)) (read-edn))]
  (prn x))

ahmed1hsn13:04:55

what if *in* is file instead?

borkdude13:04:56

@ahmed1hsn *in* is from clojure.core:

user=> (doc *in*)
-------------------------
clojure.core/*in*
  A .Reader object representing standard input for read operations.

  Defaults to System/in, wrapped in a LineNumberingPushbackReader

cldwalker14:04:56

Thanks for the above snippet. Is there a way to automatically determine if *in* has anything on it without hanging the script? This probably isn't bb specific but just curious. I think it's good script etiquette to support args or stdin without unexpected hangs

kah0ona14:04:51

@borkdude the example I provided, and you tweaked, seems to bug on print-table, is clojure.pprint/print-table not available in 0.0.88-2?

cldwalker14:04:02

Great. Thanks!

borkdude14:04:20

@kah0ona I have just added this on master. Releasing now

kah0ona14:04:28

aah ok πŸ™‚ that explains it

cldwalker14:04:42

That clj-together board is exciting. Particularly protocols as I'm guessing that would unlock usage of more 3rd party libs in bb. If there's a way to vote on issues, let me know. Otherwise I can drop a comment in an issue

borkdude14:04:39

commenting on issues is the only way to "vote" right now, I don't know of any other way

borkdude22:04:39

@cldwalker I just discovered that you can sort issues at Github by thumbs up or thumbs down. So giving a thumbs up on an issue counts as voting

cldwalker13:04:43

Good to know

olical16:04:20

Just wanted to say πŸ‘ since babashka's nREPL completion works with my new Conjure completion integration! https://asciinema.org/a/H7136YMD9J9bU7l6GZE1yVNAa so cool!

lukaszkorecki16:04:09

Is there a guide for adding library support to Babashka?

lukaszkorecki16:04:40

For example I'd like to add some bits from java.util.concurrent (Executors etc)

borkdude16:04:51

@lukaszkorecki This is documented at https://github.com/borkdude/babashka/blob/master/doc/dev.md. How is the UDP stuff doing?

borkdude16:04:38

Also if you'd like to add stuff, please make an issue with a description of why you would like to add it. I'd like to screen it first, since adding things to babashka is not free in terms of binary size. If it's useful for general purpose scripting, or compatibility with existing libraries, that's often a good reason to add something

lukaszkorecki16:04:58

@borkdude Thanks, I missed that doc somehow. As for UPD: it's been rock solid since latest fixes metal Each ECS task deploys a tiny container as a sidecar which scrapes ECS metadata endpoint and sends container metrics via statsd protocol to our graphite server. I'm doing a write up about it - and from the looks of things I can open source it too. https://share.getcloudapp.com/YEu1p5E9 (3 charts we get from that)

lukaszkorecki16:04:28

Gotcha - I guess I got a bit excited since I almost see Babashka as the low-footprint Clojure I always wanted ;-)

borkdude16:04:28

awesome πŸ™‚

lukaszkorecki16:04:59

so it's less about scripting, and more about building low-resource usage services which still deliver 80/90% of Clojure experience

lukaszkorecki16:04:11

but of course, that might go against your vision of the project

lukaszkorecki16:04:45

(in my ideal world, we just use native-image/graalvm for all our services but that's pretty challenging to pull off atm )

borkdude16:04:59

let's just see what it might add in terms of features and size. e.g. adding the postgres support was quite heavy (it added 3mb), but I think it will be very useful

lukaszkorecki16:04:12

Sure thing - I will have to set some time aside to dive into it first πŸ‘

ales.najmann16:04:26

I don't know how much is it relevant for the discussion here, but yesterday I tried something. I took hsqldb and just for sake of curiosity I tried to compiled it with native-image and to my big surprised an 8.6MB binary was spit-out with runnable server-binary (Darwin). It seems from the documentation that hsqldb support in-process connection. I wonder if it's a possible candidate for embedding and eventual alternative to (native) sqlite.

lukaszkorecki16:04:30

TIL about hsqldb - very cool. I've used H2 for this kind of stuff before

borkdude17:04:07

@ales.najmann If you would like to see hsqldb support in babashka, you can help by doing the research: create a graalvm cli that includes next.jdbc and hsqldb and make a POC like https://github.com/yogthos/graal-web-app-example/tree/postgres - it's basically because of that example that it was so easy to support postgres

ales.najmann18:04:53

Let's see how it goes. I'm not sure I'm skilled enough.

borkdude18:04:26

gotta start somewhere πŸ™‚

ales.najmann18:04:45

haha, yeah that's right πŸ™‚

borkdude18:04:18

a very minimal example would suffice, the main thing is to figure out the reflection config

borkdude18:04:13

@lukaszkorecki I now added a branch called executor that can execute this program:

(import '(java.util.concurrent Executors))

(defn run [nthreads niters]
  (let [a (atom 0)
        pool  (Executors/newFixedThreadPool nthreads)
        tasks (map (fn [_]
                     (fn []
                       (dotimes [_ niters]
                         (swap! a inc))))
                   (range nthreads))]
    (doseq [future (.invokeAll pool tasks)]
      (.get future))
    (.shutdown pool)
    (prn @a)))

(run 5 10) ;;=> 50
Binaries over here: https://clojurians.slack.com/archives/CSDUA8S6B/p1587751416010300 Binary size seems to be ok, only like 380kb added or so

ahmed1hsn19:04:25

now we can rely on java.util.concurrent instead of clojure.core.async. Does it include everything in this ns?

ahmed1hsn19:04:31

java.util.concurrent

borkdude19:04:43

ah, package. No I have only added three classes to begin with to make this snippet work. See the executor branch.

ahmed1hsn19:04:23

Thanks, still a great start πŸŽ‰

lukaszkorecki18:04:47

Fantastic - I'll check out the diff to see what's involved πŸ‘

kah0ona19:04:15

Whenever I run something in bb, I get: setrlimit to increase file descriptor limit failed, errno 22 macOS

kah0ona19:04:30

is that bb related?

kah0ona19:04:30

is that bb related?

borkdude19:04:37

I haven't seen that before. Does it happen with any code?

kah0ona19:04:27

btw the script itself does work

kah0ona19:04:21

yes it appears to even happen with bb '(+ 1 1)'

borkdude19:04:55

@kah0ona does it crash, or only give this warning?

borkdude19:04:18

can you try to start a bb repl by just running bb and then:

$ pid=$(pgrep bb); lsof -p $pid
This will list all open files by the process

kah0ona19:04:26

seems warning, script continues afterwards, so it’s just some extra output

kah0ona19:04:51

marten-sytemas-macbook:~ marten$ pid=$(pgrep bb); lsof -p $pid
COMMAND   PID   USER   FD     TYPE DEVICE SIZE/OFF       NODE NAME
bb      89181 marten  cwd      DIR    1,4      480    5126343 /Users/marten/Desktop
bb      89181 marten  txt      REG    1,4 51327260 8658390464 /usr/local/bin/bb
bb      89181 marten  txt      REG    1,4   973824 8647962767 /usr/lib/dyld
bb      89181 marten    0u     CHR   16,8 0t215324       6049 /dev/ttys008
bb      89181 marten    1u     CHR   16,8 0t215324       6049 /dev/ttys008
bb      89181 marten    2u     CHR   16,8 0t215324       6049 /dev/ttys008
bb      89181 marten    3r  PSXSEM             0t0            /cSunMiscSignal-89181

borkdude19:04:09

That looks pretty normal. I don't know what's up. Might be something in your shell?

borkdude19:04:31

Try a reboot?

borkdude20:04:39

Have you always seen this message on your system with bb?

borkdude20:04:28

What do get when you run ulimit on your shell

kah0ona08:04:48

will try all of the above, and report back πŸ™‚

liquidz.uo23:04:44

I made a simple Github Action to run clojure.test :relaxed: https://twitter.com/uochan/status/1253826555609878529