Fork me on GitHub
#babashka
<
2021-06-08
>
pbaille09:06:12

Hello, i'm trying to use babashka.process for a little thing. I would like to be able to launch a process that do not terminate and do something whenever some new output occurs from it. Is there a builtin way to do this? I've try this example from the documentation:

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))

(pipeline (pb '[tail -f "log.txt"])
          (pb '[cat])
          (pb '[grep "5"] {:out :inherit}))
but I do no see anything printing out, is it intended ?

borkdude09:06:23

@pbaille The readme explains what the problem is: buffering.

borkdude09:06:32

oh now you are using pipeline ok

pbaille09:06:42

sorry I've edited the message

borkdude09:06:19

Yes, so this works:

(require '[babashka.process :refer [pipeline pb]])

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))


(-> (pipeline (pb '[tail -f "log.txt"])
              (pb '[cat])
              (pb '[grep "5"] {:out :inherit}))
    last
    deref)

pbaille09:06:26

thank you @borkdude, I've just tried this snippet in my editor (intellij cursive) on Java 15 and clojure 1.10.0. Am I supposed to see something printed in the repl ?

borkdude09:06:20

@pbaille :inherit prints directly to stdout (`System/out`)

borkdude09:06:34

Just try (process ["ls"] {:inherit true}) to try it out what happens in your editor

pbaille09:06:36

yes it prints the output

pbaille09:06:39

in the repl

borkdude09:06:19

ok, then that works correctly.

pbaille09:06:56

ok, so I should be able to see some output from your first snippet ?

borkdude09:06:11

yes, I think so

borkdude09:06:22

maybe simplify the example for debugging purposes

pbaille09:06:22

it hangs on the last expression

borkdude09:06:54

you can remove the deref to make it not hang. I just added that to make the bb script not terminate

borkdude09:06:01

let's continue in thread

pbaille09:06:38

I got this but nothing prints

borkdude09:06:25

Can you please try this in a standalone bb script and then execute it from the terminal?

borkdude09:06:47

Or clojure script if you are using this lib on the JVM

pbaille09:06:25

yes I will do this

pbaille09:06:44

it works nicely as a script

borkdude09:06:33

Perhaps try :out *out* in your editor for the last process in the pipeline

borkdude09:06:39

else, I don't know :)

pbaille09:06:46

I've tried this

pbaille09:06:02

but didn't change anything

pbaille09:06:37

ok, thank you for your time @borkdude! I will investigate on my own and let you know if I find something

dabrazhe16:06:31

Can I connect to the bb repl in VSC Calva without manually entering it every time? it does not pick it up from .nrepl-port file

borkdude17:06:17

@dennisa Babashka does not write an .nrepl-port file since that can conflict with a Clojure process.

dabrazhe17:06:50

I can write it myself, just need Calva to pick it up from somewhere, otherwise getting this promt with no port. I have to restart the bb repl rather often as it slows down and it's a chore to type every time 1667 : )

borkdude17:06:05

@dennisa There is a snippet about writing this port file here: https://book.babashka.org/#_nrepl

borkdude17:06:45

(I think this script can be simplified/refactored now since there is babashka.process)

dabrazhe17:06:11

Have someone managed to Calva to pick it up? Or am I in the wrong channe?l :)

dabrazhe19:06:21

what's the babashka.process : ) ?

dabrazhe19:06:05

nevemind, found it in the BBook )

pez20:06:21

I haven’t tried this myself, but you can probably configure a connect sequence, nReplPortFile with the name of whatever nrepl-port file you write. See: https://calva.io/connect-sequences/

pez20:06:17

Now tried it. This setting works:

"calva.replConnectSequences": [
        {
            "name": "bb",
            "projectType": "generic",
            "nReplPortFile": [".bb-nrepl-port"],
            "cljsType": "none"
        }
    ]
Then start babashka like so:
echo 1667 > .bb-nrepl-port && bb --nrepl-server

dabrazhe08:06:16

It won’t work for me. I can see ‘bb’ in the ‘connect to repl’ prompt but the port is still empty after localhost: The .bb-nrepl-port is created fine with 1667 port in it. @pez

pez08:06:32

And you are selecting bb, not just seeing it?

pez08:06:47

Also, which directory are you starting babashka in?

dabrazhe12:06:14

Yes, selecting. BB starts in the root dir or the project, where the repl file is.

dabrazhe12:06:43

Restarted VSC just in case

pez13:06:56

I now tried this on my Windows machine, and it just works there as well… Please file an issue about this using the Help menu in vscode. Attach the setting and the command line you are using.

dabrazhe09:06:23

It's on the mac. Do you want me to file issue to Microsoft about calva )? What's the chance to resolve it? )

dabrazhe09:06:16

The will forward me to the maker of the extension, at best

pez09:06:59

There is an option there to file the issue on the extension, so it will land in Calva’s GiHub repo. The point with doing it this way is that some system info will be attached to the issue.

Joshua Suskalo20:06:48

Continuing from #clojure in a thread: I'd like to add babashka support to https://github.com/IGJoshua/farolero The key thing that would prevent it from working as-is is the requirement to be able to throw an object that isn't caught by ex-info catch clauses.

Joshua Suskalo20:06:27

If there's a way to throw an arbitrary object or extend a throwable marker protocol or something that'd be ideal, but since babashka is likely to be used in smaller projects with fewer dependencies, I may if need be just use ex-info, but I'd prefer to not as there's a requirement to re-throw exceptions from catch blocks based on a condition, and a normal catch block for ex-infos won't re-throw based on that condition.

borkdude20:06:26

How do you solve this in normal Clojure?

Joshua Suskalo20:06:43

I solve this by making the farolero-signal artifact on maven that contains a single class that extends java.lang.Error (which extends from Throwable but not Exception), https://github.com/IGJoshua/farolero/blob/master/signal/src/farolero/signal/Signal.java Then in the actual farolero lib I just extend-protocol to this class to add the functionality for storing the objects that need to get passed around.

Joshua Suskalo20:06:50

And the reason that it's in a different artifact altogether is to allow git dependencies on farolero.

borkdude21:06:44

@U5NCUG8NR So your library, does it catch only this specific exception and handle that one? Or does it install some global exception handler?

Joshua Suskalo21:06:29

It only catches this one specific exception, and it never throws the exception except if it knows for a fact that it's currently in a dynamic context where it will be caught.

Joshua Suskalo21:06:53

For the CLJS support it catches all objects thrown and re-throws them if they don't implement the protocol.

Joshua Suskalo21:06:16

In particular there is exactly one construct that throws or catches this exception, and it is block, and everything else is implemented on top of that.

(block the-block
  (some code)
  (return-from the-block :returned)
  :unreachable)
return-from throws the exception, and block catches it

borkdude21:06:07

In CLJS, can you throw any arbitrary object?

Joshua Suskalo21:06:16

Yeah, it's a feature of JS.

borkdude21:06:42

A workaround I can come up with for bb is: Make one singleton Exception instance, e.g.:

(defonce FaroleroEx (Exception. "my-special"))
and then in the catch clause check (identical? FaroleroEx ex)

borkdude21:06:56

or just use ex-info with a special field in it to mark it as a farolero exception

borkdude21:06:20

I think that may just work best for bb

Joshua Suskalo21:06:02

That's a good thought, I might do that. The key reason that I would want to throw arbitrary objects or something would just be to prevent catch blocks from interfering with this mechanism, but if I can't do that in bb, that's fine.

Joshua Suskalo21:06:39

The features already won't be 100% parity anyway because I doubt bb implements locking and so the debugger won't be the same.

borkdude21:06:54

bb does implement locking

borkdude21:06:16

btw another thought. I see you mix specs with your code. If they are non-essential to your lib, e.g. only for development, I recommend putting them aside in a farolero.specs namespace so they can be optionally loaded. This saves startup time

borkdude21:06:56

bb currently doesn't come with clojure.spec.alpha since ... it's alpha

borkdude21:06:13

but there is https://github.com/borkdude/spartan.spec as a drop-in replacement

Joshua Suskalo21:06:16

Oh, cool, maybe it can be on part with the debugger then. The main part I was concerned about is this highlighted portion: https://github.com/IGJoshua/farolero/blob/master/src/cljc/farolero/core.cljc#L1192-L1269 I guess it's worthwhile to see if it'll run unmodified on bb though. Thanks! As for the exception mechanism, I think I can just do what you suggested with an extra field on an ex-info

Joshua Suskalo21:06:45

Oh, yeah, the specs. Good to know. The reason the specs are there is just to help with macroexpansion, they aren't used at runtime at all. Moving them to a separate namespace is far from out of the question.

borkdude21:06:12

Hmm, dosync and ref etc are not in babashka. Nobody ever asked for this. Those features were cool in the beginning of Clojure but nowadays pretty much everyone uses atoms

Joshua Suskalo21:06:46

Yeah, this was the first time I'd ever found a use for refs.

borkdude21:06:32

What is the debugger object?

Joshua Suskalo21:06:33

I might be able to refactor it to use atoms, but it could be a challenge, this is all pretty sensitive code and prone to race conditions on changes.

borkdude21:06:49

you call e.g. .wait and .notify on it, what is the type?

Joshua Suskalo21:06:39

It's a unique cons of the condition and arguments.

borkdude21:06:28

I don't think .wait and .notify currently work, but I think they could be made to work, with some work ;)

borkdude21:06:37

not sure how hard it is to port dosync etc

borkdude21:06:08

probably it's possible

Joshua Suskalo21:06:33

Fair enough. I'd like to take a peek in the source of babashka/sci to see how hard adding notify and wait are. dosync etc I might be able to avoid by refactoring to use an atom, although we'll see if that ends up worth it.

borkdude21:06:53

oh gosh it does work already:

(locking 1 (.wait 1))

Joshua Suskalo21:06:16

Are numbers interned objects in bb?

borkdude21:06:12

hm, this didn't work:

user=> (def c (cons 1 2))
user=> (locking c (.wait c))
java.lang.IllegalArgumentException: No matching method wait found taking 0 args for class clojure.lang.PersistentList [at <repl>:1:8]
The interop in bb is implemented using reflection and reflection inside a native image needs some special attention sometimes

borkdude21:06:58

Are numbers interned: I don't do anything special with numbers, so I don't know

Joshua Suskalo21:06:22

Yeah, I was able to get just a bare object to work as expected with wait and notify

borkdude21:06:26

user=> (identical? 1 1)
true

borkdude21:06:52

user=> (identical? 1 (+ 1 0))
true

Joshua Suskalo21:06:36

Fair enough. Anyway, it looks like wait and notify don't work on clojure's collections, yeah, so that'd probably be where the work on wait and notify would have to be put.

borkdude21:06:38

I'm going afk now, as it's getting late. Let's revisit some other time and make issues for the things that could be examined / improved

Joshua Suskalo21:06:47

Also I'm looking at this and while the ref code could be re-expressed in terms of atoms, it'd be clunky at best.

borkdude21:06:42

btw, I think I know what the issue is. bb supports reflection on a selection of classes. e.g. this works:

user=> (def c (java.io.File. "foo"))
#'user/c
user=> (locking c (.wait c))

borkdude21:06:58

so if I add explicit support for a type then it should work

borkdude21:06:26

it just doesn't include the wait/notify methods for PersistentList

Joshua Suskalo21:06:31

Fair enough. I'll make an issue for this now.

Joshua Suskalo21:06:43

As well as a discussion issue about ref/dosync

Joshua Suskalo21:06:40

As an aside, farolero kind of completed my personal trek through the multithreading facilities that clojure provides, since before this I'd used atoms of course, core.async channels, reducers, and agents for per-endpoint ratelimits on a discord client. Finally with farolero I've also used refs. Now I guess it's time for me to learn manifold, lol.

borkdude21:06:14

I think manifold could use some help, so that would be super cool

Joshua Suskalo21:06:46

Honestly the one thing that I felt was really missing from core.async though was a promise channel that you could deref. So I've had to write one. 😛 https://github.com/IGJoshua/discljord/blob/master/src/discljord/util.clj#L93-L151

borkdude13:06:35

@U5NCUG8NR I've got dosync etc working. I'm also looking into .wait and .notify: this does work if you use an ^Object type hint, so you could try that as a workaround meanwhile.

borkdude13:06:06

If you want to try a binary with dosync etc, let me know your OS

Joshua Suskalo17:06:41

Awesome, I'll give it a try today.

Joshua Suskalo17:06:21

I'm using nixos

Joshua Suskalo20:06:32

@borkdude the commute and alter functions would be great to have too. I know farolero uses at least alter. I'll get to actually testing it once my workday is done.

borkdude20:06:30

I already added alter. Adding commute now as well.

borkdude10:06:16

@U5NCUG8NR Now the .notify / .wait stuff also works :)

borkdude10:06:10

$ time bb -e '(def x (cons 1 nil)) (locking x (.wait x 2000))'
bb -e '(def x (cons 1 nil)) (locking x (.wait x 2000))'   0.01s  user 0.01s system 1% cpu 2.033 total

borkdude10:06:25

I will just make a new release so you can use that

Joshua Suskalo14:06:14

Thanks! Sorry I didn't get to try it out last night, life just kinda got in the way. Got a kitten recently and she requires a lot of attention.

borkdude14:06:55

No worries at all, I'm glad I got it working after all

borkdude14:06:10

and if you need something then we can just make another release