Fork me on GitHub
#babashka
<
2022-04-05
>
grumplet09:04:17

Some random weirdness that I’m trying to get a handle on…. I have an Rscript that reads a CSV and attaches it like so:

param <- read.csv("params.csv")
attach(param)
This works fine when called from bash. The weirdness is that the attach fails when shelling out to the same Rscript from babashka. The read.csv call works fine there, but the attach doesn’t.

1
borkdude09:04:48

How are you shelling out exactly?

borkdude09:04:06

And what does attach do?

grumplet09:04:06

attach(dataframe) installs the dataframe columns as individual variables.

grumplet09:04:01

In the failing case those variables are all NULL

borkdude09:04:30

Can you show how you are shelling out?

grumplet09:04:49

Yep - but hold on - I have tried so many options….

borkdude09:04:12

You could also try babashka.process

grumplet09:04:02

I’m requiring:

grumplet09:04:07

[clojure.java.shell :refer [sh]]

grumplet10:04:07

(sh “Rscript” “resources/r_model_tests/kidney/waiting/adjcox.R”)

grumplet10:04:59

There are variants where I pass the current directory to Rscript, but in this variant I just set it to the correct dir in adjcox.R.

borkdude10:04:11

And why would this work with bash but not with sh? Maybe you could try:

(babashka.process/sh ["Rscript" ...])
which is similar

grumplet10:04:19

I wondered about babashka.process, but I could not see the benefit since the R code is self-contained. It reads its own files and writes it’s own results to file.

borkdude10:04:43

Maybe you can log the current directory, etc from the Rscript, to debug what's different.

grumplet10:04:07

Yeah - there isn’t a problem with the current directory.

grumplet10:04:09

OK - I’ll have a go with babashka.process…

borkdude10:04:09

Have you checked the :err output?

borkdude10:04:18

And the :exit code?

borkdude10:04:42

I would be able to take a closer look if you could make a repro in a github repo I could check out

borkdude10:04:48

and tell me how to install Rscript

borkdude10:04:12

For now you could maybe shell out to bash to call Rscript, if that works

grumplet10:04:18

Yes - the err: output is exactly what you would expect from R when it sees NULL instead of the correct value for one f the header variables.

borkdude10:04:20

(sh "bash" "-c" ...)

grumplet10:04:54

Another layer… Interesting idea….

grumplet10:04:28

OK You’ve given me some things to try. Thanks!

borkdude10:04:56

I'm pretty sure this can be fixed, given the right repro. :thumbsup:

grumplet10:04:51

I’ll come back to this shortly.

grumplet10:04:12

(sh "bash" "-c" ...) has the same attach issue

grumplet10:04:15

(babashka.process/sh ["Rscript" "resources/r_model_tests/kidney/waiting/adjcox.R"])
generates the same issue:
{:proc #object[java.lang.ProcessImpl 0x5eabdbcd "Process[pid=2775, exitValue=1]"], :exit 1, :in #object[java.lang.ProcessImpl$ProcessPipeOutputStream 0x3d98c7c6 "java.lang.ProcessImpl$ProcessPipeOutputStream@3d98c7c6"], :out "", :err "Error in adjcox(cent = tests$cent[i], sex = tests$sex[i], eth = tests$eth[i],  : \n  object 'tx_sex_m' not found\nExecution halted\n", :prev nil, :cmd ["Rscript" "resources/r_model_tests/kidney/waiting/adjcox.R"]}

grumplet10:04:58

tx_sex_m is the first column header that should be attached.

borkdude10:04:11

If you want to make a ready to go repro I can clone and execute locally, including a csv file, and instructions how to install Rscript, I can take a look. No ideas at this point.

borkdude10:04:32

Perhaps it helps searching for "Java shell out Rscript attach" or so

grumplet10:04:23

OK - I’ll see if I can isolate it. It should be simple to do.

👍 1
borkdude10:04:30

My wife is also using R, so it would be interesting for me to figure this out :)

grumplet11:04:14

Well that was interesting. The issue only appears in the Calva babashka repl. In other contexts it’s fine. Gist is here: https://gist.github.com/gmp26/b859872eeb53c753ece08c7e632f7b02

borkdude11:04:31

Nice! So it's a Calva, maybe general GUI + shelling out problem or so

borkdude11:04:45

Congrats on discovering this

borkdude11:04:52

Maybe also try with a JVM REPL

borkdude11:04:11

and with a babashka nREPL connected with lein repl :connect in a terminal maybe

borkdude11:04:29

As @U02N27RK69K hinted, it may also have to do something with tty-ish

tatut14:04:07

just merged https://github.com/tatut/clj-chrome-devtools/pull/32 and clj-chrome-devtools now works with babashka (haven't tested very extensively)

🎉 3
borkdude14:04:00

Lemme try :)

tatut14:04:19

there's a profile named bb

borkdude14:04:29

The first trouble I have with the README and get going quickly is that I can't see where connect is coming from

tatut14:04:10

(require '[clj-chrome-devtools.core :as core]
         '[clj-chrome-devtools.automation :as auto])

(def a (auto/create-automation (core/connect)))
(auto/to a "")
(auto/print-pdf a "clojure.pdf"
                {:print-background true
                 :paper-width 8.3
                 :paper-height 11.7})

tatut14:04:29

yeah, should probably add my sample script

tatut14:04:17

the connect actually needs chrome to be already up, there's a launcher ns as well which will launch a chrome instance

borkdude14:04:12

How do I launch a chrome with a debug port?

borkdude14:04:17

might also be nice to add to the README

borkdude14:04:21

I already have chrome running, I can't figure it out without restarting it

borkdude14:04:29

Now just relaunched with the port, it worked!

22:       (throw (ex-info (str "Error in command " method ": " (:message error))
          ^--- Error in command Page.printToPDF: Printing is not available
23:                       {:request payload

tatut14:04:38

you must be running --headless for PDF printing to work, don't know why that is

tatut14:04:23

check README again, I added a quickstart help that links to a script

borkdude14:04:42

For the bb script example, you can do this without the classpath hassle:

(ns devtools
  (:require [babashka.deps :as deps]))

(deps/add-deps
 '{:deps {tatut/devtools {:git/url ""
                          :git/sha "62a8aba789bbda6d954d17f2dc6f92dd74ec8619"}
          org.babashka/spec.alpha {:git/url ""
                                   :git/sha "1a841c4cc1d4f6dab7505a98ed2d532dd9d56b78"}}})

(require '[clj-chrome-devtools.automation :as auto]
         '[clj-chrome-devtools.core :as core])

(def a (auto/create-automation (core/connect)))
(auto/to a "")

tatut14:04:06

that's neat

borkdude15:04:02

Do you think this or a similar approach could also work for firefox?

borkdude15:04:08

This is already very cool btw!

borkdude15:04:14

Thanks a lot for making it bb compatible

tatut15:04:41

I don't know what kind of protocol firefox has... chrome has the CDP protocol that is described as json and you can easily autogenerate the functions

tatut15:04:20

I started this bb support because I had an idea to use AWS Lambda custom runtime to have a lambda that takes in HTML and returns back PDF

tatut15:04:57

but could also be used for all sorts of browser automation things now

borkdude15:04:02

for sure! For AWS Lambda #nbb could also be an interesting option: https://github.com/babashka/nbb/blob/main/doc/aws_lambda.md There are many pdf things for Node.js as well. But I don't want to distract you too much from this project, since I could see it being very useful for browser testing in bb + JVM

borkdude15:04:31

As for lambda: if you could move clojure.spec aside too, then this code would load even faster.

borkdude15:04:00

I might bring clojure.spec.alpha as a built-in in the future, then that becomes irrelevant for startup time, but right now it has to load a bunch of spec code at startup

tatut16:04:27

this relies heavily on clojure.spec

tatut16:04:18

well, it might not be absolutely necessary, but all the autogenerated commands have fdef specs

tatut16:04:36

I could imagine moving the specs to auxiliary namespaces and having some switch to skip loading them...

tatut16:04:44

you could even detect bb and skip them by default

tatut16:04:13

btw, regarding firefox, it does support a subset of the CDP, so you can use it as-is

tatut16:04:45

just tried, it supports the same --remote-debugging-port=<port> parameter

borkdude16:04:01

If your program relies on spec, then that's good, keep it! But often spec is loaded not for any good reason. Not in this case.

borkdude21:04:19

@U11SJ6Q0K Just spitballing here, but would it help in any way if a browser were controllable through nREPL: https://twitter.com/borkdude/status/1511456422667735042/photo/1 I feel there could be some kind of synergy between the debug protocol and the thing I'm currently working on

borkdude21:04:05

I noticed when trying firefox on the debug protocol, that after:

(def a (auto/create-automation (core/connect)))
(auto/to a "")
it did go to http://clojure.org, but then the script was waiting and the next expression wasn't evaluated

tatut04:04:30

hmm, I was able to print a PDF with firefox

tatut04:04:38

it was very much slower than chrome but worked

tatut04:04:01

about sci, you could use CDP to just do js eval on the browser

cap10morgan16:04:24

Running into what seems like a regression running main from a jar in 0.8.0 (works in 0.7.8 and prior) that at first glance doesn't appear to be related to my changes in 0.8.0 😆: https://gist.github.com/cap10morgan/65e0c9bfabdb2b161ee2cc7617a5d7ab anything jump out at you there @borkdude? or anything else you'd like me to provide?

cap10morgan16:04:20

going to see if I can make a minimal repro example

1
borkdude17:04:05

The only arg parsing changes came from the bb.edn pods stuff ;)

borkdude17:04:32

Repro welcome, we can do a follow up release if it's critical

cap10morgan17:04:44

what makes you think it's related to arg parsing, out of curiosity?

borkdude17:04:10

I thought you hinted at that, but maybe I've not read good enough

cap10morgan17:04:23

darn, thought you had noticed something I hadn't 😉

cap10morgan17:04:46

haven't been able to repro in a more minimal example yet... but I'll keep working on it

borkdude17:04:54

probably best to git bisect

borkdude17:04:08

you can just run with jvm using a git sha

cap10morgan18:04:27

hmm... don't think I've done that before. is that documented somewhere?

cap10morgan18:04:08

to clarify, I've done git bisect, but haven't run bb w/ jvm (is that bb clojure? not sure how to do that w/ an arbitrary git sha)

borkdude19:04:24

@U06FS3DLH I mean, just the process of choosing a git sha, checking if the error happens there, etc.

borkdude19:04:45

not literally git bisect, but just splitting the commit range in half, trying, etc.

borkdude19:04:59

you can just invoke babashka as a git dep + -M -m babashka.main

cap10morgan19:04:22

ah, I see. I'll give that a try. thanks!

cap10morgan20:04:51

how do you deal with the git submodules when you do that? according to jira they're not supported in deps.edn git deps

borkdude21:04:45

oh right. well, maybe use a local root + git reset then

borkdude21:04:02

I must say that I always only use babashka JVM as a local root locally

cap10morgan21:04:16

:thumbsup: makes sense

cap10morgan21:04:11

ok false alarm. it was a problem in my code after all. just strange that it behaved so differently in 0.7.8. it's possible it was a combination of things I hadn't tested before. anyway, I don't think babashka is to blame either way.

borkdude21:04:47

🎉 great to hear we didn't forget to add a test for something :)

👍 1