Fork me on GitHub
#babashka
<
2020-04-17
>
Ahmed Hassan08:04:33

how to convert json response into edn in babashka?

borkdude08:04:44

See the babashka.curl README

borkdude08:04:43

(require '[babashka.curl :as curl])
(require '[ :as io]) ;; optional
(require '[cheshire.core :as json]) ;; optional

(->
  (curl/get "" {:query-params {"q" "clojure"}})
  :body
  (json/parse-string true)
  :args)
;;=> {:q "clojure"}

Ahmed Hassan08:04:16

I want map on vector of links and store all responses on one file, how can I ensure that same response is stored only once on file (same response entry is not repeated)?

borkdude08:04:24

> how can I ensure that same response is not stored only once on file? I don't understand

Ahmed Hassan08:04:35

One way is to slurp file and merge response maps into it then spit it (without :append true ) so whole file is rewritten every time. Or better, slurp file once, merge all downloaded responses and spit it to file at the end.

borkdude08:04:42

Ah, so you want to prevent multiple concurrent writes overwriting each other?

Ahmed Hassan08:04:45

yes, ensuring integrity of each response on file.

borkdude09:04:19

You could also use a lock:

user=> (def o (Object.))
#'user/o
user=> (pmap #(locking o (spit "/tmp/foo.txt" % :append true)) (range 10))
(nil nil nil nil nil nil nil nil nil nil)
user=> (slurp "/tmp/foo.txt")
"1023475986"

Ahmed Hassan09:04:58

So, in the case of making request to link, should I obtain lock before making get request or before spit (after get)?

borkdude09:04:31

In general you should lock things as short as possible to maximize parallelism

borkdude09:04:26

But if you are writing often to that file, maybe it makes sense to batch the results in an atom and write at the end

Ahmed Hassan11:04:45

@U04V15CAJ do every call to curl/get make a new process, because I'm making 500 requests at once (`pmap` on vector) and it's slowing the laptop.

borkdude11:04:50

yes, babashka.curl shells out to curl.

borkdude11:04:07

if you don't want this, you can also use https://github.com/borkdude/clj-http-lite

borkdude11:04:49

no, but it won't create a new process.

borkdude11:04:22

it's based on HttpURLConnection, a class which is also supported in babashka. you can also just use that low level stuff

borkdude11:04:50

You can also shell out to curl yourself and then use this: > Download multiple files simultaneously https://vitux.com/how-to-download-a-file-on-debian-using-the-command-line/

Ahmed Hassan11:04:28

Response is json, so I think HttpURLConnection would be better.

Ahmed Hassan11:04:01

This class and Namespace should be mentioned in redme.md of babashka

borkdude12:04:22

Maybe you can try it out and then suggest what should be documented in the README

borkdude12:04:49

So it seems this snippet works:

(let [^HttpURLConnection conn (.openConnection (.URL. ""))] (.connect conn) (json/parse-stream (io/reader (.getInputStream conn)) true))

borkdude12:04:08

(I did it on the REPL as a single line, you might want to reformat it)

Ahmed Hassan12:04:25

So, getInputStream returns only body of response. Right?

Ahmed Hassan12:04:11

This snippet should go to docs, with namespaces.

borkdude12:04:41

so if you need the response code, you can use (.getResponseCode conn)

borkdude13:04:38

but right now you can also use clj-http-lite as a library which uses all of this: https://github.com/borkdude/clj-http-lite

Ahmed Hassan13:04:54

HttpURLConnection simply fills my requirement. There should be async mechanism too.

borkdude13:04:09

@UCMNZLJ93 babashka has core.async

borkdude13:04:21

java 11 has an async client: https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html this might be available in babashka in the future, but right now we're using java8

Ahmed Hassan13:04:05

Thanks, So I should put investment in core.async

borkdude13:04:36

learning core.async would be good

David Pham08:04:21

Is it possible to change the working directory of a script?

borkdude08:04:31

What are you trying to achieve?

David Pham09:04:08

I want to store my scripts in a folder and call them from a different one. I tried to ln -s them into my ~/bin directory but I failed to make work

David Pham09:04:00

Or somehow in a projects I have a bunch of bb files saved under /scripts, and I want to call them from /

borkdude09:04:36

@UEQGQ6XH7 I think you can use the classpath option for this. Add your /scripts directory to the classpath and require the namespaces with require

borkdude09:04:17

You can set the classpath in BABASHKA_CLASSPATH

borkdude09:04:25

or with the --classpath option

David Pham09:04:41

Ok let me try it :)

borkdude09:04:04

Note that the namespace names have to match the directory structure like in normal Clojure projects

borkdude09:04:54

Another option is to use load-file with an absolute directory filename

borkdude09:04:26

If your scripts are always relative to the directory you are running them from, you can also inspect *file* and figure out the script directory from there

borkdude10:04:29

Fun idea. Now that babashka has clj-yaml, it might actually be feasible to re-implement http://clojure-toolbox.com using a babashka script: https://github.com/weavejester/clojure-toolbox.com

borkdude10:04:46

$ bb '(-> (yaml/parse-string (slurp "projects.yml")) :babashka)'
#ordered/map ([:name "babashka"] [:url ""] [:categories ("Scripting")] [:platforms ("clj")])

David Pham10:04:36

You also get a free edn to YAML concerter

David Pham10:04:53

So all your config files can be edn :)

borkdude10:04:56

That too. The Github actions workflow for babashka itself can now be generated from a script to avoid some repetition

stijn11:04:06

Yeah, I'm planning to use babashka for generating linuxkit yml files

Darin Douglass12:04:19

Supporting clj-yaml meant I was able to remover the custom YAML formatting from our kube library :) Plus it’s smart enough to know that (str 2) needs to render as ’2’ in YAML, which helps immensely for setting up Pod env

borkdude15:04:15

$ time script/generate.clj
Written 763 projects in 214 categories to index.
script/generate.clj   0.21s  user 0.06s system 97% cpu 0.279 total

David Pham17:04:27

Nice! I started to use babashak with the classpath trick it is really cool 🙂

borkdude18:04:18

v0.0.86: https://github.com/borkdude/babashka/releases/tag/v0.0.86 contains a bugfix for babashka.curl (cc @lukaszkorecki) and some other fixes/enhancements

lukasz18:04:52

Thanks - I'm going to deploy the new version to our staging environment and confirm that the issue is fixed

bherrmann20:04:01

When babasha gets an exception at runtime, I'm not getting much about where the exception happened...

#inst "2020-04-17T20:17:47.288-00:00" HI JOJO: GET /105 HTTP/1.1
Exception in thread "Thread-7" java.lang.NullPointerException
	at sci.impl.interpreter$eval_instance_method_invocation.invokeStatic(interpreter.cljc:304)
	at sci.impl.interpreter$eval_special_call.invokeStatic(interpreter.cljc:459)
	at sci.impl.interpreter$eval_call.invokeStatic(interpreter.cljc:476)
	at sci.impl.interpreter$interpret.invokeStatic(interpreter.cljc:526)
	at sci.impl.interpreter$interpret.invoke(interpreter.cljc:503)

bherrmann20:04:36

Is there something which can tell me on which line the interpreter was chewing on when this happened?

borkdude20:04:08

Can you maybe post the stack trace as a file instead of flooding the channel?

bherrmann20:04:35

sure... I was hoping slack would auto-scrunch it.

borkdude20:04:49

yeah, happened to me too

borkdude20:04:07

about your question: can you make a reproduction of what causes the issue?

bherrmann20:04:54

I'm sure it is something stupid that I'm doing to make an NPE, I'm just wondering if I could get the line number of the script where it is happening.

borkdude20:04:25

Normally sci / bb tries to include line information in the exception

borkdude20:04:59

e.g.:

$ bb /tmp/foo.clj
clojure.lang.ExceptionInfo: Divide by zero [at /private/tmp/foo.clj, line 2, column 1]

borkdude20:04:24

But there may be a reason why that's not the case here. In that case a repro can help check if that can be fixed

bherrmann20:04:27

I wonder if it is confused by the fact this this occurs on a thread.

bherrmann20:04:01

ok, I will see if I can shrink down my use case.

bherrmann20:04:30

ser=> (.start (Thread. #(throw (RuntimeException. "blarh"))))
nil
user=> Exception in thread "Thread-1" java.lang.RuntimeException: blarh
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
...

bherrmann20:04:27

so I guess it is the default stack trace printer for a thread.

borkdude20:04:32

yeah, sci catches exceptions in the main thread and re-throws them with location information, but I'm not sure how to do that with other threads?

bherrmann20:04:37

There is a way to override the default thread handler... or perhaps I could try catch call sci?

borkdude20:04:52

yeah, maybe that would help

bherrmann20:04:28

perhaps when I (Thread. ) I should set the default thread handler to be the same one that main-thread uses

borkdude20:04:19

or maybe bb should set the default exception handler to something sensible

borkdude21:04:28

in general I think it's quite handy to wrap stuff you're doing in threads in futures in your own try/catch

borkdude21:04:42

especially with future you might not notice that something is wrong because the exception is thrown only when you deref it

borkdude21:04:53

@U0502D2GL Ah look:

(.start (Thread. (fn [] (/ 1 0)))) @(promise)
Exception in thread "Thread-1" clojure.lang.ExceptionInfo: Divide by zero [at /private/tmp/foo.clj, line 1, column 25]

borkdude21:04:13

I think the location metadata might have been lost with the function literal. I think that can be fixed.

borkdude21:04:50

you notice the difference?

bherrmann21:04:55

ok, yea, I can try that.

borkdude21:04:33

I'll try to see if I can make a fix for this

borkdude22:04:41

@U0502D2GL Fixed on master. Issue: https://github.com/borkdude/babashka/issues/362 New binaries will appear in #babashka_circleci_builds shortly for you to try.

bherrmann22:04:05

Whoo hoo, thx.

bherrmann22:04:46

Humm.... is this expected behavior?

bherrmann22:04:06

$ clj
Clojure 1.10.1
user=> (.printStackTrace)
Execution error (IllegalArgumentException) at clojure.main/main (main.java:40).
Malformed member expression, expecting (.member target ...)
user=> 
$ bb
Babashka v0.0.84-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (.printStackTrace)
java.lang.NullPointerException
user=> 

bherrmann22:04:38

I get that the (.fjfjfjf fjfjfj) thing is a java macro

borkdude07:04:54

Clojure prints > Malformed member expression, expecting (.member target ...) I'll make bb also print that.

bherrmann22:04:20

but I was suprised when I accently had a "(.printStackTrace)" without the exception arg.