This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-04-17
Channels
- # announcements (1)
- # babashka (94)
- # beginners (76)
- # calva (24)
- # cider (24)
- # clj-kondo (1)
- # cljs-dev (16)
- # cljsrn (45)
- # clojure (135)
- # clojure-europe (9)
- # clojure-france (5)
- # clojure-germany (2)
- # clojure-italy (12)
- # clojure-losangeles (13)
- # clojure-nl (3)
- # clojure-portugal (54)
- # clojure-uk (20)
- # clojurescript (55)
- # conjure (67)
- # core-async (5)
- # cursive (2)
- # datomic (10)
- # docker (7)
- # duct (22)
- # emacs (16)
- # fulcro (34)
- # graalvm (15)
- # hoplon (1)
- # instaparse (1)
- # jobs-discuss (3)
- # juxt (94)
- # luminus (1)
- # meander (4)
- # off-topic (13)
- # pathom (4)
- # pedestal (1)
- # ring (3)
- # ring-swagger (2)
- # shadow-cljs (61)
- # spacemacs (17)
- # specter (2)
- # sql (23)
- # xtdb (33)
how to convert json response into edn in babashka?
(require '[babashka.curl :as curl])
(require '[ :as io]) ;; optional
(require '[cheshire.core :as json]) ;; optional
(->
(curl/get "" {:query-params {"q" "clojure"}})
:body
(json/parse-string true)
:args)
;;=> {:q "clojure"}
I want map on vector of links and store all responses on one file, how can I ensure that same response is stored only once on file (same response entry is not repeated)?
> how can I ensure that same response is not stored only once on file? I don't understand
One way is to slurp file and merge response maps into it then spit it (without :append true
) so whole file is rewritten every time. Or better, slurp file once, merge all downloaded responses and spit it to file at the end.
yes, ensuring integrity of each response on file.
You could also use a lock:
user=> (def o (Object.))
#'user/o
user=> (pmap #(locking o (spit "/tmp/foo.txt" % :append true)) (range 10))
(nil nil nil nil nil nil nil nil nil nil)
user=> (slurp "/tmp/foo.txt")
"1023475986"
So, in the case of making request to link, should I obtain lock before making get
request or before spit
(after get
)?
But if you are writing often to that file, maybe it makes sense to batch the results in an atom and write at the end
@U04V15CAJ do every call to curl/get
make a new process, because I'm making 500 requests at once (`pmap` on vector) and it's slowing the laptop.
if you don't want this, you can also use https://github.com/borkdude/clj-http-lite
is it async?
it's based on HttpURLConnection, a class which is also supported in babashka. you can also just use that low level stuff
look at this example: https://github.com/borkdude/deps.clj/blob/64589845e014191b33d97cb1983dca504ba1dff2/src/borkdude/deps.clj#L188
You can also shell out to curl yourself and then use this: > Download multiple files simultaneously https://vitux.com/how-to-download-a-file-on-debian-using-the-command-line/
Response is json, so I think HttpURLConnection would be better.
This class and Namespace should be mentioned in redme.md of babashka
Maybe you can try it out and then suggest what should be documented in the README
So it seems this snippet works:
(let [^HttpURLConnection conn (.openConnection (.URL. ""))] (.connect conn) (json/parse-stream (io/reader (.getInputStream conn)) true))
So, getInputStream
returns only body of response. Right?
This snippet should go to docs, with
namespaces.
yes. you can view the available methods here: https://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html
but right now you can also use clj-http-lite as a library which uses all of this: https://github.com/borkdude/clj-http-lite
HttpURLConnection simply fills my requirement. There should be async mechanism too.
java 11 has an async client: https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html this might be available in babashka in the future, but right now we're using java8
Thanks, So I should put investment in core.async
Is it possible to change the working directory of a script?
I want to store my scripts in a folder and call them from a different one. I tried to ln -s them into my ~/bin directory but I failed to make work
Or somehow in a projects I have a bunch of bb files saved under /scripts, and I want to call them from /
@UEQGQ6XH7 I think you can use the classpath option for this. Add your /scripts
directory to the classpath and require the namespaces with require
Ok let me try it :)
Note that the namespace names have to match the directory structure like in normal Clojure projects
If your scripts are always relative to the directory you are running them from, you can also inspect *file*
and figure out the script directory from there
I want to store my scripts in a folder and call them from a different one. I tried to ln -s them into my ~/bin directory but I failed to make work
Fun idea. Now that babashka has clj-yaml, it might actually be feasible to re-implement http://clojure-toolbox.com using a babashka script: https://github.com/weavejester/clojure-toolbox.com
$ bb '(-> (yaml/parse-string (slurp "projects.yml")) :babashka)'
#ordered/map ([:name "babashka"] [:url ""] [:categories ("Scripting")] [:platforms ("clj")])
You also get a free edn to YAML concerter
So all your config files can be edn :)
That too. The Github actions workflow for babashka itself can now be generated from a script to avoid some repetition
Supporting clj-yaml meant I was able to remover the custom YAML formatting from our kube library :)
Plus it’s smart enough to know that (str 2)
needs to render as ’2’
in YAML, which helps immensely for setting up Pod env
ét voila, ported http://clojure-toolbox.com from Ruby to babashka: https://github.com/borkdude/clojure-toolbox.com-babashka/commit/5b516fa0e325b7c9ba973a88e4600d3bbad97c66
can you make a redirect from https://clojure-toolbox.com/ to https://www.clojure-toolbox.com/ please?
$ time script/generate.clj
Written 763 projects in 214 categories to index.
script/generate.clj 0.21s user 0.06s system 97% cpu 0.279 total
Nice! I started to use babashak with the classpath trick it is really cool 🙂
v0.0.86: https://github.com/borkdude/babashka/releases/tag/v0.0.86 contains a bugfix for babashka.curl (cc @lukaszkorecki) and some other fixes/enhancements
Thanks - I'm going to deploy the new version to our staging environment and confirm that the issue is fixed
When babasha gets an exception at runtime, I'm not getting much about where the exception happened...
#inst "2020-04-17T20:17:47.288-00:00" HI JOJO: GET /105 HTTP/1.1
Exception in thread "Thread-7" java.lang.NullPointerException
at sci.impl.interpreter$eval_instance_method_invocation.invokeStatic(interpreter.cljc:304)
at sci.impl.interpreter$eval_special_call.invokeStatic(interpreter.cljc:459)
at sci.impl.interpreter$eval_call.invokeStatic(interpreter.cljc:476)
at sci.impl.interpreter$interpret.invokeStatic(interpreter.cljc:526)
at sci.impl.interpreter$interpret.invoke(interpreter.cljc:503)
Is there something which can tell me on which line the interpreter was chewing on when this happened?
I'm sure it is something stupid that I'm doing to make an NPE, I'm just wondering if I could get the line number of the script where it is happening.
e.g.:
$ bb /tmp/foo.clj
clojure.lang.ExceptionInfo: Divide by zero [at /private/tmp/foo.clj, line 2, column 1]
But there may be a reason why that's not the case here. In that case a repro can help check if that can be fixed
ser=> (.start (Thread. #(throw (RuntimeException. "blarh"))))
nil
user=> Exception in thread "Thread-1" java.lang.RuntimeException: blarh
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
...
yeah, sci catches exceptions in the main thread and re-throws them with location information, but I'm not sure how to do that with other threads?
There is a way to override the default thread handler... or perhaps I could try catch call sci?
https://docs.oracle.com/javase/7/docs/api/java/lang/Thread.UncaughtExceptionHandler.html
perhaps when I (Thread. ) I should set the default thread handler to be the same one that main-thread uses
in general I think it's quite handy to wrap stuff you're doing in threads in futures in your own try/catch
especially with future you might not notice that something is wrong because the exception is thrown only when you deref it
@U0502D2GL Ah look:
(.start (Thread. (fn [] (/ 1 0)))) @(promise)
Exception in thread "Thread-1" clojure.lang.ExceptionInfo: Divide by zero [at /private/tmp/foo.clj, line 1, column 25]
I think the location metadata might have been lost with the function literal. I think that can be fixed.
@U0502D2GL Fixed on master. Issue: https://github.com/borkdude/babashka/issues/362 New binaries will appear in #babashka_circleci_builds shortly for you to try.
$ clj
Clojure 1.10.1
user=> (.printStackTrace)
Execution error (IllegalArgumentException) at clojure.main/main (main.java:40).
Malformed member expression, expecting (.member target ...)
user=>
$ bb
Babashka v0.0.84-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
user=> (.printStackTrace)
java.lang.NullPointerException
user=>