Fork me on GitHub
#babashka
<
2020-04-15
>
borkdude07:04:11

Well look at this. Native-image executables seem to respect -Xmx!

$ bb -Xmx128m -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+VerboseGC -e '(while true (+ 1 2 3))'
[Heap policy parameters:
  YoungGenerationSize: 13421770
      MaximumHeapSize: 134217728
      MinimumHeapSize: 26843540
     AlignedChunkSize: 1048576
  LargeArrayThreshold: 131072]
[[48 msec:  GC: before  epoch: 1  cause: CollectOnAllocation.Sometimes]
[60 msec: Incremental GC (CollectOnAllocation.Sometimes) 13259K->1019K, 0.0119090 secs]
 [60 msec:  GC: after   epoch: 1  cause: CollectOnAllocation.Sometimes  policy: by time: 50% in incremental collections  type: incremental
  collection time: 11909021 nanoSeconds]]

Ahmed Hassan07:04:49

How can I download images (or other files) from web asynchronously directly to local file system, without storing in memory using babashka?

Ahmed Hassan09:04:53

Will this script fulfill the requirements I mentioned?

borkdude09:04:33

@ahmed1hsn You can use babashka.curl with the :as :stream option. E.g.:

(io/copy
  (:body (curl/get ""
    {:as :stream}))
  (io/file "icon.png"))

✔️ 1
Ahmed Hassan09:04:19

Is this feasible way to download 500 images at once? (by mapping through a list/vector of links)

borkdude09:04:51

I think so? You can also use pmap to do it in parallel

borkdude09:04:13

Or use core.async to implement some form of throttling

Ahmed Hassan10:04:32

java.lang.IllegalArgumentException: "No method in multimethod 'do-copy' for dispatch value: [clojure.lang.PersistentArrayMap http://java.io.File]" This error is given, when I try to run your code.

borkdude10:04:50

@ahmed1hsn did you try that exact code? what version of bb are you using?

Ahmed Hassan10:04:43

Exact. babashka v0.0.78

borkdude10:04:06

try bb 0.0.84

borkdude10:04:15

there has been a breaking chance in babashka.curl

Ahmed Hassan10:04:46

Update. file downloads, but it's not opening in default image viewer in Ubuntu. Maybe it's corrupted.

borkdude10:04:36

hmm @UG1C3AD5Z could you try the above snippet as well maybe on linux?

(io/copy
  (:body (curl/get ""
    {:as :stream}))
  (io/file "icon.png"))
I do have a unit test for this comparing the length of the original and downloaded one

borkdude10:04:32

fwiw on my mac it downloads correctly:

Ahmed Hassan10:04:03

This script works perfect.

sogaiu10:04:53

seems to have worked here

Ahmed Hassan10:04:07

Oh, There was problem with args, It works now.

Ahmed Hassan10:04:44

What are trade-offs between these two methods? (1. with babashka.curl 2. with io/input-stream and io/output-stream)

borkdude10:04:18

@ahmed1hsn With your method, I'm not sure if it handles redirects correctly

borkdude10:04:30

but if it works, it works

borkdude10:04:42

@UG1C3AD5Z thanks for testing!

👍 1
borkdude10:04:36

Also tested this on Windows:

Ahmed Hassan10:04:01

It seems both methods are directly saving data to file system, instead of buffering to memory?

Ahmed Hassan10:04:01

input-stream would be useful if I want to do operations on the file, so in my case babashka.curl would be more appropriate.

Olical10:04:16

How far are we from creating a babashka based shell I wonder...

Olical10:04:29

Just a thought that popped into my head when my partner mentioned that she was learning powershell.

borkdude10:04:04

I kinda prefer a REPL and writing scripts over some custom shell DSL, but this project tries to replace shells with something Clojure-y: https://github.com/dundalek/closh

borkdude10:04:30

maybe something like that could be written as a bb script though.

sogaiu10:04:02

closh's recent talk was so worth watching

borkdude10:04:47

my talk was right after that one, very appropriate because I didn't have to explain why clojure is a better alternative to bash 🙂

borkdude10:04:04

some people called that segment of ClojureD "BashD"

sogaiu10:04:40

i liked your talk too 🙂

☝️ 1
borkdude10:04:43

before that talk there was @rahul080327 with his talk on bob (data driven docker / build tool using clojure). It was a nice trilogy

borkdude10:04:50

I wasn't fishing for compliments btw, just found it funny that there was an entire afternoon of ops/scripting things 🙂

borkdude10:04:02

and yeah, I liked Jakub's talk as well 🙂

sogaiu10:04:04

nice to have a theme like that though -- at least if you are into that stuff

borkdude10:04:27

there were two tracks, so people who were not into it had some choices 🙂

Ahmed Hassan10:04:04

Can't we accomplish everything Closh does with Babashka?

borkdude10:04:24

I think a lot of it can be written as some script maybe

borkdude10:04:25

hmz, I can't get bb on Windows working anymore. I tried re-installing that redistributable again and again.

sogaiu10:04:58

oh no, not that again

sogaiu10:04:14

is there a specific executable to try?

borkdude10:04:11

I probably messed up my Windows VM when trying to compile babashka locally in it

sogaiu10:04:33

you mentioned some other windows issues recently -- may be they are related?

borkdude10:04:15

ah now it works again, I probably installed the wrong one (x64 vs x86?)

borkdude11:04:02

Nice, they now documented this for Windows: https://github.com/oracle/graal/issues/1852

sogaiu11:04:05

finally! too bad they don't seem to give hints about automating the installation process...