Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-08-10

Channels

#babashka
<
2022-08-10
>
jmglov08:08:45

Along the lines of Franco's question, is there a way to include tasks from one bb.edn in another? Maybe by merging :tasks from one bb.edn in BABASHKA_PRELOADS or something? Or maybe this is just way too gross to consider. 😉

borkdude08:08:48

This isn't supported right now

jmglov13:08:48

I came up with a solution that is much better than what I had in mind (at least, in my mind). Will explain in my blog. 😉

jmglov13:08:02

Prepare to be horrified by the probable abuse of babashka-cli

borkdude13:08:10

Already loving it!

jmglov15:08:57

So yeah... didn't actually talk about babashka-cli in the post, just what it enables: https://jmglov.net/blog/2022-08-10-dogfooding-blambda-cli.html

jmglov15:08:08

Tomorrow. I promise!

jmglov15:08:42

(Though as you know, I'm a well known liar, so take my promise with the requisite amount of salt.)

borkdude15:08:56

Where's the code?

jmglov15:08:24

You're so impatient! 😉

jmglov15:08:38

Like I said, prepare to choke back some vomit. 🤢

jmglov15:08:07

But seriously, if you have suggestions for improving things, I would be very happy to hear them. 🙂

borkdude15:08:50

@U054AT6KT I think this is awesome, I mean it. The thing I wondered about was the spec merging but you did this manually. I wanted to fix this, but I'm glad you found a way.

borkdude15:08:23

I see you are checking required keys manually. There is now also a :restrict option for this, that can be set to either true or a collection of required keys

jmglov16:08:03

Nice! I’ll try out the :restrict option. I was using a validator before, but I didn’t want an exception thrown. Would you consider being able to specify an error handler function which would get the invalid value and the spec for the invalid opt?

borkdude16:08:26

Yeah, this is an area that needs some love probably

jmglov17:08:06

I’ll open an issue for it. If we come up with a good design, I can try to do a PR.

borkdude18:08:54

Yes please

1
jmglov08:08:31

I think maybe you were mixing up :restrict and :require, right? 

* `:restrict` - `true` or coll of keys. Throw on first parsed option not in set of keys or keys of `:spec` and `:coerce` combined.
  * `:require`: a coll of options that are required

jmglov08:08:01

:restrict true gets a little tricky with —help. I guess I'd just need to add it explicitly to my spec.

jmglov08:08:06

What do you think about :require true to mean all options are required?

borkdude08:08:48

sounds legit

borkdude08:08:17

but this means you can't have any optional options... is that what you want?

jmglov09:08:10

Exactly. It's not always what people would want, but in my case, it's exactly what I want, and easier than listing things explicitly.

jmglov09:08:22

I'll file an issue and you can think about it some more. 🙂

lepistane10:08:31

The saga continues 😄 So i've made the video split functionality - works great BUT I need to upload it to s3 now 🙂 I am using https://github.com/babashka/pod-babashka-aws 

(aws/invoke s3-client
                 {:op :PutObject
                  :request {:Bucket bucket
                            :Key s3-key
                            :Body (io/file file-name)}})
but whenever i am using bigger files (example 2.5GB) i am getting this error 
{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.aws.client/throwable {:via [{:type java.lang.OutOfMemoryError, :at [java.io.ByteArrayOutputStream hugeCapacity "ByteArrayOutputStream.java" 125]}], :trace [[java.io.ByteArrayOutputStream hugeCapacity "ByteArrayOutputStream.java" 125] [java.io.ByteArrayOutputStream grow "ByteArrayOutputStream.java" 119] [java.io.ByteArrayOutputStream ensureCapacity "ByteArrayOutputStream.java" 95] [java.io.ByteArrayOutputStream write "ByteArrayOutputStream.java" 156] [$fn__11554 invokeStatic "io.clj" 307] [$fn__11554 invoke "io.clj" 302] [clojure.lang.MultiFn invoke "MultiFn.java" 239] [$copy invokeStatic "io.clj" 406] [$copy doInvoke "io.clj" 391] [clojure.lang.RestFn invoke "RestFn.java" 425] [cognitect.aws.util$input_stream__GT_byte_array invokeStatic "util.clj" 104] [cognitect.aws.util$fn__9796 invokeStatic "util.clj" 137] [cognitect.aws.util$fn__9796 invoke "util.clj" 129] [cognitect.aws.util$fn__9781$G__9776__9786 invoke "util.clj" 126] [clojure.core$update invokeStatic "core.clj" 6185] [cognitect.aws.client$send_request$fn__10551$state_machine__5843__auto____10578$fn__10580$fn__10594 invoke "client.clj" 99] [cognitect.aws.client$send_request$fn__10551$state_machine__5843__auto____10578$fn__10580 invoke "client.clj" 96] [cognitect.aws.client$send_request$fn__10551$state_machine__5843__auto____10578 invoke "client.clj" 84] [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic "ioc_macros.clj" 978] [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic "ioc_macros.clj" 980] [clojure.core.async.impl.ioc_macros$take_BANG_$fn__5861 invoke "ioc_macros.clj" 989] [clojure.core.async.impl.channels.ManyToManyChannel$fn__684$fn__685 invoke "channels.clj" 99] [clojure.lang.AFn run "AFn.java" 22] [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128] [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628] [clojure.core.async.impl.concurrent$counted_thread_factory$reify__587$fn__588 invoke "concurrent.clj" 29] [clojure.lang.AFn run "AFn.java" 22] [java.lang.Thread run "Thread.java" 829] [com.oracle.svm.core.thread.JavaThreads threadStartRoutine "JavaThreads.java" 596] [com.oracle.svm.core.posix.thread.PosixJavaThreads pthreadStartRoutine "PosixJavaThreads.java" 192]]}}
How do i go around this? One way i can think of is using aws cli and not use pods at all but maybe i am doing something obviously wrong

borkdude10:08:59

Yeah you could shell out to the aws cli or use awyeah-api which is the aws lib ported to bb

👍 1
🙇 1
borkdude11:08:19

@U45SLGVHV Let me know how that goes

lepistane12:08:52

Hey i decided to go aws cli route because it offers me bit more options with syncing the bucket and folders + i can control concurrent options with it I didn't try awyeah-api unfortunately

👍 1
JAtkins21:11:14

For future people reading this, I've done some work with this recently. As far as I can tell, there's a bit of a memory problem with the http client on both the cognitect aws and awsyeah implementations. I can't seem to garbage collect the byte arrays 😕. I've ended up shelling out to the AWS CLI as well to ensure that I don't have memory leaks on long running processes.

borkdude21:11:39

Might be some unclosed resource problem. If you have a repro, I'd suggest filing issues with both libs, if you care about getting it fixed eventually

👍 1