Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2019-05-18

Channels

#clojure
<
2019-05-18
>
todo04:05:05

Given Clojure/JVM, cljs, graal, ... what is the 'lowest startup delay' method of running clojure on aws lambda ?

art06:05:17

@todo clojure on jvm8 is as fast as go, the jar is even smaller than go binary. But don't try to use 128mb of mem, too small. The rest is too abstract, you need to run your benchmarks depending on the case. VPC is still slow so probably if you really need lambda, you are limited to public databases like Aurora Serverless (MySQL atm, Postgres in preview), Dynamodb, Kinesis and so — the ones that give access through iam role and not force you to keep everything in a VPC. They said they should fix it some time in 2019. And this is applicable to all runtimes. From my observation, clojure does not really add any big overhead.

val_waeselynck06:05:05

> clojure on jvm8 is as fast as go @todo As fast for startup time? I very much doubt that, especially if you need to load more than just clojure.core!

art06:05:16

Dependencies is the thing to keep an eye on, right.

art06:05:32

Like I said, it does not add big overhead, but still does some. If a lambda somewhere behind Kinesis Stream, Dynamodb Stream, SQS, SNS, it's fine to have it starting a bit longer than go or node.js.

art06:05:18

I would not use lambda as an http handler tho, forgot to mention that, sorry.

borkdude07:05:18

@todo I don’t know about AWS Lambda but on my machine native with Graal is definitely the fastest. it can also be a bit limiting in terms of what it supports, e.g. no eval

Busy escaping the comfort zone08:05:53

Hi, I'm trying to change the default print output for functions to into 

(doc f)
While the following works fine in the default clj repl 
(remove-method print-method clojure.lang.Fn)
  
(defn- pretty-demunge
   [fn-object]   
     (let [dem-fn (clojure.repl/demunge (str fn-object))
            pretty (second (re-find #"(.*?\/.*?)[\-\-|@].*" dem-fn))]
      (if pretty pretty dem-fn)))
      
(defmethod print-method clojure.lang.Fn [f w]  
    (.write w (:doc (meta (find-var (symbol (pretty-demunge f)))))))
It doesn't work in lein repl (I keep getting the default #function output, any idea why this is the case? (my guess is that I need to implement an nrepl middleware but id like to find an easier way of doing this)

todo19:05:09

@art @val_waeselynck @borkdude: this is my fault for not specifying it earlier -- but yes, I want to use Clojure on Lambda as HTTP REST API / handler. This is why I care about the code start time.

todo19:05:21

What is the best way to minimize the clj/lambda startup delay?

kulminaator19:05:22

you're having in mind aws lambda ?

borkdude19:05:30

@todo as I said, on my own machine Graal is definitely the fastest startup, but I don’t know how AWS Lambda works

todo19:05:18

@borkdude: I know nothing about Graal+CLJ. Is this something that can be deployed to Lambda? (I"ve deployed Go, Rust, and Clj-JVM functions.)

borkdude19:05:10

Graal can compile to a binary for linux, MacOS or Windows. I assume AWS can run linux binaries

kulminaator19:05:35

aws lambda's jvm startup is a major pain , there is no real way around it (i have invested tens of hours onto optimizations there) ...

kulminaator19:05:06

if you want stuff that starts fast, pick either python or nodejs there. (or maybe graalvm and native-image ... maybe maybe)

kulminaator19:05:12

maybe clojurescript on top of nodejs ?

borkdude19:05:20

lumo might also work?

borkdude19:05:25

or yes cljs + node

todo19:05:44

got it, so if I want clj-like on lambda, I should look into: clj/graalvm, cljs/nodejs, planck, lumo ?

kulminaator19:05:59

even plain java stuff on the java vm is slow as hell for the first request

todo19:05:39

I can't tell if 'y' is for 'yes' or 'why' 🙂

kulminaator19:05:42

by the looks of it amazon is actually booting up a fresh jvm there and the performance of those has never been great off the line ... even my java lambdas are having trouble and they are much smaller than clojure itself.

todo19:05:33

so the overhead of planck, lumo, and cljs/nodejs should all be ~ the same right? boot a js vm + load cljs primitives ?

borkdude19:05:37

@todo if you’re really going for speed I would use a compiled CLJS -> nodejs project, no self-hosted stuff

todo19:05:36

I'm really not a fan of Node. Googling "clojure graalvm" is bringing up lots of youtube talks. Do you have a link for setting up a nice clojure/graalvm workflow?

borkdude19:05:44

or just use lein uberjar. there’s an example here: https://github.com/borkdude/cljtree-graalvm

todo19:05:44

dumb question: so during development, I'm still developing vs JVM Cloljure REPL, but when everything is done, I say "hey lein, take these java class files, and convert them over to graalvm" ?

borkdude19:05:47

@todo for questions about Graal, there’s also the #graalvm channel

borkdude19:05:07

that’s how I’ve been developing https://github.com/borkdude/clj-kondo

todo19:05:16

this makes sense; thanks

borkdude19:05:29

there are some things that GraalVM cannot handle, e.g. eval. there’s also an issue with locking. so YMMV.

daniel.spaniel21:05:21

i am curious if there is a simple way to recursively descend into a nested hash and make a copy that has certain keys removed .. i have tried clojure.walk and friends but that is good for transforming keys and not removing them

jsa-aerial21:05:29

generally, this is a perfect use case for specter. And it is totally general and will always maintain data types. In my experience/use it is simpler and far better for this sort of thing than core stuff + clojure.walk/*

daniel.spaniel22:05:37

good point @U06C63VL4 I have watched the video on specter 2 times actually and thought of it but my partner prefers to use less libraries ( BUT ) in this case it makes sense ( so thanks for suggestion )

✔️ 4
seancorfield21:05:39

update-in with dissoc?

daniel.spaniel21:05:29

oh .. let me try that one ( was not sure if that would descend the hash )

daniel.spaniel21:05:58

problem with update-in is that is expects the path to dissoc from but I am decending tree and don't know the path exactly .. i guess I could collect those paths ? but might be better way ?

ivana21:05:14

maybe assoc them by nils can help? or why you need "copy" without some keys? you have a side function, which logic depends on this keys?

Alex Miller (Clojure team)21:05:15

clojure.walk/postwalk-replace can be used for this

daniel.spaniel21:05:48

i tried post-walk .. but i could not figure out how to >> remove << just transform the value

daniel.spaniel21:05:43

so i can transform it but not remove it with the walkies function

daniel.spaniel21:05:15

i have a tree with key values and want to walk and remove certain keys

daniel.spaniel21:05:01

so yes, i have a function to determine which to remove

Alex Miller (Clojure team)21:05:12

user=> (clojure.walk/postwalk #(if (map? %) (dissoc % :a :b) %) {:a 1 :e {:c 2 :b {:f 3}}})
{:e {:c 2}}

Alex Miller (Clojure team)21:05:30

something like that to remove all :a and :b at all levels

daniel.spaniel21:05:12

oh .. wow .. trying that now

Alex Miller (Clojure team)21:05:56

basically it will (bottom first) visit every node of the tree and replace the node it's on with the value returned from the fn

Alex Miller (Clojure team)21:05:20

the if is important to only apply the dissoc on map nodes, and not on individual leaves, etc

daniel.spaniel21:05:35

ok .. got it .. it works really well .. and i love it of course, but gimme a few moments to let it soak in and then i will change the if to dissoc the keywords that i need

Bravi22:05:22

Hi everyone. I’ve got this little code 

(let [image-fields [:thumbnail_image :featured_image :display_image]]
    (cond-> updated-data
      (pos? (count changed-images))
      (assoc :images (mapv #(image-updates in-updated in-original %) changed-images))

      :always
      (dissoc :thumbnail_image :featured_image :display_image)))
and I want to apply that dissoc using image-fields but cannot figure out how to. so basically I want to do something like 
:always
(apply dissoc image-fields)
but because it’s in a cond->, I guess it ends up as being (apply updated-data dissoc image-fields). I tried it like this as well 
:always
#(apply dissoc % image-fields)

Alex Miller (Clojure team)22:05:34

rather than tacking that on the end, "always" stuff can be done at the beginning

Bravi22:05:01

do you mean up here? 

(cond-> updated-data

Alex Miller (Clojure team)22:05:11

(cond-> (apply dissoc updated-data image-fields) ... )

Bravi22:05:24

yeah that makes sense 😄 thank you

Alex Miller (Clojure team)22:05:31

cond is for conditional things :)

Bravi22:05:09

thanks again. it cleared up a few other things before that chunk of code 👍

daniel.spaniel23:05:54

i've been getting drunk on this code for a few hours

daniel.spaniel23:05:57

(clojure.walk/postwalk #(if (map? %) (dissoc % :a :b) %) {:a 1 :e {:c 2 😛 {:f 3}}})

daniel.spaniel23:05:13

to remove key/values from a tree when key matches a pattern

daniel.spaniel23:05:09

but could not figure out how to make it so that i dynamically check the keys ( to see if they match a regex ( lets say ) .. this code assumes that they are known keys like ( :a or : b ) but I don't know the keys in advance.

daniel.spaniel23:05:53

i tried oh so many things, but i could not figure it out ( even tried specter )

seancorfield23:05:43

@dansudol You have the whole map right there. You can get the keys and do whatever you need, based on them.

daniel.spaniel23:05:52

even nested keys ?? i guess you suggesting i ( step 1 ) pick out the paths i want to remove ( with my function ) then step 2 ( dissoc them ? )

daniel.spaniel23:05:51

my tree ( hash ) also looks more like {:a [{:b 1} {:c 2}] :d { :e 4}} where key values can be array of more hashes

seancorfield23:05:14

Perhaps you could explain more clearly what problem you are trying to solve?

seancorfield23:05:17

With postwalk, you can look at the whole of each hash map within the tree -- inside the (if (map ? %) ... % ... ) branch

daniel.spaniel23:05:29

just walking tree and removing key/values that match a pattern ( like if key has "ui" in it )

seancorfield23:05:43

Then, yeah, you can absolutely do that.

daniel.spaniel23:05:00

i know i can do that .. i just flailing .. but i keep trying though ..

seancorfield23:05:53

(apply dissoc % (map keyword (filter (partial re-find #"ui") (keys %)))) -- something like that?

seancorfield23:05:10

(I should pull up a REPL and try that before recommending it)

daniel.spaniel23:05:33

oh .. so your saying if you find a hash do that

daniel.spaniel23:05:43

ok .. let me slap that sort of thing in the REPL myself

seancorfield23:05:35

You may need to check that the list of keys returned that way is not empty before attempting the dissoc...

seancorfield23:05:03

Hmm, no, dissoc supports just one argument so that should work without a condition.

daniel.spaniel23:05:20

got it but your idea is to get list of keys to remove .. ok .. that interesting

seancorfield23:05:15

If your problem is "remove all keys that have ui in their name, from all hash maps in the tree" then that code solves that. But I get the impression that isn't quite the actual problem?

daniel.spaniel23:05:45

no no .. that is exactly it

daniel.spaniel23:05:48

it is that simple

daniel.spaniel23:05:04

almost got it .. gimme a few more minutes

daniel.spaniel23:05:18

this was good suggestion by the way

daniel.spaniel23:05:48

got it .. oh man .. that was 3 hours well spent .. thanks @seancorfield and @alexmiller for both suggestions .. that was a whooper