Fork me on GitHub
#babashka
<
2021-04-23
>
Karol Wójcik08:04:11

@borkdude Can I join babashka pods paths? I would like to take some default pods from ~/.babashka and some other from /.native-deps

borkdude08:04:13

@karol.wojcik I'm not sure what you want to do with the joined path?

Karol Wójcik08:04:52

I want to pack babashka runtime as an AWS Layer with babashka-aws as a default pod. Default pods will be stored in .babashka folder, but still I want allow the users to upload their own pods. Those extra pods will be stored at .native-deps directory, therefore I want babashka to look either in .babashka or .native-deps directory for pods 🙂.

borkdude08:04:32

@karol.wojcik Yes, sure. The way you would have to call those pods must be using the filename then instead of the fully qualified symbol, right?

Karol Wójcik08:04:16

I don't want the user to specify pods via the filename. From the user perspective it doubles the work to deploy an artifact. 1. Use fully qualified symbols. 2. Download and pack pods (soon via bb pods --download). 3. Rename dependencies to use filenames instead of fully qualified symbols. What I would like from babashka side is to allow extra paths for pods (like classpath).

Karol Wójcik08:04:06

Maybe via environment variable. Something like: BABASHKA_PODS_LOOKUP

borkdude08:04:55

@karol.wojcik A fully qualified symbol implies the pod is downloaded from the registry (or in the expected already downloaded location). The other option is to use the pod's filename and then it will be looked up from the system path. There is no combination of these two.

Karol Wójcik08:04:11

I don't want to combine those two.

Karol Wójcik08:04:53

I want to specify an extra "already downloaded location" for fully qualified declared pods.

Karol Wójcik08:04:19

We could use "or" in 169 and 170 line for custom babashka pods paths.

Karol Wójcik08:04:58

I would love to make a PR for it if you would like me to.

borkdude08:04:38

I don't understand the use case. Why not use the location that is already supported?

Karol Wójcik08:04:10

Ok let explain myself a little bit. Here is how babashka runtime looks like. It's packed together with following things: .babashka, .m2 and bb executable

#!/bin/sh

set -e

export _JAVA_OPTIONS=-Duser.home=.
export BABASHKA_DISABLE_SIGNAL_HANDLERS=true
export XDG_CACHE_HOME=.

./bb -cp "$(cat classpath)" -m "$(cat ENTRYPOINT)"
In .babashka i store aws-pod since it will be so commonly used with babashka runtime that it's just a waste of resources to pack this dependency on each release. All of those files will be shipped as an AWS layer which means that upon lambda invocation .zip file will be extracted in /var/task directory (layer) to which items from user lambda release artifact will be merged. Which means that user cannot pack babashka pods in .babashka, because it will remove all dependencies declared! Instead holy-lambda will pack extra user pods in .native-deps from which I want babashka to lookup for extra pods if an artifact could not be found in .babashka. Let me know, whether it makes sense to you.

borkdude09:04:22

I don't understand this part: > to which items from user lambda release artifact will be merged. Which means that user cannot pack babashka pods in .babashka, because it will remove all dependencies declared! What is it, merge or replace?

borkdude09:04:53

Are you not able to add to directories in layers, do they always replace? That seems odd to me

Karol Wójcik20:04:32

@borkdude my bad. Layer goes to /opt/var. User packed artifacts go to /opt/task. Seems that I need babashka pods classpath. ;(

borkdude21:04:36

@karol.wojcik Can you make an issue in the pods repo to clearly explain the case so I can give it some thought?

borkdude21:04:36

Thanks! I'll give it a night's sleep and will look at it this weekend

Karol Wójcik08:04:25

I can implement it on babashka side if you would like me to 🙂

borkdude09:04:10

I reacted in the issue

Karol Wójcik09:04:04

Imagine that you unzip two zips in place which both have the same directory inside.

borkdude09:04:53

I think that should merge, but doesn't it?

Karol Wójcik09:04:34

Hmm.. Actually i did not test it, but my intuition says no :D will check it out!

borkdude09:04:55

let's put it to the test

borkdude09:04:58

I tested it and it merges (like I would expect)

borkdude09:04:36

merging is not the right word: it just creates files into the directory structures, but doesn't remove any existing ones. on overwrite zip will prompt you

borkdude09:04:12

Here you have foo1.zip and foo2.zip. Both contain a dir foo with different files. After unpacking both, the foo dir has both files.

Karol Wójcik09:04:01

I hope that aws lambda does the same.

Karol Wójcik09:04:14

I will check it out and let you know!

borkdude09:04:42

I think docker works the same way, but it would be good to check

borkdude12:04:03

If someone wants to have some fun, perhaps one could find out how to get zsh (or bash) completions for a project specific bb.edn?

❤️ 5
russmatney21:04:07

I took a shot at this and have a prototype working as documented here https://github.com/russmatney/bb-task-completion

🎉 1
russmatney22:04:30

Right now it just stringifies the task definition, so looks like

russmatney22:04:11

I'm not sure the best approach for a few things - i listed those in the readme - if you have thoughts on next steps, let me know and i can take a shot at a PR, but feel free to run with this however you see fit

borkdude22:04:15

Awesome, I'll take a deeper look tomorrow :)

borkdude22:04:54

Note that there is also :doc in tasks which you could use to print

borkdude22:04:16

although I haven't used it in the case of antq's bb.edn

russmatney22:04:31

gotcha - i'll update to prefer that if it's present

Duck Nebuchadnezzar22:04:37

The stringified task definitions break with some tasks:

{:tasks
 {all {:depends [lint-eastwood]}

  lint-eastwood
  {:task (clojure "-M:dev:eastwood '{:source-paths [\"src/main\" \"src/test\"}]'")}}}

russmatney23:04:44

thanks for trying it! good find - we'll probably need to escape those quotes

russmatney02:04:26

I updated it to just remove quotes completely for now, so it should work for those cases - feel free to ping me if there are other failures

fnumatic13:04:22

Hello. I'm converting some scripts to bb task. I want to use shorthand syntax for functions. But getting an exception: "Exception in thread "main" java.lang.RuntimeException: No dispatch macro for: (" any hint?

(run!  #(cp-to % dest) (fs/glob rp "**.{css}"))
;;(run! (fn [f] (cp-to f dest)) (fs/glob rp "**.{css}") )

borkdude13:04:43

@fnumatic Since bb.edn is an EDN file you cannot use #(...) unfortunately.

borkdude13:04:42

if necessary you can split out functions into a different file and put it on your classpath using :paths

borkdude13:04:53

off topic: a nasty trick to inject code into every namespace (by modifying clojure.core) https://github.com/babashka/babashka/issues/799#issuecomment-825654751

fnumatic13:04:29

is there a short way to just copy a file to a specific directory? my bash script right now

cp $RP/index.html $DOCS

borkdude13:04:24

Does (fs/copy "foo" "bar") work?

borkdude13:04:08

(fs/copy (fs/path (System/getenv "RP") "index.html") (System/getenv "DOCS"))

borkdude13:04:16

it's not really short, but that should do it

borkdude13:04:41

you could also define a variable in :init to make it shorter

borkdude13:04:03

{:init (do (def docs (System/getenv "DOCS")))}

fnumatic13:04:57

nope. System vars are not the problem. I just cannot choose a destination dir. I always have to use explicit destdir + destfilename.

borkdude13:04:36

hmm, let me check

fnumatic13:04:02

this is my current workaround in a let block

cp-to (fn [f dir]  
         (fs/copy f (fs/file dir (fs/file-name f)) 
          {:replace-existing true}))

borkdude13:04:14

yeah this is annoying and should be fixed.

fnumatic13:04:45

good work, by the way

borkdude13:04:28

Thanks. I see the behavior is similar to:

(require '[ :as io])

(io/copy (io/file "file") (io/file "foo"))
so at least it's consistent, also with me.raynes/fs

borkdude13:04:46

Maybe we should have a function copy-into or something which copies a file into a dir

👍 1
borkdude13:04:13

so it's not a bug at least, but slightly annoying

borkdude13:04:25

@fnumatic btw if your use case it to copy a dir into another dir, then you should consider using fs/copy-tree

👍 1
borkdude15:04:56

@fnumatic please add tests. As I currently see it, the function calls itself infinitely

borkdude15:04:18

also some bikeshedding: copy-into or copy-to? any preferences anyone?

fnumatic15:04:33

would prefer copy-to

borkdude15:04:55

yeah, I think so too

borkdude15:04:36

well, copy-into is less ambiguous perhaps. copy-to could still mean that you copy file X to file Y

borkdude18:04:37

Whoops. Whenever I'm not certain about a name, I just toss a coin on Twitter. https://twitter.com/borkdude/status/1385662077600243715

lukasz18:04:19

(a new challenger appears)

borkdude18:04:21

yeah, I think copy-to is more esthetically pleasing, but copy-into is less ambiguous, so even if the poll is in favor of copy-to I may still choose copy-into based on arguments

borkdude18:04:28

cp is ambigious in itself

lukasz18:04:36

oh, so there's a difference?

lukasz18:04:51

I always liked Ruby's FileUtils module where all methods were named after shell commands

lukasz18:04:56

(mkdir, mkdir_p etc)

borkdude18:04:26

cp behaves like foo target-dir: copy into the dir, and cp foo target-dir/non-existing-or-file creates or overwrites the file, which over time you just learn

borkdude18:04:53

what's mkdir_p?

lukasz18:04:07

create directory if doesn't exist, otherwise report success

borkdude18:04:21

oh mkdir -p equivalent

lukasz18:04:26

or create a whole path structure mkdir -p ...

dharrigan18:04:56

I sorta feel that the destination should come first, given it's ...into and ...to

dharrigan18:04:12

i.e., (fs/copy-into "quux/dir" "foo/bar/baz.txt")

borkdude18:04:13

copy x into y

dharrigan18:04:23

yes, I'm aware of the implicit

Stuart18:04:37

I think destination first is confusing.

dharrigan18:04:43

but it reads better and in my mind feels better when the destination comes first

borkdude18:04:58

I think the unix world disagrees

Stuart18:04:01

I wouldnt expect that to be the behaviour

dharrigan18:04:03

you look at it and you think "I'm copying into /foo/bar/baz.txt"

borkdude18:04:59

the argument names solve that problem, copy arguments usually go from src to dest, left to right

dharrigan18:04:13

how about (fs/copy-from....)

dharrigan18:04:57

that is a perfectably reasonable choice too

dharrigan18:04:26

(fs/copy-from "foo/bar/baz.txt" "quux/dir")

dharrigan18:04:43

(fs/copy-from "foo/dir" "quux/dir")

borkdude18:04:44

The second example is already supported by copy-tree

dharrigan18:04:59

it was an example

borkdude18:04:09

sorry, I don't find it intuitive :)

dharrigan18:04:18

copy-from has better readability

dharrigan18:04:28

I'm copying from x to y

borkdude18:04:04

it doesn't explain whether you are copying from file X to a new file Y or into a dir Y

dharrigan18:04:32

that's irrelevant

dharrigan18:04:41

do you care if you do cp x y

borkdude18:04:20

babashka.fs doesn't try to emulate bash, it is based on java.nio.file and there such semantics matter

Stuart18:04:26

Everything is a file 😉

borkdude18:04:38

e.g. fs/copy is based on java.nio.file.Files/copy

dharrigan18:04:47

THe problem you are introducing the -into or -to is that the explict reading of it in english sets up a wrong premise

borkdude18:04:52

so we already have a "style guide" that we have to stick to

dharrigan18:04:16

if you read out (fs/copy-into "foo/bar/baz.txt" "quux/dir") out loud, you'll see that it just reads wrongly,

dharrigan18:04:32

ditto with to

borkdude18:04:50

(> 5 6): 5 greater than 6 (copy-into foo bar) copy foo into bar makes sense to me

dharrigan18:04:09

I'm copying into foo bar

borkdude18:04:25

foo copy-into bar

borkdude18:04:55

x op y is written as (op x y) in lisp

dharrigan18:04:03

I'm aware of lisp

dharrigan18:04:35

but you're trying to shoehorn a LISPy schematic into a file system action

borkdude18:04:58

now this is a better argument of putting the destination first: https://twitter.com/danielglauser/status/1385667548620869633 I don't agree. this is how you write functions / predicates in lisp in general.

dharrigan18:04:00

all I'm saying is that it reads wrong if you read it out, in both choices, where as the -from actually does precisely what the action is doing.

borkdude18:04:54

I think often the order of arguments can be counter-intuitive and you can explain it both ways, I'll give you that

borkdude18:04:12

(apply copy-foo dir [x y z])

borkdude18:04:58

(run #(fs/copy-to-dir % dir) ...)

borkdude18:04:45

Maybe copy-to-dir is the most explicit name we can have and, to be compatible with the other copy functions, I think it's least confusing if we put the order the same, src -> dest

borkdude19:04:12

or maybe not include it... :)

borkdude19:04:17

funny enough, me.raynes.fs didn't have this function

borkdude19:04:34

does anyone know how python or Ruby calls this function, if available?

borkdude19:04:09

Just copy in python it works for both copying into a dir and creating a new file by name

borkdude19:04:38

In fs copy already is based on Files/copy so we can't alter that behavior I think (better stick to the standard here). It's better to create another function, it's only the name that is difficult

borkdude19:04:11

ok, let's be open minded: Maybe we could change fs/copy to copy the file "into" a dir if the dest is a dir... and deviate a bit from java.nio.files. What do y'all think?

fnumatic20:04:28

That would be even better. My goal is, to get rid of various shell scripts. As of now, just copy file to file is counterintuitive. Everything is a file, as @qmstuart said.

borkdude21:04:26

ok, let's do it

borkdude21:04:23

ok, fs/copy now accepts dirs as dest: https://babashka.org/fs/babashka.fs.html#var-copy published as 0.0.4 and will be available in a new bb near you soon

🎉 3