Fork me on GitHub
#boot
<
2019-01-13
>
flyboarder00:01:52

oooo micha thats a good idea, we should include a lein->boot task by default

micha00:01:36

i think leiningen exposes a lot of that as a library

micha00:01:52

i seem to recall making something like that a long time ago

flyboarder00:01:10

i’ll look into it, seems like it would be fairly simple to slurp it up either way

micha00:01:11

like before boot had the minimal set of tasks you need, we used it as a crutch

micha00:01:34

yeah i think there are lots of ways you can specify dependencies in leiningen

micha00:01:46

like various profiles and whatnot

micha00:01:07

i don't think there is a simple single place in the project.clj to look to find dependencies

micha00:01:17

there are dev dependencies, test dependencies, etc

flyboarder00:01:51

ok, then the task would either need to replicate some of the lein config or utilize lein to parse the project file

flyboarder00:01:31

that seems rather doable, the boot-shadow task works the same way, we use shadow-cljs as a library

micha00:01:32

right, i'm pretty sure i recall loading the leiningen namespaces and using those

flyboarder00:01:55

a few other items on my radar are the dependency loading in general, i’d like to provide options for people who also use tools/deps.edn

flyboarder00:01:18

the next version of the boot-shadow task will exclusively pull deps from the shadow-cljs.edn file

micha00:01:35

yep, the boot cp stuff should be pretty good start for tools.deps integration

micha00:01:55

tools.deps enforces "pedantic mode" or whatever they call it

flyboarder00:01:51

yeah the nice thing with pods is that we can mix and match

flyboarder00:01:42

like the shadow task only pulls deps into the pod it uses for compiling, it leaves all the runtime deps to npm

micha00:01:45

that's perfect

micha00:01:45

man i'm having some docker troubles

micha00:01:50

do you know about docker?

micha00:01:30

i guess this isn't the docker channel 🙂

jaide01:01:51

What is the process for creating a custom clean task deletes all files in a folder except for .git?

micha01:01:27

boot isn't really designed for that kind of task, it's designed to leverage immutability in builds

micha01:01:49

so managing a mutable tree of files is not really someting you probably want to use boot for

micha01:01:43

in general, boot won't alter any of your files at all

jaide01:01:01

Oh, the use case is when I run build it outputs to the target/public directory and cleans it. But now I want to make the public/target a submodule to a github pages repo so cleaning is fine, I just don’t want it to delete the .git folder if possible.

micha01:01:13

boot tasks only create or modify boot-managed files in the fileset

micha01:01:18

and those are all immutable

micha01:01:56

the only place where boot intersects with your project files is when it writes to the target dir

micha01:01:11

and that is considered write-only as far as boot tasks are concerned

micha01:01:28

you don't want a task to read from the files in there

micha01:01:19

you can look at the target task, i bet it would be easy to copy and paste that with a slight modification to do what you need

jaide01:01:41

Ah ok, that’s what I’ve been looking through currently but was curious if that was the best way.

micha01:01:07

you could just add a --blacklist option perhaps

micha01:01:12

to the target task

micha01:01:23

and have it just ignore files that match a regex or something like that

micha01:01:30

like while it's cleaning

micha01:01:19

there is also the -C --no-clean option

micha01:01:27

in the existing target task

micha01:01:53

you kind of want a more granular no-clean i guess

jaide01:01:57

Yeah, I saw the no clean but cleaning is still a good idea. The blacklist option sounds like what I’m after.

micha01:01:35

👍 Should Be Easy ™️

flyboarder01:01:22

@micha I do know docker!

micha01:01:08

i'm mapping in /var/run/docker.sock in my container so i can run another container inception style

micha01:01:19

that works, i can docker in docker that way

micha01:01:46

but when i map volumes it looks like i can't map a volume from level1 docker into the level2 docker

micha01:01:55

i need to map the host directory

micha01:01:08

which makes sense, but it would be great if there was a way around that

flyboarder01:01:42

hm, I know there are volume mapping issues with docker-in-docker

micha01:01:15

this isn't technically docker in docker, i'm just making /var/run/docker.sock available in the level1 container

micha01:01:23

so it can basically call out to the host docker server

micha01:01:35

i don't run a docker server in the container

micha01:01:37

ah yes, this SO question is exactly what i'm looking for

flyboarder01:01:08

yeah it’s not technically docker-in-docker, but I think the symlink might work for you

micha01:01:31

interesting

micha01:01:48

my workaround was to just have the paths be the same in docker as on the host

micha02:01:46

haha that's an answer there already i see

flyboarder02:01:53

yeah, I am always weary of doing stuff between host/docker, I generally opt for either in docker or out of docker

micha02:01:55

zero upvotes tho

micha02:01:20

yeah my situation is i have some automation around deployment, cloudformation, and so on

micha02:01:48

so for managing the infrastructure stuff i like having a docker container with the right versions of everythying, like npm for aws lambda functions, etc

flyboarder02:01:56

yeah I like the solution you have tho, it makes much more sense instead of a random symlink

micha02:01:03

but i also have submodules for the various services, clojure services and whatnot

micha02:01:17

and those all have docker containers to build them

flyboarder02:01:37

omg, so much dockerception

micha02:01:38

so i need to run the build docker container while i'm in the infrastructure container

micha02:01:43

haha yeah

micha02:01:13

but it does achieve the goal of not maintaining a "how to prepare your system to build this service" for every service and every OS

flyboarder02:01:13

I like this, we use a “boot is the environment” thing so it’s our only hard dependency

micha02:01:18

what about like nodejs dependencies and so on?

micha02:01:52

like for lambda functions on aws i need a specific version of npm

flyboarder02:01:20

I have never needed anything but the latest version

micha02:01:07

we use a few different versions, heh

flyboarder02:01:18

# Node Build
FROM node:alpine AS node

# App Workdir
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Node Dependencies
COPY ./package.json /usr/src/app
RUN npm install

# Boot Build
FROM  AS boot

# Boot ENV
ENV BOOT_JVM_OPTIONS=-Xmx2g

# Boot Cache Dependencies
RUN boot repl -e '(System/exit 0)'

# App Workdir
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# App Files
COPY --from=node /usr/src/app .
COPY . /usr/src/app

# App Build
RUN boot build

# Node Runtime
FROM node:alpine AS app

# App Workdir
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# App Files
COPY --from=boot /usr/src/app/target .

CMD ["node", "nodejs"]

EXPOSE 8080

flyboarder02:01:28

This is the build we copy-paste everywhere

flyboarder02:01:03

so first we prep and cache as much as we can, then we download all the npm deps, then build with boot, and finally move the app to a fresh node image

flyboarder02:01:16

If only there was a build tool in clojure you could wire this up with 🙊

micha02:01:25

it's like just get docker and run the build container

micha02:01:38

haha one of the best reasons to use docker is because you can then write bash scripts without needing to work around OSX braindead bash and coreutils

flyboarder02:01:09

yeah, thats why I want to get boot running as a native image, so I can just Lisp all the things™️

jeroenvandijk09:01:38

@flyboarder +1 🙂 I think there is some common interest between this and clojure shell project closh. We've also looked into GraalVM. There are also some features in boot that closh is interested in (e.g. boot pods) (https://github.com/dundalek/closh/issues/119).

jeroenvandijk09:01:52

I believe there could be some synergy between the projects

flyboarder15:01:52

Awesome! Once I start work on splitting the pods into their own library maybe you can help me test it out

flyboarder02:01:22

without the performance hit of jvm

flyboarder02:01:14

it works but it’s highly alpha, Im going to try and get my containers moved over to the native-image and see what kind of performance issue arise

micha02:01:16

do you think graal could be used to build a program that uses for example aws sdk?

flyboarder02:01:22

hm, I think I saw something about that, but the thing about graal is that it’s closed world, so you need everything at build time

flyboarder02:01:31

I would imagine the aws sdk is loaded on demand

flyboarder02:01:35

since it’s so huge

micha02:01:41

you feed it an uberjar i guess?

flyboarder02:01:59

yeah that might work

flyboarder02:01:18

easiest way to find out is use the graal docker image and try building your app

micha02:01:48

i have PTSD from the cljs --advanced optimizations days

micha02:01:56

i may get triggered

flyboarder02:01:31

if it doesnt compile then it’s not worth investigating, with boot it compiled fine and I only had to sort out dynamic loading issues

flyboarder02:01:54

but it seems most projects get stopped at the compile step

flyboarder02:01:00

and never get to runtime issues 😛

flyboarder02:01:31

also graal isnt even v1 yet

flyboarder02:01:43

so I dont plan on releasing the boot version until that happens

micha02:01:52

i am astounded that you were able to get all the crazy classloadering to work in graal

flyboarder02:01:13

well it’s fairly simple, you delay all the dynamic stuff to the spawned jvm

flyboarder02:01:21

then it’s business as usual

micha02:01:43

ahh i see

flyboarder02:01:46

really we just shift all the static stuff to graal and compile that to native

flyboarder02:01:53

so it’s really fast

flyboarder02:01:24

but we do drop 1 level of classloader shims which is nice

micha02:01:15

that is sweet

flyboarder03:01:00

If anyone else is interested in the experimental work on boot:

martinklepsch19:01:13

@flyboarder huge kudos for taking over 🙌 Consider applying for the next ClojuristsTogether batch, I think boot would be a great candidate and there's definitely some stuff to do 🙂

simple_smile 5
flyboarder21:01:58

@martinklepsch I have put my hat in for that! Any funding would help go along way!

🔥 5
🎉 5
jeroenvandijk09:01:38

@flyboarder +1 🙂 I think there is some common interest between this and clojure shell project closh. We've also looked into GraalVM. There are also some features in boot that closh is interested in (e.g. boot pods) (https://github.com/dundalek/closh/issues/119).