Fork me on GitHub
#tools-build
<
2024-05-29
>
bhurlow20:05:38

I have a monorepo project with several deps.edn modules. Some of these modules are depended upon by an application which is shipped to AWS lambda. AWS lambda has a size limitation on the uploaded jar, so I’m running into the limit when I create an uberjar. To remedy this, we’ve been using b/write-pom to package the jar dependencies into a lambda layer, then using b/jar (not uber) to compile only our src into the shipped jar. It looks like b/write-pom does not roll up dependencies from :local/root so we’re starting to see issues where some required deps are not being included. I’m guessing I can generate a zip of jars per module, but wanted to check and see if there’s a better solution to get the full module tree pom created. Anyone have this issue?

Alex Miller (Clojure team)20:05:48

write-pom can't include local deps, because there is no way to specify that in the pom

Alex Miller (Clojure team)20:05:03

the basis includes information on the local libs and their location, so you could copy source paths from local jars into your target dir for inclusion in the final jar

Alex Miller (Clojure team)20:05:52

that code effectively exists inside the uber jar, I guess you could also uber just the local libs with a custom basis and then unjar that into your target dir

Alex Miller (Clojure team)20:05:32

if some subset or internal part of tools.build would be useful as a task, I'm happy to look at breaking it out

bhurlow20:05:53

amazing thanks @U064X3EF3. Our :local/root dependencies are pointing to deps.edn projects, not jars. It definitely seems like clojure.tools.build.tasks.write-pom has the parts we need, since the deps were interested in are in the basis

Alex Miller (Clojure team)20:05:30

write-pom just writes the pom file, I assume you actually need the contents of the local project paths?

bhurlow20:05:25

we want the local project path jars to be assembled separately from the compiled src code due to the lambda limitation

Alex Miller (Clojure team)20:05:42

that's more similar to code in the current uber task

👍 1
bhurlow20:05:48

but I guess we could copy them in first prior to compilation….

Alex Miller (Clojure team)20:05:24

the code is kind of spread throughout the uber task, but it's essentially using :libs in the basis, looking for lib paths that point to a directory (will be the :paths from local deps), then copying those libs to the target dir where the uber is built. the latter part is not hard but is careful about things like retaining file timestamps (which can affect whether compilation happens)

bhurlow20:05:38

looks like I may not really need to write a pom, ultimately I just need a zip of all the lib entries in the basis, don’t even need to compress them into a jar since lambda can work with a zip of jars to add to the classpath

Alex Miller (Clojure team)20:05:57

yeah, the pom is probably unimportant if you're not deploying it to maven

Alex Miller (Clojure team)20:05:18

I'm trying to think about what sensible part of this uber task could be extracted for reuse

bhurlow20:05:23

it was a mechanism by previous dep to achieve the packaging

bhurlow20:05:40

yea if some of the uber ns could be depended upon that would be great

Alex Miller (Clojure team)20:05:38

that's kind of bound up with all of the additional handlers that uber provides though

bhurlow20:05:29

is there a reason I can’t iterate over the :libs entry and copy each into a dir and zip those? this looks like its packing up the jar which may not be needed for me

Alex Miller (Clojure team)20:05:26

you can feel free to remix whatever you need from here, there's no magical stuff happening, it's all just basically copying files from jars and libs into a dir and zipping it

bhurlow20:05:41

totally, thanks for the support, this makes sense and I can work with those bits

bhurlow20:05:55

if there’s something particular to extract I can let you know

Alex Miller (Clojure team)20:05:18

not readily apparent to me without doing the work, but if you identify something like that, happy to look at it. lambda layer building is something that people have built (I know @U050ECB92 did a version of this, but don't think it was ever made public), would be great to consider something reusable

bhurlow22:05:24

Iterating over the :libs from basis appears to work great