Fork me on GitHub
#beginners
<
2023-10-18
>
Pavel Filipenco08:10:31

What's the best fit channel to share solutions for leetcode/etc. problems?

kennytilton09:10:28

I do not know of one. If no one else does, it sounds like a channel I would like! Some of the most informative threads on c.l.lisp followed from shared code everyone dissected. Maybe #C053PTJE6?

seancorfield16:10:05

There's a #C01J99RN4G5 channel (I don't know its contents or purpose).

Pavel Filipenco16:10:42

Seems dead, I guess I'll populate it :)

leifericf11:10:39

I have successfully compiled my first Clojure web app to a .jar file, which runs on my machine with java -jar myapp-0.1.1-standalone.jar Next, I must package it in a Docker image, deploy it to Kubernetes (in Microsoft Azure) via GitHub Actions, and then run it there. Does anyone know of a blog post or similar on how to do something like that?

futurile14:10:06

I don't see any article links - could you paste them in again? - as I'd be interested in reading about how to do this :-)

practicalli-johnny16:10:44

Multi-stage docker file is an effective way to build the Clojure app in CI and deploy a minimal resources container in the Cloud ☁️☁️☁️ https://practical.li/engineering-playbook/continuous-integration/docker/clojure-multi-stage-dockerfile/

👀 1
leifericf04:10:24

That’s awesome, @U05254DQM! Thanks for the link. I think I found an older version of that site, but the one you shared is presented much nicer.

Felix Dorner20:10:49

If I have a “multi-module” project with libA and libB as subdirectories, can I define, at the root level a common deps.edn and “include” that from the libA/libB deps.edn files, so that I get consistent versions for common dependencies like cheshire etc? Or would I maybe create an empty lib “deps” with just a deps.edn, and then depend on that from both libraries?

dpsutton20:10:58

you could make a sibling to libA and libB called common that defines them and libs A & B could use cheshire transitively from common

pithyless07:10:08

if you’re OK with a preprocessing step that updates your deps files to keep them in sync, I’ve previously had good success using this approach: https://github.com/exoscale/deps-modules

pithyless07:10:04

the shared common utils is fine until you want more granular control of which deps are “common”

Felix Dorner07:10:19

I’m just used to maven parents and <dependencyManagement> sections where we just declare the maven coords and in the children we only reference the artefact but without the version. Maybe your preprocessing does something similar to that.

pithyless07:10:54

Yeah, you can achieve something similar by using :override-deps and merging with a second "base" deps.edn file.

pithyless07:10:01

But there are many gotchas - e.g. the official clojure tooling doesn't allow passing in multiple files, so you often see hacks with overriding the path to your home deps.edn file (that is also always read and merged with your project level one).

pithyless07:10:15

But then you need to control ENV, etc.

pithyless07:10:49

I've seen this break in too many ways where some tooling, linter, editor, etc. doesn't work the way you'd like and you get lots of paper cuts

pithyless07:10:23

I like the exoscale approach where it's a preprocessing step, you can just commit the file to git, and all your tooling works without knowing about any of this.

Felix Dorner07:10:57

Thanks for sharing this!

pithyless07:10:15

Another example on the opposite side of the spectrum to solve this problem is #C013B7MQHJQ which "takes over" your classpath/project structure/etc. But then they work really hard to make all the tooling, etc. work for you :)

pithyless07:10:01

If you search for monolith or monorepo on #C6QH853H8 or maybe general slack (especially with keywords like :override-deps ), I'm sure you'll find lots of threads that go into all the nuance of what kind of paper cuts you can expect with various solutions.

pithyless07:10:36

Which is not to say that it's not worth solving; I'm #TeamMonorepo :)

Sahil Dhanju22:10:45

I've been writing large-ish transducing functions which is just a data pipeline that does a bunch of stuff and persists stuff to a db but doesn't actually produce anything in the end. To use this pipeline I've been writing this

(transduce my-data-pipeline-xf (constantly nil) input-data)
is there a better way of doing the call above? Something that executes a transduce but doesn't need this reducing function - i.e. in my case (constantly nil)

phronmophobic22:10:36

I don't think there's anything wrong with that (see run!), but it might be more beneficial to have your "write to db" function be the reducing function in this case.

👍 1
Sahil Dhanju22:10:56

I do several writes to the db within the pipeline at different steps

phronmophobic22:10:14

Maybe that's ok. Without knowing too much about the use case, I would be worried about how the pipeline handles failures if writes are happening independently.

Sahil Dhanju22:10:54

Ok great, thanks for the answer

Sahil Dhanju22:10:49

I'm scraping over a paginated API which returns json that has other paginated APIs, so I use iteration inside this transducing pipeline to go over all of these objects and persist them into my db

phronmophobic22:10:09

Ideally, you would build up either one large transaction, or maybe a sequence of transactions that get atomically applied with some appropriate error handling. A typical case where that's awkward is if you are writing to multiple destinations.

phronmophobic22:10:40

But, even for multiple output destinations, it's nice to have a reified log of logical transactions that gets produced separate from the process that applies them and does error handling.

Sahil Dhanju22:10:42

That makes sense but I'm not sure how to do this with a transducer - for example

Sahil Dhanju22:10:40

My pipeline has steps A B and C, at each step I want to write something to the db and then modify the object passed into each step. This modified object gets sent to the next step

phronmophobic22:10:46

Does the modified object depend on the result of writing to the db?

Sahil Dhanju22:10:01

in reality what this looks like is

map f
cat
write to db
map g
write to db
map h
cat
cat
filter
map
write to db
etc.

phronmophobic22:10:05

YMMV and it depends, but one approach that I like is to just pass a long a map through the pipeline that accumulates more information.

phronmophobic22:10:19

At the very end (ie. the reducing function). Takes the map and applies all the I/O

👀 1
Sahil Dhanju22:10:22

How does this map sit next to the object I'm passing

phronmophobic22:10:01

you just add another key-value to the map

Sahil Dhanju22:10:34

How does this work with cat? I definitely understand the bigger picture of what you're saying but don't fully get the specifics

Sahil Dhanju22:10:08

Mind if I dm you?

phronmophobic22:10:31

This might be a helpful convo for others.

👍 2
Sahil Dhanju22:10:44

What would be your idea for how to pass this map with a cat?

Sahil Dhanju22:10:17

Will take a look, thanks!

phronmophobic22:10:06

I haven't tested this code, but here's some psuedo code with xforms:

(comp
 (mapcat f)
 (x/multiplex [(map (fn [x]
                      {:foo x}))
               (comp
                (map g)
                (x/multiplex
                 [(map (fn [g]
                         {:g g}))
                  (comp (map h)
                        cat
                        cat
                        (filter f)
                        (map i)
                        (map (fn [x]
                               {:other x})))]))]))

phronmophobic22:10:47

It gets kind of messy, but you end up with a sequence of maps that are marked with the data. At the end, you just have something that checks the keys and writes them to the db.

phronmophobic22:10:31

I probably wouldn't write it as one big hairy transducer, but write the pipeline using a format close to the way I think about the problem and then have another function that takes the description and returns the pipeline.

Sahil Dhanju22:10:45

That makes a lot of sense

phronmophobic22:10:34

the function that takes the description and returns a pipeline can also add error handling or you can do the error handling in the reducing function that writes to the db

phronmophobic22:10:59

and you can have a policy that independently decides: • quit on first error • ignore errors • log errors, but proceed • log errors, but quit after X errors • etc

👍 1
Sahil Dhanju22:10:09

What do you mean by "description"?

phronmophobic22:10:43

Also, generating a sequence of transaction can be more efficient since it's trivial to batch and pipeline them.

phronmophobic22:10:28

> What do you mean by "description"? The way I like to think about it is if you wanted to describe your pipeline to someone else or "future you", what would you write down on a whiteboard or piece of paper.

phronmophobic22:10:11

That description can almost always be mapped to some terse edn description.

Sahil Dhanju22:10:14

Interesting, I see

phronmophobic23:10:12

But this is all just general advice and food for thought based on similar projects I've done in the past. It may or may not be overkill for your use case. Depending on number of items in the pipeline, I might just write something simple like:

(let [foos (into
            []
            (mapcat f)
            input)
      
      bars (into []
                 (map g)
                 foos)
      bazs (into []
                 (comp (map h)
                       cat
                       cat)
                 bars)]
  ;; run transactions
  )

👍 1
phronmophobic23:10:59

It does mean you will have to hold all the intermediate results in memory, but that's not always a problem.