As far as I know there is no widely accepted correct way of dealing with exceptions. Most people just use plain try/catch with (ex-info "msg" {:data :data}) exceptions. There are a handful of libraries that provide monadic error handling / containers that indicate whether or not there's a successful result inside or an error. It's also common for people to create macros that implement some generic error handling / logging according to their preferences in their application. Here are a few things that come to mind: https://github.com/scgilardi/slingshot https://github.com/cognitect-labs/anomalies https://github.com/MichaelDrogalis/dire

Regarding part 2, you might find this an interesting read: https://vvvvalvalval.github.io/posts/2018-07-23-datascript-as-a-lingua-franca-for-domain-modeling.html

thanks

you could look at the code for distinct and make a version that gives you both a in-order vector, and a set for testing membership

it would mean lifting the result list / set into a tuple I think

Interesting! I'll take a look. It turned out that clojure.string/includes? really spoke to the nature of the problem I'm solving + performed significantly faster than using some.

For some reason I was expecting it to be faster

I haven’t used it, but there appears to be a Faktory lib: https://github.com/contribsys/faktory, https://github.com/apa512/clj-faktory

would you mind elaborating on what you mean by this?

Well, spring libraries are some of the more well maintained and feature-ful java libraries. The way they are documented for usage is through annotations on classes, so even if its possible to wrangle classes in the right way dynamically its not always clear how or i am willing to bet possible

https://spring.io/guides/gs/batch-processing/

and clojure only supports annotations inside of gen-class

which is both non-interactive and obtuse

and requires special build time steps in order to use correctly

https://spring.io/guides/gs/service-registration-and-discovery/

lets say i wanted to use this in clojure

whole java library, hooks me up to service discovery stuff

Maybe im wrong, but it definitely feels like clojure's java interop falls short

thanks for the detailed clarification!

i guess this doesn't quite help then? https://clojure.org/reference/datatypes#_java_annotation_support

the only example i had seen before was inside gen-class - didn't realize there was one more type of place it could be done

i just tested it, and iiuc, ^:bar is metadata for :baz -- though this wouldn't work because :baz is a keyword which cannot have metadata. but assuming you had chosen a vector instead of :baz, the vector would have the metadata attached. the function which would handle the foo tag (set via *data-readers* typically), would be given the vector with the metadata attached.

something like the following session is what i did:

user=> (defn my-tag-fn [x] (binding [*print-meta* true] (prn x) [1 2]))
#'user/my-tag-fn
user=> (set! *data-readers* {'my-tag my-tag-fn})
{my-tag #object[user$my_tag_fn 0x256f8274 "user$my_tag_fn@256f8274"]}
user=> #my-tag ^:a []
^{:a true} []
^{:a true} []
[1 2]
user=> 

so you can see that the vector [] had the metadata attached to it and it was outputted by (prn x) within my-tag-fn

Thanks! And also thanks for sharing how you tested it. Keeping that!

np, discussing discard forms with you helped to motivate me to understand this stuff better, so ty for that 😉

You might see more responses if you're able to reduce your example to something smaller. Inspiration: https://www.youtube.com/watch?v=FihU5JxmnBg

I'm trying to create a new List<SomeObject>() in clojure, but don't know how to. When I do (List. SomeObject), I get No matching ctor found for interface java.util.List

List is a Java interface. So to create a list object, use (java.util.ArrayList.) or (java.util.LinkedList.). This will be a list of Object. You don't need to (and can't) specify the type of object.

List is an interface. Objects cannot be created. Need to use an ArrayList or something?

java.util.List is an Interface, and can't be instantiated directly. You could use: (ArrayList.) or (ArrayList. some-vec)

In addition to the above comments, don't think in Java it is possible to pass a "type" to a List constructor like that.

Assuming SomeObject is a type.

See here: https://stackoverflow.com/questions/3688730/how-to-pass-a-typed-collection-from-clojure-to-java

pre-`var` java lets you do:

List<SomeObject> lst = new ArrayList<>();

which might cause this confusion

In JVM byte code, List<SomeObject> and List<Object> are identical, so Clojure does not bother providing a way to make you believe you are doing List<SomeObject>, since the produced byte code wouldn't be doing that anyway.

Java collections are not typed, only the compiler does some checks for you.

And Java collections don't support primitives for the same reason. C++ collections are truly typed.

So, @U010Z4Y1J4Q, when writing any Java code which would accept this list (if that's what you're doing), your method sig might look something like processClojureList(List<Object> lst)

And inside the method, you'd cast it back to List<SomeObject>

When I do the following: (defn find-significant-contours [edge-img size] (def contours '()) (def hierarchy (Mat. size cv/CV_8UC1)) (def edge-img-8u (Mat. size cv/CV_8UC1)) (cv/find-contours edge-img-8u contours hierarchy cv/RETR_TREE cv/CHAIN_APPROX_SIMPLE) )

I get the error (UnsupportedOperationException) at org.opencv.utils.Converters/Mat_to_vector_vector_Point (Converters.java:542).

find-contours looks like:

public static void findContours​(Mat image, java.util.List<MatOfPoint> contours, Mat hierarchy, int mode, int method)

can't I simply use '() for the java.util.List<MatOfPoint>?

Like instead of passing in List<MatOfPoint>, I think passing in '() is causing this error.

So, the implementation is here: https://www.google.com/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentList.java

just use (java.util.ArrayList.)

Looks like '() does implement Java's List.

generics are a compile-time trick, in reality List and List<MatOfPoint> are the same after java compile is done with it

the reason why you’re getting an exception with '() is becausse the clojuree list doesn’t allow mutation operations like .add()

so you need a mutable list like ArrayList

Yeah, Clojure lists only implement the read-only parts.

If you’re already using Postgres, you may find this interesting: “ZomboDB brings powerful text-search and analytics features to Postgres by using Elasticsearch as an index type.” https://github.com/zombodb/zombodb

We are migrating away from elastisch to spandex - mostly because a) the DSL masks a lot of the actual ES queries and index setup b) spandex is built on top of the official Java client. The only downside is that it relies on core.async in few places

spandex but I am biased 🙂

@U0JEFEZH6 what is the downside of the core.async?

We don't need it :-)

For the last 4 years of running Clojure on the backend I still haven't found use for it

Interesting. I came from the front end so was new to needing threads etc and used PurelyFunctional TV courses straight to core.async lol

I can see that - core.async definitely helps somewhat with the callback hell in cljs. I'm definitely biased: the application I'm working on is not deployed as a single unit, so there's a lot of communication between separate processes, rather than internal queueing

7 years of clj + Elasticsearch experience here: For the love of all that is good and holy just use the REST API.

The official client is just that

It just adds pooling/discovery, the rest is essentially ring from the end users perspective

@U07S8JGF7 yep, that's where spandex shines, you have to use the REST stuff (also ~7 years of ES "experience")

Yeah, not enough bang for the buck IMO. clj-http is general-purpose and works fine.

caveat emptor

Always some use case where round-robin connections become important. But you can even do persistent connections pretty trivially in clj-http.

FWIW both clj-http and official ES rest client are built on top of the same apache httpcomponents libs

yep, so… why the middleman?

I take it back. There’s obv a value-add. Just minimal for the gain in my view.

As with all client wrappers - same reason why we abandoned all of them and just use clj-http for all 3rd party service interactions

Amazon being the only outlier because it's amazon

clj-http value-add is significant: general-purpose, data-oriented http access

ES client basically strips off “general purpose” and exchanges “round robin requests”

Amazon is complicated primarily by security, I think.

Arguably clj-http has also a lot of cruft you don't need to deal with when using es :). the official client is configured to work well and be very performant out of the box, saves you some fiddling, writting all the crust around clustering & co. But I get the argument for familiarity

@U4EFBUCUE VisualVM is a free alternative to YourKit. I also often find JDK tools to be really useful before/without (ever) reaching out for a profiler. Especially jcmd. Possibly combined with Java Mission Control (JMC) which got a new stable version recently. That's said, 256M isn't that much. It depends on what "a very simple app" means and how do you configure java/jvm in terms of -Xmx et al.

Thanks!

symbols with dots are treated as class names in Clojure so may be a special case here

makes sense, thanks!

Careful of leading zeros!

At World Singles Networks, we have a monorepo with about thirty subprojects. We use CLI/`deps.edn`. That's about 96,000 lines of Clojure (of which 21K is tests).

We build about a dozen separate uberjars from that for deployment to production.

Big advantage: easy to have "all" code on your classpath and loaded into your dev environment. Also: nice to have a single git SHA across all your "projects" from a release management/trackability p.o.v. We bake the SHA into every artifact so even if we build uberjars independently, we can still see exactly what point each one was built from.

Technically we have two monorepos: one for all the backend stuff (Clojure) and a separate one for all the frontend stuff (JS).

At Google size if you are a SRE facing an outage in some random service, a monorepo lets you instantly see the source code for all of google without needing to sort through thousands of project orgs and understand the transitive dependencies

Yeah, it does seem to be something some of the really large companies favor -- but that comes with its own interesting challenges since they have so much code that it won't fit on a developer's machiner all at once.

Is that true that it doesn’t fit on a dev box? A TB of text is a lot of code

Google have talked about that -- they have custom file system abstractions and all sorts of custom search tooling.

I think it must be this thread

Who else has a monorepo apart Google of the big ones?

This is quite interesting about what Netflix is having to do in a distributed model to get the benefits of a monorepo https://netflixtechblog.com/towards-true-continuous-integration-distributed-repositories-and-dependencies-2a2e3108c051

I'm seeing several articles stating that Facebook, Twitter, and Microsoft also work with monorepos.

Hum, didn't know that

So, something I don't understand when reading this article, which is bringing up good issues I think, but maybe its that I don't really get what a monorepo entails. Like, on every code change, you would still recompile and re-run the tests of the entire company's code base?

Otherwise I don't understand why it would give you "Publisher Feedback"

Like, do you basically threat the code as one big code base? Like its not just a mono-repo, but a monolithic app, except it has different "main" entry points?

So everything is always on the class-path, and there's just no dependencies, its all just in the source folder?

And running tests and compiling is thus always running the entire tests of everything together and compiling everything together?

No, of course not. That would be silly.

But I don't really have the patience to explain it. There's plenty of information out there about it.

Oh, I was actually thinking that could be nice 😅

Ya, no worries. I'll one day go and learn much more about it

Thanks for that link, it's too bad it didn't really talk how they implemented those solutions though

This, probably! https://clubhouse.io/blog/monolith-meet-mono-repo/

exactly!

What is accelerate?

I'm assuming each "project" has its own source directories, resource directories, build.boot, etc?

Oh, I'm seeing now your responses in the other thread.

We switched from Boot to CLI back in 2018.

Got it. So each subproject has its own deps.edn?

Or a top-level deps.edn with different aliases for each subproject?

We have clojure/versions/deps.edn which is our "master" deps file and we use CLJ_CONFIG=../versions to make that "replace" the user-level deps when we are in each subproject (which, yes, have their own deps.edn as well).

We have a :defaults alias in the master deps with :override-deps for all the libraries that we "pin" to specific versions across the whole (mono)repo. All tooling aliases are also in that master deps file.

So a per-subproject command is cd subproject; CLJ_CONFIG=../versions clojure -A:defaults ...

(and we have a short build shell script that wraps that so we can just say something like build test api and it adds :test to the above command)

How are dependencies between subprojects handled? Do you use some kind of internal artifact repository and declare dependencies in each deps.edn, or do you just assemble one giant classpath and let the :require s sort it out?

(I hope that question made sense.)

Mostly we use one giant classpath. Part of the benefit of a monorepo for us is not having to use artifacts for internal dependencies and therefore not having to manage that whole tangle of version updates etc.

Every now and then we spin off parts of the repo as public OSS projects, once they're stable enough and we feel they might benefit the community, but we also move those out of the monorepo and treat them entirely standalone at that point.

How do you handle situations where submodule A depends on submodule B and you have to make some breaking changes to submodule B on a short timeframe? It's not clear to me how to do this in a monorepo of the kind you're describing unless you have some kind of artifact repository.

I feel like I'm missing something. 🙂

Clojure's ethos is to not make breaking changes so there's that.

But if you absolutely have to make breaking changes to B you should also update A. Our position is that master should always be releasable.

On rare occasions we do branch builds of isolated services -- but all tests across the whole repo should still pass even on a branch, even if we're only releasing one "product".

In reality, it's just not a problem -- we just don't break APIs very often and we never find ourselves in a situation where we're rushing to release one subproject so badly that we would leave anything broken.

But, again, our basic position is: master is always releasable and we can go from commit/push to production in just a few minutes if we have to -- so we can release multiple times a day fairly easily.

That's really helpful, Sean. Thanks for explaining.

I've been doing purely Clojure for so long I've pretty much forgotten how "bad" it can be in other languages 🙂

The dream 🙂 I will say, not being in a monorepo actually does make using Clojure easier in this regard

I do (vaguely) remember working in environments where builds used to take long enough to go have lunch instead of just a coffee...

Yeah, this custom tooling is the difference between those two cases for me

It lets me do a rebuild for our main project in < 5 minutes on my laptop if I really dial it in

We have a bunch of microservices and a few libraries shared between them (all Clojure) and managing the dependency graph between the microservices and the libs is a bit of a pain in the arse - hence the pondering of monorepos.

I think the main con is going to be set up. a lot of tools assume project repos so you'll have to figure out how to make them work the "non-standard" way

Most of the downsides of monorepos come at scale, when there's too much code to fit on a developer laptop, or it's too slow for your editor to search large parts of the codebase. Google and a few others talk about that -- and all the custom tooling they've build to address it.

for instance, turning on travis CI for a repo? easy peasy. turning it on for a folder/subproject? I'm sure you can, but I'd have to look it up

Yeah @U050MP39D I had wondered that myself. We're using Circle CI but it's the same sort of challenge. As you say though, there will be ways around it but one then finds oneself fighting the tooling somewhat. IDE-wise, I'm pretty confident that Cursive will deal with it fine but we have engineers using VSCode/Calva too - I'm not so sure how that will cope.

Thanks @seancorfield, yes the problems at scale certainly make sense. Not something that we're going to struggle with I think though thankfully.

Unless you're at scale, you probably don't care about running CI only for a single folder. Assuming your tests don't take a significant period of time to run, that is.

ehhhh. it's more about one project not blocking another

With failing tests?

yeah

We have a monorepo that is set up with CircleCI. We have 51 sub-projects within the repo. To speed up CI test runs, each sub-project runs its tests in a separate job. With CircleCI's new pricing, you can have an "unlimited" number of jobs running at once. This makes it so failing tests won't block, unless I'm misunderstanding @U050MP39D.

right. I know it's possible. I was just saying you'll have to do more when you're setting things up

Ah, true. We have to generate our CircleCI config programmatically at this point.

I know people have told me before, but I still don't see the problems with a multi-repo setups

I've had people say refactoring across projects, but in Clojure, grep is my refactor tool so 😛

And I also always point out that Google did not adopt a Monorepo by choice

They migrated from a Perforce monorepo to a Git monorepo, because it was the easier path to migration

And then had to build a bunch more tooling to address the challenges of a Git monorepo

But I've never tried a monorepo, so I'll reserve criticism to when I do

I guess I'm more interested in hearing about why people even get curious about monorepo? What current issue you face with a multi-repo that make you look for an alternative?

You're on your phone, aren't you? 🙂

😎

"What current issue you face with a multi-repo that make you look for an alternative?" -- pretty sure another thread here had a discussion of that.

I know I talked about some of the downsides of multiple repos when folks were discussing this yesterday...?

For me, the key downside that I'm finding to having multiple repos is that we have several services as projects all of which use the same set of internal shared libraries - when one of those libraries adopts changes that need to filter through to all services, there then follows a laborious job of updating, testing and deploying all of the services separately. Or, in cases where a service doesn't need to be updated immediately with the change to the library, there's no easy way of ensuring that the library hasn't introduced breaking changes to those services. I suppose that one could concoct a CI job to 1) detect the library change; 2) automatically pull down all service source code 3) bump the library version in each service temporarily and 4) run the automated tests. (And I do get the whole point about just not making breaking changes to libraries but in this case these are libraries internal to the (small) team and so the overhead in ensuring backwards compatibility is just too much to consider.)

I kind of think even shared internal libraries should have a high standard. The amount of wasted effort from "migrations" greatly make up for it in my opinion.

But I also don't understand why changes to this library must filter through? Can't services just choose to upgrade whenever they care for the new functionality?

And what I don't yet understand is how a monorepo avoids this problem.

Also, that CI script you describe is what we have. In our CI pipeline for each service, the service can list out certain dependencies for which they want to automatically upgrade too and deploy on new minor version bumps. Major version bumps do not trigger this, since we use those to indicate breaking changes.

But for major versions, we'll get a warning that we are now depending on an old version

This was kind of all setup already for me, so I have no idea how difficult it is to create such CI infrastructure :rolling_on_the_floor_laughing:, so maybe I'm looking at it through rose colored glasses

a monorepo you don't have versions of the shared library, it's part of the same repo. it also removes the friction of having to set up library publishing and internal maven etc

I don't get that? How do you bootsrap the classpath?

Do you just add the source folders of your dependencies to your sources?

Or use deps local repo support?

Was that + meaning a yes to that? If so, that's definitely interesting. I have to ponder on that.

I use the deps local repo stuff fwiw

Only on lein do I see the source paths thing, and it is painful with dependencies

Reading about it more and more, both of these are kind of interesting take. So I guess that's the part I was missing to understand. It seems directed graph package manager seem to work better for monorepo, like Google's Blaze and Facebook's Buck

I just wanted to say thanks to everyone above who got involved in this conversation - it's been enlightening!

use ` and then inside do ~'?e

which means "unquote this thing which is a quoted ?e"

cool, thanks!

As an alternative, you can also do: ['?e attr "7"].

lol, indeed

What’s your interest in GraphQL? Simplifying getting data to/from single-page apps, GraphQL itself, or something else?

Lacinia is in active use and development. Alumbra seems stale. Graphql is still changing so if you want to keep up with the ecosystem that matters.

Also check #graphql

nice thank you @U7PBP4UVA