Can't you just call compile yourself with independent namespaces in parallel? Without having to patch anything.

Namespaces are changing, new connections are added, this cannot be done statically. And I never bothered to write an automatic analyzer/topological sorter for it, we didn't gain too much from parallel compilation.

Oh, I didn't mean in an automatic way. More like when you know that proj.a and proj.b are independent, you can compile the parts they both depend on, and then compile a and b concurrently.

I did some experiments with this as well, basically end result was "not worth the effort"

since I don't do any dynamic requires and such I just basically re-used the logic from shadow-cljs for parallel CLJS compilation. basically the problem is that there are very few namespaces that take a long time to compile. and very many that basically take no time at all. problem is that those slow ones almost everything else depends on, so you have to wait for those (e.g. clojure.core, core.async)

But specifically those two are already compiled, no?

not in the setup I tested, which involved a patched clojure

I think clj-kondo already tracks which namespaces required from what namespace. even if they are "required" implicitly via call to require or var usage. Not sure if it can generate topology right now but doesn't seems like a big problem to me. @U04V15CAJ does it worth to opening an issue?

an issue about what?

generation of topological analysis for required namespaces. It could be used as a base for other tools (eg. for parallel compilation)

this already (implicitly) exists, at least the data for it. https://github.com/clj-kondo/clj-kondo/tree/master/analysis there are several tools that visualize dependency graphs between namespaces and vars based on that data. • https://github.com/SevereOverfl0w/vizns - visualize the relationships between namespaces and dependencies • https://github.com/benedekfazekas/morpheus - visualize dependency graphs of variables

So, who wants a JFR recording of lots of code being compiled? Where do you think most work is spent? taking bets

What are our choices?

For some reason JFR didn't capture any cpu stack frams, so let's say, which class dominates allocations?

clojure.lang.PersistentList? :)

nope, byte arrays! (for strings)

seems like I do have some threads profiling, where analyzeSeq dominates (seems legit)

top allocations

StaticMethodExpr initialization takes quite a bit of CPU, too

I wonder if introducing a couple of caches could speed things up

I've used a parallel tools.namespace fork for 2.5 years now using ns tree analysis, as hinted in the thread Normally it has at least some benefit, although it's not spectacular It would shine the most in a large and modular codebase cc/ @U06BE1L6T

In Component I think the most common tack to take from what I've seen/heard is to make a wrapper that implements a protocol so that it can be mocked in tests.

probably both. the big issue is transaction handling if you abstract away the db connection

In integrant I can't talk about trends really, but I know in my own code I usually make the actual dependencies be functions. So if I have a DB connection then I have an ::ig/init-key for the db connection, but it's not depended on by things that use the DB. Instead I make an init-key for some specific query that I want to make, and that depends on the DB, and then code using the query depends on the function. This way in tests I can just mock the query function and don't have to worry about mocking a DB.

sometimes you want to do multiple things in a transaction, so that gets tricky if the db connection handling is abstracted away behind a (update-user! ...) or whatever protocol

Ah right, that is a good point

That is an interesting approach, thanks @U5NCUG8NR

We're perhaps an edge case but we have a Component that sets up five connection pools because we interact with multiple databases and we have multiple users across the connection pools -- including select-only on specific subsets of those databases (this is MySQL, it would be like schemas in other DBs I believe).

The code then pulls the specific "db-spec" from the Component and passes it down into functions that need a DB connection/datasource.

It makes it very easy to mock in tests and it also makes quite a clear boundary for interactions with the DB and turning them into self-contained functions. Usually the way I structure this in practice is to have a DB ns which has a dynamic variable for the DB connection, and a bunch of functions which operate on it, then the actual init-key for the queries just wraps them in a fn that does a binding of the db connection. This can compose well with transactions too since in e.g. jdbc you can bind a transaction to that dynvar and have it work as expected in composite query functions.

(we can't use a dynamic var because we have code operating on multiple connections across multiple DBs)

if the functions know which db they operate on (not a guarantee), then you could just store a map from db identifier to connection object in the dynvar. otherwise doing it as you say sounds pretty good to me

I found that having the component to wrap a connection pool and then a set of app.model.* namespaces worked way better than creating a protocol with methods like get-user, update-user and so on. For one, in your tests you technically can provide a different DB connection (H2 instead of Postgres or whatever) rather than implementing the DB protocol and all of its methods. And similarly to Sean's case - my apps always ship with two connection pools (read-only for the replica and read-write for main instance)

Our apps are so DB-heavy that the whole mock-with-protocols thing would be completely impractical for us. We have a lot of complex SQL, often dynamically built (via HoneySQL), with sufficient MySQL-specific stuff that even switching to a different DB in testing would be almost impossible

For basic stuff, yeah, that probably works well but no one's ever been able to convince me it scales in complexity -- short of actually building an ORM which... well, that way lies madness 🙂

Hmm, yea that will probably be a problem for my app as well. Had never tried it before, but was considering it.

#2 from your question is completely fine: if you have a map, you can supply it with meta-functions that start and stop it, and pass the component directly to JDBC functions:

(defn init [options]
  (->
    {:options options}
    (with-meta
     {'com.stuartsierra.component/start
      start
      'com.stuartsierra.component/stop
      stop})
    (component/using [:compA :compB])))

it really depends on the context, and sometimes adding meta to a map is just simpler rather run making a dedicated deftype/defrecord with a bunch of methods from a protocol

mock-with-protocols scales up to about presentation-slide level code 🙂

It kinda works, but not in this scenario. I swap API clients in tests for example or have a function component that returns current time etc

As a feedback on the question itself - I feel like you won't get any answers unless someone that already knows about Cloxp or GT manages to pass by and notices it. After skimming the article and the repo for a couple of minutes, I still can't clearly see what it's about.

just went over some of the cloxp videos and looks like it has quite some overlap with #flow-storm. As @U2FRKM4TW said is hard to figure out all the things the project was aiming for from the articles. I leave you a video here https://www.youtube.com/watch?v=Mmr1nO6uMzc of a recent FlowStorm demo, maybe you find what you were looking for there

Ah ok thank you for the feedback. I will update the description.

ok, now I see, FlowStorm will only help with the algorithms and data inspection part of it

and there is also https://www.datarabbit.com/

Tudors work has been an inspiration for Clerk https://px23.clerk.vision

Thank you all for sharing. I will look into clerk again. Summer vacation is coming now soon. I think I will try to look into it after that. I already have a Zettelkasten in Java and also some building blocks in clojure. I would very much appreciate a kind of interactive - online - discussion about clerk. Do you have community calls or something other related? Where people can ask questions? I mean I think I could type everything down here as text messages but I also think that one - 1 hour online session would be enough to show what I already have and then get some feedback on how to best bring that to clerk. Thank you 🙏 again.