Clojurians Log v2

Do you have common code used in 1-3?

If so, I would think about some mono repo approach like polylith. If not, seperate repos may serve better.

I don't think there's code in common

If 1-3 are independent, what's the benefit of putting them in the same repo? Some scripting to glue the parts together?

They developed naturally together but now it's getting annoying 😁

shared schemas, maybe?

That's some of it, but they're defined in the core / driver code, other pieces can consume it

with separate repos, that requires some build coordination. Build and install the driver code, then build the rest. With a mono repo, you could build everything at once.

Not that it can not be done with a build script starting the repo builds in order. But that would live outside of the repos.

The "build" could be part of the results repo, bringing in the analysis and driver as dependencies, thus pinning their versions, running the driver then the analysis on the results. I don't see how I'd need an extrinsic script then

i am personally a big fan of monorepos. even if you just have 3 separate folders for each project, if they're intended to be used together, it makes sense to me they'd be in the same repo. cf ring-core vs the other ring subprojects. (of course, ring also has separate repos for middleware so it's not exactly this)

Counterpoint, with the results inside the repo all text based search of either file names or context drowns in noise 😬

@smith.adriane I am curious to hear if you have something to add, you also maintain a repository with similar structure - Dewey

I am also a big fan of mono-repos

the dewey repo doesn't contain any of the content it extracts

I would tend to have 1&2 in the repo and have the results exist externally

My experience is that the harness/driver and analysis/visualization do tend to share code and its convenient to have access to analysis/visualization while working on the harness/driver and vice versa.

I prefer that, too What solution did you end up on for Dewey's artifacts? how are they pulled in to visualization?

I just put them on github, https://github.com/phronmophobic/dewey/releases

It's free for some reason.

so you're just doing it with the gh CLI from some machine?

For the next iteration, I plan to store them somewhere in the cloud (most likely s3) so that I can reuse results from run to run. Currently, the whole dataset is created from scratch every week. However, I plan to expand the number of repos I analyze by about 10x, so reusing previous work makes a lot of sense.

I run the analysis using AWS batch. There are multiple steps. The final step is to make a https://github.com/phronmophobic/dewey/blob/aeb25ef86ef941edfe711a9e5a1376721d1e5915/src/com/phronemophobic/dewey.clj#L57.

AWS batch is nice because you can use spot instances easily. The actual job is triggered from a lambda that is scheduled via EventBridge

Automation and running on hardware which isn't my long suffering PC is the project's stretch goal

Theoretically I'd want to test different CPU models and architectures

for checking other CPU models and architectures, github CI is fairly convenient.

I still have some details to figure out

> Ideas and suggestions most welcome, including how best to store and share the results I'd do all my source code in one repo, and publish the generated output (static site, I presume) to a separate repo. That's how I publish my website to github pages. It doesn't feel like your source code part needs a structured monorepo, so it probably does not make sense to introduce monorepo + tooling, just for this one project. However if you wish to join the "monorepo or bust" gang, then this may be just the excuse to do it. Polylith, of course, or choose-your-adventure (maybe https://github.com/adityaathalye/clojure-multiproject-example, but maybe not because https://xkcd.com/1445/ and https://xkcd.com/1205/ 😅).