Clojurians Log v2

Pro or cloud?

Pro

In pro there is no difference in time complexity between them.

D/transact waits for durable write of the tx data, but there’s no wait for durable write of index (indexing is asynchronous)

@favila, what is the cost in Cloud when running on the PCG? On a QG?

I don’t know how with-dbs are held in cloud. If they are in-memory on a certain node, it’s exactly the same as pro

we are starting to use query groups, which requires us to extract transaction functions, so we can deploy all of them to the primary compute, regardless of which query-group's code rely on them. it feels like polylith would be very useful in this case, so the logical code boundaries are not blurred with the deployment artifact boundaries.

I do not use polylith tool directly, but I impose polylith-like structure on my projects. It helps me to take things apart, think about individual components and share the components between artifacts. I can develop with everything in one REPL, but have strict control over dependencies in deployment. List of dependencies deployed to ions is absolutely minimal. Our app has - a lot of complex application logic in transaction functions (think pairing orders in a trading platform) - some helper functions for query xforms - backend server for websockets - some jobs which run as lambda functions What we do - develop with Datomic Local in projects/dev (basically depends on almost all components and bases) - build ions with minimal dependencies - build several artifacts (polylith projects) with different dependencies. Artifact can be websocket server, job for daily reports etc - share the components between artifacts (even with cljs frontend) - run a lot of tests both locally and in CI using Datomic Local Our project tree structure

bases/
   test-all/
   backend-services/
   frontend-1/
   frontend-2/
   ...
components
   datomic-cloud/
   datomic-local/
   transactor-fns/
      src/
      test/
      deps.edn
   ...
projects/
   dev/
      resources/datomic/ion-config.edn
      build.clj
      deps.edn
   ions/
      resources/datomic/ion-config.edn
      build.clj
      deps.edn
   server/
   lambda-job-1/
   lambda-job-2/

We have migrated our whole codebase to a monorepo with clojure project using Polylith. We now have a compute group and 4 query groups and everything is deployed using CircleCi. Except for some nuances which are mostly about config.edn in local vs. cloud it works well and I have been pretty happy about the whole thing. I have thought about creating an example repo w/ CI setup etc. but have never gotten to it due to lack of time. If there's interest maybe I can sacrifice some family time over the weekend and put an example out there. Until then I can probably share some things privately.

thank you for the details! it will take some time to digest it... i was wondering about the management of the ion-config.edns for example, since query groups need some of the allowed function, which are used by queries, but not the tx-fns. they need different ion app name too. then the ion-dev push/deploy wrappers, which provide the aws params and monitor the deployment status also need to be shared. i would also expect to run tests from the dev repl, but tests need fake implementations to be loaded, while a dev-local implementation should be present too while i might even connect to remote dbs thru :server-type :ion what i'm unclear on is how would the interface namespace names not collide?

yes, management of ion-config.edn is a bit of a pain in local. When deploying code to query groups you'll get an error if something is missing either from :xforms or :allow so the main inconvenience in local is just to have a config where everything exists. It is not a major issue for us at the moment but going forward I'd probably use an ion-config for local that is a merge from all the query and compute group configs and just write a test that this would always hold true. app-name has to be the same for a datomic system. We just build a separate artifact for each project and push it to the query group where this needs to go. babashka scripts handle the deployment, feedback etc. I'll try to push an example to github over the weekend. Things work pretty much as skardan explained :)

https://clojurians.slack.com/archives/CHY97NXE2/p1736514462597849?thread_ts=1736509545.748459&channel=CHY97NXE2&message_ts=1736514462.597849 i did this little clj-kondo + jet pipeline to make sure we don't leave out new tx-fns or xforms before we commit or deploy, but hasn't hooked it up still...

My approach has been to share the codebase between the PCG and QGs. I have two different classpaths that I use at the moment of deploying. Their only difference is the ion-config.edn file. I concluded that effort required to statically partition the code into that needed for the PCG and that needed for the QGs was not worth the effort -I would rather pay the price of having "union" deploy artifacts and the small pain of managing two different ion-config.edn files that are identical except for the :app-name key (@jaret is aware of my ask to allow that value to be provided at the command line when deploying). Given that transaction functions tend to leverage the same utilities, libraries and even domain code as the "regular" app, this seemed like a pragmatic approach.

The differentiation of the PCG and QG comes from (a) the transaction functions only run on the PCG, (b) my lambda entry points are configured in AWS to hit one or the other (via the lambda name) and (c) my sole HTTP direct is configured to hit just the PCG (but I'm tempted to add one for the QG; since the code bases are identical it would be easy and only the DNS name would differentiate).