Some relevent links: • https://forum.datomic.com/t/kafka-consumer-as-an-ion/823 • In Rich's talk on Ions, https://youtu.be/thpzXjmYyGk?t=5123 without using Datomic, and his response is, roughly, "well you can if you want, Datomic will be there for you if you need it".

The promise of Datomic IONs is: "Developer write functions, declare in ion-config.edn which ones need to become lambdas, ion deploy and enjoy" It is a joy to work with. Past Friday I upgraded our Datomic System with one hand, glass of wine in the other. I cannot overstate how light our Ops load is in the team. We have a terraform repo that creates the stuff we need around our Datomic System (things that eventually call the lambdas). And the Datomic System we get to ignore for the most part: Write code, see code run. The resource-hungry cost is the EC2 compute use of you instances. So start with 1 x t3.small, then scale up as needed. The Storage cost will stay at 0% if you are not using Datomic. And I mean, we get this whole topology, (literally now) for free: https://docs.datomic.com/cloud/images/topology.png

And here is a sketch of kicking of running processes at-boot of your different EC2 instances. Note the non-availability of the Datomic DB in this case:

(ns org.ions.long-running)

(defn kick-off-long-running-process-as-this-namespace-loads
  "You will _not_ have access to Datomic here.

  Only if you enter through an ION can you be guaranteed to have it"
  []
  ;; Always set the setDefaultUncaughtExceptionHandler
  (Thread/setDefaultUncaughtExceptionHandler
    (reify Thread$UncaughtExceptionHandler
      (uncaughtException [_ thread ex]
        (let [msg (str "Uncaught exception on" (.getName thread))]
          (cast/alert {:msg       msg
                       :exception ex}))))))

;; Call on-load
(kick-off-long-running-process-as-this-namespace-loads)

(defn ion-entry-point
  "Can be called to control the process above. Use an atom for communication.

  Must be bound in ion-config.edn because Datomic only loads namespaces defined in there.
  
  Hard to use to kick-off long-running processes, because every deploy restarts all EC2 instances.

  A post-deploy hook that call this ION's Lambda is possible. In that case you will have access to the Datomic DB"
  [{:keys [context input]}]
  (cast/event {:msg "Long running process being changed"}))

Datomic Cloud wants you to only use the DB-on-use, from an ION call. We work around it.

Thanks for the detailed response! One thing though, by "resource hungry", I mean how much ram/cpu does datomic use on the ec2 instances themselves when not being used? My long running process uses a lot of ram/cpu and I'm wondering if this will be a problem running on the ion instances?

Hi @U064UGEUQ , could you explain a bit more about your process? Is it a long running process that eventually terminates like a batch job or an always on process like a web server?

Yes, sorry it's a long running program that is always running, not a batch job.