First-hand experience with Regex literals not round-tripping

IME to use fressian well you need to know exactly what types you expect to serialize and set your read and write handlers up carefully

out of the box even the clj wrapper will do some surprising roundtrips

nippy is definitely an easier out of the box experience

(with clojure data)

Thanks for the comments. nippy has open GH issues related to large memory usage; transit+msgpack has open GH issues related to encoding byte[] as base64 strings. Which makes me wonder if I shouldn't just use transit+json. Maybe someone else will pitch in with thoughts; otherwise I'm left with just running some synthetic benchmarks ;]

you really shouldn’t store any blobs in datomic that are large enough to cause nippy memory issues

If you’re talking about storing a single 10mb value, that is definitely too big for datomic

#error {
 :cause Unexpected end of ZLIB input stream
 :data {:datomic.client-spi/request-id 97ac9bd9-b6cf-4d2b-87fc-d7a1289005d7, :cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message Unexpected end of ZLIB input stream, :dbs [{:database-id datomic:, :t 322871158, :next-t 322871160, :history false}]}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message Unexpected end of ZLIB input stream
   :data {:datomic.client-spi/request-id 97ac9bd9-b6cf-4d2b-87fc-d7a1289005d7, :cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message Unexpected end of ZLIB input stream, :dbs [{:database-id datomic:, :history false}]}
   :at [datomic.client.api.async$ares invokeStatic async.clj 58]}]
 :trace
 [[datomic.client.api.async$ares invokeStatic async.clj 58]
  [datomic.client.api.async$ares invoke async.clj 54]
  [datomic.client.api.sync$unchunk invokeStatic sync.clj 47]
  [datomic.client.api.sync$unchunk invoke sync.clj 45]
  [datomic.client.api.sync$eval12765$fn__12786 invoke sync.clj 101]
  [datomic.client.api.impl$fn__2619$G__2614__2626 invoke impl.clj 33]
  [datomic.client.api$q invokeStatic api.clj 351]

(I got a similar exception when using a laptop peer with a janky valcache setup (process-shared valcache, running inside a file mounted as a block device; it would transiently read a valcache file as 0 bytes, rerunning the query would always fix), but this peer server has a Proper valcache setup in the cloud. Related or red herring?)

you certainly could. you need to keep in mind that your event source will need to have logic for retrying transactions in case of failure

which is something that queues and/or things like kafka can often help with

I have used datomic on-prem’s tx-report-queue to provide an event source for other systems (i.e. they take some action reacting to a transaction). But this is definitely cutting some corners. datom updates are very granular, so you need some tx discipline (e.g. tx metadata describing the “semantic” meaning of the tx) to know what happened and decide what should be done

You can use datomic to have the ease-of-use of a normal db system where the source of truth is the graph of data at rest (vs an event-source system where the events are primary and you need to devise a projection), while using the tx-queue for some of the niceties of an event-source model for other projections or for creating commands

but if you have actual commands (i.e., perform this side effect against an external system) you are much better off using a proper command event in a proper queue that provides retries, dead-letter, etc for you. You can still have something in the middle to read the tx queue and create commands on another queue

signs you are doing it wrong: transactions whose purpose is purely to trigger an email or http request; job entities which stores the progress of a job and also drive its execution via db writes; you need to replay portions of the tx queue to get side-effects to retrigger

@U09R86PA4: oh -- so, instead of doing it wrong, I should use kafka?

put it this way: you will likely burn yourself if you try to make datomic’s tx queue drive command execution

those are signs you might be trying to do that

in those cases I think you should use a real queue and datomic is not a good replacement

but if you just want some event-sourcing niceties via the tx-queue without the event-sourcing complexity of using a queue, well defined events, defining an indexed projection, etc, you can replace e.g. kafka for your event source in some circumstances

the event vs command distinction was not something I really grasped until I did it wrong a few times (using datomic’s tx-queue it so happens)

hmm ... well, I'm pretty new to all of it, and was hoping to simplify my infrastructure as I explore the paradigm (opposed to synchronous systems and shared state through distributed RESTful services)

any advice/tutorials/resources is greatly appreciated

It is available for on-prem:

(d/q '[:find ?e :keys :e :where [(ground 1) ?e]] )
=> [{:e 1}]

absolutely available in on-prem

in clients and peers https://docs.datomic.com/on-prem/query.html#return-maps

I get 'Argument :keys in :find is not a variable' with something like this

(d/q '[:find ?c :keys :c :where [?c :customer/id "id"]])

@U05120CBV @U09R86PA4 Thanks for taking a look 😄 Glad that it is not a cloud only thing!

Your datomic client/peer version is not new enough

😮 That could be it

com.datomic/client-pro {:mvn/version "0.8.28"}

Hah, latest is 0.9.37