Clojurians Log v2
datomic
Regarding the use of d/tx-report-queue:
> It is the responsibility of your application to empty the queue. When you are done monitoring the queue, remove it by calling https://docs.datomic.com/clojure/index.html#datomic.api/remove-tx-report-queue otherwise it will continue to accumulate transaction reports and consume memory.
which process' memory is consumed? the queue consuming peer, or the transactor, or both?
follow up question: are there metrics we can use to observe this queue behaviour on either side?
The tx-report-queue is only a concept on the peer. It is an unbounded queue, which can grow unbounded for each transaction the peer is informed about by the transactor. It doesn't affect the transactor, if not consumed.
The queue is some java.util.concurrent, if I remember correctly it is a ConcurrentLinkedQueue .
For metrics, the .size method on the queue object is probably all you need.
The intention by the API is that you as a developer can get a ("singleton") transaction queue per peer-client, and it is your responsibility to either consume the queue or close it. You as a developer also have to distribute the incoming transaction notifications in your app (by some pub-sub or what ever you need).
Also beware that each transaction notification has a :db-before and :db-after, which can be (partially) GC:ed by the GC-storage method if you keep the notifications long enough. https://docs.datomic.com/clojure/index.html#datomic.api/gc-storage The storage process reclaims unused blocks in the underlying datom indexes so it might not be immediatly apparent you db version pointer is not working anymore (I haven't checked the exact behavior here). The tx-report-queue differs from the transaction result in that it don't return the tempid resolutions (since it does not know who transacted what). The transaction result is guaranteed to be delivered before the queue notification of that same transaction.
Usually you should consume the transaction notifications "immediately", as they occurs. You can chose to convert them to other data structures. It is always possible to re-create a database pointer to the version that can be deduced from the transaction result (modulo excision operations etc)
thanks, Linus!
hello folks, sometimes I am getting some errors when using datomic.client.api/connect, the error is:
{"CognitectAnomaliesCategory":"CognitectAnomaliesFault","CognitectAnomaliesMessage":"Unable to unmarshall exception response with the unmarshallers provided (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: PFG3BCUTRCASB2DEIAPIST646RVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)","Error":"Unable to unmarshall exception response with the unmarshallers provided (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: PFG3BCUTRCASB2DEIAPIST646RVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)"}or is it related to the java version and the amazon sdk being used?
https://aws.amazon.com/blogs/aws/introducing-aws-lambda-managed-instances-serverless-simplicity-with-ec2-flexibility/ seems to validate a lot of the processing model taken by Datomic Ions. Any thoughts?
link?
It's different (lambda code runs on "your EC2 instances" instead of opaquely on AWS infrastructure), but it is similar in how it blends long-lived EC2 processing and lambda-as-a-gateway.