Hi all, I am new to clojure but really like both datomic and fulcro and I am looking to use them in an upcoming project. I imagine I will be making heavy use of transaction functions as a way of performing atomic writes. However, I am worried about the statement in the docs (https://docs.datomic.com/on-prem/database-functions.html): >The transaction processor will lookup the function in its :db/fn attribute, and then invoke it, passing the value of the db (currently, as of the beginning of the transaction) Say I have 2 TxFunctions (f1 and f2) which sometimes I may perform individually but other times want to perform them together in a single transaction. I am not sure how to deal with a case where the result of f2 depends on the value of the database which may have been modified by f1. Have other people had problems with this and if so how have they dealt with them?

@nzjoel I know for a fact that there are other people in here with much more knowledge and experience than me, but I would suggest looking at the :db/cas special value for a transacted value as a starting point

It causes transactions to throw a specific exception type if the so-annotated value in the db into which you propose to transact differs from the one in the db you’re working from, essentially kicking it back to you to decide how to handle that case (exponential backoff? propagate exception out to caller? something else?)

@chris_johnson Yeah I thought about handling it like that, i guess that would take load of the TXer but seems like a "try again and hope it works this time" situation

well, sort of

:db/cas doesn’t make any decisions about what to do because it’s different for every case

sometimes you just need to ensure that nothing else is changing in flight, so you can afford to retry n times with exponential backoff, so when there’s contention all contending txns slow down until each one in turn gets to “win”

in another situation, though, something changing underneath your txn might invalidate it so thoroughly that you have to go aaaaaaall the way back up to a human to find out what to do

etc. etc.

I guess maybe the TXer could be wrapped in a higher level TXer which queues writes and can minimize contention in the actual TXer...?

just seems like it would be a common problem and have not been able to find much info about it

I wouldn’t think that’s the “right path” - I don’t know the specifics of your case but if coordination between f1 and f2 is critical enough that they have to see the same db value, at least some of the time, I would probably solve that with something like (defn f3 [db & args] (apply (comp f1 f2) db args))

not sure if both apply and comp are needed in there, but you hopefully see what I mean

compose them when they have to see the same db rather than trying to coordinate them

Thinking that getting a post on Datomic Ions to AWS Blog would be neat for visibility (ship material to https://aws.amazon.com/blogs/aws/author/jbarr/ maybe) 🙂

Hi, Is aws lambdas + clojure a good fit for http api & serving html given the startup time costs ? (looking at datomic ions tutorial)

I guess cold-start cost depends on the runtime used by the proxying Ions Lambdas

Was reading on [Datomic architecture](https://docs.datomic.com/cloud/whatis/architecture.html) and saw query groups (transactional vs. analytic vs. developer queries). Does this mean that one might use Datomic for analytic workload also? (say long running queries that possibly scan the whole dataset) Was thinking machine learning use, where a model might create predictions for whole set of customers, replacing old predictions entirely (this implies batch ingestion of data as well as queries that scan a large set of data).

I'm migrating my app which used a postgres database, and I wonder what the idiomatic way of structuring my session and user data would be in Datomic. Should I: 1) Have a session "object" (`:session/token`, :session/validUntil, :session/user-ref) which has a reference to a user. (what I have today in postgres). 2) Store :session/token as part of :user/* with db.cardinality set to many and check the insertion time if the token is valid or not.

Also, should I in general just stop having created_at and updated_at attributes for my entities, since I can look up the transaction time when I need something like that?

@claudiu There was a great talk at Clojure/West that explains Lambda overhead https://www.youtube.com/watch?v=GINI0T8FPD4. Recommended for anybody using AWS Lambda, whether via Datomic or not

@claudiu but to answer your question: as always it depends on you specific requirements 🙂

that said, there is more than one way to put code behind API Gateway. We shipped AWS Lambda proxies yesterday. If we were to ship something else with different price/performance tradeoffs, choosing it would be transparent to your application code.

Datomic ion can be used with peer API? There is HTTP/IO cost when I run a query/make a pull?

@souenzzo no, ions use the client API. With no wire overhead

Why use as a tx function a function that just inserts data? https://docs.datomic.com/cloud/transactions/transaction-functions.html#testing

@lockdown- :db/add might be slightly confusing if you’re used to other databases. It can result in a “modification” or “addition” to the new value of the db. A transaction function can also return :db/retract to “remove” data

hopefully that makes sense, I may have misunderstood your question, though 🙂

yeah, I'm referring more to classpath transaction functions

why use one instead of a function that just does (d/transact conn {:tx-data some-data}) for just inserting data that is

You want a txn fn if you require that the data you’re asserting absolutely has a stable basis

@donaldball don't understand what that means

what's is data that has absolutely stable basis?

One use case might be incrementing the number of items in a user’s shopping cart

sure, you are doing some computation there correct?

like check/using the current number of items

but this create-item function https://docs.datomic.com/cloud/transactions/transaction-functions.html#testing is just simply inserting new data

So, I get "transactor is unavailable" on a regular basis, and I see on the transactor that it's stopped because of heartbeat. I've upped the timeout, but it still happens every week or two. The client and transactor are both running in AWS, in us-west-2, and it

's using Dynamo for storage.

This can't be normal, is it?

The transaction load is tiny.

Like maybe a dozen transactions a week.

@eraserhd that is not normal at all. Are you running HA?

No.

@lockdown- that tx fn is a constructor, and could be performing validations

@eraserhd instance size?

We just bumped it from t2.small to t2.medium and gave it more memory

It might be happening less frequently? Not sure, but it's definitely happened a few times since.

@stuarthalloway got it, wasn't clear for me since the example wasn't doing any validation

@lockdown- that example could be more interesting 🙂

true 😉

@eraserhd the only common causes I have ever seen for that are underprovisioned DDB or out-of-memory. You could look for evidence of OOM in the logs. You probably already have this link but for the record: https://docs.datomic.com/on-prem/deployment.html#troubleshooting

@eraserhd you have the peer lib and transactor in the same instance?

The peer and transactor are separate instances. The peer is a bigger instance, -Xmx6G for the JVM.

There is a kind of large transaction function involved. It's updated once per deploy, and used for almost every transact.

It seems like we lost the heartbeat timeout config. Hrmm.

@eraserhd is it possible that your peer is unresponsive to the transactor because too high cpu load? We have experienced this to be a common cause

In those cases we were mislead by the "transactor unavailable" log

@eraserhd we experienced intermittent “transactor unavailable” under low load while running ours on a t2.medium. tweaking memory settings and upping the heartbeat timeout cleared them up eventually. when in non-HA mode i don’t understand the benefit of the heartbeat failover even being enabled, really

@eraserhd we build a docker image for the transactor and run it via ECS — i can send you our properties file and Dockerfile if you’re interested

@spieden that would be great! The organization is moving to Kubernetes, so the Dockerfile would also be valuable.

@eraserhd here you go: https://gist.github.com/spieden/7a5b303e6ce03bbbce765e797f236a73

we run that task on a cluster with a single t2.medium in it

(i assume you’ve got all the roles correct so i left our their defs)