datomic 2018-01-27 | Slack Archive

jdubie00:01:41

We are running a datomic transactor and using MySQL for storage in AWS. I’d like to be able to read from the MySQL database but enforce no writes at the network level. Can you run the peer library without a transactor? I’d prefer that this peer have no ability talk to the transactor. Is that possible? I know peers are probably designed to get streamed transactions from the transactor.

favila00:01:17

Use different database users+creds and a different jdbc connection string for peers vs transactor

favila00:01:22

e.g. we run google cloud MySql, the peers all use user "datomic_ro" which has only SELECT privileges; transactor uses "datomic_rw", which has only SELECT INSERT UPDATE DELETE (which is all it needs.)

favila00:01:02

so peers never see anything which could be used to write

favila00:01:29

you still need a transactor running though, to communicate not-yet-indexed tx-log data to peers

favila00:01:50

bah, I misunderstood what you want. No, there's no way to make d/transact impossible

favila00:01:12

you would have to run a peer in a trusted process and have it expose its own RO interface to other processes which use it

favila00:01:19

peer = trusted

jdubie16:01:17

Exactly I’d like d/transact to fail at a network level

stuarthalloway10:01:22

@jdubie in the peer world, you would implement read-only yourself, as @favila says. But in Datomic Cloud, is is straightforward to use IAM to make a client process read-only. See https://docs.datomic.com/cloud/operation/access-control.html#sec-2

val_waeselynck11:01:00

@drewverlee regarding testing, Datomic is about as good as it gets - consider that you won't have to 'mock' anything thanks to in-memory connections, which makes for an even better testing story than the traditional client-side testing of SQL databases. About storing business logic in the db, you should also consider that it may also cause operational difficulties (deployment, new versions etc.) compared to having your business logic share the ephemeral lifecycle of Peer code. I think one of the main incentives to use stored procedures in the SQL world is to bring the business logic closer to where the data lives, and sometimes to overcome the limitations of SQL - but you don't have those problems on a Datomic Peer, at least not for reads. This testing power and this expressiveness of Peers will probably make for much fewer reasons to use stored code than in the SQL world.

val_waeselynck11:01:54

@drewverlee so my advice would be: don't be too eager to put all your business logic in transactions functions, and don't be too strict about enforcing all invariants upfront. With Datomic you have a lot more testing, querying and debugging power to ensure the correctness of your system than with a SQL database.

Drew Verlee17:01:51

Thanks for the insights. I’m i correct in assuming the big tradeoff seems to be control for evolvability? As an extreme example of the control having business logic in the db offers, the article i post makes a good observation. There might be data in the database you only want a select few people to know exists at all (fraud detection). Another example of control that having business logic in the db, is invariants on the data. I think this is where the mental model (graph and immutable) that datomic presents is different then relational databases. In a relational db, your much more worried about someone over writing your “good data” with “bad data” in dataomic, thats less of a concern, because you can probably recover (because you never through the record away). So its likely to be more open. Your claiming that keep your business logic and db logic separate is allows for easier evolvability. In the form of easier deployments and new versions, etc.. This is the side of things i don’t fully understand. Simply because i have never hard to worry about this sort of thing before 🙂. But i feel like you have really helped me narrow in on the tradeoffs. Thanks

val_waeselynck10:01:51

@drewverlee it's hard for me to answer because it's not clear what "in the db" means. In the SQL world, "in the db" means "it executes over there" but in Datomic most of your db code is in your app process - because it can

Drew Verlee12:01:46

I'm trying to compare the relational + sql approach to the datomic + datalog in the context of the discussion around business logic. So in the conversation I talked about both.

val_waeselynck14:01:50

@drewverlee ah right, so this is more about "should I use Datomic" than "how do I use Datomic", correct?

cap10morgan22:01:28

On the Capacity Planning page, it says that if the MemoryIndexMB goes above a certain threshold, then "you need to plan carefully to avoid back pressure." Is anyone aware of what the details are behind that? What should I do if I see it going above that threshold?

cap10morgan22:01:00

If I have plenty of RAM, is it just a matter of increasing memory-index-max and memory-index-threshold?

Ben Kamphaus23:01:33

@cap10morgan increasing them might delay but likely won’t solve the problem (if there is one). Memory index MB is just an indication of how the transactor is keeping up with the load of indexing it has to do. Those two tunable knobs - threshold at which to start an indexing job and when to throttle additional writes may help you find a subtle sweet spot for how bursty your write loads might be, but if the memory-index-mb metric gets high and stays high for long periods of time upping those knobs won’t help much b/c the problem is essentially one of indexing throughput at that point, (not the degree to which your settings allow the transactor to endure bursts of writes). At worst it could make indexing/back-pressure more catastrophic when you hit the threshold (essentially halting the system for new writes until you finish a huge indexing job).

2018-01-27

Channels