@danielcompton good point, thanks

then let me rephrase, how do people deal with multiple DC’s, do people use some kind of hot-standby set up, or are there other methods I missed?

I am puzzled why a 'coordinator' component that gives a facade api for the application and works with multiple datomic dbs is discouraged. It will make datomic a viable option for applications that need to scale horizontally.

It may have to compromise on throughput which may be agreeable to some applications.

It hurts to lose an argument to some other product just because they said they scale indefinitely .... Is it too much to expect a discussion on this sour point ?

Who discourages this? It doesn't come out of the box, but I don't think it's discouraged? It's definitely more complicated though

ah I meant that to be a thread

Datomic is read scale, not really write-scale. If you have a huge amount of data but low write volume, you can use one transactor with a very large storage and multiple dbs. (Although be careful what storage backend you use. A sql backend, for eg, puts everything for one transactor in one table.) If you have high write volume, you need multiple transactors, sharding, and probably have to live without cross-tx atomic commits.

Both of these are just more awkward.

Looking at ignite. It's a more complex ops architecture (designed to run in a cluster with dedicated machines), I'm not sure how read-scaling works (I think you need more cluster members?) and it doesn't have time-travel. But it's definitely "bigger", i.e. if you have the resources you can scale it to more storage and higher write volumes than is possible with datomic

when the in-memory grid databases talk about distributed transactions, they are dealing with similar concerns , right.

If that's really what you need, then that's probably a better fit

I don't think so re: distributed transactions

It depends on their model, but usually they are trying to write the same data to multiple nodes to ensure quorum and no conflics

it's an artifact of clustering

when I talk distributed transactions, I mean more XD-like

you don't want some to succeed and some to fail, but they're not part of the same storage

keeping Ignite aside for a moment, I am trying to understand whether solving this in Datomic .. would it not make datomic available for lot more use cases .

solving what? high write volume?

suppose I am ok with low write volume, but, just that, I need to handle lot of data.

If you need to handle a lot of data, make more dbs

the same strategy that e.g. elasticsearch would take

If my data is split among 3 dbs and say I have one unit of work that needs to be stored among the three.

will that be one tx over 3 dbs (so that I get the acid over 3 dbs)

Yeah, that doesn't seem to be a problem worth solving

you could do it with your own single-writer facade

no one else could write

and you could attach metadata to each tx to correlate them together

and you could have well-defined rollback behavior if one fails

I mean, that doesn't look like it's worth it to me?

And, all that 'code' is essentially domain agnostic, right. As long as we come up with a standard way to express the needed metadata anyone else can use it.

there are lots of edge cases around rollbacks

and what if the dbs got out of sync because someone bypassed the writer facade?

I mean, yeah, I guess you could solve all those problems

we use lots of dbs in the same application, but we don't split a unit of work across two dbs

we never need a guarantee that two txs fail or succeed together

and .. it is exactly the that does not look like it is worth aspect that is puzzling to me. I understand it is complex, but, it looks like the in-memory grid guys have done it... May be my understanding of what they exactly offer is poor.

they have done it by making different tradeoffs

they have clustered architectures, custom storages, mutable data

datomic has single writer, storage agnostic, immutable data

ok. the trade offs. I wanted to understand the trade offs and the right questions to ask. Your explanation is helping me. Thank you much. I appreciate all the help you provide to datomic users.

glad to help. I'm not going to claim datomic is right for every problem

For us, we don't have big-data workloads, and immutability and history and easy administration are all very important

the low-friction, low-impedance api is good too (no complex sql orms)

but it's not the only database we use. e.g. datomic is bad at fulltext search, so we pipe datomic data into elasticsearch

got it. thank you.

Here (http://docs.datomic.com/transactions.html#monitoring-transactions) it’s stated that the :tx-data key on a transaction result can be used as a db value for a query. When trying that trick with the client API, I get Exception Not supported: class datomic.client.impl.types.Datom com.cognitect.transit.impl.AbstractEmitter.marshal (AbstractEmitter.java:176) using the exact code example from the above link. Has anyone successfully pumped the transaction result back into a query using the client API?

@cch1 You can use the :db-after as a db value for query, not the :tx-data

OK. But I was assuming the point of using tx-data is that it would obviate the need for a trip to the server.

Is there a use case for :tx-data in the client API?

all queries go to the peer server

if you’re using client

OK.

doesn’t matter what the db is

sure, you may want to examine the specific datoms created for something. maybe to get the tx-inst or save off the txn id locally for something

I am trying to find the reverse links to an entity using the datoms fn via the :vaet index, I am not sure how to navigate the results, what does it return and how do I handle those results ?

It returns a seqable (i.e. you can call seq on it, or use something that does so automatically) that returns a lazy seq of datoms from the index you specified, with components matching what you specified in the args

Individual datoms have fields that can destructured by position [[e a v tx added?]] or by key {:keys [e a v tx added?]}

thanks looking into that now

typical use would be like this:

(->> (d/datoms db :vaet :db.cardinality/one)
     (take 5))
=>
(#datom[8 41 35 13194139533312 true]
 #datom[9 41 35 13194139533312 true]
 #datom[15 41 35 13194139533366 true]
 #datom[17 41 35 13194139533366 true]
 #datom[18 41 35 13194139533366 true])

yes thats what I am getting

This is all references TO the :db.cardinality/one entity

(which is id 35)

so note the "v" slot on all results is 35

How to decode the vals in the #datoms vector ?

you can get by position or key

I see the attrribue name is returned by id, do I have to look that up ?

yes. this is the raw index, so there are no names or other niceties

aah right

d/ident is your friend here

make sense

thanks !

I feel like accessing datom fields should be better documented. all I could find was this note

> The datoms implement the [datomic.Datom](http://docs.datomic.com/javadoc/datomic/Datom.html) interface. In Clojure, they act as both sequential and associative collections, and can be destructured accordingly.

on http://docs.datomic.com/log.html

right thanks

(->> (d/datoms db :vaet :db.cardinality/one)
       (take 5)
       (map (fn [{:keys [e a v tx added]}]
              [e a v tx added])))

(->> (d/datoms db :vaet :db.cardinality/one)
       (take 5)
       (map (fn [[e a v tx added]]
              [e a v tx added])))

Was just converging on that 🙂

(->> (d/datoms db :vaet :db.cardinality/one)
       (take 5)
       (map (fn [[e a v tx added]]
              [e (d/ident db a) v tx added])))

@favila thanks that works, does the datomic console synthesize the reverse keyword names like :entity/_link, just not sure where they come from ?

I don't use the console so I'm not sure

if their "datoms" feature has names, it's because it's calling ident

How do I get a list of all the attributes in the system ?

the values of the :db.install/attribute attribute on the :db.part/db entity

thanks