Fork me on GitHub
#datomic
<
2017-05-08
>
favila01:05:37

@lorenlarsen why can't the peer create the database?

lorenlarsen02:05:25

@favila Well the peer could create the database but my thinking was that this isn't such a good idea long term when I have multiple peers. I can use that as a workaround for now though.

dm315:05:20

I’ve read the docs a few times, but I’m still confused about Datomic objectCacheMax on the Peer. I understand the cache consists of uncompressed segments on-heap and compressed segments off-heap. If so, does objectCacheMax specify the max size for the off-heap+on-heap cache? If I set e.g. Xmx500m, objectCacheMax=250m and MaxDirectMemorySize=32m, is the Peer smart enough to not blow the direct memory limit (if it even uses direct byte buffers)?

Lambda/Sierra17:05:31

@dm3 The Datomic Peer does not currently use any off-heap memory. objectCacheMax is the size of the in-heap cache space.

Lambda/Sierra17:05:03

Compressed segments are kept in the Storage Service and in Memcached.

val_waeselynck18:05:33

@nikki I don't think that's viable, because Datomic doesn't qualify as a source control system (no forking and merging, no line/expression-level diffing) - for good reasons.

val_waeselynck18:05:14

Databases and codebases are inherentenly different (despite all the parallels drawn between them), because in databases the 'snapshots' are derived from incremental changes (transactions), whereas in codebases the incremental changes (patches) are derived from the snapshot (i.e a consistent codebase)

val_waeselynck18:05:14

The reason for that is that we reason about writing data in terms of incremental events, whereas we reason about writing code in terms of a coherent whole

val_waeselynck18:05:36

at least that's what I do 🙂

nikki18:05:34

i feel like diffing should be AST-level (CST-level) in any case

val_waeselynck18:05:07

@nikki yeah would love that especially for lisps 🙂

val_waeselynck18:05:26

But I don't know if there are diffing algorithms for tree structures

val_waeselynck18:05:31

not an expert at all

val_waeselynck18:05:49

The thing is, we do like our indentation, so a diffing/merging algorithm must preserve more structure than just the CST

nikki22:05:48

lol i'm actually thinking about an AST editor

nikki22:05:58

That visualizes the code in a touch interface or something

mac0102122:05:04

Hello everyone. I watched the “Day of Datomic” videos with the presentations by Stuart Halloway. In the section on query, he briefly discusses a query that looks something like this.

[ :find ?item ?price
  :where [?item :product/costs ?price]
               [(> 256 ?price)]]
Now, it seems clear to me that, if you have a billion products in your DB, then several gigabytes of data need to flow over the wire to the peer in order to process this query (assume a cold cache) even if the result set is only going to contain a few items. Am I wrong? If so, how is this avoided? Otherwise, what is the typical approach to solving this problem when using Datomic?

favila22:05:15

@mac01021 Mac you are correct, but this is by design.

favila22:05:08

Consider: 1) network is faster than disk, 2) only the peer running a query feels the load of the query

favila22:05:46

avoiding network traffic with your db on the fast datacenter network is rarely the scalability problem

mac0102123:05:51

#2 is a clear advantage. But with most databases that I might use, the data are indexed in such a way that I could run this query without ever inspecting the vast majority of the records (via network, disk, or any other mechanism). So the time to arrive at your result set, assuming it is sufficiently small, will be logarithmic in the number of records. (Hurray for B-Trees!) Does Datomic really provide no way to perform a range query without evaluating a predicate on every single item in the database?

favila23:05:00

range queries do not examine every item

favila23:05:08

(necessarily)

favila23:05:16

e.g. your example will array-bisect

favila23:05:40

assuming :product/costs has a value index

favila23:05:33

see also the datomic.api/index-range function

mac0102123:05:57

ok awesome. Thank you