datomic 2018-04-23 | Slack Archive

Processing the Log to rewrite a database, i'm seeing some weird datoms in the log, of the form

[[17592186523036 :myapp/some-attribute :db.sys/partiallyIndexed 13194140011931 true]
 [17592186485599 :myapp/some-attribute :db.sys/partiallyIndexed 13194140011931 false]]

Obviously these have not been added by me - what do they mean, and how should I process them?

val_waeselynck09:04:53

Nevermind, I got this wrong, this were actually long-typed attributes that I mistakenly processed as if they were ref-typed.

hmaurer11:04:40

@val_waeselynck can you elaborate on this very quick? I am curious (in case I encounter a similar “issue” myself)

val_waeselynck14:04:44

@U5ZAJ15P0 I've set some attributes to have :db/fulltext true, which is irreversible and prevents excision. So now I'm rewriting my entire database by processing the Log, reading the set of datoms added by each tx and transforming it into a transaction request (which is not trivial). My problem here was caused by a bug in my processing: I was interpreting the value of a datom as it the attribute was ref-typed, when in fact it was long-typed.

prozz12:04:10

hi, some recent articles from reddit got me interested in datomic. i got question (probably very basic): where indexes are kept? is it client side? id suppose so as queries are executed client side from what i reckon. if thats the case, do you have any stats how much time does it take for datomic to warm-up and build indexes up? if those are kept in memory are there any limits in database size for this to work optimally/at all? im curious in this technology, but at the same time i feel like im missing some important concepts.

val_waeselynck14:04:19

They are primarily kept in storage, and redundantly propagated to many places, including in an in-memory cache on Peers which is where the querying happens.

val_waeselynck14:04:56

The time to warm up will largely depend on querying patterns

prozz20:04:47

thanks

jaret12:04:00

@favila @kingcode the peer api getting started is now here, under the on-prem docs. https://docs.datomic.com/on-prem/peer-getting-started.html

🙏 4

eoliphant13:04:05

Hi, I’m playing around with microservice architecture with sort of ‘lite’ event sourcing, where datomic does it’s thing internally and onyx or something dumps events in to kafka, etc based on transactions i’ve reified with the approprate meta data. In terms of sequencing, since I can do 1 transactor -> n db’s (I’m not going to be dealing with large write volumes), the various (since they are per db) onyx jobs that are extracting ‘events’ should have the correct timewise ordering right?

eraserhd15:04:51

It looks like I'm seeing significantly more memory usage since upgrading the peer from 0.9.5561.50 to 0.9.5697. Does this make sense?

gerstree16:04:10

Somehow in the back of my mind I stored the fact that after excision a transactor restart is required. The docs don't mention it, but to fully excise that fact from my memory: is it correct, that excision is an online process and no transactor restart is required? (Or maybe this was true before and it changed?)

marshall16:04:26

@gerstree restart is not required; You may be thinking of restore - that requires a restart Excision will happen “online” - it will be completely excised after an indexing job completes

gerstree16:04:11

In the case of a restore that absolutely makes sense. Thanks for clearing this up!

marshall16:04:18

no problem

gerstree16:04:57

The question was GDPR related for a bit of background. That topic takes hostage of all of Europe these days unfortunately...

octahedrion17:04:39

what's the upper-limit on the number of dbs that can be queried at once ?

eraserhd19:04:09

Is Datomic likely to significantly optimize one big bag of or expressions over individual queries in a doseq?

souenzzo19:04:07

It will "return nil" or throw?