Hi, anybody using datomic for ETL? We have a low throughput webapp, and we’d like to start collecting events for some kind of BI. We don’t need fancy lambda architecture, just a collector (eg: kafka) and an ingestor. My only concern is that we want do to some heavy-ish aggregations and I’d like to do it straight from datomic

I also took a look at onyx but we’re low throughput so don’t need that right now

I only have 1 data point for StorageGetMsec, and I'm a little confused by that. Shouldn't it be triggered fairly regularly? Does Memcache have an effect on this metric?

question on ions: is it a good idea to handle asynchronous workloads on the datomic cluster (with core.async). the use case is: based on some update in the database, another HTTP api needs to be called. the process initiating the db update should not wait on the HTTP response of the other API. I think I can use vars or atoms to keep references to channels so the core.async machinery doesn't get garbage-collected. Are there any issues with this approach? Or is it better to hand-off this functionality to a physical queue, with e.g. lamba's processing the queue (which is quite a different level of operational complexity 🙂 )?

Hi! Here in Magnet we’ve played a bit with awesome Conformity migration library by @avey_q. However we’ve had some issues with transactions getting reapplied unexpectedly so we’ve developed a simplified, atomicity-compliant version of it. We haven’t explored Ragtime by @weavejester yet, but in case it’s useful for some of you, there it is lying. https://github.com/magnetcoop/stork

hello…im having a strange (but probably simple PEBCAC) problem with unique identities. I have a transaction that includes something like:

[{:db/id pid :person/email email}
 {:db/id mid :club/member pid}]

where :person/email has the unique attribute in the schema

insert works fine, but upsert fails, and I dont understand why

the probem seems to be in referencing the entity-id of the existing item

(e.g. if I take the second add away, the upsert itself “works”

i just dont know how to reference it in the subsequent add

the error I get is:

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: :db.error/not-a-data-function Unable to resolve data function: :db/id

any guidance appreciated

@ghaskins what’s the exact data you are transacting when you get that error? that’s usually caused by missing a seq wrapper around the tx-data

i.e. (d/transact conn {:tx-data [your-data-here]}) instead of (d/transact conn {:tx-data your-data-here})

is anyone aware of a library that enables something similar to pull expressions but for EDN data?

hmm, seems like this maybe https://github.com/juxt/pull

@marshall in this case, its happening inside a txn-function, so harder to get the exact set its using

ill see if I can tease that out

@marshall good call, that was it

ty

the txn-fcn was returning a sequence in the initial case and a map (with one :db/id it it) for the update case

@spieden This is also worth checking out https://github.com/halgari/odin

@john cool, thanks

@spieden and perhaps https://github.com/wilkerlucio/pathom, although it's probably overkill for parsing just EDN.

I'm looking at upgrading Datomic cloud and have a couple of questions: (a) I'm unsure of whether a system has been upgraded before, so I'm trying to determine whether I should follow the First Upgrade path. (b) If it is a First Upgrade, it tears down the stack and creates a new one with "Reuse existing storage on create" set to True. Both "Upgrade" and this setting imply that existing data will be preserved. Is this correct?