Fork me on GitHub
#datomic
<
2018-01-31
>
laujensen07:01:55

@stuarthalloway / @marshall. We’re about ready to remove CSS/HTML from the Database (due to missing blob support), but I assume I’ll have to nuke the history of these entities in order to reduce the memory load. How do I go about that?

robert-stuttaford08:01:51

@laujensen how old is your db? you may have a better time of it if you build a new db by replaying the transaction log and eliding the data you want removed from the transactions. @stuartsierra calls this ‘decanting’

robert-stuttaford08:01:32

the only way to actually remove data is to excise it, which has very poor performance semantics. it’s not meant for large data removals

laujensen08:01:10

Yikes. Hope I wont have to set :no-history, dump the db and re-import it, then re-add history

robert-stuttaford08:01:02

do you care about the history of this data, or only the present state, @laujensen? because you could just build a new db with the present state

robert-stuttaford08:01:18

you can always keep the original db around to answer history questions

laujensen08:01:41

I do care about the history, but the large blobs we’ve saved is killing performance, so if I have to make a choice, I choose speed

laujensen08:01:49

And yes, I can find a way to fake the data

robert-stuttaford08:01:15

i guess what i’m asking is, does your app use historical data, or is it only accessed ad-hoc manually

robert-stuttaford08:01:33

if only ad-hoc, i’d say start afresh and archive the current one 🙂

laujensen08:01:22

We use it for many things, among others you can pull up past versions of any webpage on your site

robert-stuttaford12:01:14

yeah, then decanting is your best long-term option

marshall14:01:46

@dbernal ^ for the java doc

marshall14:01:54

there’s also the clojure documentation

uwo16:01:04

@val_waeselynck sync-schema would fail on a forked connection, no?

val_waeselynck17:01:02

@U09QBCNBY I'm assuming you're talking about Datomock? Why would it fail?

uwo17:01:02

oh I was just poking in the dark really. I was testing a migration against a forked connection that added unique, which of course requires adding an index first and sync-schema’ing. I had an odd failure, but it must be something on my end

val_waeselynck17:01:40

In Datomock , sync is supported and easy to implement (since the coordination is only local). From what I understand, sync-schema does less work than sync, and therefore is supported as well.

val_waeselynck17:01:59

However, the burden of not forking too early is on you;

uwo17:01:27

too early would be when?

val_waeselynck17:01:28

so maybe you'll want to sync-schema then datomock.api/fork.

uwo17:01:50

wouldn’t that defeat the point of testing the migration?

val_waeselynck17:01:44

@U09QBCNBY I'm not sure what you're doing so it's hard to answer. If you're dry-running a migration on a connection that was forked via Datomock, there's no syncing to do - just wait for the transaction to return and you're good. If you've run a migration on an actual Transactor and need to see the effects of that in a forked Connection, then you can sync-schema the real connection then fork.

uwo17:01:28

ah. hmm. so this is a dry run migration that adds unique to an existing attribute. That requires first transacting :db/index true for the existing attr, then calling sync-schema, and then asserting unique.

val_waeselynck17:01:46

@U09QBCNBY if you want to master the principles behind Datomock, here's a crash course 🙂 Imagine Datomic connections are Clojure agents holding db values, and (defn fork-conn [conn] (agent @conn)). That's it!

val_waeselynck17:01:45

@U09QBCNBY but do keep me posted if my assumptions seem incorrect, maybe Datomic does some dark magic in this case that I don't know about. Datomock basically gives you everything that datomic.api/with provides.

uwo16:01:17

man, I can’t type today 🙂

stuarthalloway17:01:11

@laujensen one other option to consider: store the blobs in Datomic with a content-address prefix in the value itself

stuarthalloway17:01:48

you would have to write application logic everywhere such values were used to strip/add the prefixes

stuarthalloway17:01:15

pretty gross, but in the spirit of covering all options…

alexk21:01:42

What’s the natural way to truncate a Datomic database? I know that I could retract every entity, but that seems like a lot of work. So far I’ve been deleting the database and creating another with the same name. That definitely “truncates” the database, but it leaves any currently-connected clients in a bad state. Truncation is useful because I want to migrate data into a clean database each time.