is GPG broke on el capitan? I am trying to work with datomic-pro and am getting "pg: gpg-agent is not available in this session” even thought that daemon is indeed running

@bkamphaus @bostonaholic: still the same error using: @(d/transact conn [[:db.fn/retractEntity (Long. company-id)]])

@bkamphaus: @bostonaholic I found our problem with retracting an entity somehow we saved the entities in the

db.part/tx

@mitchelkuijpers: that would do it. Note that to have gotten this outcome if using e.g the map form, you’d have specified a tempid for :db.part/tx in the same attr. These attributes then become attributes on the transaction entity. For annotating a transaction in the map form case you would need to supply separate maps for the attributes intended as tx annotations and attributes intended for a new or existing entity. Example (though Java) at: http://docs.datomic.com/transactions.html#reified-transactions

Follow-up : It was because I didn’t fully nuke the gpg installed by brew before installing the one recommended by leiningen.

I just realized that postgres is using 30gb of storage for a small app

I assume this is because I havent been garbage collecting, or am I messing something up bad

What is the best tutorial for someone who wants to use datomic + clojure

these docs are a mess

@erichmond: the day-of-datomic repro is at: https://github.com/Datomic/day-of-datomic — if you’re talking about the tutorial on the docs page, it’s available in clojure in the datomic directory as mentioned here: http://docs.datomic.com/tutorial.html#following-along — if you’re looking at query specifically: http://docs.datomic.com/query.html points to the clojure examples from day-of-datomic here: https://github.com/Datomic/day-of-datomic/blob/master/tutorial/query.clj

@bkamphaus: thanks, also, this datomic for 5 year olds is helping too

@erichmond: We also have the full Day of Datomic training session as a series of videos here: http://www.datomic.com/training.html

I really got a lot out of those videos, fwiw

I've heard good things about http://www.learndatalogtoday.org/

thanks, I’ll check out the videos too.

@max: you should be doing some gc http://docs.datomic.com/capacity.html#garbage-collection — you may also have to take additional steps for postgres (and other storages) to reclaim data, e.g. VACUUM http://www.postgresql.org/docs/9.1/static/sql-vacuum.html

Actually, all the querying and whatnot is pretty straightforward to me

so it looks like my vm ran out space (I had a 40gb vm)

I upped the disk space

and my database size is only growing

and the transactor is unavailable

I was looking more for “10 steps to firing up a mem based datomic connection” “10 steps to firing up a dev based datomic connection + datomic console”

will this resolve itself?

I’m realizing now, if I want to run mem, I don’t even seem to need to download that datomic.zip, etc

@max not enough info to tell. can you tail the logs to see if the txor is busy? e.g. indexing

bkamphaus: debugging this I also found that my only log file is log/2015-06-26.log.

I kept the default logback.xml

so that’s another issue

is there another place they could be?

@max: does your transactor properties file specify a different log location?

bkamphaus: ah thanks. Okay so it’s indexing

aw crap

I may have done a bad thing.

I accidentally shoved some ~860kb strings into datoms

am I hosed here?

well it definitely can kill perf stuff, and will depend on how your system is provisioned. But yeah, you definitely want to avoid large blobby stuff in datoms. options for recovery — do you have a recent backup? You can also excise that stuff.

are those fields in :avet? i.e. indexed -- that’s when it will hurt the most by far.

they are indexed

bkamphaus: in the future, if i want to store this, doing noHistory and without index would be a bad idea still?

less of a bad idea, but I’d still avoid it. Indexing it guarantees that it will be a huge perf drag. Your best option for blob/document type stuff is to put in storage directly and store the pointer/ref/key w/e for it in Datomic in the datom

or a file store, e.g. s3

@arohner: Stu has replied re: your questions/issues on bytes reported here in slack and on group https://groups.google.com/forum/#!topic/datomic/JqXcURuse1M

@bkamphaus: yeah I just saw, thanks

Datomic 0.9.5206 has been released https://groups.google.com/forum/#!topic/datomic/kEAqsjeeMaE

@bkamphaus: do you work on datomic for cognitect?

@erichmond: yes, I’m on the Datomic team at Cognitect.

very cool!

I agree that it’s very cool to be on this team. Also, typing is hard.

bkhamphaus: thanks for your help so far.

I tried to run a garbage collect and an excision of one of the attributes

my database size is still growing (33->46 gb in the past hour)

and datomic is running at 100% cpu

heres a tail of the log

so it looks like im still indexing?

@max where you’re at, you’re waiting on indexing to push through — it will have to complete before space can be reclaimed and it will probably take longer for excision, etc. (more indexing necessary) — gc also competes for transactor resources — cpu/mem.

from the log tail, seems that way

how long can I expect to wait and is there anything I can do to speed it up

my hd is now 160gb, can I be reasonably sure I won’t hit that?

how many attr val pairs were targeted by the excision?

1

I was just doing a test on one datom.

@max can you grep for successfully completed indexing jobs, e.g. :CreateEntireIndexMsec metrics, index specific completion messages, grep ":[ea][aev][ve]t, :phase :end" *.log, possible failures (just grep for AlarmIndexingFailed.

the last index specific completion was 3 hours ago

2015-07-30 11:58:23.370 INFO  default    datomic.index - {:tid 150, :I 5265000.0, :index :eavt, :phase :end, :TI 8465930.540997842, :pid 1480, :event :index/merge-mid, :count 2110, :msec 14400.0, :S -283878.0409978423, :as-of-t 961005}

I have an AlarmIndexingFailed once a minute

2015-07-30 13:15:47.572 INFO  default    datomic.process-monitor - {:tid 13, :AlarmIndexingFailed {:lo 1, :hi 1, :sum 4, :count 4}, :CreateEntireIndexMsec {:lo 16500, :hi 18600, :sum 70500, :count 4}, :MemoryIndexMB {:lo 0, :hi 0, :sum 0, :count 1}, :StoragePutMsec {:lo 1, :hi 239, :sum 11097, :count 381}, :AvailableMB 2640.0, :IndexWriteMsec {:lo 1, :hi 659, :sum 35259, :count 381}, :RemotePeers {:lo 1, :hi 1, :sum 1, :count 1}, :HeartbeatMsec {:lo 5000, :hi 5346, :sum 60427, :count 12}, :Alarm {:lo 1, :hi 1, :sum 4, :count 4}, :StorageGetMsec {:lo 0, :hi 124, :sum 2204, :count 305}, :pid 1480, :event :metrics, :StoragePutBytes {:lo 103, :hi 4568692, :sum 128385966, :count 382}, :ObjectCache {:lo 0, :hi 1, :sum 231, :count 536}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :StorageGetBytes {:lo 1853, :hi 4568435, :sum 95278692, :count 305}}
2015-07-30 13:16:47.573 INFO  default    datomic.process-monitor - {:tid 13, :TransactionDatoms {:lo 3, :hi 3, :sum 3, :count 1}, :AlarmIndexingFailed {:lo 1, :hi 1, :sum 3, :count 3}, :GarbageSegments {:lo 2, :hi 2, :sum 4, :count 2}, :CreateEntireIndexMsec {:lo 15800, :hi 17400, :sum 50500, :count 3}, :MemoryIndexMB {:lo 0, :hi 0, :sum 0, :count 1}, :StoragePutMsec {:lo 1, :hi 291, :sum 11173, :count 474}, :TransactionBatch {:lo 1, :hi 1, :sum 1, :count 1}, :TransactionBytes {:lo 102, :hi 102, :sum 102, :count 1}, :AvailableMB 2460.0, :IndexWriteMsec {:lo 2, :hi 350, :sum 36373, :count 471}, :RemotePeers {:lo 1, :hi 1, :sum 1, :count 1}, :HeartbeatMsec {:lo 5000, :hi 5003, :sum 60006, :count 12}, :Alarm {:lo 1, :hi 1, :sum 3, :count 3}, :StorageGetMsec {:lo 0, :hi 100, :sum 2151, :count 351}, :TransactionMsec {:lo 19, :hi 19, :sum 19, :count 1}, :pid 1480, :event :metrics, :StoragePutBytes {:lo 86, :hi 4568692, :sum 146567666, :count 473}, :LogWriteMsec {:lo 8, :hi 8, :sum 8, :count 1}, :ObjectCache {:lo 0, :hi 1, :sum 247, :count 598}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PodUpdateMsec {:lo 2, :hi 7, :sum 9, :count 2}, :StorageGetBytes {:lo 86, :hi 4568435, :sum 94665879, :count 351}}

@max which version of Datomic are you running?

datomic-pro-0.9.5173

can you do a failover or start/restart to upgrade to 0.9.5201 (or latest 0.9.5206) to see if the indexing job is then able to run to completion?

okay

any reason to go with 5201 vs 5206?

I’d just drop into latest 5206 if no preference, 5201 is just minimal to get past a fix for a related issue. 0.9.5206 only adds error handling/explicit limits to byte attributes

bkamphaus: I updated, am getting some out of memory errors

015-07-30 13:31:43.668 WARN  default    datomic.update - {:tid 77, :pid 10386, :message "Index creation failed", :db-id "canary-f3e9a40e-2036-4ad9-aae7-52919cced434"}
java.lang.OutOfMemoryError: Java heap space

I’m using

# Recommended settings for -Xmx4g production usage.
 memory-index-threshold=32m
 memory-index-max=512m
 object-cache-max=1g

@max some follow up q’s then — can you verify you’re using GC defaults? Either only setting Xmx, xmx as transactor args, or if using JAVA_OPTS, adding -XX:+UseG1GC -XX:MaxGCPauseMills=50 to keep GC defaults? Also, would it be possible to up -Xmx (what’s current + available on machine)?

exec /var/lib/datomic/runtime/bin/transactor -Xms4g -Xmx4g /var/lib/datomic/transactor.properties 2>&1 >> /var/log/datomic/datomic.log

that’s my datomic command

I could up the memory, should I change transactor props also?

@max I would up memory, double it if you can — maybe up object-cache-max only slightly (i.e. to 25% of heap or so, not up to 1/2 for sure). I.e. something like -Xmx 8g, object-cache-max=2g, rest same

ok

bkamphaus: the excision finished!

thanks!

@max awesome — make sure and spread out the excision the way you’d normally pipeline txes on an import

assuming you’re following up by removing more of the blobby string vals

there are only 35 attrs to excise

…my postgres db size is at 51gbs though

ah, cool, so less of an issue then. as stuff pushes through, you’ll be able to run gc (or maybe it’s already running?)

I ran a datomic garbage collect and it didn’t seem to do much, I assume I should run it again and vacuum

it runs async

but yes you should do it after excision, more segments will need to be gc’d after that

the gc-storage call when finished will log something like: 2014-08-08 03:24:14.174 INFO default datomic.garbage - {:tid 129, :pid 2325, :event :garbage/collected, :count 10558}

so, how did this happen? I had 35 blobs some of which were like a meg at most. And the rest of my data is pretty small. How did my db grow to 51gigs?

And how do I make sure it doesn’t happen again, garbage collect daily?

I don’t know how much segment churn you go through, but it does build up over time from indexing. The blobs can be particularly bad with :avet on.

Nightly may not be necessary, but you can set up a gc-storage call to run at w/e period you determine is necessary

(as a side reference, for my [relatively normal] webapp, I run it at application bootup, because only the webservers are datomic peers and they're deployed together)

and then periodically I’m assuming you’ll need to VACUUM in postgres before space is reclaimed since the deletion in Datomic will be handled/deferred by table logic in the storage

i.e. Cassandra via tombstone, Oracle space reclamation is deferred by High-water Mark stuff, etc.

cool, thanks so much for your help @bkamphaus

Weird datomic error throwing me for a loop:

airworthy.repl=> @(api/transact @db/connection [{:segue/time #inst "2015-04-09T05:32:48.000-00:00", :segue/way :out, :segue/airport 277076930200614, :segue/user 277076930200554, :db/id 277076930200690}]) IllegalArgumentExceptionInfo :db.error/not-an-entity Unable to resolve entity: Thu Apr 09 00:32:48 CDT 2015 in datom [277076930200690 :segue/user #inst "2015-04-09T05:32:48.000-00:00"] datomic.error/arg (error.clj:57)

Why does it think the date is an entity?

what is schema for :segue/time ?

:instant

and :segue/user ?

:ref

Thank you for your help @bkamphaus

@bkamphaus: I ran a garbage collect and a vacuum, but pg still says the datomic database size is 51gb. Any suggestions?

@max have you backed the db up recently so that you have a reference for how large the backup is?

@bkamphaus: 150mb

@micah: as a sanity check, I would verify that all entities in the transaction exist and that all attr keywords are specified correctly (e.g. spelled correctly) and exist, including the (assumed enum) :out entity — it may be that something else wrong in the transaction is causing it to resolve to the incorrect datom that’s transacting the date as the value for the :segue/user attr (the cause of the exception)

@max that datomic db is the only one in that instance? you haven’t e.g. run tests that generate and delete dbs (note that dbs when deleted have to be [garbage collected](http://docs.datomic.com/capacity.html#garbage-collection-deleted), also). And definitely nothing else you’re storing in postgres?

I only use memory dbs for testing, and I don’t run tests on the production instance

There is another database on the pg instance, but it’s tiny:

datomic=> SELECT pg_database.datname,
       pg_size_pretty(pg_database_size(pg_database.datname)) AS size
  FROM pg_database;
  datname   |  size
------------+---------
 template1  | 6314 kB
 template0  | 6201 kB
 postgres   | 6314 kB
 canary-web | 7002 kB
 datomic    | 51 GB
(5 rows)

@max have you restored versions of the same database when the restore has diverged? The incompatible restore is one thing I’m aware of which can potentially orphan segments so that they never get gc’d.

I don’t think so. This is the production db, so it was initially restored from a seed db, and then only backed up

it looks like a lot of the growth (20gbs worth!) happened after I ran out of disk space last night and was trying to do excisions.

DBs do pick up small amounts of cruft from operational churn, but this is well out of lines of my expectation for the size of it. Depending on what kind of outage you could tolerate, you could do a test restore from backup to a clean postgres in a dev/staging environment and seeing what the resulting table size it.

The failure to index could be contributing then, maybe leaving orphaned segments somehow. There’s always the possibility of clobbering the table and starting from a clean restore, obviously you want to backup and test a restore as I mentioned above first before considering going down that path.

Do you know what the table size was prior to running into the indexing failure?

I’m not sure, I ran out of disk space at ~30gb

I’m assuming it’s going to affect performance to keep this 51gb database around

So I did a restore on my dev system, and the pg database is 142mb after restore. I can do a restore in prod again, but I’m worried about this happening again. Any suggestions as to what to do at this point?

is it possible I hit a bug in datomic?

@max hard to speculate about a possible bug without knowing more specifics. I’m wondering how much of this can be attributed to the failures to index w/the blob-ish strings. My general advice would be to make sure and make regular backups, and configure some kind of monitoring for Alarm* events - so that you can jump in more quickly (i.e. reacting to AlarmIndexingFailed, rather than toe running out of size).

bkamphaus: that makes sense, and it’s definitely my next plan of action

we ran out of space at db size 30 gb, so there must have been some failure before that that caused that 30gb to be written

but I guess that could have been cascading indexing failures?

I think it’s fairly typical for dbs in production over time to accumulate a little bit of cruft, but nothing like the difference in size from your backup to postgres table, which is why I think it must be linked to that indexing failure. I haven’t seen another report of that much excess size, usually when I’ve looked through those concerns about size differences it’s still less than 2-3x the expected size (after account for e.g. storages with replication factors, etc.) on dbs that have been running for a long time, nothing orders of magnitude larger than expected size like this — except with whole dbs not gc’d, or gc never having been run, etc.

ok. I’ll set up better monitoring and see if it happens again

one more question: we’re not using aws, and I am using datadog for this data. Do you generally recommend to use the built in cloudwatch stuff and push that data to other services, or is integrating with a non-AWS monitoring service pretty easy?

@max we definitely have users doing both. Cloudwatch is what we use at Cognitect and test the most, but lots of people on premise just configure their own callback ( http://docs.datomic.com/monitoring.html#sec-2 ) stuff or point it at various other logging/metric tools.

@bkamphaus: Thanks for the tip. I verify everything is correctly spelled and schema-fied.