This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-03
Channels
- # bangalore-clj (1)
- # beginners (160)
- # boot (174)
- # cider (57)
- # cljs-dev (16)
- # cljsrn (5)
- # clojure (144)
- # clojure-argentina (1)
- # clojure-austin (6)
- # clojure-finland (2)
- # clojure-france (4)
- # clojure-russia (185)
- # clojure-serbia (4)
- # clojure-spec (61)
- # clojure-uk (126)
- # clojurescript (212)
- # community-development (7)
- # core-async (2)
- # cursive (17)
- # datomic (210)
- # emacs (10)
- # euroclojure (2)
- # gsoc (23)
- # hoplon (86)
- # jobs (8)
- # lein-figwheel (9)
- # luminus (19)
- # lumo (12)
- # off-topic (4)
- # om (14)
- # om-next (6)
- # pedestal (38)
- # perun (11)
- # planck (35)
- # portland-or (2)
- # ring (1)
- # ring-swagger (28)
- # specter (6)
- # untangled (6)
- # yada (2)
@sova, I've found, by the way, that I usually don't need to get the entity id when the code is written in idiomatic way
although of course that depends on what you're transacting 🙂
has anyone come across an :db.error/invalid-entity-id on valid entities before?
(d/transact (conn) [{:db/id 17592186076008, :account/owner-phone-number "047041****"}])
=> #object[datomic.promise$settable_future$reify__6480 0x49b7d935 {:status :failed, :val #error {
:cause ":db.error/invalid-entity-id Invalid entity id: 17592186076008"
(d/touch (d/entity (cal.utils.database-connection/db) 17592186076008))
=> {:account/owner-phone-number "<snip>", :account/base-commission 0.0, ...}
I'm using the dev db, deleted the data and restored a backup of prod, where we seemed to be having a bit of trouble
@reitzensteinm You cannot transact arbitrary entity ids explicitly
the only way to "mint" a new entity id is to transact with a temp id and let the db assign an actual id
@favila yes I'm sure it existed (see the second output)
what is the basis-t of the db? @reitzensteinm
it says 540270
is that a transaction count?
there is one counter inside the db which is incremented for every minted entity id. its value goes in the "t" part of an entity id (the lower bits)
ah, I see
makes sense. it's a very old entity
been in the db for nearly a year
i'm suspecting corruption of the db. I was investigating an issue where, on prod, a transaction with a tempid overwrote an existing entity
the error occurs on a backup of the prod db, restored locally
(not the same entity)
but the restored db is apparently very broken
other sanity checks I can think of: are you sure that db is actually from the correct connection? Are you connecting to both transactors from the same process with an older datomic version? (older datomic versions would reuse the connection)
do you see any 0-size files in the values/ dir of your backup? (this happened to us--a corrupted backup)
there's just the one transactor (the same process does not connect to prod)
will check the backup dir
i've deleted and rebacked up several times
it's on postgresql, hosted on RDS
the same error (writing to existing entity) does not, but I'm thinking that the other shoe is going to drop pretty soon
just the one error on prod indicating corruption (tempid reusing an existing entity)
do you mean using the datomic integrity diagnostics fn?
I'm not aware of that? I meant the blob column in the sql db will be size 0 or start with 0x00
ah I see
thanks for your help! I'll try to look for corruption and file a ticket
select * from datomic.datomic_kvs where id like "pod%" will get you the (mutable) roots, may have some interesting things in it
no 0 sized files in the backups dir, lowest are 60
is there some sort of trick for manual indexing to help me get around a terribly slow query ?
photos belong to a collection, collection belongs to organizations. I have 2M photos and I can't get the highest "foreign key" from my photos because running out of RAM.
as long as the attribute is index/true you could simply walk (d/datoms :avet :foreign/key) until you get to the largest
it's basically a user. firstname / lastname... they create a collection :collection/organizaiton and then you upload photos to your colleciton :collection/photo
:user/name :db/type :string :collection/organization :db/type :ref :photo/collection :db/type :ref
so... i guess one thing that would help would be being able to quickly iterate through an org's photos
would it make sense ( i'd hate to do it ) but to add an attribute :photo/organization
but i'm importing old data. so after import i'm trying to set the counter to the highest fk
right... i guess i'm trying to find ways though to offset some of my performance anxiety about these kinds of queries in the future
do you think I should think about duplicating the attr :collection/organization to :photo/organization ?
right... i guess it's like an index, it would be cool if you could have these kinds of indexes happen in the background
you're trying to avoid realizing a large intermediate set you only want an aggregation for
you can do that by performing the aggregation in the where of the query via a function call instead of using datalog to aggregate (which will realize the intermediate set)
Although I'm not sure how relevant this is from your discussion because I'm not sure how the entities are connected. I would have to see the original query
why are you interested in the highest entity id anyway? that seems like a strange thing to care about?
the highest foreign key... i'm migrating data from databases that used an RDBMS and i want to keep the foreign key counter going seamlessly when they migrate to new system
if you have some prior knowledge of the values you’re looking for you might be able to leverage http://docs.datomic.com/clojure/#datomic.api/index-range
@jdkealy something like this https://gist.github.com/favila/a93662a47952c5eda6708a1e5fddd791
datoms are stored in sorted order, so the :avet index will sort by a, then v, then e, then t
honestly i'm using elasticsearch for most of my queries... datomic i'm just using to give me the facts
so it may not be fast if there are a lot of datoms to seq over, but they will never all be in memory at the same time
neat side effect - you’ll populate your local cache with all the segments about your orgs and photos by running it
my point was only that if you ran that reducer on a “cold” peer by the time it ran you’d have cached potentially most of your DB
without an indexes, it ran for like an hour and i gave up, with indexes GC overhead, and this way like 10 seconds
most of the time query is great and preferred, but there are a few cases where you definitely can’t beat direct datoms
yeah that access pattern is a weakness of queries. Queries can't do the aggregation until the entire set you are aggregating over is realized
Question: what is a Datomic database value, really? Is it just an entity id or something? I'm asking because I used an exception service today to diagnose a production exception (in a Rails app), and I thought, man, it'd be so nice to be able to query the database at that point in time. I got to wondering whether Datomic database values were lightweight enough to include in the exception information, so that I could copy the value from an exception report, then go and paste it into a REPL session connected to the production database for forensic investigation.
@jeff.terrell You can recreate a db value from its basis-t, as-of-t and since-t (both usually nil). We log the t in situations like you describe
@favila - Thanks! That makes sense. Filing this little fact away for future Datomic advocacy. :-)
@jeff.terrell More technically, a database value is a pointer into one set of "root" nodes in the immutable persistent trees that make up Datomic's indexes in storage. New versions of these roots are created each time the transactor does an indexing job; old roots are discarded when you run gc-storage
. The d/as-of
call gives you a "filtered" view of the database as it existed at time t
.
👍 neat. Thanks for the explanation, @stuartsierra.