Fork me on GitHub
#datomic
<
2017-01-13
>
jdkealy00:01:44

can i restore to an existing database? I've been trying to seed my database for my clients and running and re-running my import scripts, I get the error "The name 'restore1' is already in use by a different database"... The problem is it takes over 30 mins to build and deploy my code, which i need to do when i want to create a new database

sova-soars-the-sora01:01:52

total beginner's question: what's the difference between an ident and a lookup ref ?

luposlip07:01:32

@sova ident is usually a keyword you use instead of an attribute ID. Such as :user/likes, :user/email or :object/type. A lookup ref is used to identify a specific entity. You can lookup an entity using :db/id 124312345123254 (using the entity ID), or with a lookup ref such as :db/id [:user/email "[email protected]"]. For this to work you need to have the attribute (i.e. :user/email) defined as :db/unique :db.unique/identity in your schema.

karol.adamiec10:01:19

on top of that i would add that lookup ref throws when no entity is found. if one does :db/id 124312345123254 that assumes that id got queried before, and in case it is not in db you get back nil. so you can check aot instead of catching exception...

jdkealy14:01:11

Where can i find more info on paginating using (d/datoms) ? I read you can return lazy sequences this way.

jdkealy15:01:42

n.m looked at mbrainz sample... this is really an underdocumented way to assist with pagination. very nice

mitchelkuijpers15:01:41

@jdkealy Do you have a link to that example?

jdkealy15:01:01

here's an example of something i just put together... i import all my entities to elasticsearch

jdkealy15:01:26

:content/fk is an indexed attribute... it only seems to work on indexed attributes

mitchelkuijpers15:01:30

Aha we do almost the same

mitchelkuijpers15:01:50

But we do a simple query for for example only the db/id and the company/name for us

mitchelkuijpers15:01:00

and then sort by company/name and then the drop and take

jdkealy15:01:01

just realizing i shouhld be calling _db and not my function that returns a new db in the function.

jdkealy15:01:19

are you using seek-datoms?

mitchelkuijpers15:01:22

And then do a pull on the entities you actually want to show

jdkealy15:01:41

in my case with 3M records, i run out of java heap space when i use the query api

mitchelkuijpers15:01:09

Aha that seems like a problem, if you have them in elasticsearch why not serve them from elasticsearch?

jdkealy15:01:14

because it needs to pull the whole collection... with 2M records (and enough RAM) it would take like a minute per page

jdkealy15:01:26

this is how i import them to elasticsearch

jdkealy15:01:53

on create / update, i have a hook to import to ES, but this is for my initial import

jdkealy15:01:10

also... probably will run daily just in case something gets out of sync

mitchelkuijpers15:01:26

Ah ok, we will also have to reach out to ES in the future, but currently our collections are not bigger then 10.000 luckily..

mitchelkuijpers15:01:58

Really a shame that datomic does not fix this somehow 😞

mitchelkuijpers15:01:31

Thank you for the example seek-datoms seems like something I might need in the future

jdkealy15:01:52

i wish there was one DB that could do it all though

pesterhazy16:01:25

@jdkealy if you scroll up a bit there was a lengthy discussion on pagination the other day

bballantine19:01:04

Perhaps a dumb question: trying to simply determine if two entity references refer to the same entity in a datomic query. This is non-working code shows what I’m trying to do:

(defn same-person? [db p1 p2]
  (d/q '[:find ?p2 .
         :in $ ?p1 ?p2
         :where (= ?p1 ?p2)]
       db p1 p2))
Of course I get: IllegalArgumentException Cannot resolve key: =

bballantine19:01:34

.. and this the function can be called like (same-person? ddb [:person/slug "romeo-montague"] [:person/email "”])

marshall19:01:02

try equal? instead of =

favila19:01:39

= is not a rule. use [(= ?p1 ?p2)] @bballantine

potetm19:01:56

TIL equal? is part of datalog

marshall19:01:17

my bad; I think I mis-spoke 🙂

potetm19:01:43

Ah yeah. It parses, but doesn't do what you would hope.

favila19:01:06

@bballantine be aware that if you do not know ?p1 and ?p2 are both entity ids you may be lead astray

potetm19:01:07

Yeah I was thinking what @favila said.

marshall19:01:27

indeed, @favila said what I thought 🙂

favila19:01:38

@bballantine If necessary, use datomic.api/entid to normalize, inside or outside of the query

bballantine19:01:22

@favila yeah.. been messing with that.. seems a bit clunky, but this works currently:

(defn same-person? [db p1 p2]
  (d/q '[:find ?p .
         :in $ ?p1 ?p2
         :where [(datomic.api/entid $ ?p1) ?p1-id]
                [(datomic.api/entid $ ?p2) ?p2-id]
                [?p1-id :person/slug ?p]
                [?p2-id :person/slug ?p]]
       db p1 p2))

favila19:01:23

that looks good to me

favila19:01:39

or you can just do this:

favila19:01:52

(I think, try with lookup refs)

favila19:01:05

[?p1 :person/slug ?p][?p2 :person/slug ?p]

favila19:01:44

since you only care about ?p

bballantine19:01:59

Yeah.. right, so this works:

(defn same-person? [db p1 p2]
  (d/q '[:find ?p .
         :in $ ?p1 ?p2
         :where [?p1 :person/slug ?p]
                [?p2 :person/slug ?p]]
       db p1 p2))

favila19:01:09

(and I assume slug is unique)

favila19:01:33

clojure idiom is functions that end in ? return boolean, so you may want to coerce at the end, but that's besides the point

bballantine20:01:19

ah, good point

bballantine20:01:34

Actually this works and seems more.. direct:

(defn same-person? [db p1 p2]
  (d/q '[:find ?p1-id .
         :in $ ?p1 ?p2
         :where [(datomic.api/entid $ ?p1) ?p1-id]
                [(datomic.api/entid $ ?p2) ?p2-id]
                [(= ?p1-id ?p2-id)]]
       db p1 p2))

favila20:01:59

does not guarantee types, right? these two ids may not be people

bballantine20:01:24

I guess = only works in certain context of primitive types?

favila20:01:25

(also this doesn't need a query at all)

favila20:01:44

your earlier example checked for :person/slug

favila20:01:55

that guarantees that the entity id represents a person entity, no?

favila20:01:02

(in your domain model)

favila20:01:32

this is trivially true for e.g. (same-person? 0 0) => true

bballantine20:01:34

re needing a query… it’s actually a part of a bigger query

favila20:01:30

I'm still a bit fuzzy on how datalog handles lookup refs and keywords

bballantine20:01:46

I guess to answer your other question, it could be generalized (if it was stand-alone) to be called same-entity.

favila20:01:27

[?p1][?p2][(= ?p1 ?p2)] may work as such a rule, assuming datalog normalizes idents and lookups to entids

favila20:01:36

but I would test that first

favila20:01:16

I know it will understand them in the E or A slot (not the V slot ever!) of datalog match clauses

favila20:01:00

but = treats its arguments as values not relations, so not sure what the actual values will be

bballantine20:01:08

@favila - thanks again. Was just experimenting with the last suggestion. Unless I get the entity-ids out and compare them, the entity refs don’t pass the equivalency check. As you said, it seems = is just treating the entity refs as values. i.e. [:person/slug "romeo-montague"] is not equal to [:person/email “<mailto:[email protected]|[email protected]>"]

favila20:01:33

A good habit is to call d/entid on inputs to queries or at the top of a query on its entity-id-typed arguments in :in that the query will use

favila20:01:55

then you can write the rest of the query forgetting out this sublety

bballantine20:01:00

In the end, this Imma go with something like.. Actually might turn it into a rule.

(defn same-entity? [db p1 p2]
  (some?
    (d/q '[:find ?p1-id .
           :in $ ?p1 ?p2
           :where [(datomic.api/entid $ ?p1) ?p1-id]
                  [(datomic.api/entid $ ?p2) ?p2-id]
                  [(= ?p1-id ?p2-id)]]
         db p1 p2)))

bballantine20:01:33

@favila you can call datomic.api/entid in the :in part?

favila20:01:59

no, I just mean call it on :in args to normalize them

favila20:01:04

like you do here

favila20:01:13

so the rest of the query can assume that they are actual ids

favila20:01:00

e.g. of a gotcha where this matters [?e :person/friend ?p1], if ?p1 is not an entid it will never match

favila20:01:39

because datomic won't resolve entity identifiers in the V slot

bballantine20:01:10

ok thanks again for the help and info

sova-soars-the-sora21:01:51

in Datomic is it possible to get the "max 9 " of a db? Instead of just the absolute max?

sova-soars-the-sora23:01:08

Man, I could really use some more Pull syntax examples.

sova-soars-the-sora23:01:50

(defn get-blurb-info [bid]
   (d/q '[:find [(pull (d/db conn) [*] bid) ...]
              :in $
              :where
              [?bid :blurb/title ?title]
              [?bid :blurb/link ?link]
              [?bid :author/email ?author]
              [?bid :blurb/content ?content]] (d/db conn))) 
... is there something really wrong with my Pull syntax?

sova-soars-the-sora23:01:05

I'm just experimenting trying to track down what I've got mixed up... Ideally I'd just like to get all the results possible in this case (without supplying an entity ID)

dominicm23:01:03

Yes. Ther eis.

dominicm23:01:18

You don't need to give (d/db conn) to the pull in the :find

sova-soars-the-sora23:01:26

Hmm. Okay.. but removing that still leaves (pull [*] bid) which is an "invalid-pull" ...

dominicm23:01:08

I think you want (pull [*] ?bid)

favila23:01:18

@sova @dominicm (pull $ ?bid [*])

dominicm23:01:03

@favila I don't think that is right

favila23:01:04

The pull in :find clauses uses a different argument order from datomic.api/pull

dominicm23:01:09

You shouldn't need $

favila23:01:12

which is *maddening*

favila23:01:21

you don't need it, but it accepts it

dominicm23:01:27

But you're right, (pull [*] ?bid)

dominicm23:01:41

oh really? interesting.

favila23:01:53

I only just now realized that is not documented

favila23:01:25

What I really want to know is whose bright idea it was make arg order inconsistent

favila23:01:43

I get them mixed up several times a day