Fork me on GitHub
#datomic
<
2016-07-29
>
zentrope00:07:07

The database function docs are kinda sketch on how one tx function can call another tx function. Is that possible, even?

cezar00:07:35

is there a way to efficiently read through all datoms within a given partition? I'm thinking of using indexRange and the EAVT index. But don't know how to specify the lower and upper bound. Can i somehow find out what the lowest and highest entId is for a given partition?

kenny00:07:00

@cezar: I'd question why you want to do something like that. This should do what you want though.

(d/datoms (d/filter db (fn [_ ^Datom datom]
                         (->> datom .e d/part (= part)))) :eavt)

cezar00:07:33

what you showed is a sequential scan over the whole database. If I have a 1000 customers and each in its own partition and I want to trawl through a reasonably large collection of datoms (say 10,000,000 for each customer) it will be way more efficient to hone in only on the section of the index where their specific data is located. Isn't that what partitions are for?

kenny00:07:36

Just query the database?

cezar00:07:59

queries are very eager and I find that running d/q where the result is more than about 5,000,000 entities can easily blow up a Peer with OOM even on a Peer with Xmx4g

cezar00:07:39

I prefer the lazy nature of d/datoms or indexRange or seekDatoms

cezar00:07:48

for my use case here

bkamphaus00:07:05

The locality of reference is in :eavt, you could take-while with a pred on part - http://docs.datomic.com/clojure/#datomic.api/part — starting from the beginning of the partition with seek-datomshttp://docs.datomic.com/clojure/#datomic.api/seek-datoms

cezar01:07:26

ah, figured it out. looks like seekDatoms on EAVT is the way to do this

cezar01:07:45

thanks @bkamphaus for pointing me in the right direction

cezar01:07:50

I can simply look use entIdAt with Date 0 and Date now to get all entities over a given partition

zentrope02:07:17

How do you lookup a database function so that you can call it from within another database function?

zentrope02:07:22

I ask because I find myself wanting to (mapcat (fn [x] [:my/fn db a b c]) values) which seems, odd? Hm. Maybe it’ll work.

zentrope03:07:41

Ha! It works! My tx function is huge with multiple functions, but I don’t think there’s much else I can do when updating a complicated “document-like” structure.

jimmy03:07:20

hi guys how do I avoid transacted value being printed to *out* in repl ? I import quite big amount of data and it's annoying.

zentrope03:07:52

@nxqd (do @(d/transact ….) :done) Something like that?

jimmy03:07:23

yeah totally forgot T_T thanks

val_waeselynck08:07:28

@bkamphaus: I'm having a doubt about how Datomic works: are the indexes persisted in storage are immutable trees of segments, or is this just the in-memory representation?

robert-stuttaford10:07:53

i usually do (-> (d/transact ...) :db-after) or ( .. :tx-data count)

marshall12:07:25

@val_waeselynck: it’s turtles all the way down.

marshall12:07:40

@zentrope: Yes, you can call txn functions from other txn functions. As long as they all eventually resolve to valid txdata (i.e. list of datoms)

marshall12:07:04

@cezar: Yep, as you found (and @bkamphaus indicated), the most significant bits of the entity ID encode the partition. Thus, the partition is contiguous in EAVT index.

val_waeselynck13:07:45

> it’s turtles all the way down. @marshall: sorry, I don't know what that means 😕 (not a native English Speaker)

marshall13:07:01

@val_waeselynck: Sorry. Yes, it’s immutable segments everywhere.

dominicm13:07:30

@val_waeselynck: As an english speaker, I do not understand either.

zentrope17:07:17

@marshall: Yes, it all makes sense if you think of a TX function as producing a data structure. Kinda like a macro.

nando19:07:57

I have a few questions regarding the console.

nando19:07:11

1) Does it work with mem databases?

nando19:07:17

2) From where do I get the dev-transactor.properties file?

nando19:07:17

3) The example command to start the console here http://docs.datomic.com/console.html:

bin/console -p 8080 dev datomic:
Has 2 port designations 8080 and 4334. I get that the web app that is the console is then available on 8080, but what is running or available on 4334? Do I need to start a transactor in a local dev environment on 4334? I think I read somewhere that I don’t, but it’s not clear from the explanation on this web page.

marshall19:07:23

in that example, the "datomic:<dev://localhost:4334/>“ is the URI of the database

nando19:07:45

Where do I set that?

marshall19:07:59

There is an example dev-transactor.properties file in the Datomic Pro distribution under config/samples

nando19:07:18

Not in the most recent download ...

marshall19:07:13

do you have Datomic Pro Starter ?

nando19:07:39

OH, sorry. I found it

nando19:07:49

It is in a samples subdirectory.

marshall19:07:00

if you’re using free, there is a free-transactor-template.properties file in config/samples

nando19:07:01

Thanks @marshall. When do I need to specify the transactor.properties file in a command? Do I need to start a transactor in a local dev environment?

jaret19:07:02

@nando you want to specify the file when you launch your transactor. Yes, you will need to start a transactor in a local dev env.

nando19:07:17

Is there a guide somewhere I’m missing that explains how to set up a dev environment?

nando19:07:22

http://docs.datomic.com/console.html jumps into explaining how to start the console without mentioning that you need a transactor running

jaret19:07:13

@nando The getting started section is where I went first. I didn’t make my way into the overview section until after the tutorial.

nando19:07:58

Ok, I’m more clear what I need to do now. Thanks again.

zentrope19:07:59

Using the pull api: Is there a way to turn {:foo {:id “1234”}} to {:foo “1234”} without doing any post processing?

uwo20:07:54

If I want to use an aggregate result to track down another value on an entity, will I always need to use two queries?

(defn last-edited [conn eid]
    (d/q '[:find (max ?inst)
           :in $ ?e
           :where
           [?e _ _ ?tx _]
           [?tx :db/txInstant ?inst]]
         (d/db conn) eid))

;;(the following is not what I want because “Query variables not in aggregate expressions will group the results and appear intact in the result.”)
(defn last-edited [conn eid]
    (d/q '[:find (max ?inst) ?user
           :in $ ?e
           :where
           [?e _ _ ?tx _]
           [?tx :db/txInstant ?inst]
           [?tx :user/source ?user]]
         (d/db conn) eid))

uwo20:07:16

alternatively I could just query the entire history, sort by txInstant, yadda yadda

bhagany20:07:46

@zentrope: no, pull will always do that for refs

bhagany20:07:21

@uwo: I believe you can use a subquery for this, but I don't know the right syntax off the top of my head. I think that you can put a second (d/q …) in the place of a where clause, though.

uwo20:07:41

hmm! thanks I’ll look into that

zentrope21:07:51

@bhagany: I was willing to be totally surprised that it was possible. ;) Fortunately, clojure.walk solves all problems.