Fork me on GitHub
#datomic
<
2016-10-18
>
dominicm14:10:08

http://docs.datomic.com/best-practices.html#use-pull-to-retrieve-attribute-values are there any performance hits/gains from using pull instead of {:find [?every ?attribute ?i ?am ?interested ?in] :where [[?e :blah/is 1] [?e :every ?every] ...]}? I have a lot of queries using this syntax. I'm guessing I might see some improvement simply because I can more easily re-order clases (http://docs.datomic.com/best-practices.html#most-selective-clauses-first)

robert-stuttaford14:10:09

i don’t think you’ll see a massive perf gain by doing that, but you will see cleaner code by using pull

yonatanel14:10:11

@dominicm How does using pull expression make it easier to re-order where clauses?

robert-stuttaford14:10:32

q/q to find which things, and d/pull to look at them

robert-stuttaford14:10:12

either way, the query cache will almost certainly have your data in local memory by the time :find or d/pull operate

dominicm14:10:29

@robert-stuttaford As long as there isn't a significant hit, I'm happy. I'm trying to optimise a 20ms query that we currently run 2k times… Hopefully I can get it to run once — when we actually look up the entities. But I can't understand the queries to begin with, so this seems like a good first step.

robert-stuttaford14:10:53

that sounds like fun 🙂 what are you doing for measurement?

dominicm14:10:09

That's exactly what I am using! I have been wondering if this slipping under the radar isn't an excuse to have some kind of profiling for every request in dev, with significant spikes causing the speaker to notify at you.

robert-stuttaford14:10:08

i found us using a datalog query as a sort-by function using tufte -grin-

dominicm14:10:47

https://blog.codinghorror.com/performance-is-a-feature/ Stack Overflow have this MVC mini profiler, which is just awesome. It'd be pretty cool to generate one for every request.

robert-stuttaford14:10:14

oh man i’d love that

dominicm14:10:53

https://github.com/yeller/clojure-miniprofiler I did play with this a little bit, it was quite fun 🙂

robert-stuttaford15:10:51

you just put something on the very top of my list, sir

robert-stuttaford15:10:09

it has datomic measurements? i wonder how it does that...

dominicm15:10:44

It has some really neat ways to "label" parts of your code. I guess it's just that

dominicm15:10:13

It has a custom-timing function, which could be used for datomic:

(custom-timing "sql" "query" my-query-string
  (execute-sql-query my-query))

robert-stuttaford15:10:27

ah, so it cheats 🙂

robert-stuttaford15:10:33

and makes you do the work

dominicm15:10:45

Yep, same as Tufte.

robert-stuttaford15:10:14

does it deal with laziness sanely? or do you have to doall the things yourself?

yonatanel15:10:38

I know datomic can support a "tiny" number of databases with the same transactor. Is it also true for abandoned databases that are never connected to anymore? A case might be migrating the data to a new database and leaving the old one as is.

jaret15:10:02

@yonatanel Every database running against a transactor will have to maintain a small amount of overhead. You can delete the abandoned database by calling d/delete database. To be clear, you could also leave abandoned databases out there as it is a very small amount of overhead.

dominicm16:10:10

Okay, I'm a little confused still, I have this query:

'{:find [...]
  :in [$ ?input]
  :where [[?input :e/name ?ea]
          [?aaa :aaa/name ?ea ?tx]
          [?aaa :aaa/locations ?location]
          [?location :location/addresses ?address]]}
and it was performing OK (21ms mean avg) but I needed faster as it's running 2k+ times (If I can optimize that part, I'd love to, but can't see how right now...) I figured the problem might be that I am doing an AVET lookup from ?ea on the :aaa/name value, so I added an index there, but haven't seen any performance improvement (I did sync-schema). In case it's relevant, this query is being done on a as-of db.

dominicm16:10:06

I'd like to know how to most efficiently figure out what the expensive lookup is, and how to optimize that part. I might be into difficult territory, in which case I think it's better to re-evaluate if I need to do this, but that'll require me cleaning up an even larger query..

jaret17:10:21

@dominicm Datalog is super nice for query de-composing. I recommend dropping each clause and confirming you have the optimized clause order.

jaret17:10:31

This example walks through the steps

jaret17:10:41

This will also tell you the biggest cost clause

dominicm17:10:39

@jaret Thanks, I'll have a look through that tutorial now

dominicm17:10:40

The biggest hit was 15ms from the second clause. I believe it's due to https://github.com/Datomic/day-of-datomic/blob/master/tutorial/decomposing_a_query.clj#L81 I tried to get round it by adding an index.

dominicm17:10:26

I'm finding a relationship through a shared, non-normalised value. I guess that's the hit, but I thought that was the purpose of an index. Or have I misunderstood?

dominicm17:10:49

{:db/ident :aaa/name
 :db/index false
 :db/id #db/id [:db.part/db]
 :db.alter/_attribute :db.part/db}

dominicm17:10:46

I'm so stupid

jaret17:10:59

The false?

dominicm17:10:36

Let's see if that offers any performance improvement 🙂

dominicm17:10:47

Mean time: 6.84ms. Excellent!

whitecoop22:10:40

Does anyone know if there are any other ports that I need to open up besides 4334-4336 to connect peers to a Datomic dev transactor (4336 is open to see the h2 database)? I've got a transactor running on one box and when trying to connect from a peer on another box I can see how many databases I have from the peer (`(d/get-database-names "datomic:dev://<transactor-box-ip>:4334/*")`) – so the peer is connected – but if I try to create a new db (`(d/create-database "datomic:dev://<transactor-box-ip>:4334/test-db-two")`) I get an exception (`ActiveMQNotConnectedException AMQ119007: Cannot connect to server(s). Tried with all available servers. org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:799)`). However, running the exact same create-database call on the transactor box succeeds. So it's only an issue with the peer. I'm wondering if it has something to do with a port I haven't opened that should be and that's why the peer doesn't succeed. Couldn't find anyone with a similar issue on Google. Thanks in advance for any help.

whitecoop23:10:01

Well, it's not ports. Temporarily disabling the firewall didn't fix it.