Fork me on GitHub
#datomic
<
2018-09-03
>
jdkealy00:09:41

my queries are getting really slow and i took a look at memcache, this would appear that memcache is not being used at all right ?

stuarthalloway10:09:03

Hi @U1DBQAAMB -- that looks odd, what Statistic are you graphing there?

stuarthalloway10:09:37

"getting really slow" implies change over time, what else (if anything) is changing over time?

jdkealy13:09:14

The statistic is just "Memcache"

jdkealy13:09:53

it's in cloudwatch, under the datomic namespace

jdkealy13:09:46

according to docs https://docs.datomic.com/cloud/operation/monitoring.html#metrics this metric shows the number of cache hits

jdkealy13:09:02

in regards to slowness... i have two queries that do a regex on a word to find a partial match... like an autocomplete, the queries have gotten so slow that sometimes it goes over 60 seconds and gets taken off the load balancer and then i have a range of other problems. I believe they're getting slower because the growing number of records

jdkealy13:09:47

but actually... there's well under 100k records in the query, so i feel like it shouldn't be that slow

stuarthalloway13:09:36

searching 100k records should be blazing fast

stuarthalloway13:09:49

this is On-Prem?

stuarthalloway13:09:16

I would love to see the query. Also, how big are the records? Are they bits of text? Documents? Tomes?

jdkealy13:09:25

email addresses

jdkealy13:09:31

what's on-prem

jdkealy13:09:01

yes i set it up before cloud was offered

jdkealy13:09:01

[{:find [?u], 
       :in [$ % ?orgid], 
       :where [(find-relationship ?u ?orgid)]} 
       datomic.db.Db@79b819fd 
       [[(find-relationship ?u ?orgid) 
        [?perm :permission/user ?u] 
        [?perm :permission/ent ?orgid]] 
       [(find-relationship ?u ?orgid) 
        [?o :transaction/user ?u] 
        [?li :line_item/transaction ?o] 
        [?li :line_item/organization ?orgid]] 
       [(find-relationship ?u ?orgid) 
        [?o :domain/organization ?orgid] 
        [?o :domain/user ?u]]] 17592186045798]

jdkealy13:09:53

i guess there is some other stuff that makes it more complicated

jdkealy13:09:41

oh wait sorry this is missing the email query

jdkealy13:09:13

[{:find [?u], 
 :in [$ % ?orgid [?attrs ...] ?stext], 
  :where [(find-relationship ?u ?orgid) 
                [?u ?attrs ?val] 
                [(im.models.user.queries/includes-i? ?val ?stext)]]} 
                datomic.db.Db@79b819fd 
 [[(find-relationship ?u ?orgid) 
 [?perm :permission/user ?u] 
 [?perm :permission/ent ?orgid]] 
[(find-relationship ?u ?orgid) 
[?o :transaction/user ?u] 
[?li :line_item/transaction ?o] 
[?li :line_item/organization ?orgid]] 
[(find-relationship ?u ?orgid) 
[?o :domain/organization ?orgid] 
[?o :domain/user ?u]]] 17592186045798 
[:user/name :user/email] "jdkealy@g"]

jdkealy13:09:32

(defn includes-i? [in out]
  (clojure.string/includes? (clojure.string/lower-case in) out))

jdkealy13:09:40

sorry this query references a lot of rules

jdkealy13:09:58

i had another query that didn't... but i took it out because it brought my site down

stuarthalloway13:09:40

@U1DBQAAMB to be seeing the slowdown you are describing, I would imagine that the query is considering several orders of magnitude more than 100K records

stuarthalloway14:09:38

the two most likely causes for this (if your data set is small) are covered in the docs here:

stuarthalloway14:09:22

when rules are involved, you can ensure that things are bound in the order you expect by using the "required variable" shape when creating rules https://docs.datomic.com/on-prem/query.html#rules

jdkealy14:09:37

Ok but wouldn’t a malformed cache config ration also lead to slowness?

stuarthalloway14:09:04

maybe, but that slowness would impact everything (all queries)

stuarthalloway14:09:33

100k records should fit in memory, at which point the memcache is not being consulted ever

stuarthalloway14:09:07

you could look at the ObjectCache sum and average values

stuarthalloway14:09:14

the sum would tell you how much cache you need

stuarthalloway14:09:39

and the average would tell you how often you were hitting memory, without ever needing memcache

jdkealy16:09:12

thanks stuart!

jdkealy16:09:53

there's well over 100k records in the whole db more like 10M.... sorry i wasn't being clear, i meant 100k records of from the point of the first where clause

jdkealy16:09:08

( i was thinking of my "users" like a SQL table, forgot it's nothing like that in datomic)

jdkealy16:09:09

so... i had a wicked simple query take 53 seconds today

jdkealy16:09:30

object cache is getting hit, memcache isn't, no ?

jdkealy18:09:00

sorry to keep bothering you 😐 I just wanted to point out that the exact same query is slow in one instance and blazing fast in another, i don't know if that's helpful

stuarthalloway19:09:00

@U1DBQAAMB if you are restarting processes, then the query could be slow the first time, and fast on subsequent times

stuarthalloway19:09:32

how hard is it for you to experiment with memcached disabled?

stuarthalloway19:09:53

^^ if that makes things better then you certainly know you have a memcached config problem

jdkealy19:09:11

i'm not sure if it's even enabled

jdkealy21:09:51

i mean... i have no way of seeing that it's being touched. there are in fact 80M datoms, and i believe i followed the instructions correctly for memcached support ( i used the cloud formation template and i pass datomic.memcachedServers as args to the JVM). I would just like to see one single read or validate the connection is used at all... otehrwise i would just spin down the cache and use more peers

clarice04:09:55

Hello, I have created a new database with a schema and am trying to add data to it

(require '[datomic.api :as d])
(def db-uri "datomic:)
(d/create-database db-uri)
(def conn (d/connect db-uri))

(def schema
  [{:db/ident :movie/title
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one
    :db/doc "The title of the movie"}

   {:db/ident :movie/genre
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one
    :db/doc "The genre of the movie"}

   {:db/ident :movie/release-year
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one
    :db/doc "The year the movie was released in theaters"}])

@(d/transact conn schema)
(def movies [{:movie/title "The Goonies"
              :movie/genre "action/adventure"
              :movie/release-year 1985}

             {:movie/title "Commando"
              :movie/genre "action/adventure"
              :movie/release-year 1985}

             {:movie/title "Repo Man"
              :movie/genre "punk dystopia"
              :movie/release-year 1984}])

(d/transact conn {:tx-data movies})
When I run it in the datomic REPL, I get ClassCastException clojure.lang.PersistentArrayMap cannot be cast to java.util.List datomic.api/transact (api.clj:92). Any chances that you can see what I am doing wrong?

steveb8n04:09:51

I think you are using the peer api instead of the client api. for peer txns you don’t put the entities in a map with :tx-data. try (d/transact conn movies) or try switching to the client api instead of datomic.api

👍 4
clarice06:09:43

Thanks for your insight! I am going to have to learn more about the differences between the two. 🙂

steveb8n07:09:47

yeah, all the little differences caught me out too when I migrated from peer to client. I think of it like peer is v1 and client/cloud is v2

stuarthalloway10:09:02

Hi @UCM1FJA4E & @U0510KXTU! Let us know if you find something absent or confusing at https://docs.datomic.com/on-prem/moving-to-cloud.html.

clarice10:09:05

👍 Thanks, will do. When I read the documentation again I noticed that it mentioned the Client API so that made sense. simple_smile

stijn13:09:15

hi, I see that for versions >= 402 you can choose t2.small for the datomic instances instead of i3.large, but the documentation still mentions i3.large for production topology. we have 1% cpu utilization on the i3.large instances, so having smaller ones makes sense I guess. What are the recommendations here?

stuarthalloway14:09:01

hi @U0539NJF7 -- the i3s are doing more for you than just CPU, e.g. the SSD caches keep your data hot across ion deploys

stijn14:09:59

ok, I see. So, is there a way to measure what type of instance would be sufficient?

stuarthalloway14:09:23

also the primary group instances have to do all the housekeeping work for Datomic, which is bursty and unrelated to your transaction load

stuarthalloway14:09:34

all that said, we do not currently support anything smaller than i3.large for the production topology

stuarthalloway14:09:58

are you (yet?) hosting your application code as ions?

stijn14:09:38

no, we are still at the first version and now looking to do the upgrades

stuarthalloway14:09:53

if/once you move to ions, your next tier out will disappear entirely (eliminating those instance costs, if any), and that load will be on the i3.larges, increasing the CPU utilization there

stijn14:09:16

I understand, but we need quite a refactor for that as we're running on elastic beanstalk right now

stuarthalloway14:09:57

it is definitely our intent that the baseline prod topology be a cost-effective way to host an entire application -- if you are not finding it so, I would be happy to look at the details -- you can email me <mailto:[email protected]|[email protected]>

stijn14:09:06

no, I think the cost is not extreme for our production usage right now, just wondering why the marketplace mentions t2.small and was looking if that would be an option

stuarthalloway14:09:42

backstory: the marketplace was originally designed to sell ec2 instances (not Cloud-native solutions) and then gradually upfit to support systems that use CFT and run multiple services

stuarthalloway14:09:49

as a result, the marketplace UI thinks EC2 instances are the center of the universe, and does not anticipate different topologies with different capabilities

stuarthalloway14:09:34

so there are various places where the UI suggests instance/CFT combinations that are not practicable, nor permitted

stuarthalloway14:09:16

sorry for the confusion! we are working on it

stijn14:09:15

ok, thanks for the explanation! 🙂

lilactown19:09:13

I’m inexperienced in both AWS and Datomic, so sorry if this is dumb: I’m currently using the solo config and seeing a bunch of alerts about INSUFFICIENT_DATA. anything I should be doing when I see those?

stuarthalloway20:09:50

if the system is running (e.g. you are able to follow the instructions and connect) then you are fine

lilactown20:09:31

thanks! I’ll avoid disabling them and just ignore them for now. I appreciate the reply stuart

👍 4
kenny22:09:16

Is it possible to pull transaction attributes from an entity?