Hi @U1DBQAAMB -- that looks odd, what Statistic are you graphing there?

"getting really slow" implies change over time, what else (if anything) is changing over time?

The statistic is just "Memcache"

it's in cloudwatch, under the datomic namespace

according to docs https://docs.datomic.com/cloud/operation/monitoring.html#metrics this metric shows the number of cache hits

in regards to slowness... i have two queries that do a regex on a word to find a partial match... like an autocomplete, the queries have gotten so slow that sometimes it goes over 60 seconds and gets taken off the load balancer and then i have a range of other problems. I believe they're getting slower because the growing number of records

but actually... there's well under 100k records in the query, so i feel like it shouldn't be that slow

searching 100k records should be blazing fast

this is On-Prem?

I would love to see the query. Also, how big are the records? Are they bits of text? Documents? Tomes?

email addresses

what's on-prem

vs. Cloud

https://docs.datomic.com/on-prem/moving-to-cloud.html

oh right

yes i set it up before cloud was offered

[{:find [?u], 
       :in [$ % ?orgid], 
       :where [(find-relationship ?u ?orgid)]} 
       [email protected] 
       [[(find-relationship ?u ?orgid) 
        [?perm :permission/user ?u] 
        [?perm :permission/ent ?orgid]] 
       [(find-relationship ?u ?orgid) 
        [?o :transaction/user ?u] 
        [?li :line_item/transaction ?o] 
        [?li :line_item/organization ?orgid]] 
       [(find-relationship ?u ?orgid) 
        [?o :domain/organization ?orgid] 
        [?o :domain/user ?u]]] 17592186045798]

i guess there is some other stuff that makes it more complicated

oh wait sorry this is missing the email query

[{:find [?u], 
 :in [$ % ?orgid [?attrs ...] ?stext], 
  :where [(find-relationship ?u ?orgid) 
                [?u ?attrs ?val] 
                [(im.models.user.queries/includes-i? ?val ?stext)]]} 
                [email protected] 
 [[(find-relationship ?u ?orgid) 
 [?perm :permission/user ?u] 
 [?perm :permission/ent ?orgid]] 
[(find-relationship ?u ?orgid) 
[?o :transaction/user ?u] 
[?li :line_item/transaction ?o] 
[?li :line_item/organization ?orgid]] 
[(find-relationship ?u ?orgid) 
[?o :domain/organization ?orgid] 
[?o :domain/user ?u]]] 17592186045798 
[:user/name :user/email] "[email protected]"]

(defn includes-i? [in out]
  (clojure.string/includes? (clojure.string/lower-case in) out))

sorry this query references a lot of rules

i had another query that didn't... but i took it out because it brought my site down

@U1DBQAAMB to be seeing the slowdown you are describing, I would imagine that the query is considering several orders of magnitude more than 100K records

the two most likely causes for this (if your data set is small) are covered in the docs here:

1. https://docs.datomic.com/on-prem/best-practices.html#most-selective-clauses-first

2. https://docs.datomic.com/on-prem/best-practices.html#join-along

when rules are involved, you can ensure that things are bound in the order you expect by using the "required variable" shape when creating rules https://docs.datomic.com/on-prem/query.html#rules

Ok but wouldn’t a malformed cache config ration also lead to slowness?

maybe, but that slowness would impact everything (all queries)

100k records should fit in memory, at which point the memcache is not being consulted ever

you could look at the ObjectCache sum and average values

the sum would tell you how much cache you need

and the average would tell you how often you were hitting memory, without ever needing memcache

thanks stuart!

there's well over 100k records in the whole db more like 10M.... sorry i wasn't being clear, i meant 100k records of from the point of the first where clause

( i was thinking of my "users" like a SQL table, forgot it's nothing like that in datomic)

so... i had a wicked simple query take 53 seconds today

object cache is getting hit, memcache isn't, no ?

sorry to keep bothering you 😐 I just wanted to point out that the exact same query is slow in one instance and blazing fast in another, i don't know if that's helpful

@U1DBQAAMB if you are restarting processes, then the query could be slow the first time, and fast on subsequent times

how hard is it for you to experiment with memcached disabled?

^^ if that makes things better then you certainly know you have a memcached config problem

i'm not sure if it's even enabled

i mean... i have no way of seeing that it's being touched. there are in fact 80M datoms, and i believe i followed the instructions correctly for memcached support ( i used the cloud formation template and i pass datomic.memcachedServers as args to the JVM). I would just like to see one single read or validate the connection is used at all... otehrwise i would just spin down the cache and use more peers

Thanks for your insight! I am going to have to learn more about the differences between the two. 🙂

yeah, all the little differences caught me out too when I migrated from peer to client. I think of it like peer is v1 and client/cloud is v2

Hi @UCM1FJA4E & @U0510KXTU! Let us know if you find something absent or confusing at https://docs.datomic.com/on-prem/moving-to-cloud.html.

👍 Thanks, will do. When I read the documentation again I noticed that it mentioned the Client API so that made sense.

hi @U0539NJF7 -- the i3s are doing more for you than just CPU, e.g. the SSD caches keep your data hot across ion deploys

ok, I see. So, is there a way to measure what type of instance would be sufficient?

also the primary group instances have to do all the housekeeping work for Datomic, which is bursty and unrelated to your transaction load

all that said, we do not currently support anything smaller than i3.large for the production topology

are you (yet?) hosting your application code as ions?

no, we are still at the first version and now looking to do the upgrades

if/once you move to ions, your next tier out will disappear entirely (eliminating those instance costs, if any), and that load will be on the i3.larges, increasing the CPU utilization there

I understand, but we need quite a refactor for that as we're running on elastic beanstalk right now

it is definitely our intent that the baseline prod topology be a cost-effective way to host an entire application -- if you are not finding it so, I would be happy to look at the details -- you can email me <mailto:[email protected]|[email protected]>

no, I think the cost is not extreme for our production usage right now, just wondering why the marketplace mentions t2.small and was looking if that would be an option

backstory: the marketplace was originally designed to sell ec2 instances (not Cloud-native solutions) and then gradually upfit to support systems that use CFT and run multiple services

as a result, the marketplace UI thinks EC2 instances are the center of the universe, and does not anticipate different topologies with different capabilities

so there are various places where the UI suggests instance/CFT combinations that are not practicable, nor permitted

sorry for the confusion! we are working on it

ok, thanks for the explanation! 🙂

generally not a problem! https://stackoverflow.com/questions/33639642/avoiding-insufficient-data-in-cloudwatch

if the system is running (e.g. you are able to follow the instructions and connect) then you are fine

thanks! I’ll avoid disabling them and just ignore them for now. I appreciate the reply stuart