I'm connecting to someone's Datomic On-Prem DBs, backed by DynamoDB. It's very slow to connect, and some tiny queries seem to eat RAM & never complete. Any ideas? (My current theory: the Peer is filling its cache with way too many things.)

;; Tested under:
;;   [com.datomic/datomic-pro "0.9.5561"]
;;   [com.datomic/datomic-pro "0.9.5786"]

;; "Elapsed time: 132848.045478 msecs"
(defonce ^:private db-prod
  (-> "datomic:ddb://..." d/connect d/db time))

;; "Elapsed time: 37830.972005 msecs"
(defonce ^:private db-dev
  (-> "datomic:ddb://..." d/connect d/db time))

;; Despite commenting out `sample` & `count`, query still fails on db-prod.
(defn request-that-eats-memory [db]
  (d/q '[:find [(rand 1 ?e) #_(sample 1 ?v) #_(count ?e)]
         :where [?e :foo/bar ?v]]
       db))

;; :foo/bar has only 5 entities in db-dev.
;; "Elapsed time: 2263.876834 msecs"
(time (request-that-eats-memory db-dev))

;; Consumes RAM at the rate of 70 MB/min.
;; Runs a few minutes before I abort.
(time (request-that-eats-memory db-prod))

Maybe A long time since last reindex? Do transactor logs complain about indexing failures?

Thinking out loud here, if index is very old then the log data in Txor might be big, leading to slow conn times as the peer got the log and slow queries because the unindexed portion of data is so big

I was wondering if anyone has seen this before: * I have a bunch of rules defined (example below) * Rules 1, 3, 4, 6, & 7 all work fine * Rules 2, 5, & 8 all fail * The failing rules contain ‘fn-expr(s)’ * I am running on datomic cloud * ALL rules work when running the query through the bastion (i.e. locallyi at the repl) * I get the following exception message when running through API Gatway (i.e. in my environment):

clojure.lang.ExceptionInfo:
      :db.error\/invalid-rule-fn
      The following forms do not name predicates or fns:
      (* * -)
      {:cognitect.anomalies/category
       :cognitect.anomalies/incorrect,
       :cognitect.anomalies/message
       "The following forms do not name predicates or fns: (* * -)",
       :symbols (* * -)
       :db\/error :db.error\/invalid-rule-fn}

(def sort-calcs
  
  '[;; WORKS
    [(rule-1 [?p] ?value)
     [?p :q.p/t :my-val]
     [?p :m.p/p ?value]]

    ;; FAILS
    [(rule-2 [?v ?p] ?value)
     [?v :f/m ?ms]
     [?p :q.p/t :my-val]
     [?p :m.p/rp ?r]
     [?p :m.p/p ?pa]
     [?p :q.p/t ?tl]
     [(* ?r 0.01 ?ms)  ?ra]
     [(* ?tl ?pa) ?tot]
     [(- ?tot ?ra) ?value]]

    ;; WORKS
    [(rule-3 [?p] ?value)
     [?p :q.p/t :my-val-2]
     [?p :m.p/p ?value]]

    ;; WORKS
    [(rule-4 [?p] ?value)
     [?p :q.p/t :my-val-2]
     [?p :m.p/r ?value]]

    ;; FAILS
    [(rule-5 [?v ?p] ?value)
     [?v :f/m ?ms]
     [?p :q.p/t :my-val]
     [?p :m.p/rp ?r]
     [(* ?ms ?r 0.01) ?value]]

    ;; WORKS
    [(rule-6 [?v] ?value)
     [?v :f/sp ?value]]

    ;; WORKS
    [[rule-7 [?v] ?value]
     [?v :v/dis ?value]]

    ;; FAILS
    [(rule-8 [?v ?p] ?value)
     [?v :f/sp ?pr]
     [?p :q.p/t :my-val-2]
     [?p :m.p/ma ?a]
     [(- ?a ?pr) ?value]]])

I posted this in the forum too, sorry if that feels spammy; I’m still feeling my way for what to put where… 🤷

my attempts to reach S3 in my Ions succeeds for a couple hours after deploying, but if I come back hours later, fails completely

doing another (unrelated) deploy then brings up it’s ability to access S3 again

when I execute the lambda for my Ion, I get this error:

{
  "errorMessage": "Cannot open <nil> as a Reader.",
  "errorType": "datomic.ion.lambda.handler.exceptions.Incorrect",
  "stackTrace": [
    "datomic.ion.lambda.handler$throw_anomaly.invokeStatic(handler.clj:24)",
    "datomic.ion.lambda.handler$throw_anomaly.invoke(handler.clj:20)",
    "datomic.ion.lambda.handler.Handler.on_anomaly(handler.clj:139)",
    "datomic.ion.lambda.handler.Handler.handle_request(handler.clj:155)",
    "datomic.ion.lambda.handler$fn__4075$G__4011__4080.invoke(handler.clj:70)",
    "datomic.ion.lambda.handler$fn__4075$G__4010__4086.invoke(handler.clj:70)",
    "clojure.lang.Var.invoke(Var.java:396)",
    "datomic.ion.lambda.handler.Thunk.handleRequest(Thunk.java:35)"
  ]
}

@lilactown do you have some kind of long-lived connection in your S3 ion?

OK, I could be dense but if this:

(def s3 (aws/client {:api :s3}))

which would explain a lot tbh

I’m using cognitect’s aws-api

you probably don’t want to do that in a def directly

you’ll want something like a memoized “get client” function

that makes sense! I didn’t realize it was actually creating a connection

i.e. sort of how the ion starter project handles datomic connections https://github.com/Datomic/ion-starter/blob/master/src/datomic/ion/starter.clj#L13

that error looks like trying to do something with a closed connection; which would make sense after it sits for a while

and the redeploy of another ion would cycle the process

causing your namespace-loading side-effect of creating the client again

mystery solved 😄

I’m a bit naive about how memoize actually works; if my connection drops, my assumption is that it would not gracefully reconnect but throw an error

you dont really even need it to be memoized

you can just have a ‘get client’ function

the key is not to create the client as a side effect of ns loading (via def)

but instead create it (or refresh it) when you invoke

like this: https://github.com/pedestal/pedestal-ions-sample/blob/master/src/ion_sample/service.clj#L45

should I bother trying to avoid calling (aws/client {:api :s3}) on each invocation?

i dont’ know exactly what the overhead is creating the client for that library; if it is cheap i wouldnt worry about it; if it isnt you could put it in a memoized function

I’m wondering that as well 🙂 I didn’t understand that it had some sort of long-lived connection

which begs the question how I should clean it up

if I’m opening a new connection to S3 on every invocation, and those connections are just lying around, I’m afraid I’ll end up taking up a ton of resources after awhile

Curious if anyone has had experience with utilizing Ions in a ring style app with CORS enabled? My issue is related to using the binary media type */* as the Ions tutorial recommends along with the CORS setup in API Gateway as apparently they don't play well together. If both are enabled, the preflight OPTIONS request generates an internal server error. Removing the */* binary type fixes the preflight request but then the body of all operations are returned base64 encoded. Any suggestions on what the right solution is to this?

@adammiller I had the same problem. I landed on setting up a single API gateway proxy ion endpoint instead of one per ion. Then I wrote a universal router that will handle the OPTIONS requests and CORS details. Here is an article written by Joshua Heimbach a few days ago that talks a little more about this approach. Mine is slightly different in detail, but conceptually the same. (I am using a different router) https://medium.com/@rodeorockstar/datomic-ions-aws-api-gateway-and-routing-d20a1bb086dd

Yeah, I'm already using one endpoint to route to my lambda which is served as ring app (basically) but I think handling cors at the app layer has some downsides 1) would be cost as you will be invoking lambdas for preflight requests (not huge deal), 2) Not sure it's possible to use amazon security this way, again not totally positive on this, just what I've found searching this problem where others talked about having the app layer handle cors.

Ok, yeah sorry I didn’t catch that you were on a single proxy ion from your description. I wasn’t happy with those tradeoffs either, but I decided to defer that issue for a little while because I had to move on to other ones 😅. If you come up with a solution that you like, I’d appreciate it if you share.

I definitely will, thanks for your input! I've been going back and forth on making those concessions myself as I can't spend much more time on this!

@okocim I found the answer (after a lot of searching). You have to run the following commands (apparently no way to change this in the Console):

aws apigateway update-integration \
    --rest-api-id <api-id> \
    --resource-id <resource-id> \
    --http-method OPTIONS \
    --patch-operations op='replace',path='/contentHandling',value='CONVERT_TO_TEXT'

aws apigateway   update-integration-response \
    --rest-api-id <api-id> \
    --resource-id <resource-id> \
    --http-method OPTIONS \
    --status-code 200 \
    --patch-operations op='replace',path='/contentHandling',value='CONVERT_TO_TEXT'

Would probably be nice for this to be in the Ions documentation somewhere as I'm guessing it will be a common problem for anyone who decides to host a full webapp (or api) inside Ions.

documentation related to setting up CORS, that is.

for now sounds like a good blog post 😄

@adammiller Thanks! That’ll allow me to do one of my favorite things: delete some code 🙂