Fork me on GitHub
#datomic
<
2018-08-21
>
Petrus Theron10:08:02

Out-dated link in Datomic Cloud v0.8.54 query exception: (datomic.client.api/q '[:find [?e ...] :where ...] cloud-db) throws Only find-rel elements are allowed in client find-spec, see , which links to on-prem documentation that does support find-coll return, which was confusing. Exception should probably link to https://docs.datomic.com/cloud/query/query-data-reference.html

Petrus Theron11:08:47

Does datomic.client.api/with work for prospective tx on Datomic Cloud?

(d/transact cloud-conn {:tx-data []})
=> {:db-before {:database-id "57ad2773-...", :db-name "my-db", :t 3, :next-t 4, :type :datomic.client/db}, :db-after {:database-id "57ad2773-...", :db-name "my-db", :t 4, :next-t 5, :type :datomic.client/db}

(d/with (d/db conn) {:tx-data []}) ;; also throws for actual tx-data values
=> Exception: `clojure.lang.ExceptionInfo: Datomic Client Exception {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :datomic.client/http-result {:status nil, :headers nil, :body nil}}`

Petrus Theron11:08:19

Hi @U072WS7PE. I can't seem to find any logs related to this error, but I'll PM you my log group name. I see other errors about trying to use a missing db function though. (would be cool if I could stream errors related to my queries into REPL)

stuarthalloway11:08:24

We are improving error messages with each release. Are you on the latest client and CFT?

Petrus Theron12:08:28

I'm on com.datomic/client-cloud {:mvn/version "0.8.54"} and CFT version 402.

stuarthalloway12:08:29

so the first thing I search in any Datomic Cloudwatch log is "Alert - Alerts" -- that text will find all the potentially scary stuff, if any

Petrus Theron13:08:06

Took me about a minute to find the little page icon for CloudFormation template URL on https://docs.datomic.com/cloud/releases.html, which you can't right-click on. So I had to go to web inspector to find it. (It's not obvious that clicking on it puts something in your clipboard.)

Petrus Theron13:08:29

Attempting to create a new CloudFormation stack from template triggers Rollback with reason: No export named mysystem-CatalogTable found. Rollback requested by user. (I did not initiate anything)

Petrus Theron13:08:23

Hm, I must have done something wrong - probably didn't tick "Reuse existing storage" to yes, or it didn't detect when I pasted JSON template URL

Petrus Theron13:08:55

Stack started up again, but bastion instance not created

Petrus Theron13:08:41

Why would I not see an "Enable bastion?" option for the latest Solo CloudFormation template (v409)?

Petrus Theron13:08:33

Hmm...maybe I triggered the Storage only template. Trying to fix it via "Update Stack" with the Solo template and enabling bastion triggers a rollback again, with reason Export mysystem-Subnet1 cannot be deleted as it is in use by mysystem sigh I'm going to delete the whole stack and start over again

Petrus Theron13:08:59

Can't recreate stack with same name: No export named "mysystem-Subnet2" found. Rollback requested by user.

Petrus Theron13:08:59

After attempting an upgrade to 409 by deleting my stack as per docs (maybe it wasn't a "first upgrade"?) I had to go via the marketplace to avoid the subnet error and get the VPC options again; so far it doesn't seem to be rolling back. This cloud state management stuff is so complicated.

stuarthalloway14:08:28

Wow, that is a bummer. Will investigate.

Petrus Theron13:08:59

Can't recreate stack with same name: No export named "mysystem-Subnet2" found. Rollback requested by user.

henrik14:08:19

With Ions, I’m getting random stack overflows in my code when pushing to AWS,

"java.lang.StackOverflowError, compiling:(buddy/core/bytes.clj:72:15)"
Usually something in buddy, but not the same file every time. This started happening after adding Amazonica to do some S3 stuff. This is on solo, so is it likely to be OOM errors?

stuarthalloway15:08:32

@U06B8J0AJ this is definitely solo not having enough mem

henrik15:08:11

Poor solo. Can I edit the official cloud-formation template to use t2.medium or something instead?

stuarthalloway15:08:07

No. Solo runs only on t2.smalls. I am considering other options.

stuarthalloway15:08:22

If the problem is in compilation then deploying compiled bits may help.

stuarthalloway15:08:55

The last time I checked, Amazonica is enormous, and only has a monolithic deploy.

henrik15:08:53

Yeah, I tried my best to rip out large parts of it, but no go. I will consider other options. Perhaps the REST API instead, for now.

henrik15:08:01

I’m making a prototype at my own expense to convince my colleagues to Do The Right Thing, so I’d rather avoid taking on the cost of the production topology at the moment.

henrik15:08:42

But also of course to learn because it’s awesome.

okocim16:08:47

I had similar issues with amazonica myself. It’s a nice library, but I find it a bit too clever and bulky. I ended up just writing some small functions to do what I needed in S3 using interop and the DefaultAWSCredentialsProviderChain. It works great for my needs, and I’m happy to share the code with you if you’d like.

henrik17:08:55

@UBBBNAS6T That would be fantastic, thank you!

okocim19:08:57

(ns util.aws.s3
  (:import [com.amazonaws                   AmazonServiceException, SdkClientException]
           [com.amazonaws.auth              DefaultAWSCredentialsProviderChain]
           [com.amazonaws.services.s3       AmazonS3, AmazonS3ClientBuilder]
           [com.amazonaws.services.s3.model ObjectMetadata, PutObjectRequest]))

(defn default-s3-client []
  (AmazonS3ClientBuilder/defaultClient))

(defn stack-trace-as-str [e]
  (let [sw (java.io.StringWriter.)
        pw (java.io.PrintWriter. sw)
        _ (.printStackTrace e pw)]
    (str sw)))

(defn put-object-string
  ([bucket-name key-name value]
   (put-object-string (default-s3-client) bucket-name key-name value))

  ([s3-client bucket-name key-name value]
   (try
     (let [field-puller (fnil
                         (juxt (memfn getContentMd5)
                               (memfn getETag)
                               (memfn getExpirationTime)
                               (memfn getExpirationTimeRuleId)
                               (memfn getMetadata)
                               (memfn getVersionId)
                               (memfn isRequesterCharged))
                         {:response-obj "no data!"})
           [md5-hash
            etag
            expiration-time
            expiration-time-rule-id
            metadata
            version-id
            requester-charged?] (field-puller
                                 (.putObject s3-client bucket-name key-name (str value)))]
       (cond-> {}
         md5-hash                (assoc :md5-hash md5-hash)
         etag                    (assoc :etag etag)
         expiration-time         (assoc :expiration-time expiration-time)
         expiration-time-rule-id (assoc :expiration-time-rule-id expiration-time-rule-id)
         version-id              (assoc :version-id version-id)
         true                    (assoc :requester-charged? requester-charged?)))
     (catch AmazonServiceException e (stack-trace-as-str e))
     (catch SdkClientException e (stack-trace-as-str e)))))

🙏 4
okocim19:08:13

I only ever needed stuff on the put side, so there’s not much too this, but here you go anyway 🙂

henrik20:08:49

@UBBBNAS6T Do you manually rip out all AWS libraries you don’t use, or do you find it possible to include the entire SDK?

okocim20:08:51

for s3, I’ve been relying on the dependency that’s transitively included with an ion project, so I didn’t explicitly put it in my deps.edn, but as a rule, I’m only including the parts of the SDK that I am using.

okocim20:08:55

com.amazonaws/aws-java-sdk-sqs {:mvn/version “1.11.382”}

okocim20:08:57

for example

okocim20:08:01

in this case, com.amazonaws/aws-java-sdk-s3 1.11.314 is coming through transitively because of the ion-dev dependency

👍 4
henrik14:08:32

Both deploys AND rollbacks are failing.

stuarthalloway15:08:00

Did you deploy to a strong name (git SHA) or a weak one?

stuarthalloway15:08:23

If you are deploying to the same weak name over and over, rollback cannot work because you are overwriting the thing to roll back to

stuarthalloway15:08:54

That said, the stack overflow thing can be intermittent, so it may be you got lucky the first time and now cannot get back.

henrik17:08:45

No, I consistently use git SHAs for deploy. I think you’re right in that I got lucky. Maybe AWS is rolling back to that one lucky build. I’m worried I’ve screwed something up permanently though. I’m getting failed deploys for older builds even, where Amazonica definitely was absent.

steveb8n22:08:24

I suffered that problem as well a while back. In my case, it was due to the dep warnings seen during the push step. Once I found the right combination, the stackoverflows went away. My errors were referring to Specter for the stackoverflow but that was not accurate, it’s just where it was executing when it ran out of mem.

Petrus Theron14:08:20

How do I refer to the tx-id in a transact call using Datomic Cloud?

rhansen14:08:04

"datomic.tx" if I remember correctly. It's some magic string, you should be able to find it somewhere in the docs.

👍 4
Petrus Theron15:08:25

Where do my Datomic Cloud queries actually run? Do they run on the t2.small EC2 instance, or is that just a transactor?

stuarthalloway15:08:32

Bwa ha ha. It is magic!

Petrus Theron16:08:34

So do the nodes live on my Solo topology's t2.small?

okocim16:08:10

Is there any way to install the lambda proxy in a different region than the datomic cluster? I’m trying to use the pre-generate token Cognito trigger so that I can put additional claims for data from Datomic into the idToken that Cognito hands back, but it appears that the Cognito Triggers only work with lambdas that are in the same region as the user pool. Unfortunately, in this case, my user pool is in us-east1, and my Datomic system is in us-east-2.

Petrus Theron17:08:32

How are peeps doing granular application permission filtering on Datomic Cloud without d/filter? I'd like to give my power users Datalog read-query access to whitelisted attributes, but I'm not sure how I'd do that safely.

kenny17:08:55

For those of you using Datomic Cloud and looking for a solution to local development & testing or offline usage, I wrote this library https://github.com/ComputeSoftware/datomic-client-memdb which provides an implementation of the Datomic Client protocols for the Datomic peer in-memory database. We've been using it for about a month at my company and it has made local development and testing on the CI much easier.

👍 16
Petrus Theron08:08:09

I'm still getting d/with Datomic Client Exception with no error on CloudWatch running Solo topology after upgrading to client-cloud v0.8.63:

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :datomic.client/http-result {:status nil, :headers nil, :body nil}}
Do I need to cycle in my stack?

jaret12:08:39

What version is your stack running server side? (i.e. what CFT did you launch your solo stack with). I will investigate further.

tlima21:08:13

Is there anything like an undocumented alt-port property that I could configure at my transactor, to let peers know that alt-host also uses a different port?

tlima12:08:39

Where I can file a feature request?

jaret12:08:27

@UBY44N2JG under the “suggest a feature portal” located in the top right of your http://my.datomic.com account after login. If you don’t have a http://my.datomic.com account, please let me know and I can create one for you.

Petrus Theron08:08:09

I'm still getting d/with Datomic Client Exception with no error on CloudWatch running Solo topology after upgrading to client-cloud v0.8.63:

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :datomic.client/http-result {:status nil, :headers nil, :body nil}}
Do I need to cycle in my stack?