Fork me on GitHub
#datomic
<
2020-09-29
>
armed06:09:34

Hi. Is there any way to make custom transaction function to omit operaton (e.g. return nil instead of transaction operation)? I want omit insertion of data in some situations.

armed06:09:01

I have permission entity with composite tuple (unique) on all three attributes. When I try to bulk insert list of permissions transation sometimes aborts with unique exception. I want to make something like postgres's on conflict do nothing. Here is my transaction function, which not workig obviously.

(defn try-add-permission
   [db {:keys [permission/app
               permission/user
               permission/role] :as perm}]
   (if (d/q '[:find ?p .
              :in $ ?app ?user ?role
              :where
              [?p :permission/app ?app]
              [?p :permission/user ?user]
              [?p :permission/role ?role]]
            db app user role)
     nil
     perm))

tatut07:09:28

how about returning empty vector instead of nil?

armed07:09:12

I already tried that. Got error

{:status :failed,
         :val #error{:cause "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                     :via [{:type java.util.concurrent.ExecutionException,
                            :message "java.lang.IllegalArgumentException: Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [datomic.promise$throw_executionexception_if_throwable invokeStatic "promise.clj" 10]}
                           {:type java.lang.IllegalArgumentException,
                            :message "Cannot write auth.server.cas_sync$try_add_permission@723b2a26 as tag null",
                            :at [org.fressian.handlers.WriteHandlerLookup
                                 requireWriteHandler
                                 "WriteHandlerLookup.java"
                                 48]}],
...

armed07:09:46

@(d/transact (db/get-connection) [[auth.server.cas-sync/try-add-permission perm]])

tatut07:09:10

you need to quote db fn name

tatut07:09:46

ah, it’s on-prem, don’t know about that

favila08:09:05

This is a quoting issue. The exception is related to serializing the function, which you can’t do. Your function is not being executed yet

favila08:09:41

In fact your transaction data hasn’t left the peer. What is the error you get when you quote the function name?

armed09:09:43

when I quote like this:

armed09:09:46

@(d/transact
   (db/get-connection) [['auth.server.cas-sync/try-add-permission perm]])

armed09:09:14

I get error: Could not locate auth/server/cas_sync__init.class, auth/server/cas_sync.clj or auth/server/cas_sync.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

favila09:09:27

is this function installed on your DB?

favila09:09:05

you’ll note in your link you need to make classpath transaction functions available on the classpath of the transactor. This error looks like it can’t find the function

👍 3
favila09:09:06

actually it can’t even find the namespace

armed10:09:39

@U09R86PA4 thanks, It seems that I misunderstood how transaction functions work.

tatut09:09:43

Updating datomic cloud compute group to 2020/09/23 715-8973 release, the log shows that the compute nodes don’t seem to get up after upgrade… complaining that our application code has syntax error (which it shouldn’t as it worked in previous version)

tatut09:09:54

production topology

tatut09:09:40

"Msg": ":datomic.cloud.cluster-node/-main failed: Syntax error compiling at …

tatut09:09:11

it doesn’t seem to find a required .cljc file

onetom07:10:26

has this been solved yet?

tatut11:10:49

yes, workaround with support… it seemd you can’t have paths that point to for example `“../common/src” (like we have sharing backend and frontend cljc code)

tatut11:10:02

it worked in all previous versions, but it doesn’t anymore with the latest

tatut11:10:21

workaround with symlinks seems ok

onetom11:09:10

When I connect a web ion via an API Gateway proxied thru a lambda, my ion function is supposed to receive a ring-compatible map as an argument (according to the official Ion docs). However, the map I receive only contains :headers, :server-name and a :datomic.ion.edn.api-gateway/data and /json keys, so I can't just use the typical routing libs to build my web-app or http API, because those depend on the :request-method and :uri keys of the request map. Is it a know issue? Is it something related to the Lambda proxy data format version? Is it just some kind of mis-configuration?

onetom11:09:54

here is an example request map I observed:

{:headers
                  {"accept-encoding"   "gzip, deflate",
                   "content-length"    "0",
                   "host"              "",
                   "user-agent"        "http-kit/2.0",
                   "x-amzn-trace-id"   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                   "x-forwarded-for"   "42.200.88.157",
                   "x-forwarded-port"  "443",
                   "x-forwarded-proto" "https"},
     :server-name "",
     :datomic.ion.edn.api-gateway/json
                  "{\"version\":\"2.0\",\"routeKey\":\"$default\",\"rawPath\":\"/\",\"rawQueryString\":\"\",\"headers\":{\"accept-encoding\":\"gzip, deflate\",\"content-length\":\"0\",\"host\":\"\",\"user-agent\":\"http-kit/2.0\",\"x-amzn-trace-id\":\"Root=1-5f730b24-4ac64db84deabaf53c38af60\",\"x-forwarded-for\":\"42.200.88.157\",\"x-forwarded-port\":\"443\",\"x-forwarded-proto\":\"https\"},\"requestContext\":{\"accountId\":\"191560372108\",\"apiId\":\"8g759uq7nb\",\"domainName\":\"\",\"domainPrefix\":\"8g759uq7nb\",\"http\":{\"method\":\"GET\",\"path\":\"/\",\"protocol\":\"HTTP/1.1\",\"sourceIp\":\"42.200.88.157\",\"userAgent\":\"http-kit/2.0\"},\"requestId\":\"Tn6trha8yQ0EMGg=\",\"routeKey\":\"$default\",\"stage\":\"$default\",\"time\":\"29/Sep/2020:10:23:32 +0000\",\"timeEpoch\":1601375012276},\"isBase64Encoded\":false}",
     :datomic.ion.edn.api-gateway/data
                  {:version         "2.0",
                   :routeKey        "$default",
                   :rawPath         "/",
                   :rawQueryString  "",
                   :headers
                                    {:accept-encoding   "gzip, deflate",
                                     :content-length    "0",
                                     :host              "",
                                     :user-agent        "http-kit/2.0",
                                     :x-amzn-trace-id   "Root=1-5f730b24-4ac64db84deabaf53c38af60",
                                     :x-forwarded-for   "42.200.88.157",
                                     :x-forwarded-port  "443",
                                     :x-forwarded-proto "https"},
                   :requestContext
                                    {:routeKey     "$default",
                                     :stage        "$default",
                                     :time         "29/Sep/2020:10:23:32 +0000",
                                     :domainPrefix "8g759uq7nb",
                                     :requestId    "Tn6trha8yQ0EMGg=",
                                     :domainName   "",
                                     :http
                                                   {:method    "GET",
                                                    :path      "/datomic",
                                                    :protocol  "HTTP/1.1",
                                                    :sourceIp  "42.200.88.157",
                                                    :userAgent "http-kit/2.0"},
                                     :accountId    "191560372108",
                                     :apiId        "8g759uq7nb",
                                     :timeEpoch    1601375012276},
                   :isBase64Encoded false},
     }

onetom11:09:07

my ion-config.edn looks like this:

{:allow    [datomic.ion.starter.http/ionized-app]
 :lambdas  {:app
            {:fn          datomic.ion.starter.http/ionized-app
             :integration :api-gateway/proxy
             :description "return html app"}}
 ;:http-direct {:handler-fn datomic.ion.starter.http/return-something-json}
 :app-name "kyt-dev"}

onetom11:09:54

I'm using the Solo topology (the version, which was the latest last week), otherwise I wouldn't bother with lamdba gateways if I could use the production topology.

onetom11:09:59

the docs are mentioning these /json and /data keys in a note, but just in the table above the note, they are not namespaced: https://docs.datomic.com/cloud/ions/ions-reference.html#web-ion

onetom15:09:04

thanks, I had a look, but I don't see how would it deal with my situation. it does have a great example of a protocol which converts the response body into an input stream, which I still need, because the reitit.ring/create-resource-handler just returns a java.io.File as a :body and Datomic threw some ->>bbuff conversion error as a result. for now, I just have a middleware to transform the above mentioned request map to be ring compatible:

(if-let [gw-data (:datomic.ion.edn.api-gateway/data gw-req)]
       (-> gw-req
           (assoc :uri (-> gw-data :requestContext :http :path))
           (assoc :request-method (-> gw-data :requestContext :http :method)))
       gw-req)

Joe Lane15:09:30

I want to make sure I understand, did you call apigw/ionize on your ring handler function? https://docs.datomic.com/cloud/ions/ions-tutorial.html#lambda-proxy

onetom15:09:06

I have the suspicion that or ion-config.edn doesn't need the :integration :api-gateway/proxy option anymore if I use the newer style HTTP API gateway setup (as opposed to the RESTful API style), it's just hasn't been documented...

onetom15:09:12

@U0CJ19XAM yes, that .../ionized-app is defined as (apigw/ionize app), where app is simply:

(fn [req]
      {:status  200
       :headers {"content-type" "text/plain"}
       :body    (with-out-str
                  (clojure.pprint/pprint req))})

onetom15:09:41

that's how I obtained the request map I showed above

Joe Lane15:09:37

To make sure I understand correctly, you're not using the supported integration. Is there still a problem if you use the supported one?

onetom15:09:04

what do you mean by supported integration?

onetom15:09:55

I'm just realizing that probably the Datomic docs are taking about how to integrate a web ion with the traditional RESTful API gateway, not the new "HTTP API". I'm using this "new style" gateway, because it supports JWT authorizer's out of the box, without the need to deploy a lamdba function just for that purpose.

onetom15:09:29

yes, the mentioned request map was observed when my ion-config.edn contained that : integration :api-gateway/proxy setting

Joe Lane15:09:48

I'm not sure that approach is compatible with the current mechanism for making a ring compatible handler. I think you're in uncharted territory, rife with undefined behavior.

onetom15:09:54

Probably... Thanks looking into it!

onetom15:09:38

I will probably just transition to the production topology, though I can foresee issues with the VPC link and NLB in that case, which I have just as little experience with as I have with Cognito and JWT authorizers :)

Joe Lane15:09:22

You will get a raw payload if you use http-direct

Joe Lane15:09:32

it will not be a nice map like above.

onetom15:09:16

Alternatively, I can write a replacement ionizer function for this new "HTTP API" gateway

Joe Lane15:09:08

You're free to do that, but we won't be able to support that if there are issues.

👍 3
jeff.terrell00:09:03

@U086D6TBN try switching the payload format version in the integration config for your API Gateway instance. Saying that based purely on memory so details may be off, but I’m pretty sure I got cut by this exact issue and that was the solution that I eventually found.

🤞 3
onetom02:09:44

@U056QFNM5 thanks a lot for the advice. it worked indeed and I can even see how the JWT authorizer has decoded the token!

onetom02:09:03

so i don't have to fall back to the old, RESTful-style API gateway creation

jeff.terrell03:09:02

You're very welcome. Also keep an eye out for setting cookies. I ran into an issue where the value of my set cookie header in my ring response map was a vector rather than a string. Apparently this is legal in ring, but it didn't work in an Ions context. Again, going from memory here, but I think that was right. A simple middleware to detect such values and only take the first value out of the vector worked.

👍 3
xceno10:10:39

Hi guys, just found this thread because I'm working on the exact same thing right now (trying to deploy an SPA as ion / lambda proxy) I got confused by the mismatch between the Ion tutorial and the API Gateway console, so just to clarify once more: What the Datomic Ion docs are talking about is now called REST API on AWS? And the HTTP API is a new thing, that is not officially supported?

jeff.terrell13:10:44

Yes, REST is the old kind that the docs implicitly refer to. HTTP can work, but it's not what the docs describe specifically.

xceno14:10:30

Understood, thank you!

braai engineer15:09:32

Just gotten bitten for an hour getting 401 Unauthorized for Datomic Pro due to missing XML schema in ~/.m2/settings.xml, which is not mentioned in https://my.datomic.com/account. Previously: https://clojurians-log.clojureverse.org/datomic/2019-01-30/1548890962.888000

jaret15:09:32

Sorry about that, what is missing in https://my.datomic.com/account ? I see the .m2/settings.xml described as:

;; In ~/.m2/settings.xml:
  <!-- ~/.m2/settings.xml (see the Maven server settings docs) -->
  <servers>
    …
    <server>
      <id></id>
      <username>REDACTED</username>
      <password>REDACTED</password>
    </server>
    …
  </servers>

;; In deps.edn:
{:mvn/repos
 {"" {:url ""}}
 :deps
 {com.datomic/datomic-pro {:mvn/version "${VERSION}"}}}

braai engineer15:09:45

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns=""
          xmlns:xsi=""
          ssi:schemaLocation="">

jaret15:09:36

ack! let me talk to Alex so i fully understand and I can update our http://my.datomic.com account to reflect that!

👍 3
braai engineer15:09:04

Also - and this is probably out of scope - but even after fixing settings.xml (and running clj from terminal), IntelliJ Cursive still reported a 401 for deps because it caches the 401 error for a given deps.edn (not sure if this is due to Maven or IntelliJ). Fixed after reordering any two items under :deps, then I could start a REPL.

tekacs20:10:33

I'm now debugging this 401 issue in my own case, where Datomic fails to download consistently on Github Actions for CI purposes but not locally on my machine.

Michael J Dorian18:09:25

Hey! I have a transaction that always puts one entry into datomic, and I'd like to get it to return the entity id of the new entry. I notice that the returned map contains :tx-data, which has the data I need. But I'm not sure how to read the contents of the returned datum, and, indeed, if this is considered bad best practices or not. Help appreciated!

ghadi18:09:12

the returned map also contains :tempids which is a map of tempid -> entity id

Michael J Dorian18:09:18

I'm getting an empty map on that one, do I just need to add a temp-id to the transaction?

ghadi18:09:59

paste your code/input

ghadi18:09:31

if your transaction included tempids, datomic returns the resolved ids after it transacts

Michael J Dorian19:09:34

{:tx-data [#:user{:email "e", :password "q", :name "q", :token "q"}]} ; this query is generated by (make-record :user) and executed
(def q (make-record :user "q" "q" "q" "q"))
(:tempids q)

Michael J Dorian19:09:38

Ah, ok! Just had to add :db/id "nonsense" to my query and now the map gives me {"nonsense" id} !

ghadi19:09:27

if :user/email is a unique attribute in your database, you can use it to lookup entities without entity ids