Fork me on GitHub
#datomic
<
2021-07-16
>
robert-stuttaford04:07:51

@jaret is it just me or is there some documentation missing here? https://docs.datomic.com/on-prem/schema/schema-change.html#altering-avet-index i came to read this to confirm something: to add uniqueness to an attr that has existing non-unique values, i can retract the non-unique values and then add uniqueness? i don't have to excise them?

robert-stuttaford14:07:21

just following up on this one, @jaret? šŸ™‚

jaret15:07:43

Yes @U0509NKGK you just need to retract. If there are values present for that attribute, they must be unique in the set of current assertions.

jaret15:07:13

You do not need to and should not excise them.

robert-stuttaford06:07:10

@jaret thanks pal! still think your documentation is missing some key paragraphs there, though šŸ™‚

robert-stuttaford06:07:51

"... so the two will be described together." <page ends>

robert-stuttaford05:07:50

i came to read this to confirm something: to add uniqueness to an attr that has existing non-unique values, i can retract the non-unique values and then add uniqueness? i don't have to excise them?

Ben Hammond09:07:32

I am trying to understand datomic ions :http-direct I am running 884-9095 https://docs.datomic.com/cloud/ions/ions-tutorial.html talks about using a url like

curl https://$(IonApiGatewayEndpoint)/datomic -d :hat
and I dont understand the /datomic part of the route; how does this fit in? Should the http entry point function contain some dispatch logic to process the incoming URL? is this somehow handled implicitly within ApiGateway?

Ben Hammond09:07:57

I get this error message on my experiment

{:cause "No op uniquely specified in x-nano-op header or in URI path"}

Ben Hammond09:07:15

which presumably is telling me that I dont have my dispatch logic set up

Ben Hammond09:07:02

It doesn't seem obvious how I am supposed to set it up though

Ben Hammond10:07:58

when I post to the other api gateway, the one with ions in its name I get

HTTP/1.1 500 Internal Server Error
Date: Fri, 16 Jul 2021 10:01:04 GMT
Content-Type: text/plain
Content-Length: 20
Connection: keep-alive
server: Jetty(9.4.36.v20210114)
apigw-requestid: CjrTIierrPEEPdQ=

Ion execution failed

Response code: 500 (Internal Server Error); Time: 8282ms; Content length: 20 bytes

Ben Hammond10:07:28

its supposed to return a hardcoded

{:status 200
   :headers {"Content-Type" "application/edn"}
   :body {:it :worked}}

Ben Hammond11:07:52

https://docs.datomic.com/cloud/ions/ions-tutorial.html#configure-entry-points > theĀ `:http-direct`Ā section specifies functions that will be callable via HTTP but it looks like you can only set up a single http-direct function per ion-config file; is that correct??

Ben Hammond11:07:25

so I followed https://docs.datomic.com/cloud/troubleshooting.html#http-500 and I see the message

{
    "Msg": "IonHttpDirectException",
    "Ex": {
        "Via": [
            {
                "Type": "java.lang.IllegalArgumentException",
                "Message": "No implementation of method: :->bbuf of protocol: #'datomic.java.io.protocols/ToBbuf found for class: clojure.lang.PersistentArrayMap",
                "At": [
                    "clojure.core$_cache_protocol_fn",
                    "invokeStatic",
                    "core_deftype.clj",
                    583
                ]
            }
        ],
        "Trace": [
            [
                "clojure.core$_cache_protocol_fn",
                "invokeStatic",
                "core_deftype.clj",
                583
            ],
            [
                "clojure.core$_cache_protocol_fn",
                "invoke",
                "core_deftype.clj",
                575
            ],
            [
                "datomic.java.io.protocols$fn__3594$G__3589__3599",
                "invoke",
                "protocols.clj",
                12
            ],
            [
                "$__GT_bbuf",
                "invokeStatic",
                "io.clj",
                20
            ],
            [
                "$__GT_bbuf",
                "invoke",
                "io.clj",
                17
            ],
            [
                "clojure.core$update",
                "invokeStatic",
                "core.clj",
                6185
            ],
            [
                "clojure.core$update",
                "invoke",
                "core.clj",
                6177
            ],
            [
                "datomic.ion.http_direct$encode_body",
                "invokeStatic",
                "http_direct.clj",
                66
            ],
            [
                "datomic.ion.http_direct$encode_body",
                "invoke",
                "http_direct.clj",
                61
            ],
so, is it not sufficient to retunr a plain old persistent map for an http-direct call?

Ben Hammond11:07:18

I am trying' to return

{:status 200
   :headers {"Content-Type" "application/edn"}
   :body {:it :worked}}
maybe its that :body value that is the problem

2
Ben Hammond11:07:02

er, so I think the lesson I learned is that > Lambda ionsĀ https://docs.datomic.com/cloud/ions/ions-reference.html#lambda-ionĀ a String, InputStream, ByteBuffer, or File. also applies to the :body of an http-direct response

Ben Hammond11:07:27

do let me know if that is not correct

souenzzo11:07:21

It's more like a #pedestal question maybe You can see here the supported bodies https://github.com/pedestal/pedestal.ions/blob/master/src/io/pedestal/ions.clj#L21

šŸ‘ 2
Ben Hammond11:07:44

oh is it pedestal running behind the scenes? that is helpful to know

souenzzo11:07:57

Humm... not sure now. When I started with datomic ions, pedestal was the way to go. I will re-read the docs.

Ben Hammond11:07:27

but

(supers (class {:hello :world}))
=>
#{clojure.lang.AFn
  java.lang.Runnable
  java.lang.Iterable
  clojure.lang.IHashEq
  clojure.lang.IMapIterable
  java.util.Map
  clojure.lang.Seqable
  java.io.Serializable
  clojure.lang.IObj
  clojure.lang.IEditableCollection
  clojure.lang.ILookup
  clojure.lang.Associative
  clojure.lang.APersistentMap
  clojure.lang.IKVReduce
  clojure.lang.IPersistentCollection
  clojure.lang.IFn
  java.lang.Object
  clojure.lang.MapEquivalence
  java.util.concurrent.Callable
  clojure.lang.IMeta
  clojure.lang.IPersistentMap
  clojure.lang.Counted}
and
(extend-protocol IonizeBody
...
  clojure.lang.IPersistentCollection
  (default-content-type [_] "application/edn")
  (ionize [o] (pr-str o))
...
which would have been fine

Ben Hammond11:07:23

but the protocol mentioned in the error log was .protocols/ToBbuf

souenzzo11:07:40

@U793EL04V sorry for missleading

Instead of wiring a web handler to a single function, use a routing library such as Compojure or pedestal.
^ In the end of this tutorial there is this phrase You are using the "raw ions handler", that is not related with the code that i sent to you

Ben Hammond11:07:21

I would like to know how you are supposed to support multiple routes from the http-direct ions handler; do I arrange a routing library into that http-direct handler-fn, or is something already built in?

souenzzo12:07:57

ions http handler is a single ring function. You should use compojure, pedestal, or any other ring routing library

souenzzo12:07:25

Using pedestal for example, the same service-map that you will use on ions ring handler, you can use to develop at localhost using jetty

šŸ‘ 2
Ben Hammond13:07:40

That's helpful thanks Enzzo

Bjƶrn Ebbinghaus10:07:19

Why does count for something empty, like in (d/q '[:find (count ?e) . :where [?e :db/ident :something]] db)) return nil and not 0?

favila10:07:56

Aggregation functions (count in this case) are not called when there are no results. The result set is #{} (set of no tuples). No tuples means no aggregation functions called. The . calls ffirst (ie first slot from first tuple) which is nil.

Bjƶrn Ebbinghaus11:07:11

Thank you, for the technical answer. šŸ™‚ But I really think count should return 0 for an empty set, should it not? It feels so... wrongā€¦

robert-stuttaford11:07:52

they ain't gonna change it now šŸ™‚

stuarthalloway12:07:38

so different aggregates should work differently for empty sets?

stuarthalloway12:07:05

@U09R86PA4ā€™s point is not merely technical, it is consistent across all aggregates.

favila13:07:12

@U4VT24ZM3 The semantics of what you want are unclear in the general case. Ignore the . syntax for now, because itā€™s just syntax sugar for an ffirst on the query result. The result set after :where and :with is a tuples (one per slot in the :find), unaggregated. Aggregation takes the unaggregated :find slots and produces another set with the aggregated columns unbound, and then the aggregation functions are called with what the correlated, unbound values would be, and the result of the aggregation is placed into it. Some code to illustrate:

;; :find before aggregation
(sort (d/q '[:find ?l ?n
             :where [?l ?n]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> (["a" 0] ["a" 1] ["a" 2] ["a" 3] ["a" 4] ["b" 0] ["b" 1] ["b" 2] ["b" 3] ["b" 4])
;; :find collecting sets for aggregation
(sort (d/q '[:find ?l (vec ?n)
             :where [?l ?n]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> (["a" [0 1 2 3 4]] ["b" [1 2 3 4 0]])
;; aggregation value substituted in place
(sort (d/q '[:find ?l (count ?n)
             :where [?l ?n]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> (["a" 5] ["b" 5])

favila13:07:32

so aggregation always has the shape of reducing one-or-many values to one value to place into an existing tuple. It canā€™t make new tuples in the result set, only reduce

favila13:07:51

so if (count ?n) returned 0, into what tuple would it be placed?

favila13:07:45

Hereā€™s the same code example as above, but with an empty result set:

;; :find before aggregation
(sort (d/q '[:find ?l ?n
             :where [?l ?n]
             ;; not in result set
             [(ground "Z") ?l]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> ()
;; :find collecting sets for aggregation
(sort (d/q '[:find ?l (vec ?n)
             :where [?l ?n]
             ;; not in result set
             [(ground "Z") ?l]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> ()
;; aggregation value substituted in place
(sort (d/q '[:find ?l (count ?n)
             :where [?l ?n]
             ;; not in result set
             [(ground "Z") ?l]]
           (into
            (mapv vector (repeat "a") (range 5))
            (mapv vector (repeat "b") (range 5)))))
=> ()

favila13:07:16

How could count produce 0 in this case?

favila13:07:58

The only way to get what you want is to make aggregations ā€œspecialā€ somehow when there are no unaggregated items in the :find

kennytilton13:07:04

Background: a year or so ago I was messing with Datomic and installed some its software. Now I am trying to run examples from the Helix react native world. No connection, right? But: $ cd helix-react-native ~/helix-react-native (master) $ make shadow clj -m shadow.cljs.devtools.cli watch dev Error building classpath. Could not find artifact com.datomic:dev-local:jar:0.9.203 in central (https://repo1.maven.org/maven2/) make: * [shadow] Error 1 Strange, but I am no maven/jar/npm expert. I will be happy to uninstall anything Datomic related for now, but just not sure how to track down who has that dependency. Any tips welcome. Thx! šŸ™

Alex Miller (Clojure team)13:07:31

probably something in ~/.clojure/deps.edn

Alex Miller (Clojure team)13:07:45

or ~/.m2/settings.xml

šŸ‘€ 2
kennytilton14:07:02

Bingo. Thx, @alexmiller! I never mess with ~/.clojure/deps.edn, must have been in the Datomic install as a suggestion.

Tatiana14:07:01

Hey! Perhaps someone has already tried IonApiGateway? Can't set up authentication correctly for this connection. I assumed that this was related to the IAM, but I cannot understand how to correctly configure it then so as not to receive 502. And in general, is it correct to use it to connect?

Joe Lane14:07:46

How are you setting up authentication?

Tatiana14:07:43

@U0CJ19XAM Earlier I used access gateway with IAM. I dont understand if I can use IAM here(in IonApiGateway) or I need use JWT or another way to auth.

Joe Lane14:07:11

Can you show an example your client config map?

Joe Lane15:07:20

FWIW, you shouldn't have to configure anything in API Gateway to "set up authentication", that should be handled for you separately from an api gateway authorizer.

Tatiana15:07:47

My example client config map: {:server-type :ion :region "eu-central-1" :system "*" :endpoint "*.http://eu-central-1.amazonaws.com" }

Tatiana15:07:05

Where I can set up this if not in api gateway? I try do curl from terminal with my IAM user but its dont work.

Joe Lane15:07:29

Do you have an aws profile established for your user on your laptop? If so, attach it to :creds-profile "your creds-profile" in the map.

Tatiana15:07:59

Yes I have aws profiles. And how after this I can test this right? I add :creds-profile "your creds-profile"

Joe Lane15:07:16

Wait, you're trying to connect your client to the IonApiGateway, shouldn't you be using ClientApiGatewayEndpoint

Tatiana15:07:19

Wait, Am I need add ClientApiGatewayEndpoint to :endpoint for use curl https://$(IonApiGatewayEndpoint) ? im confusion

Joe Lane15:07:23

Ah. We are talking about totally different things.

Joe Lane15:07:04

I thought you were having problems connecting via the datomic client. Is the datomic client connecting correctly?

Tatiana16:07:23

Connecting via datomic client does not work. I keep getting a "Execution error (ExceptionInfo) at datomic.tools.ops.aws/invoke!" error even though my user has been configured correctly. ClientApiGatewayEndpoint seems to be working when accessing with curl. The thing is, I'm trying to follow an updated version of Datomic cloud documentation that was released couple of days ago. I thought trying https://docs.datomic.com/cloud/ions/ions-reference.html#invoke-web-service might be useful but curl request fails with 502 Bad Gateway error. Not sure if I'm doing it the right way:)

Joe Lane16:07:28

I'm assuming you're using the recently released 884-9095 version of cloud. We create 2 API gateways for you (unless you explicitly chose not to when when deploying the cloudformation templates). The 1st API Gateway is for the datomic client, and it replaces the access-gateway / socks proxy. This is the ClientApiGatewayEndpoint output in cloudformation. You should put that url in your client config map:

{:server-type :ion
Ā :region "eu-central-1"
Ā :system "*****"
Ā :endpoint ""
Ā }
You will NOT be able to curl this endpoint and get a meaningful response (this wouldn't be secure). Separately, the 2nd API Gateway we create is an API Gateway for your ions (which you may or may not have created already, I'm not sure what your exact situation is.) Here are https://docs.datomic.com/cloud/tutorial/client.htmlwalking through how to set up your client and this https://docs.datomic.com/cloud/operation/howto.html#template-outputs describes how to get your ClientApiGatewayEndpoint

Tatiana17:07:16

Joe, thank you so much! It's clearer now. No, I haven't created my ions yet, trying to check and configure everything I can before doing that. So once deployed, my ions will be accessible through this IonApiGatewayEndpoint, right? Do I require any specific AWS IAM policy to have access? Really appreciate your help!

Joe Lane17:07:42

The IonApiGatewayEndpoint is on the internet, so however you want to auth that is up to you. But no, that IonApiGatewayEndpoint does not require any IAM policy. The ClientApiGatewayEndpoint requires the same AWS IAM policy as before this release (to secure access to your database).

Tatiana17:07:10

Got it, thank you so much!

Joe Lane17:07:54

Have fun, reach out if you run into anything else!