datomic 2019-09-19 | Slack Archive

Laverne Schrock14:09:17

When I push my Datomic Ions app to a query group running 482-8794 (using version 0.9.234 of com.datomic/ion-dev, I get a non-empty :dependency-conflicts map.

{:deps #:com.cognitect{s3-creds #:mvn{:version "0.1.23"}},
 :doc
 "The :push operation overrode these dependencies to match versions already running in Datomic Cloud. To test locally, add these explicit deps to your deps.edn."}

If I add that to my deps.edn for local testing, I get Could not find artifact com.cognitect:s3-creds:jar:0.1.23 in central (). Does someone at Cognitect need to cut a release?

jaret14:09:01

@lvernschrock would you mind sharing your deps?

Laverne Schrock14:09:45

It's pulls in a fair number of dependencies internal to our company. Let me see if I can trim it down to something you could build a classpath with.

jaret14:09:18

are you including both ion-dev and ion?

jaret14:09:12

and what version of client-cloud.

jaret14:09:36

I think I just need to see those and if you have them aliased or in the main deps.

Laverne Schrock14:09:51

In that case, it looks something like the following:

{:paths     ["src" "resources"]
 :deps      {com.datomic/client-cloud            {:mvn/version "0.8.78"}
             com.datomic/ion                     {:mvn/version "0.9.35"}
             org.clojure/clojure                 {:mvn/version "1.10.0"}
             ;; ... a bunch of other deps ...
             }

 :mvn/repos {"datomic-cloud" {:url ""}
             "central"       {:url ""}
             "clojars"       {:url ""}
             ;; ... internal repo ..
             }
 :aliases   {:tasks {:extra-paths ["tasks"]
                     :extra-deps  {com.datomic/ion-dev              {:mvn/version "0.9.234"}
                                   ;; ... a bunch of other deps ...
                                   }}
             :deployment-overrides
             {:override-deps {com.amazonaws/aws-java-sdk-s3 {:mvn/version "1.11.479"}
                              com.cognitect/transit-clj {:mvn/version "0.8.285"}
                              commons-codec/commons-codec {:mvn/version "1.10"}
                              org.slf4j/slf4j-api {:mvn/version "1.7.14"}
                              org.clojure/core.async {:mvn/version "0.3.442"}
                              com.cognitect/s3-creds {:mvn/version "0.1.23"}}}}}

Then I'd run

clj -Atasks ...

where ... runs a main method that eventually calls

(datomic.ion.dev/push {:repl-cmd "-A:deployment-overrides"})

jaret21:09:33

@lvernschrock >The :ion server-type implements this behavior, connecting as a client when remote, and providing an in-memory implemention of client when running in Datomic Cloud. Because ions are using an in-memory implementation of client it will not use your main dep of client-cloud. If you moved com.datomic/client-cloud {:mvn/version "0.8.78"} to your alias (and I recommend that you do) you would not see the dependency conflict override. It is worth pointing out here that in order to get an accurate report of the overrides section, you have to match the version of ion-deploy with the version of cloud you are running.

jaret21:09:47

so in your deps it should probably look like:

{:paths     ["src" "resources"]
 :deps      {
             com.datomic/ion                     {:mvn/version "0.9.35"}
             org.clojure/clojure                 {:mvn/version "1.10.0"}
             ;; ... a bunch of other deps ...
             }

 :mvn/repos {"datomic-cloud" {:url ""}
             "central"       {:url ""}
             "clojars"       {:url ""}
             ;; ... internal repo ..
             }
 :aliases   {:tasks {:extra-paths ["tasks"]
                     :extra-deps {com.datomic/client-cloud            {:mvn/version "0.8.78"} 
                                          com.datomic/ion-dev              {:mvn/version "0.9.234"}
                                   ;; ... a bunch of other deps ...
                                   }}
             :deployment-overrides
             {:override-deps {com.amazonaws/aws-java-sdk-s3 {:mvn/version "1.11.479"}
                              com.cognitect/transit-clj {:mvn/version "0.8.285"}
                              commons-codec/commons-codec {:mvn/version "1.10"}
                              org.slf4j/slf4j-api {:mvn/version "1.7.14"}
                              org.clojure/core.async {:mvn/version "0.3.442"}
                              com.cognitect/s3-creds {:mvn/version "0.1.23"}}}}}

Laverne Schrock13:09:36

Okay. That makes sense I guess (and it worked as desired when I tested it :thumbsup:). > match the version of ion-deploy with the version of cloud you are running I'm not sure what this means since the version numbers for com.datomic/ion-dev don't seem to line up with com.datomic/ion, the clojure client , or the query group templates.

Brian15:09:59

I am writing an extremely simple query which is returning a hard-to-understand error

[:find ?port
         :where
         [_ :blacklist/port ?port]]

This query works and returns [[443] [80] ...] however I want it to return [443 80 ...] and so I have adjusted my query to this one

[:find [?port ...]
         :where
         [_ :blacklist/port ?port]]

And I get Only find-rel elements are allowed in client find-spec, see http://docs.datomic.com/query.html#grammar I know I've done this successfully on-prem. I'm now working with cloud. I am including [clojure.data.json :as json]. Any ideas?

Brian15:09:54

[datomic.client.api :as d]

Joe Lane15:09:55

cloud doesn't have [:find [?port ...] in it's find-rel grammar

Brian15:09:27

Sigh. Any alternatives or do I just need to deal with this?

Joe Lane15:09:18

use (into [] cat (d/q '[:find ?port :where [_ :blacklist/port ?port]] (d/db (get-conn)))

Brian15:09:16

Thank you!!

Joe Lane15:09:13

NP @brian.rogers, lmk if you have any other questions or cloud hurdles. I've only ever used cloud so I've had the complement of your problem many many times 🙂

Brian15:09:27

I used that binding in my in clause. I didn't expect to be unable to use that in my find clause

Brian15:09:37

Much appreciated @joe.lane =]

Joe Lane15:09:28

Are you using ions or the client api in your application in the cloud vpc?

Brian15:09:48

Hmm well I plan on deploying this with aws code deploy so it will be an ion eventually. I am currently testing it against my aws database. I'm not sure how the two (ion vs client api) differ necessarily. At the end of the day what I need is to deploy this function (making it an ion I believe) and that this function will pull some data from my database in the vpc

Joe Lane15:09:17

In your second sentence is "... my aws database" your datomic cloud database or a different type of existing database (an existing mysql db, for example?)

Brian15:09:28

my Datomic Cloud database*

Joe Lane15:09:38

I think what you will ultimately want is to leverage datomic Ions, It's a blast to work with. Have fun!

❤️ 4

markbastian16:09:16

I am trying to use a protocol to obtain the implementation of the query function for a given datalog implementation (e.g. Datomic, Datomic Cloud, Datascript). The entire implementation for Datomic Cloud is here:

(ns foo.proto
  (:require [datomic.client.api :as d])
  (:import (datomic.client.impl.shared.protocols Db)))

(defprotocol IDatalogQueryable
  (q [this]))

(extend-protocol IDatalogQueryable
  Db
  (q [this] d/q))

When I run this I get the following error:

Syntax error (ClassNotFoundException) compiling at (src/foo/proto.clj:1:1).
datomic.client.impl.shared.protocols.Db

This does work when I use the in-mem datomic api jar or with Datascript. I can do the following to make the class load and become visible:

(ns foo.works
  (:require [datomic.client.api :as d]
            [foo.config :refer [config]]))

(let [client (d/client config)
      conn (d/connect client {:db-name "example-db"})]
  (d/db conn))

(import '(datomic.client.impl.shared.protocols Db))

(defprotocol IDatalogQueryable
  (q [this]))

(extend-protocol IDatalogQueryable
  Db
  (q [this] d/q))

It looks like if I initiate a connection the right classes are loaded and then I am able to extend the protocol. Does anyone have any ideas as to how I might make the first example work? I am assuming there is some sort of dynamic classloading going on that I need to do, but I don't want to have to initiate a test connection just do do that.

Joe Lane17:09:21

I would recommend not using that (let [client... block, you will run into this issue if you ever use that in an ion https://docs.datomic.com/cloud/troubleshooting.html#assert-failed

Joe Lane17:09:26

Are you trying to extend the datalog engine to clojure datatypes with cloud-client like how you currently can with on-prem?

markbastian17:09:51

Yeah, the let form was just done to demonstrate that creating a client will load the needed classes to support the protocol. I would not deploy that. My goal is to achieve what the former snippet is doing. My ultimate goal is to be able to have a set of queries that work across all three of datomic on prem, datomic cloud, and datascript. This isn't a general capability, so the queries I have do work with all three so long as I have the right q function. So, I need the ability to use the right q based on the implementation. Using a simple protocol to dispatch on the db type was what I was hoping for.

markbastian17:09:22

The real problem is that trying to import datomic.client.impl.shared.protocols.Db before creating a client fails.

Joe Lane17:09:31

Can you dispatch on which library is available on the classpath?

markbastian18:09:37

Potentially. The challenge with that is that I would only be able to have one implementation at a time on my classpath.

2019-09-19

Channels