Released Datomock v0.2.1. Fixed a bug causing d/transact to throw instead of returning a failed ListenableFuture, which means that you could typically see some errors in development and not see them in production. Thanks @tony.kay for spotting this. https://github.com/vvvvalvalval/datomock

hello! does anyone have any experience with migrations in datomic cloud? i've been using conformity for on-prem but it seems that the library doesn't support client API 😞

Question about the Ions starter project: should you expect to have to push your ions before you can develop against them locally?

I’m getting an error about datomic/ion-starter.edn not being on the classpath and I see it both in my REPL and in the log output from my Cloud instance:

datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"ClientSPIAnomaly","DatomicClientSpiErrorResponse":{"Status":200,"Body":{"CognitectAnomaliesCategory":"CognitectAnomaliesNotFound","CognitectAnomaliesMessage":"'datomic\/ion-config.edn' is not on the classpath"}},"Type":"Event","Tid":361,"Timestamp":1529069275190}
datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"AdopterCheck","DbId":"c4f19984-b350-4a1e-94de-5368e4b26b12","Type":"Event","Tid":382,"Timestamp":1529069276364}

which makes me think maybe you have to push …first? Maybe you have to push an “empty project” so that the ion-config.edn file exists in the Cloud instance, and then you can develop against your local one? I find this confusing.

@chris_johnson the file is read locally. deploy will upload the necessary files to s3

did you create a datomic/ion-starter.edn file?

hm. that’s confusing but in a different and more-manageable way

https://docs.datomic.com/cloud/ions/ions-tutorial.html#sec-2-1

I sure did, it’s right next to ion-config-sample.edn and the only difference between the two is that I added my system name

I restarted my REPL too

and deps.edn does have {:paths ["src" "resources"] in it

how are you starting your REPL?

clj

where are you getting that error? in local repl (when you do what)?

well, so I do have one other thing going on which I suppose will end up being the problem somehow, though I don’t see how: I added CIDER to my deps like so:

cider/cider-nrepl {:mvn/version "0.18.0-SNAPSHOT"}
        refactor-nrepl {:mvn/version "2.4.0-SNAPSHOT"}

and I start the REPL with clj and then (require 'cider-nrepl.main) and (cider-nrepl.main/init ["cider.nrepl/cider-middleware"])

and I am C-c C-e eval’ing this: (items-by-type* (d/db (get-connection)) :hat)

are you trying to invoke the ion on a Cloud instance before you’ve done a push?

indeed

that was the substance of my initial, poorly-worded question

you would need to push before you can invoke it remotely

push and deploy

so you can do truly local development (e.g., handing a map into a fn) in isolation, but in order to exercise something that uses get-connection you have to push and deploy that first

that makes sense now that I know it, but it wasn’t clear to me from the way the starter docs order things. Thanks for clearing it up. As you might guess from my log message above, my Friday Me Time™ today is all about getting an Ion or two working with AppSync and writing it up for others to see. Onward and upward! 😄

Cool. glad that makes sense now

@chris_johnson not exactly -- see https://docs.datomic.com/cloud/ions/ions-reference.html#server-type-ion

get-connection is not part of Datomic Cloud, but if it uses :server-type :ion you can dev locally without :push

also cider et al should not be in your production deps? just under e.g. :dev alias

so, I do in fact have :server-type :ion set up and I’m using the get-connection -> ensure-dataset -> get-client machinery in the starter project

I guess based on my reading of the docs I would have expected to be able to exercise all the ions in the project locally against the Cloud instance over the SOCKS proxy (which is running)

ensure-dataset appears to have transacted schema and data to the Cloud instance

and thanks for the pointer about :dev - this is my first time trying out a tools.deps based project so I took the bull-in-a-china-shop approach and said “oh look, a dependencies array! I’ll just leave these here” 😄

I want to learn Ions now and will go back and learn tools.deps in more depth after I have my blog post written hehe

I do get the same error from the stock clj REPL too:

user=> (in-ns 'datomic.ion.starter)
#namespace[datomic.ion.starter]
datomic.ion.starter=> (items-by-type* (d/db (get-connection)) :hat)
ExceptionInfo 'datomic/ion-config.edn' is not on the classpath  clojure.core/ex-info (core.clj:4739)
datomic.ion.starter=>

oh

haha, yes

@chris_johnson is that error from the Cloud or the local REPL?

I think the problem is actually right here: > I want to learn Ions now and will go back and learn tools.deps in more depth after I have my blog post written hehe

what does ( "datomic/ion-config.edn") tell you?

I was running clj without -Rdev and so I’m 99% sure I was running without ion-dev loaded at all

I’m a few seconds away from apologizing for wasting yours and Marshall’s coffee break

the error message was on point 🙂

hm, well I still get that error so that wasn’t it

It’s being reported by the local REPL but is also showing up in the Cloud logs:

datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"ClientSPIAnomaly","DatomicClientSpiErrorResponse":{"Status":200,"Body":{"CognitectAnomaliesCategory":"CognitectAnomaliesNotFound","CognitectAnomaliesMessage":"'datomic\/ion-config.edn' is not on the classpath"}},"Type":"Event","Tid":357,"Timestamp":1529072273749}

to answer your question:

user=> ( "datomic/ion-config.edn")
#object[java.net.URL 0x73a19967 "file:/Users/chris/src/ion-starter/resources/datomic/ion-config.edn"]
user=>

and slurping that does show the contents I would expect to see in the file, including a valid value for :app-name

https://aws.amazon.com/blogs/compute/introducing-amazon-api-gateway-private-endpoints/

So now one can do private services with Ions

yeah, that’s going to be Big™ - I’m sure lots of people’s Friday is figuring out how to use that to pull a bunch of their service API surface “inside the wire”

@chris_johnson did you edit the client map in ion/starter.clj ?

https://github.com/Datomic/ion-starter/blob/master/src/datomic/ion/starter.clj#L14

if you’re going to do local dev, you’ll need to set up that map

I did do that

okay, so this appears to be working:

Clojure 1.9.0
(def cfg {:server-type :ion
                       :region "us-east-2"
                       :system "datomic-cloud-appsync"
                       :query-group "datomic-cloud-appsync"
                       :endpoint ""
                       :proxy-port 8182})
#'user/cfg
user=> (require '[datomic.client.api :as d])
nil
user=> (def client (d/client cfg))
#'user/client
user=> (def conn (d/connect client {:db-name "datomic-docs-tutorial"}))
#'user/conn
(defn create-item
  "Transaction fn that creates data to make a new item"
  [db sku size color type]
  [{:inv/sku sku
    :inv/color (keyword color)
    :inv/size (keyword size)
    :inv/type (keyword type)}])
#'user/create-item
user=> (create-item (d/db conn) "test-001" "large" "blue" "shirt")
[#:inv{:sku "test-001", :color :blue, :size :large, :type :shirt}]

(note that this is just mechanically copying into the REPL the very same config map and fn def that exist in the local source file)

@chris_johnson pretty sure @marshall was right at the beginning

you cannot call a query function that you haven't deployed

okay

you can dev and test it directly, e.g.

(datomic.ion.starter/feature-item? db some-e)

but not inside a query as items-by-type* does

we probably could use more pictures describing the dev workflow

right, that makes sense and tracks with the next thing I tried, which was to defn feature-item? in #'user and re-`defn` items-by-type* to use that

which yields ExceptionInfo Unable to resolve symbol: feature-item? in this context clojure.core/ex-info (core.clj:4739)

that all makes perfect sense and for the record I was 100% sure at every point that one of the two of you was right and I was doing something incorrectly 😇

hopefully your efforts to help an under-caffeinated neophyte will persist in the logs long enough to help someone else. Thanks, as always, for being so willing to spend time in here working through issues.

I am afraid my contribution here was net negative. Sorry @marshall and @chris_johnson

nah

it was also correct, just not for this specific context i think

I agree, I think at least for me there was value in having you both assert the correct things you did and then having to convince myself they were true at the REPL. As I find often to be the case, coming at something with a half-correct mental model and a thoroughly-broken code example, with a couple of willing experts to offer advice, has taught me much more than just having the tutorial work right the first time ever would have.

well. one expert and one pretender.

i’ll let @stuarthalloway interpret that as he sees fit 😉

and indeed, as a final check before deploying anything, reloading starter.clj with this form of items-by-type* does work at the REPL (and in CIDER for that matter):

(defn items-by-type*
  "Returns info about items matching type"
  [db type]
  (d/q '[:find ?sku ?size ?color #_?featured
         :in $ ?type
         :where
         [?e :inv/type ?type]
         [?e :inv/sku ?sku]
         [?e :inv/size ?size]
         [?e :inv/color ?color]
         #_[(datomic.ion.starter/feature-item? $ ?e) ?featured]]
       db type))

💯

Completely unrelated question while you’re (maybe) still reading, @marshall - does the Peer library log anything about its use of memcached? Not the transactor’s CloudWatch metrics but the Peer library. I’m moving a service from Heroku to AWS and it doesn’t seem to be using the shiny new Elasticache cluster I set up for it, even though it was making use of Memcachier in Heroku. I tried turning the datomic logger in our logback.xml up to ”DEBUG” but I still don’t see any telemetry I can use to figure out What I Did Wrong

Yes, the peer lib will report memcache metrics as well

you need to specify the memcached server address as a command line arg

to the jvm

Yes, we do that

It will report metrics? Like, CloudWatch metrics or logging statements?

do you have peer metrics enabled?

https://docs.datomic.com/on-prem/monitoring.html#register-callback

you’d need to have a metrics callback enabled

you could use println if you just want them locally in a repl, or you can write (or use a community) wrapper to push them to CloudWatch

i.e. something like this https://gist.github.com/geoff-kruss/4504cdcf7e017d289862ab75fc856720 **I haven’t used that specifically and can’t officially endorse it, but that should give you an idea

Oh ho

No, I don’t believe we have a callback handler set

I was expecting to find stuff in the console logs because I am a bad person who doesn’t read documentation, apparently

I will try that, thanks!

if you have logging enabled (i.e. logback or something) you will likely see them there too

i’d suggest looking at the logs when you start up the app - it should record some details about the datomic config and creating the memcached connection

again, assuming your peer lib logging is enabled

https://docs.datomic.com/on-prem/configuring-logging.html#peer-logging

it should be and the fellow who set this up in Heroku in the long-long ago claims that he’s seen useful peer logging in the past by changing the same value I did in our logback.xml but I have not seen with my own eyes proof that this is true

hrm. you should just need to have logback.xml on your classpath and logback in your project deps

https://docs.datomic.com/on-prem/configuring-logging.html#leiningen-logback

yeah, we do - just knowing that it should be logging something at startup is enough for me to chew on

Is there a way to get the datomic-socks-proxy to persist across laptop sleeps?

mosh