Fork me on GitHub
#datomic
<
2018-06-15
>
val_waeselynck09:06:38

Released Datomock v0.2.1. Fixed a bug causing d/transact to throw instead of returning a failed ListenableFuture, which means that you could typically see some errors in development and not see them in production. Thanks @tony.kay for spotting this. https://github.com/vvvvalvalval/datomock

🙂 8
evangeline12:06:44

hello! does anyone have any experience with migrations in datomic cloud? i've been using conformity for on-prem but it seems that the library doesn't support client API 😞

chris_johnson13:06:22

Question about the Ions starter project: should you expect to have to push your ions before you can develop against them locally?

chris_johnson13:06:55

I’m getting an error about datomic/ion-starter.edn not being on the classpath and I see it both in my REPL and in the log output from my Cloud instance:

chris_johnson13:06:02

datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"ClientSPIAnomaly","DatomicClientSpiErrorResponse":{"Status":200,"Body":{"CognitectAnomaliesCategory":"CognitectAnomaliesNotFound","CognitectAnomaliesMessage":"'datomic\/ion-config.edn' is not on the classpath"}},"Type":"Event","Tid":361,"Timestamp":1529069275190}
datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"AdopterCheck","DbId":"c4f19984-b350-4a1e-94de-5368e4b26b12","Type":"Event","Tid":382,"Timestamp":1529069276364}

chris_johnson13:06:07

which makes me think maybe you have to push …first? Maybe you have to push an “empty project” so that the ion-config.edn file exists in the Cloud instance, and then you can develop against your local one? I find this confusing.

marshall13:06:49

@chris_johnson the file is read locally. deploy will upload the necessary files to s3

marshall13:06:10

did you create a datomic/ion-starter.edn file?

chris_johnson13:06:12

hm. that’s confusing but in a different and more-manageable way

chris_johnson13:06:56

I sure did, it’s right next to ion-config-sample.edn and the only difference between the two is that I added my system name

chris_johnson13:06:03

I restarted my REPL too

chris_johnson13:06:32

and deps.edn does have {:paths ["src" "resources"] in it

marshall13:06:33

how are you starting your REPL?

marshall13:06:57

where are you getting that error? in local repl (when you do what)?

chris_johnson13:06:49

well, so I do have one other thing going on which I suppose will end up being the problem somehow, though I don’t see how: I added CIDER to my deps like so:

chris_johnson13:06:51

cider/cider-nrepl {:mvn/version "0.18.0-SNAPSHOT"}
        refactor-nrepl {:mvn/version "2.4.0-SNAPSHOT"}

chris_johnson13:06:42

and I start the REPL with clj and then (require 'cider-nrepl.main) and (cider-nrepl.main/init ["cider.nrepl/cider-middleware"])

chris_johnson13:06:14

and I am C-c C-e eval’ing this: (items-by-type* (d/db (get-connection)) :hat)

marshall13:06:23

are you trying to invoke the ion on a Cloud instance before you’ve done a push?

chris_johnson13:06:00

that was the substance of my initial, poorly-worded question

marshall13:06:18

you would need to push before you can invoke it remotely

marshall13:06:36

push and deploy

chris_johnson13:06:40

so you can do truly local development (e.g., handing a map into a fn) in isolation, but in order to exercise something that uses get-connection you have to push and deploy that first

👍 4
chris_johnson13:06:41

that makes sense now that I know it, but it wasn’t clear to me from the way the starter docs order things. Thanks for clearing it up. As you might guess from my log message above, my Friday Me Time™ today is all about getting an Ion or two working with AppSync and writing it up for others to see. Onward and upward! 😄

marshall13:06:01

Cool. glad that makes sense now

stuarthalloway13:06:23

get-connection is not part of Datomic Cloud, but if it uses :server-type :ion you can dev locally without :push

stuarthalloway13:06:04

also cider et al should not be in your production deps? just under e.g. :dev alias

chris_johnson13:06:37

so, I do in fact have :server-type :ion set up and I’m using the get-connection -> ensure-dataset -> get-client machinery in the starter project

chris_johnson13:06:19

I guess based on my reading of the docs I would have expected to be able to exercise all the ions in the project locally against the Cloud instance over the SOCKS proxy (which is running)

chris_johnson13:06:38

ensure-dataset appears to have transacted schema and data to the Cloud instance

chris_johnson13:06:40

and thanks for the pointer about :dev - this is my first time trying out a tools.deps based project so I took the bull-in-a-china-shop approach and said “oh look, a dependencies array! I’ll just leave these here” 😄

chris_johnson13:06:16

I want to learn Ions now and will go back and learn tools.deps in more depth after I have my blog post written hehe

chris_johnson13:06:03

I do get the same error from the stock clj REPL too:

chris_johnson13:06:05

user=> (in-ns 'datomic.ion.starter)
#namespace[datomic.ion.starter]
datomic.ion.starter=> (items-by-type* (d/db (get-connection)) :hat)
ExceptionInfo 'datomic/ion-config.edn' is not on the classpath  clojure.core/ex-info (core.clj:4739)
datomic.ion.starter=>

stuarthalloway14:06:45

@chris_johnson is that error from the Cloud or the local REPL?

chris_johnson14:06:08

I think the problem is actually right here: > I want to learn Ions now and will go back and learn tools.deps in more depth after I have my blog post written hehe

stuarthalloway14:06:28

what does ( "datomic/ion-config.edn") tell you?

chris_johnson14:06:36

I was running clj without -Rdev and so I’m 99% sure I was running without ion-dev loaded at all

Alex Miller (Clojure team)14:06:45

should -R:dev btw (although -Rdev works right now)

chris_johnson14:06:13

I’m a few seconds away from apologizing for wasting yours and Marshall’s coffee break

stuarthalloway14:06:36

the error message was on point 🙂

chris_johnson14:06:18

hm, well I still get that error so that wasn’t it

chris_johnson14:06:31

It’s being reported by the local REPL but is also showing up in the Cloud logs:

chris_johnson14:06:34

datomic-datomic-cloud-appsync datomic-cloud-appsync-datomic-cloud-appsync-Compute-AKGO1YKWH8U7-i-0a275b66a345e2502-2018-06-09-17-06-06- {"Msg":"ClientSPIAnomaly","DatomicClientSpiErrorResponse":{"Status":200,"Body":{"CognitectAnomaliesCategory":"CognitectAnomaliesNotFound","CognitectAnomaliesMessage":"'datomic\/ion-config.edn' is not on the classpath"}},"Type":"Event","Tid":357,"Timestamp":1529072273749}

chris_johnson14:06:18

to answer your question:

user=> ( "datomic/ion-config.edn")
#object[java.net.URL 0x73a19967 "file:/Users/chris/src/ion-starter/resources/datomic/ion-config.edn"]
user=>

chris_johnson14:06:38

and slurping that does show the contents I would expect to see in the file, including a valid value for :app-name

viesti14:06:24

So now one can do private services with Ions

chris_johnson14:06:33

yeah, that’s going to be Big™ - I’m sure lots of people’s Friday is figuring out how to use that to pull a bunch of their service API surface “inside the wire”

marshall14:06:59

@chris_johnson did you edit the client map in ion/starter.clj ?

marshall14:06:02

if you’re going to do local dev, you’ll need to set up that map

chris_johnson14:06:19

okay, so this appears to be working:

chris_johnson14:06:27

Clojure 1.9.0
(def cfg {:server-type :ion
                       :region "us-east-2"
                       :system "datomic-cloud-appsync"
                       :query-group "datomic-cloud-appsync"
                       :endpoint ""
                       :proxy-port 8182})
#'user/cfg
user=> (require '[datomic.client.api :as d])
nil
user=> (def client (d/client cfg))
#'user/client
user=> (def conn (d/connect client {:db-name "datomic-docs-tutorial"}))
#'user/conn
(defn create-item
  "Transaction fn that creates data to make a new item"
  [db sku size color type]
  [{:inv/sku sku
    :inv/color (keyword color)
    :inv/size (keyword size)
    :inv/type (keyword type)}])
#'user/create-item
user=> (create-item (d/db conn) "test-001" "large" "blue" "shirt")
[#:inv{:sku "test-001", :color :blue, :size :large, :type :shirt}]

chris_johnson14:06:04

(note that this is just mechanically copying into the REPL the very same config map and fn def that exist in the local source file)

stuarthalloway14:06:17

@chris_johnson pretty sure @marshall was right at the beginning

stuarthalloway14:06:41

you cannot call a query function that you haven't deployed

stuarthalloway14:06:04

you can dev and test it directly, e.g.

stuarthalloway14:06:22

(datomic.ion.starter/feature-item? db some-e)

stuarthalloway14:06:43

but not inside a query as items-by-type* does

stuarthalloway14:06:25

we probably could use more pictures describing the dev workflow

chris_johnson14:06:48

right, that makes sense and tracks with the next thing I tried, which was to defn feature-item? in #'user and re-`defn` items-by-type* to use that

chris_johnson14:06:09

which yields ExceptionInfo Unable to resolve symbol: feature-item? in this context clojure.core/ex-info (core.clj:4739)

chris_johnson14:06:07

that all makes perfect sense and for the record I was 100% sure at every point that one of the two of you was right and I was doing something incorrectly 😇

chris_johnson15:06:36

hopefully your efforts to help an under-caffeinated neophyte will persist in the logs long enough to help someone else. Thanks, as always, for being so willing to spend time in here working through issues.

stuarthalloway15:06:27

I am afraid my contribution here was net negative. Sorry @marshall and @chris_johnson

marshall15:06:01

it was also correct, just not for this specific context i think

chris_johnson15:06:42

I agree, I think at least for me there was value in having you both assert the correct things you did and then having to convince myself they were true at the REPL. As I find often to be the case, coming at something with a half-correct mental model and a thoroughly-broken code example, with a couple of willing experts to offer advice, has taught me much more than just having the tutorial work right the first time ever would have.

marshall15:06:40

well. one expert and one pretender.

marshall15:06:14

i’ll let @stuarthalloway interpret that as he sees fit 😉

chris_johnson15:06:47

and indeed, as a final check before deploying anything, reloading starter.clj with this form of items-by-type* does work at the REPL (and in CIDER for that matter):

(defn items-by-type*
  "Returns info about items matching type"
  [db type]
  (d/q '[:find ?sku ?size ?color #_?featured
         :in $ ?type
         :where
         [?e :inv/type ?type]
         [?e :inv/sku ?sku]
         [?e :inv/size ?size]
         [?e :inv/color ?color]
         #_[(datomic.ion.starter/feature-item? $ ?e) ?featured]]
       db type))

chris_johnson15:06:27

Completely unrelated question while you’re (maybe) still reading, @marshall - does the Peer library log anything about its use of memcached? Not the transactor’s CloudWatch metrics but the Peer library. I’m moving a service from Heroku to AWS and it doesn’t seem to be using the shiny new Elasticache cluster I set up for it, even though it was making use of Memcachier in Heroku. I tried turning the datomic logger in our logback.xml up to ”DEBUG” but I still don’t see any telemetry I can use to figure out What I Did Wrong

marshall15:06:47

Yes, the peer lib will report memcache metrics as well

marshall15:06:15

you need to specify the memcached server address as a command line arg

chris_johnson15:06:17

Yes, we do that

chris_johnson15:06:15

It will report metrics? Like, CloudWatch metrics or logging statements?

marshall15:06:30

do you have peer metrics enabled?

marshall15:06:07

you’d need to have a metrics callback enabled

marshall15:06:39

you could use println if you just want them locally in a repl, or you can write (or use a community) wrapper to push them to CloudWatch

marshall15:06:32

i.e. something like this https://gist.github.com/geoff-kruss/4504cdcf7e017d289862ab75fc856720 **I haven’t used that specifically and can’t officially endorse it, but that should give you an idea

chris_johnson15:06:57

No, I don’t believe we have a callback handler set

chris_johnson15:06:31

I was expecting to find stuff in the console logs because I am a bad person who doesn’t read documentation, apparently

chris_johnson15:06:37

I will try that, thanks!

marshall15:06:57

if you have logging enabled (i.e. logback or something) you will likely see them there too

marshall15:06:28

i’d suggest looking at the logs when you start up the app - it should record some details about the datomic config and creating the memcached connection

marshall15:06:44

again, assuming your peer lib logging is enabled

chris_johnson15:06:13

it should be and the fellow who set this up in Heroku in the long-long ago claims that he’s seen useful peer logging in the past by changing the same value I did in our logback.xml but I have not seen with my own eyes proof that this is true

marshall15:06:42

hrm. you should just need to have logback.xml on your classpath and logback in your project deps

chris_johnson15:06:24

yeah, we do - just knowing that it should be logging something at startup is enough for me to chew on

kenny18:06:56

Is there a way to get the datomic-socks-proxy to persist across laptop sleeps?

jaret18:06:36

We had one user who made the following suggestion, I haven’t tested but others have and it seems to work: >I installed autossh instead and hacked the script to use this, and it is now much more stable (and survives sleeps of my laptop). I wonder whether it might be worth having the standard script check for the installation of autossh and if found, use that instead (and maybe print a message to the user if not found, before continuing with the regular ssh client). For anybody interested in my little hack, I just commented out the ssh command at the bottom of the script, and added the autossh one. Like this... #ssh -v -i $PK -CND ${SOCKS_PORT:=8182} ec2-user@${BASTION_IP} autossh -M 0 -o “ServerAliveInterval 5” -o “ServerAliveCountMax 3" -v -i $PK -CND ${SOCKS_PORT:=8182} ec2-user@${BASTION_IP}

jaret18:06:16

Credit to @U9HA101PY. I might copy it over to the forum as I think this is the second or third time I’ve linked it.

jaret18:06:54

as reduce says, https://mosh.org/ would probably work as well