Fork me on GitHub
#datomic
<
2020-10-15
>
steveb8n00:10:43

@onetom not sure what problem you are describing here. one useful tool is to watch the deploy in the AWS console in “Code Deploy”. That can provide useful info

onetom00:10:34

thanks! i never looked at that console yet, only the various cloudwatch logs.

steveb8n00:10:28

sure. are you also enabling api-gw level logging (as well as lambda/ion logs)? I have debugged many issues with that level of detail

onetom01:10:30

is there a way to deploy ions from a clojure repl? i tried this:

(datomic.ion.dev/-main
    (pr-str
      {:op :push
       :uname "grace"
       :region "ap-southeast-1"
       :creds-profile "gini-dev"}))
but it quits my repl after executing the operation. it would be nice to just expose the function which gets called with that map provided as a command line argument and return the printed result map, so i can just grab the :deploy-command (or rather just the map itself, which describes the next operation)

joshkh13:10:37

here's what i use, which let's me deploy from the command line to a provided group name via an alias:

$ clj -Adeploy-to-aws some-group-name
https://gist.github.com/joshkh/3455a6905517a814b4623d01925baf0e

solussd00:10:15

There is. 🙂 You can use the functions push and deploy in the namespace datomic.ion.dev Here's some code from my dev ns on a project. I think you can visually extract the important bits and ignore the project-specific ones:

(defn deploy-unrepro-build!
  ([]
   (deploy-unrepro-build! nil))
  ([system-config-overrides]
   (deploy-unrepro-build! system-config-overrides
                          (str "dev-" (java.util.UUID/randomUUID))))
  ([system-config-overrides uname]
   (let [system-config (system/get-config system-config-overrides)]
     (ion/push {:uname uname})
     (ion/deploy {:group (:pawlytics/deployment-group system-config)
                  :uname uname}))))

(defn deploy-rev-build!
  ([rev] (deploy-rev-build! rev nil))
  ([rev system-config-overrides]
   (let [system-config (system/get-config system-config-overrides)]
     (ion/push {:rev rev})
     (ion/deploy {:group (:pawlytics/deployment-group system-config)
                  :rev   rev}))))

(defn deploy-current-rev-build!
  ([]
   (deploy-current-rev-build! nil))
  ([system-config-overrides]
   (deploy-rev-build! (-> (shell/sh "git" "rev-parse" "HEAD")
                          :out
                          str/trim-newline)
                      system-config-overrides)))

solussd00:10:11

warning though: repro builds don't check that the working directory is clean like they do using the clj command.

onetom11:10:55

thanks everyone! I will give these a try!

Erwee08:10:54

Hey, coming from a typical app, if you have heavy read operations, you could spin up a sql read only replica and point your data guys there, safely knowing you wont topple your prod db. How is this generally solved in the on prem postgresql storage datomic world? Something like memcache wont offer much value, its one off huge queries being run.

favila18:10:32

How big is your table in bytes and what is your current read/write to Postgres? The load on storage is purely IO—Postgres is basically used as a dumb key-value store. It seems very unlikely (but not impossible) that this is going to be a problem.

Erwee19:10:23

We were in DDB, but it became too expensive, so we moved to PostgreSQL. DB is 50GB on clean restore, up to 100 GB, as we need to vacuum the storage

Erwee19:10:43

Its a theoretical at this point, postgresql is more than capable, im just curious what the options are, or what other folks are doing for this kind of thing 🙂

favila23:10:03

If the segment churn on your db is low, consider keeping a valcache volume around and remounting it for this big query job

donyorm18:10:49

I'm trying to use dev-local, but I'm getting this error: Unable to load client, make sure com.datomic/client-impl-local is on your classpath. Is there a way to add client-impl-local directly to the classpath? I'm not seeing a maven dependency online anywhere

kenny18:10:52

It’s probably a transitive dep. Are you on the latest Clojure CLI version @U1C03090C ? I know there was an issue with an older Clojure CLI version and dev-local.

donyorm18:10:41

Yeah I have the same issue with the newest CLI. I tried looking for the dependency transitively with -Stree, but it wasn't there

kenny19:10:37

Hmm. Well you should not need to add that dep to your cp. Can you paste your -Stree you used to launch your REPL?

donyorm20:10:48

Here's what I got

kenny21:10:36

Hmm. Not sure. Guessing someone with deeper knowledge of dev-local needs to help here.

donyorm21:10:05

Well thanks for the attempt anyway. I appreciate it!

kenny21:10:06

Only other thing that might help is posting a snippet of what you're doing to get the error.

kenny21:10:08

Fwiw, this is what my dev-local looks like from -Stree

com.datomic/dev-local 0.9.203
  com.google.errorprone/error_prone_annotations 2.3.4
  com.datomic/client-api 0.8.54
  com.google.guava/listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
  com.datomic/client 0.8.111
    com.cognitect/http-client 0.1.105
      org.eclipse.jetty/jetty-client 9.4.27.v20200227
  org.checkerframework/checker-compat-qual 2.5.5
  com.google.guava/failureaccess 1.0.1
  com.google.guava/guava 28.2-android
  com.datomic/client-impl-shared 0.8.80
    com.cognitect/hmac-authn 0.1.195
  com.google.j2objc/j2objc-annotations 1.3
  com.datomic/query-support 0.8.27
  org.fressian/fressian 0.6.5
  com.google.code.findbugs/jsr305 3.0.2
  org.ow2.asm/asm-all 4.2

kenny21:10:50

I don't see any dep in my -Stree for com.datomic/client-impl-local

donyorm21:10:04

Yeah I wonder why it's looking for that

donyorm21:10:36

oh I used the wrong type of :server-type in the config. I did :local instead of :dev-local . That would do it

✔️ 3
Lennart Buit20:10:56

Little data modeling question: Say that I have a category with tags, and these tags are component/many of this category. Now, I’d like to add a composite (tuple) key to this tag entity that says [tagName, category] is unique, but there is no explicit relation from tag -> category. Do I have to reverse this relation / lose the component-ness to add this composite key?

Brandon Olivier22:10:42

I’m trying to do a fulltext search on my Datomic instance, but I get this error:

The following forms do not name predicates or fns: (fulltext)
Anybody know why that might be? I’m following straight from what’s in the docs

Lennart Buit22:10:05

Are you using on prem, or cloud?

Brandon Olivier15:10:41

@UDF11HLKC This is local. It should be the on-prem version, but I’m connecting via the datomic api client.

Lennart Buit15:10:38

Iirc you can’t use fulltext from the client api

Brandon Olivier18:10:26

That was my suspicion, but I couldn't confirm. So I need to convert my application to use the peer server internally?

Brandon Olivier18:10:34

or I guess "should", not "need"

Lennart Buit19:10:30

That depends on what you’d like to achieve. If your goal is to move to the cloud at some point, you may want to consider sticking with the client API and instead using some other store for your full text needs. Here is a thread on the datomic forums about it: https://forum.datomic.com/t/datomic-fulltext-search-equivalent/874/6

Lennart Buit19:10:59

I can’t decide what your architecture should look like, but this is advice I’ve seen before 🙂