Fork me on GitHub
#datomic
<
2021-05-11
>
tatut04:05:19

in datomic cloud analytics, how do I configure username/password to the trino connector (https://trino.io/docs/current/security/password-file.html) ? the datomic documentation only talks about catalog .properties file and not the overall config.properties

tatut04:05:09

any pointers on trino configuration? should I just ssh into the bastion to do changes directly

Joe Lane04:05:48

It's a bit late, but I'm not clear here what you're trying to do? Datomic Analytics (as of today) still runs presto 348, which is a pre-trino version.

tatut04:05:15

ok, the same config should work there… trino is just a name change

tatut04:05:13

I’m trying to expose an analytics endpoint via load balancer to our customer, and need to configure options for the connector

tatut04:05:04

the datomic-cli analytics sync seems to only copy the catalog and metaschema and doesn’t have a way to do other config changes for presto/trino

Joe Lane04:05:34

Trust me, it's more than just a name change 🙂 Is "our customer" a 3rd party? What options exactly are you trying to configure for the connector?

tatut04:05:13

the username/password authentication and tls proxy config

tatut04:05:18

customer is 3rd party

Joe Lane04:05:49

How many databases will your system have?

tatut04:05:35

2 that need analytics access

Joe Lane04:05:06

I'm going to have to sleep on this one.

Joe Lane04:05:21

Let's reconnect tomorrow?

tatut04:05:42

sure, I think we will try just ssh’ing and modifying the config.properties in our dev test environment and see that happens

Joe Lane04:05:48

Something more useful for me would be a sample configuration / project showing that you can expose a secure presto/trino server with the config you need to your customer, taking datomic cloud out of the picture entirely (just for the sample project).

Joe Lane04:05:07

I'm sure ssh'ing will get it to work once, but it will probably not continue to work upon access gateway restart.

tatut05:05:47

added 2 lines to /opt/presto-config/config.properties.template

http-server.authentication.type=PASSWORD
http-server.process-forwarded=true
and then added password-authenticator.properties and password.db to /opt/presto-data/etc folder… that worked. Don’t know if that survives restart of the gw

jcf13:05:41

Hi all! 👋 Hope everyone is doing well today. Is there anything special I need to do to get at Math/abs in a Datomic client query? I seem to remember things like this just working, but this would have been with the peer API…

[:find ?va
 :where 
 [_ :foo/long ?v]
 [(.doubleValue ?v) ?vd]
 [(Math/abs ?d) ?va]]
When I try to execute my query I get an exception, so maybe I need to declare a dependency in my query. I'll perusing the docs now.
1. Caused by clojure.lang.ExceptionInfo
   Unable to load namespace for java.lang.Math/abs
   #:cognitect.anomalies{:category :cognitect.anomalies/not-found,
                         :message
                         "Unable to load namespace for java.lang.Math/abs"}
               require.clj:   53  datomic.core.require/anomaly!
               require.clj:   51  datomic.core.require/anomaly!
               require.clj:   67  datomic.core.require/default-resolver/fn
               require.clj:   64  datomic.core.require/default-resolver
               require.clj:   57  datomic.core.require/default-resolver
               require.clj:   79  datomic.core.require/resolve!
               require.clj:   74  datomic.core.require/resolve!
               datalog.clj: 1342  datomic.core.datalog/resolve-qualified-fn
               datalog.clj: 1336  datomic.core.datalog/resolve-qualified-fn
                 query.clj:  448  datomic.core.query/resolve-qualified-fns
                 query.clj:  445  datomic.core.query/resolve-qualified-fns
                 query.clj:  465  datomic.core.query/parse-query
                 query.clj:  452  datomic.core.query/parse-query
                 query.clj:  469  datomic.core.query/load-query
                 query.clj:  468  datomic.core.query/load-query
I'm assuming I'll need to add type hints to prevent reflection too; hoping primitive types are all good…

Joe Lane13:05:07

Try typehinting that first, it may not be able to find the right method without it.

jcf13:05:41

@U06FTAZV3 I have a ^long and a ^double hint in my query, and I'm seeing the same exception.

Joe Lane13:05:28

paste it again with these new hints?

jcf13:05:59

Can't type hint a primitive local… I thought that might be a problem.

Joe Lane13:05:28

Show me the query with the hints

jcf13:05:27

'[:find (sum ?va)
  :with ?e
  :where
  [?e :transfer/amount ?v]
  [(.doubleValue ?v) ?vd]
  [(Math/abs ^double ?vd) ?va]]

jcf13:05:56

I get a result when I ditch the use of Math/abs and sum the ?vd.

Joe Lane13:05:10

Can you show that query as well

jcf13:05:51

'[:find (sum ?vd)
  :with ?e
  :where
  [?e :transfer/amount ?v]
  [(.doubleValue ?v) ?vd]]

jcf13:05:06

That gives me back a negative double.

jcf13:05:34

clj-kondo is warning me about reflection. That's a great library!

Joe Lane13:05:36

Now go for the minimal repro.

'[:find ?va
  :where
  [(ground 42.0) ?vd]
  [(Math/abs ^double ?vd) ?va]]

jcf13:05:48

Same exception with your minimal repro.

jcf13:05:02

1. Caused by clojure.lang.ExceptionInfo
   Unable to load namespace for Math/abs
   #:cognitect.anomalies{:category :cognitect.anomalies/not-found,
                         :message "Unable to load namespace for Math/abs"}
               require.clj:   53  datomic.core.require/anomaly!
               require.clj:   51  datomic.core.require/anomaly!

jcf13:05:17

A call to Math/abs works outside of the query, which is what made me wonder if I need to whitelist the Math namespace, but I think everything in java.lang is available by default.

Joe Lane13:05:29

'[:find ?va
  :where
  [(ground 42.0) ?vd]
  [(java.lang.Math/abs ^double ?vd) ?va]]

jcf13:05:05

Adding java.lang doesn't help.

Joe Lane13:05:35

Hmm.. Can you open a support case for this so I can look into it?

jcf13:05:20

Where's the place to open support cases these days? http://support.datomic.com?

Joe Lane13:05:34

Yep, same as always

Joe Lane13:05:19

We've also got this handy format that prevents roundtrips https://docs.datomic.com/cloud/tech-notes/writing-a-problem-report.html

jcf13:05:33

@U0CJ19XAM want me to log an issue for this Zendesk error too? I can't create a password because of some janky iframe stuff from the looks of it.

Joe Lane13:05:32

Yes, that's weird.

Joe Lane13:05:49

Please include your browser details

Joe Lane13:05:42

and if possible a .har file network recording (or the firefox equivalent) of the network requests made.

jcf13:05:05

The support email that gets sent out links to a different doc on what info to provide with support requests, and it's from 2016: https://support.cognitect.com/hc/en-us/articles/215581538-Information-to-provide-with-a-support-request

jcf13:05:48

It doesn't mention providing version numbers, which is probably more helpful with a problem on top of an in-memory database. 🙂

jcf14:05:22

Quick fix for the Zendesk iframe jank is to open the iframe in a new tab, and then submit the form.

jcf14:05:03

I've logged the support request. Thanks, @U0CJ19XAM! 🙇

Yarin Kessler15:05:19

• Hi all. So I was going through the ion-starter tutorial, and ran into the following error at https://docs.datomic.com/cloud/ions/ions-tutorial.html#test-your-connection :

- Downloading: com/datomic/ion/0.9.50/ion-0.9.50.pom from datomic-cloud 
- Downloading: com/datomic/ion/0.9.50/ion-0.9.50.jar from datomic-cloud 
- Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.50 in central ()
◦ Here’s the project’s deps.edn for reference: https://github.com/Datomic/ion-starter/blob/master/deps.edn ◦ I was able to resolve this by adding full S3 access permissions to my IAM Datomic user, based on this tip from https://clojurians-log.clojureverse.org/datomic/2021-03-14/1615742989.291900. However, I’m not clear on why that helped. Why would giving full access to my S3 account help with locating an external jar? I’m completely new to Java/Maven/tools.deps ecosystem so feel free to ELI5. Thanks!

Alex Miller (Clojure team)15:05:00

the ion jars are provided in a Maven repository hosted on s3. while the bucket is public, you must have IAM creds with access to S3 to read it

Yarin Kessler15:05:42

So my Datomic user was set up according to instructions here: https://docs.datomic.com/cloud/getting-started/configure-access.html#authorize-user Is there a reason that setup doesn’t have the necessary S3 creds baked in?

Joe Lane15:05:50

FWIW, you shouldn't need S3 full access.

Yarin Kessler15:05:01

Yea, I figured I don’t need full access, but I don’t know what specifically I do need? Still confused as to why giving access to MY S3 account would affect access to a bucket outside of my account.

Joe Lane15:05:52

The issue that you haven't given your user access to read ANY S3 buckets, even public ones.

Yarin Kessler15:05:15

Say I were to grant universal read access via an AmazonS3ReadOnlyAccess policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}
What that means to me is that I have granted full read access to my buckets. What would differentiate a policy that gave read access to my buckets vs a policy that gave read access to outside public buckets?

Joe Lane15:05:03

You should restrict the resource arn I believe.

Yarin Kessler16:05:36

Tried this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::datomic-releases-1fc2183a/maven/releases/*"
        }
    ]
}
But AWS doesn’t allow it. You can’t set arn to point to outside resource AFAICT. Which means there’s no way to say “You can read S3 public buckets but not my buckets”. Which honestly aligns with my original instinct that external public buckets are public resources and so applying permissions against them makes no sense. So I’m still massively confused.

Pragyan Tripathi20:05:54

I started learning datomic/datalog today. I have following pull query that works:

(d/pull db '[*] [:block/id #uuid 0000000-0000-0000-000]) ;; based on unique
Now I want to another resolver that returns a vector filtered based on block/tags I couldn’t figure out how to write pull query for that:
(d/pull db '[*] [:block/tags :button])
The sample data looks like following:
[{:db/id "block-button-id"
  :block/id (uuid-from-string "block-button-1")
  :block/tags [:button]
  :block/display "Button Block 1"
  :block/description "Button Block 1"
   :block/value 101155069755482}
  {:db/id "block-button-2"
   :block/id (uuid-from-string "block-button-2")
   :block/tags [:button]
   :block/display "Button Block 2"
   :block/description "Button Block 2"
   :block/value 101155069755483}]
Apologies if it is a trivial question, I would appreciate any help in learning it.

Joe Lane21:05:58

Hi @pntripathi9417, I think you're looking for a query, not a pull.

(d/q '[:find (pull ?b [*])
       :where 
       [?b :block/tags :button] db)

👍 4
Pragyan Tripathi03:05:58

Thanks this helps.

💯 4
naomarik23:05:51

Anything exist that's more updated than this? https://github.com/dazld/awesome-datomic