Fork me on GitHub
#datomic
<
2020-03-11
>
onetom03:03:21

If I want to cash some datomic query results, what's the best cache key I could derive from a database? My guess would be (defn cache-key [db] ((juxt :id :basisT) db), but then there are :filt :history :raw attributes too which feel relevant, I'm just not sure what is their exact meaning My expectation would be that d/q and d/pull* and d/datoms calls would return the same results for databases if their cache-key are the same.

favila10:03:51

I think you want at least `

[(:id db) (or (d/as-of-t db) (d/basis-t db)) (d/since-t db) (d/is-history db)]

favila10:03:08

and if (d/is-filtered db) you can’t cache it at all

robert-stuttaford06:03:46

@marshall @jaret just confirming that :db/tupleAttrs requires the target schema to be transacted already, and that if you try to transact the targets with the tupleAttrs at the same time, you get :db.error/invalid-tuple-attrs anomay?

jaret12:03:20

@robert-stuttaford that is correct. You need to transact the schema first.

bamarco17:03:59

Hi, I'm attempting to follow the ion-starter tutorial. When I get to the push section I run into this problem:

bamarco17:03:07

$ clojure -A:ion-dev '{:op :push}'
Downloading: com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.pom from datomic-cloud
Downloading: com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion-dev:jar:0.9.251 in central ()

bamarco17:03:49

It seems to be finding the jar in datomic-cloud, but then fails for some reason

Alex Miller (Clojure team)17:03:05

Are your aws credentials set?

bamarco18:03:04

I'm pretty sure they are. i created a datomic database

Alex Miller (Clojure team)18:03:35

aws sts get-caller-identity

bamarco18:03:23

yes, they are set

Alex Miller (Clojure team)18:03:54

You’re not behind a proxy or anything?

Alex Miller (Clojure team)18:03:04

Well let me ask this, what version of clojure tool are you running? clj -Sdescribe

bamarco18:03:13

not behind a proxy

bamarco18:03:41

{:version "1.10.1.536" :config-files ["/usr/local/Cellar/clojure/1.10.1.536/deps.edn" "/Users/bamarco/.clojure/deps.edn" ] :config-user "/Users/bamarco/.clojure/deps.edn" :config-project "deps.edn" :install-dir "/usr/local/Cellar/clojure/1.10.1.536" :config-dir "/Users/bamarco/.clojure" :cache-dir "/Users/bamarco/.clojure/.cpcache" :force false :repro false :resolve-aliases "" :classpath-aliases "" :jvm-aliases "" :main-aliases "" :all-aliases ""}

bamarco18:03:49

no prob, I appreciate the help

Alex Miller (Clojure team)18:03:54

do you see anything in ~/.m2/repository/com/datomic/ion-dev/0.9.251 ?

Alex Miller (Clojure team)18:03:32

the output you have looks like it downloaded the pom but not the jar, but that's extra weird

bamarco18:03:31

it is an empty folder

Alex Miller (Clojure team)18:03:06

anything in ~/.m2/repository/com/datomic/ion-dev ?

Alex Miller (Clojure team)18:03:21

like ea metadata xml file?

bamarco18:03:53

nope just the empty folder

Alex Miller (Clojure team)18:03:08

I guess those are on the repo, and you see _remote.repositories files in your local repo

bamarco18:03:33

I'm not sure what you mean. where do I check for _remote.repositories?

Alex Miller (Clojure team)18:03:38

does this work aws s3 cp . ?

bamarco18:03:30

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Alex Miller (Clojure team)18:03:01

can you try export AWS_REGION=us-east-1 and try again?

bamarco18:03:57

same error

Alex Miller (Clojure team)18:03:34

that call is effectively what the clj s3 maven transporter is doing to download the file so it is definitely identity related somehow

Alex Miller (Clojure team)18:03:43

that bucket should be public read though to head/get object

Alex Miller (Clojure team)18:03:31

more specific but I'd guess this fails too:

Alex Miller (Clojure team)18:03:37

aws s3api head-object --bucket datomic-releases-1fc2183a --key maven/releases/com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar

bamarco18:03:22

yup same error

Alex Miller (Clojure team)18:03:47

I can repro if my aws credentials are set to bad values

Alex Miller (Clojure team)18:03:49

not sure how you could get bad values but the sts returns you an identity though

Alex Miller (Clojure team)18:03:17

can you try explicitly setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to your iam creds?

bamarco18:03:33

tried explicitly setting and it failed

bamarco18:03:44

curl -x :[port] .[system].[region].
does work

Alex Miller (Clojure team)18:03:22

can you try aws --debug s3api head-object --bucket datomic-releases-1fc2183a --key maven/releases/com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar

Alex Miller (Clojure team)18:03:50

in particular anything useful in the response body part

Alex Miller (Clojure team)18:03:34

I'm guessing not but you never know

Alex Miller (Clojure team)19:03:05

I don't understand why a valid iam user would not be able to access a public object in a bucket. I've pretty much only seen this with region issues but we tried that above.

bamarco19:03:06

DEBUG - Making request for OperationModel(name=HeadObject) with params: {'body': '', 'url': u'https://s3.amazonaws.com/datomic-releases-1fc2183a/maven/releases/com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar', 'headers': {'User-Agent': 'aws-cli/1.18.12 Python/2.7.16 Darwin/19.3.0 botocore/1.15.12'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'signing': {'bucket': u'datomic-releases-1fc2183a'}, 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x10d0816d0>}, 'query_string': {}, 'url_path': u'/datomic-releases-1fc2183a/maven/releases/com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar', 'method': u'HEAD'}

bamarco19:03:53

2020-03-11 14:56:05,098 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 217, in main return command_table[parsed_args.command](remaining, parsed_args) File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 358, in call__ return command_table[parsed_args.operation](remaining, parsed_globals) File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 530, in call__ call_parameters, parsed_globals) File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 650, in invoke client, operation_name, parameters, parsed_globals) File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 662, in makeclient_call **parameters) File "/usr/local/aws/lib/python2.7/site-packages/botocore/client.py", line 316, in apicall return self.makeapi_call(operation_name, kwargs) File "/usr/local/aws/lib/python2.7/site-packages/botocore/client.py", line 626, in makeapi_call raise error_class(parsed_response, operation_name) ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden 2020-03-11 14:56:05,105 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255 An error occurred (403) when calling the HeadObject operation: Forbidden

Alex Miller (Clojure team)19:03:43

seems like I'm using a much newer version of aws+python than you, but request is basically the same. doesn't shed much light though, as it's really an IAM issue.

Alex Miller (Clojure team)19:03:46

I'm not sure what else to try. maybe make different iam creds and see if those work.

Alex Miller (Clojure team)19:03:13

unless the datomic team folks have an idea of something

ghadi19:03:17

when debugging IAM or 403 issues, it is really important to know whether you are using AWS_PROFILE AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY and where your region info comes from: explicit AWS_REGION or something in the profile file

ghadi19:03:42

if you are calling this from EC2, there are other possibilities, but it looks like you're on a mac

bamarco19:03:59

yup on a mac

ghadi19:03:25

and if using AWS_PROFILE, whether your profile is an ordinary profile with credentials, or you're using something like aws-vault or an assume role profile

ghadi19:03:52

aws sts get-caller-identity will tell you who you are

Alex Miller (Clojure team)19:03:55

the region is right in the debug trace above

Alex Miller (Clojure team)19:03:04

and this bucket is public read

ghadi19:03:18

but where your credentials came from is a separate question

Alex Miller (Clojure team)19:03:23

and we tried both profile and keys

ghadi19:03:41

what does aws sts get-caller-identity return?

bamarco19:03:39

{ "Account": "<my-account-#>", "UserId": "<my-user-id>", "Arn": "arn:aws:iam::<a-number>:user/datomic-admin" }

ghadi19:03:17

is datomic-admin the user you expect to be?

ghadi19:03:35

what policies does that user have?

ghadi19:03:14

arn:aws:iam::${ACCOUNT#}:policy/datomic-admin-#{DATOMIC_SYSTEM}-{AWS_REGION} is it just ^ ?

ghadi19:03:56

because if so, that is not sufficient, even for a public-read bucket @alexmiller

ghadi19:03:26

aws s3api head-object --bucket datomic-releases-1fc2183a --key maven/releases/com/datomic/ion-dev/0.9.251/ion-dev-0.9.251.jar

An error occurred (403) when calling the HeadObject operation: Forbidden

ghadi19:03:57

^^ me creating a user + access_key, adding the datomic-admin policy to that user, then trying to download the jar ^^

Alex Miller (Clojure team)19:03:43

well that seems bad if so

bamarco19:03:50

I think this is the problem. This is my first time using aws.

ghadi19:03:00

no worries, this stuff is complex.

ghadi19:03:14

were you following a particular guide to set up datomic access?

ghadi19:03:22

if so, I'd love the link so that we can improve it

bamarco19:03:25

let me find it

ghadi19:03:09

Most developer accounts in aws have a pretty permissive s3:GetObject policy that would allow getting from the datomic-releases-1fc2183a bucket

ghadi19:03:38

thank you

ghadi19:03:40

yes, this allows access to datomic (as you see), but it does not have permissions to download the datomic jars from Maven.

ghadi19:03:44

obviously production user does not need to be an administrator, but for the purposes of this tutorial, yes

bamarco19:03:00

so I should make a dev user with admin permissions?

ghadi19:03:40

no you can keep this user and augment it with the AdministratorAccess policy https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator

bamarco20:03:38

Thanks @ghadi and @alexmiller this solved it.

ghadi20:03:04

no problem -- we will update the docs, thanks for persisting

Brian20:03:28

How do I crack open the #object[datomic.promise ...] return by a (d/transact ...) call? My goal is to check for failure/success. I see the contents of the object in my console but I'm not able to programmatically access it which I'd like to

johnj20:03:12

did you de-reference it?

Brian20:03:16

I tried (deref <obj>) but that errors out with a syntax error. Although it then seems to also print out the correct error

Brian20:03:04

It results in:

Syntax error (Exceptions$IllegalArgumentExceptionInfo) compiling at ... ; syntax error
:db.error/not-an-entity Unable to resolve entity ... ; error I want to grab from obj

marshall20:03:16

@brian.rogers synchronous transact can throw

marshall20:03:41

you’d need to put the deref’d call in a try if you want to grab the exception