Fork me on GitHub
#datomic
<
2022-06-08
>
JAtkins08:06:22

Does datomic support the concept of optionality? I have a schema where ?opt-property may or may not be present in

[:find ?always ?opt-property
 :in %
 :where 
 [:always ?always]
 [?always :opt ?opt-property]]
I've tried using or-join to bind ?opt-property to nil when it's not present, but no luck since that's an invalid datalog query.

favila09:06:41

You have to use a sentinel to represent “no value” that isn’t nil

favila09:06:08

Alternatively it sometimes makes sense to push that concern into the pull projection

pieterbreed16:06:17

I am trying to push a datomic-cloud app to a new datomic-cloud stack, ie this is the first push that I'm trying to perform on this code-base/datomic installation.

$ clojure -A:ion-dev '{:op :push :region "eu-west-1"}'
WARNING: Implicit use of clojure.main with options is deprecated, use -M

{:retry 1}
{:retry 2}
{:retry 3}
{:retry 4}
{:retry 5}
{:retry 6}
{:command-failed "{:op :push :region \"eu-west-1\"}",
 :causes
 ({:message
   "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: ENQWWR2D199SMDXB; S3 Extended Request ID: DRHqS+Bx4XxeHjliYGQ6uCgnJ/kKsTXzeH0ky20Ko9ICYGbgeo+DNAXLHlkDx6TJaLPQb/7r1hY=; Proxy: null)",
   :class AmazonS3Exception})}
I've spent the afternoon making sure I've got the latest of everything. Things like datomic cloud list-systems and datomic system list-instances <> work, as in I get results. The CloudFormation stack shows SUCCESS everywhere, I can connect from my local machine to the datomic db etc. I'm not sure how to debug this ion push issue. I have tried with different auth types; currently with an IAM user, with attached policies for AdministratorAccess, datomic-admin-<system> and an additional policy to grant s3:* on everything on the datomic-gui-<guid> bucket... but still getting this S3 permissions error above. How can I debug this?

pieterbreed16:06:30

My IAM creds are being loaded with AWS_* environment variables

Daniel Jomphe18:06:28

FWIR, envars are not supported by the ion-dev tool. Try using aws configure to set your ~/.aws/* files correctly.

pieterbreed18:06:30

Ok, I did not consider that. Thank you, I’ll test tomorrow and revert

Daniel Jomphe18:06:42

I couldn't find again in Datomic Cloud's docs where it's documented that we shouldn't use those. I might have learned that from one of the quick setup videos they published. Couldn't find those quickly either.

pieterbreed18:06:48

I must admit; I’ve been through this process once before shortly after ions was announced. I am struggling more this time around.

Daniel Jomphe18:06:53

Datomic Cloud does indeed assemble together many AWS parts. 🙂 Celebrate every successful step (even though it's often not too hard.) 🙂

Robert A. Randolph19:06:15

https://docs.datomic.com/cloud/ions/ions-reference.html#push lists necessary information for supplying AWS credentials information. We've taken note of the difficulty that you're encountering here.

👍 1
pieterbreed10:06:24

Yeah guys, I don't know... @UEFE6JWG4 Here is my current setup: • I have an IAM user, that has datomic-admin-<system>-<region> policy and AWS-supplied AdiminstratorAccess (from desperation) • Additionally I created a policy that contains access to datomic-releseas-... (after encountering https://clojurians-log.clojureverse.org/datomic/2020-07-05/1594042226.483900) • I've configured this user's credentials with a named profile using aws configureand below will show how/that it works with aws cli. Below is actual output of a shell session:

$ unset AWS_PROFILE

$ aws s3 ls -<guid>
Unable to locate credentials. You can configure credentials by running "aws configure".

$ export AWS_PROFILE=nette-prod
$ aws s3 ls -<guid>

$ aws s3 cp deps.edn   -<guid>/deps.edn
upload: ./deps.edn to -<guid>/deps.edn

$ aws s3 ls -<guid>/
2022-06-09 11:43:50        244 deps.edn

$ aws s3 rm -<guid>/deps.edn
delete: -<guid>/deps.edn

$ clojure -A:ion-dev '{:op :push :creds-profile "nette-prod"}'
WARNING: Implicit use of clojure.main with options is deprecated, use -M

{:retry 1}
{:retry 2}
{:retry 3}
{:retry 4}
{:retry 5}
{:retry 6}
{:command-failed "{:op :push :creds-profile \"nette-prod\"}",
 :causes
 ({:message
   "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: PQMXEV1DW5FMZ8S6; S3 Extended Request ID: 0EGW6n1B8a77BkB8A5rb50RXIrFaKh4qzCXNDzd9++WQYc6HLrUvQnF7Kfg36AMrtGKGv0xb76Y=; Proxy: null)",
   :class AmazonS3Exception})}
• I'm not sure what S3 bucket is being accessed here nor how to debug the permissions for that access. Clearly the user configured using the aws named profile has access to the datomic-code... bucket. • I'm not sure if this operation listing from datomic-releases-... should succeed, I'm not sure if this is the problem or not, but it does not work:
$ aws s3 ls 

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
I'm attaching various things here which might be useful.
$ datomic cloud list-systems
[{"name":"app-20220607",
  "storage-cft-version":"Unknown",
  "topology":"Unknown"}]

$ datomic system describe-groups app-20220607
[{"name":"app-20220607-Compute-<nrs-and-letters>",
  "type":"compute",
  "endpoints":
  [{"type":"client",
    "api-gateway-endpoint":
    "https://<numbers-and-letters>.",
    "api-gateway-id":"<numbers-and-letters>",
    "api-gateway-name":"datomic-app-20220607-client-api"},
   {"type":"http-direct",
    "api-gateway-endpoint":
    "https://<numbers-and-letters>",
    "api-gateway-id":"<numbers-and-letters>",
    "api-gateway-name":"datomic-app-20220607-ions"}],
  "cft-version":"939",
  "cloud-version":"9127"}]

$ clojure -Sdescribe
{:version "1.11.1.1113"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/pieter/.clojure/deps.edn" "deps.edn" ]
 :config-user "/home/pieter/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/pieter/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}

$ cat ~/.clojure/deps.edn
{:aliases {:ion-dev {:deps {com.datomic/ion-dev {:mvn/version "1.0.306"}}
                     :main-opts ["-m" "datomic.ion.dev"]}}
 :mvn/repos {"datomic-cloud" {:url ""}}}

$ cat deps.edn
{:paths     ["src" "resources"]
 :deps      {com.datomic/client-cloud {:mvn/version "1.0.120"}
             com.datomic/ion          {:mvn/version "1.0.59"}}
 :mvn/repos {"datomic-cloud" {:url ""}}}

pieterbreed15:06:32

I suspect my AWS provided best practice Control Tower guardrails are preventing access to s3 outside of eu-west-1…

Daniel Jomphe17:06:11

Looks like opening up a ticket with Datomic Cloud would be a good idea. I'd love to learn afterwards what was missing. I, too, get this:

aws s3 ls 

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
so I suspect it's not the root cause of your issue pushing. We also use a Control Tower (with all its default guardrails) and it doesn't hinder us.

Daniel Jomphe17:06:36

I feel like when I'll learn what's the issue's cause, I'll facepalm and feel like I should have been of better help to you, Pieter.

pieterbreed21:06:17

Just having someone to talk to and run ideas by is immensely helpful, thank you. gratitude-thank-you

pieterbreed22:06:44

I think I'm getting closer to cracking it. The trail of crumbs might be interesting: • this is a new AWS account, I tried to follow best practice, this means using AWS Control Tower to set up a multi-account AWS structure. • One of the "guardrails" they provide and semi-suggest (and sounded cool to me) was to limit access to any AWS resources, within certain regions only. • In our case this meant only AWS resources API calls within eu-west-1 • I believe <s3://datomic-releases-1fc2183a> is in us-east-1, so I've allowed that region in our AWS SCP too. • This has gotten me further than before. ion-dev {:op :push} fails further down the road now.

pieterbreed22:06:47

$ clojure -A:ion-dev '{:op :push :creds-profile "nette-prod"}'
WARNING: Implicit use of clojure.main with options is deprecated, use -M

Downloading: com/datomic/ion-http-direct/1.0.46/ion-http-direct-1.0.46.pom from 
Downloading: com/datomic/ion-lambda-dispatcher/0.9.34/ion-lambda-dispatcher-0.9.34.pom from 
Downloading: com/cognitect/ion-runtime/1.0.20/ion-runtime-1.0.20.pom from 
Downloading: com/cognitect/caster/0.9.42/caster-0.9.42.pom from 
Downloading: com/cognitect/http-endpoint/1.0.101/http-endpoint-1.0.101.pom from 
{:command-failed "{:op :push :creds-profile \"nette-prod\"}",
 :causes
 ({:message
   "Failed to read artifact descriptor for com.datomic:ion-resolver:jar:0.9.17",
   :class ArtifactDescriptorException}
  {:message
   "Could not transfer artifact com.datomic:ion-resolver:pom:0.9.17 from/to datomic-cloud (): Unable to execute HTTP request: Connect to  [] failed: Connect timed out",
   :class ArtifactResolutionException}
  {:message
   "Could not transfer artifact com.datomic:ion-resolver:pom:0.9.17 from/to datomic-cloud (): Unable to execute HTTP request: Connect to  [] failed: Connect timed out",
   :class ArtifactTransferException}
  {:message
   "Unable to execute HTTP request: Connect to  [] failed: Connect timed out",
   :class SdkClientException}
  {:message
   "Connect to  [] failed: Connect timed out",
   :class ConnectTimeoutException}
  {:message "Connect timed out", :class SocketTimeoutException})}
$ clojure -A:ion-dev '{:op :push :creds-profile "nette-prod"}'
WARNING: Implicit use of clojure.main with options is deprecated, use -M

{:retry 1}
{:retry 2}
{:retry 3}
{:retry 4}
{:retry 5}
{:retry 6}
{:command-failed "{:op :push :creds-profile \"nette-prod\"}",
 :causes
 ({:message
   "Unable to execute HTTP request: Connect to  [] failed: Connect timed out",
   :class SdkClientException}
  {:message
   "Connect to  [] failed: Connect timed out",
   :class ConnectTimeoutException}
  {:message "Connect timed out", :class SocketTimeoutException})}

pieterbreed23:06:45

Those last few errors were actually just a misbehaving wifi router. :push worked, :deploy worked. I think I found the well-paved road again...

pieterbreed23:06:42

I've gotten here... {:deploy-status "FAILED", :code-deploy-status "FAILED"} Will continue tomorrow.

pieterbreed07:06:15

{:deploy-status "SUCCEEDED", :code-deploy-status "SUCCEEDED"} I think we can call this "closed" now. Thanks so much for engaging.

Daniel Jomphe10:06:38

🙂 Great! Happy for you, good work!! :thumbsup::skin-tone-3:

Daniel Jomphe10:06:53

BTW it was the first time I saw those

{:retry 1}
{:retry 2}
{:retry 3}
{:retry 4}
{:retry 5}
{:retry 6}
I know they come from cognitect.anomaly's retry strategy for this kind of error condition. Your router config must have been fixed by now, then. :)

Daniel Jomphe18:06:42
replied to a thread:I am trying to push a datomic-cloud app to a new datomic-cloud stack, ie this is the first push that I'm trying to perform on this code-base/datomic installation. $ clojure -A:ion-dev '{:op :push :region "eu-west-1"}' WARNING: Implicit use of clojure.main with options is deprecated, use -M {:retry 1} {:retry 2} {:retry 3} {:retry 4} {:retry 5} {:retry 6} {:command-failed "{:op :push :region \"eu-west-1\"}", :causes ({:message "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: ENQWWR2D199SMDXB; S3 Extended Request ID: DRHqS+Bx4XxeHjliYGQ6uCgnJ/kKsTXzeH0ky20Ko9ICYGbgeo+DNAXLHlkDx6TJaLPQb/7r1hY=; Proxy: null)", :class AmazonS3Exception})} I've spent the afternoon making sure I've got the latest of everything. Things like `datomic cloud list-systems` and `datomic system list-instances &lt;&gt;` work, as in I get results. The CloudFormation stack shows SUCCESS everywhere, I can connect from my local machine to the datomic db etc. I'm not sure how to debug this ion push issue. I have tried with different auth types; currently with an IAM user, with attached policies for `AdministratorAccess`, `datomic-admin-&lt;system&gt;` and an additional policy to grant `s3:*` on everything on the `datomic-gui-&lt;guid&gt;` bucket... but still getting this S3 permissions error above. How can I debug this?

I couldn't find again in Datomic Cloud's docs where it's documented that we shouldn't use those. I might have learned that from one of the quick setup videos they published. Couldn't find those quickly either.