The specific error:

{:command-failed
 "{:op :push :creds-profile \"stt-datomic-administrator\" :region \"us-east-1\"}",
 :causes
 ({:message
   "Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: NADB9CCB20GA92EE; S3 Extended Request ID: /Pc7zz7FmXg/SLX89tm0mS10phhqKV/5aFq4PB0Kk2go4yZckW2z30WUl6GxS4mc3cADoDvIGxg=; Proxy: null)",
   :class AmazonS3Exception})}

And the command: clojure -M:ion-dev '{:op :push :creds-profile "stt-datomic-administrator" :region "us-east-1"}'

Any suggestions? The amount of esoteric effort required to get Datomic Cloud to work with AWS credentials is going to drive me insane.

Do you have anything set for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environmental variables by any chance? I think I remember having to set these to get ions to work

No. ONly in the credentials file under the default profile.

I’ll try with the env vars…

Just another shot in the dark, but I think that error might be different than the "bad credentials" one? So it could be something else besides bad creds? Not at all confident in this though

I definitely remember struggling with similar things but none of the details unfortunately

I think I saw your posts on the forum on this subject, but something made me think your solution didn’t apply.

Now I’m trying with the env vars set and getting a very different error that I have seen before…

{:command-failed
 "{:op :push :creds-profile \"stt-datomic-administrator\" :region \"us-east-1\"}",
 :causes
 ({:message
   "Library org.clojure/alpha.spec has missing :sha in coordinate.",
   :class ExceptionInfo,
   :data
   {:lib org.clojure/alpha.spec,
    :coord
    #:git{:url "",
          :sha "3d32b5e571b98e2930a7b2ed1dd9551bb269375a"}}})}

AWS_ACCESS_KEY_ID=AKIATGBBEZHEMVFNLW3 AWS_SECRET_ACCESS_KEY=Ol+NkIsbAySZmkAbrVsUN0MUpZPJ1fCIaI9YIFb clojure -M:ion-dev '{:op :push :creds-profile "stt-datomic-administrator" :region "us-east-1"}'

[wasn't my forum post] but that error message seems like progress!

(Creds fuzzed for safety)

phew haha

I dunno. I think it’s an earlier blocking error where the built creds provider can’t even read S3 public repos. But that IAM User has AdministratorAccess.

That seems like some issue related to your deps.edn? Maybe the clojure version?

clojure 1.11.1

or you clojure cli version

"Library org.clojure/alpha.spec has missing :sha in coordinate.", does this mean it expects :sha instead of :git/sha, as the earlier versions did?

Also clojure 1.11.1

oh hmmm

So that theory is one I’m considering… in which case I don’t really know the solution…. maybe downgrade clojure?

is there a spec2 entry in your deps.edn?

https://ask.clojure.org/index.php/11376/tools-graph-outdated-version-alpha-doesnt-support-coordinates this seems sort of relevent, it seems like something to do with versioning

That ^ + env var creds “solved” the problem. If anybody from cognitect is reading this, it would dramatically improve the out-of-the-box experience if the documentation were updated. Not an obscure answer in the forums (because who knows what is authoritative and current). Not something in the knowledgebase (same problem). It’s not unreasonable to try to use Ions with current versions of Clojure and AWS tooling but the magic incantations required to make it work gives the impression that Datomic Cloud is abandonware. Better yet, update Datomic cloud to work with current tooling from AWS and … Cognitect.

@U0698L2BU To clarify default profile did not work, but envars did, and what precisely did you have, then update, in your deps.edn?

So I’m still exploring all the credentials issues but for sure you need to us the :sha syntax and not the newer :git/sha syntax. For the credentials, it’s not strictly required to use env vars to convey the credentials but you do need to be wary of confusing the ion-dev credentials provider chain (not documented anywhere that I can find, but probably based on the decrepit AWS Java SDK v1). With great care, it is possible to use a named profile. When I figure out the exact incantation, I’ll post it here.

Thank you for posting all this. I’m trying to get SSO to work just for a non-Ion config. Did you ever get it working?

I got through the aws configure sso process which does indeed go through an SSO authentication and dumps out a profile file. After successfully creating the client, I get this when trying to create a database:

; Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
; Unable to load credentials into profile [profile PowerUserAccess-467579589241]: AWS Access Key ID is not specified.

I did. Things I done: used only the :sha syntax in deps.edn (not :git/sha); install env vars with STS credentials using https://formulae.brew.sh/formula/aws2-wrap. Ensure I’m SSO-authenticated.

Aren’t you amazing. Thank you.

I’m a little discouraged. I’m pretty new at all this. I tried to follow your guidance and get the same error even though I (System/setProperty env-var val) with the output of aws2-wrap --profile <my-profile-name> --export after I’ve logged in aws sso login --profile <my-profile-name>. I can successfully connect from the command line, asw s3 ls --profile <my-profile-name> for example. Can/may/should I post my code and (fuzzed) REPL output? I’ve researched my eyes out. I’ll do more when I can pop them back in.

@U05ML354JLU, are you trying to perform an Ion push from the REPL? I’ve never done that and always just run it from the command line with something like aws2-wrap clojure -M:ion-dev '{:op :push}'

If you post your fuzzed REPL, I might be able provide some guidance, but pushing from the REPL is foreign to me.

Not dealing with Ion yet. Just trying to connect to the database. Ion is my next mountain.

I take it you are trying to get the datomic cIoud client to use its equivalent of the SystemPropertyCredentialProvider (I think that’s the class) from the AWS Java SDK v1?

clj꞉life.db.core꞉> 
#'life.db.core/get-and-set-aws-credentials
clj꞉life.db.core꞉> 
Output: export AWS_ACCESS_KEY_ID=fuzzfuzzfuzz
export AWS_SECRET_ACCESS_KEY=fuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzz
export AWS_SESSION_TOKEN=fuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzzfuzz==
export AWS_DEFAULT_REGION=
us-east-1
nil
clj꞉life.db.core꞉> 
#'life.db.core/client
clj꞉life.db.core꞉> 
; Execution error (ExceptionInfo) at datomic.client.api.async/ares (async.clj:58).
; No AWS profile named '<my-profile>'
clj꞉life.db.core꞉> 

I guess that’s what I’m doing. I’m trying to connect like I did using dev-local. Maybe I’m just going about it the wrong way completely.

In my experience, the most SSO way to connect with the cloud client is to wrap your REPL session with aws2-wrap and count on the client resolving STS credentials from the environment.

The downside, I seem to recall, is that those credentials expire and you’ll need to restart the session.

Trying to inject the credentials into system properties might work, but I seem to recall running into a problem. In any case, I’m not sure of the correct names for those properties. In any case, your REPL session indicates that the client is seeing the AWS_PROFILE env var and trying to resolve its credentials…

Indeed it does expire. For the moment, I’m willing to deal with it. It helps me understand the SSO system.

The Rube Goldberg machine I have constructed actually wraps a custom credentials provider that auto-refreshes STS credentials. It has interfaces to satisfy Datomic (AWS Java SDK v1) and com.cognitect.aws.client.

Yikes. I’m not that good yet. I must be trying to build a workflow that just doesn’t make sense.

I’m not sure of the details, but SSO + Datomic Cloud is not easy or simple or fun.

Truth there.

Layer Ion push/deploy and there is another way to get confused.

Amen

First advice: scrap the JVM properties and just get the ENV vars right -that will work for Ion deploys too (unlike my Rube Goldberg).

aws2-wrap is your friend for “relaying” SSO creds to STS env var creds.

I’m both embarrassed and very grateful. I’ll have to take this super slow. I was just trying to follow the https://docs.datomic.com/cloud/tutorial/client.html#create-client. Pray I don’t wear out my welcome. Anyway… In zsh: aws sso logout, aws sso login --profile <my-profile>, using the --generate option on aws2-wrap builds a credentialsfile. Running aws s3 ls --profile <my-profile> returns what looks like reasonable results for Datomic Cloud resources I set up. Restarting the REPL seems to read those creds and results in the No AWS profile named '<my-profile>' error. I don’t think the JVM properties are even read. Is there really no better-trodden path for this process? I don’t really want to bag SSO given the down-the-road advantages.

Sometime after AWS Java SDK v1, the format changed to remove or add the word “profile” to the profile stanza (I’m pretty sure it was removed)

Ahh.

I’ll edit it and retry.

To make it findable by DC, add that manually. That’s another reason to just set the three AWS vars directly (not indirectly via AWS_PROFILE).

> OMG. The create-database came back true. The --generate creates a [default] profile in credentials. Changing default to <my-profile>does it. > > I owe you more than a lunch. > > Thank you.

I’ve been banging on this issue for so long, I hardly know what to do next. Schema installation and viewing I think via the console — whatever or wherever that is. Time to figure that out.

I’ll check. Thanks.

Again, thank you so much.

We could definitely use a better SSO story here. I am sorry you had to go through all this pain Chip and appreciate Chris for walking you through it.

CC @UEFE6JWG4 ^ This is exactly the story we discussed with Stu a few weeks ago. I'd love to have your thoughts on Monday on it

The trouble is worth the connections with @U0698L2BU, @U1QJACBUM, @UEFE6JWG4. Thanks.

If I can help document something, I’d be happy to — not that I have the whole thing sorted.

For what it’s worth: When the SSO credentials expire, I log out, log back in using the profile I need, regenerate with aws2-wrap --profile <my-profile> --generate then fix the ~/.aws/credentialsfile, replacing [default] with [<my-profile>] and wiping out the old creds. A REPL restart then gives me what I need. Not convenient. I’ll figure out convenient later. The https://github.com/linaro-its/aws2-wrap/tree/master docs section called https://github.com/linaro-its/aws2-wrap/blob/master/README.md#assume-a-role-via-aws-sso does talk about configuring the ~/.aws/config file to use the default stuff --generate creates. I just don’ t know how to deal with the Datomic (or downstream) client reading the profile names in credentials. I’ll either figure it out later or I won’t. I’m just glad SSO works.

My suggestion would be to scrap generating the profile since it’s incompatible with DC without manual touch up. Instead, wrap the start of your REPL (or IDE) with aws2-wrap and let DC pick up the env vars. I hope that restarting your REPL/IDE from the command line (to pick up env vars) is not a big deal. That hassle is what prompted me to write my sso-relay Rube Goldberg credential provider. It works well and avoids the need to restart the REPL, but it’s not even close to conventional, documented or supported.

That makes sense. Thanks. I’m sure there’s a way to restart the VS Code REPL from the command line. I’ll poke around for how. Did you publish your provider somewhere?

I have not. It lacks documentation and has poor test coverage.

Thanks. “Poor test coverage” defines my existence. Another mountain for me to climb.

The general outline is this: • Reify a thing implementing cognitect.aws.credentials/CredentialsProvider (for the awesome Cognitect AWS lib). • The thing reads cached SSO credentials managed by AWS CLI and exchanges them for STS credentials. • Wrap that thing in cognitect’s cached-credentials provider to automatically refresh the credentials when the STS ttl expires. • Reify that thing in an object supporting both com.amazonaws.auth.AWSCredentialsProvider (for Datomic) and cognitect.aws.credentials/CredentialsProvider (essentially pass-through to the cached credentials provider). It’s ugly, convoluted and still has warts: • the means of extracting cached SSO credentials was evolving quickly and did break more than once. I’m not even sure it’s officially sanctioned or documented by AWS. I’ve tried to make my provider behave much like a “regular” credential provider (e.g. respecting AWS_PROFILE to find the sso credentials) but there are some edge cases in containers I suspect. • While the STS credentials can be refreshed for a very long time (a month?), the underlying SSO credentials expire after (max) one hour and you must aws sso login again to build a new provider. So a running provider can run for a month, but you only have an hour to start it (without a new login). Not so bad, but still surprising. • When your SSO credentials have expired and you try to use, there is no clean way to trigger an SSO login from within the provider. On the bright side… • Clojure code using either the cognitect AWS client or Datomic cloud client use exactly the same SSO-derived credentials. This consistency reduces the mental model for me. Thankfully the DC client does allow one to inject a custom credentials provider! • Despite the code being a Rube Goldberg, the external effect is nice: Use standard AWS SSO credentials, AWS CLI tooling and standard (even if poorly documented) AWS CLI boto files. • Once started, the credentials provider has proven to be surprisingly resilient, including surviving laptop sleep, IP address changes, etc. Most days it’s not my running Clojure REPL code that prompts me to SSO login but rather wanting to open the AWS Console (using SSO, of course) or running tests in a separate JVM.

Short of Cognitect overhauling the internals of DC to use the cognitect AWS client, this is the best I can come up with.

Thank you again…again. That’s a weekend project for me. Does Ion require this infrastructure to get SSO to work? (not that I have any idea how to use Ion yet)

No. Ions run in EC2 instances and the standard EC2 instance credentials are used. Instance credentials are part of all* the credential provider chains out-of-the-box. SSO credentials are unavailable and probably inappropriate. Pushing and deploying ions, on the other hand, can use SSO credentials but only with something like aws2-wrap to “relay” or exchange SSO credentials for STS credentials. The ion deploy tooling is probably also based on the old AWS Java SDK v1.

I’m still learning AWS. Thanks for your patience.

d/delete-database and stop referencing any db

and it will be collected

👍