I started out creating a question here because I thought it was a problem with the Datomic Maven repo but I can download the JAR with aws s3 cp just fine. http://ask.datomic.com/index.php/546/could-not-find-artifact-com-datomic-ion-jar-0-9-48-in-central

This works:

aws s3 cp  .

$ clojure -h | head
Version: 1.10.1.739

It works on my machine. 😄

do you have AWS creds set?

On my machine, yep. Defaults and profiles. In the Docker container I had AWS_ACCESS_KEY_ID and secret exported at build time, but have since moved that so the creds are only there at runtime. Inside the GitHub Action I've not exported anything or configured AWS in any way.

Are you thinking my AWS creds are making things work on this host machine?

I read that for Maven repos on S3 there's some AWS API interaction required to get the region. I guess that could be failing.

yes, it needs to determine the bucket location

You can stick a ?region=... on the end of the :mvn/repo URL I think. I could try that.

I don't know which region the Datomic Cloud bucket is in but I should be able to find out.

it's us-east-1

I'll try that now. Thanks Mr. Miller. Uno momento.

Building a Docker container, and running a job (?) via GitHub Actions.

No joy with GitHub:

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: joda-time/joda-time/2.10/joda-time-2.10.jar from central
Downloading: commons-codec/commons-codec/1.15/commons-codec-1.15.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()

Docker sits here for a few seconds and then continues on its way:

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: tigris/tigris/0.1.2/tigris-0.1.2.jar from clojars
Downloading: javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar from central
Downloading: com/cognitect/aws/appsync/809.2.784.0/appsync-809.2.784.0.jar from central
Downloading: com/amazonaws/aws-java-sdk-s3/1.11.210/aws-java-sdk-s3-1.11.210.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()
ERROR: Service 'clojure' failed to build : The command '/bin/sh -c clojure -Srepro -P -A:dev:test:cider-clj' returned a non-zero code: 1

I'm not sure why tools.deps is downloading things twice.

Should there be two lines "Downloading" com.datomic/ion ?

yes, pom and jar

Ahh, that makes sense.

Hmm. I'm still stumped in that case. :thinking_face:

Okay, I can reproduce locally.

mv ~/.aws{,-ignore}
mv ~/.m2/repository/com/datomic/ion{,-ignore}

error in process sentinel: Could not start nREPL server: Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true
DEPRECATED: Libs must be qualified, change compliment => compliment/compliment 
DEPRECATED: Libs must be qualified, change nrepl => nrepl/nrepl 
DEPRECATED: Libs must be qualified, change refactor-nrepl => refactor-nrepl/refactor-nrepl 
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud
Downloading: org/clojure/clojure/maven-metadata.xml from datomic-cloud
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()

Interestingly there's nothing in my ~/.m2/repository/com/datomic/ion/0.9.48 directory…

I can see two .part files show up momentarily, and then they disappear after I'm told classpath construction failed.

it has to do with the aws creds of whatever user you're using

it does not have iam access to s3 ops or something like that

So to pull down the Ion dependency I need to configure AWS and ensure I have at least some S3 permissions?

you need to be using aws user credentials that don't prevent s3 use for HeadObject, GetObject, and maybe GetBucketLocation

Okay, I'll create an IAM policy that grants that access. 👍 I can test that pretty quickly with Docker. GitHub Actions will take a little longer. 🙂

Dos momentos.

the objects you're going after definitely still exist and are public in the bucket, the issue here is around IAM on the accessing user

@alexmiller I see you've answered my question over on http://ask.datomic.com. I've added the following policy and I'm seeing the same error:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::datomic-releases-1fc2183a/*"
        }
    ]
}

sorry I'm not an IAM guru

is there some kind of simulator or something that can answer whether this user has access to some resource?

I've added the credentials to my local machine and I'm trying out the operations using the AWS CLI.

Copying the JAR worked. Just checking I can fetch the bucket location.

Okay, so I'm not allowed to get the bucket location. That's not surprising though as it's not my bucket. I shouldn't be able to grant myself permission to interact with buckets owned by other people.

$ aws --profile REDACTED s3api get-bucket-location --bucket datomic-releases-1fc2183a

An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied

$ aws --profile REDACTED s3 cp  .

download:  to ./ion-0.9.48.jar

this bucket has a policy that GetBucketLocation is open to *

you may be hitting something region related though - what's the default region for this user?

buckets are inherently region-specific

this is the case where adding the ?region=us-east-1 suffix to the url should help I think

This user defaults to eu-west-2. I tried the suffix on the repo URL but no dice.

And massive thanks for your help on this, @alexmiller. :man-bowing:

can you try setting your default region creds to us-east-1?

either via the region in your credentials file or setting AWS_REGION

Coming right up! Thanks! 🙂

@alexmiller I think I've found the right combination of permissions and Docker configuration to get things working! 💥 The IAM policy I shared previously should do the trick. There were two things I had to do to get Docker working: 1. Use COPY to add an AWS directory to the container at the beginning of the build; use of environment variables is frowned up for security reasons, and is hard to get right with enviroment: and env_file: not working at build time. If you want env vars at build time you need to add ARGs to your Dockerfile and then you can set args: within your build context. 🙂 2. Replace use of the new -P option and invoke clojure -Srepro -A:dev:test:etc -e 'nil'. I mount the same ~/.aws volume at runtime but could swap out new credentials if necessary (which could make sense if you want different access at build vs. runtime). I've got a running container with all the Datomic goodness available including connecting the SOCKS proxy up to a running system. I can connect Cider from Emacs and get all the code reloading goodness you'd expect. I've created a client and I can list databases in the Solo topology running on AWS. I have to say another massive thank you for all your help with this! Thank you! And merry Christmas! 🎅 🎄

well I'm glad it's working. I'm curious why you needed the alternate clojure command, seems like if those aliases are relevant you could use clojure -Srepro -A:dev:test:etc -P

I chalked the -P up to using a newer version of tools.deps on my host machine to what's in the most recent Docker image for Clojure. I might have just gotten the order of the arguments wrong in that case. 🤷

yeah, that version could be lagging

but if you're missing -P that's a good workaround

So you basically went the interdeps-like route?

There were things I didn't like about how interdeps worked so I did not go back and look at it when I rebuilt our stuff today.

I figured you didn't want to take on some dep, that's why I added the -like suffix

We ended up with <subproject>/<subproject>-deps.edn for the templates and a <subproject>/<subproject>-deps.md5 file for the checksums so we can regen on-demand automatically in our build script.

The checksum for each subproject includes the checksum for the master file as well, so we regenerate any <subproject>/deps.edn file if either the master template or the subproject template has changed. We have not yet tackled the issue of transitive :local/root dependencies (we probably won't, unless it actually bites us).

I'll probably roll my own solution for this too, like I did in the boot era, if this remains unsupported.

Do you actually parse the EDN and merge yourself, or do you use simple string templating

Merge the EDN. I lift the :defaults aliases out of the master template and pull it's :override-deps out and then update the project template deps with those overrides.

yeah, easy enough.