Clojurians Log v2

Is the deploy failing or simply the push?

$clojure -M:test:ion-dev '{:op :push :creds-profile "<my-profile>" :region "us-west-2"}'
{:retry 1}
{:retry 2}
{:retry 3}
{:retry 4}
{:retry 5}
{:retry 6}
{:command-failed
 "{:op :push :creds-profile \"byt-wrought-test\" :region \"us-west-2\"}",
 :causes
 ({:message
   "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: ENXM8X1ZM9QYWCA2; S3 Extended Request ID: oC+OxKbi3KuJ0bqCk2kbpIvKKdJfVTApbuMtJWvvyv4NIhBCl3C1BfyY2IEwlPKb6mt1PFK3i0Q=; Proxy: null)",
   :class AmazonS3Exception})}

The push. Haven't gotten to the deploy yet.

thats saying you dont have the right credentials

I know, but I can't figure out what they need to be, or where they changed from last month.

You need just:

export AWS_ACCESS_KEY_ID="*****************"
export AWS_SECRET_ACCESS_KEY="******************************"
export AWS_SECRET_KEY="*************************************"
export AWS_REGION=us-east-1

for an account in that aws account

we recommend admin, but you can have a more constrained set of permissions, for making sure you can get back to working I'd make sure I used and admin key

The secret key and secret access key are the same key, I just have this leftover legacy of not knowing which one is AWS current requirements so i always set both

I added admin group to the user I had set up, still didn't work. I created a new user in the admin group, still didn't work.

Do you have the right region sourced? Did you try sourcing the credentials directly instead of using profiles?

I'm trying sourcing the credentials right now. But why is that different?

I am just trying to rule out your profile being wrong somehow

It technically shouldn't be different

It seems to be working now. That is strange, because these same profiles work for everything else. Why not for ion push?

Also checking if you can use the aws cli to that account

Ok it worked good

that's how I know the profiles are okay, because aws and datomic cli commands are fine

I can dig into profiles on my side and see if we aren't doing something to support it

But have to pick up kids will look tomorrow

Thanks. I exported the credentials for the new test/admin user I just created and that worked. Then I went back and did it with the original credentials I used back in July. That also worked. I was copying these keys directly from the ~/.aws/credentials file. So I know the credentials are good. I know the associated IAM user is good/has the right permissions. It seems to be something with profiles. I'll put it in my notes for next time.

Update: I was including the :test alias in my command because it included some jvm-options. One of them was "-Daws.profile=<different_profile>" and that was probably the conflict/source of the issue. I don't think any of those jvm-options are really needed, so I'm going to stop using that :test alias in connection with ion push.

One final thought: there are pages and pages of warnings that are printed out, but the push still seems to succeed. Is that normal?

Push worked. But the deploy itself failed. I checked things in CodeDeploy and it failed because the final ValidateService step failed after 5 minutes. Then everything got rolled back.

I decided to upgrade Datomic and that is also failing. Everything is conking out on the final step, the ValidateService step, of CodeDeploy.

so, the new deploy fails at validate service. That means most likely it cannot resolve your dependencies or load application code before codeDeploy Times out. The first thing to check is what changed between deploys. Ion deployment failures can show exceptions in your systems log group. https://docs.datomic.com/operation/monitoring-cloud.html If you are running on a small instance (t3.small) you can consider upgrading your instance size. But its useful to track down if you can find any error or can identify the change set between deploys. If there are deps.edn changes, perhaps its the instance size. The previous deploy is still deploying... Ions that fail roll back to the previous/last known good deploy. When you upgrade you will also deploy that last known good ion. My theory without being able to look specifically at your system. Is that your system was on a larger instance, you downsized at some point, now deploys are failing if you upgrade or if you deploy new code. If that isnt it, please feel free to shoot me an e-mail <mailto:support@cognitect.com|support@cognitect.com> or log a case https://support.cognitect.com/hc/en-us/requests/new. I can then gather specifics like what version you are using, look at your ion-config/ion/ion-dev versions and if you are comfortable sharing read only access your logs and help troubleshoot.

I haven't changed size. I've always been running t3.medium.

I'm not very skilled at using CloudWatch logs. Don't really know what I'm doing over there.

Does ion deploy cause the datomic instance to download a bunch of dependencies, or are the "passed" along, copied from the m2 repo on my machine and bundled up and sent to Datomic? They must be passed over, because the datomic instance wouldn't have access to some of my private github repos. And those were present when I first installed in May, and when I last updated in July, so that wouldn't be the cause today's troubles.

My changes in deps.edn are minimal. datomic/client-cloud went from 1.0.131 to 1.0.137, I updated versions of two private repositories, and added markdonw-clj/1.12.7 as a new dependency.

If I go poking around in the datomic-ions zip files on my machine, does it make sense to compare the "locations.edn" files between my july deploy and today? There I see some cognitect.aws dependencies are new, some jetty-9 depencies were dropped, markdown-clj, a couple of yaml, those couldn't be it. Is there something hanging on those cognitect aws dependencies that are new? Those aren't new dependencies in my deps.edn, but they seem to be new in the ion locations.edn file. I don't know, I'm just grasping at straws here trying to make sense of something that is too complicated for me.

So these are probably two separate issues. Right? Here's how I understand it. Back in July I upgraded Datomic to 1253-9431. When I try to upgrade Datomic today to 1254-9433 it's trying to use that same ion revision (09c62b) from July but failing to deploy. It tries to create 5 or 6 instances but then kills them all because they never complete validation. These are instances with newest datomic version, and my "old" ion revision from July 14. The second issue is I try and do an ion push with a new revision (582c5c) and that also fails to deploy. That is a new ion revision trying to deploy on the "old" version of Datomic. Both of these attempts are failing.

The issues could be related, when Datomic upgrades it deploys whatever running Ion you have. There could be an issue with that ion that is preventing it from validating the application code and its timing out. But yes they are two separate issues. Could you shoot me an e-mail to support? Would it be possible for me to walk you through setting up a read-only AWS account so I can see if I can figure this out by looking at your ion/logs? Without access my next steps would be walking you through version updating in your ion-project or walking back changes one by one until your current deploy works so we can identify what might be causing the failure to validate. Another dependency used to push/deploy is ion-dev. What version do you have of that?

Also happy to have a zoom or google meet and see if we can identify an issue.

disregard request to log a ticket! I see your ticket. Checking it out now.

The solution which got this to work, based on your research and suggestions, @jaret, was to move my tx-fn into a separate namespace, with only the bare minimum :require's needed to support the tx-fn. When it was in the earlier namespace, mixed in with middleware and route handlers, something in there was making datomic not happy and the ion deploy kept failing.

Glad you got it sorted Mike! Please let me know if you go to upgrade your stack and are able to successfully do that.

I will. I’m gonna focus first on testing my mobile app and once I’m happy with that then I’ll try to upgrade the compute stack and I will let you know.