I'm not sure how to debug this, I don't usually do stuff with AWS. Have you tried the regular cognitect aws-api library on the JVM to see if the same issue happens over there?

Perhaps @U0EHU1800 can shed some light on this as well, if he has time or @U7ERLH6JX since he does stuff with AWS too

@U04LJ76K6BY Have you tried in Clojure with https://github.com/cognitect-labs/aws-api and do you get the same error?

seems to work for me with a test bucket and object with awyeah api. tested with the versions mentioned in the readme. could you tell us more about the versions being used, if there is any special things about the key or path etc @U04LJ76K6BY?

there seems to be a redirect happening in your call somehow, not sure how to reproduce that. im suspecting something is up with the path, spaces or something which is confusing the redirect

could be a proxy maybe?

Thank you @U04V15CAJ, @U7ERLH6JX and @U03B2SRNYTY.. @U03B2SRNYTY, I'll try aws-api but behind the scene awyeah-api calls aws-api for S3 related functionalities.. @U7ERLH6JX @U04V15CAJ, I was thinking the same.. it is not that this code didn't work, it did work till yesterday but then I've been getting this issue.. the versions I'm using are the below..

are you using an SSO provider by any chance?

@U04LJ76K6BY This isn't true, awyeah-api doesn't call aws-api behind the scenes, it is a re-implementation, but it does use the endpoint and API data from the regular Clojure lib. Please try JVM + aws-api (cognitect) as has been recommended two times before ;)

@U04V15CAJ,, sorry.. I wanted to edit say I think it uses.. you caught me before that.. sure @U04V15CAJ, I'll try that. @U7ERLH6JX, yes.. but we created an iam user and use that to connect to the bucket

i think that the API keys of the user has expired and its possibly trying to redirect to the SSO landing page

does it work with the AWS CLI? using the same creds?

the key is not expired.. I don't know if token generated by awyeah-api is expired and it is trying to redirect.. if yes, how do I refresh the token? and also, as I mentioned earlier, (-> (aws/invoke s3 {:op :ListBuckets}) <= this one works.. if the key is expired, wont this fail?

right makes sense, can you also make sure the role has the required IAM permissions are there too? can you try this with the AWS CLI?

even before you try with the JVM and aws-api, i suspect something has changed externally

@U04V15CAJ @U03B2SRNYTY, Getting a different error when u used jvm + cognitect aws-api

I see it addressed here.. let me try => https://clojurians-log.clojureverse.org/aws/2020-03-15

@U04V15CAJ, @U03B2SRNYTY, @U7ERLH6JX It worked with jvm + cognitect aws-api and found an issue with the incorrect region.. I tried babashka + awyeah+api and it worked as well.. Thank you so much .. I wish awyeah-api has friendly error as cognitect had!!

incorrect region, I've heard this several times ;)

in my experience too when i hit cryptic errors like this, i try the AWS CLI first, that narrows it down the fastest

when I did as you (@U04V15CAJ) and Daniel (@U03B2SRNYTY) suggested, I could see the real issue.. as I said, I wish awyeah-api returned the same..

will do @U7ERLH6JX.. appreciate this help.. all of you!!

9/10 times it’s one of the 5 billion AWS knobs that’s the real thing, not so much the code 😅

lol.. btw, the weird thing is that the s3 buckets are global.. if that is the case, why would region matter is something I don't understand

buckets are region specific, you need to tell that when you create them for example:

this is to reduce latency of objects depending on where the access is from. the objects dont leave the region they are created in

Thanks again

I think AWS have changed something recently which makes some sdks more sensitive to region. Believe it may have something to do with dualstack endpoints but haven't had a proper chance to dig in yet. Boto3 (python) based stuff can be coerced into working by setting this dualstack endpoint option. (see https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html). Not sure if an equivalent exists in the java sdk and/or any thing in clojure.

There are differences between aws-api and awyeah-api as to how the AWS http requests are generated and sent: awyeah-api uses the java.net.http.HttpClient while aws-api uses one based on the jetty client. I believe there’s at least one legitimate bug in awyeah-api related to this; I just haven’t gotten around to investigating it.

The pieces that awyeah-api uses of aws-api are the individual service definitions, which are essentially just data transformation definitions, not code.

Glad Borkdude's suggestion helped. Didn't see his prior comment when I posted 😅

"it depends" ™️

not possible to say without sharing your project

You are right it might be the load I have in the :init - I had forgotten about that part.

yep, could very well be. You could wrap your init block with (time ..) to see if that's the culprit. Sometimes lazy loading namespaces could help too, using requiring-resolve. Time could be spent loading namespaces using :requires

I suspect it is the :require.

if there are expensive requires in there, replace them as follows:

(let [my-fn (requiring-resolve 'expensive.namespace/my-fn)]
  (my-fn ...))

Thanks

(java.net.URLEncoder/encode unencoded "UTF-8")

oh sorry, I didn't read closely enough, you already had that

Yeah

then I don't understand your question. Please give an example

(URLEncoder/encode "string with spaces" "UTF-8")
=> "string+with+spaces"

From the docs > for converting a String to the application/x-www-form-urlencoded MIME format I'm not sure if anything else is different in that lib...

https://stackoverflow.com/questions/4737841/urlencoder-not-able-to-translate-space-character just use string-replace "+" to "%20"

To what you'd normally expect in a URL. I'm unclear on why space ended up as + when everything else uses %20. From reading up it might be a very old version of the format?

I can do that no worries, but I'm worried there might be other character differences that mean it breaks that my tests don't cover

I guess you can copy/paste this: https://github.com/dakrone/clj-http/blob/e31fba678f8b0f668461ce9f00be458048e0e292/src/clj_http/client.clj#L167-L175

😆 Good shout!