Fork me on GitHub
#onyx
<
2017-01-22
>
isaac10:01:32

Is :lifecycle/after-read-batch working on input tasks?

jasonbell22:01:04

Has anyone experienced a problem with the Onyx Amazon S3 that fails when uploading files to paths that don’t exist?

17-01-22 11:24:33 jasebellmacmini.local INFO [onyx.plugin.s3-output:45] - Client request failed CLIENT_REQUEST_FAILED_EVENT
17-01-22 11:24:33 jasebellmacmini.local INFO [onyx.plugin.s3-output:48] - Transfer failed. TRANSFER_FAILED_EVENT
If I upload to a bucket then that works fine along with the naming function. As soon as I want something like /2017-01-01/tweets/twitter-tagname-e02afaacec1f5466d5725fed22982edfcc5bdfe6e1f6a96619264e433ac30.gz then I get the failed transfer message.

jasonbell22:01:32

I can do this from the aws cli with no issues, just wondering if I was missing something.

michaeldrogalis22:01:11

@jasonbell To be clear, are you trying to upload a bucket that doesnt yet exist?

jasonbell22:01:27

@michaeldrogalis For example :s3/bucket “mybucket” but then the name function would be (str “/” (f/unparse date-format (t/now)) “/” (:onyx.core/identity event) “.gz”)) for example

jasonbell22:01:53

the path would, normally, create the folder on the uploading of the event.

jasonbell22:01:17

If I upload to just the bucket with a filename the plugin works fine.

michaeldrogalis22:01:48

@jasonbell I would bet the leading slash is giving it problems.

jasonbell22:01:21

okay, I’ve tried various combinations but possibly not that one! 🙂

jasonbell22:01:46

I’ll give it a go in the morning, 22:50 in the UK. Thanks for the prompt heads up @michaeldrogalis

lucasbradstreet22:01:28

@jasonbell @michaeldrogalis interesting. I don’t think it matters whether the path exists as long as the bucket exists because s3 keys are the whole path

lucasbradstreet22:01:44

what Mike is saying about the leading path is probably correct though

lucasbradstreet22:01:55

Let us know how you go

lucasbradstreet22:01:05

(in the morning :))

jasonbell22:01:14

@michaeldrogalis @lucasbradstreet I’ll let you know after I’ve tried it 🙂 (in the morning) 🙂

jasonbell22:01:21

g’nite y'all

michaeldrogalis22:01:30

Sounds good. G'night