Fork me on GitHub
#yada
<
2016-03-07
>
malcolmsparks10:03:27

@imre I've committed/pushed the file upload code to master. It's in a demo called dev/upload.clj

malcolmsparks10:03:40

["/post"
       (resource
        {:id ::index
         :methods
         {:post
          {:consumes "application/octet-stream"
           :consumer (fn [ctx _ body-stream]
                       (save-to-file
                        ctx body-stream
                        (java.io.File/createTempFile "yada" ".tmp" (io/file "/tmp"))))
           :response (fn [ctx] (format "Thank you, saved upload content to file: %s\n" (:file ctx)))}}})]

malcolmsparks10:03:23

The idea here is that you provide an optional :consumer function that will override the default processing of the request body. The consumer function returns the ctx, perhaps augmented with information about what it did with the incoming request body. In this way, you could build all sort of things (e.g. direct S3 passthrough)

malcolmsparks10:03:40

[yada.consume :refer [save-to-file]]

malcolmsparks10:03:52

s is manifold.stream

malcolmsparks10:03:26

In this example, we batch up 100 8k network buffers to optimize disk writes to a file. Note the s/reduce is the asynchronous aspect of this - and b/to-byte-buffer takes the batch of 100 network buffers and creates a single NIO buffer which is written direct to the file channel.

malcolmsparks10:03:15

I was experimenting with various parameters to tweak performance last night, testing with my 1Gbyte test file. Sometimes I managed to get that uploaded through yada into /tmp in ~1second, which is extremely fast. However, I've been getting some OutOfMemory errors in Netty which indicate that there's something not right. I suspected the problem was not releasing the Netty buffers properly but after quite a bit of investigation I'm not 100% sure now. There's no indication from Netty's resource leak detection system either.

malcolmsparks10:03:53

This could be a bug/issue remaining in Netty, which is still a new version (4.1.0 RC3).

malcolmsparks10:03:17

maybe @ztellman could see something here I'm doing wrong

malcolmsparks10:03:57

but the overall reliability of handling this level of throughput was disappointing.

imre10:03:56

thanks Malcolm

imre10:03:09

it looks impressive

imre10:03:03

so if I understand correctly you can now supply :consumer which will be run instead of process-request-body right?

malcolmsparks10:03:26

yes - actually :consumer is run by process-request-body, if it exists

imre10:03:35

I meant the multimethod in yada.request-body

thomas15:03:35

@malcolmsparks: you are doing a 1Gbyte file in one second? that is very impressive indeed!!!