Fork me on GitHub
#yada
<
2015-11-16
>
stijn12:11:01

@malcolmsparks: nice change for multipart form data

stijn12:11:51

I have a question though: is it intentional that process-request-body only handles multipart/form-data and application/x-www-form-urlencoded?

malcolmsparks12:11:50

no. it should be open

stijn12:11:53

Where should you now coerce the schema for json, edn,...?

malcolmsparks12:11:05

but I've only just reached the point where multipart works so I committed!

malcolmsparks12:11:08

but haven't released yet

stijn12:11:47

i'm closing this pull request, ok? https://github.com/juxt/yada/pull/34

malcolmsparks12:11:04

I had to remove the original body protocol

malcolmsparks12:11:28

yes, the reason I couldn't merge that PR was that I needed to finish multi-part first, because body processing is now asynchronous

malcolmsparks12:11:41

because yada must support large bodies, as well as small ones

malcolmsparks12:11:54

the question is, how to support json/edn etc. again

malcolmsparks12:11:57

and make things open

stijn12:11:57

no, I totally understand

malcolmsparks12:11:12

it should be a simple matter of extracting out the protocol again and making it open for extension

malcolmsparks12:11:23

although obviously json and edn support should be built in

malcolmsparks12:11:48

body coercion is still part of multipart/form-data processing

malcolmsparks12:11:58

so it shouldn't be too much work

malcolmsparks12:11:26

I'll work on this tonight and push a few more commits

stijn12:11:48

i'll stay on my development branch until I can extend body coercion indeed

malcolmsparks12:11:49

I'd like to test that it works with large streams, perhaps your video upload use-case

malcolmsparks12:11:58

yes, forks are good simple_smile

stijn12:11:03

yes, i'm very interested in testing that!

stijn12:11:55

I'll need to see first how you can asynchronously upload data into mongo's gridfs

stijn12:11:07

(if it can be done with the java driver)

malcolmsparks12:11:12

it doesn't make sense for yada to support all these http things, but then fall over at scale - and having bits of yada that require synchronous thread blocking is going to lead to a bottleneck

malcolmsparks12:11:46

re. mongo gridfs - can you send me a link to the API

stijn12:11:46

I think that wouldn't work well with aleph's model right?

stijn12:11:28

it wraps the Java client

stijn12:11:53

I didn't look into detail yet, but saw no async api though

malcolmsparks12:11:41

there's an important protocol in multipart.clj called PartReceiver - the idea is you can hook into yada's body processing to send the individual parts (including partial pieces, if the part is large), to your own 'receiver', that could send these parts on to mongodb

malcolmsparks12:11:50

so it's a pass through

malcolmsparks12:11:17

I can see that we need a similar protocol for non-multipart bodies, where you want raw access to the incoming buffers

stijn12:11:32

yes, in my use case there's no multipart, just raw data in the body

stijn12:11:38

it looks like they're working on it right now simple_smile https://jira.mongodb.org/browse/JAVA-1282

malcolmsparks12:11:47

yes, async may be the future, but it's not always the present!

malcolmsparks13:11:13

perhaps the (far) future is super lightweight threads on the jvm, then we'd be back to synchronous models - that might be a long way off though

imre13:11:44

@malcolmsparks: did you say there is no extension point now for that?

malcolmsparks14:11:17

@imre: hi - yes, but I meant to say that this was a temporary state, and that a new extension point will be added - I've fetched your branches into my local repo and will attempt to bring your json/edn/transit work into the new model

imre14:11:18

@malcolmsparks: great stuff. If you do have trouble, leave them there and I'll update them once the new ext point is out