Fork me on GitHub
#aws
<
2015-11-02
>
cfleming04:11:14

I haven’t looked at Kinesis yet

alandipert04:11:04

it's pretty neat

alandipert04:11:10

basically kafka on AWS, if you are familiar

cfleming04:11:48

Only vaguely.

cfleming04:11:39

So would the best thing be to put data right into Kinesis from the API endpoint, and then pull off and only mark as processed when it’s successfully written?

alandipert04:11:20

the cool thing is the various consumers of the kinesis stream you may have maintain their own idea of where in the stream they are

alandipert04:11:55

and if your consumers use the kinesis consumer library to do their receiving, this is handled automatically by the library, which stores things in dynamo

cfleming04:11:55

So if that stores in dynamo, do I actually gain any robustness by using kinesis?

alandipert04:11:55

well, the robustness of aws things is a factor of scale

alandipert04:11:01

considering the scale it can go, it's very robust

alandipert04:11:31

but maybe not as robust as a smaller thing with fewer AWS dependencies

cfleming04:11:53

Yeah, I see.

cfleming04:11:35

It’s hard to tell whether it’s more or less robust than a single server with dodgy monitoring set up by a monkey (me)

cfleming04:11:02

Looks like I can’t shovel data from an API endpoint right into Kinesis, sadly

cfleming04:11:26

If I have to worry about the data consumer going down anyway, I think I’d just put it right in Dynamo

cfleming04:11:44

And then on any error just send a detailed email via something non-AWS (mailgun)

alandipert04:11:59

that sounds legit

cfleming04:11:40

Much as I would like to think that thousands of people will be purchasing Cursive per second, I suspect I’m unlikely to have a legitimate big data problem simple_smile

alandipert04:11:47

maybe immutant is in your ballpark?

alandipert04:11:57

i've looked at it for medium projects, seems like a nice way to arrange things

alandipert04:11:15

that also gives you a nice layer between you and the linode os

cfleming04:11:32

My main goal isn’t gigantic scale, it’s not having to maintain and monitor things.

cfleming04:11:07

I’m still fairly tempted by something bare-bones with just an embedded k/v store on my server, but Lambda seems like a pretty easy option.

cfleming04:11:33

And in theory, it should just stay working once I have it set up.

alandipert04:11:02

one of my favorite things about lambda is it automatically does what most bad programs need done

alandipert04:11:06

on some basis

alandipert04:11:08

which is, be restarted

cfleming04:11:02

Doesn’t it just get its dodgy state serialised?

alandipert04:11:27

especially if it's doing something useful and writing to dynamo

erik_price15:11:48

> My main goal isn’t gigantic scale, it’s not having to maintain and monitor things What about something like GAE?

erik_price15:11:59

(I’ve never actually used GAE)