Fork me on GitHub

What would your go-to technologies be if you had to 1) have a user hit an api . There are millions of hits per day. 2) store all available data related to that hit, ie all http request data for up to a few days 3) regularly aggregate stats every few minutes into a reporting layer, such as how many times a given ID is hit and save this aggregate data to a traditional database, 4) look up the id in a key value store of some sort and return the value for the given ID. 5) There is a seperate CRUD app associated with this that shouldn't have performance impacted by the call to that api. Is this a perfect case for something like Onyx or Kafka? Something else entirely? I've working my way though the Onyx tutorial project but I've never really worked with anything like this and am taking a long time to "get it". Any other tools this sound like a good fit for?


@jjttjj This looks like the typical problem Nathan Marz' "lambda architecture" was meant to solve; I'd suggest reading a bit about it to assess and possibly steal ideas


@jjttjj I agree with @gaverhae and, given that, writing hits to a persistent event streaming technology like kakfa and using something like onyx or storm to aggregate them to a database would work. Although lambda architecture has a "speed" layer component that you haven't mentioned a use case for it's worth looking at this and so called Kappa architecture too.

Drew Verlee15:06:12

@jjttjj whats the significance of this other crud app? How is it connected to the api hit? > There is a seperate CRUD app associated with this that shouldn’t have performance impacted by the call to that api.

Drew Verlee15:06:42

If the api getting hits has to service the request then the simplest thing todo would be to have it keep an in memory aggregate that writes to a db. The count might be off for various reasons, but jumping from “i need a count of api hits” to “imploying a lambda architecture” is a huge leap and might not be worth it.

Drew Verlee15:06:32

Also the distribution of hits over the day might play a role in the ideal solution

Drew Verlee15:06:08

However, if you do need to go down the distributed streaming path then i think Onyx is ahead of the flink and spark.

Drew Verlee15:06:50

Anyone have a good software book that they think ppl should read? I perfer ones on technology as opposed to ideology. Im working through data intenstive applications (again) and would hate to think i’m missing out on anything as good as that.


Technology books are lot less useful than ideology books. Technology changes too fast!


I stopped buying technology books


Most of them are useless after a couple of years.


My least favorite publishers are probably PakT and OReilly, but I think their business model is banking on the fast pace of technology. So they can keep selling the same books (for different editions/versions).


I ❤️ O'Reilly, but I only buy their more ideological books, or ones that are pretty much timeless, e.g. ones about UNIX or Vim

Drew Verlee15:06:12

I don’t really care to much about the format. I suppose i should have asked where people go to get high quality information on things their interested in. HN is to open for my tastes. I follow the MorningPaper and really like his material.

Drew Verlee15:06:57

Now i feel like the answer is “Google”

Drew Verlee15:06:56

maybe the better thing would be to follow some respected devs just so i have different perspectives then my own.


Talks and blog posts. From talks I get exposure to something which I then go and research on my own. Also talks that explain the ‘why’ are probably my favorite. I don’t tend to like talk that are full of code and seems like it is just a performance of the presenter’s living coding ability.


I think Michael Drogalis is one of my favorite speakers because he tends to delve into the why of what he does.


without getting to philosophical or abstract.

Drew Verlee15:06:17

I feel very lucky to have been exposed to all the hard work drogalis does at a very early stage in my career


Thanks for the tips everyone


I'm really enjoying "elements of clojure" so far, though it is a little on the idealogical side


The other thing about technological articles is that, usually you have a problem you want to solve, and will discover the technology when you research how to solve it


It's not as useful to keep with the technology if it's not relevant to the software you're writing.


There's all kinds of new technology going on out there


Any good super simple mobile app-maker platforms for restaurants to take orders via phone?