Fork me on GitHub
#aws
<
2020-10-15
>
orestis06:10:07

Any good/bad experiences with AWS elasticsearch? The managed offering, I mean.

lukasz13:10:39

@orestis the price 👎 but it's pretty hands off once it's all working - 0-downtime cluster migrations is something worth paying money for. What is your use case for using ES?

orestis14:10:50

Full text search as a secondary index. Could probably live with 1 node or perhaps 2 nodes for HA at some point.

lukasz14:10:02

@orestis I'll stop you right there :-) You absolutely have to run your ES cluster with min 3 nodes in AWS. single node clusters cannot survive node restarts/replacements - and AWS can trigger those at any point in time. It broke our staging environment in the early days and response I got from support was "you have to run 3 nodes, otherwise we will break it again". To be honest: I wouldn't run a single node ES for anything than local development or simple testing, it's just too fragile otherwise. Other than that, I'm pretty ok with how things work in AWS but with caveats: best to deploy your cluster within the VPC and not allow any traffic from the public internet, instead relying on security groups and VPC subnets to allow access from your applications. That also means if you want to access Kibana you will have to jump through some hoops (hosts). We use ES as the full text and faceted search as well as basic analytics backend in our product, and it's fine but again, that price 😢

orestis14:10:35

Oh. Three t3.mediums it is :)

mccraigmccraig14:10:04

if you already have k8s around then the official ES operator makes an easy alternative to the AWS managed service - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html

mruzekw21:10:36

How would I compare the price of doing this vs the managed version? I’m guessing it would mainly depend on the number of live ES containers at anytime, but is there anything else I should consider?

mccraigmccraig06:10:14

running on k8s you avoid whatever premium aws are charging over baseline ec2 for their ES service, which seems to be about 50% looking at m5.larges

mruzekw19:10:00

Wow, that sounds worth looking into! Thanks

mccraigmccraig21:10:21

it's well worth it if you are already running k8s - but if you aren't then maybe it's not something which alone would make it worthwhile to go to the effort of implementing k8s

👍 3
orestis17:10:24

@lukaszkorecki https://aws.amazon.com/blogs/database/reducing-cost-for-small-amazon-elasticsearch-service-domains/ — no mention there about single instance nodes breaking every now and then, hm. I’ll reach out to support to clarify.

lukasz17:10:56

@orestis admittedly it was a while ago and we were on ancient ES (1.7? 2.0?) but still, having run clusters by hand I just don't trust a single node anyway

orestis17:10:54

It’s just such a shame that full text search is such a headache, even for very small datasets. We are currently running an in-process solr server which is more than enough but has its own set of drawbacks.

lukasz17:10:03

what about full text capabilities in your primary data store?

lukasz17:10:26

Postgres apparently can take you pretty far, of course it's not a dedicated search engine but it is capable

orestis18:10:53

I looked into that but we also want “more like this”, tf/df and better text analysis than what Postgres fts provides. Lucene is way too low level to get decent results without a huge effort.

lukasz18:10:30

Yeah, we're in the same boat - ES has served us well, and given our setup and data set size, it's not that painful to use - and AWS' managed service takes away a lot of the pain.

valtteri18:10:36

I’ve been running single node ES in prod for a few years without issues. It’s a small project though and no-one would die if it crashed for a while.

lukasz19:10:38

Yeah, not an option for us :-(

lukasz19:10:55

(dying is not involved, but customers love to search)

valtteri19:10:38

Yep, for a popular service I’d go HA also.

valtteri16:10:42

High availability

valtteri19:10:03

How about running this in Lambda + EFS? https://github.com/valeriansaliou/sonic

valtteri19:10:22

Dunno if it would even work but somebody had this type of idea in #off-topic some time ago

lukasz21:10:15

Interesting. For a small project it could be worth a try, but ES is so battle tested, and original question was about the managed AWS service :-)