This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-10-15
Channels
- # admin-announcements (1)
- # announcements (11)
- # asami (6)
- # aws (26)
- # babashka (17)
- # beginners (119)
- # bristol-clojurians (7)
- # chlorine-clover (2)
- # cider (3)
- # circleci (1)
- # clj-kondo (10)
- # clojure (127)
- # clojure-australia (3)
- # clojure-dusseldorf (5)
- # clojure-europe (135)
- # clojure-france (5)
- # clojure-nl (8)
- # clojure-uk (6)
- # clojurescript (103)
- # clojurewerkz (1)
- # css (2)
- # cursive (5)
- # datalog (5)
- # datomic (36)
- # emacs (3)
- # events (2)
- # figwheel-main (3)
- # fulcro (1)
- # graalvm (3)
- # helix (31)
- # jobs-discuss (4)
- # leiningen (1)
- # london-clojurians (1)
- # malli (17)
- # off-topic (2)
- # parinfer (10)
- # portal (1)
- # re-frame (48)
- # reitit (2)
- # reveal (12)
- # shadow-cljs (3)
- # sql (3)
- # tools-deps (4)
- # vim (4)
- # xtdb (22)
@orestis the price 👎 but it's pretty hands off once it's all working - 0-downtime cluster migrations is something worth paying money for. What is your use case for using ES?
Full text search as a secondary index. Could probably live with 1 node or perhaps 2 nodes for HA at some point.
@orestis I'll stop you right there :-) You absolutely have to run your ES cluster with min 3 nodes in AWS. single node clusters cannot survive node restarts/replacements - and AWS can trigger those at any point in time. It broke our staging environment in the early days and response I got from support was "you have to run 3 nodes, otherwise we will break it again". To be honest: I wouldn't run a single node ES for anything than local development or simple testing, it's just too fragile otherwise. Other than that, I'm pretty ok with how things work in AWS but with caveats: best to deploy your cluster within the VPC and not allow any traffic from the public internet, instead relying on security groups and VPC subnets to allow access from your applications. That also means if you want to access Kibana you will have to jump through some hoops (hosts). We use ES as the full text and faceted search as well as basic analytics backend in our product, and it's fine but again, that price 😢
if you already have k8s around then the official ES operator makes an easy alternative to the AWS managed service - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html
How would I compare the price of doing this vs the managed version? I’m guessing it would mainly depend on the number of live ES containers at anytime, but is there anything else I should consider?
running on k8s you avoid whatever premium aws are charging over baseline ec2 for their ES service, which seems to be about 50% looking at m5.larges
it's well worth it if you are already running k8s - but if you aren't then maybe it's not something which alone would make it worthwhile to go to the effort of implementing k8s
@lukaszkorecki https://aws.amazon.com/blogs/database/reducing-cost-for-small-amazon-elasticsearch-service-domains/ — no mention there about single instance nodes breaking every now and then, hm. I’ll reach out to support to clarify.
@orestis admittedly it was a while ago and we were on ancient ES (1.7? 2.0?) but still, having run clusters by hand I just don't trust a single node anyway
It’s just such a shame that full text search is such a headache, even for very small datasets. We are currently running an in-process solr server which is more than enough but has its own set of drawbacks.
Postgres apparently can take you pretty far, of course it's not a dedicated search engine but it is capable
I looked into that but we also want “more like this”, tf/df and better text analysis than what Postgres fts provides. Lucene is way too low level to get decent results without a huge effort.
Yeah, we're in the same boat - ES has served us well, and given our setup and data set size, it's not that painful to use - and AWS' managed service takes away a lot of the pain.
I’ve been running single node ES in prod for a few years without issues. It’s a small project though and no-one would die if it crashed for a while.
What's HA?
How about running this in Lambda + EFS? https://github.com/valeriansaliou/sonic