@dpsutton Worth a read: https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html

Will do. Thanks very much

The fastest I can load data into AWS Firehose before ES starts panicking is about 4k records/s. I suspect this is because the cluster is very small but AWS managed ES is kind of weird

yeah, AWS managed ES is weird. We try very hard not to use it.

I do kind of miss running our own clusters but Firehose was super convenient and replaced a very fragile logstash setup.

I don't have a frame of reference for what kinds of ingestion rates are normal for ES.

I've always assumed ours would be on the slow side because our records are sparse

it depends on many factors, like shards, the size of the documents, etc.

Docs are like 1kb, shards are set to default 5/1, two data nodes and three masters

We could probably turn off replicas in hind sight, it takes all of 30 minutes to rehydrate the index from s3

What is the easiest way to get an ast from clojure code? I'm trying this example but its broken: https://github.com/clojure/tools.analyzer#example-usage I want to try to write an emitter to python, "just for fun".

how is the example broken?