Fork me on GitHub
#xtdb
<
2020-10-04
>
dominicm13:10:31

Curious to know if there's any benchmarks of the s3 adapter? The performance on my home connection seems very poor - it doesn't seem like a realistic thing for me to use. Wondering if performance "in the cloud" is significantly superior or something like that?

victorb13:10:46

I have no idea about the specifics here, but normally when accessing S3 from another service (like EC2), you use the internal AWS network that is a lot faster (and cheaper) than requests going through the external net.

3
dominicm13:10:25

Ah, I see. So if you weren't hosting on the AWS network, you probably want to avoid the S3 adapter for this use case. Makes sense!

refset13:10:30

Yeah you probably want to test it locally with LocalStack or something that mocks the S3 API. We don't have the numbers to share for S3's "in the cloud" performance at the moment, but I suspect that for high-throughput use-cases, using Kafka as the doc-store (as well the tx-log) is still a good default choice.

victorb13:10:41

@U899JBRPF While we're talking about performance, rocksdb for production usage = super bad idea or good enough to start out with? Mostly want to avoid Kafka right now because of the complexity

refset13:10:54

RocksDB as the tx-log or doc-store is not a good option if you are looking for real high-availability, but certainly it will scale and perform just fine.

victorb13:10:31

yeah, no, not looking for HA right now, just need something single-instance scalable for now. Good to know, thanks @U899JBRPF 👏

🙏 3
dominicm13:10:50

I think JDBC is a good option for low complexity HA fwiw :)

6