Arguing about dynamic vs static typing in the context of a single program is a red herring unless you only build systems that don't communicate with any other components.

The more components that are added to a system that don't use the same type system the less return on investment for using types there is. OTOH you don't see that when focusing on data.

The individual ‘chunks’ (files/folders) of data are small and independent (i.e. there’s no need to coordinate between hosts).

…so the main concerns would be how to fairly distribute the chunks (each chunk should be stored redundantly for HA), routing requests for a given chunk to the correct host, and being able to handle added capacity as the data grows.

there’s probably a dozen key-value stores and cloud services like s3 that would seem to fit this use case. any reason not to use one of these options?

yeah, I need the data to be on disk

as in, there’s a service that operates on the data that requires it to exist locally on disk

that might be a reason, but I’m still not convinced. what’s the service do? data transformation, service requests? how does it compare to something like hadoop, storm, or samsa?

well, basically it’s a legacy technology that I can’t modify, that performs operations of hundreds to millions of file on disk

pretty much all it does is serve requests for performing one out of a list of operations on a single data chunk (folder)

it differs from those platforms you mentioned in that it’s not really programmable in any way

interesting. not sure I can think of something like that off the top of my head

alright, yeah

it seems to me like there should be pre-packaged solutions to this problem, but I can’t really think of or find any either

like, there are distributed file systems like Gluster, but using something like that seems a bit overkill to me

really all I want is something that’ll help me distribute chunks of data across hosts as well as route requests to whichever host has the relevant chunk

yea, all the stuff that I can think of also wants to have control over the file format

so key value stores will do that, but they will store the data their own way

will it be running in the cloud?

the other option I was thinking about was to just buy a large RAID disk and circumvent the distribution issues

there might be some cloud equivalent as well

is the legacy system also going to be installed on each server in the cluster?

yeah the plan is to run it in the cloud, but I’d like to keep the door open to running it on bare metal as well

yeah

and depending on the individual file sizes and latency requirements, it might still be feasible to store everything in s3 and just stream the data that you need when you need it

hmm, it seems like you could try the RAID approach on aws, https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html. just have one server with a giant RAID disk

Well, the lower latency the better, really, so I don't think S3 would be appropriate really. I suppose the RAID approach works, but it sort of feels like I'd be building a SPOF/bottleneck for something that's trivially distributable.

I guess I could just build my own centralized routing service with a DB containing the chunk -> host mapping

agreed. there’s a lot of different tradeoffs and it all depends on the use case. a distributed system is inherently more complicated than a non-distributed system. it’s just a matter if that complexity is worth it (which it might be). having a single server with a backup ready to go would be more straightforward to build and maintain, but it’s not as elastic.

Yeah

the centralized routing service doesn’t sound too bad, but it’s easy to end up spending a lot of time on incidental complexity anytime you build a distributed system

Yeah. I think the tricky part with that approach would be how to deal with disk space running out on the hosts

There'd have to be some mechanism for moving chunks to new hosts when that happens

is the total file storage requirement growing over time?

Yeah, as well as the individual chunks

ohhh, ok

the fact that the individual files are growing also reduces your options

what’s the order of magnitude for the total storage space requirement?

Not quite sure yet. At the time being it'd be terabytes

ah ok. single server does not seem appropriate for that

Maybe a dumb answer but

https://github.com/s3fs-fuse/s3fs-fuse

this has a ton of caveats, but if the legacy software reads the whole file, does work, and dumps a whole file

then this will work just fine

I'd say try profiling the s3 solution - the amount of lifeforce you would save in operations is worth seeing it through

under limitations: > random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy this seems like it might be a deal breaker

Oops sorry missed the continuation of this thread. Thanks for the recommendation! I’ll check it out, but I have a hunch that it’s not going to be a good fit.