Fork me on GitHub
#clojure-europe
<
2022-02-11
>
dharrigan07:02:04

Good Morning!

simongray10:02:56

I need to transfer several terabytes of documents over a SMB connection through a brittle VPN… 😬

🤢 1
mccraigmccraig11:02:00

are you using rsync or similar to make your life easier ?

thomas12:02:31

can you chop it up in smaller blocks?

simongray13:02:21

@U052852ES Yeah, I am just transferring a few folders at a time, but the VPN connection will randomly close. If I could transfer it all in one go it would take around 48 hours it seems… @U0524B4UW Hm, I didn’t consider that at all. Do you think it would work with a SMB network drive?

thomas13:02:14

rsync should work with SMB. I kinda remember doing that 12 years ago

thomas13:02:36

and it zips things on the wire, if not already done so

simongray13:02:33

Ok, better try it out then. Just got disconnected again. It seems to intentionally disconnect me because I am using too much bandwidth. At least that’s the message I get.

thomas13:02:59

yeah in that case use something that can restart (which I think rsync can)

🙏 1
mccraigmccraig14:02:52

i don't think rsync cares about the type of filesystem - networked filesystems should be fine as long as they aren't weird (i think SMB will be fine, but i'm not sure about, say, S3 via FUSE). it compresses and can restart, and has some options that can help you with large files if you are seeing failures part-way through transferring large files (e.g. https://fedoramagazine.org/copying-large-files-with-rsync-and-some-misconceptions/ )

🙏 1
mccraigmccraig14:02:20

some details on how rsync calculates source/target file deltas (and so avoids transferring whole files after a previous transfer has partially succeeded): https://rsync.samba.org/tech_report/node2.html

🙏 1
dharrigan16:02:36

If you use rsync -P it'll do partial

dharrigan16:02:43

so it can be restored from that point onwards after a failure

dharrigan16:02:56

I normally do rsync -vahP <src> <dest>

🙏 1
Ben Sless19:02:10

Do it with a torrent?

simongray20:02:01

I don’t control the originating server.

simongray20:02:10

It’s a brittle samba directory connected to by some proprietary software that the IT department wrote which establishes athis connection to when you log in to their VPN. This directory contains ~2 years worth of TIFF file uploads that have been gradually put there over that period by some people operating a scanner in some basement of the Danish Royal library.

😱 1
simongray20:02:53

I’ve asked to get direct access, but they say it’s not possible.

simongray10:02:21

LMAO, it literally just lost connection at 760 MB transferred… guess this will take some weeks to accomplish

orestis19:02:25

Upload to S3 and then download at the other end?

orestis19:02:46

Or perhaps look at tailscale? Not sure if relevant.

simongray20:02:56

I don’t control the originating server.

genRaiy10:02:40

good morning

lread15:02:10

top o' the morning!

dharrigan16:02:20

😍 7
🍪 3
🐶 1
dharrigan16:02:52

Cute! Well, until next week! ttfn!

pez17:02:25

Cute overload!

borkdude17:02:48

Until I opened the picture, I was thinking, how on earth can an overload be cute, like (fn ([]) ([x])) - ok, I admit, that's cute too.

orly 3