Fork me on GitHub
#off-topic
<
2017-07-03
>
pesterhazy10:07:54

I'm looking for a way (a tool?) to enrich a git repository with binary blobs (e.g.external dependency jars, db seeds). Team members need the blobs to start the dev env, but the files are too large to be committed to git, especially if they change over time. What to do? I'm thinking ...: - sync files from Google Cloud Storage (or S3) - config file committed to git only holds pointers to current versions (hashes?) - it's fine to require developers to run bin/sync-blobs manually occasionally - limit tooling complexity. I've had bad experiences with git-lfs and git-annex, but happy to hear about reports to the contrary. Is there anything fitting this profile out there? Or am I missing any simple solutions?

carocad14:07:28

pesterhazy: maybe dumb question but can you not simply store them in a server of yours and commit a config file with the url of those files? They can be downloaded into a lib/ dir and used onwards. Or did I miss the point 😆 ?

pesterhazy14:07:02

I can, but for that I need to 1. run a server, 2. configure authentication and 3. write a script to download the right versions

pesterhazy14:07:20

I'm checking if someone has already done this before - seems like a common use case

jgeraert10:07:21

the external dependency jars can go on a maven repository

jgeraert10:07:42

if you can somehow wrap the db seeds into an maven artifact you can upload them there as well

jgeraert10:07:57

then you can just create dependencies in leiningen onto them

jgeraert10:07:03

and it will download them

pesterhazy11:07:21

@jgeraert an interesting idea, but not sure if storing a 100M db dump file in a jar is ideal

pesterhazy11:07:59

but I'll think about it. any suggestion for simple private maven hosting?

a1311:07:19

>- config file committed to git only holds pointers to current versions (hashes?) vote for this, for it's the simplest possible solution, AFAICS >- it's fine to require developers to run bin/sync-blobs manually occasionally why don't use git hooks instead?

sundarj11:07:13

im sure there's a library specifically for managing binary data in git... let me try and find it

a1311:07:13

>- limit tooling complexity. I've had bad experiences with git-lfs and git-annex, but happy to hear about reports to the contrary.

jgeraert11:07:18

@pesterhazy you can also create a simple repository on s3

pesterhazy11:07:06

@sundarj like I said I tried it but found it horribly slow (with github's servers) and github's plan is way too expensive

sundarj11:07:17

what about having a git subtree / submodule purely for binary files?

pesterhazy11:07:38

@sundarj that still requires you to keep old versions around (in .git) essentially forever, no?

pesterhazy11:07:34

I guess you have to trust the people behind http://s3auth.com?

sundarj11:07:11

it's open source, i suppose you could roll your own

pesterhazy11:07:19

hm all very interesting

jgeraert12:07:25

and of course there is nexus/artifactory/archiva

jgeraert12:07:41

but you need to host that yourself too ofc

pesterhazy12:07:32

s3auth seems dangerous (uses HTTP Basic Auth without encryption)