Fork me on GitHub
#boot
<
2016-09-10
>
sbondaryev11:09:35

I’ve just cloned a javelin repo which uses boot - what is the fastest way to start a REPL to test modifications which i made to the lib?

micha14:09:13

hi @sbondaryev you can run the javelin tests

micha14:09:10

boot watch speak test-javelin

micha14:09:21

that will run them continuously as you hack on the code

micha14:09:06

(you'll need to have phantomjs installed) to run the tests

micha14:09:59

we should probably update javelin to use the cljs test task now

micha14:09:13

@martinklepsch which tasks do you use for running cljs tests these days?

micha14:09:17

sweet thanks

martinklepsch14:09:39

but mostly snapshot releases & not so much docs so you'll probably end up reading the code 🙂

micha14:09:01

haha that's ok with me

micha14:09:51

oh there is even an example project

xcthulhu15:09:34

@richiardiandrea I wanted to follow up on a discussion from yesterday in #cljs-dev

xcthulhu15:09:16

As you know, Clojars (optionally) lets you sign JARs you push to it's artifact repository - their aught to be a way of querying their API for the registered public key for a particular user. Next, there should be a way of getting a signature from their API, and finally, once you have a public key and signature, checking is straight-forward. Almost everyone just wraps bouncy-castle - here is a wrapper

xcthulhu15:09:41

Checking signatures is far more challenging in ClojureScript, where there is a dearth of cryptographic libraries that work well with the Closure Compiler and the zoo of JavaScript platforms.

micha15:09:50

@xcthulhu boot does show you whether jars you get from maven are signed or not

micha15:09:02

actually verifying the signatures, though, is another matter

micha15:09:27

i don't know of any real solution in this area

micha15:09:58

maybe with letsencrypt though there is a way forward for the future

micha15:09:19

like currently anyone can sign a jar with a gpg key

micha15:09:38

but there is no way to know whether you should trust that key or not

micha15:09:09

we were talking about http://keybase.io a while ago

micha15:09:41

but then i realized that their whole system is based on npm modules and i just don't trust it

xcthulhu15:09:20

@micha Doesn't maven/clojars provide an API for getting the public key of registered artifact authors?

micha15:09:45

how would that solve the problem?

xcthulhu15:09:06

You would just use the repository as your authority

micha15:09:27

but why would you trust it?

xcthulhu15:09:43

You trusted it to download the artifact in the first place

micha15:09:51

not rally

micha15:09:03

i mean if that's the case then you don't need signatures right?

xcthulhu15:09:18

Well, I guess someone could mess with it on your hard-drive

micha15:09:27

the signature is to protect against a compromised clojars

micha15:09:15

like if you already trust clojars to send you the correct jars, you can just use that

micha15:09:27

which is my current situation

xcthulhu15:09:51

Like I said, a hacker could modify the jars placed in your .m2 directory

micha15:09:15

one thing i would trust would be jars signed by keys that are themselves signed by a legit CA

micha15:09:35

where you need to provide personal info to get the cert

micha15:09:36

and so on

micha15:09:41

like a SSL cert

micha15:09:23

http://keybase.io does the next best thing, which is cross reference multiple social media accounts

xcthulhu15:09:24

Eh, CAs can get hacked too. Honestly, a solution that is independent of worrying about various servers getting hacked would be CAS

xcthulhu15:09:38

Storj, IPFS, or whatever Ethereum is inventing

micha15:09:52

lol then you get neverending 51% attacks

micha15:09:13

at the end of the day it seems like you need to trust humans to keep everything above board

xcthulhu15:09:31

It's not a blockchain

xcthulhu15:09:35

It's a CAS system

micha15:09:37

it's it's purely algorithm then it can be exploited

xcthulhu15:09:06

Please sketch in greater detail how you would compromise IPFS

micha15:09:18

oh i am not the hacker

micha15:09:25

but look at etherium

micha15:09:49

i am ok with like verisign handling validation of SSL certs

micha15:09:07

i feel more comfortable with that than with a blockchain based SSL cert system

micha15:09:15

because there are people who can be sued or sent to jail

xcthulhu15:09:57

I am not suggesting using a blockchain as a certificate authority

xcthulhu15:09:25

The only reason I mentioned Ethereum is that their "Swarm" system they are building out is a CAS like IPFS

xcthulhu15:09:58

Incidentally, IPFS is already dogfooding using IPFS for versioning their upstream dependencies

xcthulhu15:09:20

I would love it if I could specify my dependencies like that

micha15:09:47

but how do you know which things to trust though

micha16:09:46

i'm talking about transitive dependencies

xcthulhu16:09:46

I need to do some chores around the house, but I would not mind discussing this matter further, especially if I can get devs working on a build-tool excited.

xcthulhu16:09:21

Oh, well either the JAR has dependencies in Clojars/Maven Central, and you are in the same boat you are now

micha16:09:28

yeah this is a huge problem and we're only seeing the tip of the iceberg

xcthulhu16:09:33

Or, all of its dependencies are specified with IPFS hashes

xcthulhu16:09:41

In which case you just go to the CAS

micha16:09:08

yeah i meanimmutability is good, i just don't see how it helps you trust the source of the immutable thing

micha16:09:32

i don't see how you can avoid delagating to some legal entity like a CA authority

micha16:09:45

something that can be taken to court if they misbehave

xcthulhu16:09:05

It's called "Content Addressable Storage". If the merkel hash of the artifact/directory you have on your hard-drives checks out, then you must have received the right thing.

xcthulhu16:09:09

Please go read the white paper.

micha16:09:16

i get it, i do

micha16:09:37

i can be 100% sure that the thing on my hard drive is the correct sequence of bytes

micha16:09:59

that does not help me know if the sequence of bytes was written by a human i trust or not

micha16:09:38

specifically transitive dependencies

micha16:09:57

how do i know if a library is pulling in a trojan horse dependency?

micha16:09:26

the dependency jar is 100% correct on my harddrive

micha16:09:38

it just contains malicious code written by a hacker

micha16:09:33

and it's a transitive dependency, so i'm not likely to inspect its code

xcthulhu16:09:36

Honestly you have such threats now. Anybody can push whatever they want to Clojars, it could potentially be very malicious.

micha16:09:48

we totally do have this threat now

micha16:09:54

it's a huge problem i think

micha16:09:21

i'm really amazed that it hasn't blown up yet

xcthulhu16:09:57

I am discussing a problem of "Well, checking OpenGPG signatures isn't a very good check if a server is compromised" which is true.

xcthulhu16:09:07

And CAS would be a valid alternative.

xcthulhu16:09:36

Rather preferable

micha16:09:59

if the signature was associated with a legit certificate authority though, you don't care if clojars is compromised

micha16:09:16

and you also know you can trust the jar, because that's what the CA does

micha16:09:35

it can revoke, etc

xcthulhu16:09:36

I've never used Let's Encrypt, do they do rigorous KYC?

micha16:09:36

there would be nothing a hacker could do when they exploit clojars

micha16:09:17

they're like a free CA you can use to get a SSL cert

xcthulhu16:09:25

That's a "No"

micha16:09:46

but not every platform includes their keys by default, so it's not really 100% ready

xcthulhu16:09:51

So you can register there, push a Jar with a trojan, and sit easy still anonymous.

micha16:09:15

how can you be anonymous?

xcthulhu16:09:25

Well, they don't bother with KYC or anything.

micha16:09:30

they do the same human verification that verisign does, as far as i know

micha16:09:54

otherwise you might as well use the mit pgp keyserver right?

micha16:09:07

man i certainly hope they do, anyway

micha16:09:18

otherwise i will block them in my browser instanly

xcthulhu16:09:48

Eh, I'm cool with source code being included in my JARs TBH and just using CAS. I'd rather have CAS for versioning anyway, since someone can always push an artifact with the same version tag to a repository and rewrite history.

micha16:09:10

yes i agree that immutable storage is pretty key

xcthulhu16:09:14

I guess if you wanted to bolt on an elaborate CA thing for legal liability that's cool too. I'm not an anarchist or anything.

micha16:09:23

but i think it doesn't really solve the whole problem

micha16:09:32

i think we need both

micha16:09:55

the immutability is more for general dependency hell avoidance than security imo

xcthulhu16:09:20

Although it does help with validation.

xcthulhu16:09:54

A certified individual would presumably need to sign some hash, might as well be a CAS hash

micha16:09:57

i wonder if it would be possible to get some kind of funding for real certs we can use to sign jars

micha16:09:11

perhaps linked to github user

micha16:09:36

that seems to be the real solution to the trust issue

micha16:09:53

because nobody can verify the source of every transitive dependency

micha16:09:12

that ship has sailed except for maybe the NSA or something

xcthulhu16:09:45

Ethereum has a janky thing using the blockchain as their CA - https://github.com/axic/mango

micha16:09:09

i think it really needs to be part of the legal system though

micha16:09:26

we're dealing with our customer's money and trust

micha16:09:20

this seems pretty sane to me

xcthulhu16:09:13

Okay, step one, get support for checking these in boot, even if it's sort of just for show

micha16:09:52

there is boot show -v, which checks whether jar is signed or not

micha16:09:07

does not do verification because that's useless the way things are

micha16:09:16

it would be false hope 🙂

xcthulhu16:09:20

Like I said, you need to do it just for show

micha16:09:04

so i think the only impediment is the overhead and cost of maintaining the CA relationship

micha16:09:23

which makes sense because there are costs to being a legitimate professional

micha16:09:52

we can build boot tooling to make the mechanics of correctly signing and verifying jars simple

micha16:09:11

we need some way to help people pay for the certs and manage them

xcthulhu16:09:12

Actually, I have been doing most of my work in ClojureScript lately

xcthulhu16:09:00

A ClojureScript GPG library is a big undertaking for me.

micha16:09:19

why gpg in cljs?

xcthulhu16:09:39

So you can have a build system that doesn't need the JVM

micha16:09:11

i would shell out to the jar tooling

micha16:09:28

like in the tutorial

micha16:09:38

the final artifact will be a jar

micha16:09:50

you definitely don't want to rewrite aether in cljs

xcthulhu16:09:59

There's no unified way of shelling out in the various JS CLI platforms.

micha16:09:38

you need the jvm to manage maven deps anyway

xcthulhu16:09:55

You can tell that to the people who want to build the JS build tool. Honestly, I have no idea when I will have the time / funding to do this sort of work.

micha16:09:57

why not use boot just for dealing with jars

xcthulhu16:09:19

I do everything with ECC anyway, RSA is terrible.

micha16:09:33

what js build tool are you referring to?

xcthulhu16:09:49

After google released their JS Closure Compiler, everyone has been excited about eventually getting a JS land build tool, since waiting for the JVM to spin up is lame.

xcthulhu16:09:19

But the mess of course is that there's no tools for dealing with JARs, not even a standard for IO for JS CLI

xcthulhu16:09:50

Nor crypto, but that's like, the last on everybody's list of thing to worry about, but it's been my living lately

micha16:09:27

yeah i don't have those problems

micha16:09:32

i write business applications

micha16:09:42

clojure startup time isn't a concern for me

micha16:09:30

when i make the UI i'm not playing around with live reaload constantly, because the UIs are for business. i don't need to have a live art performance

micha16:09:52

what i do need is robust dependency management though lol

xcthulhu16:09:57

It's not a big deal for me either. But, I would like to sort of help the W3C out in putting the right primitives in their cryptographic API, because right now that thing is useless.

xcthulhu16:09:46

Other than that and helping the crypto situation in JS in general out of kindness, I'm ready to admit it's theatre right now when dealing with JARs.

micha16:09:34

yeah i haven't seen any package manager that beats the reliability of maven really

micha16:09:06

it has some quirks but they're easy enough to understand because it always does the simplest thing when it's unsure

micha16:09:59

definitely prefer maven to npm for instance

xcthulhu16:09:48

I prefer maven to NPM as well.

xcthulhu17:09:59

@onetom This is why the common practice amongst go developers is to "vendor" their upstream developments, and why IPFS dogfoods itself in its source code

onetom17:09:42

sure, it's just a good example of a real story which shows the serious consequences of this dependency problem

xcthulhu17:09:06

Right. It's a bit tougher in go-lang than in Clojure, because you can literally put URLs in your includes, and go will fetch them.

xcthulhu18:09:48

Much as I love the idea of IPFS, there's no JVM port. Who knows when one will land...

richiardiandrea18:09:36

Very interesting, I have always wanted to have a quicker build tool, especially for dev, but I agree that a maven port to js would be a big challenge (maybe Aether is enough to port?)

richiardiandrea18:09:50

I am checking ipfs ;)

onetom18:09:14

@xcthulhu when u said https://clojurians.slack.com/archives/boot/p1473523449001059 which white paper were you referring to? can u give an URL, pls?

xcthulhu18:09:46

I enjoy that the URL uses the IPFS hash in the link to the official white paper.

onetom18:09:23

ah, this one i think i read before

flyboarder18:09:50

+1 IPFS task

micha18:09:08

yeah the go dependency troubles are pretty well known now i think

micha18:09:33

i am really happy with the robustness of maven as a package manager

micha18:09:24

it's also important to separate package management conceptually from the build process, and even from dependency management

micha18:09:11

clojars does enforce immutability to the extent possible without something like IPFS

micha18:09:02

it won't let you change an artifact you've already pushed

micha18:09:10

and you can't delete anything

flyboarder18:09:39

I think it would be a good community project to make an IPFS task that works like the old checkout task, that way boot doesnt need to care about it

flyboarder18:09:30

all we are trying to do is download a file extract the contents and add it to the fileset.

micha18:09:25

i suppose

micha18:09:46

personally i am strongly opposed to using anything but versioned dependencies in my projects

micha18:09:51

especially transitively

micha18:09:46

if people start publishing artifacts with dependencies on SHAs i think it will be a real shame

micha18:09:07

i am pretty convinced now that the problem is not a technical one

micha18:09:17

it's a hygiene problem

micha18:09:40

if you're going to publish artifacts you need to be thoughtful and move slow

flyboarder18:09:15

lots of things use sha’s for their build identifiers the are less developer friendly but more computer friendly

micha18:09:32

but dependencies are not a computer problem

micha18:09:37

they're a human concern

micha18:09:43

the halting problem

micha18:09:54

a computer can't know which version of a dependency you want

flyboarder18:09:16

right if im leaving it up to the computer i probably want the latest version, it tihnk it works better for rolling systems

micha18:09:16

and SHA as version is meaningless

micha18:09:40

the whole point of the version is to communicate information about compatibility

micha18:09:55

but that information i don't think is computable

micha18:09:03

a human needs to make a judgement call

micha18:09:14

i don't see how CAS helps at all

micha18:09:58

if you're worried about exploits and hackers you need some kind of crypto web of trust anyway

micha18:09:10

and in that case you can validate the SHA very easily without IPFS

micha18:09:34

i never want the latest version of transitive dependencies btw

micha18:09:48

that's why i don't use ruby, basically

micha18:09:23

maven is great because if it builds and works today it will build and work the same in 100 years

flyboarder18:09:30

i think it depends on why you want an IPFS file, im not arguing for it’s use with dependencies just it’s availability

flyboarder18:09:48

but again i dont think it’s boots concern

micha18:09:51

i specifically don't want to update any dependencies automatically

xcthulhu18:09:58

If you pegged to an IPFS hash, it would not update any dependencies automatically since the thing would be immutable

xcthulhu18:09:24

TBH, you can already do this sort of thing - you can just git hashes with http://jitpack.io

micha18:09:24

maven is already immutable though

xcthulhu18:09:53

@micha - I think if you email the folks at Clojars they will manually take artifacts down.

xcthulhu18:09:07

Just like over at NPM

xcthulhu18:09:28

Also, if Clojars goes down, you are SOL

xcthulhu18:09:34

Happens from time to time.

micha18:09:38

i trust toby to make the right decision with respect to deleting things

flyboarder18:09:41

they might be less likely to do this after padleft incident

micha18:09:54

the left-pad thing could never happen with clojars

xcthulhu18:09:39

Well, I trust Toby too now, since I trust you, @micha, a total stranger 😛

micha18:09:23

that's what business is built on though, which i feel very good about really

xcthulhu18:09:43

I thought Clojars was a non-profit? What on earth do they sell?

micha19:09:12

no i mean business in general

micha19:09:41

like i trust him to manage clojars and my customers trust me

xcthulhu19:09:33

Oh, okay. Anyway, I think I might have a job opportunity porting IPFS to the JVM.

xcthulhu19:09:35

I'd rather not have to trust Toby or whoever unless they are actually incentivized (with money, not reputation) to not delete stuff.

xcthulhu19:09:15

No idea what it's like over at the other repos like mavencentral or sonatype.

xcthulhu19:09:54

In python, you can just delete stuff on pypi

xcthulhu19:09:24

Somehow there's never been a left-pad type incident

flyboarder19:09:36

I’d prefer to depend on people who are not incentivized by money, someone who is already incentivized is more likely to switch sides for more incentives… NSA deals with Facebook, Microsoft, Google etc.

micha19:09:13

well i think there are separate concerns that are being conflated maybe

micha19:09:30

like the ability to delete things, IPFS is deifnitely good for that

micha19:09:41

i mean preventing people from doing that

flyboarder19:09:08

right and prevent them from replacing packages, with say modified or corrupted ones

flyboarder19:09:15

part of the problem is that a lot of issues are starting to converge

micha19:09:46

the second point i think is a crypto problem

micha19:09:00

because SHAs are meaningless

micha19:09:12

the only meaning they have is as a cache key

micha19:09:29

but there is no information about what the thing actually is there

flyboarder19:09:12

right moving from a version to an sha is kinda a regression since, like you said, it’s meant to convey info to the human, a different issue than using something like IPFS

xcthulhu19:09:53

Sounds like it should be optional.

xcthulhu19:09:33

For me, I would vastly prefer to do versioning with hashes rather than numbers

xcthulhu19:09:49

At least for security sensitive things.

micha19:09:14

i mean you already do that with maven though

micha19:09:24

the artifact's pom url is the hash

micha19:09:40

it's true that things can in theory be deleted or changed

micha19:09:07

but verifying that the thing at the hash url contains the correct contents is trivial with or without IPFS

micha19:09:29

i mean using crypto

micha19:09:47

and you need that crypto layer anyway if you have any transitive dependencies

micha19:09:21

if you get rid of hashes that encode version info in favor of SHAs you lose a lot i think

micha19:09:32

just in terms of auditing

xcthulhu19:09:02

Why? Can't you still pull down all the source code?

flyboarder19:09:15

i really enjoy the talks in #boot and #hoplon , very interesting discussions that cover a wide range of issues 🙂

micha19:09:38

sure, but you end up with kind of the same thing though, no?

micha19:09:51

each hash needs to be associated with metadata about what the artifact means

micha19:09:13

like i look at my deps tree and see all hashes

micha19:09:18

that would not be very useful to me

xcthulhu19:09:43

A human readable view would presumably have the version tags on all of your deps.

micha19:09:25

so the hashes could be an implementation detail then

micha19:09:39

because you'd need a way to be sure that a hash corresponds to a version

xcthulhu19:09:53

It's just so you can avoid having to worry about Clojars or mavencentral getting hacked or taking a thing down.

xcthulhu19:09:55

Like, because as we already established, the GPG key sigs are rather useless.

xcthulhu19:09:14

Makes you feel good I suppose

micha19:09:48

i feel like someone cares when they sign the jar

micha19:09:20

it's like the thank you note you get in return for the bread machine wedding gift

micha19:09:09

so if artifacts are hosted in IPFS you'd want some kind of index, right?

micha19:09:39

the index would need to live in the regular web i imagine

xcthulhu19:09:01

Clojars or GitHub

micha19:09:06

that's kind of interesting too

micha19:09:13

the git connection

micha19:09:24

so the index is also sort of immutable

micha19:09:31

like you could have your own copy of it

micha19:09:38

and clone, merge etc

micha19:09:17

your dependency list could be associated with a specific commit in the index in git

micha19:09:49

because if the index is modified you're still pretty screwed it seems

micha19:09:12

like for example, suppose i depend on foo/bar version 1.2.3

micha19:09:21

i look it up in the index and get some hash in IPFS

micha19:09:44

someone could change that to point somewhere else in the index later

micha19:09:17

and then it's loading a different thing for the same artifact

micha19:09:22

this is interesting

micha19:09:49

IPNS + IPFS seems like something you can build a maven wagon on

micha19:09:13

that's the part of maven that deals with remote resources

micha19:09:25

vs. the local cache

xcthulhu19:09:32

Cool. Well, truth be told, I will have to wrangle a contract to port IPFS to the JVM, which will take time.

xcthulhu19:09:43

But I will keep folks posted since it's an exciting development.

micha19:09:57

is there a fuse filesystem implementation for it?

micha19:09:09

or something like that?

micha19:09:02

anyway yeah this looks promising to me

flyboarder19:09:07

@xcthulhu I would be very interested in tracking a clojure version of IPFS to JVM

flyboarder19:09:33

I actually know someone who may be interested in helping a project like that, hes waiting for it

keatondunsford20:09:11

Btw y’all @micha @xcthulhu @flyboarder I made an #ipfs channel in here a month ago. 🙂 I’m finding myself at the intersection between Clojure world and Web 3.0 world too, but as a new developer it’ll probably be a while before I can build anything to link up to Ethereum/IPFS/Golem Network/Storj myself.

keatondunsford20:09:45

Also made an #ethereum channel. Not sure if y’all know but the EthereumJ project is a full-out Java implementation of the Ethereum node. So in theory you could run an Ethereum node in Clojure. Nothing for IPFS though from my understanding like you were mentioning. You guys should really look into the Golem Network too for basically decentralized AWS through Docker containers. It’s so cool IMO. That’s written in Python.

micha20:09:12

interesting!

madstap23:09:39

58477kllllll+,9n6u[h7n6