Fork me on GitHub
#clojure-spec
<
2017-04-26
>
mbjarland12:04:22

a followup question to one I posted yesterday, if I have a conformer from string to seq of chars:

(s/def ::layout-string
  (s/and string?
         not-empty
         (s/conformer seq)
is there any way within spec to have the conformed values turned back into strings?

mbjarland12:04:51

I take that back, turns out @dergutemoritz already answered this one

dpsutton15:04:32

spec will be versioned then?

Alex Miller (Clojure team)15:04:53

versions will be ala core.async: 0.1.<git-change-derived>

gfredericks15:04:43

the specs for clojure.core functions/macros will stay with clojure?

bronsa15:04:01

no, that's what core.specs.alpha is for AFAICT

dpsutton15:04:07

it'll be interesting to see how this impacts CIDER's support for spec. Now the tooling will have to make sure to match the version of spec that the application uses

gfredericks15:04:23

@bronsa that's a namespace in the spec artifact?

bronsa15:04:35

it's a separate contrib lib

gfredericks15:04:56

oh so you need three artifacts if you want to use clojure with the core specs?

bronsa15:04:15

I'm assuming clojure will depend on core.spec.alpha but dunno

gfredericks15:04:38

I'm just trying to figure out if/how the specs can drift from the implementations

gfredericks15:04:51

if the specs stay with clojure then that's not an issue

gfredericks15:04:31

> Additionally, this is a first step towards increased support for leveraging dependencies within Clojure. I don't know what that means

Alex Miller (Clojure team)16:04:53

@bronsa yes, clojure will depend on the two new libs

Alex Miller (Clojure team)16:04:13

but you can also specify a newer version and override the one its using

bronsa16:04:59

FWIW i have a similar dependency tree with t.a/t.a.jvm and I found that it's easier to just depend on the leaf dependency (t.a.jvm or in spec's case core.spec.alpha) and release concurrently when the root dependency needs a change

mobileink16:04:38

just curious: why not put the “alpha” bit in the version string instead of the ns?

tbaldridge16:04:04

(note: I'm speaking as a Clojure user here, not as someone who has authority on the subject)

tbaldridge16:04:18

It's answered somewhat in Rich's talk on spec: https://www.youtube.com/watch?v=oyLBGkS5ICk

tbaldridge16:04:16

But the gist is you could have alpha, beta, v1, v2, v* code all in the same artifact if you wanted. That just because spec moves to "beta" or "RC" doesn't mean the alpha API has to go away.

Alex Miller (Clojure team)16:04:29

at some point there will be namespaces that are not alpha. the alpha stuff will continue to work.

mobileink16:04:38

ah, so this is just until we get to v1?

Alex Miller (Clojure team)16:04:07

well, same idea may also apply to any breaking changes post v1

Alex Miller (Clojure team)16:04:27

whether that’s done at the fn level or ns level is a future decision point

mikerod16:04:14

The only problem I see with making libraries like spec (not part of lang) are “jar hell” dep issues

mikerod16:04:42

in JVM land libraries using deps always leads to trouble because you end up using 2 libs with 2 different version of the same dep and they are incompatible

mikerod16:04:46

(eventually)

mikerod16:04:37

However, I know Rich’s talk at the last Clojure/conj was about never making a breaking change essentially. So maybe clojure.spec will be robust enough to avoid the problem and you can always just “take the newest version” of the dep when multiple lib deps diagree on version.

mikerod16:04:15

I just know that I ran into issue with say, Plumatic schema when several libraries were using it and I was using those libraries and then schema came out with a set of breaking api changes

mikerod16:04:42

This libraries having dependencies conflict problem has been one of the greatest pains of mine on the JVM it seems. I don’t think I’m alone there though. It can get difficult to deal with. I think it often encourages libraries to not use any dependencies. Which I feel is unfortunate for code-reuse or using useful utilities for documentation like spec/schema.

mikerod16:04:17

Or in Java-land sometimes people do build-time repackaging of classes to some “internal” package name. I think it is harder in clj to confidently add a prefix to a set of namespaces though in a way that doesn’t miss anything.

seancorfield16:04:00

My experience — six years of production Clojure with a code base of now 50K lines — is that API version conflicts are actually pretty rare in Clojure libraries in the wild.

seancorfield16:04:27

They are more common with Java libraries, true, but Clojure people seem to be better about API changes.

seancorfield16:04:53

(says the man who’s guided clojure.java.jdbc through several breaking API versions, however)

bronsa17:04:53

seancorfield: c.j.j is different than most libs in that it's used much more by application code than by other libs so it's easier to avoid getting into dependency issues IMO

seancorfield17:04:31

Yes, and that’s in line with why I think the utility libraries tend to be the worst offenders here. It’s why I have serious concerns about relying on Timbre, Carmine, Faraday, etc — even tho’ Peter seems really careful about not breaking the API across versions.

seancorfield16:04:37

I think the danger comes mainly from Clojure libraries that themselves have a large number of transient dependencies where they rely on Swiss Army Knife utilities libraries for just one or two “cool functions” — which is an indictment of the questionable validity of such libraries, IMO. They should be broken up into much smaller libraries so you don’t have to pull in the whole thing for just one or two useful functions.

seancorfield17:04:00

But this is a case of “do one thing and do it well” which such libraries inherently don’t follow 😞

tbaldridge17:04:23

Right, part of the issues with "jar hell" are really problems with non-additive changes to APIs.

mikerod17:04:22

Yeah, that’s sort of what I was thinking was the answer to this concern here. It’d be great to see that workout.

mikerod17:04:58

It’s the “secret weapon” 😛

Alex Miller (Clojure team)17:04:13

the whole idea is to make all versions of a library compatible

Alex Miller (Clojure team)17:04:15

but the important thing here is that alpha is a time before this applies and breaking stuff is ok (you as a user have to track the changes)

seancorfield18:04:42

@alexmiller I asked on the list but I’ll ask here too: will the new org.clojure/spec.alpha artifact be available for a few days before Clojure master SNAPSHOT actually removes the namespaces?

Alex Miller (Clojure team)18:04:21

org.clojure/spec.alpha 0.1.94

seancorfield18:04:22

@alexmiller This hasn’t hit Maven Central yet, right?

Alex Miller (Clojure team)18:04:39

I just downloaded it from there

seancorfield18:04:58

Ah, guess it just hasn’t been indexed yet? It doesn’t show up on http://search.maven.org yet. OK, got it, thanks!

Alex Miller (Clojure team)18:04:28

as of just a few moments ago

seancorfield18:04:40

Thank you! Much appreciated!!

Alex Miller (Clojure team)18:04:54

but we are moving (slowly) towards an updated alpha release of clojure that uses it today

seancorfield18:04:15

I’ll get our codebase switched over today then…

dominicm20:04:13

@alexmiller I'm a little curious about this shift. If you depend on "1.9.0", is it part of clojure's api to expose it's transitive dependency on spec? Or should I always explicitly depend on spec if I'm using it in my lib?

seancorfield20:04:24

I would expect core 1.9 will conditionally use spec at this point… So if you have spec loaded, you’ll get better macro error messages?

Alex Miller (Clojure team)20:04:35

@dominicm clojure will have a compile-time dependency on spec.alpha so you can always rely on it being available. So you don’t need to explicitly list the new library as a dependency. You may do so though in order to specify a newer version than Clojure is depending on.

Alex Miller (Clojure team)20:04:16

@seancorfield it is not currently conditional in any way

pdlug20:04:26

Is there a mechanism in spec for namespacing keys as part of a conform? For example, a map parsed JSON will not have namespaced keys so I'm applying some functions that namespace them for me but this feels like a pretty common use case which must have a better solution

Alex Miller (Clojure team)20:04:57

s/keys has :req-un and :opt-un options for this

seancorfield20:04:13

So Clojure 1.9 will transitively bring in the spec library automatically? But you can bring in a different version yourself? How will that work if the spec library breaks the API?

Alex Miller (Clojure team)20:04:59

@seancorfield yes. yes. then you’re broken (see “alpha = things may change”).

Alex Miller (Clojure team)20:04:39

sorry, have to jump in the car but can pick up later

pdlug20:04:53

@alexmiller Any pointers as to where? I've been through that a few times, I see the examples on spec'ing unqualified keywords but not coercing to namespaced keywords as part of a conform

seancorfield20:04:03

Ah, so for clojure.spec to break Clojure itself, you’d already have to have a new public release of Clojure available (even if it is 1.10 alpha perhaps at this point)?

seancorfield20:04:55

I’m hoping you wouldn’t make a new release of clojure.spec that then required users of it to rely on SNAPSHOT builds of Clojure — you’d at least have a new alpha of Clojure available by that point?

seancorfield20:04:28

(as an active user of clojure.spec in production, this is all a bit frustrating and a little worrying!)

seancorfield20:04:54

@pdlug I think maybe Alex misread what you were trying to do? There's no automated way to produce qualified keys from unqualified ones.

pdlug20:04:35

@seancorfield Thanks, that's exactly what I was asking, that's unfortunate because this seems like a really common use case (JSON APIs, JSONB in PostgreSQL, etc.)

seancorfield21:04:38

Since any namespace qualifier you would be adding is arbitrary, I don't see how spec plays into this: you control that arbitrary key qualification.

seancorfield21:04:33

Somewhere you're converting string data to a Clojure representation of JSON. Why not introduce the qualifier at that point -- then spec with qualified keys directly?

seancorfield21:04:11

That's why clojure.java.jdbc provides a :qualifier option, for example.

mobileink21:04:29

i do it by typing (on the keyboard).

mobileink21:04:52

for a json schema i'm working with i can come up with namespaces for everything, but automating it seems like a lot more work than just typing it in.

Alex Miller (Clojure team)21:04:30

@seancorfield if you were asking about spec breaking Clojure, I don't see how we would do that. The integration points are pretty small.

seancorfield21:04:39

@alexmiller I was considering a situation where you made a breaking API change in clojure.spec.alpha that could affect Clojure itself (since it depends on the earlier, non-breaking API)

mobileink21:04:00

e.g.json schema puts "di" in schema "foo.bar". in spec: :foo.bar/di.

pdlug21:04:39

I think this would be easier if there were more functions for manipulating maps to add namespaces to keywords. I keep copying the same keywordize-keys-ns function in every project to coerce incoming JSON into keywords w/ namespaces. Would be nice to also have a rename-keys that could do unqualified to qualified mappings en masse.

seancorfield21:04:44

But if the integration surface is small, I guess that breakage is unlikely — and what I meant was that you’d already have to have a version of Clojure somewhere that used the new (changed) API in order to test/use it anyway. I just wanted to check that if you ever made such a breaking change to clojure.spec.*, you’d ensure there was a *non-SNAPSHOT* version of Clojure made available at the “same time”.

bronsa21:04:09

@seancorfield the only thing clojure depends on from spec atm is s macroexpand-check, I find it really hard to imagine a situation where that could be broken

Alex Miller (Clojure team)21:04:03

I would consider that broken :)

seancorfield21:04:05

Well, the World Singles’ main codebase now uses clojure.spec.alpha with an explicit version dependency so I’m “happy” 🙂

thheller21:04:08

@alexmiller any particular reason why the core.specs.alpha is a separate package?

thheller21:04:57

keeping it in clojure.spec.alpha or clojure.core I guess

thheller21:04:53

just wondering if there are extended plans for it

Alex Miller (Clojure team)21:04:35

You mean why in a different ns or why in a different project/artifact?

thheller21:04:45

different artifact

Alex Miller (Clojure team)21:04:07

It's logically separate and can evolve at its own rate

thheller21:04:27

ok I'm still confused how the whole fdef / macroexpand-check integration is going to work, guess I'll have to wait and see

Alex Miller (Clojure team)21:04:31

We made the decision long enough ago that I've forgotten the details tbh

Alex Miller (Clojure team)21:04:12

The code is just in a different artifact

thheller21:04:28

ok, I keep thinking there is a circular dependency somewhere

Alex Miller (Clojure team)21:04:47

spec.alpha depends on Clojure

Alex Miller (Clojure team)21:04:59

Clojure depends on spec.alpha

Alex Miller (Clojure team)21:04:32

There is a bootstrap aspect to it

thheller21:04:38

ah right, forgot the current code already does a var lookup.

Alex Miller (Clojure team)21:04:08

I find it's best not to think about it too hard :)

thheller21:04:28

hehe yeah I'm thinking more about CLJS at the moment

thheller21:04:47

I was using the core.specs for CLJS but I guess that is just as easy with them in a lib

Alex Miller (Clojure team)21:04:26

The specs are not the same though for things like ns

thheller21:04:52

well, everything besides ns is though

thheller21:04:39

but still trying to come up with a good way to intergrate them with CLJS properly. might need to copy them anyways for self-hosted.

thheller22:04:18

still trying to solve the "helpful errors" issue, currently the spec explain for something wrong in a let doesn't look much more helpful than the actual compiler error

Alex Miller (Clojure team)22:04:38

Well more to come on that

seancorfield22:04:51

@thheller I think the benefit will come from familiarity — all spec-powered errors will have a similar output format and folks will quickly get used to them (both from let and other core constructs, as well as from general libraries that use spec). And any improvements to how explain works will improve all errors from code that uses spec.

seancorfield22:04:13

I mean, yeah, it’s a lot of output, but it’s very regular and structured.

thheller22:04:39

not too sure about that

thheller22:04:45

CLJS error in demo/errors.cljs at 3:1
In: [1] val: 1 fails spec: :clojure.core.specs/arg-list at: [:args :bs :arity-1 :args] predicate: vector?
In: [1] val: 1 fails spec: :clojure.core.specs/args+body at: [:args :bs :arity-n :bodies] predicate: (cat :args :clojure.core.specs/arg-list :body (alt :prepost+body (cat :prepost map? :body (+ any?)) :body (* any?)))
:clojure.spec/args  (x 1)

thheller22:04:06

the most helpful part is demo/errors.cljs at 3:1 😉

thheller22:04:25

the problem in this case is that 1 is a number which doesn't have reader metadata, so you cannot narrow the cause down further

thheller22:04:07

(defn x 1) is the actual source so 3:1 is the start of the (defn not the 1 which would be more accurate

hiredman22:04:58

I don't follow

hiredman22:04:05

that seems pretty clear

hiredman22:04:43

1 isn't a vector, 1 isn't that other big long thing

hiredman22:04:03

each case gives the value that failed, and the predicate it failed

hiredman22:04:33

and which spec the predicate came from

thheller22:04:46

@hiredman the predicate can get cryptic as seen in the second message

hiredman22:04:20

the grammar is cryptic 🙂

thheller22:04:38

I think it is alright as well, a beginner might not be so inclined

thheller22:04:12

but my issue for not is more about more accurate source locations

thheller22:04:32

for the most part they are accurate enough though

Alex Miller (Clojure team)22:04:55

The lack of reader metadata is annoying but not an endpoint necessarily

thheller22:04:44

yeah coupled with the spec errors you can usually figure out whats wrong pretty quickly

thheller22:04:12

it kinda breaks apart if the printed val gets too large though

seancorfield22:04:53

I read that first error as “arg-list is not vector?” which is pretty clear. Even the second one is pretty straightforward: “args+body” is not a sequence of “args followed by body” — and if you (source defn) that should triangulate to the thing after the symbol being defnd is not either of those.

thheller22:04:24

... please don't get hung up on my toy example. This particular example is "good enough" with spec.

thheller22:04:47

Parameter declaration "1" should be a vector is the default error message without spec ...

thheller22:04:14

which IMHO is better than the spec error (but much less accurate)

thheller23:04:52

@alexmiller will the new spec projects/artifacts get their own JIRA projects or should issues still go to CLJ?