Fork me on GitHub
#clojure-spec
<
2017-07-02
>
wildermuthn02:07:34

I recently read an article about how Reddit (or parts of it?) are switching to TypeScript, and one of their main criteria for choosing a compile-to-js lang was it needed to be typed. Been thinking about that, and having a lack of experience with typed languages, what other Clojurists think about that argument, particularly with respect to Clojure Spec. Link: https://redditblog.com/2017/06/30/why-we-chose-typescript/ Any articles or thoughts out there about how Spec fits into this question of typed vs. dynamic?

carocad10:07:15

wildermuthn: you should probably check the core.typed library from clojure. It adds static type annotations on top of clojure code. Also afaik one of the current initiatives in google summer of code for clojure is precisely to use specs to create those static types. I think another project about that is spectrum which is already pursuing that initiative.

seancorfield18:07:34

I think developers either naturally lean towards types or not, and that influences their language choices. At World Singles, we’ve historically worked with dynamically typed languages and our forays into typed languages (C#, Scala) have not felt very “natural” to us. Since we adopted Clojure in 2011, we’ve also tried Schema and core.typed, and given up on them, then tried them both again and given up on them again. We’re heavy users of Spec, however, and we focus mostly on specifying data structures and using conform to validate (with a little coercion).

seancorfield18:07:12

Myself, personally, I’ve worked with both statically typed and dynamically typed languages across most of my career and, whilst I like the idea of static types, I don’t generally like the reality of them. I was an early advocate of Haskell (because of my PhD work on FP language design in the 80's) but I have something of a love/hate relationship with Haskell’s type system 🙂 So, overall, I think I’m one of those that leans naturally to dynamically typed languages.

carocad19:07:55

@U04V70XH6 would you mind sharing the reason why you abandoned the type annotations for Clojure? Although I have not used them I don't see a reason to abandon them completely. I think specially for libraries api it could be quite useful, even though personally I would prefer core.typed to infer the type of my arguments based on their spec I.e, go from the more generic approach to a more specific one and definitely avoid having to do both

seancorfield19:07:57

It’s very hard to provide type annotations that satisfy core.typed for some idiomatic Clojure code. Also, core.typed doesn’t do a great deal of type inference so you have to annotate a lot of code to satisfy core.typed. The former means you often need to rewrite Clojure to a less idiomatic style to satisfy the type checker. The latter means you end up with a lot of “noise” in your code.

seancorfield19:07:09

I don’t remember how many of my discussions with Ambrose about this were done on public mailing lists. Some. So maybe look in the archives (I think there’s a core.typed mailing list?).

seancorfield19:07:10

By contrast, Spec is very much purely opt-in. You can specify just what you want, and test for validity/conformance exactly where you want. You can specify just data or just functions or a bit of both. You can choose exactly what to instrument and when. And you also get the benefit of being able to generate conforming test data and conduct generative testing as and when you need it.

carocad19:07:39

I guess the marketing idea of core type is greater that its implementation. Have you actually tried spectrum? I know it is quite experimental but I think it child be a great tool, although from what I have read you also have to annotate a lot of code for it to stop complaining

seancorfield19:07:31

We have not tried Spectrum yet. It seemed a bit too experimental for production code at present.

seancorfield19:07:35

And to be fair to core.typed, it’s improved dramatically over the years and it is an amazing piece of engineering work. Clojure is just a really hard language to perform full and useful type inference on so it’s a brutally hard problem to solve. I understand that the latest work being done focuses handling the boundary between typed code and untyped code and that Ambrose is working with the Typed Racket folks on that?

carocad19:07:47

Yeah I'm looking forward to the results of this gsoc and see the result of spec/type projects (hopefully they are implemented) :)

seancorfield19:07:13

Part of my PhD work was looking at type inference so I know how difficult this problem can be with some languages 😐

seancorfield19:07:52

This was a fascinating talk at Clojure/West BTW http://2017.clojurewest.org/emina_torlak/ — back in the late 80's I was working with MALPAS which was a program verification system and we were transforming C code for analysis and verification. Emina’s talk showed both how far this area of research has come in three decades, as well as how difficult it is and how slow progress really is.

carocad19:07:13

Mind if I ask: in your opinion is the type inference that core.typed is doing "better" than for example flow for JavaScript. I guess I don't have to ask about java :/

seancorfield19:07:40

I don’t follow anything in the JS world — can’t stand the language 🙂

carocad19:07:25

Well funny because flow is written in Ocaml so I thought the techniques might be familiar or general for type inference. So I was wondering about their difference. I guess I ignore too much about the topic 😫

seancorfield20:07:00

The implementation language has no bearing on the language you're analyzing, interpreting, or compiling. I implemented both APL and my various experimental FP languages using Pascal, and I wrote the type inference engine in Prolog simple_smile

seancorfield20:07:34

Given a choice back then I'd have preferred Algol 68 to Pascal but my reviewers were not familiar enough with Algol 68 so I wasn't allowed.

carocad20:07:55

Jesus I don't even know those languages ... :/

seancorfield20:07:26

The "Pragmatic Programmer" suggests learning a new language every year -- good advice! 😈

wildermuthn01:07:14

Good info here! @U04V70XH6, did you find that using typed langs helped to do the things the article said it wanted: catch bugs earlier, make it easier to refactor, have a larger team working on the codebase?

wildermuthn01:07:39

And related, whether Spec can actually address those issues?

wildermuthn01:07:54

My cljs team uses Spec mostly to conform/validate api request/responses. But thinking about whether to expand it to be a keyword namespaced kind of typing.

seancorfield01:07:11

Sure, a type system will catch some bugs earlier (by definition, since it takes effect before your code even runs) but a type system can’t catch all classes of bugs. In the context of Clojure — working with small, pure functions on immutable data (for the most part), and building up code with a REPL — there are whole class of bugs you won’t introduce in the first place. I don’t think there’s been enough analysis in the FP arena comparing static/dynamic type systems for any conclusion to be drawn. If you compare, say, Java and Kotlin, you have two languages with static types and one catches potential NPEs (Kotlin) and one doesn’t. But if you look at Clojure, nil-punning is idiomatic and makes annotating code much, much harder because “nilable” infests all the types in many places in your code. Type signatures for generic code tend to become very complex — try writing a Spec for map (and remember its return type will be different with just a function argument compared to with one or collection arguments).

seancorfield01:07:02

Spec isn’t a type system and you’re not really comparing apples to apples if you try to compare it to one. I don’t know what Typed Clojure (in a true language sense, not core.typed) would look like but I don’t think it would be much like the Clojure we know and love today.

carocad07:07:04

@U04V70XH6 I partially disagree with what you said. Building up code in the REPL and having small pure functions sure helps but there are times where the datastructures are soo deep that creating those by hand becomes a nightmare. Specially in those cases I find a type system to be way more useful than a repl. It gives you a first overview of whether a big refactoring is going in the right direction. It is not something to rely on, of course, but I think it eases those tasks

seancorfield10:07:24

@U0LJU20SJ I'm curious about the business domain where such deeply nested structures occur? It's not something I've ever run into. I'd expect Spec to be very helpful in such a situation then?

carocad13:07:05

@U04V70XH6 take a look at the response from Mapbox Directions API: https://www.mapbox.com/api-documentation/?language=JavaScript#route-object. They need to returns lots of objects in a json api. In my case I am trying to provide a customized routing which means that I have to mimic their api response. The problem obviously arises once they change (currently they are at v5). In those cases you have to throw away pretty much every spec that you had for that and start refactoring lots of code. That is what I meant with deeply nested objects involved in a big refactoring process. I guess @U07CTDKT7 tried to raise that topic as well.

carocad14:07:11

To mention another example. Since we are not all senior developers ( 😉 ) we might just go with the first idea of what you should return at a specific place. Later on as you learn more both about the topic and about the language you realize that you need a better design for it. More often than not, those are deeply nested objects which require quite some refactor. In that case, having a static type analyzer to back you up is great. Obviously that means that the system was poorly designed from the beginning but then again … we beginners 😛

wildermuthn15:07:29

@U04V70XH6, what you are saying about immutablity, pure functions, and the repl being a bug-reducer has been my experience too (js vs ClojureScript). @U0LJU20SJ, I've had the same case with denormalized api responses. We're using DataScript to renormalize the data, but that's really tedious (using Spec has helped). In terms of refactoring, though, my impression is that not having to deal with types makes refactoring easier (but not simpler?). I'm impressed by Rich Hickey's point in his Spec talks that it is probably better to just have new functions that to modify old ones in a breaking manner (which Spec should make easier to identify).

wildermuthn15:07:42

But this would mean Spec'ing every function you ever write! :) Which makes me wonder if it is worthwhile.

carocad15:07:37

@U07CTDKT7 I actually tried to spec every function before as well and it was a nightmare. But at some I realized that specifying the API functions is actually enough. If the user facing function works as expected, then all other internal ones should also do it, otherwise the external would have failed. At least that is the way that I am approaching the problem now 🙂

seancorfield16:07:49

Agreed, re: spec’ing lots of functions. We spec key data structures (“data as API”) and we spec some functions. Nearly all of our use of Spec is explicit (`conform`, valid?) rather than implicit (`instrument`). Regarding deeply nested API results — I’d probably wrap the API and flatten results out quite a bit and then work with that everywhere else in my code, probably with namespaced keys. Easier to read and manipulate. Inside the wrapper, I’d probably use Spec to describe the exact API result to better support the transformation to a flatter data structure.

stathissideris17:07:21

Just released spec-provider v0.4.9, now with optional numerical range predicates https://github.com/stathissideris/spec-provider#inferring-specs-with-numerical-ranges