Clojurians Log v2

What makes a Lisp? YS transpiles to Clojure and Clojure is a Lisp. Is it the syntax, the AST or something else? The transpilation of ys to clj is a 7 stage process as seen here https://gist.github.com/ingydotnet/8aacb48e83d9cd9929aabc22dd3e92ea There are 2 important AST states: • https://gist.github.com/ingydotnet/8aacb48e83d9cd9929aabc22dd3e92ea#file-tmp-xh53rwp0up-L22-L28 is clojure parts still in the original yaml input structure • https://gist.github.com/ingydotnet/8aacb48e83d9cd9929aabc22dd3e92ea#file-tmp-xh53rwp0up-L42-L52 is a full clojure AST ready to Print

My answer to "Is YS a Lisp" is a resounding NO. As can be read in https://en.wikipedia.org/wiki/Lisp_(programming_language), Lisp notation is either M-expressions car[cons[A,B]], like used in Mathematica, or indeed S-expressions. YS uses YeS expressions, which is not Lisp, but nevertheless retains the "Code is Data" paradigm.

it can use ysexprs and other sugar forms that translate directly to sexprs. Certainly there are other Lisps that do similar?? You can also code YS in all sexprs...

@markus.agwin not refuting you just growing the convo

My answer is currently resounding MAYBE???

Well, I didn't realize I would be pulled into a religious war 🤣. I don't think I can give a definitive answer to this. However, I can provide some thoughts from my experience of personally implementing mal in a few dozen languages. I think there are two aspects that make a language Lisp-like: syntax and internal operation/mechanisms. The syntax level and questions of homoiconicity are actually the harder level to answer for this question. I've had many discussions over the years with colleagues (including early Clojure contributors) about what exactly that term means and everybody has a different opinion. One aspect of homoiconic languages is that you should be able to round trip from source code to AST and back to source code. Almost no popular/modern Lisp complete fulfills this requirement due to macros. Reader macros in particular mean that you can't get back from the AST to source code. But even regular macros break this ability. You could in theory track macro information in metadata or something to be able to get back to the original but I don't know of any language that does that. fexprs are one way of getting the power of macros in a more homoiconic way. If YS is thought of as a set of complex macro transformations from the original source to the "true" source form, then you could argue that YS has a form of homoiconicity I suppose. As far as I can tell, YS doesn't have a full built-in macro system yet right? I think full AST macros and a native eval meta-circular evaluator (function that can evalulate YS code directly) is probably important for YS to make claim to a Lisp heritage. The ability of macros to dynamically generate and evaluate an AST during program read/compile/eval is definitely less easy to use if the AST differs a lot from the default syntactical representation, however. I've even heard it argued that Clojure's hash-map literals mean that the language is not fully round-tripable and therefore it is not 100% homoiconic because key order is not maintained. That's the sort of bike-shedding that happens when you try and nail down something like homoiconicity. The second aspect of internal operation/mechanism seems like it would be even harder to answer. But over the years I've come up with a good rule of thumb: how easy and concise is it to implement a mal interpreter in that language. What I've found in practice is that the more "Lispy" the internals of a language are, the easier it is to implement mal using that language (the language syntax is much less important when it comes to this). And likewise, creating a mal implementation of mal in a language will expose which parts of the language are Lisp-like and which are not. To make it concrete, Ruby is a very Lisp-like language. It very much feels like Ruby is a Lisp language with a object-oriented skin around it. For example, compare the core library of mal implemented in https://github.com/kanaka/mal/blob/master/impls/ruby/core.rb as opposed to https://github.com/kanaka/mal/blob/master/impls/java/src/main/java/mal/core.java . I haven't done an implementation of mal using YS (nor used YS in anger at all TBH), so I can't really say how Lisp-like YS feels. From poking around in the code and docs my guess is that the internals/operation are probably quite Lisp-like, especially given it's inspiration in mal and the fact that it compiles to Clojure/SCI. Whew, that became a bit long-winded. I probably should have broken up that more but that's what comes to mind. Was that type of non-commital answer what you were looking for 😄.

@clojurians115 that was a gorgeous response and I appreciate you taking time to do it. No war at all. I've only stated that YS might be a Lisp and might be homoiconic. Really I just wanted to start some fun conversation. No other agenda for this. @phill brought up homoiconicity here in April: https://clojurians.slack.com/archives/C05HQFMTURF/p1713486686334549?thread_ts=1713474866.964849&cid=C05HQFMTURF and @danielmgerson was interested when I visited him in London in May https://clojurians.slack.com/archives/C05HQFMTURF/p1717843326903329 I have more to respond to but will use separate msgs...

Yeah, I haven't read back in the channel, but I'm sure my elaboration above was probably repeating a lot of points already made here.

In the end, I think you'll need to finish your mal implementation in yamlscript and then you can tell us all how "Lispy" it is 😉. BTW, on the code-as-data front, it does strike me that there might be an opportunity for an alternate syntax of YS that is closer to sexprs. I.e. something that translates more directly to the intermediate "build" form of the AST. Not saying you should do that, but it might be interesting to explore.

More about that in https://clojurians.slack.com/archives/C05HQFMTURF/p1723577426565109 Definitely going to do the YS/MAL PR sooner than later. I'll just port my Perl (a very Lisp language that was the main inspiration for Ruby!). I suspect I could get the PR done in a few hours. ^^ Total Hacker Hubris 😕

> I think there are two aspects that make a language Lisp-like: syntax and internal operation/mechanisms. Totally agree. YS has a Lispy model, but usually not a Lispy syntax. Roundtripping source->ast->source isn't a thing yet, but is on my radar. I'd have to pick a canonical style for various forms, where a Lisp barely needs to. But that's not uncommon in languages like Go where where canonical code style is nearly enforced... When I get around to a full nrepl story I'll likely need to take this on...

exactly that term means and everybody has a different opinionSame perception. I was (pleasantly) surprised when @phill was saying (essentially) that YS was "homoiconic where it mattered"

> Ruby is a very Lisp-like language. I was in (Perl inventor) Larry Wall's kitchen talking with him and his wife Gloria when it came over the radio that John McCarthy had passed away. Larry told a funny story about a very early Perl presentation where he described Lisp as having all the visual appeal of oatmeal with fingernail clippings on mixed in. Unbeknownst McCarthy was in attendance and nonplussed, but Larry said "I only meant Visual appeal; otherwise Perl was completely inspired by Lisp. Of course these days Perl is usually compared to a dumpster fire 😂 From the Ruby wikipedia page: > According to the creator, Ruby was influenced by https://en.wikipedia.org/wiki/Perl, https://en.wikipedia.org/wiki/Smalltalk, https://en.wikipedia.org/wiki/Eiffel_(programming_language), https://en.wikipedia.org/wiki/Ada_(programming_language), https://en.wikipedia.org/wiki/BASIC, https://en.wikipedia.org/wiki/Java_(programming_language), and https://en.wikipedia.org/wiki/Lisp_(programming_language).

That's a great anecdote. I've never met Larry Wall but from what I know he seems like an very interesting person.

Likewise McCarthy, although a very different personality I would guess.

My father is also a linguist by trade so Larry's writings had a deep resonance for me. I was a perl enthusiast early on. Less so after learning python and then later Clojure. A lot of the perl philosophies have come to be seen as anti-patterns.

> As far as I can tell, YS doesn't have a full built-in macro system yet right? Not yet. It has internally hard coded macro-like functions that happen mostly at the 6th (transform) stage of compilation. There have long be plans to introduce a user facing defys system for writing YS functions that alter the AST(s) during compilation. One main thing preventing this atm is deciding the AST is stable. Can't ask people to change their defys s every time we change the AST. Way less to worry about in a Lisp! > If YS is thought of as a set of complex macro transformations from the original source to the "true" source form, then you could argue that YS has a form of homoiconicity I suppose. Interesting. YS is a fancy YAML "loader" which the spec defines in stages. It loads YS in all the normal ways a spec compliant loader is encouraged to, but adds a couple stages. Then sprinkled in are some tweaks to make it DWIM. The idea is that eventually we have a stable AST (or maybe an AST API abstraction) and then all the special tweaks are pulled out and reapplied as defys macros. This means of course that users can change a lot of how YS looks just like I'm doing now. And in a lexical (probably file based) scope. Execution/compilation order also has difficulties to solve. I think ClojureScript's macros had similar hurtles. YS being Clojure can still define Clojure defmacro macros, but of course that's not what we're talking about with defys.

> My father is also a linguist by trade so Larry's writings I recall realizing well after uni that I probably should have doubled majored in Linguistics 😄 Full respect to those that have. Larry was always more into linguistics than programming languages, I've observed. Gloria is also a linguist. They met in Uni!

Agree with Markus's sentiment. I also don't want to get stuck in definitions. If you use YAMLScript as a macro language for YAMLScript and it has first class REPL support, then I'd say yes it is. Or rather that's the interesting part of homoiconicity that I care about...

The heck. Even Einstein developed his General Relativity under a guiding Principle. Turns out this "Mach Principle" had come in many formulations, all blurred, and in the end some solutions of GR obeyed some of the Mach-versions, in some situations not at all. Einstein kept insisting on the Principle, calling solutions not obeying as "non physical". In that sense homoiconicity is an example of the guiding principle no creator can do without.

> first class REPL support @danielmgerson when I think we've accomplished that, I'm going to ask you to confirm

I suspect @danielmgerson and @phill will have things to say here. 🙂

This is a tough one. I rewatched the https://www.youtube.com/watch?v=o7zyGMcav3c&t=1335s on the subject. Hmmm. I would give the "it depends" answer. I'd say, tentatively, we will see in its use whether YAMLScript is homoiconic. I think any language that is alive is defined in a large part via it's use. If people are going to write Macros for ys in Clojure, I'd say: "homoiconic? rather not". Moreover, I'd like to make the distinction between 1) libYamlScript and 2) YS-Cli. with 1) libYamlScript, YS is used to generate JSON-structures, so the data resulting out of the code is never homoiconic. All the more so, the reason that ys is used within, say, Python, makes the whole discussion about homoiconicity ridiculous. with 2) YamlScript CLI, one can do ys --load --yaml so the produced data can be made homoiconic. All in all, I am not sure whether it is really necessary to clarify this point. Better settle it case by case referring to the concrete problem that is solved using YAMLScript.

> I’ve always suspected that a Lisp could be encoded in JSON (thus YAML since JSON is YAML). Lisp code is data, after all. 😃

That's a bit oversimplifying things. Imagine rewriting all your Clojure using only EDN syntax. How would you go about it? That's what miniMAL is doing but for a minuscule subset of Clojure!

To be usable, you need more syntax that you can get out of JSON/EDN. YAML has hidden gems to exploit here. 🙂

To me an interesting aspect of it is that YS is first parsed (not loaded) as YAML. A YAML parser turns YAML text into a stream of 10 different events (typed structs of information). So that's what YS gets to work with and decided what means what.

A new language designer gets to work with a stream of characters. Endless possibilities but also endless decisions required to make. YS has only YAML events, limited but quite useful. JSON has a maps, vecs, strings, nums, bools, and nil. So minimal. At least EDN adds keywords (and comments!) 😄

Yeah, my comment was very much tongue in cheek. > Imagine rewriting all your Clojure using only EDN syntax. How would you go about it? To me Clojure syntax and EDN syntax look very similar. But I am not a language designer.

@pez, sometime (in your copious spare time) try rewriting https://github.com/BetterThanTomorrow/calva/blob/published/src/cljs-lib/src/js_cljs/core.cljs in EDN for me. 😉

Looking over https://github.com/edn-format/edn EDN adds a pretty big handful of things over JSON.

@clojurians115 ednMAL? 😉

@ingy Haha, that's just Clojure minus reader macros and minus regex and plus tagged types. Or to put it another way, EDN is the round-trippable part of Clojure syntax.

Here is my first pass at ednMAL (higher than most other Lisps on the homoiconicity scale) 😈:

user=> (require '[clojure.edn :as edn]) (loop [] (let [l (do (print "> ") (flush) (read-line))] (when l (println (pr-str (eval (edn/read-string l)))) (recur))))
> (+ 2 3)
5
> ( (fn [a b] (* a b)) 7 8)
56

> Haha, that's just Clojure minus reader macros and minus regex and plus tagged types. @clojurians115 derp, I totally spaced that EDN was that capable.

$ ys -pe 'clojure::edn/read-string("'\''foo")'
'foo

iirc it's why ys compilation doesn't compile its anonymous function syntax to clojure's anonymous function syntax.

$ ys -ce '\(%1 * (%2 % %3))'
(fn [& [_1 _2 _3]] (*_ _1 (rem _2 _3)))

It's great seeing how far you've come! The exorcism examples are illuminating.

@delon exercism (like exercise) not exorcism (like demons)

not sure that was the best name choice for them, but I guess a little cringe helps people remember it 😄

Is it #exercism we’re talking about?

Ah cool! Thanks @pez!

I'll have to announce there when I launch.

actually I might have some questions about expectations for them

I think @porkostomus can help you with expectations. He’s been in rather close contact with the core Exercism team, iiuc.

I think #announcements is the place to announce a YAML Exercism track, btw.

and there of course. (YAML*Script*) 🙂

Yeah, haha.

Interesting. That's exactly what I didn't want YS to look like. I wanted programmers to like coding it.

ys -c <(echo '

!yamlscript/v0
say "Hello": 2 + 3

')
(say "Hello" (+_ 2 3))

SPOILER ALERT: This link contains all the current solutions and test files for the upcoming YS exercism track. (They are public in a public repo anyway) This is a good way to review a bunch of real YS programs as the language sits today... https://gist.github.com/ingydotnet/aebc6d93f284394ddc861ff78a4e84ec I'm very happy with almost all of it. But also interested in suggestions!

I should mention that almost all new ideas I get for YS come from programming in it. Just like that first time I ported my perl mal code to YAML 😄

YAML has all these "features" in the spec that normal users don't bother with. Having invented YAML I know all the nuances and put them to good use in YS. I've always suspected that a Lisp could be encoded in JSON (thus YAML since JSON is YAML). And I've made project DSL ASTs from JSON in the past. There's these various parts of YAML that allow it to be turned into a fairly clean looking programming language, but its not immediately obvious even to me. It's been a fun problem to work on...

(didn't realize I wasn't threaded there...)