Clojurians Log v2

I’m practicing REPL Driven Development in Python. You’re not interacting with a running program in the same way as clojure, but the workflow is pretty much the same. I’ve written about it in this post (and also in a couple of Polylith-related posts): https://davidvujic.blogspot.com/2022/08/joyful-python-with-repl.html

Hum, I'm thinking maybe Mathlab and Mathematica might count. Julia might also count: https://docs.julialang.org/en/v1/manual/workflow-tips/ I would say R for sure.

But, as far as I know, Erlang and Elixir don't really offer repl driven development. They have shells (which sometime they call REPLs), but so those Java, Python, etc., and many other languages, but they're not really integrated into the development flow and meant to build the program as it is running. Well, it seems maybe Elixir has something akin to it. Hard to find by googling haha. But when I've done Elixir before, I did not see how to do repl driven development with it.

Python is pretty dynamic :) importlib.reload works great in my experience. I made a thing for reloading python source code without restarting the python interpreter in it. https://docs.python.org/3/library/importlib.html?highlight=reload#importlib.reload https://github.com/teodorlu/hotload

It's not as good as using the REPL (in my opinion), but it gets pretty close. Reloads take about 5 milliseconds on my system, and stays that low even when you pile on slow dependencies. The tradeoff is that you have to decide which modules to reload, somehow.

My minimum expectations are: 1. State can be maintained between reloads, or selectively cleared 2. Reference to the reloaded code are updated so they now refer to the new code Do you get that in Python?

Yes

But you cannot "send a form to the REPL", only save files and have those changed files be picked up. At least with the thing I wrote.

https://factorcode.org/

With Erlang and Elixir, one can connect to a running system, inspect its state, and modify code while running, albeit it's not very common to work in that mode.

@teodorlu The doc you linked says: > • Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired. That would fail my #2 no?

Consider importlib.reload a low level primitive that reloads one namespace (one module in Python terminology). It doesn’t do any magic, just reloads one thing. If you’re working with two modules, a and b where a depends on b (`a` contains import b), you’ll need to reload b first, then reload a. So in a sense, this is weaker than what you get with Clojure, because Python doesn’t have vars (as far as I know). When I use hotload myself, I keep track of those dependencies myself. I create a python module dedicated to reload what I need, and to run the code that I need. Then I use that module “as a REPL”. I save to see what I get. I could work with tests, prints, or just a bunch of asserts. As I noted, I prefer working with Clojure, where this kind of interactive / dynamic development has good built-in support 🙂 Edit: read your comment again. I agree, Python fails nr 2! Or at least, it fails nr 2 without a userspace solution. And you could (possibly) build that yourself.

While Python does have first-class functions, it does not encourage or enforce immutable data and is more geared towards working in the object-oriented paradigm. I'm not fixed on which attributes a programming language must exhibit to be considered "functional," but I feel facilities for working with immutable data are essential. https://www.modular.com/mojo, i.e., "Python 4.0," provides facilities for immutable data and supports optional static typing. And all valid Python code is (will be) valid Mojo code (that is https://docs.modular.com/mojo/why-mojo.html#compatibility-with-python).

Right, I mean, it's better than nothing, but I would not say it lets you do REPL-driven-development without those two criteria. I mean, that's just my personal qualification. I feel it's a bit like saying Java is a FP lang, or Clojure is an OO lang, yes, you kind of can use them like that, but it's not the full experience that would really let you understand why people like those things.

Agreed—it’s more interactive than a compile-run-loop, but it’s not REPL-driven development in my eyes.

> With Erlang and Elixir, one can connect to a running system, inspect its state, and modify code while running, albeit it's not very common to work in that mode. That's true, I feel it should be possible to make a REPL, not sure why there isn't one. Does Clojerl have a REPL?

It does, so at least it's possible, though not sure if it had to implement Vars as well, or it's just using Erlang's own code-reload

> I feel it should be possible to make a REPL, not sure why there isn’t one It’s possible to create a REPL for Python too, an editor-integrated REPL that you can send forms to. There are Emacs packages that provide just that. I tried using them, and I just didn’t like the workflow. Without clearly delimited syntax (s-exrepssions), re-evaluating parts of the code felt weird. I’d rather have more control myself, and stick to reloading full files. Then I tried, and that attempt turned into hotload 🙂

Interesting! I was not aware Clojerl had a REPL. I don't know why Erlang/Elixir people have not adopted a more interactive REPL-driven workflow since it is doable. It just hasn't caught on over in the Erlang/Elixir camp, I suppose.

Jupyter Notebook is kinda sorta a REPL if you squint and hold your nose.

Or… Rather, an attempt of implementing interactive programming, not a REPL.

This is what you would use I guess in Elixir: https://hexdocs.pm/iex/IEx.Helpers.html#r/1 Unclear to me if it properly reloads meeting my two criteria, or not.

These two articles show how it works a bit more in-depth: https://blog.appsignal.com/2018/10/16/elixir-alchemy-hot-code-reloading-in-elixir.html https://blog.appsignal.com/2021/07/27/a-guide-to-hot-code-reloading-in-elixir.html

In general, the attitude in the Erlang/Elixir community is: "This is super cool and a powerful feature, but you rarely want to actually use it. Use it only when strictly necessary." Similar to how we talk about macros in the Clojure community. Coincidentally, Erlang and Elixir also have similar facilities for macros, and they also echo this sentiment. 😅

I thought only Elixir had macros, or maybe they're just more convenient and Erlang also has them?

Well, I can see not wanted to hot-swap code in prod. But REPL-driven-development is for development time. So maybe there are caveats?

This is one example of repl-driven-development that meets my criteria: user=> (defn foo [a] (println (+ 10 a))) user=> (future (loop [] (foo 1) (Thread/sleep 1000) (recur))) 11 11 11 user=> (defn foo [a] (println (+ 100 a))) 101 101 101

It's not even that it knows to reload dependent code, it's that the actual references are updated even if the code is running. That's what the symbol resolution/var mechanism enables.

@teodorlu Ya, the lisp syntax really helps for RDD, because it makes it so easy to choose the granularity of what to eval. I often will eval subset of the code inside my function, it helps me with implementing the function.

AFAIK Erlang beats Clojure on dynamism. You can connect to and inspect a running CLUSTER, deploy code to it atomically with rollbacks, etc I don't know many Erlang developers so I don't know how they like to work but the tools are all there

Yeah, https://erlang.org/documentation/doc-4.7.3/doc/extensions/macros.html 🙂

That's a good point about S-expressions being easier to evaluate selectively in an interactive environment. I hadn't thought of that before.

Lisp syntax has nothing to do with a REPL, just with metaprogramming via macros, but other dynamic languages have macros or other solutions, like Tcl There are homoiconic langs like Red which get it right without s-exprs completely orthogonal

I was riffing off what @teodorlu said above: > […] I just didn’t like the workflow. Without clearly delimited syntax (s-exrepssions), re-evaluating parts of the code felt weird. […]

I think an important distinction is clearly parsable - you can trivially parse and symbolically manipulate Tcl code, if you squint even a bit Red code looks like S-Expressions. I think that's what matters - no impedance mismatch between the human reader's understanding of where the program parser's expression begin and end

Take a math expression. 2 + 2 / 3 * 10 (+ 2 (* (/ 2 3) 10)) How do you evaluate a part of this? You would be forced to highlight 2 / 3 if you wanted that sent to the REPL. With Lisp syntax, you just put the cursor after the paren. It's a lot more ergonomic and quick. So I do think Lisp syntax is ideal for interactive evaluation, because it really lets you navigate and indicate what chunk to evaluate most quickly.

Erlangs cluster machinery is impressive, and so is hot reload. But last time I checked, the recommended deploy method was a load balancer fronted fleet. Cluster setups are apparently difficult to manage, and the deploys can be trickier, so it seemed to be only recommended of you really need distributed computation with high level of communication between nodes. It also doesn't really give you anything for repl driven development. I guess it means it could, but still, I find I'd want RDD over the ability to production inspect/hot swap.

Also, Erlang and Elixir, I'm pretty sure, cannot hot swap a function, only a module. So the code reload isn't as granular as in Clojure. So in some ways you could argue Clojure is more dynamic, depending on your perspective.

They can, they can even hot swap a tail recursive function and keep the loop's state, basically the whole cluster's state is updated atomically

PHP gets plenty of valid criticisms, but it definitely has a fast feedback development experience.

It also got a few things right that I think are often overlooked based on its other quirks: • all state is stored in the database • state is updated via transactions • reads are done via relation algebra (ie. SQL) • views are mostly pure-ish functions of the state

I'm surprised no one has mentioned small talk. eric normand and gene kim have a series where they dive into the glamorous toolkit, https://www.youtube.com/watch?v=JuLZBocJ5jQ&list=PL4aJznIeHN6xtVARPWgT8SkbB2hh1I_0h. I would say small talk leans much more object oriented than functional, but is one of the best examples of interactive development

Objective-c has a very https://developer.apple.com/documentation/objectivec/objective-c_runtime?language=objc. You can basically do anything you want at runtime including reflection, loading new frameworks, creating new classes, etc. However, its usage is not typically interactive at all.

Oh, yeah! I forgot about Objective-C. If I'm not mistaken it was highly influences by Smalltalk.

That's what I've heard as well

and there's some old stuff that is really neat, https://jackrusher.com/classic-ux/

https://www.youtube.com/watch?v=rf5o5liZxnA

Check out "interface builder" section from the hci demos.

If you search for "flow based programming". Many of those are quite functional and are typically live coded. https://noflojs.org/ is one example.

I know people have their thoughts on excel, but it does IMO get a number of things right: • pure functions • immutable data • dataflow • interactive development

There are more experimental efforts: DarkLang https://www.youtube.com/watch?v=kylAbEPcadw

This is an awesome talk that covers a lot of the latest work for interactive development in academia, https://www.youtube.com/watch?v=1gGd7pKSpRM

Hopefully, this isn't too much of a link dump. I could keep going, but I'll stop here. Interactive development is something I've thought a lot about and there's still lots of unexplored design space!

It's all good! I've saved the links to watch on my flight from Norway to Japan on Wednesday if my five-month-old son can sleep 😅

@ben.sless I can't find any doc or info that shows you can reload at a more granular level than a module in Erlang? Do you have a link? It seems you can only swap in a new module, and references to the old module update to the new. But the old module still references itself.

My take is that Mojo was designed for the "pre-generative AI world" where lots of engineering teams and data scientist teams were building big codebases around a range of ML (SVM,Random Forests,RNN,CNN etc) with lots of bespoke tuning for their usecase . The new Generative AI world has focused on a small number of very powerful models (e.g. Transformers/Attention/Diffusion) and most engineers don't touch or tune these models. You don't need mojo to make Generative AI models better: • You only need a small subset of python to write these models (e.g. GPT can be written in 2,000 lines). • Most of the work needs to be run on GPU (using Torch/Tensorflow or raw CUDA) - Mojo can't make a GPU run any faster. • Some of the models are being re-written into C++ with raw CUDA for better performance (e.g. llama.cpp) - this still only gives a small improvement (e.g. 10%) over pytorch implementations. You don't need mojo to work with Generative AI: • Most users are using generative AI over REST - which means you can use any programming language (including Clojure!) with Generative AI. • Generative AI is incredibly slow and compute intensive - so most teams are not doing "big data" with it. I absolutely wish them luck with the project- but I can't help but feel generative AI is actually a set back for them.

From the landing page, it sounds pretty great. Now, it's still Python, so from a syntax/semantics standpoint, it's not super interesting. But from a runtime point of view, it seems to be amazing, like taking Python and re-implementing it to be performant, multi-threaded, fast, with stronger types and maybe static types, and additional syntax/semantics to let you do more low level programming with it as well. Honestly, wished this was done to Clojure 😛

He's got the chop to do it too. He made LLVM, Clang, Swift. So he got the know how to do this.

> Our core mission for Mojo includes innovations in compiler internals and support for current and emerging accelerators, but don’t see any need to innovate in language syntax or community > [...] > Embracing Python massively simplifies our design efforts, because most of the syntax is already specified. We can instead focus our efforts on building Mojo’s compilation model and systems programming features

Yeah, Chris Lattner is the GOAT regarding low-level language and hardware things like this. Chris and I have corresponded over LinkedIn and email, so I asked him about Clojure the other day. One of the main reasons for putting in the effort with Python and making Mojo a true superset is to lower the barrier of widespread adoption, given that Python has "won" in AI/ML. But I also would have loved to see this in Clojure. The Julia community was similarly saddened.

Perhaps we, the Clojure community, might be able to do some kind of interop thing with Mojo or its underlying infrastructure.

I listened to it too, loved it. So much juice. Great to hear the perspective of someone who understands both compilers and the use cases for fast code. Starting out with a solid high level language and extending it to be speedy when you need it to be makes a lot of sense to me. I'd say that @chris441 advocates for a similar approach using clojure. Just... Clojure is already flexible (data, macros), so there's no need to fork the language to start speeding things up. If you're curious, you can start https://m.youtube.com/watch?v=zYNlZXTV14E or https://m.youtube.com/watch?v=ralZ4j_ruVg.

> Perhaps we, the Clojure community, might be able to do some kind of interop thing with Mojo or its underlying infrastructure. > Might be interesting, there's already libpython-clj. Though Mojo is very early software, so I'd probably want to wait a bit for things to evolve before I spend too much time on it personally. https://github.com/clj-python/libpython-clj

I'd be very interested to hear what Chris Nuernberger thinks about Chris Lattner's work on Mojo.

I'm not sure if I've got this right. Still, it seems to me most of the complicated stuff and heavy lifting with Mojo will be in the interface to various hardware components, i.e., determining where to execute some piece of code and producing the most efficient instructions for specific hardware devices. I wonder how tightly coupled those lower-level innards will be to the Mojo language itself. If the coupling is loose, which I suspect it will be, knowing Chris Lattner, then perhaps those facilities will be available to other languages as well. If so, @borkdude will likely bang out some magical #sci based interface in a couple of weekends 😂 #data-science would go nuts for something like that.

I think Lattner mentioned plans for making a way to call Mojo code from normal python. If they make that, we could (possibly) just call that using existing libpython-clj functionality. Libpython-clj let's you call into python from clojure. It works by running a full fledged python interpreter next to the Clojure process. I was thinking of starting a Mojo process instead of a python process, and calling into that with the existing libpython-clj interface. Hosting sci on Mojo should be possible too! Then you don't need a JVM. Though sci hosted on Mojo might give worse performance than normal JVM Clojure calling into Mojo through libpython-clj. Though borkdude, Chris N or Chris L might prove me wrong!

Well, it seems that if you have Python code, you don't really get any benefits from Mojo, it'll literally run using CPython. But you can, one module at a time, rewrite to Mojo. And Mojo should be able to compile itself to many hardware accelerator and CPU. I guess we'll have to see what is different about Mojo from Python here. But it also means, it's unclear if it could host Clojure. First off, it might not be that dynamic, so maybe no REPL. And then maybe there's no IR, so you'd need to compile Clojure to Mojo. We'll have to wait and see.

Given the state of Python re: 2.7 vs 3.x, I would worry that this is just going to fragment things even more...? There are still Linux distros shipping with 2.7 (and core infrastructure that only runs on 2.7), as I understand it

@seancorfield Lattner discusses this in the podcast above episode as well. They expressly want to avoid the Python 2 → Python 3 nightmare. Lattner does not call Mojo “Python 4” for that reason; those were my words. Their goal is to be a true superset of Python, and their lofty goal is for all Python code, including hybrid Python + C libraries, should “just work” out of the box, as far as I understand.

So is Mojo a superset of Python 2.x or of Python 3.x?

Good question. I'm not sure! I don't recall hearing about that specifically.

According to the podcast episode, Mojo is a strict superset of Python 3 :)

In my mind, I have tagged Mojo with “Rust, but it’s Python with optional strict typing” 😅

The wiki says it runs Python code using CPython, so it should be 100% compatible that way

@didibus If I understood correctly, the long-term goal is to not use CPython at all, but it's required until Mojo is "done."

I feel this: https://docs.modular.com/mojo/why-mojo.html#compatibility-with-python implies a bit otherwise. Unless they mean that eventually they think it'll be 100% compatible. But I get the impression it maybe wouldn't? Like would the C/C++ interop also work exactly the same? I understand the, we will support 100% of the syntax and constructs of Python, and be a superset. But still, that can mean that many lib don work perfectly because of all kind of minor differences. So I thought they meant that basically python files can be mixed in with mojo files, they'd run with CPython, and mojo files would run with mojo, but it would be transparent to the programmer and both would have access to each other. I might have gotten that wrong though.

Especially this part: 1. We utilize CPython to run all existing Python 3 code without modification and use its runtime, unmodified, for full compatibility with the entire ecosystem. Running code this way provides no benefit from Mojo, but the sheer existence and availability of this ecosystem will rapidly accelerate the bring-up of Mojo, and leverage the fact that Python is really great for high-level programming already. 2. We will provide a mechanical migration tool that provides very good compatibility for people who want to migrate code from Python to Mojo. For example, to avoid migration errors with Python code that uses identifier names that match Mojo keywords, Mojo provides a backtick feature that allows any keyword to behave as an identifier Together, this allows Mojo to integrate well in a mostly-CPython world, but allows Mojo programmers to progressively move code (a module or file at a time) to Mojo. This is a proven approach from the Objective-C to Swift migration that Apple performed

Ah, right. My bad. I suppose the CPython dependency will always be there to maintain compatibility with those libraries partially written in C that do not migrate the C-parts to Mojo. Although I suspect most major libraries will migrate to Mojo to have the whole codebase in one language and for increased performance. But there will likely always be some Python libraries or code bases that remain partially written in C.

So https://www.modular.com/blog/weve-raised-100m-to-fix-ai-infrastructure-for-the-worlds-developers (totaling $130M) to develop Mojo and other related things. I've never seen a new programming language receive so heavy financial backing. It will be interesting to observe what happens.

What?! You can see my view above that I'm a bit bearish / unconvinced by them at the moment. But clearly investors aren't! What do you think I've missed or have they missed?

Probably so much crap runs on python that plenty of companies stand to make lots of money if mojo serves as a drop in replacement

Good point. There are a few different implementations for Python (CPython, PyPy etc) [https://wiki.python.org/moin/PythonImplementations] but most companies are happy just going with the main implementation which isn't the fastest. The original presentation for Mojo also spent a lot of time talking about machine learning in particular.

globally speaking, that investment of $100m will likely pay for itself in infra cost savings, but can that value actually be captured by the company undergoing getting the funding? hm, almost as though investment in programming languages is better thought of as a public good than a profit generating investment 🧐

OR The investors are also invested in tech firms which are heavy users of Python Investors work at a different scale than most people

that's still a public-good justification imo; lots of other public goods increase the market value of things (e.g. transit improvements and real estate)

As the resident cynic https://www.generalcatalyst.com/portfolio

Every one of those companies is capable of making bank from AI, doing so with low costs is a force multiplier

I feel there's a lot of AI use that large models and generative AI are not a great fit for no ? Many enterprises I feel have specific needs over their own datasets for very precise tasks. Are LLMs and generative AI really better here? Seems both overkill and possibly harder to leverage and get the expected results no?

This is probably pushed by VC - even if one company benefits from it, it makes up for the investment on all others by orders of magnitude

> likely pay for itself in infra cost savings, I guess my point earlier was that there was already much faster ways to run Python for free (e.g. PyPy) - but most companies don't use it. A lot Python code for calls out to C++ code for compute intense elements anyway so that won't benefit really. Another data point, if companies use the mainstream cloud which often are 10x more expensive for compute vs bare metal alternatives - they probably don't care much about the compute cost. > Are LLMs and generative AI really better here? LLMs basically beat all other AI models for NLP/text - across practically every usecase (translation, summarisation, classification, question answering, text generation etc etc) - so LLMs are the go to model now for all these tasks. LLMs doesn't work so well on structured data and timeseries - but I think you will see Enterprises switch their focus away from these and focus on text/NLP for a while.

The VCs poured a lot of money into crypto and the bottom is falling out of that market -- but a lot of crypto companies are pivoting to AI and VCs are following them and pouring money into AI now. /cynic My experience with VCs, here in the Bay Area, is that they'll pour huge amounts of money into anything that is "hot" -- but often won't fund good, solid ideas and/or mainstream tech because they are so focused on the (potential) high return of a few unicorns and don't want "safe" investments. (I have a very cynical view of VCs -- and a lot of the "startup culture" they've created in this area)

Mojo now claims to be https://www.modular.com/blog/outperforming-rust-benchmarks-with-mojo.

Julia people claim the implementation is incomplete and meh/10

Yeah, and Rust people complain the source code is not available for review. And that they "forgot" to turn on release mode, etc.

It would be nice if they included a numba version too to compare