being able to use remote inspection tools and debugging tools is an immense superpower of Clojure (and other lisps). there are a number of problems that come up - some are not technical so much as organizational or policy related, like who gets to repl into production and what can they do there.

esp when your production is, for example, using PII or regulated data

but putting those important concerns aside, there are interesting technical problems with getting data over the wire to a local tool, and we've been working on some stuff in this area this summer (still some wip) but eventually we hope to have more building blocks here

I'm excited to hear what comes of that work. Thanks for sharing. 🙏

@U0DJ4T5U1 I'll put my hand up as someone who runs (socket) REPLs in production and uses them for "interactive debugging". I'm contemplating ways to expand that without adding too much "bloat" in terms of dependencies in production. nREPL isn't too bad (no deps) so we'll probably add that. Portal brings in Transit (which includes Jackson 😞 and msgpack etc) and Google gson and http-kit so it's... OK... but a bit more than I really want in all our production apps. So we'll probably add nREPL and some code to start it based on a JVM property, like we do for Socket REPLs today (since that gives us more options in terms of clients to connect). And we may add Portal to our Admin (internal) app since we could ssh tunnel and use a web browser against it I think which would give us visualization.

In the course of doing some SOX-compliant / regulated FinTech work I’ve had it be acceptable to get “read only” (Ruby) console on the prod instance — which was really not about the running Ruby interpreter but about RO/RW connection creds to the DB/state, with an audit log available of who was escalating to RW DB access. It would be super amazing to have some trusted IAM layer for the interpreter and/or DB query engine, which would both networking, security and operations all much easier!

(Which presumably creates a design problem for modeling both (hand-waving) something like “operations,” “objects,” “actors” and “permissions” — but hopefully in CLJ land that’s all describable via namespaces, functions, data, schemas and specs…? e.g. (handwaving)`:db/sensitivity db.sensitivity/pii` for a particular attribute)

There's Clojail and a few similar attempts at making restricted evaluation environments, but in a regulated env, I think you'd have to stick to a pure R/O DB connection and a separate app/REPL that had just that DB connection and no access access to the running apps themselves.

But if you have any state saved into global atom then you can't have R/O without a jail :thinking_face: i.e. you can't connect to the same process then, you have to start your own. But then it's not actually production anymore

I've been using REPLs in production in every Clojure production system I've used and it was of tremendous help when inspecting, hot-patching or migrating data. Of course you have to be super careful!

I think there's a few different problems that people envision when we start talking about this problem: 1. Experimenting with new things using production data & infrastructure 2. Troubleshooting production system problems 3. Updating existing production systems

at my current company, we have a specific service that provides a REPL into our production stack. you can then send commands to different services that we have deployed. this is very helpful for solving (1), somewhat helpful for (2), but only slightly helpful for (3), since it basically allows us to query, do migrations & fix data via our service APIs, but doesn't allow us to change the code that's running there

I've patched a function once. Which would have been much harder if direct linking was enabled, since you need to track down every (transitive) usage and also patch that callsite

(I should also say, we have strict process around who gets access to this service)

for troubleshooting issues we also use tools like metrics, distributed tracing, and logs. the REPL helps us query the system as it understands itself, but the runtime behavior is often better captured by these specialized observability tools.

when updating existing systems, we use feature flags to roll out the change and use the same observability tools to observe our changes.

a REPL could perhaps augment these things, but IME so many orgs fail at getting the above two things online & working, and IMO they are far more necessary than a REPL

IDK, maybe having an easy REPL connection would allow teams to add more observability ad-hoc when production issues occur?

and that could lead to more adoption

We have a repl in all our environs, and honestly it's saved us a number of times. > add more observability ad-hoc > Absolutely. Usually when third party services are failing for insane reasons. One where auth0 was returning a 500 when a request was made with an elastic apm header. (They patched it after a few days) I've also used it to hotpatch critical issues (or even just using it to disable features) temporarily while working on a full PR. Be careful doing it, but good God I wish I had it in every language.

Just today I was REPLed in and tweaking code, as it happens — not on a true production server, to be fair, but on a server used for external-facing demos. > What repl flow has worked for you such that production feels alive and something you can easily communicate and inspect? To Rich’s point, how can we explore it further? Personally my dream is of having a true Common Lisp-style breakloop in Clojure, as described https://mikelevins.github.io/posts/2020-12-18-repl-driven/. There are significant obstacles due to the JVM’s exception handling and lack of continuations. I was able to take a short stab at it https://clojurians.slack.com/archives/C0617A8PQ/p1645556992259289 but couldn’t get there. I may or may not have the chops or the knowledge to pull it off at all, but I sure wish that someone could (and I’ll probably keep trying periodically as I have time). Other items on my workflow wish list (not all from historical Lisp/Smalltalk environments but in the same spirit): rich time-traveling debuggers. REPLs that automatically preserve the last state of local bindings in every function that’s been called and make it available on-demand for inspection and/or calling functions with complex inputs from the REPL (in other wonds: REPLs that automatically do what the wonderful https://github.com/vvvvalvalval/scope-capture lets you do manually). REPLs with affordances to make it easier to move REPL experiments into finished code (here’s https://github.com/eggsyntax/reconstructorepl). REPLs with rich built-in introspection along the lines of https://practical.li/clojure/clojure-cli/data-browsers/portal.html and https://github.com/vlaaad/reveal and my own (more limited but fully in-repl) https://github.com/eggsyntax/datawalk.

Thanks for sharing @U077BEWNQ lots of good ideas in there. I should be using the scope-capture lib all the time...

I regularly develop our back office stack with a REPL, and do ad-hoc data queries to the production db (using all of our functions that generate queries etc). However, getting debug access to the production servers is complicated mainly because of the traffic (how can I capture the exact request I want?) and the scaling (which server will get to answer my request?). There’s probably ways around that but I haven’t paid too much attention since most of the time we can reproduce locally with a copy of prod data if needed.

@U7PBP4UVA those are interesting questions. Your right, cross-server questions can't be solved by asking only one server. Many of the questions i run into typically could be. As to capturing the right request, the right solution would probably depend on the context, but i often have to do that locally for various reasons and i usually set up a log via a vector in an atom. But i think the flowstorm debugger would be a great help here to given it can instrument. To that point, instead of bringing the repl to production, if you can bring all the traffic to the local env, i think that would serve 80% of the same purpose (the other 20% being config diffs between envs that could be rather annoying to notice). Maybe setting up a kafka esq type log that the server reads from instead of getting the HTTP request directly?

For seasoned clojurist, have you ever found any other languages that made you want to leave clojure? I'm mostly just wanting to learn a new language... and stretch my brain, but its not easy to find anything appealing after clojure. Julia maybe comes close. Looking for practical suggestions, not purely academic languages.

Nope. Idris was an interesting brain stretch, though; it takes sophisticated typing rather further than even Haskell. Erlang/Elixir seem really interesting to me also, though I haven't used them.

@U0J3J79FE I worked my way through "Seven Languages in Seven Weeks" (and "Seven More Languages..." but that's not really as interesting), and I've also tried to pick up a new language every couple of years per "The Pragmatic Programmer", just to see what's out there. Nothing so far has made me want to leave Clojure. I think if someone forced me to do a different job, I could live with Kotlin. But I've learned a lot of languages over the decades and nothing makes me happier than Clojure. That list, going backwards, is roughly Kotlin, Rust, Go, Elm, Python, Ruby, Standard ML, Clojure, Erlang, Scala, Groovy, CFML (ColdFusion), Java, Perl, C++, C, COBOL, FORTRAN, APL, Prolog, Lisp, Occam, various assemblers, pl/p, Pilot, PL/1, Algol-68, Pascal, BASIC, Algol-60. I may have missed a few 🙂 CFML is a dynamic, compile-on-demand (to JVM bytecode), OO scripting language with some metaprogramming and a bunch of "functional" features too. You might be surprised at what modern CFML looks like! 🙂

(that list doesn't include things like Io from 7Li7W because I didn't really dig deep enough into those, if I didn't already know them)

Things to stretch your brain: APL, Prolog.