Clojurians Log v2

is there a way to go into the debugger on failing test?

so I can debug it. until i’m putting in more and more print statements and then running the tests, waiting 10 or 15 minutes, test fails (or not), keep running over and over until fails.

then putting in more print statements.

I’ve tried to find out which input causes the failure, but until now, when I run with that same input, as a small single test, it doesn’t fail.

> is there a way to go into the debugger on failing test? Yes, put a breakpoint in clojure.test/do-report when (:type m) is :fail. > waiting 10 or 15 minutes I'd first work on reducing the test suite size while retaining the failure. Ideally, you should be able to trigger it every few seconds, not multiple minutes. Do your tests have any kind of a shared state? Including the file system.

or class generation/compilation

loading namespaces in different orders can also be stateful

there is indeed shared state. I’m memoizing several functions. As far as I recall, the only shared state is in function memoization, unless I’ve forgotten something.

(clojure's own tests fail prey to this)

But even without a debugger, a test failure should have all the information about its location and why it has happened. There will be some value that doesn't satisfy some condition, that value would come from some function - you're interested in what's going on there and elsewhere upstream, not what's going on at the point of the failure.

well the test does print the input which causes it to fail. however, when I simply run on that input it doesn’t fail. So somewhere there is some accidental global state I’m guessing

I didn't mean the input for the test. I meant the input for (is ...).

Here’s the test after I’ve refactored it to get as much information as possible.

(deftest t-discovered-261
  (testing "discovered test case 261"
    ;;   actual: java.lang.AssertionError: Assert failed: 187: dfa not equivalent with self rte=(:contains-every :epsilon (= (1 2 3)) (satisfies seq?) (:and java.io.Serializable java.lang.CharSequence (:contains-every (:? (:contains-any)) (:and (:? :epsilon)))))
    (doseq [r1 ['(:contains-every :epsilon
                               (= (1 2 3))
                               (satisfies seq?)
                               (:and java.io.Serializable
                                     java.lang.CharSequence
                                     (:contains-every (:? (:contains-any))
                                                      (:and (:? :epsilon)))))
                '(:cat (:contains-every (:not (:cat (member a b c 1 2 3)))
                                        (satisfies seq?)
                                        (:? :epsilon))
                       (:or (= (1 2 3)) (:cat :epsilon (:or (= -1)))
                            (:+ :empty-set))
                       :sigma
                       (:cat (:or (:cat (:and)) :empty-set)
                             (:or :sigma (:contains-every (:+ (:and))))
                             (:cat :epsilon (:not (:contains-none)))))]]

      (with-compile-env []
        (let [dfa (rte-to-dfa r1)
              dfa-complement (xym/complement dfa)
              dfa-not-rte (rte-to-dfa (list :not r1))
              eq1 (xym/dfa-equivalent? dfa
                                       dfa)
              eq2 (xym/dfa-equivalent? dfa-not-rte
                                       dfa-not-rte)
              eq3 (xym/dfa-equivalent? dfa-complement
                                       dfa-not-rte)
              ]
          (if (not eq1)
            (let [abbrevs-eq1 (dot/dfa-view dfa "eq1")
                  abbrevs-eq1-xor (dot/dfa-view (xym/synchronized-xor dfa dfa) "eq1-xor"
                                                :abbrevs abbrevs-eq1)]
              (pprint [:eq1 eq1])
              (pprint [:abbrevs-eq1 abbrevs-eq1])
              (pprint [:abbrevs-eq1-xor abbrevs-eq1-xor])
              ))
          (is eq1
              (cl-format false
                         "z1: dfa not equivalent (returned ~A) with self r1=~A" eq1 r1)))))))

tests don’t really take 10 minutes. I’m using time make tests and it finally prints the following

408.983u 3.472s 3:41.31 186.3%	0+0k 0+0io 733pf+0w

3 min 41 sec wall time?

Unfortunately all this gives us exactly zero usable information because we don't know what the underlying functions do. If (is eq1) fails then it means that (xym/dfa-equivalent? dfa dfa) returns a false'y value. Why? No clue - it all depends on what dfa is exactly and what the implementation of xym/dfa-equivalent? is exactly. It's this repo, right? https://github.com/jimka2001/clojure-rte Are the failing tests in the master branch?

@p-himik, yes I showed the code because you commented about the input of (is …) I knew well that seeing the is would not be revealing. What should I say? “No, i’m not going to show you because it won’t be helpful”? Sorry perhaps I should have said that.

I need to debug this myself. just was hoping for some ideas about how to do this better than what I’m doing.

w.r.t xym/dfa-equivalent? yes, it seems to my my equivalence function should be able to decide that an object is equivalent to itself. indeed.

looks like it has a bug, but curious that the bug does not seem to be deterministic. There are some places where I iterate over a hash, and perhaps there’s a bug there, as it may not always iterate in the same order.

I didn't mean that you need to show us the code. :) What I meant to say is that putting a breakpoint where is fails is useless - it will not give you any new information. You already know which is reports a failure, you already know the call chain up to that point. You might not know some specific datums inside that call chain - that's where the breakpoint should be, not inside the machinery of is.

> It’s this repo, right? https://github.com/jimka2001/clojure-rte

yes that’s the repo. indeed.

in the test, when it fails, I have inserted a call to dfa-view when generates a png of the image of the DFA and opens it in a web browser. I’m trying to figure out why it fails to be equivalent to itself.

but not sure why it only fails sometimes and not always or never

wrt (is …) my idea was to breakpoint there and then examine the object interactively. or maybe call equivalent? again.

perhaps such a strange bug SHOULD take days to debug.

I have a feeling I’m going to learn something important about my program once I figure out why this is happening.

I'll try to reproduce it. Just FYI - the trash command isn't available everywhere. I do not have it, for example. If it's a truly temporary file, as indicated by the name tmp-1, I'd just use rm.

will rm work on windows?

perhaps I should some something old os.this.and.that.remove.directory….

on the mac trash just moves to the trash and I have to empty the trash now and again.

I could of course abstract over this.

good suggestion

OK, now the problem seems to have gone away.

4 runs, and no error.

Ah, Windows, right. No clue. In that case, .

not sure if that will delete a directory. I’ll have to do a bit of research

but yes, thanks for the suggestion

BTW, the function where I’m calling “trash”. What it is doing is opening two csv files, one very small one and one large one, and merging the files together alphabetically, (or perhaps numerically) but deterministically.

perhaps there’s a better way to do this using the clojure csv package???

should be a linear operation because the large file stays sorted.

If you don't need to order strictly field-wise and can get away with combining row-wise (i.e. without any regard for separators and escapes), you can even get away with just plain line reading, sorting, and writing.

I don’t find a merge function which is similar to sort but assumes the two inputs are already sorted.

merge should have linear complexity, while sort has n-log-n complexity

but using UNIX sort -m means I don’t have to read the data into JVM memory

nevertheless, doing it all in clojure would eliminate the silly OS/MAC/LINUX/Windows incompatibility.

BIZARE that the failure has stopped happening with the most recent debugging tweak statement that i’ve added.

One unrelated thing - you depend on metasoarous/oz which depends on a lot of stuff. Not the end of the world, but I would never add something like this to my dependencies. Maybe only inside a dev-only alias. But not as one of the main dependencies. Regarding merging sorted collections - I'd use something that already exists, like com.google.common.collect.Iterables/mergeSorted in Guava or org.apache.commons.collections4.CollectionUtils/collate in Apache Commons Collections.

i wonder whether i’m still using oz.

it was using it for vega plot, but i’m moving away from vega writing stuff in gnuplot instead

I don’t understand the comment about adding to dependencies. If I have functions what use vega plotting, and I want to test them, then doesn’t that need to be reflected in the dependencies?

If the main functionality of your library depends on something then yes, of course you need that something in your dependencies. If a secondary but still user-facing functionality of your library depends on something, you can either add that something to your dependencies or document that users of your library must add that something to their dependencies along with your library. That way, people that don't need a particular secondary functionality won't have to deal with extra dependencies. Of course, anyone can also use exclusions, but that's still something that should be documented. If a dev- or test-only part of your library depends on something, that something should be in a separate alias that isn't used when deploying the library to Clojars or anywhere else.

I reproduced the issue. When the test fails, the result of (synchronized-xor dfa-1 dfa-2) has 16 states. When it succeeds, the DFA has 6 states. I have no idea about anything in your project, but I assume that's not a normal behavior, right?

BTW, I think I have a clue as well.

look at state q12[2,2]

it has to out-going transitions with the same label. this should never happen.

Nah-nah, you won't make me think! :D I'm just using my spinal cord here.

this is a bug.

this is a very big clue.

still not sure why it only appears some of the time.

Ah, even the results of (rte-to-dfa r1) are different.

well it loops over r1.

I need to go home. i’m still in the office.

I’ll look more tomorrow.

it could be that this is caused by looping over the entries in a hash table. that might account for why results are different in one case. i’ll look further later.

it shouldn’t matter, if my logic is correct, but if there’s a bug, then it very well might matter.

rte-construct/find-all-derivatives is the earliest point where the results diverge.

Tinkered with it a bit more before going to sleep. I don't know if it's the problem, but in my eyes it's a problem. The way I set up the experiment: 1. Start two REPLs in the debug mode 2. Run all the tests in the first one till the failure is triggered - that REPL session is now primed to always fail that test 3. Only ever run the failing test in the second REPL, so the second REPL remains "pure" 4. Evaluate the same forms in both REPLs and find where the differences are So, the finding (run in the "bad" REPL):

(let [n '(not (= (1 2 3)))]
  [(gns/canonicalize-type n)
   (util.util/fixed-point n
                          (fn [td]
                            (gns/-canonicalize-type td :dnf))
                          =)])
=> [(not (= [1 2 3])) (not (= (1 2 3)))]

@p-himik, I finally found my error. how long did it take? it was an error that i fixed long ago in the python and scala versions of the code.

The error was the following. I have two type designators called = and member A type such as (= 3) represents the singleton type containing only 3. A type such as (member 1 2 3) represents the type with three elements, namely 1, 2, and 3.

The problem is that 3 and 3.0 are equal according to the clojure = function. but they have different type. The bug is that the membership predicate for the = and member types need to check equivalence and also type.

I fixed that bug in the python code at least 4 years ago.

Ah, I see. Glad you found it!

it’s a bit sad to search for something for a couple of weeks (well not full time for two weeks) and then find such a small thing.

It could be that something like the FlowStorm debugger would be helpful here. But hard to tell for sure given the amount of macros in your code.

wow, you’ve done a lot of work.

yes, as you pointed out, I do need different semantics for vectors and lists. For example, I need to distinguish the following.

(defn x [a b c] (list a b c))

(defn x (a b c) (list a b c))

Not that much work actually. :) Maybe half an hour. If you need to distinguish seqs based not just on the content but also on their type, = cannot be used since (= [1] '(1)).

yes, but clojure uses = for memoization, right? so if I want (foo '[1 2 3]) to have different semantics than (foo '(1 2 3)) then i simply can’t use clojure.core.memoize . Perhaps I need to do some deep thinking about this. I can easily distinguish [1 2 3] from (1 2 3) but it is more difficult to distinguish (((((((((([1 2 3])))))))))) from (((((((((((1 2 3)))))))))))

BTW did you delve into my function find-simplifier ? it iterates over a sequence of transformation functions until it find one which has changed the input value, i.e. returned a value which is not equal to its input, and then considers the input to have been simplified. I ask myself whether it should be testing for = or for identical?

if so, I’d need to go back and carefully check all the call sites that they are never returning a copy.

which might be impractical for recursive structures.

Yes your observation is (I believe) on-point. (-canonicalize-type '(not (= [1 2 3])) :dnf) returns its first argument, and (-canonicalize-type '(not (= (1 2 3))) :dnf) returns ITS first argument. but when memoized, they stop doing that.

😞

> With identical? you have to make sure that your simplifiers don’t change anything at all when they cannot simplify something: indeed.

> You use the inflexible clojure.core/memoize only in a single location - rte-tester/test-rte-canonicalize-nullable-1. > In the rest of the places, you use stuff from the clojure.core.memoize namespace, ahhh, I didn’t realize I was doing that. I wonder whether there was a reason. It’s probably a bug.

But more importantly - what exactly makes you certain that you need memoization in the first place?

TL;DR. Given an expression (genus type designator) there is an expensive procedure to simplify it. And then later, how can I know whether it is already simplified? I’d like to avoid re-invoking this expensive procedure.

with regard to the image I sent yesterday: the two transitions labeled t6 (exiting from is indeed (= (1 2 3)) I suspect one is supposed to be (= [1 2 3]).

> ahhh, I didn’t realize I was doing that. I wonder whether there was a reason. It’s probably a bug. If you don't specify custom memoization parameters and don't use multithreading, clojure.core/memoize and stuff from clojure.core.memoize/... behave identically. It's just that different parts of your app might end up using different caches, which shouldn't be a problem if your functions are pure. > The issue is that these recursive simplifications can be expensive in theory. > [...] > This is one motivation for memoizing. A motivation indeed but, strictly speaking, not a reason. In theory, I should avoid using cat on files because doing that might blow up my terminal emulator due to the amount of data. In practice, it's pretty much never the case. Although usually because I simply know which files it can be used on... In any case - IMO the code should first be written to be correct and as simple is reasonably possible, with some long-term planning. Using dynamic variables and memoization goes against that, unless you already have practical cases where non-memoized code cannot perform. I remembered another anecdote - the PLINK 2 toolset, a part of it that calculates the coefficient of inbreeding for potentially large inputs. We used it extensively to calculate the COI for 3-5 thousands of dogs per run and each run took multiple hours, with a decent probability of crashing. Conceptually, COI computation is just a bunch of arithmetic operations with cells in an N by N matrix, with a result being a triangular matrix of the same size. So, for thousands of dogs it should be millions of cells with maybe tens of millions of operations - nothing that should take hours. I delved into the code, and what did I see? An R package wrapping a C library with an insane amount of complexity in attempts to save memory and make things faster. With memory-related bugs that I could see with a naked eye. I ended up writing pretty much the definition of COI, just in Java and not in English. Around 20 lines of code, most of which is just a necessary ceremony. The code was able to churn through 30k dogs in 3 seconds, with no crashes, no memory errors. All without any kind of clever optimizations. Of course, some problems demand dynamic programming with caching. My point is that things should be kept simple, and complications of any kind should be justified beyond any shadow of a doubt. > if I rewrote it today I’d probably choose to use sexpressions for the reader syntax, and some constructed data structure wrapper as form to indicate, trustable data. If you have plans for this software, such a rewrite might still be the best time investment. But you're the better judge here. > TL;DR. Given an expression (genus type designator) there is an expensive procedure to simplify it. And then later, how can I know whether it is already simplified? Another option would be to use metadata. I would probably use it to signify "yes, I have already been simplified, don't touch me". But it would help only with particular instances of expressions. Two different instances of the same expression can have different metadata.

Just to make sure - memoization isn't that bad by itself if used properly. Using it properly is already kinda tough given that you need to evict specific cache entries based on some strategy. Unless your code is something short-running or with a very limited domain. But with any kind of impurities, it becomes nigh impossible to use properly. Dynamic variables being one of the worst kind here. They're bad even by themselves, without memoization. There are some cases where they are genuinely useful and worth it, but those are very few and far between.

I seem to recall a reason (not sure whether in retrospect good or bad reason) for not using the default memoization. I wanted the memoized data to be GC-able. Ie, memoized values which are not referenced anywhere should be subject to garbage collection. I sometimes have out-of-memory errors on large data structures.

This discussion makes me think that maybe I DO want (1 2 3) to be treated the same as [1 2 3] . Either they should be treated the same or differently, but my code should have defined semantics about that, and the semantics should be testable and tested. These experiments (and your observations) point to the fear that my code is not consistent in this regard.

I think the docstring is wrong on this function.

(defn-memoized [canonicalize-type-2-arg canonicalize-type-2-arg-impl]
  "We cannot memoize canonicalize-type because its return value depends
  on the value of the dynamic variable, so  canonicalize-type calls
  canonicalize-type-2-arg with that value as 2nd argument.  This
  function is a fixed-point wrapper around -canonicalize-type."
  [type-designator nf]
  (fixed-point type-designator
               (fn [td]
                 (-canonicalize-type td nf))
               =))

the 2nd argument is not a dynamic variable.

Just in case - that's just the first "top-level" difference that I noticed. There were other difference, but those were downstream. They might or might not be caused by (...) turning into [...]. I haven't investigated. > I wanted the memoized data to be GC-able Neither of the memoizations offer that themselves. Maybe the core.memoize version lets you customize it to support weak references, but I don't know. Looking it up in this channel shows a discussion with you from 2021. :D https://clojurians.slack.com/archives/C03S1KBA2/p1625828966075500

Thanks for the link. yes I remember that conversation now. the clojure tests were taking a looooooooong time compared to the scala and python tests. so I was memoizing some things to try to see which memoizations would speed up matters.

My workflow when something is slow: 1. Profile the code on slow workflows and see what takes the longest time 2. Improve impls of functions that can be sped up in obvious ways 3. Instrument other functions in a way that reports execution time and inputs 4. Run slow workflows, gather and aggregate the instrumentation data 5. See if there are any patterns and act accordingly It could be that a function is slow but most of its inputs are distinct - memoizing that function will only make things worse. It could be that there are distinct degenerate cases that can be isolated and treated differently Or yes, it could be that just slapping memoize on top of some function will make everything better. But it's not always the case, and without accurate measurements it's easy to make something better while making everything else worse.

I am starting to remember what my code is doing with dynamic variables. Rather than memoizing global functions, instead, I memoize anonymous functions and and bind them to dynamic variables. That way, when a dynamic variable goes out of scope, its memoized data is GCed.

It's logical but painful to read, to be honest. Just because it would be insanely hard to reason about reliably. I'd use regular globally memoized functions and simply manually clear the cache when appropriate. clojure.core.memoize/... lets you do this.

Now the docstring makes sense.

(defn-memoized [canonicalize-type-2-arg canonicalize-type-2-arg-impl]
  "We cannot memoize canonicalize-type because its return value depends
  on the value of the dynamic variable, so  canonicalize-type calls
  canonicalize-type-2-arg with that value as 2nd argument.  This
  function is a fixed-point wrapper around -canonicalize-type."
  [type-designator nf]
  (fixed-point type-designator
               (fn [td]
                 (-canonicalize-type td nf))
               =))

dammit. I took out one of the memoized functions, and my tests are taking a looooooong time to run. normally they take less than 5 minutes… I’ve been waiting about 30 minutes so far.

Why are you shooting in the dark? Why not do it properly? It will take maybe an hour or two, but you will get results that you would be able to rely on, not just some guesswork.

what do you mean by properly?

The workflow that I described above - instrumentation, profiling, analysis.

I’m not sure how to do what you suggest actually. I know why the code is slow. it’s because the algorithms have exponential complexity. What I’ve done in the mean time is identified what I think is the simplest case that fails. I’ve created a simple test case for the failure. I was attempting to turn off memoization to see if the test cases fail without memoization. but in the mean time i notice that the global tests take a long time.

So, what if you don't turn off memoization globally then, but instead: • Memoize only the functions that directly suffer from the exponential complexity ("directly" meaning that any functions that use such functions but don't introduce any exponential behavior should not be affected) • Use plain global vars instead of that mix of dynamic vars with lambdas with memoization • Clear the memoization cache where appropriate. E.g. each time a public API function is done with its work. Or maybe manually between tests, if multiple functions from the public API can operate on the same data (in this scenario, the cache cleaning function would have to become a part of the public API)

(defn-memoized [canonicalize-type-2-arg canonicalize-type-2-arg-impl]
  "We cannot memoize canonicalize-type because its return value depends
  on the value of the dynamic variable, so  canonicalize-type calls
  canonicalize-type-2-arg with that value as 2nd argument.  This
  function is a fixed-point wrapper around -canonicalize-type."
  [type-designator nf]
  (fixed-point type-designator
               (fn [td]
                 (-canonicalize-type td nf))
               =))

(defn canonicalize-type
  "Simplify the given type-designator to a stable form"
  ([type-designator]
   (canonicalize-type type-designator :dnf))
  ([type-designator nf]
   #_(canonicalize-type-2-arg type-designator nf)
   (fixed-point type-designator
                #(-canonicalize-type % nf)
                =)
   ))

before playing with the caching, i’d like to identify more clearly what the error is. it might not even be related. Somewhere in the code I am computing two types which are not disjoint, but the code believes are disjoint. This could be related to memoization, but it could be my misunderstanding of the java type api.

in which case it might even be a bug in the Scala version of the code.

A Tangential issue would be that I don’t know how to use the profiler. I’ve used a couple of profilers with Common Lisp, a deterministic and a statistical profiler. I would definitely benefit from knowing how to do this in Clojure.

I'd use https://github.com/clojure-goes-fast/clj-async-profiler, it's quite simple. But not that useful if some functions take minutes to run - instrumentation would be more useful there.

I wanted to try and clean it up, removing dynamic variables, but quickly realized that it would take way too much time. You use them not only for cache (BTW I didn't even realize there was soft cache, so it shouldn't create any problems for you. Not sure why you see OOM without a "manual" sort of cleanup - I'd investigate that instead of trying to circumvent it) but also for "regular" things that aren't related to caching. That's just incompatible with caching, full stop. Even if all the tests work and you can find no flaws in your code, there could trivially be some specific input with some specific dynamic bindings and a specific sequence of calls that makes everything explode. It's just impossible to reason about in full certainty - exactly like that Music21 situation.

That is, unless I'm reading it wrong and all the dynamic vars that aren't specifically for caching actually don't cross the caching boundary. Meaning, something like

(def ^:dynamic *foo* nil)

(def f (memoize (fn [x]
                  (do-stuff x))))

Made the changes to the memoized functions, ran it a few times, no OOMs or errors so far.

Ran 316 tests containing 542802 assertions.
0 failures, 0 errors.

Oh, and each run takes around 6 minutes.

Tried with a resource monitor running in the background - barely any RAM is getting used. Here's the patch if you want to try on your end.

I think i’m making a fundamental error in my logic. I’m assuming that if x satisfies predicate p and if x==y then y satisfies p but that is false. [1 2 3] equals (1 2 3) but (1 2 3) satisfies seq? while [1 2 3] does not.

my code thinks the following is a partition of the universe:

(= (1 2 3))
(and (not (= (1 2 3))) (satisfies seq?))
(not (satisfies seq?))

(= (1 2 3))
(and (not (= (1 2 3))) (satisfies seq?))
(and (not (= (1 2 3))) (not (satisfies seq?)))

> Here’s the patch if you want to try on your end. Thanks for the patch. I understand what you mean now.

hey about the patch. I can’t apply it. Could you tell me which sha1 you built it against? I’m getting the messages:

error: patch failed: src/genus/genus.clj:162
error: src/genus/genus.clj: patch does not apply
error: patch failed: src/genus/genus_spec.clj:96
error: src/genus/genus_spec.clj: patch does not apply
error: patch failed: src/rte_construct.clj:74
error: src/rte_construct.clj: patch does not apply
error: patch failed: src/rte_tester.clj:158
error: src/rte_tester.clj: patch does not apply
error: patch failed: src/xym/xymbolyco.clj:256
error: src/xym/xymbolyco.clj: patch does not apply
error: patch failed: test/rte_test.clj:241
error: test/rte_test.clj: patch does not apply
error: patch failed: test/xymbolyco_test.clj:269
error: test/xymbolyco_test.clj: patch does not apply

I’ll try to apply the patch to that old commit and then merge forward.

or would it work to send a pull request?

The SHA should be in the patch itself. A PR would work, but it wouldn't be any different, plus more hassle for me. :)

In your patch I see 3 sha1s, none of which I see at the top of my repository. I don’t find any of e47a92d, 3055546, or 100644

Ahh, my bad, I had a wrong assumption about Git patches. I used c39b80c8 as the base for the patch.

well i’ve never appied a patch before. But I think I’m doing it correctly.

Perhaps I’ll just do the edits by hand. that’s not so many.

Gimme a sec.

No bloody clue what happened, to be honest. Try this one, with git apply --check --ignore-whitespace --ignore-space-change.

worked. thanks.

Phew.

Temporal is nice but very heavy duty if that's the only thing you want. https://github.com/aphyr/tea-time might work for single process stuff

fwiw, cljdoc has been using tea-time for years.

one nice thing about Temporal's scheduler is that it handles overlap policies, i.e. you get to decide what happens when a scheduled workflow would start before the previous one finished https://docs.temporal.io/schedule#overlap-policy

There was a discussion some time back in #find-my-lib: https://clojurians.slack.com/archives/CQT1NFF4L/p1756417023865259

I was literally just checking that out. I met @smith.adriane at the conj and we ended up talking about it. It's a cool project

Another clojure specific text editor you might like to check out is https://github.com/fredoverflow/clopad It's smaller scoped and written in Java but seems pretty neat for it's narrow goal. The creator has some great clojure vids on yt too where he uses it (ex. https://www.youtube.com/watch?v=TaazvSJvBaw)

In love the Fred Overflow channel!

Another editor. Wasn’t Light Table the “killer app” of days past?

Tbf, light table was a really cool project too

did light table also run in the same context as the clojure JVM?

Light Table is the main inspiration for Easel, I think.

clopad looks interesting. I don't remember seeing it. I'll have to check it out! Easel is definitely influenced and inspired by previous projects like Light Table. More recently, https://gtoolkit.com/ has also been exploring some interesting ideas.

The Light Table UI was in cljs using Electron.

for me scripting the editor in JVM Clojure would be definitely a step up from elisp

Scripting the editor in JS Clojure is a step up from elisp too.

one step at Clojure domination at a time

But to me it seems like scripting it in JVM Clojure is a step up from doing it in JS Clojure. And also, Easel seems to be so much more than scripting my editor. Easel is more of an idea, or a set of of ideas. And as V told us: Ideas cannot be killed.

who is V?

V for Vendetta.

Ideas certainly can take a hit from wars etc. Esperanto took a huge blow by WW2

anyway that's a tangent ;)

Yeah, don’t get me started on wars. 😃

would easel need something like CIDER or is easel so different that it doesn't even make sense because you can just do without it or so?

I'm tempted to buy a streaming package from the conj to watch the talk but I have so little time that I probably won't get to it before it hits youtube

There's a lot of CIDER related stuff available as libraries (like https://github.com/alexander-yakushev/compliment). I'm seeing how far I can get without something like nrepl and socket repl. There's some fun things you can do in-process, but I think supporting an out of process REPL will happen at some point. There's a a lot of reasons why you may want a separate process (or processes) for your dev repl (eg. cljs, bb, etc).

true

also clojure-lsp support would be nice eventually

but now that we don't have to write all that integration stuff in elisp, that will be so much easier

Easel probably is a pretty awesome environment to hack on clojure-lsp and a lot of other Clojure tooling.

Has anyone ever shared the URL to the Easel project? Google results showed a video but its description didnt include a link to the project. Maybe its not public yet?

https://github.com/phronmophobic/easel Easel is in alpha. I do most of my work on the dev branch which might point to other deps that need to be released.

Easel doesn’t do much besides provide a place for tools to work together. The main project this year was clobber, a library for building code editors, https://github.com/phronmophobic/clobber/

last year (or was it two years ago...?) i moved https://github.com/clj-commons/clooj to clj-commons with the hope of getting it active again, but it turns out editors are really hard to build

i'm looking forward to easel getting a "full" release (for whatever version of full adrian thinks applies lol)

It’s a design goal to try to make all the tools that easel uses (editor, terminal emulator, data inspector, etc) available a la carte without requiring Easel.

I am working to get Easel out of alpha into beta. For productivity tools, there is a pretty high bar for reliability and even if you have a minimal setup, there is still a baseline of features needed. I'm still working through the list of baseline features. As an example, there's no "goto line" or "find and replace". The thing these features have in common is that they both require user input (ie. line number and search/replace strings). That's not hard to do, but I'm still mulling the general strategy for user input (mini buffer, custom UI per feature, etc).

are you thinking like something like a mini-buffer in emacs?

or more like a popup like in vscode

(I find vscode too distracting with all the popups tbh)

I also prefer the mini-buffer from emacs. I can't think of anything better, so I'll probably start with something mini-buffer like.

one thing that annoys me from emacs from time to time is that it can get slow with big files. wonder if you have any ideas in that area

sublime(2?) is supposedly very good with this

> one thing that annoys me from emacs from time to time is that it can get slow with big files. wonder if you have any ideas in that area clobber feels good for long clojure files (lots of lines), but still sucks at files with long lines. Making long lines fast shouldn't be too bad, but will require changes to the underlying https://github.com/phronmophobic/bifurcan/blob/master/src/io/lacuna/bifurcan/Rope.java implementation. Currently, files are reparsed incrementally on each key stroke. tree-sitter is fast enough that this works pretty well in general. tree-sitter also supports splitting up parsing into multiple calls. It's also possible to move grammar parsing to a background thread. Right now, I'm focusing on features that support editing a large majority of clojure files, but I did try to choose technologies and approaches that can be extended to support the long tail of text editing.

thankfully, long lines aren't too common in clojure

yea, I haven't prioritized it because I haven't needed it, but I do want to implement it at some point.

If there's someone that likes algorithms and wants to hack on the Rope implementation, there's plenty of fun features and optimizations to do.

I think the quick-pick menus in vscode is a genius invention. Would love if it also had a minibuffer, but anyway.

oh yea, that does seem pretty neat

I also haven't implemented the equivalent of emac's M-x which seems similar to the quick-pick menu.

@smith.adriane It would be cool if you came to Dutch Clojure Days and to babashka-conf the day before to show your SCI REPL in your phone: https://www.meetup.com/the-dutch-clojure-meetup/events/312079164/?slug=the-dutch-clojure-meetup&eventId=312079164 It's a pretty long flight though, I get it ;)

If you apply for a talk at DCD they might pay for you flight, you could present easel. And then the day before your SCI REPL project at babashka-conf :)

(just a wild idea, no pressure to actually do it, would be cool though)

Sounds like a good time!

When's the next bb conf? 👀

click the link above