tree-sitter gotcha: although it looks like the rules: { ... } part of the grammar looks like a js object, order matters for the very first key-value pair. getting this wrong can (will?) cause chaos.

re: ignore markers and tag markers -- conceptually, i've been thinking of them as follows (hopefully correctly, but who knows?). for tags:

#my-outer-tag #my-inner-tag 'a

#my-outer-tag (#my-inner-tag ('a))

for discards, what i've currently got in the tree-sitter grammar (adapted from Tavistock's) is:

discard_form: $ =>
      seq($._discard,
          optional($.discard_form),
          $._non_discard),

here $._discard is basically just #_

so for:

#_ #_ 1 2

With the edit it makes sense. 😃

he he -- sorry for the confusion 🙂

So what happens with something like:

#my-outer-tag #_ #my-inner-tag 'a 'b

my guess: #my-inner-tag 'a is "swallowed" by #_ so #my-outer-tag's associated reader function(?) is passed 'b for processing

here is an example:

user=> #uuid #_ #inst "1985-04-12" "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
#uuid "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"

Cool. That's what I guessed.

sadly, the current grammar i have doesn't appear to handle it though

Don't try this:

#uuid #_ #uuid "84098d2d-2ed5-4de5-bd4a-c4b967f9e468" #uuid "8ce4afa3-1599-4a98-b642-59eb5e9e83b8"

i don't know what happens when #uuid is handed an actual uuid

Which is the same as this, naturally, which never returns either

#uuid #uuid "8ce4afa3-1599-4a98-b642-59eb5e9e83b8"

Nothing happens. Nothing as in things stop happening. 😃

lol

well, thanks to this discussion, i have more grammar hacking to do...and i thought i was close to finishing 🙂

Same here. To both sentiments.

However, I have my crude, line based, scanner, and you have that sophisticated super tool. 😃

actually, i'm not terribly fond of tree-sitter to be honest for certain tasks -- your scanner can be more flexible.

My scanner can be almost anything I want. Which is nice. However, the line based part of it... I wish I had lots of time so I could get rid of that!

it's pretty amazing that it can do so much even being line based -- i was surprised to be sure

have you considered looking at what edamame or tools.reader does?

I have looked in many different places. Often very shallow, though.

The scanner isn't of my creation, btw. I have modified it a lot so have some right to call it partly mine, but it was Matt Seddon creating it. I think it was the 100th time he created one, so he really knows these things at a level that boggles my mind. He is creating a new one as we speak, for a new LISP that he is creating at the same time. Haha, I guess he just likes to have work.

I am nudging things so that he hopefully will create a non-line-based thing this time. And DFA. Then I plan to bring that into Calva, replacing my current thing.

it's good if he enjoys what he's doing -- getting this stuff working can be fun, but it can be frustrating when there are no official specs

i've got some notes from digging through clojure / clojure-dev google groups archives and jira issues to try to track down what is supported by the reader