Fork me on GitHub
#babashka
<
2019-11-06
>
borkdude20:11:24

edamame, the parser used in sci and hence babashka, can now parse the entire core namespaces of clojure and clojurescript. to support this, support for auto-resolve has been added. see the section under "Auto-resolve keywords" here: https://github.com/borkdude/edamame

dominicm20:11:24

It might be cool for edamame to create a parallel location data structure. That would solve the problem with locations for numbers. The question is how you would link the original data structure to the input. If you were using zippers you could traverse them in parallel.

borkdude20:11:58

@dominicm If you want something like that, you should probably use rewrite-clj

dominicm20:11:24

But then I wouldn't have all of the cool edamame features :)

borkdude20:11:17

what's a feature you would like from edamame in rewrite-clj?

dominicm20:11:23

The convenience I think.

dominicm20:11:45

Or rather, the getting a data structure rather than a rewrite structure

borkdude20:11:18

with rewrite-clj you can still call sexpr to get the sexpr. I have no plans to do a parallel universe parsing feature right now 🙂

borkdude20:11:23

the reason I added the locations was mostly because of unresolved symbols in babashka. there are hardly problems with number literals so the locations of those I don't need as much

sogaiu20:11:00

there is some caution required for use of sexpr in rewrite-clj, right?

borkdude20:11:33

in the case of potentially invalid code, yes

borkdude20:11:03

or mostly when you try to analyze something and then call sexpr on nil, when the user has forgotten a value where something is expected

borkdude20:11:23

also you lose all the location metadata when calling sexpr

dominicm20:11:52

In aero, if an argument is a number, it would be useful to point to it after the fact.

dominicm20:11:59

"This should be a string"

borkdude20:11:18

you could point to the nearest neighbouring or wrapping element, at least it's close 🙂

borkdude20:11:48

the same is happening right now in clj-kondo: we do datalog linting, but we don't know the exact locations within the datalog expression because the datalog parser library doesn't return that

borkdude20:11:59

it's not perfect, but probably good enough

borkdude21:11:52

@dominicm hmm, spec and malli include paths into the data structures when something is wrong. using such a path into the "parallel" locations data structure, would maybe work

borkdude21:11:36

dunno if aero has these paths

dominicm21:11:34

It does internally, but tweaks may be needed to retain the location data. I may just create "conf" on the same kind of principles.

dominicm21:11:12

RE edamame, how can I add to the available namespaces while reading? I'm assuming I'll need to use the dispatch table.

borkdude21:11:53

maybe it would help if edamame added key -> location metadata as metadata on parsed maps, so {:a 1} <-> {:a {:row 1 :col 16}} or something. not sure if I like it yet

borkdude21:11:21

RE edamame while reading: can you give me an example of what you're trying to do. I might have an idea, but example would be better to see first

borkdude21:11:57

there's an example in sci how to do this I think: https://github.com/borkdude/sci/blob/c8c89a63aa490106dffc3d4c856d44ba52a83e8d/src/sci/impl/parser.cljc#L18 but again, not sure if this is what you're looking for (and it's not exposed officially, so might break)

mauricio.szabo22:11:47

Do you have any benchmark of Edamame, comparing it to rewrite-clj(s)?

borkdude22:11:18

no, but since they are different in project goal it's maybe not fair to compare them

borkdude22:11:53

I expect edamame to be a little faster than rewrite-clj since it creates less overhead, but could be wrong

borkdude22:11:27

@mauricio.szabo Informally:

$ clj -Sdeps '{:deps {rewrite-clj {:mvn/version "RELEASE"}}}'
Clojure 1.10.1
user=>  (require '[rewrite-clj.parser :as p])
nil
user=> (require '[ :as io])
nil
user=> (def clojure-core-src (slurp (io/file "test-resources" "clojure" "core.clj"))
)
#'user/clojure-core-src
user=>  (def rewrite-clj (time (p/parse-string-all clojure-core-src)))
"Elapsed time: 284.394016 msecs"
#'user/rewrite-clj
user=> (require '[edamame.core :as edamame])
nil
user=> (def edamame (time (edamame/parse-string-all clojure-core-src {:all true :auto-resolve {:current 'clojure.core}})))
"Elapsed time: 110.019838 msecs"
#'user/edamame

borkdude22:11:41

user=> (time (dotimes [i 100] (edamame/parse-string-all clojure-core-src {:all true :auto-resolve {:current 'clojure.core}})))
"Elapsed time: 4825.793358 msecs"
nil
user=> (time (dotimes [i 100] (p/parse-string-all clojure-core-src)))
"Elapsed time: 14629.100979 msecs"

borkdude22:11:57

from this informal test you could derive that edamame is roughly 3x as fast