Fork me on GitHub
#clojure-dev
<
2021-02-26
>
denik20:02:48

is there way to find all non-local bindings of a clojure form? For example:

(non-local-bindings
  '(fn ([{:keys [items ks]}]
        (let [foo :bar]
          (conj [foo] (not.locally/bound))))))
;; should return:
;; => #{clojure.core/fn clojure.core/conj not.locally/bound}

noisesmith20:02:44

@denik my first hunch is to use (keys &env) and filter out everything in that

noisesmith20:02:54

(what remains is not local)

denik20:02:08

@noisesmith since the form is quoted that wouldn’t work or would it?

noisesmith20:02:02

it would require jumping through some hoops if it's possible, I don't know anything directly in clojure that answers your question - maybe some feature of tools.namspace?

noisesmith20:02:01

sorry I was thinking of tools.analyzer and mis-remembered the name (answer might still be no)

👌 1
hiredman20:02:24

Tools analyzer would be the thing, the general term for this is closed/free

hiredman20:02:57

Well I guess free, you are looking for the free variables in an expression

noisesmith20:02:52

you could make something with tree-seq I bet, but you'd need to hard code every binding form to make it work

hiredman20:02:04

Which toops.analyzer will do, and a ton more

hiredman20:02:21

Nah, tree-seq won't work

borkdude20:02:36

@denik you could also use clj-kondo which will report non-locals as unresolved symbols

denik20:02:37

that could work. does it provide output as data?

denik20:02:03

I’m already using sci 😛 this will a borkdude-heavy project 😄

borkdude21:02:15

@denik Feel free to drop by in #sci if you have questions

🙏 1
hiredman20:02:01

You need to analyze in the lexical context, which tree-seq will peel away

noisesmith20:02:08

my thought was the "children" function could return an empty list for non-binding forms, and a list with locals filtered otherwise

noisesmith20:02:12

but that might be flawed

hiredman20:02:11

In general it is not an easy analysis to do, if you can avoid it

hiredman20:02:50

It is really impossible before macroexpansion and after macroexpansion things like fn are gone, replaced with fn*, which as a special form is usually not considered free

borkdude20:02:13

$ cat /tmp/example.clj
(fn ([{:keys [items ks]}]
     (let [foo :bar]
       (conj [foo] (not.locally/bound)))))
$ clj-kondo --lint /tmp/example.clj
/tmp/example.clj:1:15: warning: unused binding items
/tmp/example.clj:1:21: warning: unused binding ks
/tmp/example.clj:3:21: warning: Unresolved namespace not.locally. Are you missing a require?

hiredman20:02:59

The clojure compiler sort of analyzes names to determine which of three states it is in: free (a free name that doesn't resolve to a var is an error) closed over (compiles to an instance field lookup) or local (gets a slot in the methods stackframe)

denik20:02:37

that could work. does it provide output as data?

hiredman20:02:15

I have some code from a sort of replacement for core.async's go macro I was writing that uses tools.analyzers postwalk-transforms feature to collect free variables https://gist.github.com/hiredman/5644dd40f2621b0a783a3231ea29ff1a#file-yield-clj-L655-L679

👀 1
denik20:02:51

@borkdude is there a way to use clj-kondo on forms directly

borkdude20:02:13

@denik yes, let me create an example

🙏 1
denik20:02:16

also using sci, so if they share a ctx I could plug that in

borkdude20:02:16

@denik

(require '[clj-kondo.core :as clj-kondo])

(def example '(fn ([{:keys [items ks]}]
                   (let [foo :bar]
                     (conj [foo] (not.locally/bound))))))

(def findings
  (:findings (with-in-str
               (pr-str example)
               (clj-kondo/run! {:lint ["-"]}))))

(require '[clojure.pprint :as pp])

(pp/pprint findings)
[{:type :unused-binding,
  :filename "<stdin>",
  :message "unused binding items",
 ...

denik21:02:08

thank you. looks like the symbols pat of a string. I’ll try to find lower-level fns that return the symbols

borkdude21:02:59

@denik we could add that to the data. I think that could be useful too.

💯 1
borkdude21:02:15

what are you trying to accomplish with the symbol once you find it?

denik21:02:41

I’m storing function bodies in datalevin, example here: https://clojureverse.org/t/datalevin-powering-environment-and-runtime/7243

denik21:02:15

and want to resolve symbols from the db and replace them with something that is invocable

denik21:02:57

also wondering if ctx can be shared between the analyzer (kondo) and sci (evaluation)

borkdude21:02:12

@denik What you can do maybe is: use sci/parser + sci/parse-next which will get you the function s-expression. Then you can do some processing on that (postwalking, replacing) and then you can feed that to sci/eval-form

borkdude21:02:32

This is also how you can implement a REPL in sci: https://github.com/borkdude/sci#repl But you can do the processing step in between the parsing and evaluation.

denik22:02:16

that works for parsing into edn form but how would it solve the local-bindings problem?

(sci.impl.parser/parse-next
  sci-ctx
  (sci.impl.parser/reader (str '(fn ([{:keys [items ks]}]
                                     (let [foo :bar]
                                       (conj [foo] (not.locally/bound))))))))
=> (fn ([{:keys [items ks]}] (let [foo :bar] (conj [foo] (not.locally/bound)))))

borkdude22:02:05

It doesn't solve that part, but that's the place to potentially fix it. Why would you store s-expressions with unresolved vars/locals?

denik22:02:51

exactly, I wouldn’t! But since I store forms in a database (not namespaces) I need to find unresolved symbols in the db and replace them with their value and throw and error otherwise

borkdude22:02:39

is an unresolved symbol a local or a var in your problem? and is it namespaced? and what would you replace it with?

borkdude22:02:56

I'm trying to understand the problem, not entirely clear to me yet

denik22:02:01

I’ll do my best to explain! If I can use sci’s ctx it is a symbol (namespaced or not) that is not available in ctx’s :namespaces

borkdude22:02:04

And how do you know what to replace it with?

denik22:02:24

yes I have a function that inlines the form

borkdude22:02:46

but how do you know what to replace with what? can you give an example?

denik22:02:20

sure,

(snipf all-tweets
       []
       (->> (datoms :aev :com.twitter/tweet-text)
            (mapv (comp entity :e))))

; =>
{:var cells.lab.code-db/all-tweets,
 :id "1MCsHLjz56",
 :created-at #inst"2021-02-26T22:19:12.408-00:00",
 :updated-at #inst"2021-02-26T22:19:12.408-00:00",
 :form (clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e)))),
 :db/id 38}

(snipf all-tweets-ui
       []
       (into [:div]
             (map (fn [{:com.twitter/keys [tweet-text tweet-author]}]
                    [:div (str tweet-text " by " tweet-author)]
                    ))
             (cells.lab.code-db/all-tweets)))

; =>
{:var cells.lab.code-db/all-tweets-ui,
 :id "1YiWEQrLlG",
 :created-at #inst"2021-02-26T22:19:28.533-00:00",
 :updated-at #inst"2021-02-26T22:19:28.533-00:00",
 :form (clojure.core/fn
         []
         (into
           [:div]
           (map (fn [#:com.twitter{:keys [tweet-text tweet-author]}] [:div (str tweet-text " by " tweet-author)]))
           ((clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e))))))),

denik22:02:09

all-tweets-ui uses all-tweets which has been added to the db through the snipf macro. we can see that the form of all-tweets-ui inlined the form of all-tweets

borkdude22:02:23

so it's more or less a dependency problem?

denik22:02:54

dependency-resolution, yes. anything that is not provided in sci’s context will have to get inlined from the db. if it doesn’t exist, it should throw

borkdude22:02:56

I think you should solve this differently and store some information on which other vars the var depends and load those first

borkdude22:02:04

you can possibly store "require" expressions along with the fn expressions, to ensure the namespace gets loaded first

denik22:02:39

hmm, it’s less about loading and more about existence of a form for a given var in the database. it would be great if this could be figured out during analysis

borkdude22:02:38

in sci you could try to eval the form, catch the exception and try to load the other form with data from the exception, maybe

denik22:02:55

none of the forms should need to be evaluated to know whether some of their contained symbols need to be replaced

borkdude22:02:01

evaluating a fn expr is more or less the same as analyzing the fn body

denik22:02:08

am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this?

borkdude22:02:42

sci does have a separate analysis step but this is not exposed. you will still get evaluations for macros that are expanded for example, so it isn't guaranteed to be side-effect free

denik22:02:59

hmm ok time to ponder this for a bit. thanks so much for being helpful!

borkdude22:02:01

> am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this? this is more or less what happens during analysis

denik22:02:07

I looked at it earlier in sci and unfortunately the analyzer closes over function arities so that I cannot inspect them

borkdude22:02:34

what you can inspect is the parsed form

denik22:02:16

(sci.impl.analyzer/analyze sci-ctx
                             '(fn ([{:keys [items ks]}] items))
                             )
#:sci.impl{:fn-bodies [#:sci.impl{:body [#object[clojure.lang.AFunction$1 0xfbaf833 "[email protected]"]],
                                  :params [p__46761],
                                  :fixed-arity 1,
                                  :var-arg-name nil,
                                  :fn-name nil,
                                  :arglist [{:keys [items ks]}]}],
           :fn-name nil,
           :arglists [[{:keys [items ks]}]],
           :fn true,
           :fn-meta nil}

denik22:02:34

this is the hack that currently works:

(defn- local-syms [params]
  (set
    (sp/select
      (sp/walker symbol?)
      params)))

(defn- resolve-arity-tail [arity-tail]
  (let [params     (->> arity-tail :params :params (mapv second))
        local-syms (local-syms params)
        body       (-> arity-tail :body second)]
    (apply list params
           (clojure.walk/postwalk
             (fn [x]
               ;; FIXME need something that ignores all local bindings
               (if (symbol? x)
                 ; fixme should resolve from sci' ctx
                 (if (or (local-syms x) (resolve x))
                   x                                        ; provided
                   (if-let [{:keys [form]} (get-db-var (namespace-sym x))]
                     form
                     ;; FIXME
                     (do (println (str "Couldn't resolve " x ", assume locally bound"))
                         x)
                     #_(throw (ex-info (str "No :form found for " x) {}))))
                 x))
             body))))

(defmacro snipf [name & f-bodies]
  (let [[arity# tail-or-tails#] (s/conform ::clj-specs/fn-tail f-bodies)
        ari# (case arity#
               :arity-1 (resolve-arity-tail tail-or-tails#)
               :arity-n (map resolve-arity-tail tail-or-tails#))
        f#   (conj ari# `fn)]
    `(db/transact-entity!
       code-conn
       {:var  (~namespace-sym '~name)
        :form '~f#})))

denik22:02:27

but it would not catch vars that don’t exist in either the sci ctx or the db at def-time

borkdude22:02:17

with parsed, I meant what you get back from parse-next. the analyzer is an impl detail, the output from that isn't meant for third party consumption

👌 1
borkdude22:02:04

So what is the use case of saving arbitrary snippets of code out of context in the db? Can you add the requirement that people can declare some kind of dependency so the function snippets become more standalone and easier to execute?

borkdude22:02:31

I'm afk now

denik22:02:00

it’s a little cumbersome but that could be done. I thought the namespaced symbol itself could be that requirement

denik00:02:24

got it to work with a combination of walking the expression and replacing namespaced symbols and evaling the fn form to catch errors

denik00:02:46

(defn resolve-db-syms [form]
  (clojure.walk/postwalk
    (fn [x]
      ;; FIXME need something that ignores all local bindings
      (if (and (symbol? x) (namespace x))
        ;; inline form if namespaced sym found
        (if-let [{:keys [form]} (get-db-var x)]
          form
          x)
        x))
    form))

(defmacro snipf [name & f-bodies]
  (let [f-bodies# (resolve-db-syms f-bodies)
        f#        (conj f-bodies# 'fn)]
    ;; eval fn def to throw syntax errors
    (sci/eval-form sci-ctx f#)
    `(db/transact-entity!
       code-conn
       {:var  (~namespace-sym '~name)
        :form '~f#})))