I’m already using sci 😛 this will a borkdude-heavy project 😄

@denik Feel free to drop by in #sci if you have questions

thank you. looks like the symbols pat of a string. I’ll try to find lower-level fns that return the symbols

@denik we could add that to the data. I think that could be useful too.

what are you trying to accomplish with the symbol once you find it?

I’m storing function bodies in datalevin, example here: https://clojureverse.org/t/datalevin-powering-environment-and-runtime/7243

and want to resolve symbols from the db and replace them with something that is invocable

also wondering if ctx can be shared between the analyzer (kondo) and sci (evaluation)

@denik What you can do maybe is: use sci/parser + sci/parse-next which will get you the function s-expression. Then you can do some processing on that (postwalking, replacing) and then you can feed that to sci/eval-form

This is also how you can implement a REPL in sci: https://github.com/borkdude/sci#repl But you can do the processing step in between the parsing and evaluation.

that works for parsing into edn form but how would it solve the local-bindings problem?

(sci.impl.parser/parse-next
  sci-ctx
  (sci.impl.parser/reader (str '(fn ([{:keys [items ks]}]
                                     (let [foo :bar]
                                       (conj [foo] (not.locally/bound))))))))
=> (fn ([{:keys [items ks]}] (let [foo :bar] (conj [foo] (not.locally/bound)))))

It doesn't solve that part, but that's the place to potentially fix it. Why would you store s-expressions with unresolved vars/locals?

exactly, I wouldn’t! But since I store forms in a database (not namespaces) I need to find unresolved symbols in the db and replace them with their value and throw and error otherwise

is an unresolved symbol a local or a var in your problem? and is it namespaced? and what would you replace it with?

I'm trying to understand the problem, not entirely clear to me yet

I’ll do my best to explain! If I can use sci’s ctx it is a symbol (namespaced or not) that is not available in ctx’s :namespaces

And how do you know what to replace it with?

yes I have a function that inlines the form

but how do you know what to replace with what? can you give an example?

sure,

(snipf all-tweets
       []
       (->> (datoms :aev :com.twitter/tweet-text)
            (mapv (comp entity :e))))

; =>
{:var cells.lab.code-db/all-tweets,
 :id "1MCsHLjz56",
 :created-at #inst"2021-02-26T22:19:12.408-00:00",
 :updated-at #inst"2021-02-26T22:19:12.408-00:00",
 :form (clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e)))),
 :db/id 38}

(snipf all-tweets-ui
       []
       (into [:div]
             (map (fn [{:com.twitter/keys [tweet-text tweet-author]}]
                    [:div (str tweet-text " by " tweet-author)]
                    ))
             (cells.lab.code-db/all-tweets)))

; =>
{:var cells.lab.code-db/all-tweets-ui,
 :id "1YiWEQrLlG",
 :created-at #inst"2021-02-26T22:19:28.533-00:00",
 :updated-at #inst"2021-02-26T22:19:28.533-00:00",
 :form (clojure.core/fn
         []
         (into
           [:div]
           (map (fn [#:com.twitter{:keys [tweet-text tweet-author]}] [:div (str tweet-text " by " tweet-author)]))
           ((clojure.core/fn [] (->> (datoms :aev :com.twitter/tweet-text) (mapv (comp entity :e))))))),

all-tweets-ui uses all-tweets which has been added to the db through the snipf macro. we can see that the form of all-tweets-ui inlined the form of all-tweets

so it's more or less a dependency problem?

dependency-resolution, yes. anything that is not provided in sci’s context will have to get inlined from the db. if it doesn’t exist, it should throw

I think you should solve this differently and store some information on which other vars the var depends and load those first

you can possibly store "require" expressions along with the fn expressions, to ensure the namespace gets loaded first

hmm, it’s less about loading and more about existence of a form for a given var in the database. it would be great if this could be figured out during analysis

in sci you could try to eval the form, catch the exception and try to load the other form with data from the exception, maybe

none of the forms should need to be evaluated to know whether some of their contained symbols need to be replaced

evaluating a fn expr is more or less the same as analyzing the fn body

am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this?

sci does have a separate analysis step but this is not exposed. you will still get evaluations for macros that are expanded for example, so it isn't guaranteed to be side-effect free

hmm ok time to ponder this for a bit. thanks so much for being helpful!

> am I wrong in thinking that the sci-ctx and a form-walker that ignores local bindings should be enough to do this? this is more or less what happens during analysis

I looked at it earlier in sci and unfortunately the analyzer closes over function arities so that I cannot inspect them

what you can inspect is the parsed form

can I?

(sci.impl.analyzer/analyze sci-ctx
                             '(fn ([{:keys [items ks]}] items))
                             )
#:sci.impl{:fn-bodies [#:sci.impl{:body [#object[clojure.lang.AFunction$1 0xfbaf833 "clojure.lang.AFunction$1@fbaf833"]],
                                  :params [p__46761],
                                  :fixed-arity 1,
                                  :var-arg-name nil,
                                  :fn-name nil,
                                  :arglist [{:keys [items ks]}]}],
           :fn-name nil,
           :arglists [[{:keys [items ks]}]],
           :fn true,
           :fn-meta nil}

this is the hack that currently works:

(defn- local-syms [params]
  (set
    (sp/select
      (sp/walker symbol?)
      params)))

(defn- resolve-arity-tail [arity-tail]
  (let [params     (->> arity-tail :params :params (mapv second))
        local-syms (local-syms params)
        body       (-> arity-tail :body second)]
    (apply list params
           (clojure.walk/postwalk
             (fn [x]
               ;; FIXME need something that ignores all local bindings
               (if (symbol? x)
                 ; fixme should resolve from sci' ctx
                 (if (or (local-syms x) (resolve x))
                   x                                        ; provided
                   (if-let [{:keys [form]} (get-db-var (namespace-sym x))]
                     form
                     ;; FIXME
                     (do (println (str "Couldn't resolve " x ", assume locally bound"))
                         x)
                     #_(throw (ex-info (str "No :form found for " x) {}))))
                 x))
             body))))

(defmacro snipf [name & f-bodies]
  (let [[arity# tail-or-tails#] (s/conform ::clj-specs/fn-tail f-bodies)
        ari# (case arity#
               :arity-1 (resolve-arity-tail tail-or-tails#)
               :arity-n (map resolve-arity-tail tail-or-tails#))
        f#   (conj ari# `fn)]
    `(db/transact-entity!
       code-conn
       {:var  (~namespace-sym '~name)
        :form '~f#})))

but it would not catch vars that don’t exist in either the sci ctx or the db at def-time

with parsed, I meant what you get back from parse-next. the analyzer is an impl detail, the output from that isn't meant for third party consumption

So what is the use case of saving arbitrary snippets of code out of context in the db? Can you add the requirement that people can declare some kind of dependency so the function snippets become more standalone and easier to execute?

I'm afk now

it’s a little cumbersome but that could be done. I thought the namespaced symbol itself could be that requirement

this would work if it was possible to pass a resolve-symbol function https://clojurians.slack.com/archives/C06E3HYPR/p1614378661043400?thread_ts=1614373036.036400&cid=C06E3HYPR

https://github.com/borkdude/sci/blob/master/src/sci/impl/evaluator.cljc#L417

got it to work with a combination of walking the expression and replacing namespaced symbols and evaling the fn form to catch errors

(defn resolve-db-syms [form]
  (clojure.walk/postwalk
    (fn [x]
      ;; FIXME need something that ignores all local bindings
      (if (and (symbol? x) (namespace x))
        ;; inline form if namespaced sym found
        (if-let [{:keys [form]} (get-db-var x)]
          form
          x)
        x))
    form))

(defmacro snipf [name & f-bodies]
  (let [f-bodies# (resolve-db-syms f-bodies)
        f#        (conj f-bodies# 'fn)]
    ;; eval fn def to throw syntax errors
    (sci/eval-form sci-ctx f#)
    `(db/transact-entity!
       code-conn
       {:var  (~namespace-sym '~name)
        :form '~f#})))