clojure 2021-09-07 | Slack Archive

all the defs are near.

I’m working on a system to calculate features for machine learning models, immediately before the model makes a prediction (think Spark, but much simpler). One use-case for it is “here’s a bunch of data, give me all the features you can possibly can calculate with it”. I thought metadata could be a way to enable this, by mapping functions to data points, but I’ve never worked with metadata extensively and was wondering if this is not a good idea. Example of a feature:

(defn email-name
  "Lower-cased name of an email address (the bit before @)"
  {:tags ["email"]
   :dependencies [:email]}
  [{email :email}]
  (-> email
      s/lower-case
      (s/replace-first #"@.*" "")))

at system start I would collect all public functions, create a map and then when it receives a request

{:input-data {:email ""
              :phone 12345345
              :name "John Doe"}}

it searches for all functions that can use :email , :phone , :name

p-himik08:09:28

Just to be clear - you want to use metadata to mark functions, and not to store the result of those functions along with the input data, right?

Santiago09:09:36

correct 🙂 I just want to have a way to search/index these functions e.g. later I’ll probably add some sort of UI so a user knows what features are available in the system with the tag “email” or whatever

Santiago09:09:29

the functions themselves are free of side-effects and their result is sent back to the requester (HTTP, CLI, …) and potentially written to a database or queue, but not stored internally

p-himik09:09:22

I think metadata sounds reasonable then. Alternatively, you can make it more explicit by creating another function, something like reg-fn, that would directly write the association between :email and email-name somewhere. This way, you wouldn't have to scan any namespaces for such functions - should be easier to extend. And to facilitate that, you could have a macro that does both defn and reg-fn in one go.

viesti09:09:51

maybe even multimethods, or put the data about the association into a map (say {:args ... :handler (fn...}, behind a def in some namespace, and the have a "handler" that receives the input, looks at the map in the def and calls the associated function

p-himik09:09:49

I imagine some functions will require multiple fields - multimethods aren't really suitable for that case.

☝️ 2

✅ 2

Santiago09:09:41

although it’s a valid solution, I want to avoid writing an explicit map mapping args<>handlers because that means a user would need to add code in two places. With metadata the user would write once a function and the answer to “what is needed to use this function” is in the function def. @U2FRKM4TW having a reg-fn type thing was exactly my idea, but I would run it on system launch after scanning namespaces. Something like

(->> (vals (ns-publics 'feature-loading-ns))
     (map meta)
     (filter #(contains? % :dependencies))
     (map reg-fn)
     (into {}))

a macro feels a bit magical because it’s writing somewhere the user doesn’t know about, or did I misunderstand?

p-himik09:09:04

ns-publics requires for that namespace to exist - you will have to load it anyway. So it won't be different from using reg-fn. A macro will just call reg-fn itself, so that users don't have to write an extra statement.

Santiago10:09:28

hum so what are you imagining reg-fn is doing? :thinking_face: I think I’m fully getting the point 😅

p-himik10:09:53

(def registry (atom []))

(defn reg-fn [f deps]
  (swap! registry conj {:fn f, :deps deps}))

Or something like that.

Santiago10:09:52

ok so we were thinking about the same 🙂 cool

DG16:09:00

Anyone here running Cursive with WSL 2? I'm trying to run or debug my project with the files in the WSL home directory (as suggested by Microsoft for WSL 2), but seems like Cursive doesn't play well with that. I've created a simple Java project (no Cursive) and it works the way it's supposed to. Seems like Cursive doesn't know how to run the JVM on WSL (which probably should be executed prefixing wsl.exe to the command line). The project's JDK is correctly configured to use the one from WSL as per Intellij IDEA's documentation. This is the error I receive (screenshot below). Any pointers?

p-himik16:09:52

If it's Cursive-specific, then there's #cursive

DG16:09:20

Oh, didn't know about that. Will post it there, thanks

👍 2

2021-09-07

Channels