Fork me on GitHub
#portal
<
2023-06-05
>
Sam Ritchie13:06:33

@djblue here’s how I install any npm deps from deps.cljs entries on the classpath: https://github.com/mentat-collective/Clerk-Utils/blob/main/src/mentat/clerk_utils/build/shadow.clj#L70-L90 using this code from shadow: https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/cljs/devtools/server/npm_deps.clj I think it would be nice to extract this out into some prep function in portal.api… wdyt?

Sam Ritchie19:06:35

(ns emmy.portal.deps
  "Borrowed from shadow-cljs and clerk-utils."
  (:require [clojure.data.json :as json]
            [clojure.edn :as edn]
            [clojure.java.shell :refer [sh]]
            [clojure.string]
            [ :as io]))

(def ^:private windows?
  (clojure.string/starts-with?
   (System/getProperty "os.name")
   "Windows"))

(def npm-cmd
  "System-specific NPM command, tuned for Windows or non-Windows."
  (if windows? "npm.cmd" "npm"))

(defn dep->str [dep-id]
  (cond (keyword? dep-id)
        ;; :some/foo? :react :@some/scoped?
        (subs (str dep-id) 1)

        (symbol? dep-id)
        (str dep-id)

        (string? dep-id)
        dep-id

        :else
        (throw (ex-info (format "invalid dependency id %s" dep-id) {}))))

(defn get-deps-from-classpath []
  (let [deps
        (-> (Thread/currentThread)
            (.getContextClassLoader)
            (.getResources "deps.cljs")
            (enumeration-seq)
            (->> (map (fn [url]
                        (-> (slurp url)
                            (edn/read-string)
                            (select-keys [:npm-deps])
                            (assoc :url url))))
                 (into [])))]

    (vec (for [{:keys [url npm-deps]} deps
               [dep-id dep-version] npm-deps]
           {:id (dep->str dep-id)
            :version dep-version
            :url url}))))

(defn resolve-conflicts [deps]
  (vals
   (reduce
    (fn [acc {:keys [id] :as dep}]
      (update acc id #(or % dep)))
    {}
    deps)))

(defn read-package-json [install-dir]
  (let [package-json-file (io/file install-dir "package.json")]
    (if-not (.exists package-json-file)
      {}
      (-> (slurp package-json-file)
          (json/read-str)))))

(defn is-installed? [{:keys [id]} package-json]
  (or (get-in package-json ["dependencies" id])
      (get-in package-json ["devDependencies" id])
      (get-in package-json ["peerDependencies" id])))

(defn install-deps [deps]
  (when (seq deps)
    (let [args (for [{:keys [id version]} deps]
                 (str id "@" version))
          install-cmd ["npm" "install" "--no-save"]
          full-cmd (into install-cmd args)
          full-cmd (if windows?
                     (into ["cmd" "/C"] full-cmd)
                     full-cmd)]
      (println (str "running: " (clojure.string/join " " full-cmd)))
      (println
       (:out
        (apply sh full-cmd))))))

(defn install-npm-deps!
  "This command:

  - Triggers an `npm install`
  - Installs any dependency referenced by a `deps.cljs` file on the classpath
  - Installs these dependencies into the calling project's `package.json` file.

  Unlike `shadow-cljs`'s native `npm-deps/main`, this command also
  installs [[shadow-npm-dep]], negating any need to tell the user to have
  versions match or to remember to do this install themselves."
  []
  (let [package-json (read-package-json ".")
        deps         (->> (get-deps-from-classpath)
                          (resolve-conflicts)
                          (remove #(is-installed? % package-json)))]
    (when (seq package-json)
      (println "Running npm install...")
      (println
       (:out
        (sh npm-cmd "install"))))
    (install-deps deps)))

Sam Ritchie19:06:01

@djblue this is a slimmed down version of what’s in shadow

djblue19:06:50

I'll take a look later today. I think my main reservation is automatically updating a users package.json without asking first :thinking_face:

Sam Ritchie19:06:10

yeah for sure

djblue19:06:10

Unless the deps lived under .portal/package.json :thinking_face: Then I wouldn't care

Sam Ritchie19:06:39

I guess we can install and remove shadow’s --save

Sam Ritchie19:06:47

or explicitly flag NOT to save

Sam Ritchie19:06:39

the reason I ran npm install and THEN did the shadow thing was that if someone had, say, mafs in their package.json already, shadow marks it as “already installed” and skips the phase

👍 2
djblue19:06:56

How would you feel about .portal/node_modules? Do you think that would cause issues?

Sam Ritchie19:06:40

maybe search both?

Sam Ritchie19:06:39

the two cases are: 1. I already have a dep installed, in which case I would expect portal to Just Work 2. I don’t know anything about npm or dependencies, and I want to run a (prepare!) command that will install all dependencies (without me having to know what they are, what versions etc)

djblue19:06:34

Okay, I need to think about it a little more but I definitely want to make this easier for everyone 👌

Sam Ritchie19:06:31

sg, let me tighten up this code

Sam Ritchie19:06:05

edited the above to make it use --no-save and do nothing if there are no deps on the classpath

Sam Ritchie21:06:11

(ns emmy.portal.deps
  "Functions for resolving npm dependencies declared in `deps.cljs` files on the
  classpath.

  This code is a simplified version of code in
  thheller's [shadow-cljs]()
  library and
  the [clerk-utils]()
  library."
  (:require [clojure.data.json :as json]
            [clojure.edn :as edn]
            [clojure.java.shell :refer [sh]]
            [clojure.string]
            [ :as io]))

(def ^:private windows?
  (clojure.string/starts-with?
   (System/getProperty "os.name")
   "Windows"))

(def npm-cmd
  "System-specific NPM command, tuned for Windows or non-Windows."
  (if windows? "npm.cmd" "npm"))

(defn dep->str
  "Given a dependency name (specified as a key in the `:npm-deps` map of
  `deps.cljs`), returns an npm-compatible (string) name for the dependency."
  [dep-id]
  (cond (keyword? dep-id)

        ;; :some/foo? :react :@some/scoped?
        (subs (str dep-id) 1)

        (symbol? dep-id)
        (str dep-id)

        (string? dep-id)
        dep-id

        :else
        (throw
         (ex-info (format "invalid dependency id %s" dep-id) {}))))

(defn get-deps-from-classpath
  "Returns a sequence of maps of the form {:id <str> :version <str> :url
  <url-of-this-deps.cljs>} for every dependency specified via the `:npm-deps`
  key of each `deps.cljs` file on the classpath."
  []
  (let [xform (map
               (fn [url]
                 (-> (slurp url)
                     (edn/read-string)
                     (select-keys [:npm-deps])
                     (assoc :url url))))
        files (-> (Thread/currentThread)
                  (.getContextClassLoader)
                  (.getResources "deps.cljs")
                  (enumeration-seq))
        deps  (into [] xform files)]
    (for [{:keys [url npm-deps]} deps
          [dep-id dep-version] npm-deps]
      {:id (dep->str dep-id)
       :version dep-version
       :url url})))

(defn resolve-conflicts
  "Given a sequence of dependencies, returns a sequence of dependencies distinct
  by ID generated by picking the first one on the classpath.

  Not ideal, but this is how shadow currently does it!"
  [deps]
  (vals
   (reduce
    (fn [acc {:keys [id] :as dep}]
      (update acc id #(or % dep)))
    {}
    deps)))

(defn read-package-json
  "Returns a map with the contents of package.json if it exists in
  `install-dir` (default \".\"), {} otherwise."
  ([] (read-package-json "."))
  ([install-dir]
   (let [package-json-file (io/file install-dir "package.json")]
     (if-not (.exists package-json-file)
       {}
       (-> (slurp package-json-file)
           (json/read-str))))))

(defn is-installed?
  "Returns true if the supplied dependency is already present in some entry in the
  parsed `package-json`, false otherwise."
  [{:keys [id]} package-json]
  (or (get-in package-json ["dependencies" id])
      (get-in package-json ["devDependencies" id])
      (get-in package-json ["peerDependencies" id])))

(defn sh-print
  "Takes a vector of command pieces, prints the command and prints the results."
  [command]
  (println (str "running: " (clojure.string/join " " command)))
  (println
   (:out
    (apply sh command))))

(defn ^:no-doc install-deps
  "Given a sequence of dependencies, runs `npm install --no-save` for each
  id/version combination."
  [deps]
  (when (seq deps)
    (sh-print
     (into [npm-cmd "install" "--no-save"]
           (for [{:keys [id version]} deps]
             (str id "@" version))))))

(defn npm-install!
  "This command:

  - Triggers an `npm install` (if ./package.json is present)
  - Installs any dependency referenced by a `deps.cljs` file on the classpath

  The `npm install` makes sure that if some dependency specified in a
  `deps.cljs` is already present in `package.json` that it actually ends up in
  `node_modules`."
  []
  (let [package-json (read-package-json)
        deps         (->> (get-deps-from-classpath)
                          (resolve-conflicts)
                          (remove #(is-installed? % package-json)))]
    (when (seq package-json)
      (sh-print [npm-cmd "install"]))
    (install-deps deps)))

Sam Ritchie21:06:16

@djblue here is a documented, tidied up version

Sam Ritchie21:06:21

all of the options code works great @djblue. Any chance we can get a version cut with that code added to SCI?

djblue22:06:27

If you mean the reagent sci code, it should be released in 0.41.0, I think you might still be on 0.40.0?

Sam Ritchie22:06:56

I meant the options code

Sam Ritchie22:06:09

For use-options inside of sci

Sam Ritchie22:06:35

I added that in a local copy of portal and it works great

Sam Ritchie17:06:26

perfect, I merged the portal work yesterday, I’ve got free time this afternoon to get the rest documented and merged… then a big docs page…

Sam Ritchie17:06:39

@djblue I want to get calva set up and try out portal integration, and maybe try out github codespaces

Sam Ritchie17:06:36

I’d love a skim if you get some time @djblue, just to make sure I didn’t do anything silly

Sam Ritchie17:06:59

presumably some of this could get extracted and live as its own plugin

💯 2
Sam Ritchie17:06:07

the tex viewer is the most obvious one, with support for emmy expressions stripped out of course

Sam Ritchie17:06:52

I didn’t make this change in time: https://github.com/mentat-collective/emmy-viewers/blob/main/src/emmy/portal.clj#L33-L36 the shift from explicit slurp to requires

djblue17:06:07

For notebooks to work correctly, make sure to use the portal.nrepl/wrap-notebook nrepl middleware. I'll take a look through the code when I get a chance 👍

Sam Ritchie18:06:43

okay sounds good