Well done

Most of it is just a matter of preference imo, but something you might consider changing is get-setting . It reads the entire config-file from disk every time a key is extracted, so given that e.g. :local/repo-root-dir is read on every repo you clone, that's gonna do a lot of IO. You might consider reading the settings into a hashmap once with def , then look up keys in it when needed Other than that, process/sh supports varargs, so if you change the signature of sh-out->json to take & args then you won't have to wrap your shell commands in lots of string stuff

Thank you, @U0422G22932! Those are some good improvement suggestions. I actually tried to make the function sh-out->json take & args, but I struggled to get it working (got some confusing errors), then abandoned that idea to finish the script and forgot to come back to it. I'll see if I can get that working now. Also, that's a very good point about excessive IO when reading settings! I totally missed that. Thank you for the suggestion.

Don't know if it helps, but the process functions now also take an explicit {:cmd [...]} option in which you can provide the strings

so (sh {:cmd ["foo" "bar"]})

Oh, yes, that might also make it easier to tidy things up a bit, @U04V15CAJ! Thanks for the tip.

Hmmm. I'm struggling with using & args here.

(defn sh-out->json
  [& args]
  (-> (process/sh {:cmd (vec args)})
      :out
      (json/parse-string true)))

(defn get-github-repo-data
  []
  (sh-out->json "gh repo list" (:github/org-name settings)
                 "--language" (:github/repo-language-filter settings)
                 "--source --no-archived --limit 1000 --json sshUrl"))

(get-github-repo-data)

["gh repo list" "myorg" "--language" "C#" "--source --no-archived --limit 1000 --json sshUrl"]

@U01PE7630AC what bb version are you using

you might have to upgrade if it's older than a few weeks

Ahaaa, that might explain a few things. I'm using this version:

bb --version
babashka v1.3.179

the :cmd thing was introduced in 180 I think

You can still do it the "old" way like this

(defn sh-out->json
  [& args]
  (-> (apply process/sh args)
      :out
      (json/parse-string true)))

You have a mixed arg list, by the way, which might cause problems

(sh-out->json "gh repo list" (:github/org-name settings)
                 "--language" (:github/repo-language-filter settings)
                 "--source --no-archived --limit 1000 --json sshUrl")

true that

Ooooh! I thought each parameter and its value had to be together as one string, as in the example with "--limit 1", that won't work. Let's see…

Easier to either tokenize everything, or pass the entire thing as a single string

I don't recommend stringifying arguments yourself, since those arguments might contain spaces

e.g. if you want to pass "--title" "foo bar" and you pass a single concatenated string this becomes "--title" "foo" and "bar" while you want to pass "foo bar" as a single string

so just stick to separate strings as much as possible

I got rid of the exception like this:

(defn sh-out->json
  [& args]
  (-> (apply process/sh args)
      :out
      (json/parse-string true)))

(defn get-github-repo-data
  []
  (sh-out->json "gh"
                "repo"
                "list"
                (:github/org-name settings)
                "--language"
                (:github/repo-language-filter settings)
                "--source"
                "--no-archived"
                "--limit"
                "1000"
                "--json"
                "sshUrl"))

{:proc #object[java.lang.ProcessImpl 0x65fde901 "Process[pid=99528, exitValue=137]"],
 :exit 137,
 :in
 #object[java.lang.ProcessImpl$ProcessPipeOutputStream 0x38b10a43 "java.lang.ProcessImpl$ProcessPipeOutputStream@38b10a43"],
 :out "",
 :err "",
 :prev nil,
 :cmd
 ["gh" "repo" "list" "myorg" "--language" "C#" "--source" "--no-archived" "--limit" "1000" "--json" "sshUrl"]}

bb --version
babashka v1.3.181

Works on my machine, with my username (and no language-filter since i dont have your config)

(defn sh-out->json
  [& args]
  (-> (apply process/sh args)
      :out
      (json/parse-string true)))

(defn get-github-repo-data
  []
  (sh-out->json "gh" "repo" "list" "joakimen" "--source" "--no-archived"
                "--limit" "1000" "--json" "sshUrl"))

(get-github-repo-data)

;; => ({:sshUrl "[email protected]:joakimen/dotfiles.git"}
;;     {:sshUrl "git@github..

note: :exit 137

this usually means "out of memory"

you can insert a process/check in between if you want to crash on non-zero exit codes

Ah! What in the world… I restarted VS Code and the REPL (via Calva), and now it works.

Thank you so much for the help, guys! Sorry to waste your time on a "fluke."

It's all good 😄

I've pushed a new version of my script now, with your suggested changes 🙂 Much better! https://github.com/leifericf/leifs-utils/blob/main/src/leifs_utils/git.clj

For comparison, https://martdegraaf.github.io/posts/consulting/git-clone-all-repos-azure-devops/#clone-all-repositories, which only gets repos from Azure DevOps (not GitHub), and presumably only works on Windows 🙂 Babashka is sooo nice!

I tried to adopt Power(S)hell a while back, I could never get into it. Absolutely has its uses when it comes to managing services, servers, working with Active Directory etc, but for a general purpose script language.. 🙅.

Indeed! Also, compared to the PowerShell script, my Babashka version is parallelized, and each function may also be used independently. And it can be made into a Task (my next to-do), etc. And it's cross-platform.

But I guess I should also make it do git pull instead of git clone if the repo already exists on disk. And maybe also add some output to the terminal to show the user what's going on whey they run it.

Yeah, but then you're suddenly dealing with a slightly different can of worms (dirty repos, missing upstreams, ..) Sounds useful though

Yeah, that's true. There are many other issues. I also want to write some functions to find all Git repos from a root directory, then search through the code and commits (code diffs) to find certain classes, method usages, etc., across all projects and their history. This is useful because we often must apply similar fixes to different microservices. We often need to find a different microservice where a fix has already been applied, etc.

Is there a similar version for GitLab? I'd like to be able to checkout multiple Groups. @U01PE7630AC

My script currently does not work with GitLab, but I see https://docs.gitlab.com/ee/integration/glab/also have a CLI. It should be relatively straightforward to adapt my script to use glab repo list (https://gitlab.com/gitlab-org/cli/-/blob/main/docs/source/repo/list.md), and Git commands to mass-clone repos from there. But I don't use GitLab, so I can't say for sure.

You should only have to change this one function:

(defn get-github-repo-data
  []
  (sh-out->json "gh" "repo" "list" (:github/org-name settings)
                "--language" (:github/repo-language-filter settings)
                "--source"
                "--no-archived"
                "--limit" "1000"
                "--json" "sshUrl"))

And maybe you need to change this function if GitLab's CLI doesn't return a JSON-response with the key sshUrl:

(defn clone-all-repos
  [repos]
  (->> repos
       (extract-key :sshUrl)
       (pmap clone-repo)))

all-ns only returns the namespaces that are already loaded (exist in memory), not those on disk

I see. Cheers on the speedy reply. I'm trying to grab vars with specific metadata. Can you think of an simple alternative route to do this programmatically?

you can use tools.namespace to scan existing files on the classpath to get their namespaces

There it is! That's perfect and well within grasp. CHEERS again!

$ bb -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "RELEASE"}}}' -e "(require '[clojure.tools.namespace.find :as f]) (f/find-ns-decls-in-dir (io/file \"src\"))"
((ns cherry.compiler (:require [cherry.resource :as resource] [cherry.internal.deftype :as deftype] [cherry.internal.destructure :refer [core-let]] [cherry.internal.fn :refer [core-defmacro core-defn core-fn]] [cherry.internal.loop :as loop] [cherry.internal.macros :as macros]

(replace RELEASE with a proper version)

What's with the fork you've got of tools.namespace?

no longer needed

where did you find this fork? I should archive this if it's not already

https://babashka.org/toolbox/ links to https://github.com/babashka/tools.namespace

ok, I'll remove that, thanks

While I've got you, do you know of anyone who's does their CI/CD pipeline with bb fns? Particularly in GH actions?

I had the bright idea to just do it as babashka fns in an interceptor chain. It sounds simple enough, but I would listen to wisdom before jumping in.

Yes, I do this all the time, but mostly using babashka tasks: https://book.babashka.org/#tasks

you can look at the bb.edn of this project for a good example: https://github.com/clj-kondo/clj-kondo.lsp

I just type bb publish on the command line and then everything is kicked off

or you can run it from CI

Yeah, I'm cool with the tasks idea. I was more leaning towards defining simple fns and then building chains based on plain data which would lead nicely into a build pipeline that is stored in a plain vector in an edn file and checked into git for the whole gitops thing....

Yeah, that task route does give some cool cli integration thought. I like those doc strings

That's another way to do it. You can hook up tasks to any function you define elsewhere in your bb code

Hmmm many thoughts, thanks for the help. I'm too amped up about today's work to keep chatting. You're the best. Much love.

Fwiw we're moving our CI to bb as well. I think it's a match made in heaven

nbb + ink perhaps? https://github.com/babashka/nbb

https://rattlin.blog/bbgum.html

https://github.com/lispyclouds/bblgum

nbb + ink is also a good option

thanks folks, playing with ink here cause what I need is a terminal UI that gets updated upon events, I couldn't find how to do that with gum

https://github.com/babashka/nbb/tree/main/examples/ink