how do y'all go about performing major refactors?

any recommendations as I embark on my own?

(major being about 5500 lines of source total, lol)

I'd want to start out with pretty good test coverage, to be honest, for a refactor of that scale.

Also, "refactoring" for me tends to be small, incremental changes that I can verify in the REPL, via spec, or a small set of tests/test expressions. Tackling 5,500 lines sounds more like a "rewrite"...

I'm mostly attempting to spread what is currently one giant namespace across 11 files into a bunch of namespaces, as dealing with the interconnectedness is reaching high complexity

my goal is to actually change as little as possible, more some much needed organization

Depending on what the code currently does, maybe you can come up with some scenarios where you could generatively test the old code and the new code produce identical results for random, conforming input data?

Not a bad idea in general! this is a game-engine, so generating correct random data is a little harder. Thankfully, I have a fairly extensive integration test suite so I can rely on that for now

thanks for the ideas!

As a sanity check on namespace size, we have just under 300 src namespaces in our codebase at work. Less than a dozen have more than 1,000 lines. Only one exceeds 2,000 lines (it has 2,017 lines). The total is 65k lines.

holy crap

I'm glad to know "many namespaces" works even when larger

We also have just over 300 test namespaces, totaling just under 20k and only two namespaces exceed 1,000 lines (one is just over 1,000, one is just under 2,000).

Code organization is an ongoing issue. We're pretty much constantly looking at small refactorings to improve navigability and cognitive load.

that's my biggest issue: navigability and cognitive load

We started with a pretty flat structure and some fairly large namespaces. Then we moved to a more nested structure and smaller, more numerous namespaces.

interesting, thank you

Clojure build/config 50 files 1000 total loc
Clojure source 276 files 64691 total loc,
    3065 fns, 625 of which are private,
    411 vars, 29 macros, 56 atoms,
    504 specs, 19 function specs.
Clojure tests 307 files 19396 total loc,
    4 specs, 1 function specs.

We have lots of little EDN files for configuration -- and each subproject in our monorepo has its own deps.edn file.

That's super cool

Hi 👋 how's #bangalore-clj doing??

@thegobinath going gud

I like flat and big namespaces personally.

I was wondering if anyone else would chip in. I've heard pros and cons of both.

We're reorganizing our code because some namespaces have become sort of "kitchen sinks" of stuff that's only vaguely related to the original purpose (and should have been put elsewhere in the first place).

We've also evolved our thinking a lot on just naming stuff in the seven years we've been working on this codebase.

(it still seems amazing to me that we have had Clojure in production for seven years now!)

We joke at work that ‘everything is in utils’

Yeah, it's really hard to avoid having some sort of "util" namespace I think 🙂

The con of a big flat namespace is that reloading takes ages

and that the likelyhood of having to reload that huge NS is steeply increased

util is another name for “junk drawer”

premature splitting of namespaces is a sin imo. I prefer to grow an ns until natural splits reveal themselves.

lots of util namespaces often contain a significant number of fns which never see reuse across a codebase in my experience. And when they do it is often limited to a couple instances.

not saying that’s a rule, but I’ve actually dumped disparate namespaces back into one big namespace to tease out the right splits after the fact.

Premature splitting forces new names into existence which often aren’t the names that capture what’s actually going on in the domain.

I strongly believe applications are better organised first by vertical/features then inside them horizontally. There are many reasons the main being: 1. People tend to work on a feature at a time. 2. Parallelising work is easier to do by feature, so structuring your code by feature means less merge conflicts. 3. Code is much easier to navigate, and it’s way easier to find things and on-ramp people. You can get a good idea on what an application does by looking at it’s top level namespaces. 4. When creating new code it’s much more obvious where to put it; which leads to greater consistency. Cross cutting concerns can be colocated under an app.concerns, or even another root namespace, i.e. an app-lib ns; and eventually moved into separate libraries if it makes sense. The biggest barrier to doing this, is that most clojure app templates organise horizontally by the framework first, e.g. they create you an app.handler, app.model namespace first. I also agree that you shouldn’t prematurely split; beyond giving each feature its own ns.

so a trivial app.feature ns may contain the view and model code (still separated but together)… at some point you may break it out into app.feature.view and app.feature.model. A good reason to do this is when you start wanting to test the models and views independently; so it will give you parallel test ns’s.

Libraries on the other hand make often make more sense to organise horizontally… though usually each library is a horizontal anyway.

@rickmoynihan have you seen the behaviour programming paradigm?

https://lmatteis.github.io/react-behavioral/

I don’t think so

The demo can explain it better, but it's essentially about implementing each feature independently

looks like it’s based on CSP — or something similar

but that’s just based on a 20 second glance

It is possible to use functions in lein project.clj file. But it doesn't work here: :essthree {:deploy {:type :uberjar :bucket "uberjars.foo" :path "bar" :artifact-name ~#(str (:version %)) }}

~#(str (:version %)) function gets converted to a string

Any suggestions for a Clojure library doing CLI args parsing with the ability to add arbitrary sub parsers like Python's argparse? I tried tools.cli which seems too low level and have to write a lot of manual parsing and boilerplate myself. Also tried https://github.com/l3nz/cli-matic which is based on tools.cli and is quite nice and declarative but it doesn't allow nested parsers yet.

@rahul080327 I saw this yesterday. I don’t know if it supports what you want, but it looks cool. maybe it works from clojure? https://picocli.info/

it seems to be annotation-heavy and no idea how that interacts from clojure

I see there’s a Clojure issue here: https://github.com/remkop/picocli/issues/231 so probably not suitable right now

I think clojure could work fine with that via interop

@borkdude Those were precisely my thoughts too. The best contender for a Clojure wrapper I see is https://argparse4j.github.io/ but was hoping if there is a more data driven solution out there

write one based on that? seems pretty useful

could definitely try that. picocli's features seem quite cool though. @dominicm whats the best way of doing a Java interop with java annotations?

I think it's reify

But I'm trying to go off the top of my head

right. Im not familiar with this kind of interop. Will try looking into it.

How about commons CLI?

https://commons.apache.org/proper/commons-cli/

@dharrigan Im not sure if it supports subparsers

kk

Hm what’s wrong with tools.cli? https://github.com/clojure/tools.cli#in-order-processing-for-subcommands

> Note that the options to log are not parsed, but remain in the unprocessed arguments vector. These options could be handled by another call to parse-opts from within the function that handles the log subcommand.

Define options, parse, check if there’s an arg, parse its options against another sub-command definition

@artpsdev I felt its too low level and involves me writing a lot of boiler plate for validation and dispatch. https://github.com/l3nz/cli-matic built on top of it has a more declarative API and spec validation, but hides the subcommand aspect

the problem came from writing

(defn validate-args
  "Validate command line arguments. Either return a map indicating the program
  should exit (with a error message, and optional ok status), or a map
  indicating the action the program should take and the options provided."
  [args]
  (let [{:keys [options arguments errors summary]} (parse-opts args cli-options)]
    (cond
      (:help options) ; help => exit OK with usage summary
      {:exit-message (usage summary) :ok? true}
      errors ; errors => exit with description of errors
      {:exit-message (error-msg errors)}
      ;; custom validation on arguments
      (and (= 1 (count arguments))
           (#{"start" "stop" "status"} (first arguments)))
      {:action (first arguments) :options options}
      :else ; failed custom validation => exit with usage summary
      {:exit-message (usage summary)})))

plus a dispatcher like this:

(defn -main [& args]
  (let [{:keys [action options exit-message ok?]} (validate-args args)]
    (if exit-message
      (exit (if ok? 0 1) exit-message)
      (case action
        "start"  (server/start! options)
        "stop"   (server/stop! options)
        "status" (server/status! options)))))

Let the root command validate common args and sub-commands validate their own args so the life is simpler.

Isolation is always good

i totally agree. what im having trouble with is writing all the (cond ...) and the ifs by hand. And that makes it more error prone. cli-matic for example gives me the ability to define the whole thing in a map:

(def CONFIGURATION
  {:app         {:command     "toycalc"
                 :description "A command-line toy calculator"
                 :version     "0.0.1"}

   :global-opts [{:option  "base"
                  :as      "The number base for output"
                  :type    :int
                  :default 10}]

   :commands    [{:command     "add"
                  :description "Adds two numbers together"
                  :opts        [{:option "a" :as "Addendum 1" :type :int}
                                {:option "b" :as "Addendum 2" :type :int :default 0}]              
                  :runs        add_numbers}

                 {:command     "sub"
                  :description "Subtracts parameter B from A"
                  :opts        [{:option "a" :as "Parameter A" :type :int :default 0}
                                {:option "b" :as "Parameter B" :type :int :default 0}]
                  :runs        subtract_numbers}
                 ]})

also gives me the function dispatch too

I’m guessing the consensus might be that this is overkill and/or maybe too low level … but I think from now on, whenever I need to parse anything (including command line stuff), I’m going with instaparse. It’s so awesome and you have so much control. instaparse + a cond statement ftw 😉

Does anyone know how to find from which remote repo (ex. enonic, maven, clojars) that a project dependency wall pulled from using leiningen?

https://memegenerator.net/img/instances/x300/84483335/what-does-quote-say-to-repl-not-now.jpg

Regarding quotes

(meta ^:m 'm) returns nil

so replacing the ' with (quote is the only way out or is there something else?

with-meta works

I’m often confused by the ^ meta reader, it seems to have some surprising behavior in certain cases (like this) so the only time I use it now is in a def to attach meta to a var

otherwise, I use with-meta explicitly

^:m 'm is attaching meta to the compile time expression (quote m)

what you want to do is '^:m m

the behavior of ^ here is perfectly consistent

another way to understand what is going on here is to use 1 instead of m

user=> ^:foo '1
1
user=> '^:foo 1
Syntax error reading source at (REPL:3:0).
Metadata can only be applied to IMetas

it might be entirely consistent 😅 I just find it doesn’t match what I want to do most of the time, so my tiny brain figured out with-meta is usually what I want

with-meta operates at runtime, ^ at read-time

gotcha

(in fact, ^ is just with-meta at read-time)

I’m actually not sure why '^:m works

it expands to (quote ^:m m)

anyone have the @hiredman ticket about iterating over a paginated collection?

can't remember enough to find it

ah found it. https://dev.clojure.org/jira/browse/CLJ-1906.

I wonder if that’s a use case for nav / datafy in actual application code

interesting idea

Hoping someone here might have some experience: I'm trying to set a log level inside a log4j.properties file just for a specific clojure namespace. Is there an easy way to do this that doesn't involve using Java interop to set the logging level inside the clojure code itself?

is it just me or does 400-500ms for 800 insert-multi! rows in jdbc seem… like a lot

are you sure that is time is being burnt in jdbc and not in your db?

i’m not sure, i just have an adhoc profiling call around the insert

any ideas on how i can dig further?

If I recall correctly — but I am by no mean an expert — insertion performance depends on how many indices there are on your rows

i have a primary key on 3 of 4 columns in the table

maybe there’s a way to get an explain out of jdbc?

You could try inserting the same rows using the mysql client/psql, see how long that takes. I wouldn’t expect something like jdbc to add any large amount of overhead. But this is just a hunch, you should ask Sean Corfield when he’s around!

@gtzogana How exactly are you inserting multiple rows?

insert-multi!

So you said. What exactly does your call look like?

so it’s provided with a vector of column names, and then 800 tuples for row values

if that’s what you mean

If you provide multiple maps, it does multiple inserts (per the docs). If you provide a vector of column names and a vector of rows (a vector of vectors of row values), it tries to do a single insert -- but the actual behavior is, like so many JDBC-related things, database-specific. For example, PostgreSQL requires that you provide a special option.

what does the option indicate?

The auto-gen'd docs haven't been updated for a while (infrastructure issues, I believe), but I've recently added this paragraph to the docstring for insert-multi!

Note: some database drivers need to be told to rewrite the SQL for this to
  be performed as a single, batched operation. In particular, PostgreSQL
  requires :reWriteBatchedInserts true and My SQL requires
  :rewriteBatchedStatement true (both non-standard JDBC options, of course!).
  These options should be passed into the driver when the connection is
  created (however that is done in your program).

So if you're on PostgreSQL or MySQL, you'll need to provide the appropriate option in your db-spec when you create your DB connection.

(and, of course, they're different options -- thank you JDBC 😞 )

BTW, there's #sql for SQL-specific / JDBC-specific questions -- which I'm much more likely to see as I don't have that channel muted (but I do have this one muted).

(JDBC is one of the biggest piles of frustrating, non-standard, quirkiness I've ever had to work with!)