Clojurians Log v2

one of my worst api using experiences was using a tag-soup wrapper that wrapped tags in a lazy seq. it created threads for parsing the tags, and didn't free any threads until you read to the end of the document.

create the resource outside the seq context so that it can be released by normal gc scope rules even if the lazy seq is never fully consumed

Tellman’s Elements of Clojure has some things to say about what goes in a single namespace iirc

"In theory, a namespace can hold an unlimited number of functions as long as none of them share the same name. In practice, namespaces should hold functions that share a common purpose so that the namespace lends narrowness to the names inside it." -- page 20 And, perhaps more importantly: "A large number of namespaces is taxing for our readers; if we have ten tables in a database, creating ten different namespaces just so we can write europa/get rather than db/get-europa has questionable value. Therefore, we should add new namespaces only when necessary. By questioning the need for new namespaces, we implicitly question the need for new datatypes and data scopes, which will lead to simpler code overall." -- page 20 (edited to fix spacing issues from copy'n'paste from the PDF of the book)

I would not follow the one function per file approach. Although I really like the idea, I think it does not matter. It is a non technical, non structural decision with no real impact. It does not make programming easier, but punishes you when you want to rename something. It only affects how you see the function in the project structure. Nothing else. For me that is not worth the extra effort. An outline view does the same. What I really want is programming in blocks. Each function is a block (maybe hold in a database). But one function per file does not bring you there,

I'd get motion sickness from jumping back and forth all those files (if I were to do that in my projects)

I think in general you'd structure your codebase and arrange stuff into namespaces/modules/etc in a way that supports the task at hand and reflects the architecture. Single function namespaces and naming them all f would initially look like polymorphism to me, and if that's all the library does, why not, as long as form follows function and it's not the tail wagging the dog. Performance being mentioned, I think the vector arithmetics example would seem like an easy thing to do a bit of benchmarking on and A/B test single-function-namespaces vs. all related functions in single namespace

Here is the kind of code I have to write many times. it is ugly, but I understand it pretty well. It just seems to me there ought to be a more beautiful way. The issue is that line-seq returns a lazy sequence of lines. If that value gets returned from find-baby-lines then the resource gets closed before the caller can read from it. I show this to my students as an example about how state and laziness are not friends of eachother. That being said, I really don’t know an elegant way to handle this.

(defn find-baby-lines
  "Fine the lines from the csv resource corresponding to
  babies of a given name for a given year and state"
  [name-target gender-target state-target]
  (let [txt-file (format "US-baby-names/namesbystate/%s.TXT" state-target)
        s (io/resource txt-file)]
    (assert s (format "resource not found: %s" txt-file))
    (with-open [r (io/reader s)]
      (assert r (format "cannot create reader from %s" s))
      (filter (fn [[state-raw gender year-raw name-raw count-raw]]
                (and (= state-raw state-target)
                     (= name-raw name-target)
                     (= gender gender-target)))
              (map
               (fn [line] (split line #"[,]"))
               (line-seq r) ;; WARNING this returns a lazy sequence
               )))))

@p-himik yes, if I understand your comment correctly, indeed wrapping a doall around (line-seq r) will give me the value I need, but defeats the purpose of the lazy sequence. Of course doall could be wrapped around (filter …) but inside (with-open ...) instead, which is probably the best fix for this particular situation.

Yeah, the major downside to this is losing out on locality of behaviour - namespaces work best when they group a bunch of domain-related stuff to read and review all in the same place (re. human cognition and chunking - to a compiler of course it makes no difference) And then you'd be introducing a layer of coordination tax and risk of drift across conceptually related bits of code - unless the dependency graph is very carefully designed I imagine you'll end up with orders of magnitude more nodes and it could get out of hand very easily

Java gets away with one-class-per-file because it's enforced on a language level and gets lots of IDE / tooling support that way, which is definitely not the case for Clojure

doall in my code is just to demonstrate that it works via REPL. It has nothing to do with running some code when the seq is consumed - it can be consumed via any other means. But yeah, that code shouldn't be used anyway. :) with-open in your code should be outside of find-baby-lines and whatever code that consumes its result.

Another aspect where I think this would feel awful is REPL driven dev - it's natural to treat the namespace as a single unit you can eval in one go, with functions scattered around different files you'd be fighting the system and throwing in :reload-all all over the place

I’m not sure whether this makes it more obvious or more confusing. I’ve grouped the boilerplate into a single function.

(defn with-baby-names
  "`unary` is a unary function which will be called with a lazy sequence
  of lines from the resource.
  `unary` should be something like this (fn [lazy-seq] ... (for [line lazy-seq] ...))"
  [state-target unary]
  (let [txt-file (format "US-baby-names/namesbystate/%s.TXT" state-target)
        s (io/resource txt-file)]
    (assert s (format "resource not found: %s" txt-file))
    (with-open [r (io/reader s)]
      (assert r (format "cannot create reader from %s" s))
      (unary (line-seq r)))))

(defn count-babies-3
  "Count babies of a given name, given gender, given state, but all years"
  [name-target gender-target state-target]
  (with-baby-names state-target
    (fn [lazy-seq]
      (reduce + 0
              (for [line lazy-seq
                    :let [[state-raw gender year-raw name-raw count-raw] (split line #"[,]")]
                    :when (= state-raw state-target)
                    :when (= name-raw name-target)
                    :when (= gender gender-target)]
                (Integer/parseInt count-raw))))))

I know this is for educational purposes, but I would probably do something like:

(require '[clojure.string :as str]
         '[ :as io])

(with-open [lines (line-seq (io/resource
                             (format "US-baby-names/namesbystate/%s.TXT" state-target)))]
  (into []
        (comp
         (map (fn [line]
                (str/split line #",")))
         (filter (fn [[state-raw gender year-raw name-raw count-raw]]
                   (and (= state-raw state-target)
                        (= name-raw name-target)
                        (= gender gender-target))))
         (map (fn [[_ _ _ count-raw]]
                (parse-long count-raw))))
        lines))

you could also further factor it like:

(defn map-parse-rows []
  (map (fn [line]
         (str/split line #","))))

(defn filter-state-name-gender-rows [state name gender]
  (filter (fn [[state-raw gender year-raw name-raw count-raw]]
            (and (= state-raw state)
                 (= name-raw name)
                 (= gender gender)))))

(defn extract-row-count []
  (map (fn [[_ _ _ count-raw]]
         (parse-long count-raw))))


(with-open [lines (line-seq (io/resource
                             (format "US-baby-names/namesbystate/%s.TXT" state-target)))]
  (into []
        (comp
         (map-parse-rows)
         (filter-state-name-gender-rows state name gender)
         (extract-row-count))
        lines))

rather than doall, I think forcing evaluation using into (even if you don't use transducers) is more readable

(into [] my-lazy-seq)

I see that you are using split. I assume that it is from clojure.string. I would recommend not refering function names. It makes it hard to tell which vars are in clojure core from which vars are in other namespaces.

Integer/parseInt can be replaced with parse-long.

Especially for educational purposes, I would separate 1) splitting the rows by , and 2) filtering rows by name/state/gender into distinct steps

If you are trying to avoid transducers, I would probably do something like:

(with-open [lines (line-seq (io/resource
                             (format "US-baby-names/namesbystate/%s.TXT" state-target)))]
  (->> lines
       (map (fn [line]
              (str/split line #",")))
       (filter (fn [[state-raw gender year-raw name-raw count-raw]]
                 (and (= state-raw state-target)
                      (= name-raw name-target)
                      (= gender gender-target))))
       (map (fn [[_ _ _ count-raw]]
              (parse-long count-raw)))
       (into [])))

you could still move filtering and extracting the count into their own functions

(defn filter-state-name-gender-rows [state name gender rows]
  (filter (fn [[state-raw gender year-raw name-raw count-raw]]
            (and (= state-raw state)
                 (= name-raw name)
                 (= gender gender)))
          rows))

(defn extract-row-count [rows]
  (map (fn [[_ _ _ count-raw]]
         (parse-long count-raw))
       rows))

(with-open [lines (line-seq (io/resource
                             (format "US-baby-names/namesbystate/%s.TXT" state-target)))]
  (->> lines
       (map (fn [line]
              (str/split line #",")))
       (filter-state-name-gender-rows state name gender)
       (extract-row-count)
       (into [])))

https://ce2144dc-f7c9-4f54-8fb6-7321a4c318db.s3.amazonaws.com/reducers.html may be an interesting read

(it predates the introduction of transducers in clojure.core, but transducers woth with CollReduce as well as reducers do)

The main difference with the other stuff in this thread is it pushes the with-open into a custom type implementing CollReduce

yea, I think that's a good way to do it

I was tempted to recommend https://github.com/cgrand/xforms/blob/78076f8cd078ebb336ed9047df873ebc2ecf3aa1/src/net/cgrand/xforms/io.clj#L19, but I wasn't sure about transducers + external libraries.

oh, yeah, I always forget about xforms