Clojurians Log v2

😂 Which episode is this?

Stealing @neumann's thunder a bit here: new episode dropped early this morning in my podcast app, so I listened on my drive

It was just too funny a quote to not share

Ha! Something to look forward to for me!

I’m glad you like it! I was relistening to the episode to write up notes and I had forgotten about that. I started laughing pretty hard myself! It’s funny how much your past self can surprise your future self. 😆

I was headed to bed last night when all of a sudden I realized that I had never hit publish on the episode! Oops!

I went and published it right away, and I'll be posting announcements shortly.

Hi @neumann, @nate thanks a lot for taking the time to reply. so given we want to do some logic on a large http response, how would one typically go about that? thread macro, start with http client giving the response, maybe in a map with a single :raw or :response key, then enrich that map with a tight "entity", maybe with a schema , and then do the logic? so the final map would be sth this?

{
 :raw { ... } ; the full big bag of response data
 :person/name "adam" ; tight entity extracted from the big response map, added here by the enricher
 :person/age 34
}
; logic is then done mostly on the :person entity, while the :raw is there for easy introspection and easy enriching in future

I have some small bits where I've extended Portal to provide a dashboard into a long-running process, but for the most part my Portal set up is just Portal itself. It's an amazing tool for inspecting runtime values quickly.

Just catching up with this one. There was something early on that caught my attention. That in Clojure, data and functions compose separately. In retrospect, this is obvious. But when you said it, yet another piece clicked in for me. Thanks!

Hey 👋 Just listened to the episode. Thanks a lot for that breakdown of common kinds of pure functions, it's the first time I've heard of it. It seems to me that just the knowledge of such kinds directly helps with naming and decomplecting, and also nicely complements the categorization that https://www.youtube.com/watch?v=ROor6_NGIWU. Thanks for the podcast, it was a great listen.

@marcel187 Thank you for your feedback! I'm happy to hear you enjoyed it and you're finding it helpful. As for Rich's talk, are you referring to the following categories ("Avoiding Objects" slide)? • Transform • Move • Route • Record/remember • Keep separate • flow vs places

Yes, yes, that is the one; I should have probably included that in my first response

@marcel187 Approach B is exactly what we're talking about! Yes, it feels strange at first, but it allows you to build up the entire request so 100% of it is specified as pure data and all that's left is to just execute it. I'll give you another real world example. I've been working with the https://developers.cloudflare.com/api/operations/stream-videos-retrieve-video-details. It requires authentication, so I want to factor that out. First, I'll define some endpoints I want to work with. I create pure functions for just the part unique to each endpoint.

(defn check-status-req [id]
  {:method :get
   :path id})

(defn delete-req [id]
  {:method :delete
   :path id})

(defn full-req
  [cloudflare api-req]
  (let [{:keys [api-key account-id]} cloudflare
        {:keys [method path], throw? :throw} api-req]
    {:async true
     :method method
     :uri (format "" account-id path)
     :headers {"Authorization" (str "Bearer " api-key)
               "Accept" "application/json"}
     :throw throw?}))

(def cloudflare
  {:api-key "super-secret"
   :account-id "42424242"})

(full-req cloudflare (check-status-req "abcdefg123456789"))

{:async true
 :method :get
 :uri ""
 :headers {"Authorization" "Bearer super-secret"
           "Accept" "application/json"}
 :throw nil}

@(http/request (full-req cloudflare (check-status-req "abcdefg123456789")))
@(http/request (full-req cloudflare (delete-req "abcdefg123456789")))

(defn request!
  [cloudflare req]
  (-> @(http/request (full-req cloudflare req))
      (update :body #(json/parse-string % true))))

(defn check-status!
  [cloudflare id]
  (request! cloudflare (check-status-req id)))

(defn delete-stream!
  [cloudflare id]
  (request! cloudflare (delete-req id)))

(check-status! cloudflare "abcdefg123456789")
(delete-stream! cloudflare "abcdefg123456789")

I know we don't get a chance to share much actual code, so hopefully that's helpful!

@marcel187, I have found raw data to be useful in a couple of scenarios. For data collection, I always save the raw data because it has everything in it. I create a loader (an "ETL") that just pulls out the useful parts into a data structured used by the application. That approach is useful because I can always create another loader or extend my current one and then go back and re-ingest. For transient information, such as an API call, I will save the raw data for debugging purposes. I've used rolling logs for that kind of information. I can search it or load it when trying to figure out what went wrong. If I hit an especially strange edge case, I will have data that can reproduce it!

As for rich keys. I tend to use them for entities, but I do not generally use them for bags of entities. For me, it has more to do with something I want to pull apart or use together. So, for the "bag of entities", I can pull them apart easily using destructuring:

(let [{:keys [account request response]} data]
 ...)

(let [{name :person/name, :user/keys [role groups]} account]
 ...)

I’m on vacation, so I have a short reply for the moment… I usually have very tightly defined “entities” that are more like a record. (They can even be defrecord for performance.) I also have pretty big maps for collecting lots of data together. I view the http response as one of those. There are all kinds of top level keys in there. The structure of the values can be tightly controlled, or not, depending. Eg “:headers”

For the record-like things, I have all the functions for it in a namespace. I tend to think of that as a strong entity. I may even have schemas for it that I check on function call boundaries.

But for the open maps, it’s all about the specific keys you care about. Fetch out what you care about and know the structure of.

Those open maps are more like scratchpads of data than a data model. It’s a convenient way of grouping a bunch of data. For me, it tends to be at the top-level of the call stack and whatever sequence my code is going through.

As for visualizing deep things, you can “select-keys” to just see the parts you want, or use portal to browse the data interactively.

@marcel187 @jason.bullers Some short replies from my phone. I’ll be back at my computer next week.

@nate may be able to elaborate on his Portal setup.

Hey @neumann, thanks a lot for the reply. I've been thinking about this approach a lot. Inspired by the 099 - Repl Your World, I have started exploring much more stuff with REPL. This includes several json-based REST API servers we have to deal with at work. For that, I've put together a thin wrapper for my REPL fiddling: https://gist.github.com/mkrcah/50554447517520f4c4954f9430d0f1fa One of the things I'm wondering is how to conceptually handle the side-effecting function; see the two approaches below. I think there's a guiding principle that I yet need to discover, so I'm now listening to the Effectively Isolated episodes, trying to get better grasp on composing side-effects.

; v1 has a side-effect at the start, and requires a full request; but I can eval the full-req only if needed
(defn req-v1! [full-req]
  (-> full-req
      (http/request)
      (friendly-res))),

; v2 has a side-effect married from both sides to prepping the request and prepping the response; more ergonomic perhaps; but I cannot see the full request
(defn req-v2! [req server-conf]
  (-> req
      (full-req server-conf)
      (http/request)
      (friendly-res)))

(comment
  (req-v1! (full-req get-status server-conf))
  (req-v2! get-status server-conf))

Perhaps more on the debugging side of things. Let's say that sth goes wrong with the check-status!

(defn request!
  [cloudflare req]
  (-> @(http/request (full-req cloudflare req))
      (update :body #(json/parse-string % true))))

(defn check-status!
  [cloudflare id]
  (request! cloudflare (check-status-req id)))

(check-status! cloudflare "abcdefg123456789")

@marcel187 Thanks for continuing this thread. You definitely have some options if something goes wrong. Generally, I don't like to make helper functions that do anything to the response. I like to get all of it back. For example, the babashka.http client includes the request map in the response map under the :request key. Also, you can pass :throw false in the request map to keep it from throwing when it gets non-2xx responses. Even if it does throw, you can call ex-data to get all the information back out.

So, if you had some issue in the way you constructed the request, you could probably find it that way.

However, the pure functions are totally deterministic, so you can always evaluate a one-off full-req like so:

#_(full-req get-status server-conf)

Even in the code you have above, for your req-v1! part, you can still evaluated the (full-req ...) form without evaluating the entire form.

Even if the library doesn't help with that, you can always assoc in whatever context you want, or even use https://clojure.org/reference/metadata if you can't assoc.

(defn request!
  [cloudflare req]
  (let [request (full-req cloudflare req)]
    (-> @(http/request request)
        (update :body #(json/parse-string % true))
        (assoc :request request))))

A map is completely open data. The idea is to put as much context as you want in the map. Since it's all persistent data structures, you're not using more memory. You might be keeping some things in scope a little longer, but it's extremely helpful to have more context when debugging.

@marcel187 I'm happy to keep chatting about this.

> A map is completely open data. The idea is to put as much context as you want in the map. > @neumann I think this is a really hard one to feel comfortable with. Coming from Java, it's natural to think of maps as entities (and even in Clojure, it seems you do think of them this way, to a point at least). It always seems smelly to just dump things in there that don't really have much, if anything, to do with the entity itself. It's kind of like if I defined a Person class like:

class Person {
  String name;
  int age;
  Map<Object, Object> extraStuff;
}

(Thank you very much for continuing this. I'm on a short family vacation till the end of this week, I will look into your replies next week. )

Hi @neumann Thanks again for replying here. I've listened to the latest episodes (Testify and Extractify) and it starts to come together. For data collection, I like the idea that you mentioned of having working data that gives us sth to reason with. I've refactored my code accordingly and see the benefits you were mentioning. For rolling logs and transient data, I don't fully get what you mean, but I believe the topic of transient data for debugging will be discussed this week on the podcast, which I'm all much looking forward to 🙌 Thanks also for that metaphor of tearing off part of the tree 💡

@neumann Thanks for the replys, it feels great to be chatting with you here after listening to the podcasts. > Generally, I don't like to make helper functions that do anything to the response. Yeah, it seems indeed that there's generally no harm in keeping the data in there for production code. However, for fiddling around, I was running into usability issue and had to look into a gigantic map in order to see just the information I needed. Maybe there's a better way for that that I don't know of? > but it's extremely helpful to have more context when debugging. I've noticed you have updated the body with the parsed json, overwriting the raw api response. What if the response was malformed, wouldn't the update make it harder to find the cause? At the same time, perhaps it's a stable API which never returns malformed body, so one might cut a corner there? @jason.bullers I think I'm slowly starting to get this idea of an open map, although coming from typed scala, it also feels quite unfamiliar. For my personal finance app, I have a vector of transaction maps and I want to add information to each transaction from other sources, such as exchange rate. My initial idea was to use another data structure that would nest the raw transaction map, but then I though: wait, how would it be if I'd put all those pieces of data into a single flat map and use namespaced keys? It felt unconventional at first, but I think I start to slowly feel the potential powers of it. (still have to listen to the episodes about maps though)

Listening to episode 026 (one call to rule them all), there was this big aha moment at the end: keep side-effects as high in the call stack as possible and try to adhere to that principle and see where it takes you. coupled with a single entry to api client, it feels that it all starts to very slowly come together: separate request building (pure fn) from request execution (side-effect), which in turn enables the single entry api client, and opens the power of clojure core for the pure part, and relates also to IO side-effect to be up the stack trace instead of being buried deeper.

Going through your answer, I see the pragmatism and advantages of the two uses-cases you mentioned, thanks for enumerating those. As I think about what you wrote, two questions emerged: • Using raw data: Does your business logic ever reach to the raw request/response, or do you always go through the parsed known entity and keep the raw data only for diagnostics/forensics? ◦ always going through parsed data gives us control, less surprises, consistency in key naming; but it creates a layer between the raw data and domain logic that might consume mental bandwidth and require code to write. what are your thoughts about this? • Rich keys vs nested? ◦ In your open bag of entities example, the IO is a rest api, and you used a map of :request, :response and the known entity :account , where the known entity :account is flat with rich keys, as per https://clojuredesign.club/episode/045-why-have-derived-fields-in-data-when-i-can-just-calculate-derived-data-as-needed-with-a-function/ and https://clojuredesign.club/episode/051-maps-maps-maps/. ◦ However, in the log serie (https://clojuredesign.club/episode/029-problem-unknown-log-lines/), the IO was a file, and you put together in one flat map with rich keys both the raw data (`:raw/line`) and the known entity (`:log/date, :log/time`, etc) ◦ When to use an open-bag of entities and when a flat map with rich keys? ◦ Thinking on this, maybe: if raw data is complex (maybe 5+ fields, has headers/metadata/etc), use nested open entity, otherwise use flat map with rich keys.

@neumann, That is great news, I'd love listening to that episode. Thank you for taking the time to reply and continue this discussion. I need time to think about and process what you wrote, I will get back till the end of the week.

@john.t.richardson.dev Thanks for sharing! I'm happy to hear it clicked! In my own bizarre imagination, I sometimes picture a class as an egomaniac that just can't let data be the focus of attention. When someone walks up to Data and tries to talk to it, Class has to jump in front of Data to answer all the questions. "I'll take it from here." In that world, Class only has significance because of Data, so Class can't let Data go! If Data breaks free, Class is just one of many others opining on Data, not Data's one and only. See? Bizarre imagination! 😂

There's a dilemma about decomplecting that I keep pondering about after listening to the composition serie, and I'm quite curious how others approach this. (sorry if this is a question with an obvious answer, I'm only getting into the clojure way of coding) Say that one needs to call a REST API, say with a clj-http. Such a call might have two concerns: 1. constructing the request (url, method, body, etc) - functionality is pure, without side-effects 2. calling the API, with a side-effect How to capture this in code? Two approaches come to mind: • Approach A is to mix the two concerns in one function • Approach B is to separate two concerns into one pure function and one side-effect function. Here's a real-life example from an app I'm building right now:

; Approach A: Mixing concerns
(defn fetch-from-public-api [currency-pair period]
  (let [url (str "" (:start period) ".." (:end period))]
    (client/get url
      {:accept :json
       :as :json
       :query-params {:to (:quote currency-pair)
                      :from (:base currency-pair)}
       })))

; Approach B: Separating concerns
(defn prepare-request [period currency-pair]
  {:method :get
   :url (str "" (:start period) ".." (:end period))
   :accept :json
   :as :json
   :query-params {:to (:quote currency-pair)
                  :from (:base currency-pair)}})

(-> (prepare-request period currency-pair)
    (client/request))

@marcel187 I like the idea of talking about this for a podcast. I made a note of that. Thanks for the suggestion In my own code, I have a couple of cases where I use open maps: 1) an open bag entities, and 2) an open entity. Consider this:

{:request { ... } ; the full HTTP request
 :response { ... } ; the full HTTP response
 :account
 {:person/name "adam"
  :person/age 34
  :user/role :moderator
  :user/:groups [:alpha, :gamma, :omicron]}}