get-in/update-in with a defined path not enough?

an example of what you have and what you'd like it to look like is always helpful

Meander and Specter are the two that I see people talking about. We don't use them (at work) -- we don't find any need for that sort of thing.

I'm greenfield, so my understanding of the problem is growing as well as my needs. I've got a large map of attributes that I'd like to grow as so... {:attribute-name {:datomic [:short :schema :def] :type :boolean :migrations {:csv {:table :table-name :column :col-name}} :fulcro {:reports [:names :names] :allowed-components [:components :list]} :alexa true :aliases #{:short-name :spanish-name :etc} }} I moved the datomic type out of the vector into it's own higher level key yesterday. Then had to deal with the cascade of tiny changes. The ultimate desire is to get this map as well defined and powerful as possible, but I'm reaching a point where I'd like the process to be less manual and more automatic. @U04V70XH6 I'll give it a look.

I'd skip adding a dependency first and see if you can formulate some helpers that just use plain get-in and update-in. Both libraries mentioned are great, but may be heavy-handed depending on what kind of read/update you need.

Seeing the structure is useful, but (and I don't mean to harp on it) if you have a specific input and output you want to be cleaner, it's useful to see a mock-up of “I have {…}, and I want a function that produces {…} from it”

So, I’m not sure how much this applies to you but

I’ve often noticed that people use nesting to ape namespaces.

Taking your example:

{:attribute-name {:datomic [:short :schema :def]
                  :type :boolean
                  :migrations.csv/table :table-name
                  :migrations.csv/column :col-name
                  :fulcro/reports [:names :names]
                  :fulcro/allowed-components [:components :list]
                  :alexa true
                  :aliases #{:short-name :spanish-name :etc}}}

So you can take a lot (but not all) of that nesting and flatten it.

This might get rid of a lot of update-ins and get-ins.

fwiw my response was based on reading that they had a lot of thread first to access keys

@U07S8JGF7 That is actually a thought I had. I've had the same thought that @devn had first, but I'm leaning with keeping the map only 1 level deep and using the naming convention you outlined. The top level keys are already doing the same. like :motor :motor/stages :motor/bit-to-bend all in the first level. I like your style, one refactor and then it's simple from there.

or at least it seemed like that's what they were saying. agree with what you're saying either way

get-in / -> :k :k — tomato tomáto

fair but you could let out a path like [:k :k] and reuse it, which is subtly different imho

I’m not sure it ever occurred to me to do that 😄. But it does make sense.

fwiw I’ve become a big fan of cramming a bunch of stuff into a map over the past few years. Namespaced keys allow you to shove all the context you could ever need into one place.

Qualified keys in maps can often go a long way to flattening structures out -- and they have the benefit of being both more descriptive and providing more context. It's why next.jdbc using them as the default behavior (and why I strongly encourage people to stick with the default -- and get used to qualified keys). :table as a key means very little, :migrations.csv/table provides a lot more context.

We use qualified keys in maps a lot at work to disambiguate things that would otherwise just be :id or :member-id (in what context?!?) etc.

I thought of you (and next.jdbc ) the other day when I found a buddy of mine works at a place that does (camel -> kebab -> snake -> kebab -> camel) for every request.

Such insanity to rename things. So glad there are tools out there that push you toward the sane thing!

I remember a long time ago Zach Tellman showed a massive performance discrepancy between people who felt the need to keywordize string keys so they could call kws as fns

Hahaha... and next.jdbc will provide kebab <-> snake builders if you have the CSK library on your classpath... but only because I got tired of people complaining about having to write those builders themselves 😐 I don't see the point, myself.

on the server side to be more specific

So far, the good advice I've gotten goes towards flattening out the data structure more along many avenues, this is looking promising on the fulcro client-db and integrant fronts too. I'm still learning Clojure, but I like the philosophy. I don't know why I asked. @U07S8JGF7's advice is too simple.

Thanks for the nudge in the right direction. I've been complicating things too much this week.

@devn That's why I encourage folks to try to work with plan in next.jdbc and reduce over results: that lets you completely avoid the whole Clojure map / keyword construction stuff!

me too!

This is how I feel about basically all of Rich’s talks.

“Wait. You can just… not do that?”

I'm not familiar, could you say a bit more?

About Rich’s talks?

sorry that was meant for Sean

I was curious about avoiding the performance penalty

Yeah I was just looking that up as well. https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.2.780/doc/getting-started#plan--reducing-result-sets

Turning ResultSet into vectors of hash maps with qualified keys has an overhead -- in time and memory -- but plan produces a Reducible instead. When you reduce over it, that gets the Connection, runs the query, and then provides a thin hash-map-like wrapper around ResultSet that calls .getObject() directly using a simple label, so you can process a (large) result set without creating any hash maps (or vectors) if you can formulate your processing as a reduction.

ok, one vote for making people opt out of doing it this way

this is clearly better

sorry we’re fully off-topic now, but I did not know about this, and it's definitely an improvement

You can do this in clojure just fine, I suggest you follow the Haskell in the paper more closer, and don't admit mutable state like your atom

Monads and applicatives are perfectly doable in clojure

There's a lot of missing APIs for transients that have not been implemented yet. The one I miss most is subvec.

Clearly, it seems people made due without, I think its non-trivial to implement each one on a transient, and it is rare people need to go that way, hence why it is still not implemented

clojure.core has some examples. E.g. some has "Returns the first logical true value of (pred x) for any x in coll, else nil."

“logical true” sounds good, there was something that didn’t like about “truthy” and i think it’s because truthy is more used in js land than clj land

logical true sounds more clojury, I’ll go with that, thanks @U2FRKM4TW!

The official docstrings only use “logical true” and “logical false” to talk about implicit boolean conversion.

Truthy/falsey is something that emerged in the community, not sure where exactly. It's in Joy of Clojure but I'm not sure if that's the origin or not

Definitely not specific to Clojure or originating from JoC. Would be interested to know the true origin though.

well, the "truthy" word I think came from Colbert's old show :)

not sure when it crossed over to Clojure

Wasn't truthy a holdover from old lisps that js people adopted because js was based on scheme?

Just say what it returns, trust the reader to know those values are truthy. That's my take at least.

What ideas have you explored?

something like (apply list (concat (butlast lst) [(update-fn (last lst))]))

Obvious question first: why/what's your use case?

Imagine a macro where you get a list and you want to augment the data in the last item in the list

what’s special about the last item?

is the list a fixed length? like these are positional parameters?

no it’s not fixed length

last item is interpreted specially by some code down the line

Is it a user facing macro or just for you?

Possibly user facing

Ok this seems to havent got a simple answer

Is this list a queue or other familiar pattern?

that’s very general, I think specifics would help here. it may be that a list is not the right data structure, if you need random access to the last item often.

Given that I expect small size there I can use recursion to update it and construct a new list

Well thanks anyway

macros are probably the only place I'd use butlast lists have linear traversal costs

I had a think about this recently, unless I have to I'd rather not expose a macro API. Is it possible to do it with a function?

(def some-list '(1 2 3 4 5 6 7 8))
=> #'user/some-list
(defn update-last [coll f]
  (loop [result []
         [item & more] coll]
    (if (seq more)
      (recur (conj result item) more)
      (list (conj result (f item))))))
=> #'user/update-last
(update-last some-list inc)
=> ([1 2 3 4 5 6 7 9])

You’ve got a vector in a list. You need to apply list.

The way i think of it, the reason for a list is because the items are meant to be considered in order. Good you use a stack? A stack is designed to give you the last item added to a collection.

Pop the item, update it, put it back on.

I'll give you one better - you can convert it into a single function, using keep-indexed as a replacement for filter + map-indexed.

But if you still need a transducer version and need to produce a lazy seq, just wrap it into sequence.

And if you don't need a lazy seq, wrap it with (into [] ...).

to make it explicit, (into [] (comp (filter #(.isFile %)) (map-indexed vector)) xs), right?

Right, although personally I'd still use keep-indexed as a transducer.

keep-indexed might not do it because the meaning of index changes.

you can use sequence or eduction

to run the transducer

with eduction you can skip the comp

> keep-indexed might not do it because the meaning of index changes. Ah, of course, good catch.

I'll have to take a look, for cljs work you typically just have to use a var to trap state for inspection. I have been curious how it might feel to have a more traditional debugger there.

I think it can be useful for people to understand the compiler (or any clojure codebase) since you can see the values, something that is pretty hard to understand by reading code because of dyn typing

@U4E9AHMGW ^ you could try this.

I've been using this to debug a project I've been working on and it's really nice! I used to use sayid for the same purpose occasionally but it was very tricky to use. Flow storm works so well I'm starting to pop it up more and more often. Nice work @U0739PUFQ!

glad to hear you find it useful already! if you have any questions, or ideas let me know!

A couple ideas so far: • The green color indicating the currently active/selected form is very low contrast vs the black basic text, so it's hard to see which is selected. Higher contrast would be an improvement, for instance a lighter shade of green, etc. • It'd be nice if I could filter traces somehow from within my code. I saw in your demo that you can search within the UI, but it'd be nice if I could do something like the cider debugger and attach metadata that can make that filtering more powerful and easier to control • sometimes you need to right click and "show function calls", but it'd be nice if just clicking or double-clicking could perform default actions a little more smoothly...and I don't risk accidentally un-instrumeting when I meant to view.

Another one: Instead of a dropdown for "Print all args", maybe checkboxes to turn on/off arg0, arg1, ... argn? It'd be nice if it refreshed the currently selected function when changed, as well.

great ideas @U01D37REZHP thanks!, will add them to my notes and then create some github issues

> It'd be nice if I could filter traces somehow from within my code. I saw in your demo that you can search within the UI, but it'd be nice if I could do something like the cider debugger and attach metadata that can make that filtering more powerful and easier to control I'm not sure what you mean by that one, can you elaborate?

It would be nice if you could bookmark things? Each time you say come back here quickly for example.

Hey @U0739PUFQ, somehow I didn't see your question before. Anyway I was looking through your commits at what changed and it seems like you nailed it! Thanks, I'm going to update and try it some more! 🙂

hey np @U01D37REZHP, https://www.youtube.com/watch?v=cnLwRzxrKDk that short video demos the features added on the last release

@U0739PUFQ, that looks really cool. I'm wondering, does the #ctrace work with exceptions? Or do I need to specifically catch those and then add then mark them to be traced? I'm asking as there are some cases where libs generate exceptions, but they're not super clean about it, so you just get things like null pointer exceptions with no stacktrace or anything, so this would be super useful to pin down where problems are occurring.

thanks @U0JUM502E! #ctrace doesn't do anything special about exceptions, is just like #trace but disables tracing so you can selectively activate them with ^{:trace/when ...} . I'm not sure I follow, can you please elaborate a little more on the exception case? Maybe describe the problem and how you see a tool helping with it?

Oh, just that as I mentioned in certain cases when an exception occurs you just get a bare exception. Which makes it very challenging to work out what's causing it. I've built up over time a series of techniques for determining what causes them, but they're still rather painful to source. So perhaps I've not thought about #ctrace in the right way, but with #trace is there a way where if something like an exception does occur, I can easily backtrack to work out where it's coming from through the trace?

oh I think I understand now. Currently it isn't doing anything, but a previous version used to have that feature. I removed it because it was slowing down execution too much. Instrumentation was basically surrounding every expression in a (try ... catch) so it could report the exact expression that was throwing it. Maybe I should add it back optionally some how. I'll add it to my notes. Thanks!

Ok, cool. I mean even just being able to mark certain boundaries as dangerous similar to how you mark functions for instrument might be useful? You already allow functions to be re-run.

That reminds me, not sure if I can reproduce it, but the refresh symbol is supposed to be re-run right?

If so, I had some cases where doing that would instead clear all of the traces collected.

yes that should re-run the original expression. Maybe you can create a issue with that specific case? sounds like a bug

I'll have to see if I can reproduce it 😃

It was a repeated problem, but not sure how to create a minimal case, it was happening in one of my larger projects.

oh, ok I'll also try to reproduce, I haven't use that functionality a lot, only when I instrument entire codebases from flow-storm.api/cli-run

Yes, I've not been doing that, primarily calling api functions from the repl

Hmm, ok, so I'm trying out the new version. Some odd behaviour (this is inside a clojure project, the project itself is public, so happy to help you repro this stuff, but not sure if I have time to make specific repro cases) • calling fs-api/instrument-forms-for-namespaces it doesn't seem to try and activate the forms in a namespace, skipping some. (It does activate some, so something is working) • calling fs-api/instrument-var, it can respond with Couldn't find source for symbol, with the symbol I'm giving being fully namespace qualified. • #trace on a top level form in the source appears to work fine. • In the Browser, selecting a form to instrument doesn't necessarily instrument it.

@U0JUM502E hmm could be very helpful to see examples of that, there are obviously bugs there. instrument-var uses clj.repl/source-fn inside to find the source of the function. This only works with functions that where loaded from a file, so repl defined functions will not work with istrument-var`

there are a bunch of reasons why istrument-forms-for-namespaces` can skip functions, I'll improve the errors and warnings there so it is easier for users to see why it can't instrument some specific form. One example happens if the form is too big, because clojure has a limit on how big a form can be to evaluate it and instrumenting adds a lot of expressions to the form, so sometimes it ends up bigger than what clojure eval accepts.

@U0JUM502E one thing you can use for quickly finding what form fired a exceptions is moving to the last trace (I'll add a button so that can be done with one click)

Here's another idea. The instrument-forms-for-namespaces function is handy while debugging, but having a whole bunch of namespaces instrumented quickly becomes overkill. I think it'd be useful to have a corresponding un-instrument function available.

@U01D37REZHP yeah, that would be useful, I'll be adding an option to instrument entire namespaces from the browser, where you can also temporaly disable instrumentation. But maybe will add both. Thanks for the idea

@U0739PUFQ btw, using this a bit more, there are definitely some issues / fiddlyness in some areas. I could put together a gist or something if that would be helpful? One definitely missing thing is these are static, if there was some way to inspect them, as at times there's fundamental info hidden within this. Also maybe on a tooltip to def or somewhere, state that it defaults to defining the var in the user ns. Otherwise this is really something for debugging fairly fiddly stuff like datascript!

@U0JUM502E Yeah, I haven't made locals inspectable yet, I should add a context menu so we can define them on the repl also. For now one way to inspect a local binding value is to click on the code, over the form that is generating their value. I'm also aware of some issues with locals showing there even if they should be showing yet, it is on my todo list to fix it. Good idea to remember the user that the var is going to be defined under user ns.

I'm trying to improve the ergonomics to the point it is a replacement for print statements in almost every situation, not just for complex codebases debugging.

Not sure if you want to support this, but being able to say, what is the path difference between these two runs, would be interesting.

@U0JUM502E hmm that is a interesting idea to think about, how do you think we can visualize it? and what do you see as a use case for it?

I just created #flow-storm channel if you are interested in discussing this ideas, or anything debugging related

So my goal is to create/revise regexes on symbols and keywords (or at least their string forms), and it would simplify things if you don't have to explicitly filter out chars that aren't allowed anyways

This is an area that is intentionally underspecified to allow for future growth

i.e. you don't box yourself in saying "these chars are allowed which means these other chars are not allowed"?

yes, some things are not not allowed :)

Seems like the (disappointing) upshot is that I should not simplify my regexes, at least if I want to keep them implementation-agnostic

(Especially since there's like, what, two characters that are explicitly disallowed?)

(ns nlp-ws.nlp-results.concepts (:require [clojure.tools.logging :as log] [buddy.core.codecs :as codecs] [buddy.core.kdf :as kdf] [buddy.core.nonce :as nonce] [cheshire.core :refer :all] [cheshire.generate :refer [add-encoder encode-str remove-encoder]] [scout-nlp-ws.db.nlp_data :as nlp]) ) (defprotocol ConceptHelpers (describe [x])) (defrecord Concept[concept_id concept_name domain_id vocabulary_id concept_class_id standard_concept concept_code valid_start_date valid_end_date invalid_reason] ConceptHelpers (describe[this] (:concept_name this)))

Here is the record defined called Concept

And I am trying to import this into the following namespace

(ns nlp-ws.model.surgical-complications (:require [scout-nlp-ws.nlp-results.concepts :as concepts]) (:import [scout-nlp-ws.nlp-results.concepts Concept]))

And this is the error

the name of the class will have _ instead of -

in general importing classes created with deftype or defrecord is something I would recommend against doing (there are places where it is unavoidable though)

if you are just creating new instances of the type, use the factory functions ->Concept in your case

okay

thank you!

> that worked!