There is probably some reason you're not doing this, but may as well ask for the sake of better understanding your needs: if it needs to be a Java class and have a lot of Java stuff going on, why not write it in Java?

I'm generating it from Clojure data

https://clojuredocs.org/clojure.core/gen-class

basically I'm setting up bindings so I can call my code from other languages via GraalVMs polyglot features, and GraalVM requires annotations

@UCHFND0T0 yeah I've actually set this up using gen-class now, and it works for methods, but AFAICT gen-class won't let me create fields

maybe look into the ASM ecosystem?

you can also dynamically make gen-class calls and have them work in AOT

Hmm yeah, that seems like it might be a bit overkill for this use-case though? I don't really need control at the bytecode level.

Haven't used it, but https://github.com/jgpc42/insn looks like it might do what you need. Support annotations on fields: https://github.com/jgpc42/insn/wiki/Annotations

Suppose we work at a company with in-house customer relationship management software written in a full Clojure stack. When a customer speaks with a CSR, one of the first things that happens is the the software prompts the CSR to ask the customer for any pieces of information we're missing – mailing address, phone number, etc. One day, a customer calls in and the software tells the CSR we don't know their last name. The CSR asks, and the customer replies, "Oh, I don't have a last name. My name is just Cher, first name only." The CSR leaves that field blank and submits the form. The system sees the blank field and says, "I guess we still don't know it," and simply omits the ::last-name key in the map that gets passed down the stack to our database. A week later, Cher calls in again with another problem. Once again, the CSR is prompted to ask her for her last name. Now Cher is annoyed: "I told you last week, I don't have a last name. Didn't you put that into your system?" This CSR is savvy and recognizes that this is a shortcoming in the software. You receive a feature request: we need to distinguish between "I don't know this person's last name" and "I know for certain that this person does not have a last name". How do you represent this distinction in your maps? How do you spec it? Is this not well and truly a "Maybe String"?

Isn't that the same distinction as a key not being there and the value being nil?

That's what I'do do anyway: set the last name to nil if it is explicitly not there. But yea a nullable thing is kind of a poor mans maybe

I would think so – and I've specced it that way – but in the talk I linked, Rich seems to be opposed to that. At the timestamp I linked it at, the slide has this line of code:

(spec/def ::model (spec/nilable string?))

IIRC spec has a special thing for nullable fields by the way

right

if this system defines a domain where customer model might not have ::last-name then unlikely you will get validation exceptions in case the key is missing. The model is simply defines a set of facts about every customer: like {::first-name "Foo"} or {::last-name "Bar"}. Both valid examples from my point of view unless those keys are not used as primary indices

@U04V4KLKC I'm not sure you quite understood the scenario. There is no validation exception. The problem is that we need to distinguish between accidental absence and essential absence, and I'm struggling to figure out how to fit that into the philosophy of specs that Rich is articulating.

Isn't the “we don’t have a last name, please ask the customer”-part the problem? There seems to be an implicit understanding that you should have a last name

i got "Once again, the CSR is prompted to ask her for her last name." as "validation" exception.

@U0190EXLXU1 Yes, exactly. There's a problem in our data model. How do we resolve it while staying within the bounds of the philosophy that Rich endorses?

Just.. don’t prompt the CSR to ask for last names in the first place?

Perhaps I’m fixating on the (simplified) example 😅

You suggest this and the product owner shoots you down: "No, we often have missing information, and we need to try to fill it in when possible. Can't you just make it so that the software understands the difference between 'I don't know this information' and 'this information doesn't exist'?"

Well you know... maybe really smart people who design systems from a hammock might miss a few real world use cases, so I say if in your system the distinction between types of missing information is important, encode that information. That it's an antipattern in most cases doesn't mean nillable in illegal, it exists to be used.

It would be easy enough for me to simply reject what Rich is trying to explain and embrace s/nilable or Maybe or whatever the equivalent is in any given language, but then I'd miss out on understanding this unfamiliar perspective.

IMO the antipattern is setting a field to nil when it could just as well be absent. But nillable is right there for you if nil is a useful value in your domain.

"nothing is a maybe string" – this is something he said after a long time working in real-world systems where surely he's encountered problems shaped like this

or simply use empty string for last name 😉

I would still not use nil for that though, but some random ::tickle-me-no-last-name . I don’t think there’s anything wrong with following advice only when it works for you :-)

@U04V4KLKC Ha, suppose we could do that. That feels Go-y in a bad way, imo. I guess it could work, but that makes it feel like her name is "Cher [moment of silence]", you know?

Hehe in Julia nil is called nothing making his statement ambiguous. "a nothing values is a maybe string" or "there exist no maybe strings"

@U0190EXLXU1 For sure, and until I change my mind I'll continue rejecting this advice and doing things as I've always done. But I know that there's something here I fundamentally don't understand. I'm watching the talk to learn a new way to think about software, not to evaluate it against my established ways of thinking.

@U029J729MUP true, that might feel as a wrong idea but at the same time you are not mixing different data types in the same field. For me string and nil nether goes together because they can't represent each other (to compare with empty sequence and nil as an example).

@U029J729MUP how do you distinguish between not-there and not-existing values right now? For map structures, this seems to make them kind-of-useless if using nil

> For me string and nil nether goes together because they can't represent each other @U04V4KLKC Isn't that the merit of using an explicit nil? It does not represent any string, and instead represents exactly the case of there being no string whatsoever.

@U0190EXLXU1 In cases where I need to model a distinction between "I don't have it" and "it doesn't exist", usually I represent the former as an omitted field and the latter as a field with a value of nil

and then you will be forced to respect nileness of the data in case of sending the data to DB or printing.

Could use a special sentinel value like a keyword, but that feels philosophically the same – it's a nil by another name, and still violates the stricture of "nothing is a maybe string"

Maybe Rich would not see it that way, but then I'd struggle to understand where he sees the distinction

my personal take away from the talk is "data should not lie". A field ::last-name should be associated with something that application recognise as a last name and nothing else, nil including.

So then how would you solve the problem in the scenario?

(s/def ::first-name string?)
(s/def ::last-name string?)
(s/def ::customer (s/keys :req [::first-name ::last-name]))

Okay, but we have to figure out a concrete map to pass to the backend where we're somehow informing it that the ::last-name is not simply missing, but is absent by design. We don't want Cher to be asked for it every time she calls in just because our system doesn't understand the difference between "the customer never told us this" and "the customer told us that it's nil"

sorry, I corrected the spec. it should have :req instead of :opt

both fields required and must be a string. Absence of any of them - is empty string.

Hrm. That does strictly speaking satisfy the principles Rich is articulating, but it feels like it violates other ones. If that's really The Hickeyan Way to do it, then maybe I simply disagree with it. But I would be really surprised by that. Surely an empty string is a specific concrete value, not a placeholder representing the absence of a value?

> "the customer never told us this" and "the customer told us that it's nil" I don't think system should know the difference.

The system must know the difference. The customers are complaining because it doesn't.

not trying to claim "this is Rich Hickey way" 🙂 that is exclusively my understanding

Oh sure. Rich isn't here, so other folks' interpretations are all I have. Yours seems compatible with what he's saying, so it's the best candidate I have so far for how to understand the talk.

> The system must know the difference. The customers are complaining because it doesn't. (edited) may I ask how they could discover that system doesn't know the difference?

haven't read the whole scrollback, but if Cher doesn't have a last name, and the login system requires a first & last name, then Cher can't login

making a Maybe is like making an empty cell in the database. the info is either there or not

the solution isn't 'better' types, it's making the login system robust to Cher, Madonna, Brazilian futbol players

> haven't read the whole scrollback, but if Cher doesn't have a last name, and the login system requires a first & last name, then Cher can't login depends how the system is implementeed. I don't see any problem with sending the form like that {"first-name":"Cher","last-name":""}

> may I ask how they could discover that system doesn't know the difference? The scenario revealed that problem. At the outset, we have sparse data – very common in customer relationship stuff. Cher told the CSR that she doesn't have a last name. The field was left empty, and that was submitted to the backend. That was her answer! It should be final! But because the system is modeling sparse data – "I don't know her last name" – and explicitly non-existent data – "I have no last name" – in the same way, the system will ask over and over again what her last name is.

> the solution isn't 'better' types, it's making the login system robust to Cher, Madonna, Brazilian futbol players Okay, but how do we make it robust? The context is a talk about spec, but really spec is just an effort to formally describe whatever data model we might have had anyway.

ew to last-name=""

make it more robust = by only requiring first name

her last name is not Null, nor "", nor None

Okay, so we relax the backend's requirements. It will accept {"first-name":"Cher"} as a valid payload and update only that field. The next time Cher calls in, the system still doesn't know her last name, so it once again prompts the CSR to ask her. It is still failing to distinguish between "it doesn't exist" and "I don't know what it is". The original problem has still not been solved.

sounds like you didn't update the system cohesively

Can’t you turn it around? Use a sentinel value for ‘still-need-to-ask’ of some-sorts

@U0190EXLXU1 I think that would work fine, but afaict it's just another way of saying "a last name is a Maybe String", and I think it's still doing exactly what Rich is saying not to do

You can key it into a :please-still-ask-for

what we really need is another nil-like value! how about undefined?

the key piece here is whether the system knows the entity is complete or not

Or store a :already-asked-for somewhere: but it quickly gets really complicated if you don’t know all the valid transitions for the data (now, and in the future with new requirements/fields). I would probably never overload nil for this though, because it makes your map useless in e.g. destructuring

sentinel values the user can't possibly enter is the only real solution here, imo

I think using a sentinel for "idk, we need to ask" and omission for "it doesn't exist" is fraught. It would imply that a map of {} is an assertion to the effect that no information about anything exists, rather than the usual idea that a map might simply have sparse data

This might be a dumb idea, but how about you keep track separately which fields needs input from the user? Not just checking if a particular field is nil/missing/””? If the user tells you she/he has no last name you clear the appropriate flag etc

@U0178V2SLAY So our map is something like {::first-name "Cher", ::first-name-finalized true, ::last-name-finalized true}?

Something like that. Or keep the names of the missing fields in a set or whatever.

It has the advantage of not complecting “no value” vs “value missing”, and can be interpreted on its own

just accept the absence of information without having to put a box around it

it's liberating 😂

@U050ECB92 Okay, but we still need to distinguish between two kinds of absence!

then mark the record as complete

ostensibly you wouldn't make it through a registration flow without a complete record

That's dangerous. What if a new field is added later? What if the front end is out of date?

for whatever your system deems "complete"

so yay to flags and ew to empty string ? ))

haha -- or no flag at all

To say "this record is complete" is not to say "this field here is absent by design", it is to say "all fields we don't have are absent by design" – which is dangerous when you might not know about every possible field. It makes the addition of a new field a breaking change that requires lockstep upgrades of all applications that understand this entity.

yeah, it's not a good design, imo

I don't really love the idea of having separate flags, but it does at least keep completion of each field isolated from other fields

Making this all ‘self-contained’ in a single key (or lack thereof) means that any choice you make, you are stuck with it going forward; in that sense having it in a separate key makes future choice (“Oh let’s not bother random people for random personal info ever, unless it is a hard requirement for legal purposes”) also possible through a simple mechanical data migration

That's a good point. I guess we can make a case that bundling all of these absences into the semantics of a single key complects ontology with epistemology.

Instead apply vexillology ;)

Although I'm not sure whether the standard Clojurian idea of a sparse map that only contains the things we know really escapes that complecting

You added a requirement about having to ask things: somehow noting that you have asked something (or should not ask something) seems like a proper thing to store in the data

I said it's a good point but I think I'm talking myself out of believing it now. Presence is epistemology – we have a fact, we know a thing – while value is ontology – here is what that fact is. If the fact is "there is no thing", shouldn't we represent that directly?

But that is not what you are doing: “I will not give you my last name, please don’t ask again” is also an answer you can expect

Right, and {::last-name nil} asserts a fact (because the key is present), and the fact being asserted is that no last name exists. It's a statement about non-being, which feels a bit vacuous, but it's still a meaningful thing to say. Meanwhile {} asserts nothing at all.

Either way, I think the “should-not-ask-this-question” can not be directly derived from the data, unless you store it specifically.

> “I will not give you my last name, please don’t ask again” is also an answer you can expect This feels like it's shifting the problem from what exists to what we know. When Cher tells us she has no last name, that is a full and complete answer to the question. The question does not remain open, and the field does not remain an unknown. We have our answer: there is no last name.

Whichever choice you make today might not be the right one tomorrow. Just as you might add (or remove) fields, so should your “can/should/may I ask for this information?”-requirements be able to change over time. Gender seems like a fine candidate of a piece of data that will provide work for the foreseeable future, in adjusting processes and rules where deemed necessary

I think if we try to model this in a way that also captures refusal to answer, then we're conflating two different scenarios: a question whose answer is permanently unknown versus a question whose answer is known and happens to entail absence.

Data is just data 😄

Okay, I guess we can make the case that we don't need to actually care about that distinction

> Data is just data Feels like we bottomed out

And rather than modeling whether we know it, we can just set that aside and do exactly what you're suggesting: model whether to ask it. Maybe we ask because we suspect it's out of date (a business rule might periodically invalidate mailing addresses), maybe we ask because it's a new field, maybe we ask because we never got an answer, and maybe we don't ask, because we've been told that the empty value (however we model that) is correct.

I think this satisfies the scenario I gave just fine, and I appreciate your perspective. Something still doesn't feel quite right – I feel like our systems will, at least sometimes, need to model, "I know this fact, and the fact I know is that X doesn't exist" – but I'll have to do more thinking on whether I can actually identify a case for that.

i’ve never met rich. as far as i know he eats his own boogers and chews on coffee cup lids

i think what is valid is to say that if you access the info :last-name and get this

dont know | known to not exist | exists
  nil     |       nil          | "abc"

then you have overlapping domain things

Yes, for sure. Whatever we do, we can't simply conflate those two things.

if clojure maps were implemented in a way where an accidental nil were not something you would run into then maybe it would be a different question

like if a Map gave an Option<T> and you stored Option<T>, you can get an Option<Option<T>>

no overlapping domain

Well, Clojure maps do let you distinguish between a key which is absent and a key which is set to nil

we just usually don't

it technically can but its in the etherspace of data representation

i think what you want is a known sentinel

dont know | known to not exist | exists
  nil     |      :none         | "abc"

no overlapping domain, no contains? check, fits better with clojure’s apis

and then you can have nil values always mean “dont know”

Yes, Jelle was suggesting that earlier. I think that amounts to just an ersatz Option, and is really just the same thing as the {}/`{::last-name nil}` distinction, but just happens to have better ergonomics in Clojure. In principle it's not different afaict.

yes.

an Option is Some(T) + None which is == to T + A where A is not in the domain of T

again, this is a stranger’s philosophy that is embodied most strongly in a library that isn’t finished

ignore it

Yes, true. It could be that if he had full and explicable answers to these questions, he would have implemented it by now!

correct me if I'm wrong, but when using a sentinel(s) to indicate something (in the absence of the actual data), the consumers must now separately deal with it (instead of just using the :last-name field as a string or whatever)

correct

@U0178V2SLAY Yes. But this is just a way of encoding a distinction which the problem domain has already said is important – however we choose to represent it, we're going to have to deal with it either way.

you can encode it so that the knowledge of the distinction is higher up by enforcing an “object contract”

(defn last-name [person if-not-known]
  (get person :last-name if-not-known))

Reasoning from first principles, there is no reason to accept someone else's opinion about how to do something just because of their status. FWIW other people with lots of experience building real world systems explicitly support Option/Maybe types in their systems: https://github.com/metosin/malli#maybe-schemas

but again now we are in a fuggin more than 100 item thread

this is a waste of human potential

@U051V5LLP That's not what I'm doing though. I'm not accepting his opinion, I'm working to understand what it is. Then I will decide what I think of it.

Well, he's not on here much, so good luck getting a reply.

I'm also not trying to ask him. I'm asking others for their interpretations. The same way you might discuss a book and ask, "What do you suppose the author meant by this? What are its implications?"

Well it seems like the discussion listed the alternatives - you somehow have to track the flavor of the unknown data.

wait

i think we just invented javascript

undefined and null

we did it

I'm sorry to say someone beat you to that joke

oh goshamn it

it was me

I made the bad joke

Just gonna throw my 2c in because who doesn’t like throwing oil on a fire I think that to understand Rich’s argument you have to understand what he’s arguing against. Most languages/databases force you to represent the absence of a value, and most of the time, people use null to represent that absence. If we approach this from first principles in Haskell for example, I’d argue that the type of each of these fields is:

Field a = Present a | Unknown | IntentionallyOmitted

can people please have long arguments about my opinions? I have a lot of them

I have a different solution to the problem: a :full-name / :canonical-namefield. In the case of individuals with first, middle, and last names, it would be a concatenation of all three. In Cher's case, it would be the mononym. At the cost of some redundancy and needing to do some lookups, the system can distinguish between "known to not have a last name" and "missing last name" by comparing :full-name with the other name fields. in fact, you could replace top-level fields like :first-name and :last-name with a :name field that's a disjoint type (I'm using malli syntax for convenience here). this has the benefit of being extensible to https://en.wikipedia.org/wiki/Spanish_naming_customs that prevail in other languages:

[:or
 [:mononym :string]
 [:first-last [:map [:first :string] [:last :string]]
 [:conventional-spanish [:map [:given :string] [:patronym :string] [:matronym :string]]

the problem isn't really the name issue, though, it's how do you represent IntentionallyOmitted vs Unknown for fields. the name issue was an example just to explain that in some cases this matters

I can't make a conclusive case for this, but as some others in the thread may have suggested before, I have a very strong suspicion that other cases where we think we have "intentional omissions" are where there are unstated assumptions in the data model. so I think other cases where we need to distinguish between "intentionally left out" and "missing" end up being like my name example upon further inspection.

there may be other reasons that there is no field and it matters that we indicate it

I mean, another way to approach this is to decomplect 😉 the features. :last-name is present if present, absent if not, and then you have a separate :last-name-confirmed. When you’re looking at the data for its own sake, :last-name is just data. When you want to power the “should we ask” feature, you have the data for that too, and an absent :last-name plus a true :last-name-confirmed means they’re Cher

I think there is an option we aren’t considering here

(def first-name 0)
(def last-name 1)

(defn create-person [first-name last-name]
  [first-name last-name])

(get (create-person "Abc" "Def") last-name)

> We could use nil, but that risks confusing other engineers who are used to nil meaning “absence”. In this case, I’d probably use a sentinel keyword. But that’s not an ersatz Maybe, it’s a representation of one of the three possible states for that field in the system. Eh, I still think it's an ersatz Maybe. Perhaps more precisely an ersatz Maybe Maybe. We've given it more precise semantics than "it might not be there!" by directly encoding why it isn't there, but this still runs up against one of Rich's central theses: a name is just a string, not a maybe string. I don't think giving a more precise name to that optionality addresses his essential point in saying this.

Seems like you’d prefer the :last-name-confirmed approach?

I mean, my preference is the sentinel value, but I'm trying to figure out what kind of approach falls out of Rich Hickey's domain modeling philosophy

Tracking "have we figured out this value?" as a separate datum has come up before in this thread and seems to fit, but feels weird to me in ways I don't quite yet know how to articulate

Like, "I know that this value doesn't exist" is just as much a knowable fact as "I know that this value is X", and it seems wrong to me that we would express these in different places. But maybe this really is the implication of Rich's philosophy and I just disagree with it. I've been hoping for an aha moment where I see something that looks tidy (even if I ultimately prefer something else).

Don't you think you might want this way of expressing yourself, because it is the easiest thing in other languages? Not saying the approach is wrong in all cases, but perhaps you need to reconsider how you decompose problem statements. To me it's not a given that I often need to differentiate 'don't know’ vs ‘does not exist’; an issue with most of the proposed alternatives is that they are not generic, but perhaps we don't want generalised solutions for problems that fundamentally only occur in highly specific situations

For what it's worth, if you generalize the problem from Boolean confidence to percentage confidence, it starts to look more like the 2-key approach: {

:sample-value 12
 :sample-confidence 0.89}

{:last-name {:value nil, :state :confirmed}}

not a fan of that, but that's what we're talking about, essentially. we need states of fields represented somehow

it sounds a bit like you need to model the individual attributes (of some hypothetical entity) as more than just a single key-value pair. because we could then start asking all sorts of questions about the attribute (do we know the value? why don't we know it? when did we last try to retrieve the value? how did we ..) edit: cora was faster 🙂

In a more general sense, this is an instance of the metadata problem: programming languages and databases aren't great at representing information about information. The RDF-inspired model that's underneath Datomic and a lot of clojure’s design aesthetic famously is terrible at this. On the other hand, it’s kind of a niche need.

@U0190EXLXU1 I've got several competing thoughts on that. From my own perspective, it's a universal fact that Cher has no last name – this isn't context specific, this is simply the shape of the domain and is a fact we should be able to assert for anyone interested in knowing it. From the perspective of trying to interpret Rich's philosophy, the goal here is context-free domain modeling, so while I'm sympathetic to sticking to narrow solutions when generalizations don't make themselves apparent, I don't think it helps me make sense of what is being endorsed in that talk.

re: everyone else Yeah, "whether and the way in which we have this value" is starting to feel like more than just a boolean proposition...

> The RDF-inspired model that's underneath Datomic and a lot of clojure’s design aesthetic famously is terrible at this I am unfamiliar with this fame. Have any resources you can point me to on breakdowns of where it breaks down?

also interested

Because it's not as if it's an esoteric problem – most PATCH services will run into it

just serialize all values to include their clojure metadata and stick any relevant info in there! problem... solved?

(contains? my-map ::last-name)
;; => :its-complicated

Uh, we may want to start a new thread for that 😆 and I may have been overstating its famousness. It’s famous among people who have tried to model metadata with RDF, which I can’t imagine is a particularly large club. I’ve run into it because of a knowledge modeling side project, and the lack of that capability drove me to explore more general approaches, like those offered by knowledge engines/knowledge bases like https://vaticle.com/. For a concrete example of this problem, consider a knowledge base like Wikipedia that contains facts like United States founded-in 1776. You’d want to back that fact up with a citation, but how would you model the citation? You really want to use the triple itself as the subject of another triple, but RDF doesn’t support that. Instead they offer reification, which no one likes:

r isa reified-node
r subject United states
r verb founded-in
r object 1776
r citation ...

in datomic i think you would use the transaction id as the entity of another triple

so “these facts” came in via transaction “t” and then you can attach the citation to “t”

Not quite, Datomic’s transaction ID refers to 1…many assertions. When you use it as a subject you’re asserting about the transaction, not about a triple. It’s the right place for metadata like “who added this” and “why’d they do it”, not for triple-specific metadata

an orthogonal but also useful concept

I agree that what we are trying to model here is the domain of a sum-type [Value | Unknown | NoValue]. By definition, you would need a way to represent 3 different states. Datomic is not a good example, because attribute values need a specific type (e.g. string), cannot be nil, and cannot be a sum-type (e.g. string OR sentinel-keyword). But, I would argue you are not breaking some unforgivable rule of idiomatic Clojure, if you decide you need more nuance in your domain and be more strict about what it means for map values to be nil or sentinels. (And you can use runtime validators like spec and malli to keep everyone in check). A practical use-case of this in the wild is the Pathom library that deals exactly with this problem: a resolver must differentiate to the caller between "I have no value, go ask someone else" and "there is no value, don't bother asking anyone else". In the internals it uses a sentinel keyword to represent this third-state, but the public API actually expects the developer to be careful in what they return: a map without a key is "Unknown - keep searching", and a map with the key and a value of nil is "NoValue - end of story".

To sum up, there’s a difference between “data modeling” and “knowledge modeling”. Data modeling is (usually) about storing data to serve a specific need, whereas knowledge modeling (usually) is about modeling everything that is known to allow you to answer arbitrary questions in the future unknown to you at present. Knowledge modeling is a lot harder and the tooling support is worse. Once in a while, data modeling problems acquire a knowledge modeling flavor, and when that happens no matter what it’s gonna be a little messy (the above is all my opinions, I haven’t heard this stance articulated by anyone else in the industry)

Thanks Max, that's all really useful perspective. I guess maybe the ultimate answer to my question is: Rich's model doesn't have an answer to this, but that's just because it's a largely unsolved problem in modeling.

And familiar solutions – like a JSON PATCH endpoint distinguishing between an omitted value and a null one – might work fine in their specific contexts but tend not to be generalizable.

Although I think I'd dispute that data modeling and knowledge modeling are separate – but I might admit that we can often pretend they are. I think at heart this might be a manifestation in software engineering of a general problem in philosophy: in some ways the object of our knowledge seems to only ever be an idea and never a thing in itself, so whenever we try to write down what we know we find it impossible to completely disentangle from the subjective details of the manner in which we know it.

Right, I think this is a good example of the difference between inherent complexity and accidental complexity. Modeling knowledge in a general fashion is inherently complex, whereas modeling enough data to drive a particular application is usually not (even though we often make it accidentally complex). YAGNI applies, until of course, it doesn’t.

Another thing you might be interested in looking into is open world/closed world assumption in knowledge modeling. In a closed world, the fact is true if it exists and false if it doesn’t. In an open world, the fact could be asserted true, asserted false, or not in the system at all meaning it’s unknown. Turns out it’s really hard to build applications on top of an open world assumption

Yeah, that makes sense as a generalization of what we've looked at here

An example: OWL, an ontology language for RDF, says it uses an open world assumption, but as far as I can tell, most people use it as if it uses a closed world assumption

The purist in me is frustrated by this because I know that doing something that you know is "wrong" in the domain model means shipping a fast MVP at the cost of a high likelihood of expensive changes later. But I guess if philosophy hasn't figured out the relationship between knowledge and truth after ~2500 years, I can't expect computer science to have done so after less than a century.

I came up with (defmacro foo [x] (let [x (if (symbol? x) @(resolve x) x)] ...)), but I'm curious to hear if there's a better way

https://www.braveclojure.com/writing-macros/ Reread this chapter really careful.

(->> positions
     (map #(sum (abs (- % pos))))
     (reduce +))

Math/abs is also v slow so I wrote my own, but that doesn't seem to be the slowest by far. just the (- % pos)

I'm guessing this is because of doing a bunch of typechecking/casting?

there's no unchecked-minus

ah unchecked-subtract

it's not checked/unchecked, it's boxing

boxed math (which you're doing) is about 100x primitive math

I see

any tips on how to unbox things?

all sequence functions will box (they only take Objects) so this almost always ends up as a loop/recur

what's positions? coll of ??

coll of ints

(defn sum
  [n]
  (reduce + (range 1 (inc n))))


(defn abs
  [n]
  (if (neg? n) (- 0 n) n))


(defn part2
  [input]
  (let [positions (->> (string/split input #",")
                       (map #(Integer/parseInt %))
                       (sort))
        max-pos (last positions)]
    (loop [pos 0
           least-fuel (reduce + (map sum positions))]
      (if (= max-pos pos)
        least-fuel
        (let [fuel (->> positions
                        (map #(sum (abs (- % pos))))
                        (reduce +))]
          (recur
           (inc pos)
           (if (< fuel least-fuel)
             fuel
             least-fuel)))))))

would be best to parse into an int[] (or long[]), and then refactor anything that's seq ops into either areduce etc or loop/recur

i'll try that

ty!

and then in the loop/recur, you want to ensure that things are unambiguously prims in the loop decl and in the recur, but really you want primitive longs (Clojure doesn't really fully support primitive ints)

(set! *unchecked-math* :warn-on-boxed) can also help find things

sum and abs are also not very good for perf. Polymorphic fast abs is coming in 1.11 too ;)

i def started from "good enough to solve the problem" and am now working my to something speedy 😛 profiling led me to the boxed math first

naively trying Math/abs to my own abs impl cut the run time in half

Math/abs is a hotspot intrinsic so should definitely be part of any use - that's going straight to hand optimized assembly code

but should also ensure it's getting a primitive input, not doing unboxing

FWIW @U4YGF4NGM, in this particular case (advent of code 7b) I found that just adding type hints improved performance by an order of magnitude (10s to 1s) for me, e.g.:

(defn distance [^long a ^long b]
  (Math/abs (- a b)))

didn't change anything for me AFAICT but I think it's because of what alex alluded to, the seq ops automatically box values going into my distance fn

is that right?

yes

the above at least is getting unboxed (- a b) :)

but really, you want to arrange for the whole thing to be unboxed primitives

a very mechanical replacement of the above seq ops with arrays/areduce:

(defn sum-fast ^long
  [^long n]
  (/ (* n (+ 1 n)) 2))


(defn part2-fast
  [input]
  (let [^"[J" positions (->> (string/split input #",")
                             ;; according to alexmiller long support is better
                             (map #(Long/parseLong %))
                             (into-array Long/TYPE))]
    (loop [^long pos (reduce max positions)
           ^long least-fuel (areduce positions
                                     i ret (long 0)
                                     (+ ret ^long (sum-fast (aget positions i))))]
      (if (zero? pos)
        least-fuel
        (recur
         (dec pos)
         (let [^long fuel (areduce
                           positions i ret (long 0)
                           (+ ret
                              ^long (sum-fast (Math/abs (- (aget positions i)
                                                      pos)))))]
           (min fuel least-fuel)))))))

strangely, even after converting all of the seq ops in part2 to areduce, the run time stayed roughly the same until I also factored all seq ops out of the sum

the first impl was running about ~6-7s on my laptop. part2-fast runs in about 35ms

It's usually about 2 orders of magnitude different

The next level of this is to start checking the bytecode

Speaking of boxed math, it doesn't warn on Util.equiv when it probably could, I can produce a simple repro

are there any libraries that help with this kind of operations?

looks like a subset of what numpy does for python

(defn string-char-predicate
  [p]
  (fn charset-pred ^Boolean [^String s]
    (let [n (.length s)]
      (loop [i 0]
        (if (= i n)
          true
          (if (p (.charAt s (unchecked-int i)))
            (recur (unchecked-inc i))
            false))))))

(defn string-char-predicate*
  [p]
  (fn charset-pred ^Boolean [^String s]
    (when-let [n (some-> s .length)]
      (loop [i 0]
        (if (= i n)
          true
          (if (p (.charAt s (unchecked-int i)))
            (recur (unchecked-inc i))
            false))))))

Please use the #adventofcode channel for AoC discussions.

@U04V70XH6 while the impetus of the question was AoC, i figured because i was trying to figure out the root cause of my performance problems w/ some pretty generic Clojure code that #clojure made more sense

i'll avoid posting full solutions in the future

Yeah, I figured this was borderline -- after we'd posted in #admin-announcements about it -- but wanted a reminder for everyone in the thread.

the boxed math warnings will not catch everything - it's not actually possible unless doing those with full runtime info I don't think

if you look up the original ticket there are more details in there (and at least one open spin off for ret types in particular)

I don't have the time to figure out if the case above is detectable by what the boxed math warnings cover

This is a great use case for spec

Do you just assert that the data matches the spec and use that assertion as documentation?

You could, you could also turn this assertion on at run time

i.e. (s/fspec foo :args ,,, :ret (s/keys ,,,)) You could write it right below your function, it's both documentation and a validation which can be turned on at run time or dev time

or you can use malli - same deal

it only asserts when you instrument your functions which most people only do at dev time

but yes, the spec serves as documentation

The doc macro at the repl will also include spec information

Datomic dev-local

I've been getting into https://github.com/wotbrew/relic

(xtdb/start-node {}) https://xtdb.com/docs/

Obviously #relic is going to be awesome when done 😉 , but you could use h2 + honeysql https://www.h2database.com/html/main.html

datascript

@U064UGEUQ: Quoting the relic README: > Did you try https://github.com/noprompt/meander, https://github.com/clojure/core.logic, https://github.com/tonsky/datascript and every graph-map-database under the sun but still do not feel http://curtclifton.net/papers/MoseleyMarks06a.pdf? Yep, describes how I feel.