Yeah, I’m not a fan of Amazon in general. There are so many horror stories 😞 I can understand why people use their services, purely from a business and engineering perspective though.

Sure. But in the case of Elastic, it was extremely exploitative. At least IMHO.

Amazon has a bullying track record. Imagine a world where large businesses like Amazon would lead by example, in ethical terms, sustainability and so on. It’s not all black and white, but I feel like it has to get better.

Personally I found 's narrative on the 'mythology of open source' (present in various pieces) clarifying for a case like this one. The popular mythology is that OSS is all rainbows and kittens, reality is that you're giving your work for free, with some benefits and tradeoffs. Delivering OSS, by most licenses doesn't imply that its consumers have to contribute back in any way. Which is why e.g. Datomic being closed source makes perfect sense :)

success in open source is the ecosystem around a thing, not necessarily the thing itself, imho

that is why lucene is the success story here and elastic is just one part of its awesomeness

The sentence “Referential transparency means you can replace the function with its value and get the same output.” should be something like “Referential transparency means you can replace the function with its evaluation result and get the same program behaviour.” My understanding is that a referentially transparent expression/function could have local state that doesn’t leak out, and it wouldn’t be considered strictly pure in this case. The term “referentially transparent” is insofar useful that you can reason more easily about evaluation, refactoring and so on. But “pure function” or just “function” is for all practical purposes the same thing. A great example of this is SICP. There you program a whole bunch of stuff before mutable state is introduced. It’s a big AHA moment because you realise how much you “lose” when your expressions are not “referentially transparent”. I don’t personally use the term though and know of almost nobody who does. “Functional” is clearer and sufficient, sometimes you need to add “immutable data structures” to be super clear.

I have not gone completely down the rabbit hole, but there are a few academic articles I found several years ago that I do not have the knowledge to understand, as I don't know much about things like formal methods for specifying and proving properties about programming language semantics, but there appear to be subtleties in the definition and meaning of the term "referential transparency" when you get down into the details.

I don't have a cut-and-dried answer for you, but if formal programming language semantics is your thing, or you've got a friend who likes that stuff, I recorded a few references when I was looking into its meaning a few years ago, here: https://github.com/jafingerhut/thalia/blob/master/doc/other-topics/referential-transparency.md

Right, that’s a good point. It’s an academic term. For me, this suffices (so far): Can you still apply the substitution rule (as in SICP).

Interesting! Ill put your article on my reading list!

And the article I wrote does try to give some concrete examples in simple Clojure and Haskell code that I think shows that when you use the substitution rule, or "replace thing X with thing Y, which are equal", then now it becomes very important what your definition of "equals" is, of which there is not only one definition.

Most useful definitions of "equals" in programming "care" about some aspects of two values/objects/whatever being the same, but intentionally ignore some other aspects, and allow those to differ, but still call two things equal.

e.g. Clojure = cares about the values of two collections, and recursively that the values that compose those collections are also equal, but it intentionally, by design, ignores metadata on those collections.

So you can't substitute one object X that is = to another object Y in an expression like (meta X) and expect to be guaranteed the same result, because = intentionally ignores metadata.

this looks one of those blog posts where the author thinks they learned a new concept, and you realize in reading it that they still don't quite get it (meaning the top post here, not anything anyone linked)

I hope I made it clear in my article that there are a lot of things I still dont' get 🙂, although I did enjoy writing it and making some things clearer in my understanding, I think.

Interesting. Also meta programming (macros, eval etc.) differentiates between the result and the original form/expression.

yeah, lisps leave the door open for a type of wacky mutability that most strongly typed languages don't (the side effect of a function can be changing the definition of another function)

I guess, as with everything related to programming the question is: what are the assumptions 🙂

I just reread my own article after not having looked at it in a long while. I had forgotten the quote from a book I found at the end, the book called "Concepts in Programming Languages", which IIRC is about formal programming language semantics. The quote I found that intrigued me was this one: "

The reason referential transparency is subtle is that it depends
on the value we associate with expressions.  In imperative
programming languages, we can say that a variable x refer to its
value or to its location.  If we say that a variable refers to its
location in memory, then imperative languages _are_ referentially
transparent, as replacing one variable with another that names the
same memory location will not change the meaning of the program.
On the other hand, if we say that a variable refers to the value
stored in that location, then imperative languages are not
referentially transparent, as the value of a variable may change
as the result of assignment.

is the difference between pure function and referential transparency is due to "closure" mechanism?

If I understood more precisely what you meant by those two terms, I might be able to answer you 🙂. Here is a concrete example from my article I linked above for thought: Do you consider the built-in Clojure function meta pure or not? This isn't a trick question. I personally consider it a pure function, in the sense that it does not use information from anywhere in the system except that contained in the parameter value you pass to it, and it has no side effects -- it simply reads and returns one field value in the collection you give it, the field that holds the metadata.

Is meta referentially transparent? That depends upon what you mean by referentially transparent, but my linked article has a very short Clojure REPL transcript, probably containing no behavior at all that would surprise anyone who knows what meta and = do in Clojure, but that demonstrates that two collections x and y can return true for (= x y) , but false for (= (meta x) (meta y)). Does that violate referential transparency, or not?

Again, an honest question, not rhetorical. But it seems to me that perhaps a consequence of many definitions of referential transparency is: if two expressions have equal values, you should be able to substitute one for the other in any other expression, and get the same result. If referential transparency implies that, then meta is not referentially transparent, if your definition of equals is Clojure's =

hmm, ok it's beyond the schema I had in mind. It is nearly clear for me. But, @U0CMVHBL2, the function could trigger different behavior depending on meta data. So, in my understanding, the meta data are part of the value. The one in the input function, necessary to determine the result, and the one in the result for the same reason but for the next function.

So metadata on a collection can be used in a Clojure function to affect its behavior, definitely true. There is a very real sense in which metadata is "part of a collection's value". But the important point is that clojure.core/= always ignores the metadata. So when you talk about referential transparency, do you only want to allow substitutions of one value for another if they also have the same metadata?

If so, that is a more strict definition of equals than clojure.core/= .

I dont find any reference on my phone. But I thought there were a more strict equality including meta like identity? or somerthing like that

Finally this article is quite clear, no identity like this does exist https://clojure.org/reference/metadata

Clojure's identical? function tells you whether two things are the same object in memory, which if true implies that their values and their metadata (if they have any) must be the same. If identical? returns false, it still might be true that the collections have contents that are clojure.core/= to each other, and have metadata that are clojure.core/= to each other. identical? cannot be used to find that out for you, though, and neither can clojure.core/= (at least neither of those functions by themselves can do it, you can write such a function yourself pretty easily)

Yes but that is a smell... should not hack equality concept...

Well, I said you could write such a Clojure function if you want, but doing so would not change the built-in definition of clojure.core/= . Agreed there are ways (e.g. changing Java code implementing Clojure) to change definition of clojure.core/= , but if you did that, you might be violating all kinds of assumptions that Clojure library authors are relying upon, and you effectively have a different variant of the language. I would not recommend it. 🙂

Interesting discussion. I'd like to chime in and summarize my understanding of the problem and said terms in the hope of moving a bit closer to intuitive understanding of those or at least having a glaring mistake or omission pointed out wherever I made one -- please feel free to correct me! Afaics, some of the reasons why those terms are confusing and why there seem to be so many ever somehow different definitions⁷⁸ of both, are: • referential transparency is commonly defined as property of expressions, purity as property of functions, and while expressions and functions are related they are not the same¹; • both rely on some notion of evaluation² and defined w.r.t. some evaluation context which often stay implicit³; • in programming the term function is often used to refer to different things interchangeably⁴; My takeaway from all of the above would therefore be: • whenever someone else uses above terms, ask to make sure I (and they!) know and can agree on what those mean; • if that is not possible, assume "referentially transparent" to mean "side-effects-free"⁵⁶ and "pure" to mean "side-effects-free"⁶ and "mathematical function of its arguments" when applied & evaluated; • whenever I want to use those terms, define what they are as precisely as need be to eliminate possible misunderstanding in communication. ¹ Hence, if we introduce a term to carry some meaning for both, we either have to define it for some common abstraction of the two or at least really make sure its meaning stays consistent, but latter without former can still be confusing afaiic. ² Note how in order to allow for e.g. definition of what would be not substitution invariant, the concept of evaluation result needs to account for not only values of expressions which are to be substituted, but also for side effects. ³ Are values of (+ 1 2) or (/ 0 0) safe to substitute in e.g. (let [+ println] (+ 1 2)) resp. (when false (/ 0 0))? There's no way to tell without knowing what the values of free variables such as println or / are, or in case of vanilla / without being a bit careful with the evaluation model. ⁴ Afaiic, in Clojure, a "function" is "value of expression (fn [args] body)" -- but we also use the term to refer to, among other things, the expression itself, or its body. Something like the concept of substitution invariance would be defined for or apply to those three very differently though. ⁵ Iianm under "natural" evaluation concepts it's not very hard to show that for expressions, substitution invariance is equivalent to introducing no side effects when evaluated (w.r.t. to all contexts preserving bindings of free variables occuring within⁶), and that for functions this translates said property to their body. ⁶ Either way, it's unclear to me whether it's common to refer to expressions with free variables as referentially transparent (edit: it seems not) ⁷ https://en.wikipedia.org/wiki/Talk:Referential_transparency#Incorrect_definition ⁸ Whole thread at https://www.reddit.com/r/haskell/comments/xgq27/uday_reddy_sharpens_up_referential_transparency/c5mbuzk/

But "pure function" or just "function" is for all practical purposes the same thing.

I would categorise rand as a procedure rather than a function

Ah, interesting. Goes to show just how much nomenclature varies, I guess 🙂 But now I'm curious -- how would you characterize the difference between a function and a procedure?

Pascal did explicitly separate these two. https://www.pascal-programming.info/lesson7.php

In clojure if there is a bang “!” at the end of a symbol, it usually implies that it is a procedure

Didn't pascal differentiate on basis of return value, i.e. procedures simply not having one?

I think so. A stricter definition would be that a procedure has effects and a function does not and returns something.

There are also other names that are interesting: methods, processes etc. And there is a name that should have semantics attached to it but in most languages they don’t: callback (handler).

Those are all very different things and I sometimes feel like it would be beneficial to separate them more. I imagine linters, type checkers, syntax highlighters etc. could provide more help for differentiation and reasoning.

I see, interesting! I was under the impression that it's common to refer to all of those as function in Clojure¹ (and in many "functional" PLs), and that every function returns a value -- I guess if we can agree on common terms for different kinds of [such] functions, then there's little need to introduce names for properties like "having [no] side effects", "depends on its arguments only" etc.. Last, if I may ask: what would you consider appropriate names for (fn [x] (+ x n)) or (fn [x] (doto x println))? And would you consider map to be free of side effects or not? ¹ https://clojure.org/guides/learn/functions (interestingly enough, there pure is used to mean free of side effects only :)).

I consider the first and map a function, you can pollute them through the environment (closures) obviously. The second would be a procedure. But I’m not an authority on such things 🙂

My take is that (fn [x] (+ x n)) can be considered a pure function if you somehow know that n is immutable (and +, too, by the way). If either n or + can be changed over time, then it is no longer a pure function.

map is free of side effects and it is a pure function, if the function f you give to it is, and the sequence/collection you give to it is immutable.

That’s a sensible definition. My gripe is that it isn’t (fn [x] (+ x n))’s fault that it isn’t pure 😄

Many things (not all) things in Clojure can be considered "pure, but only if the functions you pass to them are pure, and only if the parameter values you pass to them are immutable, and only as long as you do not redefine or mutate any Vars they refer to", e.g. + or n in (fn [x] (+ x n)).

The substitution property seems to me to depend very heavily on what your definition of "equals" is, as my linked article intends to demonstrate with simple examples from Clojure. e.g. meta in Clojure is a pure function by every definition and test I can think of, but it does not preserve substituting equal parameters for equal parameters, if your definition of equals is clojure.core/=

I think Clojure's meta would satisfy the substitution property if you changed your definition of equals to (defn custom-= [x y] (and (= x y) (= (meta x) (meta y)))) 🙂

@U0CMVHBL2 Iianm there's less direct connection between substituion invariance (edit: see below) and function application being invariant under in-language equality than seems to be at first for two reasons: • technically, substituting value of evaluation and comparing results would amount to something more along the lines of (let [x (y)] (= (meta x) (meta (y)))) • equivalence used when considering substitution property should be defined not in terms of equivalency of in-language values, but in terms of evaluation model, since we want to consider difference in evaluation results which in general include not only an expression's value but also the side effects of evaluation. Consider this: nil and (println "Hello") both evaluate to same value but they are not substitution equivalent under most useful definitions of equivalency. And there is no way we can tell the difference inside our program

But substitution invariance relies upon some definition of equality, doesn't it?

even if it isn't the in-language one.

And I don't see why the definition of referential transparency has anything to do with being restricted to something like (let [x (y)] (= (meta x) (meta (y)))). I thought that the intent is that you can substitute equals for equals anywhere, not just in special expressions like that.

Ad equality: yes, we certainly need one afaics and it cannot be in-language as long as we want to be able to consider side effects, otherwise nil would be equivalent to (println "x") and ((defn p [] 3)) would be equivalent to 3. As to your last point: that is because as I see now we use two very different definitions of substitution invariance (or R.T.) of an expression -- one defined in terms of substituting "equals for equals" (expressions with expressions or expressions with their values?) inside the expression, the other via substituting the expression itself by its value. Afaics both interpretations occur¹² but surely will not be equivalent. I reckon fwif it's at least clear that side effects seldom make things easier, not even talking about their absence 🙂 ¹ https://en.wikipedia.org/wiki/Talk:Referential_transparency#Incorrect_definition ² Whole thread at https://www.reddit.com/r/haskell/comments/xgq27/uday_reddy_sharpens_up_referential_transparency/c5mbuzk/

My next takeaway from this shall be that said concepts are useful to think about and tinker with structure and interpretation of PLs (pun intended), but less useful for communicating using those concepts relying on any kind of preexisting precise common notion behind them -- sorry, OP 🙂

It certainly seems to be the case that many people use the term "referential transparency" in the context of programming languages in very different ways from each other. "pure function" seems to have less variability in how it is used.

And from several things I have read about the earliest uses of "referential transparency" in the context of programming language semantics, it is perfectly possible to define "referential transparency" and semantics of a language like C/C++ such that in the context of those semantics, C/C++ have that referential transparency property.

Yes, looking into common usage(s) of R.T. had been a fun and interesting excursion into what seems to be a can of worms -- I think the stackoverflow thread, which I found to be far more amusing than helpful, is a good reflection of that. As to the term pure, I think most definitions I've seen are a combination of «free of side effects» and «mathematical function of its arguments» -- would you be aware of other properties the last two might not capture?

Am I aware of other properties of a pure function that those two phrases don't cover? None come to mind, but I haven't tried looking for counterexamples very long.

Fair enough 🙂 As I said before, this whole thread had been quite interesting. As well as your collection of links @U0CMVHBL2!

Glad you found something of use to you there. I think perhaps it would be better if my article did not try to link the ideas of "referential transparency" and "x equals y implies (f x) equal (f y)". That latter property seems like it might already have a better different name for it. It seems there might be some relationship between what Haskell programmers call referential transparency, and that property, but I'm not sure what that relationship is.

This page calls that "x equals y implies (f x) equals (f y), for all f" the "substitution" property, but it might have a more precise name, too.

Oops, forgot to give a link for "this page": https://www.reddit.com/r/haskell/comments/1njlqr/laws_for_the_eq_class/

Afaiic atm, any defitinition of r.t. is as good as any other, since they are as many as they are rare 🙂

(Which is not to say that different definitions cannot be equivalent)

If I would suggest one thing at all concerning your article, it would be that it probably might be useful to also define what "referentially transparent" means when you use it. This usually leads straightforwardly to a definition of "referential transparency", while the other way round is often ambiguous. E.g. iianm your definition would allow for two interpretations: (i) calling an expression "referentially transparent" if it itself when substituted would yield equivalent results as well as (ii) if substituting or any of its subexpressions would do so.

"it might be useful to also define what "referentially transparent" means when you use it". -- one of the things I came to believe as I wrote the article is that I am not clear on what it means. 🙂

Do you have a particular definition of "referentially transparent" you would use, should you write an article about it?

Usage of r.t. in haskell as far as I understand it (I'm only tangentially familiar with it) is much better suited for «x equals y»-in-language-based definitions for one reason: that its type system encodes most kind of side effects or their absence. Hence, in haskell, we have a guarantee that if x equals y then evaluating either x or y will produce same side effects. And that is why in clojure this definition might be less useful -- because values of nil and (println "3") cannot be not equal in-language. Of course, there are still side-effects we might care about which even haskell's type system will not encode -- e.g. do we care about amount of memory consumed during evaluation or not? What if one of the expressions consumes more memory than my machine has? That's why generally some outside notion of equivalency and some model of evaluation is needed, the other reason being that not every outside value is an expression which can be substituted (e.g. functions as values in clojure).

My takeaway from all of this is that you get to define what r.t. means. Of course, that is the case with any other term -- but here you even would be hard pressed to find an agreed upon definition you could run at odds with 🙂

fwiw, if I had to define "referentially transparent", I'd go with some notion of "free from side effects" as definition I think

I guess I did not say it, but when I think of the property "if x equals y, the (f x) equals (f y) for all function f", I believe I am implicitly thinking that x and y range over all in-language immutable values, not over all expressions in the language, and all functions f means all pure functions f you can write in the language.

So I never even thought of the possibility of x being (println "3")

i.e. I was (without saying it explicitly) excluding such a possibility.

Well, substitution only makes sense inside expressions afaics, not inside values -- so the substitution property has to be defined in terms of expressions somehow. Usually something like "if x and y evaluate to same values, then we want to be able to substitute x for y" (or "x for its value" etc.). Then in clojure we would have expressions nil and (println ""), which both evaluate to same value. But we would not necessarily want to substitute one for the other in our program. In haskell on the other hand the two could never be equal because they would belong to disjoint types -- but I might be wrong here.

"if x equals y, the (f x) equals (f y) for all function f" is different in that it does not operate with substituions (as it is)

but I think I might now understand better where you're aiming at with this formulation

As I said, I think what I meant by that property is not at all clear how it relates to other definitions of referential transparency.

But I did put a lot of weasel words before it 🙂. "At least some people seem to take the view that referential transparency in a programming language means something similar to this:"