Hi peter. Regarding Nippy, I checked our project and it's using 3.2.0. What is the background for recommending an update?

Hi there! Updating from v3.2.0 should not be a problem, though I’d suggest always testing in your own environment to be safe. Reasons for the recommendation to update: • v3.4.2 includes some important upstream fixes in internal dependencies. • v3.4.1 introduced a new internal mechanism to help smooth changes to data formats for users that do rolling updates - and I’d like to get everyone on this at some point so that in the future I can rely on most users having the feature present. There shouldn’t be any downside to updating, except: • Serialized output may change for some users (as mentioned in release notes). That’s not generally a problem except for the very small minority of folks that depend on specific serialized byte values. • The usual maintenance hassle of updating and testing (which I realise is non-zero!).

Thanks for the explanation. After the upgrade, my CI comments as follows. Looks good to go!

So, I know that it is sometimes very useful to break things, but in this library’s case (and as a library maintainer myself), I think it is a very very dangerous mode of release to adopt that creates a huge amount of potential work for downstream users. We use carmine heavily, and it serializes with nippy. Much of the data we store in redis is relatively long-lived and is also quite large. So, having a break change that makes all of that data stop working would be horrific, and since nippy is a nested dependency it creates quite the difficult situation for dev teams to track what might kill production in very serious ways. Even a change in the byte format is serious, in that it can prevent a production rollback from behaving properly. We’re talking many GB of data here in redis, and we rely on trustworthy deploy/rollback semantics. Why can’t the API have opt-in semantics to breaking changes? Or new nses that implement new functionality that people can port to/use when they decide they need them? I know this creates a bit of headache on the library maintainer and documentation effort, but especially for a library that is used the way nippy is it really seems like break versioning is a very bad idea.

@U0CKQ19AQ Hi Tony - just to clarify, what breaking change/s are you talking about here? > Even a change in the byte format is serious, in that it can prevent a production rollback from behaving properly. We’re talking many GB of data here in redis, and we rely on trustworthy deploy/rollback semantics. Could you clarify what exactly your concern is? Nippy has never promised precisely stable byte output, and has had several changes over the years as new types were introduced and other improvements were made. These changes are completely transparent to almost all users. I make the warning in the release notes for the benefit of the tiny minority of users that I know specifically test for byte output for one reason or another. Re: rollback support - this just isn’t something that can be completely guaranteed without locking down the supported types. If Nippy vX introduces support for a type that is not supported by vX-1 - and you’re writing those new types during a rolling update, you are inherently going to have trouble. Again, I point this out in the release notes out of an abundance of caution so that folks can be aware and take a moment to consider if it might affect them. I’m trying to dissuade people from mindlessly bumping a number in the deps.edn, but to actually make a moment to think about possible effects. > Much of the data we store in redis is relatively long-lived and is also quite large. So, having a break change that makes all of that data stop working would be horrific, and since nippy is a nested dependency it creates quite the difficult situation for dev teams to track what might kill production in very serious ways. I agree, and understand this very well - which is why Nippy releases have been so conservative over the years. I have many terabytes of data frozen with Nippy 12+ years ago that is still readable just fine today - and that I intend to keep readable indefinitely. Nothing has changed on that front. If anything, I’ve become more conservative over time since Nippy is now so widely used. I believe you might be responding here not to any change in Nippy, but in an escalation in the release note warnings to apply common sense before changing software in production that may indeed affect long-lived data.

Yeah, I know you do excellent work. Love your stuff. I also understand the nature of a user opting to use new features (new types) causes a breaking change for them, but that need not be an inherent property of the library itself (e.g. enabling new types could is inherently opt-in…existing code won’t be using them yet, so the library supporting them isn’t a problem). I’m more just talking philosophically. It just scares me when a library that is used to store data has a policy of using break versioning, and was suggesting that any “breaking” change could actually be handled differently.

> having a break change that makes all of that data stop working Where are you getting the impression that your data is suddenly going to become unreadable?

Perhaps this is just a documentation issue: I definitely see you trying your best to maintain API-level stability on data encode/decode..but break versioning implies you reserve the right to break it, no?

perhaps just a bit of documentation that says something like “Core clojure data types and primitives are guaranteed to be byte-level stable”

> I’m more just talking philosophically. It just scares me when a library that is used to store data has a policy of using break versioning, and was suggesting that any “breaking” change could actually be handled differently. I think there’s some possible confusion here on terminology. You seem to cite the use of “break versioning” as somehow indivitive of a casual attitude to introducing breaking changes. That is absolutely not the case. It’s just the semantic https://www.taoensso.com/break-versioning I use for version numbers, and have used (unchanged) for many years. We are absolutely on the same page philosophically, especially with respect to Nippy.

at least some kind of well-defined boundary around what we can trust to be forward/backward compatible

(I use break versioning most of the time as well, in pretty much the same way as you’ve documented) same philosophy, though, it’s there because sometimes it is really unavoidable. I go to herculean efforts to avoid it.

> at least some kind of well-defined boundary around what we can trust to be forward/backward compatible Sure, I’m happy to add some docs on this. Most importantly (and as always)- data written with Nippy today, will always be readable with all future versions of Nippy. Only the most extreme exceptions would be made (e.g. if some major security vulnerability were involved), and in those cases would be VERY clearly indicated and justified, and a workaround provided to still get at the data. Nippy is never, ever going to intentionally invalidate your data. I’m sorry if anything gave you that impression (/ heart attack). > it’s there because sometimes it is really unavoidable. Just to clarify - the term “break” in “break versioning” is not at all to imply that breakages are somehow easier to make or more frequent. Versioning scheme and willingness to break things are orthogonal. And willingness to break things is completely context dependent. Breaking the readability of long-held data would be absolutely unjustifiable in any universe, except for extreme security exceptions. And even then, there’d need to be a viable workaround provided.

We’re on the same page. Just nice to hear it said explicitly 🙂 decades of getting “surprised” makes me extra paranoid.

Totally fair to be paranoid, I’ve had the same traumatic experiences 🙂 Will get a clear promise documented on the README next time I’m on batched Nippy work 👍

Thanks for your time, Peter. Great talking with you, as always

Likewise Tony 👍

@U0CKQ19AQ Quick updates: • I’ve added a new section on https://github.com/taoensso/nippy/wiki/2-Operational-considerations to the README and Wiki. • I’ve updated the v3.4.2 https://github.com/taoensso/nippy/releases/tag/v3.4.2 to hopefully make it clearer that this is a non-breaking update for the vast majority of users. And that “breaking changes” will anyway always be limited to minor API changes or possible changes to https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output (which has actually never been an official part of the API or implied to be stable, so the term “break” is anyway relevant only to rare off-label use).