I forwarded this to the #C0H28NMAS channel where someone is more likely to see it...

yup it's indeed down, but there seems to be a mirror on maven

@U0479UCF48H do you have a link?

https://mvnrepository.com/repos/clojars

It should be back up now. The JVM process was at 100% CPU usage, and was unresponsive. Killing it and restarting it fixed it. This impacted the http://clojars.org UI, deployments, & search. http://repo.clojars.org was still able to serve artifacts, but directory listings would not have worked. The mvnrepository page for Clojars is new to me. Is that actually a mirror of the repo, or just an index of clojars? If it is a mirror, I don't see how to use it.

I found https://www.statuscake.com/ quite simple to use if looking for an alternative (I have no affiliation or kickbacks from the company)

Thanks! I'll check that out. I've also opened a support ticket with http://uptime.com

Maybe it's not what you're expecting, but the best approach for such cases is using a migration library. Migratus would be a great choice. It tracks the sql files and applies them to the database.

That do seem the best solution. Even more, it has an example using next.jdbc . Awesome. Yeah, I didn't expect it. But, if it's the way, what we gonna do, right? 😆. More stuff to learn.

I think it depends on what the files are. If you're actually doing migrations then a migration lib is useful. It will track which migrations have succeeded on which databases and allow you to manage the state of your migrations over time. But if you want to ingest data from a mysql dump, for example, then using the mysql command line tool will perform much better. It will allow for things like disabling things like referential key integrity during the import under the assumption that things will be fine by the end of the import. Also, I think that the mysql jdbc driver requires that you don't execute more than one sql statement at a time to help prevent sql injection attacks (or at least, it used to) so you may need to do something like split your string on ; if you're intending to manage more than one alter statement in the same file. This isn't true for the command line tool, and I suspect is a problem solved by migrations libs like migratus or flyway.

With-connection => with-open & get-connection

Do-prepared => execute!

But, yeah, use a migration library instead of rolling your own would be better.

I ended up sshing into the remote and dropping the files with \i , with postgres . It was only two files. But, I would rather have an automated way of doing it, in case I have to repeatedly setup something like that. When the time comes, I will use migrations, for sure. I wasn't successful at first try... So, I will keep manual, for now.

Thanks for the help and to tip me on the right direction, with context.

(changed to postgres because I got a free db hosting for the mean time.)

Actually, assert works just fine. No need to be clever here.

Once any given version of a library is loaded, you can't dynamically load a different version of that library (without a massive amount of classloader complication). For any given invocation of add-libs, the behavior is deterministic -- but loading deps is inherently stateful so it will be order-dependent in some situations by definition.

(and given the intended use case, this is a reasonable trade off, IMO)

Thanks Sean. I have a background in OSGi, including having worked in Eclipse's Equinox implementation, so I naturally wonder about what conversations have happened around classloader hierarchies so multiple versions of the same library can coexist and be depended-on by the right things? (Terrible sentence, Dave, but hopefully the intent is clear?)

e.g.: I'm relatively new to #C5H6JJQKZ and don't want to cowboy-ride into the forum with a bunch of assumptions. 😄

OSGi is its own special thing... 🙂

and scary :)

and massively complected

...but

Something like Java's 'one-jar' seems like a more reasonable scope. Same principle; similar implementation...

...much simpler

I think there is an OSGi "module" for Clojure but I don't know if it is maintained any more?

https://github.com/talios/clojure.osgi

add-libs is for interactive dev, if you want deterministic versions put it in deps.edn

Yeah; I looked at that. I don't want OSGi because Clojure is already basically a container with strong opinions about how classloaders work. And I'm already bald. (Yes, really.) 😄

I think the JVM is inherently a difficult environment for systems which dynamically load code (and are intended to also be long-lived processes)...

(and I'm not sure that OSGi is worth the pain -- it essentially chased me out of another tech stack because of the pain it caused :rolling_on_the_floor_laughing: )

Loading different versions of the same library as dependents for different libraries to solve the dependency diamond problem is... problematic in general. I can only think that it could be done as a result of explicit overrides either by the end user specifying what libraries to load multiple copies of, or by metadata within a library specifying that it is compatible with other versions of the same library being loaded simultaneously. Otherwise you run into issues of shared resources being misappropriated.

@U5NCUG8NR In theory, true. In my experience with OSGi, logging is the main place this tends to happen.

(OSGi really pushes one toward a shared-nothing FP-based design.)

I mean maybe the majority of cases are fine, but in the cases where it's not the case that gets hairy really quickly, and I'm speaking mostly because I am an author of several different libraries that would break horrendously if you loaded two different versions of it.

Cool! Tell me more.

(Thanks for your input. Genuinely curious here.)

Well for one I'm author of #C02CB8WHLV8 which is an error handling and control flow library that relies on a particular java class which it uses in catch blocks in order to implement that flow control. Loading two different versions as dependencies of different libraries now means that you cannot 1: use farolero to handle conditions from both libraries simultaneously, 2: have code from library a that uses condition logic get passed as a higher order function argument to library b that uses condition logic, 3: have farolero interact correctly with fmnoise/flow for railway oriented programming as both loaded versions will separately try to extend that mechanism in conflicting ways.

For two there's #C02EEAUHSJJ which allows the use of native code from clojure with little ceremony and it would likely suffer from extensive classloader issues if two different libraries loaded different versions of it, especially in cases where you wish to pass lambdas as function pointers to native code.

For three there's discljord which is a bot library for discord which while it wouldn't break just from being loaded twice, if two substantially different versions (but still within the same major version) are loaded there is a non-zero chance that the internals of the library have completely changed, resulting in breakage if a constructed discljord channel is passed between libraries built against different versions of the library since it has undergone multiple complete rewrites of the internals while maintaining non-breaking changes of the API.

All three of these would result in strange behavior that would be exceedingly difficult to track down.

It's true that some libraries can't be loaded more than once. OSGi's solution is to annotate those libraries as singletons, and the loader/solver does its best to make everybody happy.

Right, and that's what I was saying, in order for this to work in general you need some kind of annotation. I personally believe that it is more generally correct to annotate things that are not singletons in order to reduce the amount these sorts of shared resource issues, as they are I suspect in general more difficult to trace than version conflicts are.

True. So what I think I'm hearing is that it's a hard problem that nobody has had the confidence (arrogance?) to attempt for Clojure and if I want this for Clojure, I'll have to do it myself and make a proposal to the community? 😉

I wouldn't say that it's a completely unaddressed problem. It's not one I'm aware of anyone attempting anything for outside of the above clojure.osgi, but there has been some acknowledgement of the problem and discussion of some possible ways to handle some subset of the problem.

Right. How many copypastas do we have of Pomegranite's code, for example?

LOL

In Rich's Spec-ulation talk he mentions the possibility in the future of having two dependencies that refer to disjoint sets of a dependency's vars each getting the version they care about.

just catching up, is there an open question here for me to answer

Hi Alex:

> How many copypastas do we have of Pomegranite's code Why would anyone copy'n'paste it, given it's a library that any code can depend on and use? (just like tools.deps is in the non-Pomegranate world).

@U064X3EF3 I was looking at CLJ-2761 and noticing that it's using an algorithm of first-loaded-version of a library wins. Having experience with OSGi, I started "speculating out loud" about if/how worthwhile it is to allow multiple versions of a given library (loaded through a classloader hierarchy).

@U04V70XH6 That quip was because I've run into multiple forks of Pomegranite in the wild while reading Github. (Clojure projects on Github are often my favorite reading material; sure beats reading about politics.)

@U064X3EF3...and since I'm relatively new to Clojurians I was also asking about what conversations have already happened since I don't want to be That Cowboy Coder riding in... 😉

Given that you're loading libraries into a running program (the REPL), if you used a classloader hierarchy to allow multiple versions of each library to be loaded, I think you'd pretty quickly run into problems of type comparisons failing due to "the same" class being in multiple classloaders? (i.e., unable to coerce ClassA to ClassA because the two pieces of code are in different classloaders) I mean, we can already run into this in a REPL when reloading namespaces sometimes... yes?

this could (potentially) be a real big problem with libraries that offer Component interfaces

@U064X3EF3 Are any of the design docs/discussions around add-lib stuff publicly accessible, such as the design spreadsheet? (I think that's the core question here)

No, but the public tickets summarize the highlights. For this, it has always been targeted at repl use and best effort loading and trying to handle multiple versions of a lib is out of scope

@U04V70XH6 The main place I've run into this kind of thing is when running multiple Clojure instances from different classloaders. This is why Boot only allows one to pass serializable (basically EDN) data structures between pods. This is because each library only has visibility on the version of the code it explicitly depends on.

@U064X3EF3 Understood about the current API use-case. And from an engineering/product perspective this makes a lot of sense to me. Searching "Classloader" in both Jira and Confluence isn't picking up anything that appears to be relevant. Is there a better search term I should be using?

Have there already been discussions about (eventually) supporting explicitly-versioned libraries in Clojure (in addition to Joshua's reference to Rich's talk)?

If so, what would I search for to find them? (TIA)

All of that is speculative and doesn't yet exist as tickets I suspect.

As for tickets that summarize what's relevant to this discussion, Alex means look at the tickets that have been addressed as part of Clojure 1.12 alphas so far.

https://clojure.org/releases/devchangelog

@U04V70XH6 That's what I figured. (This would be a great time for me to be explicit that I'm not intending to criticize what we do have for its intended purpose. 🙂 )

I think it's fair to say that a lot of Clojure's design happens offline, with Rich and Stu and the core team, and isn't generally available online. Our experience with Clojure over the years has tended to make us fairly confident that they know what they're doing 🙂

(some of the "spreadsheet" design documents have been available for some features in the past but those have been the exceptions, I'd say -- and a lot of features that end up in Clojure have been going through an extended design process for years by the time they arrive in the language and tooling)

That's what I've noticed as well. I'm a fan of conservative design iteration, including not having too many cooks in the kitchen at the beginning. (I've been around Clojure professionally on and off for about 7 years now; I just haven't had much occasion until now to hop on Clojurians--and for a number of years worked at a company where Slack was blocked. LOL. 🙂 )

Thanks, @U04V70XH6, for the welcoming introduction. 🙂

Slack blocked? OMG! 😱

@U0587N61TMH what is the problem you're trying to solve? is it specifically to do with having multiple instances of the Clojure runtime in separate classloaders in a JVM?

@U064X3EF3 From the bottom up: A lot of library version conflicts can be solved if each individual library gets its own classloader, and that classloader's delegation algorithm delegates first to the classloaders of its explicit dependencies, then back to the context/system classloader. Each library only sees classes in precisely the versions that it depends on. (I know Clojure's classloading is a bit more complex than this.) From a community perspective, I can see value in just having this capability built into the platform. From the top-down: I'm trying to push Clojure as hard towards Smalltalk's dynamism as I can in order to see what works and what breaks. I think there may be some more low-hanging fruit from the REPL-driven development perspective here. On me: I spent 10 years working on corporate Eclipse RCP (OSGi) rich client/desktop applications so have some experience with this particular architecture--within the Java space. I also founded Eclipse's Visual Editor Project (GUI builder project), which dealt directly with Eclipse's OSGi implementation of these things. (VE was the first non-IBM project at http://Eclipse.org.)

@U0587N61TMH Just so I can build a model in my head, if library A.1 depends on B.1 and C.1 and library D.1 depends on B.1 and C.2, how many classloaders do you get? Five? (since B.1 is "shared" between A.1 and D.1 but C.1 and C.2 are different versions?)

So the problem you’re trying to solve is transitive version conflicts

@U04V70XH6 5; yes.

@U064X3EF3 Yes; as much as is possible for a system that is incompletely specified up-front.

We’ve instead taken the approach in the community of creating a culture of non breaking version updates, such that the newest version is always the best, and then you don’t have conflicts

We have no plans (now) to build a class loader system in the language to support multiple versions of a lib

@U064X3EF3 Fair enough.

at least for now it still seems true that a "lib" doesn't exist at runtime, right? Does the add-lib facility change that much or just bolt on a little class loader bit to tie into tools.deps which does have a concept of a lib?

Thanks for letting me know.

@U5NCUG8NR Assuming it's an updated implementation of add-lib from tools.deps, yes. (I read its source code yesterday.)

That's what I expect.

Good to know it doesn't change much in that regard.

I'd like to thank everyone who participated on this thread. It's been helpful and informative, and I appreciate the time everyone here spent discussing and helping.