Fork me on GitHub
#clojure
<
2019-10-04
>
Cora (she/her)04:10:43

is there anything in clojure for installing global packages with executables like you can with npx and rubygems?

seancorfield04:10:35

I'm not sure what that question means in the context of Clojure @corasaurus-hex

seancorfield04:10:06

When you ask for a dependency, it is fetched and cached into the local Maven repository cache in your home directory. After that, any process you run under your account can reuse that dependency without fetching it again.

seancorfield04:10:00

You can use an environment variable or command-line option I think to specify a different location for that cache... so I guess you could use a shared location across multiple accounts -- if that's what you mean by "global"?

Cora (she/her)04:10:07

I mean I can do gem install rake and then run rake from anywhere

Cora (she/her)04:10:20

it makes a binary available in the path

seancorfield04:10:29

If you have a dependency cached, you can use it from any project.

Cora (she/her)04:10:04

that's not exactly what I mean, though

Cora (she/her)04:10:11

but I get your meaning

valerauko04:10:32

you mean like installing leiningen globally

seancorfield04:10:36

The JVM doesn't have "executables" -- it has class files on the classpath which java "executes".

Cora (she/her)04:10:54

right, like lein being installed globally

Cora (she/her)04:10:00

lein is an excellent example

seancorfield04:10:15

That's just a matter of having lein on your PATH tho'...

seancorfield04:10:34

And that's nothing to do with Clojure/JVM's view of the world.

Cora (she/her)04:10:36

yes, and other languages provide package managers that let you install those off of the shared package repo

Cora (she/her)04:10:48

I was wondering if there was something like that in clojure

Cora (she/her)04:10:51

sounds like there isn't

seancorfield04:10:00

lein (and boot and clojure/`clj`) are shell scripts/executables that are outside the Clojure/JVM ecosystem.

seancorfield04:10:25

Maven Central and Clojars are those "shared package repo"s.

seancorfield04:10:12

But there are no "executables", just libraries that the JVM can use.

valerauko04:10:19

i don't know of a way to install clojure "apps" that way from maven, but if you have a need for it

seancorfield04:10:23

The JVM -- the java command -- is the executable.

valerauko04:10:26

then hacktoberfest is the perfect time

Cora (she/her)04:10:02

it's just a thing I find useful in other ecosystems that's noticeably missing here

andy.fingerhut04:10:23

When you do gem install rake, is rake a native machine binary, or is it a file containing Ruby source code with a #! as the first line?

seancorfield04:10:57

So it requires the ruby executable already installed

Cora (she/her)04:10:59

in rust it's the former

andy.fingerhut04:10:03

So if that command after the #! doesn't exist on your system, then it can't run.

seancorfield04:10:13

Just like Clojure requires the java executable already installed

Cora (she/her)04:10:13

which is to be expected, really

andy.fingerhut04:10:31

You can write similar things with Clojure source code if you want, using a first line like #! /usr/bin/env clj ...

andy.fingerhut04:10:49

Those will work anywhere on your path, as long as clj and a JVM are also on your path.

Cora (she/her)04:10:17

so, a package manager that makes those sorts of things available is a real possibility

Cora (she/her)04:10:14

it came to mind because I install clj-kondo and lein via homebrew

Cora (she/her)04:10:22

and I could see a lot more things like that being useful

andy.fingerhut04:10:24

The syntax gets a bit weird for that one.

Cora (she/her)04:10:39

ahhh yeah, I see

seancorfield04:10:51

@corasaurus-hex but clj-kondo is an executable

seancorfield04:10:08

Like I said: lein and other things are not Clojure

Cora (she/her)04:10:25

sure, but it could be built locally by a package manager on installation

Cora (she/her)04:10:29

like with cargo in rust

andy.fingerhut04:10:31

I believe @borkdude creates native executables from Clojure source using GraalVM, or some such wizardry I haven't looked into

Cora (she/her)04:10:44

that's what the readme says, at least

Cora (she/her)04:10:06

@seancorfield for sure, it's outside of clojure at the moment, but it need not always be the case

valerauko04:10:12

package.json files can have a "bin" bit that defines how to run it huh

Cora (she/her)04:10:36

ruby gems have a bin directory, or an exe directory

seancorfield04:10:39

So, yeah, you can use homebrew or similar to install scripts and executables... but that's completely independent of Clojure.

valerauko04:10:39

you'll need to come up with some way to define that for lein/boot if you were to do it

Cora (she/her)04:10:42

it's in the gem specification

valerauko04:10:27

or just use the target/xxx.jar as the default

andy.fingerhut04:10:39

I mean, there is brew install clojure, so presumably one could create Homebrew things that depend upon the Homebrew thing named clojure also being installed.

Cora (she/her)04:10:41

@seancorfield I don't understand what you mean by outside of clojure. If I make a package manager that does that for clojure then it's part of the clojure ecosystem

andy.fingerhut04:10:40

Sorry, I'm new to Homebrew, and don't know the lingo there yet.

seancorfield04:10:02

@corasaurus-hex what does "for clojure" mean in your sentence above?

valerauko04:10:30

a tool that produces an "executable" (from the user's point of view) from clojure source code

valerauko04:10:44

even if said executable is just an alias for java -jar target/app.jar

Cora (she/her)04:10:45

and makes it globally installable, with the executable on the path

Cora (she/her)04:10:07

like is done in ruby and npx and cargo and in php and cpan and etc etc etc

andy.fingerhut04:10:15

e.g. creates an Uberjar with a one-line shell script that runs java with that uberjar named?

Cora (she/her)04:10:42

I suppose there are lots of possibilities there

Cora (she/her)04:10:04

it's an interesting idea, and it seems there's some level of need for it

valerauko04:10:12

i think it shouldn't be hard with a lein plugin

valerauko04:10:19

chances are there's already one (last commit 2009)

andy.fingerhut04:10:21

If you don't mind the Clojure source code being in the "executable", that seems to achieve the effect, one one way (there are others)

andy.fingerhut04:10:49

just as rake includes all or most of its source code in its executable.

Cora (she/her)04:10:06

in the case of ruby it does the dependency resolution at runtime

Cora (she/her)04:10:40

but the deps are installed to the system at package install time

andy.fingerhut04:10:21

clj / clojure just do the late binding one more step -- they find the deps over the network after you invoke the command?

seancorfield04:10:14

Bear in mind that you already have clojure -Sdeps '{:deps {somegroup/someartifact {:mvn/version "RELEASE"}}}' -m some.thing which will fetch that library (if not already cached) and then run it's some.thing/main function.

Cora (she/her)04:10:30

sure, but typing rake is a lot nicer

seancorfield04:10:49

What I mean by "outside the Clojure ecosystem" is that what you need/what is to be able to install a shell script that runs clojure with specific arguments.

Cora (she/her)04:10:53

but! rake or whatever could just be calling that

Cora (she/her)04:10:47

I don't know that it's outside the clojure ecosystem if it's catering to the same ecosystem, but I won't belabor that point

andy.fingerhut04:10:01

I have about 3 tiny shell scripts in my path that do nothing but run a clojure command with long args I don't want to type twice.

seancorfield04:10:04

We already have plenty of package managers (outside Clojure) that can find and install shell scripts -- just like brew install lein or brew install clojure -- and all you need to write are the shell scripts to invoke libraries.

Cora (she/her)04:10:05

it just sounds like a useful tool with some cool possibilities

seancorfield04:10:24

But we already have such tools.

Cora (she/her)04:10:30

oh but we don't

seancorfield04:10:35

They're just "outside the Clojure ecosystem".

seancorfield04:10:47

Homebrew, Scoop, Chocolatey, etc.

Cora (she/her)04:10:44

I don't get your point, I think it would be extremely useful to have something in the ecosystem that catered to clojure's needs

andy.fingerhut04:10:51

Are you trying to make something where you can guarantee that after step X, you can disconnect from the Internet and the program is all local?

Cora (she/her)04:10:53

yes people solve it with dozens of tools now

valerauko04:10:00

something like homebrew specifically for clojure libs

Cora (she/her)04:10:34

@andy.fingerhut I think that would be the nicest version of it

Cora (she/her)04:10:41

no runtime surprises then

andy.fingerhut04:10:50

Because if working after disconnecting from the Internet is not a requirement for you, I don't see why shell scripts a handful of lines long that invoke long clojure commands don't do what you want.

valerauko04:10:38

i'd lein uberjar standalone it then make a shell script that does java -jar uberjar and puts it somewhere on the PATH

andy.fingerhut04:10:59

And even if working after disconnecting is a requirement, you could say "run the program once and let it download the rest of itself before disconnecting from the network"

valerauko04:10:04

i guess it could be very useful for making trivial little tools in clojure

Cora (she/her)04:10:19

or even non-trivial ones

valerauko04:10:35

considering the startup time of clojure, preferably non-trivial ones actually haha

Cora (she/her)04:10:25

I mean of course it's all about path management and simple wrapper scripts that dispatch to a larger program, that's mainly what it's about in all these other languages, too

Cora (she/her)04:10:48

but they end up being valuable in these other languages and make writing and distributing certain kinds of tools a lot easier

👍 4
Cora (she/her)04:10:01

anyways, I'll put that in my pocket for now and maybe revisit it later

Cora (she/her)04:10:29

thanks all, g'nite

seancorfield04:10:55

Given clojure's user-level deps.edn and "global" aliases in that, I can't say I find the need for what Nate is talking about...

seancorfield04:10:50

It would only automate such a tiny step in the process (basically adding a new alias to deps.edn and perhaps a shell script that invokes clojure with that new alias).

andy.fingerhut04:10:33

At least from one perspective, mvn/lein/boot/clj are Clojure package managers, if you consider JAR files in your $HOME/.m2 directory as installed packages.

andy.fingerhut04:10:54

They just don't have "uninstall" or "list installed packages" commands

seancorfield05:10:33

"list installed packages" = ls -d ~/.m2/repository/*/* 🙂

seancorfield05:10:32

ls -d ~/.m2/repository/*/*/[0-9]* if you want all the installed versions 🙂

seancorfield05:10:30

For years I resisted installing homebrew on my Mac -- too much "magic"... And for years I installed stuff manually because that's what I'd been used to for decades of using computers.

andy.fingerhut05:10:31

Yeah, I got tired of that some time around 2000 🙂

andy.fingerhut05:10:02

Partly because I figured out that if things got weird or seemed broken, and I had a couple hours to kill, I could rm -fr /opt/local and start over from scratch.

Jakub Zika07:10:47

Hi everyone, do you have some tips how to make uberjar small as it can be? Is there some Lein plugin that will tell that this required namespace is not used etc? Thank you

lxsli07:10:32

I use https://github.com/snoe/clojure-lsp which can tell me when required namespaces aren't referenced. I doubt you'd actually shed libraries like that though unless the code is Extremely Legacy.

lxsli07:10:43

Otherwise I'd follow Java JAR minimisation techniques - reduce the # of classes you create etc

jumar08:10:24

Most of the bloat actually comes from dependencies so I'd cut out stuff you don't really need or is available via another simpler libs.

jumar08:10:46

Perhaps the question is what is your motivation for minimizing the jar.

✔️ 4
acron08:10:23

Morning all, I remember once encountering a convention when writing Clojure... something along the lines of when should not be the final form in a function, it should be substituted with a single branch if...but I don't remember the reasoning for this. Does anyone recall this or what on Earth I'm talking about?

plexus08:10:38

never heard of such a thing @U052SB76M. I could imagine someone arguing for using (if test then-branch nil) to be explicit about the return value, but honestly I think when is just as clear, and more clear then a single branch if

acron08:10:47

Maybe I am misremembering it and that was the point being made, as that would certainly be a reasonable argument. Cheers, I'll think I'll go with this 😛

acron08:10:04

But yes, I agree with you in any case

rickmoynihan08:10:16

FWIW: I understand @corasaurus-hex’s desire, to essentially package tooling with a project. Though we just use aliases in our deps.edn that specify a :main-opts and :extra-deps for pulling in java/clojure tools and services. And this works really well. However it doesn’t work for stuff that is compiled with graalvm, e.g. clj-kondo… I’d also like to package, compile with graalvm and ship as downstream executables some of our custom tooling in a similar way. I did notice that mvn repos can in principle store arbitrary artifacts (not just jars) — so that might be one such mechanism.

borkdude08:10:39

@rickmoynihan you can also put your binaries in a Github release and use bash. this is what I do for several projects, next to brew. if you want to keep those binaries private, you could store them in s3 behind some credentials and use bash.

borkdude08:10:59

fwiw brew also works for linux nowadays

plexus08:10:58

I feel old now. I'm happy with apt-get or tarballs. Still figuring out what "snap releases" are.

borkdude08:10:49

getting a binary into the apt-get system is way way harder than brew. I haven't been able to do this for clj-kondo

borkdude08:10:29

snap is a system by canonical which basically runs your binary in a protected/managed system where the user has to give explicit permission to access the filesystem, network etc.

plexus08:10:47

oohhh thanks 🙂

plexus08:10:02

yeah getting stuff into Debian/Ubuntu is far from trivial, big kudos for the people working on having Clojure and Leiningen in there. It all has its reasons but it's not something I want to navigate for fun. which is why I'm quite content with a tarball

borkdude08:10:01

I think arch linux might have a similar model to brew, it's much more accessible for people just wanting to share their binaries

rickmoynihan08:10:01

@borkdude indeed. It’s simple enough in principle it’s just a lot of of extra setup you need to do… each package manager etc requires its own installation in CI or locally in dev. They each need different keys / auth methods etc, on both the production and consumption side, and this needs to be repeated in each downstream project. It’d be nice if you only had to do that for one of them, and then everything could be resolved. i.e. leveraging maven/tools.deps for other artifacts.

rickmoynihan08:10:44

Though I also understand the desire to keep things simple. i.e. I’m not requesting this as tools.deps feature… it’d just be nice to leverage the same auth paths / transport mechanisms it uses.

borkdude08:10:03

@rickmoynihan this is what my brew repo looks like: https://github.com/borkdude/homebrew-brew I host several binaries there now

borkdude08:10:33

each binary is installable by just doing brew install borkdude/brew/clj-kondo etc

borkdude08:10:21

I don't need to push anything into the "main" brew package system, it's just a git repo

rickmoynihan08:10:04

Thanks. I’ll take a look.

borkdude08:10:13

I got this idea from @roman.bataev, so credits go to him 🙂

rickmoynihan08:10:57

Don’t get me wrong I think your setup is great… it’d just be nice if it were easier though; without having to involve external tooling.

borkdude08:10:52

Understood, don't have the answer to that 🙂

rickmoynihan08:10:26

I was just wondering if it would possible to essentially just put a tar ball in a maven repo; and unpack it out of the repo into a local ./bin directory. Then you could manage it with tools deps, and the addition of a single -A:install-executables alias… in theory.

borkdude08:10:04

I don't see why not. You can upload basically anything into a jar.

rickmoynihan08:10:26

it doesn’t even need to be a jar.

rickmoynihan08:10:49

You could put the statically linked binaries in their directly if you wanted.

rickmoynihan08:10:06

and perhaps have a separete repo for each platform you need to support.

rickmoynihan08:10:20

or use classifiers

borkdude08:10:37

yeah, that's what brew is also doing for mac and linux. so I'm just going for that right now

rickmoynihan08:10:23

which totally makes sense btw! I’m just thinking outloud about tooling I wish we had 🙂

rickmoynihan08:10:35

and how it might work.

borkdude08:10:08

it reminds me of https://github.com/magnars/optimus which bundles clj-v8 to run JavaScript

rickmoynihan08:10:36

interesting… though I guess that’s using v8 as a native lib rather than an executable

Dmytro Bunin14:10:53

Does anyone know if it’s possible to make a phraser for a spec, which has set as a predicate? Eg.

(s/def ::operator #{"a" "b"})

alexmiller14:10:01

what's a phraser?

Dmytro Bunin14:10:58

maybe not the best place to ask

jsa-aerial16:10:42

Re: phrase & expound. The key difference is that phrase is specifically intended to enable the construction of 'end user' level error messages while expound is specifically targeting developers. In it's niche/space, phrase is impressive.

Eduardo Mata18:10:45

Howdy y'all I am having trouble to troubleshoot a NullPointerException that is caused when a Clara Rule is fired on a Record. The record contains no null values, but it tells me it has null values

Eduardo Mata18:10:26

I don't want to post the whole long exception trace. I posted my issue in clara-rules github, https://github.com/cerner/clara-rules/issues/437

noisesmith18:10:42

record lookup can return null for a key that doesn't exist

Eduardo Mata18:10:26

Can you elaborate more?

noisesmith18:10:17

I can't help right now with the clara specific stuff - I know nothing about their DSL, but clojure lets you look up any key on a record, with a null value if the key isn't present

noisesmith18:10:11

also, as a nitpick, idiomatically _ means "a value that must be assigned but isn't used", so intentionally using it as a binding in as-> is confusing

noisesmith18:10:07

(update _ :volume #(/ v 42)) - I would expect an arity exception here - this function doesn't accept any args

Eduardo Mata18:10:53

My bad yes, I saw the typo

noisesmith18:10:18

what guarantee do you have that both rate and volume will be present and non-nil?

Eduardo Mata18:10:12

100% guarantee every entity passing through that rule will have rate and volume. They are added before even going through the rate

hiredman18:10:22

I would start by looking to see what is returned by (get @evnts "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12")

Eduardo Mata20:10:51

I will look in to this probably you are right

Eduardo Mata21:10:33

You were right. The Event was not in the atom @evnts. I can't really tell what happened.

Eduardo Mata21:10:05

However, now it is there and I still get the same error!

hiredman21:10:45

the question then is, is it always there, or is it just there when you are checking and not there when the rule runs

Eduardo Mata21:10:21

The event is not always there. But when the event is in the atom, the rule runs saying there is no event with that id.

hiredman22:10:06

are you sure it is the same id? if it is not always there, how do you know that rule only runs when it is there?

hiredman22:10:01

you pass in an immutable collection of events to clara to start with, and those are the events clara is basing its decisions on

hiredman22:10:08

not the events in your mutable atom

hiredman22:10:45

so if the event is in the immutable collection you pass in to start, then you remove it, things will break when that rule fires

hiredman22:10:27

which is why I was suggesting that having your atom off to the side was a bad idea. clara internally manages its own database of facts based on the initial stuff you pass in, and you should use that and manage it with assertions and retractions, the atom thing off to the side without anyway to keep it in sync with clara's view is a bad idea

Eduardo Mata22:10:49

So I noticed that the Event is there, with different id!

Eduardo Mata22:10:24

You are saying I should not use the atom at all?

Eduardo Mata22:10:45

How should I apply changes to my event? directly to the ?event?

hiredman22:10:09

I would ask in #clara my immediate thought is to retract the old event from clara's database and assert the new event, but that will actually cause problems because of clara's truth maintenance.

hiredman22:10:45

truth maintenance is something like, clara keeps track of which facts cause which new facts to be asserted, and if a fact is retracted all dependent facts are retracted as well

hiredman22:10:37

I think the architecture that makes the most sense for a rules engine program is something: 1. gather information about the world 2. feed that information in to the rules engine 3. the rules engine runs until it reaches a stable state and decides on some actions 4. query the rules engine for the actions, do them, and loop to 1

hiredman22:10:54

so instead of changing the properties of the event, you might assert some fact which describes the action of changing the event, then the rules engine exits, then you query its db for actions, then you perform those actions on the events, then you feed the new events into a new iteration of the rules engine

Eduardo Mata22:10:47

Wow, haha I have a hard time understanding what you just said. Maybe a visual example?

hiredman22:10:17

so like, instead of swaping or associng whatever directly manipulating the event, you have your rule assert a fact like (->Action event-id some-function-that-makes-the-change-you-want), then once the rules engine exits, you run a query on its database to return all the actions, you loop over all the actions applying the function to the event with the given id, and then you take your updated events and feed them back in to the rules engine again and you do that until you get no actions to run out of the rules engine

hiredman18:10:51

my guess is it returns nil

hiredman18:10:46

the confusing thing the exception shows an event with a volume and rate, but your rule ignores that event and uses its id to look up another value in @evnts and actually uses that

hiredman18:10:25

so likely that value doesn't have a volume or rate

hiredman18:10:52

so like, the some? condition on the rule means nothing

hiredman18:10:40

also to get knit picky, Event is a record, not a protocol

hiredman18:10:23

#clara isn't super busy but usually pretty helpful. I have only played with clara a little, so don't have too much experience, but I think you are asking for a bad time if you pass in an immutable collection of events, and then sort of ignore that and fiddle around with your atom of events

noisesmith18:10:28

that sounds right - also the fact this happens under load means this could be one of those "don't deref inside the thing already using the atom" situations - it could be a condition that happens when something failed a swap! and retried for example

hiredman18:10:59

it sounds like you are trying to multithread the thing too, which likely means you are running multiple independent clara rules engines all banging on the atom, and likely doing it in a none safe way

hiredman18:10:02

I suspect the best thing to do is to restructure your rules to not do anything, but generate some kind of job specifications of what is to be done, run the rules single threaded, and then take the job specs and do whatever parallel stuff you want to do

borkdude21:10:37

user=> (apply distinct? [1])
true
user=> (apply distinct? [])
Execution error (ArityException) at user/eval7 (REPL:1).
😕

alexmiller21:10:24

seems like that's what you should expect from the docstring

borkdude21:10:11

sure, but who reads docstrings... oh right, read the docs 😉

borkdude21:10:43

but it's kinda inconvenient that you should check for non-empty when using this with apply

seancorfield22:10:14

Given that (distinct []) works and produces () it does seem a little odd that (distinct?) is not allowed /cc @U064X3EF3

seancorfield22:10:24

(and the docstring talks about "no two of the arguments" but it already allows a single argument)

alexmiller22:10:48

Those seem like very different functions to me, but I would have no problem with a no-arg arity

seancorfield22:10:11

I must admit, I've never felt the need to check whether a bunch of things are distinct from each other so up until @borkdude mentioned it, I didn't actually know there was a distinct? testing function. And, from the name alone, I think I would have expected it to work on a collection anyway (distinct? coll) rather than (distinct? x y & more)

borkdude22:10:51

I haven't used distinct? that much, but it was used in a custom EDN reader which first checks if all keys are distinct before actually creating a hash-map. It crashed on the empty map 😕

borkdude22:10:05

One argument not to change it is that it's consistent with =. (apply = []) also excepts.

seancorfield22:10:44

Yeah, I think it's one of those edge cases where there are good arguments on both sides.

seancorfield22:10:18

You could argue that it shouldn't accept a single argument, given the docstring 🙂

yuhan02:10:29

I think one line of reasoning behind throwing an exception is that both the statements (apply distinct? []) and (apply = []) would be vacuously true

yuhan02:10:54

ie. any assertion you can make about the members of an empty set, the opposite assertion is also true - probably not a good situation to have

Brian21:10:43

I am using (json/read-str "{\":id\" : \"ebc7b483-1558-4b9c-8f8c-c9af3d1d37eb\"}") which returns a map that looks like {":id" "ebc7b483-1558-4b9c-8f8c-c9af3d1d37eb"}. How can I get that to be :id instead of ":id"? Should I manually (keyword ":id") afterwords? (I'd just pass in "id" instead of ":id" if I keyworded it manually)

seancorfield21:10:16

@brian.rogers Which library are you using for json? Most of them have an option to "keywordize" keys.

Brian21:10:26

Clojure.data.json

noisesmith21:10:45

that would give you ::id as a key

seancorfield21:10:47

Oh, no. I see what you mean. Well, ":id" is a string containing : already

seancorfield21:10:05

So, whatever produced that string is doing the wrong thing.

noisesmith21:10:12

you can use a custom key generating function

noisesmith21:10:36

you might want that string to be coerced to uuid as well

seancorfield21:10:05

I would expect you to start with "{\"id\" : \"...\"}" instead.

Brian21:10:46

Yes @seancorfield I think you've given me the solution. I will pass in "id" and then use (`clojure.walk.keywordize-keys ...)`

Brian21:10:55

Thank you!

Brian21:10:09

And I'll coerce that uuid after that =]

noisesmith21:10:39

there's an optional arg, you don't need to walk afterward https://github.com/clojure/data.json#converting-keyvalue-types

noisesmith21:10:52

(json/read-str "{\"a\":1,\"b\":2}"
               :key-fn keyword)
;;=> {:a 1, :b 2}

noisesmith21:10:15

and your :value-fn could check for uuids :D

Brian21:10:50

Oh wow! That's even more perfect thank you @noisesmith!