Fork me on GitHub
#announcements
<
2021-09-28
>
borkdude10:09:04

Announcing: neil https://github.com/babashka/neil A CLI which adds common features to your #clojure deps.edn projects. It's installable with Homebrew. If this is useful to you, I'd be happy to receive more features. Neil is made with #babashka :)

clojure-spin 20
gratitude 9
🔥 8
👍 4
leiningen 3
🎉 3
Karol Wójcik10:09:33

Would it be possibile to manage configs from graal-config with neil? Or it's not a scope of this tool?

borkdude10:09:15

Perhaps, haven't thought about it yet.

borkdude10:09:42

The first thing I'd like to fix is the formatting of the deps.edn which is a bit off.

eskos12:09:47

So the name is a donk at Lein(ingen)? In that context, is the envisioned scope to gather all the new low level things such as new build tools and deps.edn and combine them into an actual build integration tool through neil or is it just a way to have shorthands for aliases?

borkdude12:09:04

@U8SFC8HLP I plan to add the ability to inject tasks into a bb.edn and then you can use bb test to run tests in your project for example. This gives you some of the UX that lein has but still with the possibility to tweak things.

👍 2
borkdude12:09:40

but I want to make this optional so this tool can be used in a non-opinionated way

eskos12:09:27

Righto 🙂 This matters to me only in the sense that I want to understand where this whole uneaseness with Lein getting disfavored is going to go in the long run… I’m not so sure on the non-opinionated part, but that’s a separate discussion. Your goals for this sounds good 🙂

borkdude12:09:53

we'll see how it evolves :)

borkdude12:09:26

I find myself adding build.clj files to deps projects because depstar is deprecated so that fulfills the goal of this tool for me right now

borkdude12:09:51

for fun we could add neil test :only foo.bar which then invokes clojure -M:test -n foo.bar or so :P

borkdude12:09:13

I'm willing to receive a PR for that :P

Kevin12:09:24

I'd like to see neil being able to run tasks. Being able to manage (most) thinks within an easy to understand tool would also be good for beginners

Kevin12:09:59

clj cli just doesn't seem friendly enough (compared to lein)

metal 1
borkdude12:09:47

like neil test :only foo.bar ?

Kevin12:09:55

Yeah exactly

borkdude12:09:06

yeah I agree

borkdude12:09:11

PR welcome :)

Kevin12:09:29

I'll see if I have the time budget 😬

borkdude12:09:45

hehe no worries, no hurry

borkdude12:09:55

just collecting ideas is already valuable

Kevin12:09:01

Might want to think this through a bit more

Kevin12:09:25

Would be great to have configurable tasks (where the CLI tool shows you all options and such)

borkdude12:09:41

This is already what bb tasks is, have you used that?

Kevin12:09:53

hmm no I haven't

Kevin12:09:56

I'll look into that

lread13:09:00

Cool! You have a real skill in naming your projects!

Johan Thorén14:09:23

Naming is actually one of the hardest things in programming.

3
Steven Deobald17:09:22

Adds calendar invite for 2031. "See who remembers why this thing is called neil."

😄 1
jumar06:09:53

@U04V15CAJ I tried to install neil with

brew install babashka/brew/neil
but it failed:
==> Tapping babashka/brew
Cloning into '/usr/local/Homebrew/Library/Taps/babashka/homebrew-brew'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 44 (delta 7), reused 43 (delta 6), pack-reused 0
Receiving objects: 100% (44/44), 5.42 KiB | 1.81 MiB/s, done.
Resolving deltas: 100% (7/7), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
No available formula with the name "[email protected]". Did you mean [email protected] or [email protected]?
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
Expected to find class NeilAT008, but only found: Neil.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
No available formula with the name "[email protected]". Did you mean [email protected] or [email protected]?
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
Expected to find class NeilAT009, but only found: Neil.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/neil.template.rb
No available formula with the name "neil.template".
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/neil.template.rb
Expected to find class NeilTemplate, but only found: Neil.
Error: Cannot tap babashka/brew: invalid syntax in tap!

jumar06:09:12

I had an ancient version of babashka so tried to upgrade it just in case but neil install is still failing.

borkdude06:09:53

Weird. Can you make an issue? Perhaps you need to update brew as well?

jumar07:09:35

brew is typically auto-updated and an explicit update didn't help. I created https://github.com/babashka/neil/issues/5

jumar07:09:48

Btw. manual installation works fine.

borkdude07:09:41

Thanks. I'll look into if I can reproduce it. Can you also mention the brew version, macos or linux version?

borkdude09:09:43

@U06BE1L6T Something else was wrong, now also fixed

borkdude10:09:00

And added another feature:

$ neil add dep borkdude/sci :latest-sha true
$ cat deps.edn
{:deps {borkdude/sci {:git/url ""
                      :git/sha "a0dd7fa2a087ed1f524fd22cb73139f571ff469d"}}
 :aliases {}}

Karol Wójcik10:09:04

Does this feature works somehow with our graal-config? 😄

Karol Wójcik10:09:15

If yes we can put the usage in the description : )

borkdude10:09:35

@UJ1339K2B Hmm, we could add neil add dep clj-easy/graal-config :deps/root taoensso.timbre

Karol Wójcik10:09:19

Oh, yes pleeease ❤️

Karol Wójcik10:09:19

But probably we have do differentiate the graal-config from the regular dependencies, since it's multi artifact deps.edn monorepo

borkdude10:09:00

we have to make a separate command for it :)

borkdude10:09:28

neil add graal-config taoensso.timbre

borkdude10:09:44

but we also support :as com.github.clj-easy/graal-config-cheshire

borkdude10:09:18

or wait no you can just use that name

borkdude10:09:40

no that won't work

borkdude10:09:48

so :as should be good

Karol Wójcik13:09:37

I have tried:

neil add dep com.github.clj-easy/graal-config :deps/root config/com.taoensso/nippy :as com.github.clj-easy/graal-config-nippy :latest-sha
But this add a dependency with {:mvn/version nil}

borkdude13:09:30

Well this option isn’t implemented yet. Also you should provide true to latest sha

borkdude14:09:37

I'll work on this now

borkdude14:09:28

@UJ1339K2B Should work now:

$ neil add dep com.github.clj-easy/graal-config :deps/root config/com.taoensso/nippy :as com.github.clj-easy/graal-config-nippy :latest-sha true
$ cat deps.edn
{:deps {com.github.clj-easy/graal-config-nippy {:git/url ""
                                                :git/sha "7a2cc5b1b1652580e26d990cf5bac2ed48247b7e"
                                                :deps/root "config/com.taoensso/nippy"}}
 :aliases {}}
in neil v0.0.11

borkdude14:09:34

oops one bug

borkdude14:09:53

the git/url isn't correct

Karol Wójcik14:09:19

But, we almost got it 😄

borkdude14:09:21

Should be fixed in v0.0.12

eskos15:09:06

This is so minor irrelevance but wouldn't :sha/latest read better than :latest-sha :)

eskos15:09:27

Dunno which part would even be responsible of that in this stack...

borkdude15:09:25

I'm not sure if all options should be namespaced keywords ;)

borkdude15:09:36

git/sha, sha/git, ...

vncz16:10:41

Question: did you think about integrating this as a Clojure tool?

borkdude16:10:20

@U015VUATAVC I have considered this, but one of the goals of neil was to see if it's feasible to package a bb script as a standalone brew/scoop/nixos package

borkdude16:10:46

I might make a tool version of this later on. But one of the benefits of having it as a bb script is that it's fast. I don't want to wait a few seconds to add something to my deps.edn

Kelvin21:09:37

At Yet Analytics, we've recently open-sourced a core product written in Clojure - lrsql (aka SQL LRS). This project uses HugSql to interface with three different SQL backends (H2, SQLite, and Postgres) to store learning data. As part of the lrsql development process, we released colossal-squuid, an open-source Clojure(script) library that generates sequential UUIDs (SQUUIDs) which follow the https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format. We use SQUUIDs in lrsql as primary key IDs, where their strict monotonicity makes them useful as query cursors. lrsql repo: https://github.com/yetanalytics/lrsql colossal-squuid repo: https://github.com/yetanalytics/colossal-squuid

catjam 14
🚀 10
😻 1
🎉 3
Ivar Refsdal09:09:12

Great idea and project! It will surely be useful for the company where I wok. I submitted an issue now that proves and proposes a fix for a race condition.

pithyless07:09:33

@U02FU7RMG8M are you familiar with mulog's flake [0] and if so, can you share some insight on pros/cons of choosing one library over the other? Asking as someone who has not studied the code for colossal-squuid yet, so feel free to just respond "RTFM". :) [0] https://github.com/BrunoBonacci/mulog/blob/master/mulog-core/java/com/brunobonacci/mulog/core/Flake.java

Kelvin15:09:48

Never heard of it before so I'm reading through it, and two three things stood out: • Flake.java is part of a larger library, which in turn has a bunch of dependencies. Not only is colossal-squuid one library dedicated to generating SQUUIDs, I designed it to not have any dependencies at all (Clojure(script) and test deps excepted). • The IDs Flake.java generates are not UUIDs, which may not be optimal for applications that prefer UUIDs (e.g. certain DBMSs like Postgres have special optimized UUID types). • colossal-squuid has Clojurescript/JS support, while Flake.java is for, well, Java only.

👍 1
Kelvin17:09:42

One con of colossal-squuid vs Flake.java is performance; these are times measured on Java 11.0.10 on my machine (MacOS High Sierra with an i7 processor):

(criterium.core/bench (squuid/generate-squuid))

Evaluation count : 245862600 in 60 samples of 4097710 calls.
             Execution time mean : 247.688222 ns
    Execution time std-deviation : 16.444680 ns
   Execution time lower quantile : 235.489134 ns ( 2.5%)
   Execution time upper quantile : 284.262079 ns (97.5%)
                   Overhead used : 6.380512 ns

Found 3 outliers in 60 samples (5.0000 %)
	low-severe	 2 (3.3333 %)
	low-mild	 1 (1.6667 %)
 Variance from outliers : 50.0896 % Variance is severely inflated by outliers

(criterium.core/bench (str (squuid/generate-squuid)))

Evaluation count : 214467540 in 60 samples of 3574459 calls.
             Execution time mean : 289.453769 ns
    Execution time std-deviation : 21.778366 ns
   Execution time lower quantile : 273.336325 ns ( 2.5%)
   Execution time upper quantile : 340.385667 ns (97.5%)
                   Overhead used : 6.380512 ns

Found 8 outliers in 60 samples (13.3333 %)
	low-severe	 3 (5.0000 %)
	low-mild	 5 (8.3333 %)
 Variance from outliers : 56.7680 % Variance is severely inflated by outliers
So generating a SQUUID is about 7 times slower than generating a Flake, and with string conversion it's about 3.5 4 times slower.

🙏 1
Kelvin17:09:57

For purposes of fair comparison, I rebenched Flake generation on my machine. Generating flakes w/o string conversion was similar to the listed time (35 ns), but w/ string conversion it was better (69 ns).

Ivar Refsdal06:10:31

I will chip in with a small observation: Flake provides monotonicity per thread, whereas colossal provides monotonicity per JVM. I doubt that makes a big difference in practice for an application, but I find the latter neater. I'm no performance expert, but I suppose that (synchronization guarantees) might also explain some of the performance differences. Side note: μ/log looks like a great and interesting project. Another option for squuids can be the datomic api. Downsides: closed source. No easy way to get from millis to squuid as far as I know. https://docs.datomic.com/on-prem/clojure/index.html#datomic.api/squuid

👍 1
pithyless16:10:59

Thank you all for the observations, this is exactly what I was hoping for. I was considering porting flake to CLJS, so colossal library arrives on the scene at a great time. :) (Datomic - and DataScript - both have a squuid function, but unlike flake and colossal, they make no attempt to be monotonic).

❤️ 2
pithyless16:10:39

Thanks @U02FU7RMG8M and @UGJE0MM0W for following up! gratitude

zeitstein11:12:46

@U02FU7RMG8M Is there a way to obtain the timestamp from an already generated squuid?

Kelvin14:12:50

Here's how you can do it in Clojure:

(let [squuid (generate-squuid)
      squuid-fst-48-bits (bit-shift-right (.getMostSignificantBits squuid) 16)]
  (java.util.Date. squuid-fst-48-bits))
It's just a matter of extracting the first 48 bits and coercing them into a timestamp. For future reference, generate-squuid* returns a map that includes the timestamp used for generation for you.

zeitstein14:12:20

Could I trouble you for the CLJS version? Sorry, really unfamiliar with bit manipulation 😕

Kelvin19:12:42

Unfortunately it'll be harder in CLJS since JavaScript UUIDs are stored as strings, so you'll have to do string parsing before you can do anything interesting with them.

gratitude 1
Kelvin22:12:41

I'm pretty sure you can hack up a solution by using a tool like https://github.com/domske/uuid-tool to parse the UUID into a number array, then perform array manipulations and arithmetic to get an integer that you can convert into a Date object. But it would be ugly.

Kelvin22:12:40

Anyways since you did mention you were unfamiliar with bit manipulation, let me explain what's going on in that code snippet: 1. .getMostSignificantBits simply takes the "most significant" half of a UUID, i.e. the ones with the greatest value. (Think of how in the number `12`, the `1` is the "most significant" digit while the `2` is the "least significant" one.) For example, the most significant bits of the UUID `017ddefd-951d-8485-a6cf-6820da6769c3` are `017ddefd-951d-8485`. 2. bit-shift-right literally shifts bits in a bit array to the right; the `16` refers to by how many bits they were shifted. For example, in the bit array `0000111100001111`, if we shift by 4 bits, will become `000011110000`, since the last 4 bits were shifted "off the map." Likewise, bit shifting `017ddefd-951d-8485` will result in `017ddefd-951d` (note that each hex digit is 4 bits long, since that's how many bits are needed to represent any number between `0` and `f`/`15`). 3. The `java.util.Date` has a constructor that accepts a long value (which all Clojure ints are), so if given 017ddefd-951d it instantiates a Date instance with value `2021-12-21T21:55:00.509-00:00`.

gratitude 1
zeitstein14:12:42

Thank you so much, really kind of you!

zeitstein15:12:22

Alright, this seems to do it:

(-> (squuid/generate-squuid)
      str
      (subs 0 13)
      (clojure.string/split "-")
      (clojure.string/join)
      (js/parseInt 16)
      (js/Date.))
Thank you, @U02FU7RMG8M! Are you interested in offering such a function in the library?

Kelvin15:12:57

Perhaps. I'll have to think about it since I'd want to keep colossal-squuid minimalistic, but if people would like such a function I'd do it. Also, I like how for your cljs solution you didn't use any external libs - you kept the spirit of colossal-squuid being dep free!

🙂 1
zeitstein15:12:09

It's simple enough to write on your own (having seen it), but the implementation would be several lines (re: minimalism). Datomic/Datascript include such a function, which speaks to demand, perhaps?

zeitstein16:12:16

Would be happy to provide a PR, if you'd like.