This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-15
Channels
- # announcements (20)
- # babashka (281)
- # beginners (13)
- # biff (8)
- # calva (20)
- # cider (5)
- # clj-commons (1)
- # clojure (46)
- # clojure-boston (1)
- # clojure-europe (6)
- # clojure-losangeles (24)
- # clojuredesign-podcast (3)
- # clojurescript (12)
- # datomic (1)
- # events (1)
- # fulcro (12)
- # guix (2)
- # honeysql (1)
- # integrant (1)
- # introduce-yourself (1)
- # rdf (16)
- # reagent (3)
- # reitit (14)
- # releases (3)
- # sci (28)
- # shadow-cljs (122)
- # specter (1)
- # tools-deps (10)
- # xtdb (6)
yay! got more schema working now! https://twitter.com/borkdude/status/1525818490485096448
If you wanna try:
$ bash <(curl ) --version 0.8.3-SNAPSHOT --dir .
$ ./bb schema.clj
(ns schema)
(require '[babashka.deps :as deps])
(deps/add-deps '{:deps {prismatic/schema {:mvn/version "1.2.1"}}})
(require '[schema.core :as s])
(def MySchema {:a [s/Int]})
(prn (s/validate MySchema {:a [1 2 3]}))
(try (s/validate MySchema {:a [:foo]})
(catch Exception e
(prn (:error (ex-data e)))))
Very nice! I'm going through the schema code to see if I can improve the bb support from there https://github.com/plumatic/schema/pull/440/files
@U055XFK8V Pushed a fix for record (non-)equivalence to babashka master now. So I think you can remove the workaround in schema
@U055XFK8V macOS binary should be installable now with the same invocation as earlier
I'm going to check out your branch, I lost my earlier work where I got the tests running :(
the defrecord fix works I think! currently investigating a locals clearing issue. something to do with the infinite-arity-fn-test
test
Here's a witness for the locals clearing (I think) https://github.com/plumatic/schema/pull/440/files#diff-fe966952e8b139238a859638669987516020fd5f9520243788d79a1987723e3dR257
./bb test
ERROR in (infinite-arity-fn-test) (/Users/ambrose/Projects/schema-local-dev/bb-test-suite/test/cljc/schema/core_test.cljc:)
expected: (= 5 (f 4))
actual: java.lang.AssertionError: Assert failed: input-checker-sym local cleared!!!
G__5086
at java.lang.reflect.Constructor.newInstance (Constructor.java:490)
sci.impl.Reflector.invokeConstructor (Reflector.java:310)
sci.impl.interop$invoke_constructor.invokeStatic (interop.cljc:57)
sci.impl.analyzer$analyze_new$reify__8934.eval (analyzer.cljc:880)
sci.impl.analyzer$analyze_throw$reify__8890.eval (analyzer.cljc:777)
sci.impl.analyzer$return_if$reify__8849.eval (analyzer.cljc:669)
sci.impl.analyzer$return_do$reify__8163.eval (analyzer.cljc:136)
....
Reprod in CI: https://github.com/plumatic/schema/runs/6493118067?check_suite_focus=true#step:9:19Oh, interesting I get this error on JVM: Error: Exception in thread "main" java.lang.RuntimeException: Conditional read not allowed, compiling:(schema/macros.clj:243:72)
Looks like bb is more lenient?
Settled on checking babashka.version
🙂 if I changed it to .cljc, cljs tries to compile it...
I don't even understand what the issue is, do you have a tiny repro of that, or is it not important?
Not yet, tried several times. Basic issue seems to be local
being cleared in code like (let [local (delay)] (fn [] ... @local))
but I can't find a minimal reprod.
In particular local
is nil at @local
and throws a NPE. Assuming it's related to locals clearing.
Thanks, this is really good for improving SCI. About foo.bar` resolving to
user/foo.bar
, that's interesting. Apparently (def foo.bar 1)
is allowed, but maybe just by accident.
Ah I see, so:
Symbols containing / or . are said to be 'qualified'.
This is why no further resolution takes place probablyThis might be the reprod:
./bb
Babashka v0.8.3-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
user=> (defn minimal []
(let [f (let [cleared 1]
(fn ([a] (inc cleared))
([a & bs])))]
(assert (= 5 (f 4)))))
#'user/minimal
user=> (minimal)
java.lang.NullPointerException [at <repl>:3:22]
Smaller:
user=> ((let [cleared 1]
(fn ([] (assert cleared))
([& bs]))))
java.lang.AssertionError: Assert failed: cleared [at <repl>:1:1]
Interestingly aliases can contain dots:
user=> (require '[clojure.set :as c.set])
nil
user=> `(c.set/union )
(clojure.set/union)
Yes, I'm guessing the thinking is that it might be an unresolved reference to a class.
OK, the reader issue is pushed to SCI and bb now. The rest I'll look into later this week
It seems:
Ran 95 tests containing 506 assertions.
2 failures, 0 errors.
only 2 failures left, not sure what those are@U055XFK8V Exciting stuff. Let me know if there's anything left to do. If not, maybe we can launch a new bb soon which will be able to run schema!
I guess we could wait for the PR to be merged and a new version of schema to be released
😄 Yes, I'll work on the PR. I'm a maintainer on schema now, should be lower friction.
Another problem related to marker protocols: https://github.com/babashka/sci/issues/743
Another issue with syntax quote https://github.com/babashka/sci/issues/744
It's an edge case, records aren't really classes in SCI, since SCI cannot create new classes, so those things are implemented differently
but these are a bit implementation-detail-ish, I don't know if I might change this in the future
yeah, that works:
user-foo=> (defrecord Foo [])
#'user-foo/Foo
user-foo=> user_foo.Foo
user-foo/Foo
I don't think that'll work... since then I would have to create something of type Class that can give a different name
(not (= (record schema.core_test.Foo {:x Any, (optional-key :y) Int})
(record schema.core-test/Foo {:x Any, (optional-key :y) Int})))
Why is the var #'Foo
needed here? Could we just return a symbol 'schema.core_test.Foo
from defrecord and class references like Foo
?
Is defrecord
returning #'Foo
instead of 'Foo
an oversight? I was hoping for a general pattern like "record classes in bb are like clj, except sometimes they are symbols". Here it's a var.
Might be worth testing how symbol resolution works on a record that's imported from another ns.
Tried it, syntax quoting it doesn't work, resolves in the current ns on an imported record.
Double-checked:
$ clojure -M:babashka/dev -e "(defrecord Foo []) (= Foo 'user.Foo)"
true
nice, that fixed one test. another popped up due to Explainer =>
schema.core-test/Explainer`
It should be fixed by the current build anyway https://github.com/babashka/babashka/commit/6089496d9c8c1a58941c884d8ebc715382412bf9
The test fancy-explain-test
started failing. I think it's because the left side of the (is =
uses class resolution for the record class name, and the right side uses syntax quote. So was "working" before you "broke" one side. Just a coincidence, if you'd have fixed both problems at once, it would have passed.
there's a problem with the pprint changes (just uncommented the test), getting a SOE. investigating..
maybe you can make sense of the trace https://github.com/plumatic/schema/runs/6514340379?check_suite_focus=true#step:9:21
@U055XFK8V Will continue tomorrow, I'm out of dev budget for today :)
@U055XFK8V All solved now, I think
Just tried it, SOE is fixed but:
FAIL in (pprint-test) (/Users/ambrose/Projects/schema-local-dev/bb-test-suite/test/cljc/schema/core_test.cljc:)
expected: (= "(maybe Int)" (str/trim (with-out-str (pprint/pprint (s/maybe s/Int)))))
actual: (not (= "(maybe Int)" "{:schema\n {:p?\n #object[clojure.core$integer_QMARK_ 0x2104eb01 \"[email protected]\"],\n :pred-name integer?}}"))
So somehow the print-method isn't correctly registered... wonder where that's defined
So this example works:
(defrecord Foo [x])
(require '[clojure.pprint :refer [simple-dispatch]])
(defn print-awesome [_foo]
(println "<<<<foo>>>>"))
(defmethod simple-dispatch Foo [o]
(print-awesome o))
(clojure.pprint/pprint (->Foo 1))
This works too:
(defrecord Foo [x])
(require '[clojure.pprint :as pprint])
(clojure.core/defmethod pprint/simple-dispatch Foo [_]
(pprint/write-out "<<<<foo>>>>"))
(clojure.pprint/pprint (->Foo 1))
./bb
Babashka v0.8.3-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
user=> (defprotocol P)
#'user/P
user=> (defrecord R [])
user.R
user=> (defmethod clojure.pprint/simple-dispatch user.P [_] (println "PRETTY"))
#object[clojure.lang.MultiFn 0x6c3bc635 "[email protected]"]
user=> (clojure.pprint/pprint (R.))
{}
nil
Right! I was just trying:
(ns foo)
(defrecord Foo [x])
(ns bar
(:require [clojure.pprint :as pprint]))
(clojure.core/defmethod pprint/simple-dispatch foo.Foo [_]
(pprint/write-out "<<<<foo>>>>"))
(clojure.pprint/pprint (foo.Foo. 1))
but that did workThis still works for me:
(ns user
(:require [clojure.pprint :as pprint]))
(defrecord Foo [x])
(clojure.core/defmethod pprint/simple-dispatch user.Foo [_]
(pprint/write-out "<<<<foo>>>>"))
(clojure.pprint/pprint (->Foo 1))
./bb
Babashka v0.8.3-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
user=> (defprotocol P)
#'user/P
user=> (defrecord R [] P)
user.R
user=> (defmethod clojure.pprint/simple-dispatch user.P [_] (println "PRETTY"))
#object[clojure.lang.MultiFn 0x6c3bc635 "[email protected]"]
user=> (clojure.pprint/pprint (R.))
{}
nil
clj
Clojure 1.11.1
user=> (require 'clojure.pprint)
nil
user=> (defprotocol P)
P
user=> (defrecord R [] P)
user.R
user=> (defmethod clojure.pprint/simple-dispatch user.P [_] (println "PRETTY"))
#object[clojure.lang.MultiFn 0x7cedfa63 "[email protected]"]
user=> (clojure.pprint/pprint (R.))
Execution error (IllegalArgumentException) at user/eval171 (REPL:1).
Multiple methods in multimethod 'simple-dispatch' match dispatch value: class user.R -> interface user.P and interface clojure.lang.IPersistentMap, and neither is preferred
Right, so :
(ns user
(:require [clojure.pprint :as pprint]))
(defprotocol Schema)
(defrecord ASchema []
Schema)
(defrecord Foo [x])
(defmethod clojure.pprint/simple-dispatch user.Schema [_] (println "PRETTY"))
(prefer-method clojure.pprint/simple-dispatch user.Schema clojure.lang.IRecord)
(prefer-method clojure.pprint/simple-dispatch user.Schema java.util.Map)
(prefer-method clojure.pprint/simple-dispatch user.Schema clojure.lang.IPersistentMap)
(clojure.pprint/pprint (ASchema.))
Oh, so a simple-dispatch on a protocol... damn, I only supported it for a record now I think ;)
the same logic is used for print-method
, is there anything special in sci to handle that?
https://github.com/plumatic/schema/blob/1116d1c68347b980ca03a7d4458213b4e13323ae/src/cljc/schema/core.cljc#L123-L124
yes, protocols aren't really types in SCI, because again, SCI cannot really create new JVM types
I added another test, this branch isn't being hit either https://github.com/plumatic/schema/pull/440/commits/e79497a4dc32171129b189572d1088292ae18270
FAIL in (print-method-test) (/Users/ambrose/Projects/schema-local-dev/bb-test-suite/test/cljc/schema/core_test.cljc:)
expected: (= "(maybe Int)" (str/trim (with-out-str (println (s/maybe s/Int)))))
actual: (not (= "(maybe Int)" "#schema.core.Maybe{:schema #schema.core.Predicate{:p? #object[clojure.core$integer_QMARK_ 0x2104eb01 [email protected]], :pred-name integer?}}"))
isa?
might be the root cause https://github.com/babashka/sci/issues/750#issuecomment-1133075809
Why doesn't this print "P"?
(ns dude
(:require [clojure.pprint :as pprint :refer [simple-dispatch]]))
(defprotocol P)
(defrecord R [] P)
(defmethod simple-dispatch dude.P [_]
(println "P"))
(prefer-method simple-dispatch dude.P clojure.lang.IRecord)
(prefer-method simple-dispatch dude.P java.util.Map)
(prefer-method simple-dispatch dude.P clojure.lang.IPersistentMap)
(prefer-method simple-dispatch dude.P java.lang.Long)
(extend-protocol P
java.lang.Long
(explain [_] "yo"))
(pprint/pprint 1)
I think it's a bad idea to change the pprint-ing of types you don't own anyway. It seems schema has to attach the pprint-method to something in order to trigger explain. And that something seems to be records.... right?
So would it be sufficient if (for now) if SCI made simple-dispatch + protocol work only for records that implement that protocol?
I guess a workaround for the current version would be to do:
(defmethod simple-dispatch Maybe [x] (pprint/write-out (explain x)))
which would be a bit tedious but it would work and you could probably remove the prefer-methodsE.g. this works as is in clojure and bb (master) without prefer-methods:
(require '[clojure.pprint :as pprint :refer [simple-dispatch]])
(defprotocol Schema
(explain [_]))
(defrecord Maybe []
Schema
(explain [_]
"M"))
(defmethod simple-dispatch Maybe [x] (pprint/write-out (explain x)))
(pprint/pprint (->Maybe))
> Why doesn't this print "P"?
I believe it's because isa?
uses Class/isAssignableFrom
, not clojure.core/extends?
.
> So would it be sufficient if (for now) if SCI made simple-dispatch + protocol work only for records that implement that protocol? That's a good start yes. I can manually add that code in schema.
You would only have to add code as of now, not if SCI would support the "good start", then it would work as is, I think?
Your proposal sounds perfect, I can't think of any other cases that are important tbh?
So to recap: 1. schema could make it work with bb master by adding the explicit record print-methods 2. SCI could potentially fix protocol printing for records, then you would not have to change anything
I will try 2 but not sure how easy it is, but we'd still have 1 as a fallback and it would not be incorrect to add it
I'm going to try 1), I think this approach is needed for CLJS support for these two multimethods (currently the extensions are #?(:clj ..)
.
yep exactly. Only important for libs that provide new schemas. Probably can revisit this if we find another library that needs this pattern.
Right, so in CLJS:
(require '[clojure.pprint :as pprint])
(defprotocol Dude
(explain [_]))
(extend-protocol IPrintWithWriter
Dude
(-pr-writer
[_this w _]
(.write w "hello")))
(defrecord DudeR []
Dude)
(pprint/pprint (->DudeR))
you would still have to provide the specific implementation for IPrintWithWriter
on the record, each time, right?looks like print-method doesn't exist in cljs and cljs.pprint/simple-dispatch is basically a closed system..
so it'll be sci-specific for now, but potentially useful for cljs if those restrictions are lifted.
What about extending IPrintWithWriter
on each record, or does schema already do that?
for pprint, I found this: https://stackoverflow.com/a/15183440/6264
I guess so:
cljs.user=> (cljs.pprint/pprint (s/maybe s/Int))
{:schema {:p? #object[cljs$core$integer_QMARK_], :pred-name integer?}}
At least the bb version won't be behind CLJS :)I guess people could just use:
(str (s/explain (s/maybe s/Int)))
"(maybe Int)"
if they wanted the pretty oneyeah, no cljs printing support at all except explain
. thanks, I'll use those two approaches!
for pprint: cljs.pprint/simple-dispatch uses a fixed dispatch table, I don't think the same approach works as clojure.pprint/simple-dispatch. Can't see any handy alternatives.
the good news is that the bb support is almost done, but I can't figure out why IPrintWithWriter doesn't work in cljs:
./bb test
....
FAIL in (simple-validated-defn-test) (/Users/ambrose/Projects/schema-local-dev/bb-test-suite/test/cljc/schema/core_test.cljc:)
expected: (= +simple-validated-defn-schema+ (s/fn-schema simple-validated-defn))
actual: (not (= (=> (both Str (pred odd-str?)) (both (pred odd?) long)) (=> (=> (maybe (=> Any Any)) (protocol s/Schema)) [(=> (maybe (=> Any Any)) (protocol s/Schema))])))
FAIL in (simple-primitive-validated-defn-test) (/Users/ambrose/Projects/schema-local-dev/bb-test-suite/test/cljc/schema/core_test.cljc:)
expected: (= +primitive-validated-defn-schema+ (s/fn-schema primitive-validated-defn))
actual: (not (= (=> long (both (pred odd?) long)) (=> (=> (maybe (=> Any Any)) (protocol s/Schema)) [(=> (maybe (=> Any Any)) (protocol s/Schema))])))
....
Ran 113 tests containing 587 assertions.
2 failures, 0 errors.
@U055XFK8V Looking good. But now what, still failing tests for bb? ;)
LOL yes, the 2 remaining tests looked a real pain to debug without pretty printing, so they've been failing this whole time.
we've been making progress, I kept uncommenting tests and we've been knocking them over 🙂
Ok I think the problem is that schema holds a global registry of Class => schema to lookup defn schemas. I assume all defn's in bb are the same class and every defn just clobbers each other. This is probably the most fragile part of schema IMO.
I think I can just use the cljs approach in bb, which is to associate fn identity with a schema instead of its class.
./bb test
Testing schema.core-test
Testing schema.macros-test
Testing schema.coerce-test
Testing schema.experimental.abstract-map-test
Testing schema.test-test
Testing schema.utils-test
Ran 113 tests containing 587 assertions.
0 failures, 0 errors.
So, how reliant is schema on record "classes" being a symbol etc in bb? I'll be back tomorrow, need to sleep now
it may be impossible actually, the map implementation takes over. I'll just comment it out.
do a grep for :bb
in my branch, you'll see some class?
calls broadened to (some-fn class? symbol?)
. it's not too bad.
I wonder if we could/should make SCI/bb user code more resilient against this implementation choice. What if we want to move the record representation from a Symbol to a deftype
later on for example
I guess a more cross-platform way would be to do:
(-> (str klazz) (str/split #" ") last)
to get the class name for both Java classes and for SCI recordsI guess SCI could offer an API to interrogate this, like:
sci.core/record-class?
or:
babashka.core/record-type?
and
babashka.core/type-name
which either gives the class name or the record name as a string?I'm trying to implement type-name
now...
(defn type-name [x]
(if (and (symbol? x)
(-> x meta :sci.impl/record))
(str x)
#?(:clj (class x))
#?(:cljs (.-name x))))
maybe nil
should be interpreted as: don't know, you're on your own. it seems getting the name from (defrecord Foo [])
is impossible in CLJS right now
Perhaps record-type-name
is a better API name and then I'll only give back a string for records and nil for other things
user=> (sci/eval-string "(defrecord Foo []) (sci.runtime/record-type? Foo)")
true
user=> (sci/eval-string "(defrecord Foo []) (sci.runtime/record-type-name Foo)")
"user.Foo"
> but how do I do this in CLJS properly, I wonder It looks like schema just gives up in this case and returns the entire prototype https://github.com/plumatic/schema/blob/1116d1c68347b980ca03a7d4458213b4e13323ae/src/cljc/schema/core.cljc#L199
> Let me know if that works for you
it feels like this abstraction could be a bit cleaner. it seems too tied to records, I think I want an abstraction over
1. "is this a fake class/prototype created due to limitations of sci" and
2. "what is the name of this class/prototype, if any?".
could we reify these metadata symbols as records themselves? like provide a sci.runtime.Type
record that responds to a type-name
fn that returns a nilable string. That way if you decide to fake classes for other things (definterface, defprotocol, deftype etc), you can build on top of Type
without requiring changes in schema. and anywhere you normally do (extend-protocol ... Class)
you can do (extend-protocol ... sci.runtime.Type)
also, without clashing with existing (extend-protocol ... Symbol)
.
Even if I add
;; see
;; pprint does not order between IPersistentMap and IDeref
(prefer-method pprint/simple-dispatch clojure.lang.IPersistentMap clojure.lang.IDeref)
I still get
: Multiple methods in multimethod 'simple-dispatch' match dispatch value: class babashka.process.Process -> interface clojure.lang.IPersistentMap and interface clojure.lang.IDeref, and neither is preferred task.util
when running process
@i There is now a new namespace for this. Add (require '[babashka.process.pprint])
and it should solve the problem
Hello babashka channel 👋 I'm trying to write a small tool that takes user input twice. The second time I try to read from stdin fails because the stream is closed. I think I'm missing something obvious ^
This may give you some ideas: https://gist.github.com/borkdude/82dcdd36a1e61ef36f19221876e7b1b6
Thanks, that's perfect! Out of curiosity what am I'm doing wrong below. task below waits for fzf input, but then does not wait for read-line input:
;; Invoked with `find * -type f | bb some-task`
some-task {:requires ([babashka.process :as p])
:task
(let [fzf (p/process ["fzf" "-m" "--print-query"]
{:in :inherit
:err :inherit
:out :string})
proc (:proc fzf)
pick (:out (deref fzf))
_ (prn "pick result" pick)
_ (do (print "input: ") (flush))
read-input (read-line)]
(prn "read-line-input: " read-input))}
So this works:
{:tasks {some-task {:requires ([babashka.process :as p]
[babashka.fs :as fs]
[clojure.string :as str])
:task
(let [fzf (p/process ["fzf" "-m" "--print-query"]
{:in (str/join "\n"
(map str (remove fs/directory? (fs/list-dir ""))))
:err :inherit
:out :string})
proc (:proc fzf)
pick (:out (deref fzf))
_ (prn "pick result" pick)
_ (do (print "input: ") (flush))
read-input (read-line)]
)}}}
I think you could do the input argument to fzf via a command line args rather than via stdin
I need to allow the user to use the fuzzy finding interface of fzf ... where they type and see how the list filters. (as opposed to passing a string before hand via command-line-args
How do I use the medley dependency on a machine which has no internet connectivity? I have transferred the medley jar manually to my project/deps folder on machine annd used :
{:paths ["src"] :deps {medley/medley {:local/root "deps/medley-1.4.0.jar"}
But still when I run bb -m foo.bar, I get the error:
Error building classpath. Failed to read artifact descriptor for org.clojure.jar:1.11.0 .
Note that I also previously manually transferred clojure tools v1.11.0 to the machine as without it, there was another error message for clojure tools being missing.
@mukundzare This is because you're still going through the tools deps machinery to fabricate a classpath which probably tries to pull in clojure 1.11 somehow. I think for no internet connection it's best to make an uberjar and use the --classpath uberjar.jar
option
You can make such an uberjar with bb uberjar uberjar.jar
on a machine which does have internet
Alright, I will try that, thanks!
I always get confused by a few of these conventions, but I will try and sort through them
Like how does --classpath differ from :paths and :deps in the bb.edn
And why is there an additional BABASHKA_CLASSPATH variable on top of it?
:paths
and :deps
are similar to how you use them in deps.edn
. When you invoke Clojure, it will calculate the classpath from them, which, if you haven't done that before, usually needs the internet.
Those options were introduced a while after we already had --classpath
and BABASHKA_CLASSPATH
: before you had to manually construct the classpath using $(clojure -Spath)
The environment variable allows you to invoke bb as you would do it locally as when you still had internet and plays nice with things like Dockerfiles.
All of this is also documented in https://book.babashka.org/
Now it's quite clear to me! Thanks a lot.
I refer the babashka book multiple times a week but I sometimes get quite stumped by a few of these concepts even after reading and re-reading them, it's great to have your guidance in this slack channel @borkdude 😀
@mukundzare You're welcome. I did not mean to say RTFM, I'm happy to shed some extra light on this :)
I like your humility. I might be posting here more often.👍