This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-13
Channels
- # aleph (3)
- # announcements (2)
- # aws (48)
- # babashka (93)
- # beginners (101)
- # bristol-clojurians (1)
- # cider (3)
- # clj-kondo (17)
- # cljdoc (1)
- # cljsrn (3)
- # clojure (208)
- # clojure-dev (2)
- # clojure-europe (19)
- # clojure-italy (18)
- # clojure-losangeles (16)
- # clojure-nl (8)
- # clojure-spec (21)
- # clojure-sweden (8)
- # clojure-uk (57)
- # clojuredesign-podcast (6)
- # clojurescript (10)
- # code-reviews (6)
- # core-typed (1)
- # cryogen (7)
- # cursive (38)
- # datomic (34)
- # duct (13)
- # emacs (13)
- # fulcro (16)
- # funcool (2)
- # graalvm (1)
- # lambdaisland (5)
- # luminus (8)
- # lumo (1)
- # malli (2)
- # off-topic (12)
- # pathom (9)
- # re-frame (13)
- # reagent (11)
- # ring (3)
- # shadow-cljs (15)
- # sql (19)
- # tools-deps (61)
- # xtdb (23)
is there a way to list the signatures of a protocol?
What are signatures in protocol?
Maybe not what some people would call "short", but very useful.
Here is a fun quote from Rich Hickey's talk in 2008 "Clojure Concurrency": "So that is the first: hard to understand, test, and reason about. But if you put concurrency in the mix, there is no doubt about it. Mutable objects are a disaster. They are a complete catastrophe, and I am going to show you exactly why. But I want you to understand that you do not have to believe me. I even carried a book this whole way. I just drove, so I did not actually carry it. But has anybody read this book? "Java Concurrency in Practice". How many people have read this book? Oh, boy, and all of these people are programming in Java. You have to read this book, because it is a fantastic book. And it is a book that will scare you to death, because what it basically says is: this is impossibly hard. It is just so difficult to get right."
https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/ClojureConcurrency.md
@U3JH98J4R the javadoc for package java.util.concurrent
is a good start
Hello, Iām probably missing something here, any idea ?
(refer 'clojure.core :exclude '(val))
=> nil
(def val 1)
WARNING: val already refers to: #'clojure.core/val in namespace: user, being replaced by: #'user/val
=> #'user/val
I should not have this warning ??I think when you start REPL, clojure.core
is included by default, so that refer
statement doesn't have any effect.
thank you @U2FRKM4TW , iām aware of the possibility to exclude from an ns form, here the beginning of the refer
docstring
refers to all public vars of ns, subject to filters.
filters can include at most one each of:
:exclude list-of-symbols
:only list-of-symbols
:rename map-of-fromsymbol-tosymbol
Maybe there is an exception for clojure.core + :excludeAs I said, when you execute REPL, clojure.core
is already there - refer
'ing it again does nothing.
You can try combining it with in-ns
because it will not load clojure.core
in that new ns.
$ clj -r
Clojure 1.10.1
user=> val
#object[clojure.core$val 0x2375b321 "clojure.core$val@2375b321"]
user=> (in-ns 'test)
#object[clojure.lang.Namespace 0x736ac09a "test"]
test=> (clojure.core/refer-clojure :exclude '[val])
nil
test=> val
Syntax error compiling at (REPL:0:0).
Unable to resolve symbol: val in this context
thank you for the idea @U2FRKM4TW
Hello ! I'm runnig windows 10 and cygwin, I wanted to use : https://github.com/clj-python/libpython-clj
so my first step was : clone repo then : lein install
but this gives me an error: http://java.io.IOException: Cannot run program "clojure" (in directory...
@andreas.scheinert if you don't get answers here, there's a #libpython-clj -- also the data science folks are pretty active over at zulipchat
thank you sogaiu
for the time beeing , maybe I took a wrong step to begin with, I'm used to lein and project.clj files , but carin meiers examples use deps.edn
could you point me to a tutorial or something howi turn her deps.edn example project to a lein project.clj one?
this is her example project: https://github.com/gigasquid/libpython-clj-examples
@andreas.scheinert i don't have a tutorial like that on hand, but perhaps this sample from the libpython-clj creator that uses project.clj might be of some help? https://github.com/cnuernber/facial-rec
thats so simple I wonder why its not included in the "official" example. It worked thanks @sogaiu!
https://clojure.atlassian.net/browse/CLJ-2056 (efficient alternative for (first (filter pred coll))
) was rejected on the grounds that linear search leads to poor performance.
However, I often find myself in a situation where I need to do it, especially when dealing with data from external sources. Is my best bet to just keep doing some variant of (first (filter pred coll))
, or is there an alternative way to approach this problem?
I was looking at a jar file created using lein uberjar
, and to my astonishment, it contains clj
files as it is. How does java command handle it? Where can I find more information on it? Or am I looking at wrong place for .class
files?
Nooooooo š®
It means, compilation phase includes "reader" as well?
TIL it. Thanks @nxtk!
I definitely need to watch it, it's been recommended second time in last few days. š
also it's good to know that clojure's minimal compilation unit is a top level s-expression\form, but there is some AOT functionality present like gen-class
, usually for calling clojure from java.
What is "a top level s-expression\form"? If you could elaborate..
take a look at any clj source file you will see that there is a sequence of top level forms separated by new line (ns ...) (defn ....) "string" 5
Oh okay. That makes sense. Thanks!
Iām trying to follow this tutorial for using Docker and Postgresql but Iām getting this error when I follow the steps. https://circleci.com/blog/package-a-clojure-web-application-using-docker/
user=> (dev)
CompilerException java.lang.IllegalArgumentException: Must hint overloaded method: toArray, compiling:(clojure/core/rrb_vector/rrbt.clj:282:1)
Seems like you have an old version of rrb-vector
with JDK 11 or newer. Try adding [org.clojure/core.rrb-vector "0.0.13"]
to your dependencies.
Is (seq big-lazy-seq)
lazy? We use this pattern often: (when (seq x) (process x))
. Weāve been having Java Heap memory errors in a job that processes a big lazy sequence in batches (~1M records, 4 jobs in parrallel). Removing the (seq records)
solves our problem. Memory usage would keep growing and GC would fallback on old-gen. These are our jvm opts:
-Xmx${max_heap_size} \
-XX:+UseG1GC \
-XX:MaxGCPauseMillis=100 \
-XX:-OmitStackTraceInFastThrow \
-XX:+HeapDumpOnOutOfMemoryError \
-Duser.timezone=UTC \
-Duser.country= \
-Duser.language=en \
-Djava.rmi.server.hostname=localhost \
And here is a JMX screenshot of running the job with and without it.
Our understanding was that this was the recommended way to check if an iterable is empty, is that still valid / is there something else going on here?@yannvahalewyn as far as i can see, seq is not lazy, it will realize LazySeq into a Cons seq
Object ls = sv;
sv = null;
while(ls instanceof LazySeq)
{
ls = ((LazySeq)ls).sval();
}
s = RT.seq(ls);
}
return s;
Hmm, thanks for replying. Iām pretty confident seq will and should stay lazy though.
(defn my-inc [x] (println x) (inc x))
(take 10 (seq (map my-inc (range 200))))
;; prints out 0-31
(do (seq (map my-inc (range 200))) nil)
Same behavior thoughThanks @nxtk for thinking along!
We have the same issue if we do (when (first records))
. The thing that confuses me is that next we (partition-all 500 records)
and that causes no issues whatshowever. This indicates there must be an issue upstream of some sort, but Iām not sure what I am looking for. Does anybody have some ideas what may cause this behavior?
too demonstrate
(defn my-inc [x] (println x) (inc x))
=> #'sicp.core/my-inc
(def xs (map my-inc (range 5)))
=> #'sicp.core/xs
(realized? xs)
=> false
(def ts2 (take 5 (seq xs)))
0
1
2
3
4
=> #'sicp.core/ts2
(realized? xs)
=> true
well it will realize up to 32 elements, for memory optimisations
my repl is broken š , can you try with 100?
So weāre looking for something wrong that makes it so that using (for [batch (partition-all 500 x)] ...)
wonāt blow my laptop up, but (when (seq x) (long-process x))
and even (when (first x) (long-process x))
will :thinking_face:
where long-process is the batch loop being wrapped by the when clause
and x
is a lazy seq which comes from a streamed json from s3. To be clear and avoid XY problems, all streaming works without issues without those when
clauses. And what we need is to see if there is something to process, and that is essential (itās a sync job, old data gets deleted and replaced).
as a wild guess does it output something, where xs is your big-lazy-seq
(filter (comp (partial not= LazySeq) second)
(map-indexed #(vector %1 (type %2))
(filter seq (take 1000 (iterate rest xs)))))
@yannvahalewyn (import clojure.lang.LazySeq)
Wel it is a clojure.lang.LazySeq
But it appears that something is making it unlazy, removing the seq
keeps it lazy which is still boggling my mind
but we solved it in another way though
now itās more about curiosity š
Iām not sure what you mean, can you elaborate?
at one point you can have something like that
(def xs (lazy-seq (cons 1 (cons 2 (lazy-seq (cons 3 (cons 4 nil)))))))
yes, all objects are from the same source, and there is not cons or concat upstream. Itās an all or nothing thing in this case. youāre never sure ofc, but the source is one big json array being streamed and parsed from s3
yeah thats why i proposed to run above code
(filter (comp (partial not= LazySeq) second)
(map-indexed #(vector %1 (type %2))
(filter seq (take 1000 (iterate rest xs)))))
=> ([1 clojure.lang.Cons] [3 clojure.lang.PersistentList])
I actually already did this š
Nothing came up sadly
haha enjoy!
Thanks for helping!
We solved it by counting the imported objects at the end of the transaction before persisting, and skipping out if it was empty though. So Iām gonna keep on with my other work now š . Thanks again!
Could it be that (when (seq x) ...
creates a local variable and that using (seq x)
makes the code hold onto the head of x (and cached values)? Just my guess after some googling.
Edit2: This is not the case.
(defn process [x]
(count (second x)))
(defn doit [x]
(when (seq x)
(process x)
#_(first x))) ; uncomment this to create OutOfMemoryError
(comment
(doit (split-with #(< % 12) (range 1e8))))
From this I think we can conclude that seq
in itself is not the problem.
Ref:
https://clojure.org/reference/lazy#_dont_hang_onto_your_head
https://stackoverflow.com/questions/15994316/clojure-head-retentionInteresting discussion š
hello! it seems to me that I have a problem with my clojure installation, currently I only have lein installed and when I try to run "lein repl" on this repo ( https://github.com/clj-python/clj-template) I get: http://java.io.IOException: Cannot run program "clojure" ...
I did and it claimed success
yes that gave me:
clojure Error: Could not find or load main class clojure.main Caused by: java.lang.ClassNotFoundException: clojure.main
There's a million things here that could go wrong to cause such an error. Without being actually able to look around in your environment, I'd recommend just removing anything that you can find on clojure
in your PATH
and installing it again.
Maybe I try tomorrow, thank you for your help!
assoc-some
can be handy https://github.com/weavejester/medley/blob/a4e5fb5383f5c0d83cb2d005181a35b76d8a136d/src/medley/core.cljc#L38
Generally I find it a tight lib, with genuinely 'missing' stuff
user=> (update {} :foo (fn [v] (or v 1)))
{:foo 1}
user=> (update {:foo 2} :foo (fn [v] (or v 1)))
{:foo 2}
Also, (fn [v] (or v 1))
is the same as (fnil identity 1)
. Assuming you don't care about false
values.
~@
can only act on a sequence, the symbol b
is not a sequence (even if its runtime value is), '(list 1 a)
is a sequence of 3 things, the symbol list
1
and the symbol a
yep, that's as I understand it
one thing I find quite helpful for understanding macros that you might not have seen is macroexpand
, running for example: (note the backtick before the expression)
(macroexpand `(m (list 1 a)))
produces:
(clojure.core/prn clojure.core/list 1 user/a)
Is there a simple hello world example of using tools.deps with docker?
I canāt seem to run clj with this docker image. https://hub.docker.com/_/clojure
And there is no clear examples of getting up and going with docker that I can find.
I thought I would be able to get going quickly but seems I will need to learn docker before I can get going
It would be helpful to understand why do you want to run Clojure in docker - is for shipping production workloads? Working locally? By default clojure image ships with Leiningen, you can start it with docker --rm -it clojure lein repl
im using tools deps
I need to use docker because thatās how everything is deployed at work and they want it that way for consistency. Iām unfamiliar with this area as I am coming from front end.
That still doesn't answer the question - do you have some sort of backend written in clojure that you need to run locally via Docker (totally doable)? Do you want to work on a Clojure project and use Docker to develop a Clojure application (AFAIR not doable ergonomically)?
Ah sorry @lukaszkorecki I see what you mean. I have to develop against a Postgresql database locally and deploy it to AWS, likely several microservices all doing very much the same thing. The purpose is data preprocessing for data scientists.
@grounded_sage Why not develop in the most convenient way for you and then package the apps as uberjar files and run them inside Docker with java -jar ...
?
Then you don't need anything inside the container except a JVM.
Iām unsure how all of this works so I canāt really comment. All I know is they want me using docker and pretty much every google search for docker and clojure results in. ā¢ Donāt use docker ā¢ We use docker but are not sharing how we did it - ( for whatever reason that may be, which is okay but somewhat frustrating compared to other lang ecosystems )
Usually Docker is used for packaging and deploying things, or running things like databases or already built services/apps
I'm not sure if you can develop in Docker, at least because of the file synchronization issues
Is there simple examples of a deploy?
That's a whole different can of worms, out of Clojure's scope really - do you have many hosts? one? Where are you deploying (aws? google?) - can you actually deploy etc etc etc
I mean there is this example. But it doesnāt run for me. https://circleci.com/blog/package-a-clojure-web-application-using-docker/
If you produce an uberjar from each of your Clojure apps, all you care about is "deploying Java apps with Docker"
Yea this seems like itās going to be the best option. I think I just need to learn Docker more because trying to copy/adapt what I find online is simply not working
Whoever the "they" is in your statement "they want me using docker", if they do it, they should have working examples for what they already do, I would think.
But there are plenty of jobs where knowing something about docker is essential.
Well itās the CTO of where I work. At present I am the only Clojure dev amongst all python devs. So my challenge is integrating into a unified way to work with their system and show the value of Clojure
https://dzone.com/articles/run-simple-jar-application-in-docker-container-1 -- from the part about adding HelloWorld.jar to Docker
You'd want whatever base Java Docker image your company is using but it's effectively as simple as develop in Clojure locally, build an uberjar (using whatever build process you prefer: lein
, boot
, CLI/`deps.edn`) and then create a basic DockerFile
that adds your JAR file to a Java-based docker image.
Yea this is starting to make sense. Docker is just meant to be a container for deployment after all
I run Clojure in ECS Fargate, and I donāt even bake my uberjar into the image. Instead I use two containers, one that downloads the jar from S3 into a place accessible to the other container. The other container is a plain openjdk container with zero modifications
my install method for java apps is similar, a freshly booted instance will figure out by itself what it needs to pull from s3 and run
and obviously asg iam roles guarantee access to the s3 objects accordingly per app
I do this because I have no need to add anything to the java docker image besides my jar, and complicating my CI with docker build/push
isnāt worth it
Does a postgresql db being included increase the benefit of Docker?
All good. I am a design/front end developer turned back end. I donāt know enough to know the right questions :)
I guess I am tripped up because I see all these dockerfiles downloading dependencies and building the apps
Yeah, I've no idea why all the tutorials seem to be so complicated. Hopefully that one I linked you to on DZone is simpler?
Was exactly what I was looking for @seancorfield thank you!
A common way to develop services is to have postgres and other non-Clojure services running in separate docker containers, because that way you can quickly start them up without installing them and ensure you have the same version and similar configuration as when it is deployed, etc. However, the clojure service you are developing is not run inside a docker container because itās much easier to start a REPL. When deployed, you might want to run your Clojure service inside of docker container to standardize and version the deployment environment, or use services like kubernetes / ECS / etc. It will typically talk to databases and other services that are outside of its container, though
Thanks @lilactown thatās a very clear response.
At work, I have Docker running MySQL (Percona 5.7), Elastic Search, and Redis -- docker-compose up
i I need to get a dev env up and running -- then do all my Clojure dev directly via the REPL from my editor (including running tests etc), with occasional more complete test suite runs via the terminal/command-line. Then we build (uber) JAR files for each process and that's run via java -jar
in production (we run on servers but it would be identical in Docker really) and all the database/search services are all running elsewhere in production (on a mixture of private and public cloud).
Our setup is somewhat similar: locally compose manages databases, rabbitmq etc and also can run our services streight from docker. In production we use managed services as much as possible, and deployments are just ansible runs pulling new versions of the container and restarting them. Our Dockerfile for Clojure uses multi stage builds with lein doing uberjarring and final container being just the JDK running the uberjar
@lukaszkorecki Iām curious as to what the multistage builds side of things are. Though it seems that most people using a docket file with build commands are using lein
https://github.com/kwladyka/fy-stock/blob/master/Dockerfile here is good example how you can use multi stages
Can i see via REPL which protocols a Record is implementing ?
@matthewlisp (ancestors RecordName)
That will show all the types the record implements including protocols.
user=> (defprotocol Foo (bar [x]))
Foo
user=> (defrecord Quux [y] Foo (bar [_] y))
user.Quux
user=> (sort-by str (ancestors Quux))
(java.lang.Object clojure.lang.Associative clojure.lang.Counted clojure.lang.IHashEq clojure.lang.IKeywordLookup clojure.lang.ILookup clojure.lang.IMeta clojure.lang.IObj clojure.lang.IPersistentCollection clojure.lang.IPersistentMap clojure.lang.IRecord clojure.lang.Seqable .Serializable java.lang.Iterable java.util.Map user.Foo)
user=>
See user.Foo
in the ancestors list?@seancorfield ancestors is a superset of supers?
i think i wasn't clear about what i want
i want to know what protocols an INSTANCE of a record implements
if that's even possible
and if the output come along with this huge list, it won't be of much use to me š
(ancestors (type instance))
i wanted only what i've written
It's a set, you could easily remove the standard ones that all records implement.
amazing
@nxtk ancestors is a superset of supers, yes. I've never actually used supers
-- I had to doc
it in the REPL and then look at the source
of ancestors
to see it calls supers
if it is passed a class instead of a tag .
user=> (defrecord base [])
user.base
user=> (def base-types (supers base))
#'user/base-types
user=> (apply disj (supers (type (->Quux 42))) base-types)
#{user.Foo}
user=>
^ @matthewlispso beautiful....
note this won't tell you if someone has installed a protocol externally for a record
and because that's an open set, there is no way to answer the general question (although you could answer it for specific protocols)
it's just for local protocols defined by myself
it's enough
True. Now you can extend protocols via metadata on objects, that changes the game a bit.
i'm making a little guide for my team-mate, we use Integrant lib at work
turns out we use Records + Protocols combo, and i was searching a way to dynamically find out which is the protocol that a Component in the system is implementing, so this way he can found it's "methods"
could be better if i could print out the name of the function(s) the protocol is specifying
but that's enough for now
user=> (defprotocol P (a [_]) (b [_]))
P
user=> (:sigs P)
{:a {:name a, :arglists ([_]), :doc nil}, :b {:name b, :arglists ([_]), :doc nil}}
Do protocols and spec play nice together? I have spec predicate
(partial satisfies? protocol/ToHoney)
When I'm conforming a larger document which contains a record that satisfies the protocol, I get the following exception:
java.lang.UnsupportedOperationException: Can't create empty: pyze.query_generator.pql.HoneyLiteral
at pyze.query_generator.pql.HoneyLiteral.empty(pql.clj:70)
at clojure.core$empty.invokeStatic(core.clj:5247)
at clojure.core$empty.invoke(core.clj:5241)
at clojure.spec.alpha$every_impl$cfns__2235$fn__2241.invoke(alpha.clj:1277)
at clojure.spec.alpha$every_impl$reify__2254.conform_STAR_(alpha.clj:1291)
at clojure.spec.alpha$or_spec_impl$fn__2115.invoke(alpha.clj:1089)
at clojure.spec.alpha$or_spec_impl$reify__2118.conform_STAR_(alpha.clj:1100)
at clojure.spec.alpha$or_spec_impl$fn__2111.invoke(alpha.clj:1065)
at clojure.spec.alpha$or_spec_impl$reify__2118.conform_STAR_(alpha.clj:1100)
at clojure.spec.alpha$conform.invokeStatic(alpha.clj:164)
at clojure.spec.alpha$conform.invoke(alpha.clj:160)
Spelunking a bit into spec source, it looks like this condition is triggered by including the protocol spec within an s/or
I think there is actually a ticket about this problem
it's a little weird
https://ask.clojure.org/index.php/3338/clojure-spec-attempts-to-make-empty-records?show=3338#q3338
it's a bug, and I've spent some time working on it in the past. it does not have an easy solution.
it's not the protocols, it's the data being passed in I believe
oh ok. look like you already have a repro case but let me know if there's something i can do to help
Anyone knows what's up with core.typed? I thought I'd look what its state is, and to my surprise, the repo seems to get regular, frequent updates. But to my disappointment, the documentation hasn't been updated in years and is completely useless...
Ambrose just started working on it again recently and has been re-merging previously split libraries
I am not sure he's at a point where the docs can be regenerated yet
so I think you've just caught it at a particularly bad time
cc @ambrosebs in case he might see this