This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-03-24
Channels
- # announcements (10)
- # aws-lambda (9)
- # babashka (14)
- # beginners (166)
- # calva (63)
- # chlorine-clover (4)
- # cider (40)
- # cljs-dev (4)
- # clojure (182)
- # clojure-europe (18)
- # clojure-italy (5)
- # clojure-nl (5)
- # clojure-spec (17)
- # clojure-uk (55)
- # clojurescript (11)
- # core-async (12)
- # cursive (23)
- # datascript (5)
- # datomic (19)
- # emacs (4)
- # fulcro (46)
- # graalvm (2)
- # hoplon (2)
- # joker (3)
- # juxt (1)
- # keechma (2)
- # leiningen (20)
- # malli (1)
- # meander (7)
- # nrepl (1)
- # off-topic (72)
- # pedestal (6)
- # re-frame (15)
- # reitit (7)
- # shadow-cljs (34)
- # sql (14)
- # testing (14)
- # tools-deps (11)
- # tree-sitter (1)
- # vim (14)
- # xtdb (19)
- # yada (3)
Idea: https://github.com/porkostomus/reel Do you think this makes any sense? I get lots of crazy ideas but I'm not sure if I've jumped the shark here.
Interesting. Have you looked at Liquid?
There's probably a really easy way to do it, but I'm implementing the low-level terminal stuff in C just for fun:stuck_out_tongue_winking_eye:
yeah the idea is basically a "multiline ed"
oh you mean actually look at the source for ed?
Then you can start looking at acme, the plan9 editor, which is basically a gui front end that feeds commands to an ed style backend
oh cool!
not sure if I made it clear, the low-level stuff is to get the terminal into raw mode, to facilitate full cursor movement, like Vim, but without clearing the screen or anything
I initially thought that to be not in the spirit of this, since the point is to not break the flow of your shell activity. But I could see it supporting extension modes (kind of like how vi began as the visual mode for ed), in which case Neovim would be the ideal candidate!
there used to be a website for searching for particular usages of a fully qualified symbol across public libraries right? what was it called? iirc the source was made open when they stopped hosting?
like hoogle for clojure
i think it was https://crossclj.info/
ah yeah
thanks
i'm looking for something i can self host across a bunch of private repos
but thank you, gives me a starting point
there's also this. i forget the name but its quite lovely to me: https://aleph.io/aleph/literate.html
marginalia i think
yeah that's right. not sure if it fits your needs but thought i would toss that out there
thanks š I'm trying to inventory uses of a library across many projects
i can resort to grep -r and whatnot but having something alias aware would be nice
Also clojure-lsp can find references. Might be able to use it programmatically and build up a dB of usages youāre interested in
Essentially the complement of https://github.com/borkdude/carve/blob/master/README.md
neat š yeah maybe i'll make a cli tool that can do it against local dirs or github users/orgs
@rutledgepaulv may be you'll find this portion of clj-kondo to be of interest: https://github.com/borkdude/clj-kondo/tree/master/analysis
ooh var-usages and ns declarations do look helpful
Not sure where to post this but the link to spec.alpha is broken at https://clojure.org/guides/faq#_spec It's currently https://github.com/clojure/alpha.spec, should be reversed I think
No, the guide is pointing to the currently released version of spec. https://github.com/clojure/alpha.spec is "spec 2" and still in development. Some things in the guide will work differently in spec 2.
oh, I totally misunderstood what you were talking about. fixing.
Is there a terser way to write this? Assuming value-of-interest
is a fn-argument and input to some
is an arbitrarily sized map.
(some #(when (= (:name %) "value-of-interest") %)
[{:name "example"} {:name "value-of-interest"])
Isn't this just filter
?
(filter #(-> % :name (= "value-of-interest"))
[{:name "example"} {:name "value-of-interest"}])
;; => ({:name "value-of-interest"})
Caveat, filter
will return a sequence instead.Iām getting the first result of filter. I was mostly just wondering if there was some clojure function left to discover. Like using when-some
in some fancy way or something.
I would probably write this as (first (filter #(= "value-of-interest (:name %)) coll))
because it makes it very clear at a glance what is happening. (And it is about equally succinct.) I don't know of a shorter way to write this with core functions.
Yea this seems like the best option tbh
Another caveat of using filter
- even if you use first
, it will still create a longer sequence thus calling the predicate multiple times.
That's a fair point. For my personal taste, I would still use filter
for the improved readability unless there was a nontrivial, measured performance penalty for the choice.
Yea itās not a very large vector. If it was a very large vector I think some
would make sense.
Do check out core.match. It can make your matching logic more succinct. https://github.com/clojure/core.match/wiki/Basic-usage#map-patterns
I would build up your own functions so that it can be terse. like seek
or find-first
or ((index-by :name data) "value-of-interest")
To be clear, filter
will consume 32 items at a time; it will not call the predicate on every item in the collection up front (unless the sequence has 32 items or less).
i pretty rarely seek for a specific item. tend to use indexed data structures and retrieve what i want via simple map lookups (almost always only ever iterate a seq once)
And that's the exact reason why something like find-first
doesn't exist in the stdlib. There was a ticket for fast implementation, and it was declined with that reasoning.
yeah, just pick the right data structures and it's often a non-issue
Yea Iām doing this now. Went further into the code to find what is depending on the datastructure and changed how it is destructured.
IMO find-like functions are useful and I have several in my utilities namespace. Yes, you could create a faster access path, but sometimes that's not the best thing. If it were always the best thing, that is an argument for not having some
.
Hi all! I'd like to ask about the experiences of other fellow clojurians... We are trying to build a system using a (not-so-)micro-service architecture in clojure. However, we are struggling with RAM consumption... Running a even the simplest JVM clojure app inside a docker container takes, at least 300MB of RAM, and that is after following all the advice we could find around about clojure and JVM performance tuning in containers (particularly, we are aware of the issues with earlier JVMs not knowing the correct memory limits with cgroups and addressed them). So that why I'd like to ask here, to see if someone else is building apps in a similar architecture, and if their memory usage correlates with ours.. I keep feeling that's too much memory for what our very simple services are doing. Something that crossed my mind is that maybe moving to NodeJS clojurescript would help with memory consumption? We could afford migrating some of our services to cljs without a lot of effort... Any other tips are very much welcome!
have you used a memory analyzer to actually look at your heap?
300MB seems to me pretty low for a Clojure app, especially when talking about the whole JVM process memory consumption; I doubt you can go much less than that
does not seem that low to me - I've run apps with less, but it really depends on what you keep in memory
How much less? I don't remember anybody recommending less than -Xmx256m
for production.
It's definitely on my TODO list, I've just started taking a look into this seriously. I just wanted to ask first to know if this kind of memory usage is normal, or we could get lower than that
If not using something like native image, which can be pretty restrictive on what things you can do...
But Alex's question is still valid. Have you tried multiple different -Xmx
values? How low you can go? What kind of errors/behavior do you observe?
just now, I've been doing some experimentation, and using the experimental ZGC plus tight limits on memory (around -Xmx500m per service), we can get as low as 250MB. Using the default GC just seems to make the JVM take all the memory available at start and never release.
I'm actually very surprised by this! At some point we just thought using the JVM meant paying 1GB of RAM upfront. But still, seeing our python services takes as low as 15MB makes me feel envy š
I'm running one of my simple clojure apps with -Xmx256m
and it works just fine. It might be worth exploring why your tiny clojure app cannot do that.
And I've just tried it with -Xmx128m
and it seems to work well.
It may depend on particular GC too - if you want the GC to return memory to OS then G1GC (since JDk 12), Shenandoah and ZGC (not sure from which version, might be only JDK14) and of course Azul's C4 support that.
Shenandoah might be better positioned to your use case (memory constraints), also because ZGC doesn't support compressed oops
my experience with G1GC in our system is that memory is never returned to the OS :S But maybe I misconfigured something.
it should be in newer jdks with g1gc
I was looking into Shenandoah and indeed looks like our best option! But sadly, the JDK included with the clojure docker images seems to lack support for it š
Ah, so at least G1GC should support that: https://openjdk.java.net/jeps/346
But perhaps don't be too obsessive about that. Could be good to figure out why you got OOM - did it come from your app code?
Also setting some goal could be helpful - you for sure aren't going to achieve 15MB memory footprint but perhaps it even doesn't make much sense
And I was definitely not attempting to achieve a 15MB memory footprint in anything JVM š As you say, it probably doesn't make sense either.
And if you want to use ZGC, then I'd highly recommend upgrading to JDK 14: https://malloc.se/blog/zgc-jdk14
Note that both ZGC and Shenadoah (afaik) are still "experimental" - they could potentially break (or silently corrupt) your app in some rare circumstances; In practice, I think, many people are already using them just fine.
Funny enough, I've got better results (on Mac with JDK14) running with the default G1GC when trying the ZGC - was able to run with as little as -Xmx18m
and the whole process is taking about 140m
an -Xmx lower than 300-ish makes the system go very cpu hungry and eventually dies with an OOM error
you're just guessing at this stuff when you can easily go look at your heap and understand it
indeed š however, my experience with looking at clojure heaps with VisualVM is just seeing hundreds of instances of the clojure basic collections. Any recommendations on how to get better insights?
well you need to look at why those are being held (path to root)
I've spent more time with YourKit
I don't know what your app does, so you need to combine your app knowledge and impl knowledge with what you see in the analyzer to see whether it makes sense to hold 300mb of stuff. if not, maybe there's a bug.
head holding, etc
just going to concur with everything Alex is saying @jsanchezf . The JVM isn't known to be super memory efficient but I've run Clojure < 100MB
@ghadi @alexmiller @jumar thanks for the advice! š I'm going to look at the heaps with a profiler now, as it seems the obvious next step!
I have found that to be an immeasurably useful way to compare what happened in the heap between two transactions if you can isolate to that level
if you can't explain all the differences, that's usually a bad sign
Iām watching my colleagues at our scala shop debate back and forth on thread pool settings. Is this something that clojure needs to deal with often?
I've worked with a web app + mysql which have problems handling number of requests above normal. We have to tune the webserver & JDBC thread pool settings.
I should be specific as well, they were discussing āmultiple execution contextsā. But yes, Iāve definitely seen the need to bump settings on the JVM and JDBC settings
what can i use/ where can i start, to load java classes into clojure and use it's methods?
how to download username, password and anti forgery token based website using clojure? i have used clj-http. Can anyone show me a way to do this? i am kind of noob in this.
I am a bit confused with some
. In my server rendered hiccup I have this.
(some #(not= "idle" (:status %)) jobs)
It renders correctly the first time. Then I trigger the job and it evaluates again. But after that it is trapped at true. It doesnāt seem to evaluate again
I have these two things in my template.
[:h1 (some #(not= "idle" (:status %)) jobs)]
[:h3 (map :status jobs)]
At first
⢠:h1 is blank
⢠:h3 is just idle idle
I trigger the job.
⢠:h1 is true
⢠:h3 is just running idle
Job completes
⢠:h1 is true
⢠:h3 is just idle idle
@UJRDALZA5 it is in hiccup using Ring/Compojure. We have it refreshing every 2 seconds
Yea. I mean this is just debugging purposes atm. But some should return to a nil response.
Well the map
clearly shows that all jobs are idle
So to me itās like some is caching itās result and then just not running againā¦?
prn
will give a literal representation, just wanted to check the type of "idle"
is still a string. Just a random thought.
Damn strings
it was being updated with "idle "
Note the space at the end lol
an equivalent problem would be something like: check for a number existing in a sorted vector without using =, >, or <
I guess technically you can call seq on the sorted set, and walk through the seq using = to compare, which isn't technically running the comparator, but then also won't be doing the same check
I'm looking for order (want to avoid sorting repeatedly) but also for fast membership checks.
I haven't read all followups to see if this has already been suggested, but you could maintain a sorted collection to avoid resorting, and also a hashed collection for usually-faster membership checks, and keep their elements in sync with each other. Those two things could be wrapped inside a deftype or similar if you wanted to hide those kinds of details
I'd assumed that equivalence was implemented by the hash function and the sorting was really about when you seq
Bad assumption in this case. Maintaining the sorted order and using a hash function for lookups essentially means you need two different trees, one for maintaining the structure of the sordidness, and the other for the structure of the hash values, selected several bits at a time (5 bits per tree level in Clojure's hash-map implementation)
I have a vector with a set of ids
and I want to query the database to return a vector with the results. I don't care about the order, I mostly want to make sure it happens simultaneously. Therefore, I am considering pmap
as a parallel computation.
(def bodyMap {:location "ABC123", :flowerIds ["flowerOne" "flowerTwo" "flowerThree]})
(defn get-elements-from-db
[bodyMap]
(let [results (flatten (pmap #(get-from-db (:location bodyMap) %) (:flowerIds bodyMap)))]
do-something-or-whatever with resuls))
I am not sure if this is a "smart" or proper way to do it. Some results might take longer than the other, it is hard to tell which one will have more "data".
Edit: I am using NoSQL such as Elasticsearch. A query is done by using search
@contact904 You could just do a single SQL query with id IN (?,?,?,...)
for the list of IDs.
That would be a lot more efficient that running a separate query for every single ID.
I am not using SQL but NoSQL, elasticsearch to be more specific.
because I won't know how many flowerIds
I will have in the vector, I could have 1 or 100s
But anyways, somebody who is more an expert in Clojure can talk about pmap
, but batching as many as possible together in the same query will be way faster for Elasticsearch.
We use ES heavily at work and we use "id in list" / "id not in list" style queries a lot.
100s of IDs isn't a problem (1,000s might be).
@seancorfield there is no cases where 1k ids will be query.
So Iāve been doing exercises in Clojure without reading a ton about the language. Iād like to read through a book that covers things that wouldnāt be needed for exercises through Exercism. Anyone have a suggestion on a book like that? Learn You a Clojure For Great Good Clojure for the Brave and True is OK, but Iām wondering if itās the best option.
Heh, I totally got the book title wrong in my original post!
I like http://clojurescriptkoans.com/ Last time I tryied Exercism I feed that the exercises force you to think in procedural way, so I dont recommend it.
Ooh nice, Iāll check that out! I enjoy learning things in a more hands-on manner, but so far the exercises Iāve been doing havenāt been pushing me to learn as much as I would have hoped.
I'd recommend either Living Clojure (Carin Meier) or Getting Clojure (Russ Olsen). Those are both supposed to be really good for learning Clojure.
Programming Clojure 3rd Ed is a bit more advanced. Then Clojure Applied for "real world" stuff.
Thanks Sean! Iāll check one of these over the āWFH breakā and see how far I can get.
I liked The Joy of Clojure a lot many years ago. Maybe not the best one for starters, but still very good
I second The Joy of Clojure, especially if Clojure won't be your first programming language
@U2J4FRT2T Just finished the koans you linked, that was pretty cool! There were a couple concepts I havenāt ran into before, Iām interested to see how they can be used going forward
I started reading the preview chapters for The Joy of Clojure and am really enjoying it! Itās refreshing to have a book be faster paced and be focused on the things Iām actually wanting to learn for a new language. Thanks for the tip!
Iām stuck at home for the forseeable future, might as well learn a new language about it
Stuart Halloway talks about simulation-based testing in https://github.com/cognitect-labs/transcriptor. Anyone knows what he means by this? > For testing code with nontrivial state I recommend simulation-based testing instead
Here's Cognitect's white paper on that https://cognitect.com/sim-testing.pdf
(I don't know if they ever released Simulant... I do seem to recall Stu talking about it at Conj one year tho'...)
https://github.com/datomic/simulant is one example @seancorfield
Ah, I never thought to look under the Datomic org. Thanks @ghadi!
It did š We did a quite big consulting project with simulant for an ecommerce platform.
Cognitect did several large simulation testing consulting projects, some using simulant, some that were conceptually similar
And I believe simulation testing was done first by folks at Cognitect on Datomic?
really the ideas are the important part, simulant itself is really mostly valuable if you happen to be using Datomic (which was not the case for all the companies we did projects with)
Always good to know that it was applied across multiple projects, versus only one, of course.
I'm fairly confident everyone here has used one or all of the systems we worked on at some point :)
https://www.infoq.com/presentations/Simulation-Testing/ was I think the first talk we did about it
https://www.youtube.com/watch?v=N5HyVUPuU0E was another good talk by Mike Nygard, who did a lot of work on Simulant
any tips on how to make integrant
to read config from a different edn
file while running lein test
? the only solution Ive found is using environ
and lein-environ
to generate a .`lein-env` from the project.clj
test profile
why not ask it to load a completely different filename?
you don't need environment variables for that, just make sure you can replace or paramaterize the config loading function
how can I detect in clojure that the project is being ātestedā? so I can (if (= env :test)
and load a different file?
don't implicitly load the config, have a function that can be called differently from a test
if the test file is loading the env, it should be asking for the config tests use
or at least, that's how I'd do it