Fork me on GitHub
#clojure
<
2020-03-24
>
Bobbi Towers01:03:33

Idea: https://github.com/porkostomus/reel Do you think this makes any sense? I get lots of crazy ideas but I'm not sure if I've jumped the shark here.

seancorfield01:03:12

Interesting. Have you looked at Liquid?

Bobbi Towers01:03:48

Yeah, that's definitely an inspiration

4
Bobbi Towers01:03:09

There's probably a really easy way to do it, but I'm implementing the low-level terminal stuff in C just for fun:stuck_out_tongue_winking_eye:

hiredman01:03:05

I would check out the ed editor

Bobbi Towers01:03:47

yeah the idea is basically a "multiline ed"

Bobbi Towers01:03:32

oh you mean actually look at the source for ed?

hiredman01:03:23

I mean like, just implement ed in the repl

hiredman01:03:01

You don't need to do any of the terminal stuff

hiredman02:03:07

Ed can run in any clojure.main/repl because it just needs streams

hiredman02:03:45

Then you can start looking at acme, the plan9 editor, which is basically a gui front end that feeds commands to an ed style backend

💯 4
Bobbi Towers02:03:18

not sure if I made it clear, the low-level stuff is to get the terminal into raw mode, to facilitate full cursor movement, like Vim, but without clearing the screen or anything

dominicm08:03:54

I wonder if you could embed neovim into this... That would be novel.

Bobbi Towers01:03:52

I initially thought that to be not in the spirit of this, since the point is to not break the flow of your shell activity. But I could see it supporting extension modes (kind of like how vi began as the visual mode for ed), in which case Neovim would be the ideal candidate!

rutledgepaulv02:03:39

there used to be a website for searching for particular usages of a fully qualified symbol across public libraries right? what was it called? iirc the source was made open when they stopped hosting?

rutledgepaulv02:03:38

like hoogle for clojure

dpsutton02:03:26

but its down now 😞

dpsutton02:03:40

fantastic site there

rutledgepaulv02:03:57

i'm looking for something i can self host across a bunch of private repos

rutledgepaulv02:03:12

but thank you, gives me a starting point

dpsutton02:03:00

there's also this. i forget the name but its quite lovely to me: https://aleph.io/aleph/literate.html

rutledgepaulv02:03:42

marginalia i think

dpsutton02:03:28

yeah that's right. not sure if it fits your needs but thought i would toss that out there

rutledgepaulv02:03:45

thanks 🙂 I'm trying to inventory uses of a library across many projects

rutledgepaulv02:03:58

i can resort to grep -r and whatnot but having something alias aware would be nice

dpsutton03:03:52

I bet clj-kondo could be wrangled to do something interesting here

dpsutton03:03:32

Also clojure-lsp can find references. Might be able to use it programmatically and build up a dB of usages you’re interested in

rutledgepaulv03:03:13

neat 🙂 yeah maybe i'll make a cli tool that can do it against local dirs or github users/orgs

sogaiu04:03:09

@rutledgepaulv may be you'll find this portion of clj-kondo to be of interest: https://github.com/borkdude/clj-kondo/tree/master/analysis

rutledgepaulv04:03:54

ooh var-usages and ns declarations do look helpful

em05:03:49

Not sure where to post this but the link to spec.alpha is broken at https://clojure.org/guides/faq#_spec It's currently https://github.com/clojure/alpha.spec, should be reversed I think

👍 4
Alex Miller (Clojure team)12:03:57

No, the guide is pointing to the currently released version of spec. https://github.com/clojure/alpha.spec is "spec 2" and still in development. Some things in the guide will work differently in spec 2.

Alex Miller (Clojure team)13:03:27

oh, I totally misunderstood what you were talking about. fixing.

👍 4
grounded_sage11:03:31

Is there a terser way to write this? Assuming value-of-interest is a fn-argument and input to some is an arbitrarily sized map.

(some #(when (= (:name %) "value-of-interest") %) 
      [{:name "example"} {:name "value-of-interest"])

hindol11:03:27

Isn't this just filter?

(filter #(-> % :name (= "value-of-interest"))
        [{:name "example"} {:name "value-of-interest"}])
;; => ({:name "value-of-interest"})
Caveat, filter will return a sequence instead.

grounded_sage11:03:51

I’m getting the first result of filter. I was mostly just wondering if there was some clojure function left to discover. Like using when-some in some fancy way or something.

Chris O’Donnell11:03:33

I would probably write this as (first (filter #(= "value-of-interest (:name %)) coll)) because it makes it very clear at a glance what is happening. (And it is about equally succinct.) I don't know of a shorter way to write this with core functions.

👍 4
grounded_sage12:03:48

Yea this seems like the best option tbh

p-himik12:03:12

Another caveat of using filter - even if you use first, it will still create a longer sequence thus calling the predicate multiple times.

Chris O’Donnell12:03:23

That's a fair point. For my personal taste, I would still use filter for the improved readability unless there was a nontrivial, measured performance penalty for the choice.

hindol12:03:19

I vote for filter as well. The example with some is a little difficult to parse.

grounded_sage12:03:22

Yea it’s not a very large vector. If it was a very large vector I think some would make sense.

hindol12:03:02

Do check out core.match. It can make your matching logic more succinct. https://github.com/clojure/core.match/wiki/Basic-usage#map-patterns

rutledgepaulv12:03:24

I would build up your own functions so that it can be terse. like seek or find-first or ((index-by :name data) "value-of-interest")

👍 4
Chris O’Donnell12:03:27

To be clear, filter will consume 32 items at a time; it will not call the predicate on every item in the collection up front (unless the sequence has 32 items or less).

rutledgepaulv12:03:14

i pretty rarely seek for a specific item. tend to use indexed data structures and retrieve what i want via simple map lookups (almost always only ever iterate a seq once)

p-himik12:03:59

And that's the exact reason why something like find-first doesn't exist in the stdlib. There was a ticket for fast implementation, and it was declined with that reasoning.

rutledgepaulv12:03:28

yeah, just pick the right data structures and it's often a non-issue

grounded_sage13:03:30

Yea I’m doing this now. Went further into the code to find what is depending on the datastructure and changed how it is destructured.

jumpnbrownweasel13:03:57

IMO find-like functions are useful and I have several in my utilities namespace. Yes, you could create a faster access path, but sometimes that's not the best thing. If it were always the best thing, that is an argument for not having some .

Setzer2213:03:11

Hi all! I'd like to ask about the experiences of other fellow clojurians... We are trying to build a system using a (not-so-)micro-service architecture in clojure. However, we are struggling with RAM consumption... Running a even the simplest JVM clojure app inside a docker container takes, at least 300MB of RAM, and that is after following all the advice we could find around about clojure and JVM performance tuning in containers (particularly, we are aware of the issues with earlier JVMs not knowing the correct memory limits with cgroups and addressed them). So that why I'd like to ask here, to see if someone else is building apps in a similar architecture, and if their memory usage correlates with ours.. I keep feeling that's too much memory for what our very simple services are doing. Something that crossed my mind is that maybe moving to NodeJS clojurescript would help with memory consumption? We could afford migrating some of our services to cljs without a lot of effort... Any other tips are very much welcome!

Alex Miller (Clojure team)13:03:10

have you used a memory analyzer to actually look at your heap?

jumar13:03:57

300MB seems to me pretty low for a Clojure app, especially when talking about the whole JVM process memory consumption; I doubt you can go much less than that

Alex Miller (Clojure team)13:03:28

does not seem that low to me - I've run apps with less, but it really depends on what you keep in memory

jumar13:03:13

How much less? I don't remember anybody recommending less than -Xmx256m for production.

Setzer2213:03:04

It's definitely on my TODO list, I've just started taking a look into this seriously. I just wanted to ask first to know if this kind of memory usage is normal, or we could get lower than that

jumar13:03:03

If not using something like native image, which can be pretty restrictive on what things you can do...

jumar13:03:25

But Alex's question is still valid. Have you tried multiple different -Xmx values? How low you can go? What kind of errors/behavior do you observe?

Setzer2213:03:12

just now, I've been doing some experimentation, and using the experimental ZGC plus tight limits on memory (around -Xmx500m per service), we can get as low as 250MB. Using the default GC just seems to make the JVM take all the memory available at start and never release.

Setzer2213:03:30

I'm actually very surprised by this! At some point we just thought using the JVM meant paying 1GB of RAM upfront. But still, seeing our python services takes as low as 15MB makes me feel envy 😭

jumar13:03:56

I'm running one of my simple clojure apps with -Xmx256m and it works just fine. It might be worth exploring why your tiny clojure app cannot do that. And I've just tried it with -Xmx128m and it seems to work well.

jumar13:03:03

It may depend on particular GC too - if you want the GC to return memory to OS then G1GC (since JDk 12), Shenandoah and ZGC (not sure from which version, might be only JDK14) and of course Azul's C4 support that.

jumar13:03:35

Shenandoah might be better positioned to your use case (memory constraints), also because ZGC doesn't support compressed oops

Setzer2213:03:51

my experience with G1GC in our system is that memory is never returned to the OS :S But maybe I misconfigured something.

jumar13:03:59

What JDK?

Alex Miller (Clojure team)13:03:10

it should be in newer jdks with g1gc

Setzer2213:03:17

I was looking into Shenandoah and indeed looks like our best option! But sadly, the JDK included with the clojure docker images seems to lack support for it 😞

jumar13:03:28

Again, what JDK? 😉

Setzer2213:03:39

yes, sorry, JDK13

jumar13:03:51

Ah, so at least G1GC should support that: https://openjdk.java.net/jeps/346

jumar13:03:34

But perhaps don't be too obsessive about that. Could be good to figure out why you got OOM - did it come from your app code?

jumar13:03:35

Also setting some goal could be helpful - you for sure aren't going to achieve 15MB memory footprint but perhaps it even doesn't make much sense

Setzer2213:03:04

I'll look into the source of OOMs, maybe there's some clue there.

Setzer2213:03:57

And I was definitely not attempting to achieve a 15MB memory footprint in anything JVM 😅 As you say, it probably doesn't make sense either.

jumar13:03:58

Yeah, stacktrace could be helpful 🙂

jumar13:03:20

And if you want to use ZGC, then I'd highly recommend upgrading to JDK 14: https://malloc.se/blog/zgc-jdk14

jumar13:03:56

Note that both ZGC and Shenadoah (afaik) are still "experimental" - they could potentially break (or silently corrupt) your app in some rare circumstances; In practice, I think, many people are already using them just fine.

jumar13:03:15

Funny enough, I've got better results (on Mac with JDK14) running with the default G1GC when trying the ZGC - was able to run with as little as -Xmx18m and the whole process is taking about 140m

Setzer2213:03:50

an -Xmx lower than 300-ish makes the system go very cpu hungry and eventually dies with an OOM error

Alex Miller (Clojure team)13:03:41

you're just guessing at this stuff when you can easily go look at your heap and understand it

Setzer2213:03:46

indeed 😅 however, my experience with looking at clojure heaps with VisualVM is just seeing hundreds of instances of the clojure basic collections. Any recommendations on how to get better insights?

Alex Miller (Clojure team)13:03:21

well you need to look at why those are being held (path to root)

Alex Miller (Clojure team)13:03:30

I've spent more time with YourKit

Alex Miller (Clojure team)13:03:42

I don't know what your app does, so you need to combine your app knowledge and impl knowledge with what you see in the analyzer to see whether it makes sense to hold 300mb of stuff. if not, maybe there's a bug.

Setzer2213:03:18

Many thanks, I'll try that!

ghadi13:03:31

just going to concur with everything Alex is saying @jsanchezf . The JVM isn't known to be super memory efficient but I've run Clojure < 100MB

ghadi13:03:56

focus on understanding workload & dependencies

ghadi13:03:13

avoid pulling down magic cmdline flags

Setzer2213:03:17

@ghadi @alexmiller @jumar thanks for the advice! 😄 I'm going to look at the heaps with a profiler now, as it seems the obvious next step!

Alex Miller (Clojure team)13:03:14

yourkit lets you take heap snapshots and diff them

wizard 8
Alex Miller (Clojure team)13:03:55

I have found that to be an immeasurably useful way to compare what happened in the heap between two transactions if you can isolate to that level

Alex Miller (Clojure team)13:03:03

if you can't explain all the differences, that's usually a bad sign

jumar13:03:56

You can also use the -XX:+HeapDumpOnOutOfMemoryError option

jmckitrick14:03:37

I’m watching my colleagues at our scala shop debate back and forth on thread pool settings. Is this something that clojure needs to deal with often?

mavbozo14:03:13

I've worked with a web app + mysql which have problems handling number of requests above normal. We have to tune the webserver & JDBC thread pool settings.

jmckitrick14:03:50

I should be specific as well, they were discussing ‘multiple execution contexts’. But yes, I’ve definitely seen the need to bump settings on the JVM and JDBC settings

MatthewLisp15:03:39

what can i use/ where can i start, to load java classes into clojure and use it's methods?

Anik Chowhdury15:03:31

how to download username, password and anti forgery token based website using clojure? i have used clj-http. Can anyone show me a way to do this? i am kind of noob in this.

grounded_sage16:03:14

I am a bit confused with some. In my server rendered hiccup I have this. (some #(not= "idle" (:status %)) jobs) It renders correctly the first time. Then I trigger the job and it evaluates again. But after that it is trapped at true. It doesn’t seem to evaluate again

bfabry16:03:17

this seems unlikely to be related to the some fn, which is pretty simple

hindol16:03:18

How is a change in jobs triggering a re-render? Is this in Reagent?

grounded_sage16:03:32

I have these two things in my template.

[:h1 (some #(not= "idle" (:status %)) jobs)]
     [:h3 (map :status jobs)]
At first • :h1 is blank • :h3 is just idle idle I trigger the job. • :h1 is true • :h3 is just running idle Job completes • :h1 is true • :h3 is just idle idle

grounded_sage16:03:03

@UJRDALZA5 it is in hiccup using Ring/Compojure. We have it refreshing every 2 seconds

hindol16:03:19

So, after job completes, you want :h1 to disappear?

grounded_sage16:03:51

Yea. I mean this is just debugging purposes atm. But some should return to a nil response.

hindol16:03:29

Can you put a (prn jobs) somewhere on page? Does that change?

grounded_sage16:03:48

Well the map clearly shows that all jobs are idle

grounded_sage16:03:03

So to me it’s like some is caching it’s result and then just not running again…?

hindol16:03:29

prn will give a literal representation, just wanted to check the type of "idle" is still a string. Just a random thought.

👍 4
grounded_sage16:03:55

it was being updated with "idle "

hindol16:03:06

Yeah, pr/prn are great debugging tools. Works with deeply nested structures too.

grounded_sage16:03:07

Note the space at the end lol

dominicm17:03:22

Is it possible to test for sorted set membership without running the comparator?

hiredman17:03:35

an equivalent problem would be something like: check for a number existing in a sorted vector without using =, >, or <

hiredman17:03:36

I guess technically you can call seq on the sorted set, and walk through the seq using = to compare, which isn't technically running the comparator, but then also won't be doing the same check

dominicm17:03:38

I'm looking for order (want to avoid sorting repeatedly) but also for fast membership checks.

andy.fingerhut19:03:02

I haven't read all followups to see if this has already been suggested, but you could maintain a sorted collection to avoid resorting, and also a hashed collection for usually-faster membership checks, and keep their elements in sync with each other. Those two things could be wrapped inside a deftype or similar if you wanted to hide those kinds of details

dominicm17:03:24

I'd assumed that equivalence was implemented by the hash function and the sorting was really about when you seq

andy.fingerhut20:03:18

Bad assumption in this case. Maintaining the sorted order and using a hash function for lookups essentially means you need two different trees, one for maintaining the structure of the sordidness, and the other for the structure of the hash values, selected several bits at a time (5 bits per tree level in Clojure's hash-map implementation)

Eduardo Mata17:03:39

I have a vector with a set of ids and I want to query the database to return a vector with the results. I don't care about the order, I mostly want to make sure it happens simultaneously. Therefore, I am considering pmap as a parallel computation.

(def bodyMap {:location "ABC123", :flowerIds ["flowerOne" "flowerTwo" "flowerThree]})
(defn get-elements-from-db
  [bodyMap]
 (let [results (flatten (pmap #(get-from-db (:location bodyMap) %) (:flowerIds bodyMap)))] 
   do-something-or-whatever with resuls))
I am not sure if this is a "smart" or proper way to do it. Some results might take longer than the other, it is hard to tell which one will have more "data". Edit: I am using NoSQL such as Elasticsearch. A query is done by using search

seancorfield17:03:37

@contact904 You could just do a single SQL query with id IN (?,?,?,...) for the list of IDs.

seancorfield17:03:54

That would be a lot more efficient that running a separate query for every single ID.

Eduardo Mata17:03:11

I am not using SQL but NoSQL, elasticsearch to be more specific.

bentrent17:03:37

Why not use a built in query in elasticsearch? Like a terms query?

Eduardo Mata17:03:24

because I won't know how many flowerIds I will have in the vector, I could have 1 or 100s

bentrent17:03:32

that is fine?

bentrent17:03:52

If it can be more than 10k, then batching once you get more than 10k is necessary

bentrent17:03:23

100s is not that big of a query to hit ES with

bentrent17:03:53

But anyways, somebody who is more an expert in Clojure can talk about pmap, but batching as many as possible together in the same query will be way faster for Elasticsearch.

seancorfield17:03:56

pmap is almost always the wrong solution.

👍 8
seancorfield17:03:28

We use ES heavily at work and we use "id in list" / "id not in list" style queries a lot.

seancorfield17:03:41

100s of IDs isn't a problem (1,000s might be).

souenzzo17:03:24

a bit of propaganda, but #pathom solve this kind of lookup problem

Eduardo Mata18:03:10

@seancorfield there is no cases where 1k ids will be query.

Eduardo Mata19:03:43

Terms query worked wonderful

4
Albert Eng17:03:20

So I’ve been doing exercises in Clojure without reading a ton about the language. I’d like to read through a book that covers things that wouldn’t be needed for exercises through Exercism. Anyone have a suggestion on a book like that? Learn You a Clojure For Great Good Clojure for the Brave and True is OK, but I’m wondering if it’s the best option.

Albert Eng17:03:33

Heh, I totally got the book title wrong in my original post!

souenzzo17:03:40

I like http://clojurescriptkoans.com/ Last time I tryied Exercism I feed that the exercises force you to think in procedural way, so I dont recommend it.

Albert Eng17:03:48

Ooh nice, I’ll check that out! I enjoy learning things in a more hands-on manner, but so far the exercises I’ve been doing haven’t been pushing me to learn as much as I would have hoped.

seancorfield17:03:15

I'd recommend either Living Clojure (Carin Meier) or Getting Clojure (Russ Olsen). Those are both supposed to be really good for learning Clojure.

seancorfield17:03:10

Programming Clojure 3rd Ed is a bit more advanced. Then Clojure Applied for "real world" stuff.

Albert Eng17:03:49

Thanks Sean! I’ll check one of these over the “WFH break” and see how far I can get.

Alex Turok18:03:15

I liked The Joy of Clojure a lot many years ago. Maybe not the best one for starters, but still very good

4
isak19:03:05

I second The Joy of Clojure, especially if Clojure won't be your first programming language

Albert Eng21:03:05

@U2J4FRT2T Just finished the koans you linked, that was pretty cool! There were a couple concepts I haven’t ran into before, I’m interested to see how they can be used going forward

Albert Eng13:03:48

I started reading the preview chapters for The Joy of Clojure and am really enjoying it! It’s refreshing to have a book be faster paced and be focused on the things I’m actually wanting to learn for a new language. Thanks for the tip!

👍 4
✔️ 4
Albert Eng17:03:42

I’m stuck at home for the forseeable future, might as well learn a new language about it

didibus20:03:36

Stuart Halloway talks about simulation-based testing in https://github.com/cognitect-labs/transcriptor. Anyone knows what he means by this? > For testing code with nontrivial state I recommend simulation-based testing instead

seancorfield20:03:59

Here's Cognitect's white paper on that https://cognitect.com/sim-testing.pdf

seancorfield20:03:32

(I don't know if they ever released Simulant... I do seem to recall Stu talking about it at Conj one year tho'...)

seancorfield21:03:31

Ah, I never thought to look under the Datomic org. Thanks @ghadi!

didibus21:03:12

Cool, I'll look over it

didibus21:03:23

Seems it was a bit experimental, wonder if it panned out for them or not

benoit21:03:58

It did 🙂 We did a quite big consulting project with simulant for an ecommerce platform.

Alex Miller (Clojure team)21:03:46

Cognitect did several large simulation testing consulting projects, some using simulant, some that were conceptually similar

andy.fingerhut21:03:30

And I believe simulation testing was done first by folks at Cognitect on Datomic?

Alex Miller (Clojure team)21:03:47

really the ideas are the important part, simulant itself is really mostly valuable if you happen to be using Datomic (which was not the case for all the companies we did projects with)

andy.fingerhut21:03:56

Always good to know that it was applied across multiple projects, versus only one, of course.

Alex Miller (Clojure team)21:03:55

I'm fairly confident everyone here has used one or all of the systems we worked on at some point :)

Alex Miller (Clojure team)21:03:13

https://www.youtube.com/watch?v=N5HyVUPuU0E was another good talk by Mike Nygard, who did a lot of work on Simulant

plins23:03:18

any tips on how to make integrant to read config from a different edn file while running lein test ? the only solution Ive found is using environ and lein-environ to generate a .`lein-env` from the project.clj test profile

noisesmith23:03:15

why not ask it to load a completely different filename?

noisesmith23:03:01

you don't need environment variables for that, just make sure you can replace or paramaterize the config loading function

plins23:03:40

how can I detect in clojure that the project is being “tested”? so I can (if (= env :test) and load a different file?

noisesmith23:03:13

don't implicitly load the config, have a function that can be called differently from a test

noisesmith23:03:34

if the test file is loading the env, it should be asking for the config tests use

noisesmith23:03:09

or at least, that's how I'd do it