What I love about this approach is that you cut through all the noise to work with the actual data from the webserver rather than some imaginary mock data. And, as you fix the code, you can test and see it fixed right then. It's also totally compatible with remote REPLs — I can fix stuff like this in our staging environment directly (which is where all our data is)!

I'd recommend trying tap> along with a tool like Portal where you can refer back to the tapped values at any moment. As for your approach, there are these projects: • https://github.com/AbhinavOmprakash/snitch • https://github.com/borkdude/deflet (actually the inverse but could be helpful) • https://github.com/henryw374/defoclock All were found with https://phronmophobic.github.io/dewey/search.html, I just happen to remember that they exist.

I can see two options here: 1. instead of def forms, use tap> and add-tap listeners. 2. there is my library called bogus which interrupts the execution and shows a GUI window with locals and repl: https://github.com/igrishaev/bogus It's totally editor-agnostic

With bogus, you can hang in the middle of HTTP request or DB transaction, have a cup of tee and think

have tried the tap> approach, but the issue for me is that the data is no longer in place where I can use it at the REPL (as in the comment form in my example); i.e. it's for reference only. Basically, portal is a slightly nicer println (which I still use it for). Will check all the links out, thank you, @U2FRKM4TW @U1WAUKQ3E!

https://gist.github.com/hiredman/a2630ea6153d06840a2723d5b2c9698c throw a repl over tap

one of my colleagues just shared https://github.com/vvvvalvalval/scope-capture with me, which seems close

do you perhaps have some context for that dark magic @U0NCTKEV8? 😄

You put a call to send-repl wherever, passing whatever context information

Then in the repl call wait-for-repl, and when execution reaches send-repl, wait-for-repl connects your current repl's in and out and err to the repl created by send-repl, with whatever context bound to *1

Execution at send-repl halts until you exit the repl it sent

Which, uh, I usually in just some terminal so ^D

We monkey-patch this in, for some sugar. The local ddef works well for me.

(ns clojure.core)

(defmacro ddef
  "Sugar:

  (ddef x y z) -> (def x x) (def y y) (def x x)"
  [& names]
  (cons
    'do
    (map
      (fn [n]
        `(def ~n ~n))
      names)))

that's very clever @U0NCTKEV8, thanks for sharing!

i like how minimal that is Danie!

@U1WAUKQ3E i really like bogus, however i'm often working on a jvm process inside an aws ec2 instance, so i wouldn't be able to see the view, right

We do the same thing as danie, but also have a 0-arity version that maps over (keys &env). Sometimes leads to confusion if a local is shadowing a top-level def 😄

haha yes, the shadowing risks are substantial

@U0509NKGK I see, yes, SWING causes troubles sometimes.

Once upon a time I tried to solve it with X-display but no luck

I use tap> with REPLey so I can view them in the browser

seems like a deep rabbit-hole @U1WAUKQ3E 😅

https://github.com/tatut/REPLey (shameless plug)

cool project @U11SJ6Q0K! i like the blend of REPL + rich data viz you have there, this is sort of like portal + repl

Portal can do all this very simply. I configure it to capture all evaluation and inspect the atom it provides if I wish to do things with the last value. Comparing Portal to println suggests a lack of understanding of Portal. Using a shared var as a local tends to end up in code examples and leads to bad practices by people learning Clojure.

thanks @U05254DQM i'll check Portal more closely. it's been a while so I'm probably quite out of date

@U0509NKGK some useful pointer for portal For automatic tap> of evaluation: https://cljdoc.org/d/djblue/portal/0.48.0/doc/guides/nrepl Portal history in the REPL : https://cljdoc.org/d/djblue/portal/0.48.0/doc/ui-concepts/history#portal-atom I also send logs to portal too: https://cljdoc.org/d/djblue/portal/0.48.0/doc/guides/logging/%CE%BC-log Create a new project using in the https://practical.li/clojure/clojure-cli/projects/templates/practicalli/service/ if an example is needed on how to put this together with a custom user namespace that launches portal listening to the nrepl evaluations and all mulog log events

amazing, thank you!

@U0509NKGK http://www.flow-storm.org/ can also do that with a couple of clicks. This is how to do it https://jpmonettas.github.io/flow-storm-debugger/user_guide.html#_locals. And here is a video demo of how defining those looks https://www.youtube.com/watch?v=Mmr1nO6uMzc&t=281s

thank you @U0739PUFQ!

I strongly second Flow Storm

It's easy to get running and is useful both for quick debugging as well as for digging into complex bugs in large codebases where you haven't yet isolated the problem.

thanks!

I’ve gotten good mileage out of https://github.com/gfredericks/debug-repl

thanks @U0VP19K6K - i like that I don't need a local UI to use that!

(-> (java.lang.management.ManagementFactory/getRuntimeMXBean)
    (.getUptime))

https://docs.oracle.com/en/java/javase/15/docs/api/java.management/java/lang/management/RuntimeMXBean.html

does java record a time it launch's then and the above give you how long it has been running for ? My main goal is to try out a load of java flags and see how much difference it makes to the startup, so it important the time is before compilation, I will take a look at the link as it may answer that question :)

kind of sounds like it does I guess one way to find out 🙂

If you are doing it in a controlled environment where you can start/stop the app freely, then (System/exit 0) at your "point in code" and wrapping the whole process launch with time sounds like a low-tech solution.

Alternatively, you can record the result of date +%s into an ENV variable prior to launching your Java/Clojure process, and then subtract it from the current time at your "point in code"

The solution thheller gave seems to work nicely as I can just leave a log statement in place to tell me launch time, I have seen these params mentioned in a few places to increase startup time, -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xverify:none how ever reading up about them I believe only XX:+TieredCompilation is required and the timing seems to confirm thats the only flag the gives me a much improved startup time

thanks for the suggestions, now to see if i can find any other flags to help me out, if any one has any suggestions that would be great 🙂

-XX:+TieredCompilation you are trading off runtime performance though

-XX:+TieredCompilation doesn't hurt performance on its own. It is -XX:TieredStopAtLevel=1 that is a tradeoff.

that was my understanding as -XX:+TieredCompilation just does the most aggressive level

in a way startup time is more important that run time for me, mainly because cloud run shutsdown a node and when a request comes in spins it up so thie inital startup time is very important, it was causing a +20s delay adding in the options though takes 15s away so that's an improvement

Optimizing a pure JVM Clojure app for a Lambda-like environment could be a Sisyphean task. You might want to check out GraalVM native compilation or something like Babashka for this task.

If I was starting from scratch then I would agree, things are never that easy though 😞

using google ads library which does not play nice with graal or babashka, in fact libraries and babashka are a big hurdle you have to craft your app to only use libraries it supports,

I think CRaC could solve it but I could not get it to work, I can create the snapshot but the restore fails with little information as to what went wrong

I'm low tech, I would just have a bash script pass in the launch time as a parameter or an env, then, in your code read that time and take your own time stamp when your far enough into the launch process to meet your needs.

https://aws.amazon.com/blogs/compute/starting-up-faster-with-aws-lambda-snapstart/ may help you here?

SnapStart seems intresting sounds like CRaC but tied to amazon lambda's which is a shame where as CRaC would work in any container if I could get it working 🙂

Oh, I thought you were working on lambda specifically. Maybe I misread

working on GCP cloud run in this instance

Have you considered https://github.com/clojure/core.memoize or core.cache? Just to get the low hanging fruit out of the way. I've had similar questions too for which those weren't perfect solutions but I'm trying to remember the details of those situations

The problem reminds me of the Bloom filter. Wrap your heavy function with a wrapper that produces bloom-filter check.

hard to answer without knowing details Possible solutions: • Use memoize with time to live or last N inputs • Use not only user input data for cache, but also data from DB or other to identify cache • Use cache and invalidate it as an event if someone will change any of data needed for processing

I'm hesitant about out-of-the-box caching and memoization (and stuff like bloom filters) because the results that would potentially need to be cached are fairly large in-memory datasets rather than the outputs of individual function calls. I should mention that my use case is data analysis work, not a data pipeline dealing with a high velocity of change from a live system.

> airly large in-memory datasets rather than the outputs of individual function calls. not sure what you mean, https://github.com/clojure/core.memoize will cache output. If you are worry about catching input, because it is big: I am not sure if it cache input as it is or some kind of hash. But even if it does, then as I remember there is a way to write own cache identification which can be some kind of hash on data. PS I am not sure about last sentence. I can be wrong. It was some time ago when I read it.

FWIW, Python has this: https://joblib.readthedocs.io/en/latest/generated/joblib.Memory.html > All values are cached on the filesystem, in a deep directory structure. So while you're using a value, it's in memory. When you stop using it, it's GC'ed but the cache still has it on-disk. Perhaps something like that exists for Clojure or JVM as well, I don't know.

all in all if your input data are big use kind of hash to identify the same input data vs storing all input data. Before trying to code it on your own check if https://github.com/clojure/core.memoize already do it. I don’t remember and can’t verify it now.

I think #CL85MBPEF might be able to model the solution correctly by using continuous signals

I actually read through the missionary docs earlier today! It seems like it may help solve the dataflow problem but not the idempotence/"execute once" aspect.

@UFTRLDZEW m/memo solves the idempotence part IIUC

I missed that on my first read! I will have to try it.

clerk’s defcached and with-cache give you clerks caching semantics as a library, should be quick to try and see if it works for your use case.

https://clojurians.slack.com/archives/C02165ZDU9E/p1689457541576359

I think an http://ask.clojure.org question about this would be nice so maybe something could be done to the api to handle this kind of usage better

looks like a ticket already exists https://clojure.atlassian.net/jira/software/c/projects/DJSON/issues/DJSON-50 has existed for years, has a patch on it, and is marked as Major

If that's the cause, why does the code block with three {} seem to work? > is marked as Major I believe I've seen Alex mentioning somewhere that the priority field isn't actually used. But I could be imagining it.

because reading empty top level maps can just proceed a character at a time without doing pushback buffer stuff

@U064X3EF3 there is no http://ask.clojure.org mirror of this question so we can't vote for it.

you can see if you switch to calling the private -read and doing your own pbr wrapping the weirdness goes away:

(let [jsons (.getBytes (str
                         (json/write-str {})
                         (json/write-str {:a 42})
                         (json/write-str {})))]
  (with-open [rdr (-> (io/reader jsons)
                      (java.io.PushbackReader. 64))]
    (dotimes [_ 5]
      (prn (#'json/-read rdr false ::eof {})))))
{}
{"a" 42}
{}
:user/eof
:user/eof

amusingly the default pbr constructor fails because data.json uses more than 1 character of pushback, but there is another ticket with a patch that is like "we only need 1 char of push back, so we don't need the generality of a java pbr"

this intersects with the other ticket we have about PBR, LNPBR for core which I've been working on a bit recently, but we're going to push it out of 1.12. there are multiple use cases here and backwards compatibility concerns so that needs to be sorted through carefully. if there's ever a jira without an ask, feel free to make it (usually these get made in the other direction, but this happens occasionally)

https://www.clojure-toolbox.com/

You could also look at the list of libraries that get embedded into babashka for those kinds of things: https://book.babashka.org/#built-in-namespaces

Clojure toolbox is usually a good place to start. You can also ask in #CQT1NFF4L or search on https://phronmophobic.github.io/dewey/search.html

the jvm has multiple io libraries built in (http://java.io and java.nio) and two libraries for date parsing (java.text.DateFormat, java.time.format.DateTimeFormatter)

@U08JKUHA9 Re: "babashka namespaces" Yea when I was working on a BB script was when I noticed a lot of useful namespaces. It felt good to have all of them listed in the docs, just within my grasp, so I was curious if there was something more like it for the greater CLJ ecosystem. Re: "Clojure toolbox" That looks really great, and I'm also happy to see that it's being maintained (judging by the Github repo activity)

@UDHL22ZFE For some of the babashka namespaces, they are just clojure projects that can be used well in normal projects too. For example, babashka/fs, babashka/process.

@U01EB0V3H39 That's a very good idea for a channel, I will keep that option in mind 🙂 Re: "Search through github" that looks OK at first glance, but I believe it takes just # of stars in consideration? increases/decreases in starring isn't being used (trendiness) nor are downloads from main sources, so it's hard to make use of that

@U0NCTKEV8 Yea I'm aware that Java does have its own libraries. For instance JDBC is very popular JVM library among CLJ users from what I can tell. I was referring to the ability to discover these options for someone who not necessarily knows everything or doesn't have guidance

Fwiw I think most Clojure users use next.jdbc rather than using jdbc directly

Though lots of ppl use java.time directly rather than one of the wrapper libraries

As someone who just contributed a bunch more libraries to the clojure-toolbox list, I will caution that, even tho' the site/repo is maintained, it's very much up to the community to request new libraries be added to the listing -- and also request that unmaintained libraries get removed. Some of the libraries listed on clojure-toolbox were added long ago and are not widely used now and/or no longer maintained -- the "best choice" for a given domain changes over time, and there may well be several very different but very well-maintained options for some domains.

Re: JSON -- I can think of at least three options in that latter category (`data.json` -- the Clojure Contrib lib, cheshire, and jsonista). We use data.json at work because it has zero dependencies (and we generally try to prefer libraries that don't drag in half the Internet!).

@U04V70XH6 Re: "Clojure Toolbox" Firstly, thanks for your contributions! :hugging_face: So basically there's a room for improvement in the overall maintenance of the lists (of recommended/popular tools) yes? Do you think there's a way to increase the visibility of this project? have it be linked to from more sources perhaps? have it mentioned more often (e.g. in talks)?

Well, the problem with trying to make it more visible is that then people will assume it's the "best" source of library recommendations -- and I suspect there are some very-widely used libraries that aren't listed (because no one has thought to suggest adding them) and the presence of old/unmaintained libraries will become more of a issue that will cause complaints 😕 A "curated" list is never going to be inclusive of all good options and is likely to be inclusive of bad options too, so my advice is to treat it as just one possible way to find libraries, along with asking folks in various online communities what they prefer and why, for any given problem domain.

The reason I ended up adding a bunch of libraries to clojure-toolbox was part of my ongoing overhaul of http://clojure-doc.org which had its own libraries page and included quite a few that were not on clojure-toolbox! As part of producing the PR for the latter, I checked the "liveness" of each of the projects (on the clojure-doc list) and only submitted ones that seemed to be still "alive". In addition, clojure-doc had a one-line description for nearly all the libraries and the toolbox previously had no way to support that -- now if you hover over library links, you should see the description for libraries that have them (but a lot still don't).

This is the old clojure-doc page before I stripped out a lot of unmaintained stuff: https://github.com/clojure-doc/clojure-doc.github.io/blob/e7dec3dacf542b2a20dd9a8124dfb4eba395c08a/content/md/articles/ecosystem/libraries_directory.md 😞

I wonder if some of the maintenance of clojure-toolbox can be automated. E.g., check all the github repos, see if the repository is archived, or last commit was 12 years ago, etc. Or maybe we could add more signals, like number of dependent open-source projects, if that would be useful.

Yeah, there was some discussion of that, either in the PR/issues around my contributions or perhaps directly with James, I can't remember. I expect he'd be open to folks contributing that sort of automation via PRs...

@U04V70XH6 Yea, I was gonna suggest the idea of an automated advisory system that: • checks status (up-to-date'ness) of libraries currently listed • checks statistics of up-and-coming (trendiness) of other packages of CLJ in general (something based on stars over time on repository, or downloads over time, or amount of mentions as a keyword over time etc.) (Was gonna send that right after my previous message, before an internet outage broke out)

@UDHL22ZFE, another list of tools can be found here: https://github.com/razum2um/awesome-clojure#awesome-tools-in-clojure

Many 'everyday' functions can be found in the Clojure standard library. It really is quite substantial, so I always start there first https://clojure.github.io/clojure/ There is also the #CQT1NFF4L channel for opinions about libraries. I find an Internet search usually provides a few options (usually picking the cleanly documented libraries)

@U03CPPKDXBL Thanks for the suggestion! I do however notice that this is not strictly for libraries. There are products, tools, experimental projects, and basically everything that has to do with Clojure. I wanted something more focused on libraries.

@U05254DQM The core namespaces are enough until they aren't 🙂 Having some community-coordinated collection of curated libraries for common tasks can benefit everyone, and a Google search not always lands you on the best choices. Spending time choosing the right tool also takes time and distracts from what you may be doing. That's how I see it and why I was asking this question.

Well, the toolbox is about as close as we've got now. It just needs more willing community members to help review the current list for outdated stuff and figure out how to add new stuff. As I said earlier, I'm sure James would welcome the help.

I would be interested to see a list of what everyone thinks is the set of every day programming languages libraries. As with any programming language there are a large variety of libraries available and people have various opinions about them (as can be seen across many discussions in Slack and elsewhere)

I had submitted a proposal to Clojurists together to work on a project allow easier discovery of appropriate libraries for Clojure, although this would still be an opinionated view of the Clojure libraries (certainly initially)

"every day ... libraries" is going to depend a lot on your domain I think? For "web apps", you're going to have Ring + some routing lib + maybe some lifecycle/dependency lib + probably some database libs (SQL or something else). For CLIs, you're going to maybe have some argument parsing library, maybe some lifecycle/dependency lib (if your CLI is complex enough), maybe database libs. Data science is going to be a whole different ball game (no idea about the libs there). FinTech, same. I'd say maybe logging is probably common to everything? For me, Component, tools.logging, data.json, next.jdbc tend to be a given for everything. Ring, Compojure, Selmer for web apps.

Hmm, these days I might throw Aero in there as a given for everything (for config).

macros get unevaluated forms.

(defmacro x [colls]
  (prn colls)
  (count colls))


#'user/x
user> (x (list* [1 2] [3 4] [5 6] [[7 8] [9 0] [11 12]]))
(list* [1 2] [3 4] [5 6] [[7 8] [9 0] [11 12]])
5

colls will be a list with the following unevaluated values: • the symbol list* • the vector [1 2] etc.

Interesting. So is using eval the thing to do there?

what are you trying to do?

why not just call count ?

(count (list* [1 2] [3 4] [5 6] [[7 8] [9 0] [11 12]]))
=> 6

Oh, it's part of a larger macro, that's just the part I was confused about

It kinda depends on what you're trying to do. You can have your macro output code that calls count on the result and also outputs code that further processes the result.

there might be niche reasons for calling eval (eg. working around clojurescript's lack of tangible runtime), but it's usually not the right option.

Alright I think I understand

For literals like vectors, maps, and sets, you can determine their length at compile time, but it's not possible for lists that represent calls to arbitrary functions. Depending on the use case, it might be possible to determine the length of forms for some function calls (eg. list*).

Will (count ~(list* []....))` Be useful? Like this

(defmacro x [colls] 
 `(count ~colls))

Could be, it will give me something else to try because I'm still stuck... I've never really written macros before

Oh. Might I recommend something I wrote? Macrobrew: Clojure macros distilled https://abhinavomprakash.com/posts/macrobrew/

Oh nice! Thanks a lot, I've been trying to read more stuff. I went through Mastering Clojure Macros last week, and, well I haven't mastered them yet

Ahahahaha. Macros are fun and you'll get there. Most importantly write macros and play around. 😄

I never had a use for them before, but now I'm trying to build a Clojure interpreter so all of a sudden I'm in the deep end