Yes, let's assume you call that function, then use java interop on the result. How is the compiler going to know it's a string? You can annotate the call site, or you can annotate the return value

To clarify, I mean (defn foo ^{:tag String} []) over (defn ^String foo [])

Ah, sorry The former just adds this metadata to the vector object at read time The latter to the symbol and won't do anything to help with type inference iirc

defn is a special case, both annotate the return type, but on the arg vector is "preferred"

Both are used in clojure core libraries

You can use Java interop just fine, (.lock my-lock).

Yes, the interop would work. But if things are not wrapped into Clojure's core lib then I start to think why. Maybe there is a real difference of some sort. Why don't we have a macro for that? :thinking_face: Maybe synchronized is somehow better? P.S. I found that I can use Thread.holdsLock(obj) (I didn't check that but it looks like it should work): https://stackoverflow.com/questions/61557491/how-to-check-if-thread-holds-the-monitor-in-intellij#61652514 So both could probably work for my needs of "only one thing from these can happen at the same time" as I always carry around that same object. But then I remember people saying "just use ReentrantLock and don't bother with synchronized"... I was always wondering why.

Because all of Clojure multithreading is built in a way to avoid the need to manually handle locking

Clojure doesn't wrap a lot of Java constructs. Doesn't mean that those constructs aren't useful. No clue about the "`synchronized` vs ReentrantLock" though - I haven't used locking primitives in years, I'd have to read it all over gain to be able to tell.

https://www.sw1nn.com/blog/2012/04/11/clojure-stm-what-why-how/

@U03KLA1A00K I am very happy about Clojure's wrapped primitives but this time I have to lock things myself as I have two in-mem DBs that I have to partially keep in sync in an unusual way. There is no way pmap or core.async can help me this time 😄 I also think STM can't help me here too as I don't control refs myself. @U2FRKM4TW Yes, but then also they wrap some important high-use functions such as subs (`.substring`). So sometimes it makes sense from importance point of view but sometimes it's wrapped simply for supporting special syntax. :thinking_face:

IIRC, STM has its scope, and not all concurrent things fall into that scope, so you can't just replace all locks with STM. There have been some discussions here about it. Also, STM is not used that much: https://clojurians.slack.com/archives/C03S1KBA2/p1531169303000322

@U028ART884X Anecdotally, I use subs about 2-3 times a week, while I use locks about once every 5 years. Which one, would you say, deserves its own function more? :)

@U028ART884X Ah I thought you were a beginner just coming fom Java's locking world. (And I have to admit I just skimmed the article and thought it would cover all primitives). It looks like you'll have to create your own macro for that case, though? Is there no way to take a step back and reorder your calls to simplify stuff? Maybe use futures to get the locking effect? Do you have to deal with shared resources? Transactions at DB level?

@U2FRKM4TW I didn't know about subs function for a long time and I always used .substring. The way I learned about that function was when I was sent feedback from a home exercise where one of the negative points was that I used .substring everywhere 😄 But I would've put that as a suggestion, not a defect of the code. @U03KLA1A00K Beginner or not the same questions matter. Sometimes a detail can mean something very important. For me locking seems alright but for self-documentation I may want a macro. I mostly try to avoid macros until I really really need them. I also think that by specifying a lock object I increase the understanding that something important is happening. Yes, it's transactions at DB level. They aren't SQL transactions, they're way simpler. I have Primary and Secondary DB. Primary pipes some data via a callback to the secondary DB. Secondary DB can also accept insertions by itself and these insertions have to lock the Primary from writing into Secondary. The difference between the instances is that one is deleted on reboot and the other one is not. Weird-custom-cache-thing.

That's really some fascinating design 😀 We had a similar issue with a customer-facing API that caused database writes and callback hooks. Some of them would cause a lot of work, but the customer at some point automated a workflow so at regular intervals they bombed us with concurrent request. We had one advantage: the API route always contained an entity ID. We used this to create a map containing per-entity action queues of thunks instead of directly accessing the DB. This performs essentially the same as locking, but was much more predictable in the execution. I don't know if this can be applied to your situation, but sometimes it helps to view a problem from different angles.

I'm splitting the DB into two and then I'm doing locking to sync them together. The split is needed to reclaim the performance that I lost by inserting the ephemeral things into the DB as I had one DB before my current refactoring. This refactoring will save-up storage (and network, and insertion-lag) and delete transactions that I don't want to keep (the reboot loses the state of the Secondary node). So I'm not yet sure if I can split the inputs into multiple streams as my DBs have only single-loop when processing txs. Also I don't have prod yet so I don't have so much lag so that this would be useful. :thinking_face: Also probably I wouldn't be able to do it anyway as my data is intertwined very much :thinking_face:

Actually I could use a core.async channel. But I want to keep things consistent. At least for now.

Viewing the updates as input streams would really suggest an async channel, right. Which part of the system are aynchronous anyway? Just the DB updates, or inputs also?

Currently I want the nodes to be synchronous as it will make my life easier. The nodes can be made to be eventually consistent with each-other by not having a lock but instead pushing the updates into the second one via a channel. But then the direct inputs into the secondary node will not be possible (easily) and I want to have them. And there are more things that are going on.

So why not multiplex that channel and input from both?

Because both nodes store completely different things and they don't share any data in between. So there is no way to have only one source of events.

Ah. Well. That's a drawback

But that's also the purpose of the split 😄

I'd still go with the stream-from-multiple-sourcese approach, but instead of passing pure data into it, I'd pass thunks, so the channel acts only for synchronisation

What are thunks? Grouped transaction batches?

Basically it's just a delayed function. You pass the function which has to be evalueted next, in your case the function causing the next DB update. https://en.wikipedia.org/wiki/Thunk

avoid synchronized entirely. It's never better. 1) biased locking is no more 2) synchronized is always worse than ReentrantLock. From JDK19 there will be Virtual Threads and to use ReentrantLock(s) is one of the performance consideration that goes with this JDK addition.

Actually in my occasion biased locking would work very well as I know that the majority of the data flow would go between the nodes. Very large majority. And it would also go one way only so only one piece of the code would be running almost all the time.

So I may counter what littleli says here. Yes using core locking constructs is likely to perform better under virtual threads, but that requires that you're going to use jdk 19 or 21 as lts. 21 would be next year and there's time for consideration in the mean time. If you're still on jdk 8 or 11 then you can consider if you're likely to upgrade in the future. Using core's locking construct is perfectly viable, it is reentrant so it likely will act the same way as your reentrant lock would, and it only uses core constructs which may be beneficial for your codebase. On the other hand making a with-lock macro that wraps a code block in a lock is not a very high maintenance burden and is one of the cases where macros aren't just okay, they're encouraged (that is to say: macros are always okay if they introduce useful new flow control constructs that do not obscure the flow of code). I also offer that Clojure not wrapping something is not really grounds to avoid it, as Rich specifically talks about how clojure devs should know and use the java.util.concurrent library, and until clojure 1.11 you were expected to use java.lang.Math directly.

it's really not a counter 🙂 as I basically agree with you. Btw biased locking was disabled with JDK15 but if it helps or not is a subject of measurement. I believe it's under -XX option:

enabled: -XX:+UseBiasedLocking -XX:BiasedLockingStartupDelay=0
disabled: -XX:-UseBiasedLocking

I would just use a future proof way to deal with such cases and we already know what this is. That's all.

Yeah it started as wanting to be a counter but after elaboration it just ended up being "i agree with caveats"

https://gist.github.com/Hendekagon/15f791a37035e61320786f9afaa8c914

Hey, I'm relatively new to clojure and using VS Code (Calva). There isn't a function that does it without adding to commands or keyboard shortcuts. Im trying to add the function to an "onClick"

Sounds like you’re doing it in Clojurescript and not Clojure. I think @U0CKDHF4L’s example is specifically Clojure.

I think you need to port something like this javascript example to Clojure: https://www.w3schools.com/howto/howto_js_copy_clipboard.asp

ask in #clojurescript

Ok thanks anyways guys

@U03KB1C9N8M works for me like this:

[:button {:on-click (fn [e]
                         (.writeText js/navigator.clipboard "wow"))}
    "Click me!"]

I would maybe make a column for function true/false or so

@U04V15CAJ but then all the function-specific attributes should still probably go in another table, rather than introducing a bunch of nullable columns to support the extra function-specific data, yes?

Not sure. A json column could also work.

Hmm, I didn't realize this. If a var is used to associate some symbol with a memory address, how does it work for anonymous functions? Is there anywhere I can read about it?

anonymous functions are just that, anonymous. They're not named, but have a body just like any other function. You can't refer to them by name so you either have to just pass them as arguments to other things or call them immediately, or you save it into a local binding.

under the hood there is a name given to these functions, but it's not human readable and it's not saved into a var, just a java class

I see. So there is some internal reference for it that clojure maintains under the hood, but that's different from vars which are essentially an interface for humans to be able to name things and keep track of them. Clojure does all the same for anonymous functions (and vars), just without the human-readable name part. Does that sound correct?

it uses the constant pool, yeah, more or less. I'd say it's still important to keep functions separate from vars though because I don't think having phantom "vars" for every anonymous function is a good idea

Okay, thank you. Some valuable insights here.

Most notably defreloadable, a macro I propose as a teaching tool at the end of an unpublished blog article on writing reloadable code.

I really need to get my blog up so I can publish this stuff.

There's an optional linter in clj-kondo to warn about that :)

I've sometimes wished I did it when I'm searching a project for functions that use a certain keyword.

#cljfmt might know

The main reason to do this would be for IO, is that your intended usecase?

IMO, trying to recover/enforce strict semantics with lazy sequences is a trap.

To some extent that may be true but the sequence operations are very convenient for a lot of sequential IO. For this reason I've made a function sequence* which is a transducing context specifically designed for lazily-realized IO.

If you care about when items are produced/consumed, then I would avoid lazy sequences.

(defn sequence*
  "Non-chunking lazy sequence transducer context.

  This acts like [[sequence]], but has characteristics specifically designed for
  working with IO in a transducer.

  "
  ([xf coll]
   (let [rf (xf conj)
         f (fn f [coll]
             (lazy-seq
              (loop [coll coll]
                (when (seq coll)
                  (let [out (rf [] (first coll))]
                    (if-not (reduced? out)
                      (if (seq out)
                        (concat out (f (rest coll)))
                        (recur (rest coll)))
                      (seq @out)))))))]
     (rf (f coll))))
  ([xf coll & colls]
   (let [rf (xf conj)
         f (fn f [colls]
             (lazy-seq
              (loop [colls colls]
                (when (every? seq colls)
                  (let [out (apply rf [] (map first colls))]
                    (if-not (reduced? out)
                      (if (seq out)
                        (concat out (f (map rest colls)))
                        (recur (map rest colls)))
                      (seq @out)))))))]
     (rf (f (cons coll colls))))))

Here's a transducing context for lazy IO, use it if you like

I'm not aware of a good reliable way to make a lazy sequence non-chunked (and to stay that way) otherwise.

I do

(->> (fetch lot of stuff)
     (filter yet-unseen?) ; depends on an atom
     (map run-some-expensive-check-and-update-seen) ; update the atom
     ...)

I'm not sure how exactly this would prevent unwanted io unless the idea is that you'll actually specify mutably that you want no more stuff inside that map. Seems prone to error.

Well (filter yet-unseen?) checks the incoming stuff against a set (stored in an atom) and filters out stuff already checked. Then in the next step we run the check and if the results are relevant, we add that stuff to the set so that we do not check it again so step 3 feeds back into step 2.

If you have a requirement of user control of realization, don't use lazy seqs

right. It makes sense, it just feels a little dangerous is all.

@U064X3EF3 I can appreciate the sentiment but a lot of library code has been written to work on lazy sequences. Lazy IO is a footgun, but I don't think it's a stretch to say that sometimes the foot's worth less than the deadline and that's a lot of code that doesn't have to get re-written for some bespoke thunk mechanism.

the dangerous part is why it makes sense. lazy seqs make no guarantees about when or how much is realized.

that's fair. Working with the lazy-seq macro like this is technically relying on an implementation detail.

You can use all the transducer stuff like map, filter, etc without lazy sequences.

that's true, and doing side effects in an eager transducing context is generally going to be more correct

Wouldn't switching to transducers fix the original problem, regardless of laziness because it would remove any buffering between the filter and the map?

No in theory because it's undefined if calling first and rest on a lazy seq realizes one elt or one hundred (and it being undefined is why chunked sequences comply) and so lazy-seq may create a sequence witch realizes an amount different from expected. No in practice because clojure.core/sequence realizes the first argument prematurely from a pragmatic standpoint. @U0P0TMEFJ

so in practice you need a sequence impl that doesn't do that and which relies on a concrete type that specifically realizes one element per call to first/next, which in practice lazy-seq with a single cons call in it whose rest is a recursive call (like in my sequence* impl above) does this, but that relies on an implementation detail of lazy-seq so could more correctly be implemented with a reify.

I may have misunderstood, but I thought that the original problem was because there was a chunked sequence being returned

(->> (fetch lot of stuff)
     (filter yet-unseen?) ; depends on an atom - returns chunked seq
     (map run-some-expensive-check-and-update-seen) ; update the atom
     ...)

(sequence (comp (filter yet-unseen?) (map run-some-expensive-check-and-update-seen) ...) (fetch lot of stuff))

either way, it might be better to join the filter and map operations using keep so the read and writes are joined together??

You are right Ed, transducers solved my problem here

I am sure I will have this question again so I recorded highlights form this conversation at https://holyjak.tumblr.com/post/689730176615120896/can-i-un-chunk-lazy-sequences-to-realize-one 🙂

cool ... glad you got it sorted 😉

Me too :)

Omniscient debuggers like sayid https://github.com/clojure-emacs/sayid

Thank you 🙏

no problem

@U061E2UBT you can try FlowStorm https://github.com/jpmonettas/flow-storm-debugger/

it was created specifically for that kind of debugging

@U0739PUFQ wow great work, thank you brother

thanks! I hope it helps, if you have any questions let me know

wow

i really like your demo: https://www.youtube.com/watch?v=YnpQMrkj4v8

exactly what i wanted

thanks!

any feedback and ideas are welcome

@U0739PUFQ the right half of this screen makes me wonder if there's opportunity for interplay with Portal: https://github.com/djblue/portal. either in borrowing ideas on how data is displayed, or exporting data to be displayed there somehow.

yes, that is my TODOs already

I thought about portal integration (which you already kind of have via the def button)

of course!

but I think I'll write that functionality (copy portal) so you don't need to have too many windows open

the latest version allows you to define any value you are seeing for the repl, then you can tap> it to portal

my favorite thing about the demo is seeing all the ways you added to navigate an overwhelming amount of breadcrumbs

kept pulling more and more things out of your sleeve lol

my idea with FlowStorm is for it to be a debugger but also a "reverse engeneering" tool

reverse engineering in the sense of creating a mental model of a sometimes unkonwn codebase, and being able to do that from execution traces instead of reading code, which is not that easy with dynamically typed langs

I’ve been endeavoring to study the clojure compiler, and there's not too much written about it, so reverse-engineering is kind of the task is

yeah I'm also on the same path

I would love to use FlowStorm for something like that, but it is written in Java

but if you are into understanding the ClojureScript or the ClojureDart compiler something like FlowStorm should help a lot

oh, I missed that point that it couldn’t reach the java parts, so I see why you chose the cljs compiler in the demo

yeah, because FlowStorm can only instrument Clojure code

my post from last week or so: https://clojureverse.org/t/resources-for-learning-how-the-clojure-compiler-works/9059

I don't think something like FlowStorm can be created for most languages, Clojure has a pretty special combination of immutability (you need for tracing), being expression based, and easy to instrument by code rewriting

> https://clojureverse.org/t/resources-for-learning-how-the-clojure-compiler-works/9059 nice, thanks, will take a look

I'm going to give a talk in a few days here https://www.meetup.com/london-clojurians/events/285940432/

maybe you are interested

RSVP’d, thanks 👍