It would work on futures, but futures cache the result and thus will aggregate memory if not disposed of carefully. A better way might be to simpliy (.start (Thread. (fn [] (println "hello"))). As a word of warning, I used a similar approach in the past and went back to awaiting the success of the side effect, as my main code overtaking the side effects started causing race conditions.

That's what i'd do as well. if you care about it completing you could add a @f to await completion somewhere where (def f (future ...))

Thanks, in both approaches, what would happen if

I run this in a doseq, potentially firing off like a dozen of these calls? would that cause things to choke, ie. halting the whole webapp

(I’m doing some performance tweaks on a particular slow part in my webapp, that gets triggered when a user clicks a button)

if you deref inside the doseq, then yes it would block that thread. webapps should work around it with more threads but its a resource hog

As always, that depends on the machine, the length of the seqs,... but basically I agree with listpyclouds

what do you mean with ‘work around it with more thread’. Or would that mean using channels/go-loops/parking etc

When you're speaking about a webapp, you mean the backend part, right?

yes

so a webserver on the JVM works with a threadpool, assigning a request to a thread. when the doseq happens that thread blocks

server side, Ring based

more requests should have other free threads to execute

Every doseqq will block a thread, so that is a factor to consider

if too many such requests happen, you could block the pool completely and/or run out of RAM

Project Loom with Virtual Threads from JDK 19 aims to solve these, specially when theres a lot of waiting on external things like IO.

(defn my-req-handler
 [req]
 (let [db     (get-connection) 
       orders (generate-orders)]
  (doseq [o orders]
   (orderdb/create-order! o db))))

So in essence, the ‘create-order’ call is slow

And it generates about 10-20 orders in that doseq

of course, there’s batch insertion on the database level, but for the future, i’d like to know a general strategy for dealing with this

Re: project loom: how would that translate to clojure/Ring? or will it ‘just work’ (tm)

I haven't tried either of these, however: If you're using vm >= 19, https://github.com/teknql/tapestry may be a nice solution. https://funcool.github.io/promesa/latest/index.html also recently added support for loom virtual theads.

just work(tm) is the aim of it, Jetty 10 has experimental support for it

ah ok, i’m using http-kit btw

we run an older VM atm, but we ’re thinking of upgrading most of our infrastructure, so this is good to consider

re: blocking txns, being very pedantic, id do it in a separate service and connect it via a queue. return an id of the txn, allowing the caller to check status asynchronously

that way the caller never blocks and you can handle failures gracefully

I second the queue thing but was missing the part with a checkable ID

yeah, trade off being more involved code

and you mean, separate service, as in, separate machine/app right?

separate process even

But gain is f&f and non-race-conditional behaviour, no overgrowing thread pools

because, if it’d be in the same app

yeah its a bit more code but has much more reliability guarantees

then the problem remains? or will the queueing system manage its own threads making sure it’s not clogging the system

I agree, it ’s probably the best direction to take

The service can be another app or just another thread

id off load it to something like rabbitmq or sqs

ah yeah i figured that yes

could it be done inside the application, with ie. some queueing lib that manages its own resources?

the clojure way de complect to simpler parts 😄

yeah i know… but the code of the doseq is so simple 😄

or is it easy

it’s probably easy

its easy

haha

dang

before doing it all, id say deploy the app, gather some metrics and measurements, both human and machine and then have specific actions?

something like https://github.com/layerware/pgqueue comes to mind as starters

At work, we do this all the time. We use a queue to manage these “jobs”. In your example, the whole doseq form would be a job. https://github.com/msolli/proletarian is the job worker system we use (only for Postgres, though).

{:foo2 {:foo :bar}, :foo nil} is what I get from the second, which looks right

are you referring to something other than clojure.core/merge ?

Yeah, same:

Clojure 1.11.1
user=> (merge {:foo2 {:foo :bar}} {:foo nil})
{:foo2 {:foo :bar}, :foo nil}

Dang, I had a typo, sorry:

(merge {:foo2 {:foo :bar}} {:foo2 nil})
;;=> {:foo2 nil}

It's not a deep merge

There are a couple things going on here. First is that merge is not a deep merge. It's just a single map being merged into another single map. You might be looking for deep-merge , which can be found for instance in medley. Second is that nil is not an empty map, so even with a deep merge, that nil will override whatever is in :foo2 ... this can be worked around by using empty maps instead.

Alright, in that case why does this work?

(merge {:foo :bar} nil)
;;=> {:foo :bar}

Basically because merge takes a map and merges other maps into it. Clobbering the original map with a stray nil would be decidedly impractical.

Alright, so some minor inconsistency for much greater practical value. Makes sense and I can definitely live with that! To give you some context, we actually have a deep-merge at work that I want to modify slightly and I was trying to understand why merge was designed like this first.

I don’t think it is inconsistent, a map is also a sequence of map entries, which nil is an empty sequence of

“…java-osx@” I assume, that you are using a mac? If so is it an ARM-based M1? I had trouble in the past with some libraries. For some libraries I needed a x86 Java

learning maven, seems there is this packaging thing: https://maven.apache.org/pom.html#packaging

this is intel mac, though I have arm one too

Is the sqlite binary for arm or x86 architecture?

ok its 8 years old. So probably x86

libsqlite4java-osx is for x86, there's libsqlite4java-osx-arm64/aarch64 for arm

putting

<dependency>
       <groupId>com.amazonaws</groupId>
       <artifactId>DynamoDBLocal</artifactId>
       <version>1.20.0</version>
    </dependency>
...
  <repositories>
    <repository>
      <id>dynamodb-local-oregon</id>
      <name>DynamoDB Local Release Repository</name>
      <url></url>
    </repository>
  </repositories>

0% tree ~/.m2/repository/com/almworks/sqlite4java/libsqlite4java-osx
/Users/kimmoko/.m2/repository/com/almworks/sqlite4java/libsqlite4java-osx
└── 1.0.392
    ├── _remote.repositories
    ├── libsqlite4java-osx-1.0.392.dylib
    ├── libsqlite4java-osx-1.0.392.dylib.lastUpdated
    ├── libsqlite4java-osx-1.0.392.dylib.sha1
    ├── libsqlite4java-osx-1.0.392.pom
    ├── libsqlite4java-osx-1.0.392.pom.lastUpdated
    └── libsqlite4java-osx-1.0.392.pom.sha1

but a deps.edn thing doesn't download native libs

0% cat deps.edn
{:deps {com.amazonaws/DynamoDBLocal {:mvn/version "1.20.0"}}
 :mvn/repos {"dynamodb-local-oregon" {:url ""}}}
0% clj
Downloading: com/almworks/sqlite4java/sqlite4java-win32-x64/1.0.392/sqlite4java-win32-x64-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/libsqlite4java-osx/1.0.392/libsqlite4java-osx-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/sqlite4java-win32-x86/1.0.392/sqlite4java-win32-x86-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/libsqlite4java-linux-amd64/1.0.392/libsqlite4java-linux-amd64-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/libsqlite4java-linux-i386/1.0.392/libsqlite4java-linux-i386-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/sqlite4java/1.0.392/sqlite4java-1.0.392.pom from central
Downloading: com/almworks/sqlite4java/sqlite4java/1.0.392/sqlite4java-1.0.392.jar from central
Clojure 1.11.1
user=>
0% tree ~/.m2/repository/com/almworks/sqlite4java/libsqlite4java-osx
/Users/kimmoko/.m2/repository/com/almworks/sqlite4java/libsqlite4java-osx
└── 1.0.392
    ├── _remote.repositories
    ├── libsqlite4java-osx-1.0.392.pom
    └── libsqlite4java-osx-1.0.392.pom.sha1

1 directory, 3 files

so maybe something interesting going on with that packacing thing

maybe I should ask this in #C6QH853H8

ah, it's been asked once https://clojurians.slack.com/archives/C6QH853H8/p1582290885135600

Tdeps only supports jar packaging right now

Would recommend checking out some other popular languages websites and seeing what ideas we can borrow from them e.g. • https://www.python.org/ • https://go.dev/ • https://www.scala-lang.org/ • https://www.rust-lang.org/ A few ideas that I think would be good: • Show some example code (I clicked around and didn't find any very easily!) • Demonstrate that it is a successful and growing language • Show that it is easy to learn • Demonstrate that Clojure is the future of programming and programmer productivity • Demonstrate that is is widely used and a good choice for teams that want to get ahead of their competition. • Compare it to other languages and show why it is better. • Show off the cool libraries and things built with Clojure

I don't think we need to open yet another thread on this. we've already done this analysis, we have a big list of ideas and todos for the front matter, some that have been acted on, some that will be worked on as we have time. I don't think it's easy for people outside the core team to work on this as it really requires direct collaboration (and we're mostly on holiday this week and working on other things right now)

Okay Alex thanks for the update, sorry I missed that thread. I focus on the Practicalli TODO list instead. Thanks.

the argument passed to select-keys is called keyseq - it can be anything seqable

user=> (select-keys {\a 1 \b 2} "ab")
{\a 1, \b 2}

that is so sick

I also keep forgetting vecs are associative 😅

(let [word "hello"]

(select-keys (vec word) (filter even? (range (count word)))))

Keywords are interned so you’re keeping memory for anything you make

They are reclaimed if unreferenced and under memory pressure but…. if you’re in that situation, you’re probably in bad shape already

And keywords that are millions of chars long are really hard to remember and type.

@alexmiller: is this what you mean with interned: “For keywords, hash values are calculated and cached when the keyword is first constructed. When looking up a keyword as a hash key, it simply returns the precomputed hashed value.” ?(quoted from: https://stackoverflow.com/a/31503297/1204047 ) The keywords are used as codes/values of expressions for what I am trying to do, so it may actually help to have them cached and be able to do fast equality checks, although they would be mostly in the range of 10k chars and very rarely >1 Mio.

It seems like a very strange usecase for keywords to be long. The intended usecase for keywords is to act as names for things to be referenced in your code. What are the aspects of keywords that you need for your project that wouldn't be handled just as effectively by strings and an LRU cache from core.cache?

By interning, I mean caching. But this seems like an abuse of the keyword cache and likely to eventually leave you someplace bad. If you want a cache or an object pool, I think you’d be way better off making one explicitly so you could control it

Keyword equality checks are fast because they are identity checks, but that’s a property you can also get with other jvm values. String interning is likely also relevant here and string deduplication in jvm gc’s may help you but there a lot of factors likely to influence that

Yeah I had a feeling that this may not be a good idea after all, so I guess strings may be the safer choice. The codes basically look like this: :NUIMNNIINUNUNNNNNUIMNUIMNUNUNUNUNUIMNNIINUIMNNIINUIMNUIMNUIMNUIM and most of them have a length of just 1 to 256 chars, but they can grow exponentially in 4^n. I have different functions that break them apart, transform or permute them for nested maps that act as value tables (which made me think about using keywords), embed them in Hiccup-like expressions and so on.

You may actually want to look at something like https://github.com/IGJoshua/ropes or clojure's vector-of :char

Depending on the exact operations that are performed

ropes allow you to have a low memory footprint and high performance when your primary operations are splitting and re-combining sections of these long strings

clojure's vector-of produces vectors that follow clojure's semantics for vectors but with a primitive representation to reduce memory usage and improve performance.

Thanks for the suggestion, I will take a look at that! I am fairly new to data structures since I am self-thought and only recently went deeper into computer science, so I am also in permanent risk of premature optimization due to lack of experience. :)

The ropes library doesn't have a formal public release or exported docs yet because I was planning on doing more with it before then, but it's in a robust enough state to at least be used for reference. I recommend them over vectors when you're doing large numbers of concatenations, splits, and other similar operations. You can depend on it with a git dependency.

Unfortunately I haven't found any other libraries that provide the same functionality for Clojure as the only other ropes library is byte-ropes only.

Seems like a good fit. I have seen in the GH issues that you are interested in porting it to cljc and since this is what I am using for my project (I plan to have compatibility with ClojureScript) I can only experiment with it for now and maybe use it later if you are still interested and have the time to implement cljc support.

Yeah, the main issue that I have with CLJS support so far is that I don't know off the top of my head how to ensure that CLJS regexes will work with them. Knowing that someone is interested in ropes however can help push it up my priority list and get it CLJS support.

Ah, at least from my research so far it is not possible to implement Regex compatibility in JavaScript with ropes, which means that the cljs variant will be somewhat hamstrung in terms of functionality as it will require you coerce it to a string before running regex functions over it.

Oh, that is unfortunate. However, I think I’ll only consider such specialized data structures when my library reaches a point where it actually matters and then I may focus on Java for such performance-critical tasks anyway.

When I think about it, maybe instead of strings it would be better in my case to use vectors of keywords, since the codes I represent only consist of 4 distinct values, so something like [ :N :U :I :M … ] could maybe take advantage of caching while only having to store these 4 values. And for the transformations I need vectors anyway, so no need to convert from strings all the time…

That may be better, yes. This is actually also a supported usecase for my ropes library, but as you said it needs cljs support for your project.

If you have a limited number of symbols (it looks like you have only four symbols = 2bits per symbol), and are under memory pressure, maybe you should consider using one byte per symbol (which is wasteful, but anyway) or encode the symbols in bytes, or, my favourite - longs. But with this encoding you would have to mark how long the ”string” of symbols is in the long(or byte) if they are not the full length of the container. Then you can make concatenations with bit shifts/rotates and and-operations. Fun but quite elaborate.

Half way there would be to encode the symbol string in a bigint, but ensure that there is always a largest bit marking the end of the string that should not be part of the string itself

The initial message in this thread had me thinking “Ah yes, we’ve made all sorts of other databases, why not KeywordDb?

Why use files when a single keyword can do the job? 🙂

have you set the --enable-preview JVM flag on your program?

Yes I have, I used a new one just now clojure:temurin-19-tools-deps-jammy and it worked!

Thanks for your time

Glad it worked!

fyi VirtualThread is not a public class, and code shouldn't directly try to load it

I am not, I am using a virtual thread factory

ok, cool

The error might be related with an import done automatically when i was trying to understand how everything worked