Fork me on GitHub

I'm confused about when to use what style of namespaced keys. I know how they work, I just don't know what is best from a maintainability standpoint. For example is :user/email a good namespaced key? Or should it be namespaced to the company such as Or perhaps namespaced to the project or even a project file namespace such as Should any way be used exclusively, or are there use cases where either style might be used? I'm thinking that :user/email still might result in naming collisions if two systems deal with user in a different way. I'm hoping to figure out an optimal solution for systems spanning many projects in an organisation.


I think I remember a Rich Hickey talk on this but I don't remember which one


project level is pretty good, I don't care very much for using the names of code namespaces for keyword namespaces


I think a good rule of thumb is to imagine all your data that exists in memory exists in a sql database, the namespaces would be the table names and the names would be the column names


and then imagine everyone's data is in the same global table space, and what would you need to name your tables to avoid collisions


When you say everyone's data, are you referring to company level namespace like


that would be one possible solution to that, prefixing all your table names with some string that you "own"


another solution would be using uuids, which would work but are not very friendly


Another factor to consider @caleb.macdonaldblack is how "global" your data is going to be. Is this just a map of data that will exist inside your company? Inside just this project? How unique does the name need to be?


If you produce it from a library you put out there for everyone to use, it needs to be "more unique" so that it can't conflict with any similar data used by any of the users of your library.


I know I've used ::


And the issue was that it coupled our key names with the namespace


So I'd say be careful about that


Right so it looks like there is no real convention around this. And I should just go with what makes the most sense given the factors.


Some examples: If I need a db uri for my users db in my company I could namespace like this :db.users/uri. I wouldn't want to use :db/uri as I might have many databases in my company. I also didn't namespace to the company because this key will be accessed only within the company And I could use :user/email if users in my company are global among all projects. But if my company had two products I would namespace the user attributes accordingly such as :project1.user/email & :project2.user/email but only if some of the code was shared in any way.


That's where my thinking is after considering the feedback


This issue of keyword namespaces is very confusing to me. What advantage is there to making the keyword completely unique across, for example, the organization? In other words, what harm is there in using :user/email for the email column in two different DBs in an organization? Is the keyword alone really used to determine which table an attribute belongs to?


Maybe :person/email would be an even better example.


@mark540 If there was 2 databases for users (employees, clients) then the business logic for dealing with the two types of users would probably be different. And the data is stored is two different places. To me this because of the 2 datasources, I would think to namespace to two separate namespaces. The namespace gives me the ability to inspect my data and decide if I want to do something with a client or an employee. And I can tell the difference because of the namespace. If there was no difference, then there would be no reason to store client & employee separately or differently (with some technical exceptions such as security). Because they're essentially the same I would namespace them the same. I'm still working all this out but that's my thinking.


I understand what you're saying. I'm just surprised that there wouldn't always be context available (client vs employee) and that the keyword alone must be unique to get the context.


Yea that's something I've though about also. For example the namespaces seem redundant for this data structure {:orgs [{:org/employees [{:employee/paychecks [{:paycheck/id 1}]}]}]} because each level up contains the context. Perhaps the namespace allows you to avoid passing in the context or more information than that is needed. For a paycheck/id=1 instead of giving {:employee/paychecks [{:paycheck/id 1}, you just give {:paycheck/id 1} and the namespace allows you to determine the context.


I think you must be right.


But it would be interesting to know what specific benefits others have found for the unique keywords.


Yea I'd like to know also. I've never written code like and I'd like to know if others how found it practical for production use


The most practical scenarios I can think of is dynamic spec validation. It could just lookup the spec for all its keys and avoid knowing any context apart from the keys


I've never done that though so I'm not sure how if at all it would work


Or even if it's a good idea. I feel like you would want to explicitly specify what spec you're validating against


OTOH, a spec for :person/email would probably work for email addresses in all tables.


However you could still do that with unique keys and some mapping.


But your thinking raises another question for me. So far I've assumed that the namespace of keys in a map would be the same. Are there any use cases for them to be different? Perhaps what your suggesting might also be a correct way of doing this. {:person/id "123" :person/name "Caleb" :employee/number "001"}


As long as there would be no way that the information stored in the shared keys would need to be handled differently I would think


Yes, I've wondered about that. But I don't have enough experience in Clojure to say. Someone with more experience will probably chime in. I can admit that unique keywords give you the most flexibility for treating them differently, if necessary, just don't know specifically when it's necessary.


Yea I agree. How far is too far. Perhaps there are times when you would use them and times when you wouldn't.


When you're looking at nested data structures, remember that sometimes you'll just be passing part of it around -- so you don't always have the parental context.


Qualified keys are designed to "avoid conflicts" and to "uniquely identify an entity in a given context". So they really are about context.


For example, at work, we have a billing subsystem and we generally have all the keys in maps associated with billing qualified with wsbilling. The context we need to convey is that "this data all belongs to billing", so the maps within that system tend to have keys like :wsbilling/member-id, :wsbilling/initial-amount and so on -- unique within the context of billing and the systems that it interacts with.


Another part of the system deals with multiple "layers" of domain logic so it uses qualifiers that identify the different layers.


Right that makes sense. And expanding upon mixing namespaces in a map as I previously suggested. Looking back on work with datomic, your entities wouldn't contain maps with mixed namespaced keys. They would nest them under a ref.


Some of these qualifiers match actual namespaces, some don't.


So perhaps the second nesting would be better than mixing namespaces


The idea is you use whatever qualifiers make sense for your data in your context. Making these "as unique as they need to be".


@seancorfield And would the use of qualifiers matching actual namespaces be almost exclusively used for keywords private to the namespace? Are there other reasons you might do this?


(which is why there are no specific rules of thumb about this whole thing 🙂 )


Yea Clojure is very interesting in that regard. It's like the complete opposite of something opinionated like ruby on rails. Everyone sort of does what works for them. It does make it a little difficult if you're new however as its hard to copy what other people do because everyone does it so differently. I imagine it's an experienced developers dream though as they can do everything exactly how they like it.


It means you need to understand more of the abstractions and concepts, yes.

Robert A. Randolph04:01:50

@caleb.macdonaldblack It's more so that people program with abstractions, and the concrete code that comes out varies


Thanks Sean, I see your point about nested data structures. And it is good to know that there is no standard convention. Another question: If you have both :employee/email and :client/email, don't you sometimes want to treat all emails the same or similarly, and in that case do you use the keyword name to identify it as an email? Just curious if you'd encountered that situation.


@audiolabs That blows my mind a little bit. When deciding on how to implement something I've usually started by figuring out what worked for me in the past or finding out what other people have done. Pretty much just lots of practice. Perhaps to improve myself I should spend some more time reading up on the theory.

Robert A. Randolph04:01:20

@caleb.macdonaldblack It's not so much about theory, as much as it's about understanding 'what needs to happen'


@mark540 If you've named them :employee/email and :client/email it'll generally be because you want to be able to tell them apart in a particular context. If you want to do something email-generic, you can pass either value to a function that just takes "an email address" or you can construct a new map containing the bits you might want under more generic names.


When you get into specs, you also have to consider whether these fields have the same underlying spec or not.

✔️ 5
Robert A. Randolph04:01:30

@caleb.macdonaldblack I find that when programming in Lisps, especially clojure, I rarely need to think about writing code. I spend most of my time thinking about architecture or specification. I find this also leads to better applications.


For example, your spec for :employee/email might be a regex that has a fixed @ domain name part (for your company).


@caleb.macdonaldblack Clojure certainly rewards thinking at a higher level of abstraction a lot of the time... and can punish thinking at a lower level sometimes 🙂

Robert A. Randolph04:01:15

@seancorfield I feel like that's a bit unfair, because the flexibility of the language allows you to do far-reaching optimizations with lower-level code changes. Perhaps more accurate to say that Clojure punishes premature optimization more than other languages?


My takeaway from this conversation is that I am free to use keyword namespaces for whatever purpose is most suited to my app, which turns the focus back to the app data model. It's a nice freedom. Thanks everyone for the comments and help.

✔️ 5

Likewise thank you for the help


@audiolabs OK, yeah, that's what I was reaching for really.


But also if you try to dive in and think about problems the way you might in other languages -- where mutable variables and loops are the basic building blocks -- you'll find Clojure working against you.


So you often have to step back and think in terms of collections rather than elements, and abstractions rather than implementation, in order to get the "shape" of a Clojure solution.

✔️ 15

wondering which is more intuitive for people to read, dispatching async work in future or async/thread , any opinions ? i don't really care for results that much but i need to dispatch a bunch of i/o bound threads and wait for all of them to finish


and count of threads is likely to be bigger than core count because i/o has latency 😞


it appears they both easily scale beyond the count of cores available


I am doing some timeseries analytics with clojure, and I have noticed some strange performance issues in loading data. I load timeseries from CSV files (size 500KB each), and I find that slurping data in takes 10ms, and then the CSV parsing takes 100ms (I parse timestamp, and floats). I compared this to incanter csv load (which takes 300ms, so worse). I used taoensso.tufte for performance tracking. I wonder if there are any ideashow to improve performance of such stimple things? If reading from disk takes 10ms, then the parsing should be perhaps another 10ms, but not longer, as disks typically are the bottomleg. I spend a lot of time just to get simple benchmarks, because I had to remove all the lazyness via "doall". Any ideas on how to speed Clojure up? Any tricks or recommendations?


did you measure the timings before and after jit got kicked in as well ?


I assume the Java JIT runs the first time a function is called.


I ran my performance numbers many times.


So I dont think this is the culprit.


they dont get compiled on the first time


that number is either in hundreds or thousands before java kicks it's jit into motion


but if you had files of 500kb then yes it should have kicked in at some point


can you try to isolate the slowness part by excluding parts of parsing ?


date parsing does sound a little bit of annoying , so i would try to avoid that at first


I now track everything


if it turns out to be the slowpoint you'll figure out what to optimize


But I still miss about 50% of the total time.


how many rows do you have in those 500kb ? and how many columns ?


sorry, I could not post to this thread my performance snippet.


What is really bizarre that the slurping is so fast,


and the parsing so slow.


It is order of magnitudes opposite how it should be.


this is the source


i tried this over here


generating a csv file with

for((i=0; i < 25000; i++)); do echo "`date -Iseconds`,$i.25,\"anytext\""; done > somecsv.csv


and then tried this code to read the result:

(defn do-parsing[]
  (let [data (slurp "somecsv.csv")]
    (last (csv/read-csv data))))

(defn time-parsing []
  (time (do-parsing)))

; (time-parsing)


indeed seems like feeding it into the read-csv added 200ms of execution time


bizarre, isnt it?


I thought that perhaps the clojure csv library is somehow not performance optimized,


So this is why I tried incanter.


But incanter is even slower.


i would rather look for a pure java csv parser and wrap it into something comfy


it looks to me that already splitting the lines apart by linebreaks is slower than slurping it from the disk


i tried to parallelize the work

(defn do-parallel-parsing[]
  (let [data (slurp "somecsv.csv")
        rows (clojure.string/split-lines data)
        parsed-rows (pmap csv/read-csv rows)]
    (last parsed-rows)))

(defn time-parallel-parsing []
  (time (do-parallel-parsing)))

; (time-parallel-parsing)


and that was a bit faster but still slow by all means 🙂


and had a chance of introducing bugs due to linebreaks within cell values


perhaps nobody every uses csv files in clojure


so this might be the reason.


but even the split lines here took the execution time to 50ms already before any csv action


line splitting can be a performance cost,


people use csv for sure 🙂 but yeah probably they don't go after the milliseconds but have more slowness somewhere else


if it is not done correctly.


because one has to create thousands of sub strings


and this is expensive.


my thinking is,


that csv parsing should be a fraction of the time it takes to read it from disk.


I have written a library in c# for .net


that had that performance.


so something is really wrong here.


But I have no clue what.


I am too much novice of clojure to find that out.


i tried to use java's own string splitting .. yes it's in a class of it's own


When I wrote my c# csv parser,


(defn do-parallel-parsing[]
  (let [data (slurp "somecsv.csv")
        rows (into [] (.split data "\n"))
        parsed-rows (pmap csv/read-csv rows)]
    (last parsed-rows)))


just like that


I bascically created my own string class,


my example went from 100ms to 50-60 range


that allowed you to take sub-strings.


so I have avoided all the senseless copying of strings.


and in the parsing,


but now my son is requesting my attention ... good luck (and yeah. get an efficient java lib for the heavy loading)


I wrote my own parser,


by reading from my own datastructure


Thanks a lot kulminaator!


Happy new year!


btw. after giving the jvm a good chance at jit compiling the whole thing i reverted the code pretty much to original


looks like the repl is just getting into my way here 🙂


with code like this

(ns silly-csv-parser.core
  (:require [ :as csv])

(defn do-parsing[]
  (let [data (slurp "somecsv.csv")]
    (last (csv/read-csv data))))

(defn time-parsing [_]
  (time (do-parsing)))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (mapv time-parsing (range 1 25)))


running the (time (do-parsing)) from the repl was always like 200ms


however , lein uberjar and then running that jar was a wastly different result


$ lein uberjar
Compiling silly-csv-parser.core
$ java -jar target/uberjar/silly-csv-parser-0.1.0-SNAPSHOT-standalone.jar 
"Elapsed time: 301.065242 msecs"
"Elapsed time: 66.629456 msecs"
"Elapsed time: 112.770513 msecs"
"Elapsed time: 46.826564 msecs"
"Elapsed time: 54.771694 msecs"
"Elapsed time: 47.163652 msecs"
"Elapsed time: 45.761238 msecs"
"Elapsed time: 46.835782 msecs"
"Elapsed time: 49.504536 msecs"
"Elapsed time: 37.032187 msecs"
"Elapsed time: 37.907508 msecs"
"Elapsed time: 44.113467 msecs"
"Elapsed time: 34.527459 msecs"
"Elapsed time: 34.381777 msecs"
"Elapsed time: 35.217125 msecs"
"Elapsed time: 34.771945 msecs"
"Elapsed time: 40.187299 msecs"
"Elapsed time: 34.952462 msecs"
"Elapsed time: 35.637608 msecs"
"Elapsed time: 34.520076 msecs"
"Elapsed time: 35.482058 msecs"
"Elapsed time: 34.636292 msecs"
"Elapsed time: 35.011732 msecs"
"Elapsed time: 35.886221 msecs"


you can see pretty well where the jit and proper memory management seem to kick in 🙂


I thought that the REPL compiles the same way how it would be compiled to a JAR. I don't get it...


But thank you so much! I honestly would never have believed that such great differences possibly could exist between repl and jar!! 👍❤️


5000 rows, 6 cols


Hi! I am in a bit of a pickle. I would like to use generative testing as part of my test suit and to this end wrote the following test:

(deftest generative
  (is (successful? (stest/check `term->ast))))
When running this, however, I get the following error:
Exception: java.util.concurrent.ExecutionException: Syntax error compiling at (clojure/test/check/clojure_test.cljc:95:1).
This only happens if the assertion fails; if it passes everything is fine. Any idea why?


That exception comes from defspec in clojure.test.check.clojure-test -- which doesn't match the code you shared @U5ZAJ15P0


@seancorfield I am most definitely running deftest; I just tried it again after clearing the cache :thinking_face:


What is the rest of the stack trace?


perhaps the fact that I am using kaocha as a test runner is the issue :thinking_face:


nah it’s not; I am getting the same error with vanilla lein test


There is a lot of machinery in that stack trace beyond clojure.test and I see you're also using respeced.test$successful_QMARK_.invokeStatic (test.cljc:104) which seems to be what is actually leading to the error here...


@seancorfield problem solved in the end; it was an issue with lein test monkey-patching clojure.test :face_with_rolling_eyes:


It's not compatible with respeced?


@seancorfield if you are interested (thanks to @U04V15CAJ for finding this). The issue is tangential to respeced.


Also test.check v0.10.0-alpha3 doesn’t seem to have the issue


respeced has nothing to do with it, it’s a clojure.test.check + lein issue


ah yes, tangential wasn’t the right choice of word; I meant they were unrelated 🙂


well, they are related, so I get it. but the problem is really with lein’s monkey patching I guess


@U04V15CAJ Cool. Was just curious when I saw it in the stacktrace -- I hadn't looked at the code.


@U04V15CAJ do you use test.check v`0.10.0` yourself?


no, but I haven’t used lein for testing for a long while


Lein's monkey-patching has caused problems with other tooling in the past (and, lately, ~/.lein/profiles.clj seems to be the bane of every beginner's life).


(also haven't used Leiningen for a long time -- switched to Boot three years ago and switched to clj/`deps.edn` this year)


I’m going along the same path. For work we’re using boot still, but for newer personal projects I tend to choose TADA


We were very heavy Boot users until this year. We just started having too many problems with it 😞


being able to use git and local deps really makes a difference


we don’t have many problems with it at work, so no need to migrate for now.


@seancorfield I wanted to use Boot but there was a nice template project for Fulcro + shadow-cljs with lein


@seancorfield what do you use now? deps.edn?


See 14 minutes ago in this thread 🙂 "(also haven't used Leiningen for a long time -- switched to Boot three years ago and switched to clj/deps.edn this year)"


So we run tests with Cognitect's test-runner


I use that one also, if I’m not using my own test runner script.

Kari Marttila14:01:15

I really love Clojure and Clojure REPL is the most productive environment I have ever used. Now I encountered one puzzle with Clojure REPL that I can't figure out how to do it. I needed to create a couple of gen-classes for a Java API I'm using via Clojure/Java interop. Before those gen-classes using Clojure REPL was a breeze. If I wanted to refresh all namespaces in Clojure REPL I just called function (do (require '[ :refer [refresh]]) (refresh)) ... in Clojure REPL. But after implementing those gen-classes I get error: "namespace 'mygenclass' not found after loading 'mygenclass". I was wondering if there is some standard procedure or best practice with gen-classes I have missed?


@kari.marttila By the way, which REPL are you using? Are you using :gen-class in the ns declaration or a free-standing (gen-class)? Anyway, I do not know c.t.n.repl, but I sometimes see that error message, in Cider, after trying to reload a ns that has errors in it OR trying to reload a ns whose name does not correspond to its filename.

Kari Marttila15:01:25

I'm using Cursive REPL with IntelliJ IDEA. I really love Cursive. I have configured all kinds of hotkeys for IntelliJ/Cursive and using those hotkeys I can jump back and forth between editor and REPL, send S-tokens from editor to REPL etc. I really love it. This refresh issue with gen-class is the only thing that causes a bit of nuisance.

Kari Marttila15:01:25

I'm using :gen-class in the ns declaration. Example in comment thread.

Kari Marttila15:01:42

Example: (ns simpleserver.util.azuregenclass.session (:import ( TableServiceEntity)) (:gen-class :extends :constructors {[] []} :init init :prefix "bean-" :state state)) (defn bean-init ([] ))


IIRC, this is related to, and would be fixed if that patch was applied


Nicola would know for sure

Kari Marttila17:01:02

Ok. Thanks for info. Let's hope that patch will fix this issue some day. 🙂


went and upvoted the issue, since I happened to stumble on genclassing lately too


more upvotes might help 🙂

Kari Marttila18:01:02

I signed in to Clojure JIRA and upvoted as well. 🙂


could anyone provide some pointers on a macro I’m trying to write? I want a function or macro that I can call with a map of let bindings (symbols to values) and then execute some form in the body of that let. I have it working for some cases but not all.


you cannot do that in the general case


you are trying to build a structure at compile time that depends on a value at runtime


hm is there no way to achieve the same goal? eval some code with different values for particular symbols as supplied by some map?


you're existing implementation is also broken for the case were it "works" because it ends up double evaling


what you are describing is a function


code parameterized by different arguments


lol. yeah in abstract I guess. in this case I’m taking code as input and want to replace the meaning of some things using data. Should I just be walking the form and making replacements instead of trying to keep the same symbols but assign different values than what they’d otherwise resolve to?


I guess maybe that’s better

hiredman19:01:17 is a library that might be helpful (I've never used it, I would be more inclined to write an interpreter for a dsl then to try and transform clojure expressions and call eval on them)


thanks, I’ll take a look