2. Beyond installing Node.JS (and NPM), the guide suggests just running the following to get shadow.cljs:

npx create-cljs-project my-project

A bit farther down I see "If https://shadow-cljs.github.io/docs/UsersGuide.html#_installation globally, you can use the shadow-cljs command directly.

$ shadow-cljs help

So...it really seems like I should do that global installation step.

If you don't go global, you can just use npx shadow-cljs help

I had the same confusion in that guide

What are the advantages / disadvantages of npx v global?

I was confused as I'd never used npm or NodeJS before

I usually install it as a dev dependency in any project

Well the big disadvantage of global is that if you work in various projects, they may rely on a specific version of shadow-cljs

I do see the suggestion for npx shadow-cljs help. but if I run that I get:....

npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/shadow.cljs - Not found npm ERR! 404 npm ERR! 404 'shadow.cljs@latest' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\Ady\AppData\Roaming\npm-cache\logs\2020-05-23T2358_07_860Z-debug.log Install for [ 'shadow.cljs@latest' ] failed with code 1

I think you typed shadow.cljs not shadow-cljs

Ahhh...thanks.

but still...the entire guide is written assuming the 'global installation', so...sounds like I should.

It's up to you, if that would make it easier to follow along it's worth it.

Global is more convenient, and for a tool you want to install, I feel it makes sense. You can re-install things as local later if you need a specific version for a specific project I believe

Makes sense. I will. I had a 3rd question...now what was it?...

Oh yeah...

3. The http://clojurescript.org quick start guide reads that you need Java 8 and teh cljs.jar file

I asked here where I supposed to put the cljs.jar file, and someone told me I actually shouldn't download it.

So I skipped that and by the time I had installed Node.JS (and NPM) and shadow.cljs (by creating a project, so not actually installed), I kind of figured the cljs.jar file would have appeared somewhere by now.

I did a search in the project folder, though, and it's not there. So...what am I supposed to do? Just ignore the fact that I do not have what the main clojurescript website tells me I need? ...or just download it anyway, and if so put it where?

Wait, are you following both the http://clojurescript.org tutorial and the shadow-cljs tutorial?

I was going to follow the http://clojurescript.org quick start guide (which I did last year on a different computer) just to refresh my memory about project folder set-up and compiling etc.,...until I was convinced by people here just to use shadow.cljs

I had barely started into the http://clojurescript.org guide, just installing JDK 8 (actually adoptOpenJDK 8 ). Did I not even need to do that??

JDK 8+ is required for both shadow and the http://clojurescript.org tutorial. However, the http://clojurescript.org tutorial is only the minimal working setup. Meaning you wont be able to connect cider (or most editors) to the REPL out of the box.

Shadow-cljs is nice because it can grow with you. The defaults are good for getting your feet wet, but if you decide to use it for more serious projects you'll be able to work with it and reap benefits of what it can do. The http://clojurescript.org tutorial may get to the point of writing cljs -> js but you'll likely soon outgrow it.

I'd love to know how Cognitect folks build cljs projects -- what tooling they use?

I think the getting started guide might be outdated for windows. Since then, both lein and tools.deps can run on windows and pull in ClojureScript for you, and I'd assume shadow as well

@didibus There's no official Windows solution, hence the official docs must still suggest a more manual approach.

(for what it is worth, I've run the cljs quick start guide on Windows Powershell using the CLI tools in their alpha state and it is possible)

Hmm I do have access to a virtual windows machine I use for gaming. I'm going to try installing shadow-cljs on it and see what I run into.

Ya that's fair. I guess lein doesn't count as official, and tools.deps powershell is alpha. My favourite on windows by the way is using the bb port https://github.com/borkdude/deps.clj

@jayzawrotny @didibus @seancorfield Sorry. got sidetracked there. Eccentric J, good to know that I shouldn't try to do both. I actually hadn't decided what I was going to do next yet. Still slogging through set-up it seems. Good to know there's one thing (the http://clojurescript.org tutorial) I can take off my list, and I should just keep going down the shadow.cljs tutorial page. Didibus, if I remember correctly the http://clojurescript.org quick start guide uses Leiningen.

It does not use lein

So can anyone explain the apparent lack of a cljs.jar file on my system (or where it might be), and what I need to do about it, if anything?

It doesn't use lein? Hmm. Well it was about a year ago I did it, so I guess after I did it I must have tried out Leiningen for some reason.

Both lein and deps will cache jars under ~/.m2/repository

I'm not using lein or deps. People recommended I use shadow.cljs instead.

(for CLJS, which I'm going to try for a while before I try Clojure)

Should I just not use the shadow.cljs guide if I'm on Windows, by the way? It seems to assume Linux.

I just tried to do the global installation as specified, and I get this:

npm install -g shadow.cljs npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/shadow.cljs - Not found npm ERR! 404 npm ERR! 404 'shadow.cljs@latest' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\[...]\AppData\Roaming\npm-cache\logs\2020-05-24T0228_55_872Z-debug.log

Actually if I go to the website https://registry.npmjs.org/shadow.cljs I get the same error.

It was pointed out earlier that it is shadow (dash) cljs not shadow (dot) cljs

I could have sworn I copied and pasted that. My mistake! Thanks.

no worries. if you want to see a presentation about how to use shadow-cljs, a sample app, and that app deployed on netlify check out https://github.com/dpsutton/asg-ignite-presentation

Thanks. Reading it now.

Still wondering about that maybe missing cljs.jar file, by the way...

Shouldn't it have installed with shadow?

I'm just going to run a search on my whole drive in the meantime.

shadow-cljs should handle that for you, the jar is not necessary. You should only be using the shadow-cljs CLI.

Wha..? No jar?

Did I also not need to install Java 8 then?

You do need Java 8

By the way, does it really have to be specifically 8, or can it be higher (I think 14 was suggested on a couple of JDK sites).

8 or 11 are recommended. others most probably work but how many battles do you want to fight

i think i'm using 13 with no issues.

11 does has a few small incompatabilities with various tools here and there and if you already have 8 installed I would just stick with it for now.

Oh I don't want to fight any extra battles. lol I just can't remember where I got the impression that the version number even mattered. Maybe a guide meantioned 8 but was written when 8 just happened to be the newest.

8 all the way. thanks.

You need at least version 8, that's all.

i thought version 10 never really worked for some reason? don't really know but i think most people recommend the long term versions which are 8 and 11

8, 11, 14 are safe choices for most Clojure stuff.

The inbetween versions are very short-lived now, the way Oracle is developing Java/JVM stuff so I personally would avoid them (and recommend beginners avoid them, in case they do cause problems).

We still have a few things on 8 but mostly we run everything on 11, although I'm developing on 14 now, mostly as a compatibility test for rolling all our systems up to 14 at some point.

Is there some kind of security argument for going with 14 instead of 8?

No.

alrighty then. 8 it is

At least, none that I know of. 8 is still actively supported by several companies and organizations. e.g., AdoptOpenJDK.

Ah...so 8 gets updated with lower-order version increments?

https://adoptopenjdk.net/ shows update 252 is the latest version of 8 they're offering.

OK. so lower-numbered versions are not necessarily older. Got it.

Well, JDK 8 as a whole is older than 9, 10, 11, 12, 13, 14 🙂 But each major version continues to get minor updates for security and buf fixes.

@dpsutton I'm reading through your guide/presentation, and...stop me if your intended audience isn't people who have only done a thing or two in JS and CLJS, but...I am a bit unclear about a couple of things already:

1. Your very first suggested command is 'mkdir app', but I have no clue what folder that should be made in.

Looking ahead to the next steps, of creating a few files...

2. There is quite a bit of (boilerplate) text to be put in the file, but no explanations of the text. Some of it (e.g. the html) I can follow, but it would be nice to know what some of it means.

More importantly, though...

https://shadow-cljs.github.io/docs/UsersGuide.html

(this is a lightning talk to get you up and running in 5 minutes. everything should seem sensible and is explained in the users guide)

3. After the "Create shadow-cljs.edn" step" there is a "Index.html" step. Should that be "Create Index.html"? ...and according to the little tree diagram should that be in a subfolder of the 'app' folder called 'public'?

Same question for the 'Hiccup' section, except it doesn't even have a file extension or a tree diagram to go by.

Hey, I'm actually going to try walking through that guide of yours now, @dpsutton. Thanks for the clarifications in the sidebar. Quick ultra-basic question (for anyone) before I start: Is there any issue with the top folder name (project name?) containing underscores or spaces? I just have noticed a-lot-of-hyphens-everywhere-in-project-names-in-multiple-online-guides.

The top folder is independent of the contents. Spaces in directory and file names seem almost always wrong to me

I mostly live in Windows at the moment, but having used Linux a bunch of times I typically use underscores all over the place instead of spaces in file and folder names (in Windows). I just remember Linux having trouble with spaces.

So...not actually asking if anything would choke on spaces, but actually that question about underscores.

The top level directory is independent of the project. It can be named whatever you want

Source directories and files need to use underscores due to requirements/limitations of the jvm

underscores or hyphens, right?

Underscores. Use hyphens in the namespace names but the files need underscores

😕 Of course it's different. OK thanks.

I am extremely unclear about the term 'namespace' by the way.

I've come across it so much during all the CLJS videos I've watched, but never with any explanation. Is it just variable scope, like within-procedure vs global?

https://clojure.org/reference/namespaces

read the reference first

That ref is hard to understand. I guess I'll wait until the topic comes up more organically during something I'm doing.

For now my guess is just that namespaces prevent two variables of the same name (maybe one in a library and one in a program using it) from being accessed at the wrong times instead of the other one.

@factorhengineering Every Clojure function, etc. is created within a namespace. The built-in Clojure functions are mostly defined in the namespace clojure.core. Some additional string-related ones are in a namespace called clojure.string. Typically when you start a library or application, you create one or more namespaces of your choice, (not clojure.core or anything else used by an existing library you want to use, but it is easy to avoid conflicts), and create your code in them.

And yes, the purpose of namespaces is to let you use whatever names you want within your own namespace, and not conflict with the same names chosen in the namespaces created by others.

thanks

Not everyone does this, but one way to ensure uniqueness of namespacee names is to take a DNS name like http://mydomain.com that you or the organization you work for owns, and use namespace names that begin with com.mydomain, e.g. com.mydomain.mynewlibrary

@dpsutton One more dumb question about your guide: When you say this file is in this folder:

src
└── app
    └── main.cljs

you don't know that

is the 2nd 'app' in the path meant to always be exactly 'app' in any project, or is it meant to be a repeat of the project name (the top level folder name)?

it could be FHE/src/app/main.cljs

I mean if I were following your guide exactly, i.e. with mkdir app

mkdir app

but this is an important concept called the classpath. by default src is the classpath root. clojure will look for the namespace app.main in a folder called app and a file called main.cljs from the classpath roots

In your case it's ...\app\src\app\main.cljs

so for a file to have the namespace app.main, it needs to be at src/app/main.cljs

so the first app is irrelevant. if that makes sense

but if the top level folder were foo, would it be \foo\src\app\main.cljs or \foo\src\foo\main.cljs ?

That second one would have a namespace of foo.main

Ah. I think you answered it above. it's the 1st one

yes?

yeah. the namespace and the hierarchy go hand in hand

"clojure will look for the namespace app.main" Got it.

and namespaces need hypens but filenames get underscores is the one place they diverge

almost always trips people up at least once

and just to be clear, I think I heard that namespaces CANNOT have underscores. It is also true that filenames CANNOT have hyphens, or can those have both?

I have to say, the namespace reference is terrible if you're trying to learn Clojure -- it talks about vars and mappings but it says nothing about project structure and how that relates to namespaces.

Namespaces can have underscores, as can file names. hyphens in Clojure namespace names are searched for in files that have hyphens replaced with underscores.

thanks @andy.fingerhut. I just tested it and namespaces can indeed have underscores but i've never seen anyone do this

It is unusual, certainly, but not illegal

(I've seen beginners do it "by accident" because they get confused about the rules)

Just to cover file names and folder names, is this correct: • file names can have underscores but not hyphens • folder names can have underscores but not hyphens • namespaces can have hyphens but not underscores (and really the hyphens and just replacements for underscores in the corresponding file names and folder names)

namespaces can have either

file names that have hyphens will not be found by Clojure when require'ing namespaces.

But don't namespaces essentially refer back to folder names and file names?

The file system will let you create such file names, of course, it just isn't productive for Clojure development 🙂

well yes. lol. just talking clojure here

This is maybe unnecessary info for you at this point, but you can create namespaces dynamically in Clojure that have no files on the file system to store them.

Namespace foo-bar.quux => filename <classpath>/foo_bar/quux.clj (or .cljc or .cljs

Don't worry about that too much right now, if it is overload of detail

(where <classpath> is going to be your src folder or your test folder typically -- for dev/test they're typically both considered, for production just the src folder, depending on how you set up your project)

^ What Sean said

@seancorfieldThat's what I was thinking. Good to see it in an example.

By the way, are the top level folder name and the project name synonymous. I'm guessing yes, but still...

i've never made a dynamic namespace. nor worked in a code base where it happened

not necessarily

notably, you can clone any repo into whatever foldername you want

for instance, git clone [email protected]:clojure/core.match.git custom-name will clone the core.match library as custom-name. The project will behave just as if it were saved into a directory named core.match

does top-level-folder-name = project-name ?

No, not necessarily.

no. as demonstrated above

OK, but usually?

unless you take specific steps to break that?

The "project name" comes into play if you are publishing a library to Clojars/Maven (a repository). Those have a group ID (like your company name or your GitHub username) and an artifact ID (the actual project name). e.g., fhe/my-project, seancorfield/clj-new. That doesn't have to correspond to anything in the source code but it will be part of the descriptor that is used to deploy the project to a repository. It's somewhat common to have the namespace structure of the project match the group/artifact tho'. So the fhe/my-project project would be in a top-level folder called my-project in which would be src and test folders (typically) and inside src you'd have fhe/my_project.clj with a namespace fhe.my-project -- does that help?

If you have the :new alias in your ~/.clojure/deps.edn file pointing to the clj-new library, you can see this in play by doing:

$ clojure -A:new app fhe/my-project

Ehh...maybe? Actually I don't get this part: inside src you'd have fhe/my_project.clj

Do you mean inside src you'd have fhe/my_project/src/my_project.clj

?

EDITING

no. I mean exactly what I say. Try the command above.

Ugh. This thing will not let me edit for some reason.

Trying to edit my post is flash-scrolling me hours back in the chat for some reason.

What I mean was:

See this:

seanc@DESKTOP-QU2UJ1N:~/clojure$ mkdir temp
seanc@DESKTOP-QU2UJ1N:~/clojure$ cd temp
seanc@DESKTOP-QU2UJ1N:~/clojure/temp$ clojure -A:new app fhe/my-project
Generating a project called my-project based on the 'app' template.
seanc@DESKTOP-QU2UJ1N:~/clojure/temp$ tree my-project/
my-project/
├── CHANGELOG.md
├── LICENSE
├── README.md
├── deps.edn
├── doc
│   └── intro.md
├── pom.xml
├── resources
├── src
│   └── fhe
│       └── my_project.clj
└── test
    └── fhe
        └── my_project_test.clj

6 directories, 8 files

And

seanc@DESKTOP-QU2UJ1N:~/clojure/temp$ cat my-project/src/fhe/my_project.clj
(ns fhe.my-project
  (:gen-class))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (println "Hello, World!"))

Actually I don't get this part: inside src you'd have fhe/my_project.clj Do you mean inside src you'd have my_project.clj ?

Is see the big tree you posted. So why is it repeating the top folder name in the subfolder of src??

It's a common convention, that's all.

Go back and read my explanation, then read the commands I showed and their output. Then go try that yourself.

Also, didn't we just talk above about the fact that the top folder name doesn't matter and is not repeated and clojure looks for app.main (meaning /app/main.cljs)?

I'm getting more confused now,.

Right, it is not related. It's just common practice for them to match.

Again, read what I said about project names vs namespace structure above.

The top-level folder name, the project "name" (group/artifact) and the namespace structure are all independent -- but it is common practice to connect them like this.

What I gather from that was that you could decouple top folder name from project name if you really wanted to.

clojure -A:new app fhe/my-project

Only the namespace structure and the file path structure (in src and test) are tied together.

I don't get the line above btw.

Why would you name your app a name with a slash in the middle??

Go read the clj-new project README where that is explained.

Libraries -- on Maven/Clojars -- have a group ID and an artifact ID and they are separated by a /

For example, org.clojure/clojure -- group ID org.clojure (a reverse domain name for ) and artifact ID clojure ... and org.clojure has several projects: clojure, clojure.string, java.jdbc etc.

@factorhengineering I have never seen a project or library name with a slash in the middle. Is there something specific that someone else said that leads you to believe they were recommending that?

@andy.fingerhut he's misreading the example I gave.

Yes, in seancorfield's example, he has:

clojure -A:new app fhe/my-project

So that would be group ID fhe and artifact ID my-project (by convention)

Why not just

clojure -A:new app my-project

That is explained the in README for clj-new. Please go and read it.

OK, but if one is not trying to produce a library, and is just working on smaller thing, that slash and 'group ID' would not be needed I guess.

It's a good practice to get into.

I have not even heard of clj-new before. I think it's down the reading list for me. 😕 Right now I'm just trying to correctly follow @dpsutton’s guide.

If your project name, top-level folder, and namespaces follow that convention, your code is less likely to conflict with code from another library.

THAT statement made complete sense to me.

I thought you had talked about clj-new the other day -- when you were asking about deps.edn and creating new projects using that...

If lots of people are uploading libraries there could be conflicts unless the namespaces are unique, so the convention is to have a unique identifier PART of the name (assigned to people or organizations) and a more descriptive PART of the name. That would make sense to me.

Libraries have group IDs and artifact IDs -- those must be unique.

so the 2nd part has to be unique too? That makes less sense to me

The combination must be unique.

If those are already unique why even have group IDs

OK, so the artifact IDs are NOT, on their own, unique. all good. ok

The group ID + artifact ID together must be unique. They are the "coordinates" of the library.

group-id/artifact-id -- in Leiningen's project.clj you see [group-id/artifact-id "1.2.3"] for dependencies... and in Clojure CLI / deps.edn it is written group-id/artifact-id {:mvn/version "1.2.3"}

As for whether you mentioned clj-new to me way back, if you did I didn't get that message. All I remember about deps..whatever it's called is that it was a built-in thing that does some of what Leiningen or shadow.cljs does.

Sean can correct me if I have this wrong, but I'm pretty sure that the group ID + artifact ID naming scheme was originated by Maven, a tool/system for publishing and distributing Java libraries, that Clojure/Java does not try to replace, but to build on, because it is common to want to use existing published Java libraries in Clojure/Java programs.

Inside a library you can have any namespaces -- but again, you want your code to not conflict with someone else's code so following group-id.artifact-id in general is a useful convention (for a file in src/group_id/artitact_id.clj)

Right, the two conventions are separate but essentially overlaid in order to keep code from conflicting, both at the library artifact level (group/artifact) and in the Clojure source (`group.artifact`).

OK. Makes sense. Thank you.

It would be... somewhat surprising... if you depended on a library as [foo/bar "1.2.3"] and then end up having (:require [quux.wibble]) to bring in its code... library names and namespaces are not usually that disconnected.

Now I feel like I need to pull myself out of this rabbit hole. lol. I mean I'm sure I'll use the info, just not right this second. i want to get through https://github.com/dpsutton/asg-ignite-presentation/blob/master/shadow-cljs.org and maybe https://shadow-cljs.github.io/docs/UsersGuide.html now and then finally actually make something.

I'm crying to actually write some of my own code at this point. lol. So...much...set-up.

I think it's important to understand the project basics (for Clojure) in order not to get lost in ClojureScript's ecosystem...

It's why I always recommend beginners learn basic Clojure stuff first and then move on to ClojureScript

(It's why I also recommend not trying to learn both Clojure and Emacs at the same time)

Otherwise you can get rather overwhelmed... as I think you have at times?

So many things to install, so many things to figure out about how to install things, and even with the tools installed this setting up a project or whatever is needed to actually get a cljs file for me to edit in front of me is even more confusing.

Yes, there are a lot of moving parts.

Clojure isn't "easy".

I understand your Emacs comment, but that bit of learning I willingly signed up for, having tried Emacs for a bit way back. At least it doesn't feel like pandora's box like this set-up stuff does.

Hmm...I guess I mean a Matroska doll

Actually, what I really mean is a hydra. lol

Emacs is all of those and more 🙂

I'm at peace with the file opening and saving and weird versions of copying and pasting in it now. I can just use it for that (and CIDER when I figure out what that exactly is) for the moment.

I first started using Emacs back in its 17.x and 18.x days... then moved on to other editors and when I learned Clojure (ten years ago) I was using TextMate! After a while I came back to Emacs (23.x I think) for a few years, then switched to Atom in 2015 and that's what I use now.

...and put off learning vim for the moment as well!

My knowledge of vim is pretty minimal. I know barely enough for basic editing.

All that investment in Emacs AND you do Clojure and STILL you moved away from it? That's sad to hear, from the perspective of someone who tried hard to identify the best programming language and editor to learn and picked Emacs as the editor. lol

About 80-90% of Clojure programmers used Emacs a decade ago when I first picked up Clojure -- but that percentage has dropped slowly each year since I think... I'd have to check the annual survey results...

It’s 45/45/10 or so emacs cursive and now vs code

Emacs was down to 45% in 2020.

But those are of course market share and hopefully the entire clojure user base got bigger :)

The 2010 survey suggested about 70% Emacs usage so I was slightly off above.

Oh well. I have tried org-mode and like it a lot. Good tie-breaker.

I never got into org-mode but I know people who absolutely love it.

I get the love but thankfully my favorite Notebook app, Notion, just became free for personal use.

Emacs dropped slightly to 43%, IntelliJ/Cursive rose slightly to 32%, and VS Code with Calva had the biggest increase to 10%.

Yeah, Atom is still very much a minority choice but I really do like it. And I love that the latest version of Chlorine (for Atom) lets me customize it via ClojureScript (using sci so there's no compile step and changes apply immediately).

Is there any way to export chat logs from this? I'm trying to get back and look at the answers about filename underscores (yes I'm already fuzzy on the details!), but this interface keeps unloading older chat messages. I hate that. It's all or mostly just text! Keep it all in memory!

All the chat here is mirrored to Zulip and also to the ClojureVerse log... I'll get you the URLs...

Zulip: https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/beginners

So...Zulip is easier (possible) to export from compared to Slack?

Is it better? Are you on that too? I don't really want to sign up for yet another thing. lol

ClojureVerse log https://clojurians-log.clojureverse.org/beginners

Yes, I'm on Zulip. It's much lower traffic than Slack. But it has unlimited archive/search.

Many channels here are archived to Zulip (it's one-way: what you post on Zulip does not appear here)

Some channels are mirrored to Matrix/Riot as well but I think that's more focused on mirroring IRC which still has hundreds of active users.

(and there's a Discord community, and Telegram and various others 🙂 )

FWIW, I login to Zulip using GitHub so it's not like I had to signup for anything new really.

Thanks for the log link. Much more searchable (though lists and probably other things are messed up by the logging.).

also the IRC tip. I have HexChat open already just to get general Emacs help. Didn't know I more clojure goodness was just a /join away.

Note sure if @dpsutton is still here, but I just ran this and got this, which doesn't lookequite right:

npm i react create-react-class react-dom shadow-cljs npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3. npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN [email protected] No description npm WARN [email protected] No repository field. + [email protected] + [email protected] + [email protected] + [email protected] added 119 packages from 191 contributors and audited 119 packages in 10.867s 1 package is looking for funding run npm fund for details found 0 vulnerabilities

There’s a warning but I’ve been ignoring it. The other warnings are warnings about your own package.json file. About no description and repository information. Completely benign

I have a beginner concurrency question. I'd like to update a Java array faster. Right now I do this:

(doseq [i (range (count pxs))]
        (aset-int pxs i (nth pxs-sorted i))
        (when (= (rem i 50000) 0)
          (println i)))
      (println "done setting pixels")

I would be interested in training my skills with clojure.test.check, and I have trouble to train my skills at writing properties. I read Clojure Applied from Alex Miller, watch the talks from the conj and from John Hughes at Clojure West and the latest one about pure function (how to specify it). Does anyone know if there is a repository (or something else) where there would be some faulty code and we would use test.check to write properties to find all the possible bugs?

i’m writing a macro that I want to accept a quoted list, e.g. '[Foo] , but because the macro is receiving the reader form (quote [Foo]) i’m iterating over it incorrectly. What would be right way to grab the list? I tried unquote but it gives me unbound function error, so doing this and it works (if (= (first tags) 'quote) (second tags) tags)

@alidcastano That is correct. ' is just a reader shorthand for (quote ,,,) so the form is actually read in as (quote [Foo]) -- macros do not evaluate their arguments.

You could just pass [Foo] and your macro would not evaluate it anyway.

(the first rule of Macro Club is Don't Write Macros -- so be sure that what you're trying to do can't already be written as a function instead of a macro)

Hi, looking for an advice: should I use clojure.spec or stick with plumatic/schema on new projects?

@yokoyama.km Personally, I'd recommend clojure.spec -- it is useful for a lot more things, in my opinion, and it's also "official" since it's from the core team.

Is spec2 compatible with specs.alpha?

We use it heavily at work and I wrote up the various different ways we use it https://corfield.org/blog/2019/09/13/using-spec/

Spec 2 isn't directly compatible with clojure.spec.alpha (aka Spec 1) but Spec 2 isn't ready for use yet

@seancorfield - thanks. This is a useful article

(ns me.specs-playground
  (:gen-class)
  (:require [clojure.spec.alpha :as s]))

(s/fdef print-digit
  :args (s/cat :just-an-int int?))
(defn print-digit
  [just-an-int]
  (println (str just-an-int)))


(defn -main
  [& _]
  (print-digit "asds"))

@prnsml argument-checking is enabled by calling instrument

Thank you, @seancorfield. And should new projects use Spec 1 or Spec 2 (although it is considered not ready for production yet)?

@prnsml See https://clojure.github.io/spec.alpha/clojure.spec.test.alpha-api.html#clojure.spec.test.alpha/instrument

Spec 2 is still changing a lot. It's not really ready for any usage yet. Unless you are deliberately exploring/testing it to provide feedback to the core team 🙂

Anyone got any experience of CORS for CLJS using cljs.ajax to a http-kit server

what libraries I should be looking at, or just anything I can read at all to help

Question, I’m trying out several JSON packages, however only with org.clojure/data.json decoding i can easily translate datetime string to actual DateTime

what is the idiom to do this in Clojure for other packages?

should I first just import the full data set and then do updates in the data set after the fact?

example being cheshire*

@patrick.glind Each library will have its own mechanism for controlling date conversions.

the weird thing is, I started a new clojure app to try out all kinds of JSON libraries

however don’t really see the option to convert decoding json data to the right ‘type’

with clojure/data.json I can do this:

(json/read-str json-file

but for cheshire and jsonista

i don’t see that kind of feature

JSON doesn't support dates as a datatype

http://dakrone.github.io/cheshire/cheshire.core.html#var-parse-string

You need to use your own scheme on top

yeah, it comes in as a String

which is fine

i just want to immediately convert it to DateTime

same as I would do with a CSV file

but then i figured, like with data.json, there should be an idiomatic way

to convert on import

The problem is that there isn't a objectively "correct" way of knowing what date/time type a JSON string represents. For a start, there are several different ways of representing dates in the JVM.

According to the Cheshire readme, the decode function can take a custom decoder that handles custom parsing based on the field name

See: https://github.com/dakrone/cheshire/issues/49

In 2.0.4 and up, Cheshire allows passing in a function to specify what kind of types to return, like so:

;; In this example a function that checks for a certain key
(decode "{\"myarray\":[2,3,3,2],\"myset\":[1,2,2,1]}" true
        (fn [field-name]
          (if (= field-name "myset")
            #{}
            [])))
;; => {:myarray [2 3 3 2], :myset #{1 2}}

I don't know if it changed since that ticket.

@didibus, what is the clojure idiomatic way to convert it after import?

@seancorfield, yeah I saw that in the doc’s

however that feature is for array: http://dakrone.github.io/cheshire/cheshire.core.html#var-parse-string

[key-fn array-coerce-fn]

i tested to see if i could make it work, but no luck

Oh, I missed that part. Sorry.

i totally understand if the packages don’t convert the data, just do the import part

i’m just curious how to process it afterwards idiomatically

Hum.. are you, actually reading the doc, are you sure it's not for map?

hmm

Is this json you get from elsewhere? Or you encode it yourself?

i get it from elsewere

parsing is no problem with all 3 libraries i’m working with

The best would be to encode the date to a map with a custom encoder. And then I think you can use that custom decoder function to decode it back to date

From elsewhere, hum, so how can you tell the difference between a string with a date and just a string of text?

it’s just that so far, only data.json supports converting types on import

the key is :created_at (i convert to keyword)

In any case, you can post process it with clojure.walk

"created_at": "2018-04-24T14:51:11.220-04:00"

Oh I see, then what Sean said from the doc, you sure it doesn't work?

i’m able to convert it with this:

(defn json-value-reader [key value]
  (if (= key :created_at)
    ;; (java.sql.Date/valueOf value)
    ;; (java.time.LocalDate/parse value)
    ;; (read-string (str "#inst" value))
    (java.time.ZonedDateTime/parse value) ; 
    value))

for data.json

The Cheshire readme is confusing -- decode is just an alias for parse-string and that only works for arrays.

I would just stick to clojure.data.json then.

yeah, i was getting to that point

but was hoping since i’m a beginner, it would be me that’s missing something haha

my next thought was: after import: just do a mass ‘update’ on the data, convert the string->ZonedDateTime

but wanted to find the idiomatic Clojure way

I think that's your best bet

You can do a clojure.walk post processing pass I guess

i’ve never worked with that

Unless we're missing an obvious feature in Cheshire

http://clojure.github.io/clojure/clojure.walk-api.html

it’s the same with jsonista

https://metosin.github.io/jsonista/jsonista.core.html#var-read-value

for decode there are very few options

As a beginner, it's reasonable to use the org.clojure libs as a first step if they do what you need. Otherwise you can get overwhelmed by all the choice (Clojure has so many libraries for any given problem).

thanks guys

the weird thing is, i was trying to use the org.clojure library for parsing a ~30mb XML file

however that caused a lot of performance trouble ha

Ah, and it was very slow?

I'm not suggesting using it (definitely stick with clojure.data.json for now) but something I've used before with some success for this sort of thing for more complex data structures is Metosin's spec-driven transformations https://github.com/metosin/spec-tools/blob/master/docs/10_spec_transformations.md

org.clojure for json works fast enought for me 🙂

Yeah, well, you're going to trade off performance for functionality in a lot of situations I suspect 🙂

haha true true

btw @seancorfield, i liked your post on next.jdbc, next time i work with a database i will definitly try your library!

I feel we are missing something. Kind of surprised Cheshire and jsonista wouldn't support custom decoder/encoders at this point. Doing a post processing pass seems way slower, and they're all about performance

i was thinking the same thing, confusing..

Probably because they're both based on Jackson, a Java library, and that doesn't expose the feature -- or they're both so focused on performance that they don't want arbitrary value transformations?

i will try some more things

thanks for your help

1 thing that is very confusing to me as a beginner though is how the clojure.xml library is having terrible performance for me

like i said, it’s a ~30mb file

and i can read it, run parse on it

When you recur a function you're not creating a new thread, so a with-local-vars binding will persist through recursive calls correct?

I mean, to be fair, that's pretty big :p

Lots of editors would choke on that size of file

haha, alternatively I found BaseX which parses it in split second and I can run XQuery commands on it

yeah, i understand that parsing/reading the file can take some time

but for some reason, afterwards I can’t really work with the XML as it brings my REPL to a halt

maybe i’m doing something wrong

(cheshire/parse-string
 (cheshire/generate-string {:bar (java.util.Date.)}))

@dpsutton as far as i know, it’s a string

what is the idiomatic way to ‘walk’/‘process’ an XML file with Clojure?

should i try to think about it like a map like dataset

or more

like the clojure.zip library

where you go through the dataset

where you do like (zip/down), (zip/right) etc

Data.xml is built on top of Java Stax library

https://docs.oracle.com/javase/tutorial/jaxp/stax/why.html ?

Zippers are nice, but need to load the whole thing in memory

So 1mb of XML probably becomes 10mb once loaded in memory for zipper

o wow

Data.xml doesn't load it all in memory, by using Stax under the hood

But how you query the XML will matter

i was googling for an XQuery library earlier today to work with in Clojure ha

If you do a big search, even though it's lazy, it'll still need to load everything it searches over

ok

So it might be related to how you're querying over it, for the slowness I mean

Also, at the REPL, trying to print anything too big will make everything super slow, your terminal/editor cannot handle large amount of text

latest thing i am considering is creating a server, for example with BaseX, which takes the xml file as a database, and then just query an API with clojure for the right data

yeah, i noticed that ha!

unfortunately even running (first …) on the xml file after the 1st tag

halts the REPL

How big is that tag?

The thing is, the Repl will keep trying to print things, and printing lazy things realizes them

i don’t know the size

of the tag

So if (first tag) returns a LazySeq for example, or something lazy, and you print that, it'll recursively realize it all and print everything

Try:

(def ftag (first xml))
(type ftag)

I've never used data.xml, but I think from the doc, you would call parse and then use either tree-seq or postwalk over it to extract what you want

am trying this now

(type (first xml-raw))
clojure.lang.MapEntry

reading the file this way: (def xml-raw (-> filename io/file xml/parse))

i tried the example from http://clojure-doc.org/articles/tutorials/parsing_xml_with_zippers.html, with io/resource, however that returned nil so didn’t work

(count xml-raw)
3

(first xml-raw)
[:tag :datafile]

(second xml-raw)
[:attrs
 {:xsi:noNamespaceSchemaLocation "wiitdb.xsd",
  :xmlns:xsi "",
  :xmlns:xs ""}]

opening the 3rd brings the REPL to a halt ha

so i can’t show that one

What is the operations you're looking to do? Just like explore the document from the Repl?

at first i was trying to do that yes, but quickly found out that wouldn’t work

now i just want to convert the XQuery I created into a Clojure ‘function’ or somthing

I'd say if that's the case, for big XML files, tens to hundreds and above megabytes, probably using Berkeley DB XML would be best.

with learning clojure, i’m taking every opportunity to use it ha

so i found this ~30 mb xml file with Wii games data and figured, let’s create an app ha

For a single query over the data, you should be able to use data.xml. But the challenge is writing an efficient query for it that only pulls in the minimum needed

so here is the working XQuery:

let $search-for := 'mario'

for $game in /datafile/game
where contains(lower-case($game/@name), $search-for)
where $game/type = ''
order by $game/@name

return <a href=''>{data($game/@name)}</a>

And at the REPL, you will constantly be at risk of accidentally printing large amount of the xml

yeah, thankfully in Calva i can interrupt running evaluation, saved me several times ha

Try with this: https://clojuredocs.org/clojure.core/xml-seq

I think you should be able to recreate that query with that

that looks very interesting, i will try right away

i get the feeling this library is out of date, https://github.com/clojure/data.xml

i know that Alex was updating all the Clojure libraries to 1.0.0 that were working some time ago

this one is on 0.0.8, it might not be stable yet

@patrick.glind use the 0.2.0 alpha version

If you're used to versioning in other tech, Clojure's approach can look very strange

i am using deps.edn, can’t seem to be able to find the library on Clojars

ok i will use that

yeah i read about that some time ago, also that in Clojure land, last commit > 1 year ago doesn’t mean out of date haha

Contrib libraries are not on Clojars. They're on Maven.

org.clojure/data.xml {:mvn/version "0.2.0-alpha6"}

aah

why is their so many options for everything in clojure

(we switched from lein to boot in 2015 and then to CLI/`deps.edn` in 2018)

i found this one just before you posted: https://clojars.org/org.clojars.michihuber/data.xml

it's really not helpful for beginners.

I feel a curated list of libraries for common tasks is needed.

http://clojure-toolbox.com

yeah i was going to say the same thing haha

my ‘workflow’ is usually to go to clojars to find the actual deps.edn ‘line’ to use

I have a bunch fo things I want to build, and I keep trying to build them in Clojure, but invariably hit a problem where i have no idea what library to use or some tooling just fails and somethjing that I could do in an hour or so ends up with me losing hours and hours or days trying to get it working in Clojure and I just give up

and I go do it in C#

@qmstuart Clojure is not easy. It's simple, but simple is not the same as easy.

Clojure is hard to learn and hard to master. It isn't for everyone.

I think it's mostly hard to master because the quality of documentation is quite poor though

There was a lot less documentation when many of us learned it in the past. It was even harder to learn then.

There are so many books, tutorials, and videos nowadays by comparison.

Yes, there is a lot of documentation, but a lot of it is quite shallow I find

It's OK for something to be hard to learn/master. A violin is hard, for example (a common comparison with Clojure as a "sharp tool").

I haven’t had any issues with documentation quality. I think one of the hurdles is that the documentation might be for java code that a clj library is wrapping (or js code that cljs is wrapping)

so it’s not always clear where to look, but I think I’ve gotten so used to it, that’s not really a pain point for me personally

There is certainly an underlying assumption behind a lot of the material that the developers have a fair bit of programming background. But it's slowly improving over time.

being familiar with both java and js can provide a huge boost for using clojure/clojurescript

Anyways, in case it was missed, I posted a link to http://clojure-toolbox.com which is a (lightly) curated list of libraries organized by category, in case that's helpful to either of you @patrick.glind or @qmstuart

@patrick.glind This tutorial seems pretty good. And it shows how to parse and query a 44GB XML file in Clojure: https://blog.korny.info/2014/03/08/xml-for-fun-and-profit.html

awesome, thanks!

i will look at that tomorrow

i was able to get part of the way working

although the REPL crashed on xml-seq while following https://clojuredocs.org/clojure.core/xml-seq

ha

(ns wiitdb
  (:require
   [ :as io]
   [clojure.data.xml :as data-xml]))

(def filename "wiitdb.xml")

(def xml-raw (data-xml/parse (io/reader filename))) ; this works..

(xml-seq xml-raw) ; crashes repl

this is what i have now

I'd expect that to crash the REPL: you're asking it to print (and therefore realize) the entire lazy sequence.

Yup, you basically can never let the Repl print the XML or any big parts of it or it'll choke

ok

You could probably print (first (xml-seq xml-raw)) or maybe even (take 5 (xml-seq xml-raw))

The tutorial I linked has a to-short-xml convenient util that will print the node without it's children which you can use

@seancorfield yeah that works, doing take 5

@didibus, it’s on my todo list for tomorrow’s reading 🙂

thank you guys so much for your help!

it’s bedtime here in Netherlands haha

👍

Good luck