Fork me on GitHub
#clojure
<
2020-11-11
>
kanwei03:11:25

@alexmiller hmm, seems like the download issue is that it's trying to download from central, instead of clojars

kanwei03:11:38

any idea why it would do that?

alexmiller03:11:32

it's probably actually not - the download error message just prints the last (or maybe first) repo it tried (it tries all of them)

alexmiller03:11:07

you can"remove" the central repo by doing {:mvn/repos {"central" nil}}

alexmiller03:11:42

if you want to make sure, but really I think that's probably a red herring

seancorfield03:11:10

@alexmiller Remind me, does t.d.a look at Maven or Clojars first?

seancorfield04:11:04

Cool, thanks. That's what I thought. Which is good, because then no one can shadow a Maven artifact on Clojars to "hijack" it (given that it doesn't do signing etc, whereas Maven does).

didibus04:11:54

The Clojars doc I remember also says that you cannot create an artifact with a group-id that exists on maven central

didibus04:11:29

But it does mean that someone could upload to maven possibly and shadow Clojars

didibus04:11:50

But maven does have a bit more of a manual check, so maybe that be trickier to make happen

seancorfield04:11:57

Oh, interesting. Good to know. I guess that makes sure folks don't upload a "shadowed" artifact and then wonder why it never gets picked up (because tooling favors Maven).

seancorfield04:11:24

Uploading to Maven is definitely a bit more controlled.

didibus04:11:50

After me thinking about it all, if I ever release a Clojure lib, I would probably just release it on Maven.

alexmiller04:11:23

tools.deps always searches maven central, then clojars, then other stuff

didibus05:11:08

Is there a predicate for anything that is "list" like. Something that would be true for vector, list and seq, but not map and nil ?

didibus05:11:37

Hum, ya that might work. I guess I would want it to include sets as well, but I think I can live without

Ben Sless06:11:17

but a set isn't list like, it's unordered 🙃

didibus06:11:39

Ya, but that's why I said list like 😛

didibus06:11:00

Basically I'm looking for a pred for unordered containers of scalar

didibus06:11:24

Or more like, they can be ordered or not I don't care, just a coll of scalar

didibus06:11:45

So nil would not count, and map would not count, because map contains key/value pairs

Ben Sless06:11:48

(and (coll? x) (not (map? x)))

didibus06:11:22

Oh, I forgot about coll?, ya ok that works

didibus06:11:00

Is this a bad idea:

(defn defalias
  "Aliases the given alias-sym in the current namespace to a new namespace
   of `<current-namespace>.alias-sym`."
  [alias-sym]
  (let [sym (symbol (str *ns* "." alias-sym))]
    (create-ns sym)
    (alias alias-sym sym)))
I'm thinking of using this, because often time, I have some entity in a namespace that I want to spec, and I want the keys to be :my-current-namespace.entity-name/entity-attribute But I don't want to actually create a physical file and namespace just for this entity which would only really contain a spec. And I'm too lazy to type out the full namespace in the s/keys. Also, I want it to be dynamic, so if I ever change the namespace name I'm in, I want the entity to reflect. Like so:
(defalias 'bar)

(s/def ::bar/foo string?)
(s/def ::bar/baz map?)

(s/def ::bar
  (s/keys :req-un [::bar/foo ::bar/baz]))

alexmiller07:11:12

just fyi, I am currently working on a solution to this (lightweight alias) for Clojure with Rich (probably 1.11)

👏 1
parrot 1
didibus07:11:05

That be great, because like @U0NCTKEV8 points out, tying your keyword qualifiers to your code namespace can be brittle, the risk that your data gets stored or moved around, and that it now no longer maps to your code namespace is there (and we had this issue happen to us). And I don't want to go Java style, where now I have a gazillion files for my data model. My other options were either, go unqualified, but then your data isn't self-describing anymore. Or go with short qualifiers like :entity-name/field-name but that can conflict. And so going with long qualifiers is a drag having to type it all out. So, I still feel I personally need to hammock all that right now, and haven't yet found a best practice I'm happy with for Spec usage. But I trust you and Rich must have had plenty of time in that hammock 😄

didibus07:11:14

I think something like a lightweight qualifier might solve my issue. Fundamentally, I want to say: "this is the spec for this key". But I want this to be resilient to me moving the spec around, so even if one day the spec is in namespace a.b if tomorrow I move it in f.g I don't want the spec key to change, so my old data will still say, the spec for this key is a.b/key. Same thing on the key side, if I change where I create my map, I don't want the keys to change to saying their spec have moved, because it may or may not. So where the keys or specs are defined/created in the code shouldn't matter. What matters is the name of the spec in the global spec registry. So going with :entity/field over using :: does solve the above, but because the Spec is a global registry it is at risk of conflicts. So ideally I'd want my spec to be URIs like namespaces: com.organization.application.context.entity/key. But now its just a whole lot of typing and risk of typos. So ya, I think if I could alias keyword namespaces, that are not code namespaces, then I could have nice URIs for all my keywords and my specs, which are fully separate to the namespaces I use in my code.

hiredman06:11:25

Using :: is gross

☝️ 1
didibus06:11:22

What do you do instead?

hiredman06:11:11

Not use it?

didibus06:11:41

Well, I mean then what namespace do you use in your spec

didibus06:11:03

Do you still use the current namespace, but fully type it out?

hiredman06:11:24

It seems like people who insist on using it 1. Saw the spec docs use it and 2. Insist on using code namespace names for keyword namespaces

hiredman06:11:35

:whatever/foo

hiredman06:11:18

The spec docs do it for the sake of example brevity, and using code namespace names for data like that is like naming tables in a database after the code that first uses it

didibus06:11:05

Well, I do that sometimes. But here's my problem. Say I have an account entity in one place, so maybe I have :account/name, :account/id, etc. Now I have another account entity somewhere else, but it is different. So how do I distinguish them? So if I have insight on the entire app and am a solo dev, maybe I remember that I already specced an account, so I need a different name now. But if its a big code base with many devs, I could accidentally mess with some other :account spec

hiredman06:11:14

;didibus-account/whatever

hiredman06:11:05

Make whatever prefix you want, but tying it to code namespaces is bad

didibus06:11:16

And now we're back at my defalias 😛, I guess I don't need to have it use *ns*, I could have it use wtv, but the point is I'm too lazy to type out my "unique-prefix" everytime

hiredman06:11:55

Then copy and paste or use an editor with completion

didibus06:11:01

So I'm creating a fake namespace, just so I can leverage :: to expand to my prefix

hiredman06:11:39

We use a lot of namespaces keywords at work, I don't know when I last type one out in full not at the repl

hiredman06:11:07

Using :: at all makes your data brittle and it will be different depending on the value of *ns* at read time

didibus06:11:29

Hum, I mean yes I could copy/paste or use auto-completion. But I'm wondering, is my little defalias util thing to take advantage of :: a bad idea? Or would that be fine.

hiredman06:11:38

So you might copy something to another file, and now boom wrong thing

💯 1
hiredman07:11:05

Or you copy something from a code file to a configuration file, etc

didibus07:11:35

I agree with you there, I've been bit by :: in keywords before too, there's some namespaces that have a comment: never rename, because we have prod data stored with the namespace name and it break the code to change

didibus07:11:08

But my solution is actually to not namespace my keywords in my data at all, so I use req-un

didibus07:11:20

So this is only for writing s/keys spec and all that

hiredman07:11:54

If you are not using namespaces keywords then maybe don't use spec

hiredman07:11:30

req-um exists, but spec really wants you to namespace keywords

didibus07:11:28

I haven't found it a big issue. The spec has to be namespaced, but your data can be as you want. I still use it to generate data for tests, validate, and document things

didibus07:11:30

What prompted me with this problem here is, I actually already have a ::type spec, but I need another one for some other map whose :type key is not the same, so it needs its on spec. So my instinct was to do ::wtv/type

didibus07:11:00

But that doesn't work, and I thought, ok, well I need a unique namespace for this spec

didibus07:11:19

By the way, as an aside, because I've been debating with myself if I want to use unqualified keywords, or something more like :entity-name/field-name as you say. Have you found any benefit to the latter? I felt when I thought about it, I couldn't find a use case where I'd be making use of the namespace on my keyword

Steiner12:11:58

hey, I want to join the discussion of Chez Scheme, I can't join them in freenode now, anyone know other channel ??

zilti12:11:15

No idea, only ever used/contributed to Chicken Scheme, is the channel private or does it just not exist?

manutter5112:11:18

You might want to ask in the #other-languages or #other-lisps channels.

Steiner13:11:54

@zilti it just tell me can't join in the channel

zilti13:11:02

Maybe you need to have a registered Freenode account. If that doesn't help, it is a private channel.

kanwei15:11:36

@alexmiller Found the issue - turns out my project deps.edn had an old http://central.maven.org/maven2/ that wasn't resolving anymore. However the packages still tried to download and just got corrupted. Any way to improve it so it rejects the package outright if it fails the checksum instead of keeping the corrupted version? might bite someone in the future.

alexmiller16:11:08

I don't actually think that's knowledge I have access to via the maven libs but if you can file a question at https://ask.clojure.org I will make an issue and look into it

kanwei16:11:29

cool, done

zalky16:11:47

Hi folks, I'm wondering if someone can help me understand what I'm doing wrong when trying to parse large XML files lazily. I'm using clojure.data.xml/parse, and I keep getting GC overhead errors, mostly likely from retaining the head of some lazy sequences. I've tried to reduce the problem for something simple. The following works and uses very little memory:

(with-open [r (io/reader "path-to-huge-file.xml")]
  (->> (xml/parse r :skip-whitespace true)
       :content
       first
       :content
       (#(nth % 3))
       :content
       count))
Whereas the following uses massive amounts of memory and eventually throws a GC error:
(with-open [r (io/reader "path-to-huge-file.xml")]
  (doseq [node (->> (xml/parse r :skip-whitespace true)
                    :content
                    first
                    :content
                    (drop 3))]
    (count (:content node))))
I'm not sure what could be hanging on to the head in the second example. Any help would be much appreciated.

alexmiller16:11:58

the seq returned by ->> effectively contains all but the first children, so the entire "rest" of the xml document

zalky16:11:29

Thanks for the response, Alex! So I actually want to process all the children at that depth, one at a time, lazily. Shouldn't the doseq help do this without retaining the head? I don't even get through the first one (it's a really big seq), basically it runs out of memory on what is effectively (#(nth % 3)). And the result of ->> should be lazy, so my understanding of the problem is that it should not matter that it contains the "rest" of the xml document. We don't even get to the rest of the document.

zalky17:11:46

Basically I want to avoid:

(dotimes [n figure-out-what-n-is]
  (with-open [r (io/reader "path-to-huge-file.xml")]
    (->> (xml/parse r :skip-whitespace true)
         :content
         first
         :content
         (#(nth % n))
         :content
         process-lazily)))

alexmiller18:11:19

sorry, was in a meeting

alexmiller18:11:56

I don't remember how much of the xml stuff is lazy - I think reading xml off a stream is but it's not lazy in depth

alexmiller18:11:23

but it might depend what's around that with-open - how are you actually using the result?

zalky19:11:51

At depth 3, I have a very large lists of small nodes. I want to process each node independently into a small RDF graph and store in a db. The (dotimes [n figure-out-what-n-is]...) approach, while slow, actually does technically work, and is not the end of the world. But I was also worried that I had misunderstood something fundamental about the nature of laziness with respect to holding on to the head of lists.

timsgardner19:11:17

Not sure if this is a clojure question or a node question, but I'm running a clojure program (via lein run -m) as a spawned child process in an electron project. When the electron program closes the JDK sticks around indefinitely, wondering how to really close it.

didibus19:11:49

Probably you need to call shutdown-agents and/or System/exit

timsgardner19:11:25

I'll try that, thanks

borkdude20:11:45

@U3Q8MMREX I also have some logic for this in https://github.com/babashka/babashka.process, see destroy-tree

timsgardner20:11:46

Turns out I was also neglecting to call process.kill('SIGINT') from node

timsgardner18:11:31

though ideally I wouldn't have to, the node process should be able crash without clojure sticking around

borkdude18:11:30

@U3Q8MMREX Babashka supports killing all sub-processes on shutdown. It has great startup time. So you could consider booting node from bb and then kill all on shutdown

timsgardner18:11:12

I might switch to it, thanks

borkdude18:11:54

An example using clojure:

$ bb -e "(require '[babashka.process :as p]) (p/process [\"clojure\"] {:inherit true :shutdown p/destroy-tree}) @(promise)"
Clojure 1.10.1
user=> (+ 1 2 3)
6
user=> ^C%

borkdude18:11:03

You can also do this in Clojure proper btw, babashka/process is also available on the JVM. Or you could just write the JVM interop code yourself