Fork me on GitHub
#beginners
<
2023-02-21
>
ackerleytng05:02:38

help me wrap my head around this: I'm building a scraper and trying to use core.async. I have two channels in and out. (list n) returns a vector of data, and (get data) takes data and enriches it. The plan is to call (list n) where n is 0, then 1, and so on, writing the returned vector to channel in, until (foo n) returns []. I am using async/pipeline-blocking to pipe from in to out, calling (get data) on each item. Once it returns [] I want to close the in channel, but (get data) should keep processing on all the rest of the items in channel in. Unfortunately once in is closed, the pipeline also closes out and the final output is missing values... How should I structure this? Is async the wrong tool for this task?

rolt08:02:46

there is a close? argument in the pipeline functions if you dont want to close out when in is closed. But the thing is: I dont think you need that. Even if out is closed you can still access the values in it

rolt08:02:30

oh I think I see your problem. I guess what you want is for (get data) so set a flag instead of closing, and for (list n) (or whatever is feeding the in chan) to close the in chan only when that flag is set.

rolt08:02:54

that way the feeder will write the pending value before closing the chan

ackerleytng16:02:48

I'm using onto-chan to write the data into the channel, and there's a close? flag that allows me to close the channel after writing, so I am closing only when there's no data returned from (get data)

ackerleytng16:02:47

I think the issue could be that pipeline-blocking closes when the input channel closes instead of when there is no more to take from the channel.

ackerleytng16:02:30

What's a good way of closing an async channel when the last data item in the channel has been processed?

Ben Sless16:02:37

The idiomatic solution is the producer closes the channel, not the consumer

phill17:02:18

If you want to make it known that a consumer has finished consuming, then you can use a second channel for that notification.

rolt18:02:12

do you want to stop the process early in case (get data) returns [] for some reason ? or do you want to process every value from the coll you're passing to onto-chan ? If this is the second case, just don't close anything, everything will be closed automatically

ackerleytng04:02:55

(list n) will always eventually return [] since at some point there will be no more data to list (`n` is the page number). So I have to use onto-chan multiple times, once for each n that returns more than one item on the vector

ackerleytng04:02:18

Thanks, I guess I will have to get the producer to notify the consumer that the last item has been processed.

rolt08:02:15

then i don't see why you have missing values. If it's your producer that closes the in chan, everything should be processed and eventually appear in out.

rolt08:02:38

just tested it and i was completely wrong. pipeline does stop processing when in is closed, it does not "drain" the chan 😕

rolt09:02:20

it's probably easier using claypoole's upmap function then

ackerleytng16:02:51

Thanks for testing it for me @U02F0C62TC1

Matthew Twomey05:02:43

I’m stuck (again) on something fairly simple. I have a directory structure for a utility lib that looks like this:

src/com/beakstar/util/base64.clj
src/com/beakstar/util/time.clj
src/com/beakstar/util/numbers.clj
src/com/beakstar/util.clj
The “stuff” inside the util folder is where all my functions are. Each file in there is in a namespace named after the file. When I use this library from a (separate) project, I want to be able to bring everything in using a single (require '[com.beakstar.util :as utils]) statement. Right now, I have to use individual statements: (require '[com.beakstar.util.base64 :as base64]) (require '[com.beakstar.util.time :as time]) (require '[com.beakstar.util.numbers :as numbers]) I was trying to “wrap” the stuff using a src/com/beakstar/util.clj file, but that doesn’t seem to be the right approach. Anyone have any tips here? Additional note: I’d also like the individual functions directly accessible under “util” - such as util/money (a function in the numbers file).

Sam Ritchie06:02:22

You may get some naysayers, but Potemkin was built for this use case: https://github.com/clj-commons/potemkin

Matthew Twomey06:02:08

Huh - ok, so it’s actually a limitation as things currently are. Ok thanks, will check out potemkin. In most languages I code, I tend to build up fairly large sized utility libraries and I like to organize things by library - but also in some cases (such as a “utility” library), I like to split things up into separate files as well. I just don’t want to have to “pay for that” by needing to do all these requires’ when I use it.

Matthew Twomey06:02:22

I’m also using this lib in babashka, so I tried the load approach in a wrapper library (along with in-ns but babashka doesn’t support load so that wasn’t working.

practicalli-johnny09:02:16

I assume that requiring all the other namespaces in src/com/beakstar/util.clj file would load them when util.clj itself is loaded, although the functions in those namespaces would need to be called by their fully qualified name, com.beakstar.util.time/function-name . So that doesn't seem to be particularly useful. For clarity I include the specific namespaces with an alias or refer specific functions if the namespace is predominantly about using those functions.

kennytilton09:02:54

Ah, @U017QJZ9M7W, I have been looking for exactly this, which describes my problem precisely: https://github.com/clj-commons/potemkin#import-vars Thx! What are the naysayers saying?

pavlosmelissinos10:02:08

> I don't like any of the Potemkin import-* functionality. Every time I am working against libs like this, "go to source" ends up in the wrong place. From my perspective, this is a bit of complexity and redirection that serves little purpose other than to confuse those looking at the implementation. Using public/private vars and implementation namespaces is sufficient to cover every case I've found useful. I think this is a pretty valid argument to make. If you're working on your own and don't plan to share your code with anyone, do whatever the heck you want, sure, but for what it's worth I'm not sure I'd want to be part of a team that uses this pattern heavily. What's wrong with being explicit?

pavlosmelissinos10:02:36

I don't like "util" namespaces either. What does it even mean, does it describe namespaces that are unrelated to your problem domain? Why is it a special case though? I'd just call it org.project.time or whatever it's supposed to do, what does the util in the name really offer? Also, in the scenario that you end up reusing the same N namespaces over and over again: it's not a big deal really, you just state your dependencies, right? You're being explicit and honest about it. Do you really need to hide it behind a level of indirection just for your convenience? If this bothers you though, it could be a code smell; perhaps you haven't really decoupled your namespaces well enough. Like, if you find that 10 namespaces require the same 5 namespaces, then maybe all the functions should be in the same namespace or split in a different way. Without seeing the code this is very handwavy obviously... But hopefully it makes some sense.

4
☝️ 2
skylize15:02:39

To me "util" means generic tools that make building things easier. If you are building a bird house, you keep your hammer in the toolbox. You don't make an extra hole in your birdhouse to store your hammer in.

skylize15:02:37

A util namespace give you a place to put such things without outsourcing them to a dependency.

pavlosmelissinos15:02:52

But in this case you are the one building this "util" and you're making it part of your source code (the birdhouse, in your story). The toolbox analogy is closer to using a third-party library I think. You didn't build it yourself, it was there before you started and it's indeed a tool (still, clojure.edn is a much more useful name than a hypothetical clojure.utils)

kennytilton15:02:13

"What's wrong with being explicit?" I think the potemkin doc explained the problem very well: as a developer, I like small files where functionally similar stuff lives together. As a user, I just want to require one or two namespaces. And the ways around this have the tail wagging the dog interms of code organization. I wonder if team Clojure ever considered implementing sth like the Common Lisp package system? https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node111.html :thinking_face:

skylize15:02:24

The third party libs are Home Depot. My own util lib is the toolbox. A util namespace is a mini-toolbox with only the tools actually needed by this library, without extra stuff that's not needed. Tools defined right in the namespace they are used is a lot of noise in the business logic.

pavlosmelissinos15:02:32

> as a developer, I like small files where functionally similar stuff lives together. As a user, I just want to require one or two namespaces I agree. I just think that it's the responsibility of the developer to expose the right functions to the user. Just make an API namespace that will include all the functionality that the user will need

kennytilton15:02:06

"Just make an API namespace that will include all the functionality that the user will need" The funny thing is that Matrix has a small API but, divided by functional area, they end up in 4-5 namespaces. So to lower that number, I have to abuse my code organization. Tail wags 🐕 .

pavlosmelissinos15:02:15

> The third party libs are Home Depot. My own util lib is the toolbox. A util namespace is a mini-toolbox with only the tools actually needed by this library, without extra stuff that's not needed. Tools defined right in the namespace they are used is a lot of noise in the business logic. Is it really? If your concern is the user, then you can make an interface/api namespace, like I said before. As a developer I very rarely have said "I wish I had more source files to keep track of". When I feel I have to break down logic, it's only for applications that have e.g. multiple business cases (e.g. UIs with multiple screens) and therefore actually require code reuse (shared namespaces). utils is very convenient when that time comes because you don't have to think of a descriptive name (I've certainly done it more often than I'd like to admit) but it's lazy.

pavlosmelissinos15:02:20

> The funny thing is that Matrix has a small API but, divided by functional area, they end up in 4-5 namespaces. So to lower that number, I have to abuse my code organization. Tail wags I don't get it, how does the API affect your implementation?

kennytilton15:02:10

I wrote, and use, Matrix. So following my small-file/functional-grouping preference I ended up with a dozen namespaces, 4-5 of which are needed by any client app.

pavlosmelissinos15:02:17

Looking at Matrix I can't tell what is the API and what is implementation, can you point me to those 4-5 API namespaces to understand what you mean?

kennytilton15:02:02

I would, @UEQPKG7HQ, but the problem seems so well understood. As I said, the potemkin doc nails it exactly:

Clojure namespaces conflate the layout of your code and your API. For larger libraries, this generally means that you either have large namespaces (e.g. clojure.core) or a large number of namespaces that have to be used in concert to accomplish non-trivial tasks (e.g. Ring).

The former approach places an onus on the creator of the library; the various orthogonal pieces of his library all coexist, which can make it difficult to keep everything straight. The latter approach places an onus on the consumers of the library, forcing them to remember exactly what functionality resides where before they can actually use it.
And here is the require from a tutorial I am writing:
[tiltontec.cell.core :refer [cF cF+ cFonce cI cf-freeze]]
    [tiltontec.cell.integrity :refer [with-cc]]
    [tiltontec.model.core
     :refer [mx-par mget mset! mswap! mset! mxi-find mxu-find-name fmu fm!] :as md]
    [tiltontec.web-mx.gen :refer [evt-md target-value]]
    [tiltontec.web-mx.gen-macro
     :refer [img section h1 h2 h3 input footer p a
             span i label ul li div button br
             defexample]]
    [tiltontec.web-mx.style :refer [make-css-inline]]
A classic is the integrity NS. That is crucial, gets its own NS, but thanks to my other bias -- transparency -- only rarely do we need to get explicit it about it via with-cc. btw, the example ^^^ also drags in Web/MX, but there we go: I cannot hide all the MX and Cells dependency from Web/MX users. See the problem?

pavlosmelissinos15:02:46

The user shouldn't have to require so many namespaces, I agree. But, like I said, I believe this should be the responsibility of the library author. If every user ends up reusing all of these, why not just create a single API namespace that contains all these functions/macros?

skylize15:02:00

> utils is very convenient when that time comes because you don't have to think of a descriptive name (I've certainly done it more often than I'd like to admit) but it's lazy. The standard library is fairly extensive, but always has gaps. Think about where you would put a function like map if Clojure didn't provide it. (Many languages don't, so shouldn't be that hard to imagine.) If your project has a small-ish handful of generic things have nothing to do with the problem domain, and look like reasonable (or even semi-reasonable) additions to the std lib, then it makes perfect sense to group them together as such. util/`utils` is just shorthand for my-personal-additions-to-a-more-extensive-std-lib-the-will-never-actually-be-standard. (Of course some people go a bit farther, adding things here that don't seem generic enough to me. But the underlying concept is theoretically the same.)

skylize15:02:28

> If every user ends up reusing all of these, why not just create a single API namespace that contains all these functions/macros? from the quote shown above >

The former approach places an onus on the creator of the library; the various orthogonal pieces of his library all coexist, which can make it difficult to keep everything straight.

Matthew Twomey16:02:41

This is a great discussion. I can see why it’s contentious. The potemkin doc does indeed describe exactly my situation and perspective. If I were to add anything to that description it would be this: Imagine I have a set of generic utilities that all “should” live in util and I’ve put them there. Now my users (really me) just need to include util. That’s where I started. However as the developer of util, I actually want to break it up so that it’s easier to group similar things together (so I don’t have one huge file with big comment dividers in an attempt to organize it). The only reason I’m breaking it up is for developer (again me) clarity and organization. I don’t want my “user” to now have to include a dozen things.

skylize16:02:27

A few days ago, Alex suggested using https://clojuredocs.org/clojure.core/load . Apparently Clojure uses it internally to separate some code out into different files, while still exporting under the same namespace. Have not tried this myself. so no idea what edges to watch out for.

Matthew Twomey16:02:08

Yeah - I found a good example of using load also. Unfortunately it’s not supported by babashka, which is the main place I’m using this lib for now. Potemkim, however, works fine.

skylize16:02:03

Did you bring that up in #CLX41ASCS? borkdude might want to add support.

Matthew Twomey16:02:11

Was just going to.

kennytilton16:02:03

Oooh! Oooh! @U0250GGJGAE, got a link? I never quite sorted out the "load" solution. It sounded as if we were supposed to omit an ns declaration altogether in the little files. So my user sees everything in one NS (yay!) but...so do I? :thinking_face:

borkdude16:02:15

Why use load if you can also just use require?

kennytilton16:02:23

I think require would mean a separate NS for the required little files. The goal is delivering their functions as a single NS, to avoid massive require lists. @U04V15CAJ

☝️ 2
borkdude16:02:41

I think you could avoid this with require as well. The essential part is in-ns , not load vs require

Matthew Twomey16:02:34

That does not work - without load there is no way to get the functions in the other files to show up in the namespace.

borkdude16:02:04

I don't get what you mean with "showing up". This works:

borkdude@m1 /tmp/bbt $ cat bb.edn
{:paths ["src"]}
borkdude@m1 /tmp/bbt $ cat src/foo.clj
(ns foo)

(defn my-foo []
  1)

(require 'bar)

(prn (bar-foo))
borkdude@m1 /tmp/bbt $ cat src/bar.clj
(ns bar)

(in-ns 'foo)

(defn bar-foo []
  (my-foo))

borkdude@m1 /tmp/bbt $ bb src/foo.clj
1
borkdude@m1 /tmp/bbt $

kennytilton16:02:39

Ah, this was the missing piece for me: (in-ns 'matrix.core) goes in all the little files. But now a little file, where it currently requires a littler file, has to do "load" as well. I am so glad I never abandoned another habit: tiny little prefixes on names that redundantly, until now!, differentiate names, and identify their sub-context.

Matthew Twomey16:02:20

@U04V15CAJ What we’re after with this (using your example) is to now require just foo and have access to foo/bar-foo.

Matthew Twomey16:02:29

(the same way in the clojure core example I linked you don’t have to require clojure.core.core_proxy, clojure.core.core_print. You just require clojure.core).

Matthew Twomey16:02:28

This is achievable in clojure.core by use of the combination of in-ns and load.

borkdude16:02:11

@UP7RM6935 Again, load vs require isn't the essential part here. It already works in the above example.

(ns baz)

(require 'foo)

(prn (foo/bar-foo))
$ bb src/baz.clj
1
1

borkdude16:02:07

This pattern isn't very well supported by clj-kondo (linting) but it does work in both clj and bb

skylize16:02:50

@U04V15CAJ Edit: corrected version below works, while this one is broken due to ID10T error. :face_palm:

Matthew Downey16:02:37

(@U04V15CAJ you tagged me instead of @U0250GGJGAE, but I'm realizing our names are quite similar, hello!🙂)

🙏 2
borkdude16:02:56

That's slack suggesting a different person, damnit.

borkdude16:02:04

Why do all those "social" apps do that

Matthew Twomey16:02:26

Looking more closely at your example @U04V15CAJ

borkdude16:02:41

@U90R0EPHA Does your code work differently in clj? I would expect not. Since use-foo doesn't contain the var foo/bar-fn without prepending foo

skylize16:02:59

Slack's locality bonus (meaning more likely to suggest someone in the thread already) is too low a number.

borkdude16:02:18

@U90R0EPHA yes, that's the example I gave earlier. foo contains all the vars from the other required namespaces due to the usage of in-ns . That's not what you did.

skylize16:02:53

My bug there. It does work.

borkdude16:02:26

I'll rest my case. require = essentially load + a no-op when the file / namespace was already loaded. There's no reason you should ever use the more low-level load over require unless you want to force reloading, which you can also do with require + :reload.

Matthew Twomey16:02:40

Let me try this again - I swear I tried it 6 ways to Sunday last night.

skylize16:02:49

@U0250GGJGAE A working example (from @U04V15CAJ’s suggestion) that seems to provide behavior required by original post.

👀 2
skylize16:02:31

Even though the file is bar.clj and the bar namespace is thus used require the contents of that file, all the rest of the code after (in-ns foo) actually gets written into the foo namespace instead of to bar.

Matthew Twomey16:02:22

Grrrr arrrrr eerrrr. Yes, that worked (tried it in my actual lib / setup). I’m quite happy about this, but frustrated I couldn’t figure this out lol. Thanks @U04V15CAJ , @U90R0EPHA, et al.

❤️ 2
skylize17:02:48

How would you be expected to know that? Beyond the requirements of a rudimentary "This is a REPL“ tutorial, and occasionally some really esoteric hacks, nobody ever talks about anything to do with namespaces that fails to align with the tradition of 1-file = 1-namespace.

❤️ 2
borkdude17:02:15

Which is imo the best approach

2
Matthew Twomey17:02:59

It is the approach I usually use. There are just some cases where I find it easier to organize in this way.

borkdude17:02:58

Another way to organize is to re-export vars (without relying on any magical tools like potemkin). E.g. take a look here: https://github.com/clojure/tools.build/blob/243aa947a97dc17a25d0223422ffdd2ce9fe1bcb/src/main/clojure/clojure/tools/build/api.clj#L277 It is a little bit more verbose, but tools like clj-kondo, Cursive, etc will work for you instead of against you :)

borkdude17:02:49

potemkin is supported by clj-kondo btw. and it also works in bb (only the potemkin.namespaces namespace)

borkdude17:02:09

If you ever have the plan to support CLJS forget about potemkin or in-ns

Matthew Twomey17:02:51

Oh - cljs doesn’t like in-ns ? That’s good to know.

borkdude17:02:42

CLJS is a bit more static when it comes to namespaces than JVM Clojure

Matthew Twomey17:02:53

(I don’t need potemkin anymore now that I understand how I can use in-ns)

borkdude17:02:04

The "manually re-exporting var" method always works, this is why I recommend doing that by default.

kennytilton17:02:20

"If you ever have the plan to support CLJS forget about potemkin or in-ns" How did I guess that? :rolling_on_the_floor_laughing: Matrix is .cljc. sob

pavlosmelissinos17:02:10

> Clojure namespaces conflate the layout of your code and your API. For larger libraries, this generally means that you either have large namespaces (e.g. clojure.core) or a large number of namespaces that have to be used in concert to accomplish non-trivial tasks (e.g. Ring). > The former approach places an onus on the creator of the library; the various orthogonal pieces of his library all coexist, which can make it difficult to keep everything straight. Well, I understand the argument @U90R0EPHA; I just disagree. This quote makes it sound like shaping your API as a library owner is a responsibility that shouldn't be yours. That's just wrong imo but I suppose we'll have to agree to disagree. (sorry for taking so long, just returned home)

Matthew Twomey17:02:22

Well, as I said in the other thread, I’m really glad clojure core doesn’t also make me require clojure.core.core_proxy, clojure.core.core_print , ..etc (even though it’s separated into individual files).

borkdude17:02:38

ClojureScript has clojure.pprint in one single file and imo it's easier to find things that way

borkdude17:02:15

I'm often on github trying to find a function in the clojure.pprint file and then have to navigate clojure_pprint/whatever.clj files until after a fourth attempt I find the right one

Matthew Twomey17:02:55

Yeah I can see a lot of pros and cons here. I understand why it’s contentious.

skylize17:02:43

I like the idea of an alias macro that binds a var into the current ns, along with all of the meta from the symbol used to find it... Except it keeps the line number meta of where it is aliased (not the source definition), so "go to definition" will bring you to the aliasing (making the alias explicit to the consumer), and User will need to "go to definition" again to see the underlying code. Not at all clear to me what negatives that would invoke.

borkdude17:02:53

Some languages have support for it, like JS and Haskell

kennytilton17:02:22

"This quote makes it sound like shaping your API as a library owner is a responsibility that shouldn't be yours. " Actually, shaping the API nicely is what we are discussing. And I am not hearing any solutions. Especially for a cross-CLJ library, where the load/in-ns trick is a non-starter. You know, with the JS port of Matrix I got frustrated with the require mechanism and just said to hell with it and whomped everything into one source. 🙂

skylize17:02:30

> This quote makes it sound like shaping your API as a library owner is a responsibility that shouldn't be yours. > The point is not at all that the author should not be responsible for API design. The point is that the author should have to work under the constraint that external API is concretely coupled to and internal organization.

plus_one 2
pavlosmelissinos17:02:27

> Well, as I said in the other thread, I’m really glad clojure core doesn’t also make me require clojure.core.core_proxy, clojure.core.core_print , ..etc (even though it’s separated into individual files). These aren't the only two options though. As the library author (in this case the clojure core team), you can do any of the following: 1. Keep all the code in a single ns 2. Put your code in an impl namespace and expose the user-facing functions in an api namespace 3. Split your code into multiple namespaces and expose the user-facing functions in an api namespace 4. Split your code into multiple namespaces and let the user figure out what to use 5. Use potemkin I'm personally against 4 and 5. 4 is bad UX, 5 is complex and can break in some contexts. 1, 2 and 3 on the other hand have their cons of course but they're vastly outweighed by their simplicity imo. > The point is that the author should have to work under the constraint that external API is concretely coupled to and internal organization. I suppose I'm not seeing that. As soon as you have an API, you should design the implementation around it, not the other way around. I don't understand how this constraint is established, in e.g. clojure.core.

Matthew Twomey17:02:13

The option they chose though, is not on your list. It’s basically similar to #2 (or #3), just with out additional boiler plate wrapping an already complete function.

skylize17:02:07

> Split your code into multiple namespaces and expose the user-facing functions in an api namespace > I think this solution is the spirit of the thread. So what methodology do you suggest for doing that?

pavlosmelissinos18:02:18

> So what methodology do you suggest for doing that? Well, by hand. Just do (def api-name #'impl-ns/name) or even (defn api-name [<API signature>] (impl-ns/name <impl signature>)) if you want different signatures. This way you can still have multiple APIs. depending on clear use cases or different versions (e.g. org.name.library.api.v2) and separate, curated docstrings, specs, etc. The main drawbacks of this approach is that you lose the metadata (which hasn't been that important to me) and the users have to "go to definition" twice (which is a minor inconvenience at best). You can still attach the metadata of an impl function to an api function (with alter-meta! et al) but it should be doable manually too.

pavlosmelissinos18:02:36

> with out additional boiler plate wrapping an already complete function What's boilerplate to you I consider thinking hard about your contract with your users. Both are valid views I believe and I'm not arguing I hold the truth or that I'm perfect. Far from it, I make this mistake all the time; design is hard.

Matthew Twomey18:02:22

> Both are valid views I believe and I’m not arguing I hold the truth or that I’m perfect. Far from it, I make this mistake all the time; design is hard. Yes yes. I didn’t mean to suggest otherwise. I think your view is perfectly legit and I apologize if I sounded adversarial. There’s room for all of us 🙂

pavlosmelissinos18:02:15

> I didn’t mean to suggest otherwise You didn't! 🙂 I just wanted to establish that I'm only expressing my opinion 😉

❤️ 2
skylize18:02:12

I think it's not always just a question of boilerplate. Quite often I want to design a contract for external use, and then use that same contract internally. I have no need or desire to introduce a second internal-only contract when the one I made for consumers already does what I need. If all I want is to expose that same contract from a more convenient location, then a using a wrapper call to do that wrapper must be kept in sync with the implementation, adding a new place for bugs to crop up. And the loss of metadata for the external API is no small loss (docstrings in the editor come to mind here) so that will need to be dealt with in some form or another, adding even more opportunity for bugs. All of that adds no value of it's own. It only heavy-handledly works around an unfortunate restriction of being in Clojure.

skylize19:02:37

That's not to say that can't be a good solution if your internal code generally has different needs from the same algorithms.

pavlosmelissinos19:02:34

> Quite often I want to design a contract for external use, and then use that same contract internally. I have no need or desire to introduce a second internal-only contract when the one I made for consumers already does what I need. Why do you need a contract internally? You are the author, just use the implementation. > the loss of metadata for the external API is no small loss (docstrings in the editor come to mind here) Ah, docstrings are a special case! User documentation is different from developer documentation. They require different language and degree of information, they shouldn't be the same.

pavlosmelissinos19:02:17

Once again, I don't consider all this boilerplate/a boring job that should be automated. It's part of design.

skylize19:02:54

> Why do you need a contract internally? You are the author, just use the implementation. > Exactly. Because I've already designed a contract for external use, I've already got a function I can just call. But for internal organization, I might want to group that function with things the user has no interest in, rather than next to -main > Ah, docstrings are a special case! > Not a special case. Just an obvious and common case that happens to be particularly pertinent to the external API.

pavlosmelissinos19:02:09

> Not a special case. Just an obvious and common case that happens to be particularly pertinent to the external API. It's a special case because you shouldn't want to just "echo" the technical documentation in the API namespace in the first place. For other metadata, sure but not for the docstring. And if you had to choose one docstring to keep, it should be the one in the API, not the technical one. > Exactly. Because I've already designed a contract for external use, I've already got a function I can just call. But for internal organization, I might want to group that function with things the user has no interest in, rather than next to -main I'm not sure I follow. The contract itself is something that exists in the implementation as well, you can organize that however you (and your team) want, as long as the contract stays intact.

skylize20:02:46

I have a function foo who's signature is already equal to the external API contract for foo. In this case, moving that code into a foo-impl and introducing another function foo just to call foo-impl verbatim is nothing more than adding a place for bugs to creep in through failure to keep the 2 functions in sync. But that seems to be your suggestion if I want to expose foo externally in a different namespace than where it is stored. Beyond that, perhaps you also have a convenience function easy-foo that calls foo with useful defaults. What namespace do you put that in? If you put it in -main, then you have yet another thing to keep in sync across multiple files. When working on foo, how do you even remember you need to be concerned about easy-foo? If you put it with impl, then now you have another function that needs to be wrapped in -main. Now you have a 5 functions to keep synced (`core/easy-foo`, core/foo, foo/easy-foo, foo/foo, foo/foo-impl) where 2 should have been sufficient.

kennytilton20:02:03

Seeing some bright spots here. 1. The elaborate docstring can go on the API version of a function. 2. The double-bounce to find the source behind a function should happen only when the library user needs to see my implementation. That should not be a regular thing. There, I feel better. Uh-oh. Is this going to work for macros?! 🙀

skylize20:02:15

> The elaborate docstring can go on the API version of a function. > Still an "onus on the creator". Why shouldn't get to benefit from your own docstrings in the editor? > The double-bounce to find the source behind a function > I see no problem with the double bounce. This shows the user explicitly what's actually happening. "You are accessing this function here. But this is just pointer. So go deeper to see what it points to."

skylize20:02:44

Looks like I misquoted Alex: He suggested load-file, not load. Don't know if that has any impact on availability in CLJS or Babashka?

kennytilton20:02:00

The only comments I write for myself are reminders of edge cases that forced a code oddity.

skylize21:02:18

Yeah, I don't think that's what I meant. The docstring is for the consumer. But if using the function internally, you are also the consumer. Why should you not get help from your editor to remember what you wrote 6 months ago, the same as a consumer of your library would?

pavlosmelissinos21:02:14

> I have a function foo who's signature is already equal to the external API contract for foo. > In this case, moving that code into a foo-impl and introducing another function foo just to call foo-impl > verbatim is nothing more than adding a place for bugs to creep in > through failure to keep the 2 functions in sync. But that seems to be > your suggestion if I want to expose foo externally in a different namespace than where it is stored. I'm sure I'm missing something but if you're the library author just use impl foo (the namespace where foo is stored, as you say), no need to introduce a third instance. > Still an "onus on the creator". Why shouldn't get to benefit from your own docstrings in the editor? Your own docstrings are for you, the API docstrings are for the user. The internal docstrings can explain the technical whys in all the gory details, especially if something is unintuitive. If a technical decision doesn't affect the user however, it doesn't belong in the API. Different goals. > The docstring is for the consumer. But if using the function internally, you are also the consumer. Why should you not get help from your editor to remember what you wrote 6 months ago, the same as a consumer of your library would? You are not "a consumer", you are the author. If you don't remember what you wrote, a quick look at your code (and your tests) should help you remember. If that's not enough you can still look at the docstring of the API to freshen up your memory but that won't be something you'll rely on regularly. If that's not enough either, well, then maybe that part of the code deserves a rewrite, what can I say... I really don't see the problem.

Matthew Twomey21:02:19

Not really in this part of the conversation, but I myself can say that 2 years after writing a lib for my own use, it might as well have been written by someone else lol. It’s basically a 3rd party lib at that point (I may have spend hours writing it, but I’ve since spent dozens of hours using it).

2
Matthew Twomey21:02:07

Can’t tell you how many times I look back at something I’ve been using and think: “I wrote that? I don’t even remember writing that…”

pavlosmelissinos21:02:49

2 years? I've written code 2 months ago that I don't remember writing... 😄

Matthew Twomey21:02:00

ha ha ha lol fair.

kennytilton21:02:55

Have you all forgotten Tilton's Law? "Spend more time on function and parameter names than on the algorithm." As for the algorithm, if it has to change at all it will be rewritten from scratch anyway.

godfather 2
😀 2
Matthew Twomey21:02:04

That’s a good one!

skylize22:02:16

> I'm sure I'm missing something but if you're the library author just use impl foo (the namespace where foo is stored, as you say), no need to introduce a third instance. > You're right. easy-foo can call foo-impl instead of foo, but we've still doubled from 2 to 4 functions, spread across 2 different files with no added value. > Your own docstrings are for you, the API docstrings are for the user. The internal docstrings can explain the technical whys in all the gory details, especially if something is unintuitive. If a technical decision doesn't affect the user however, it doesn't belong in the API. Different goals. > Maybe different goals. Maybe exactly the same goals. If the docstring for the external user is sufficiently informative, went should I be required to write another docstring for me to benefit from the one I already wrote for the user? > You are not "a consumer", you are the author. If you don't remember what you wrote, ... . > Cannibalism. 😄 If a function is written, and I am calling it how is it any different from another consumer calling it. At that point, I am both the author and the consumer. Same as if I wrote that function in another library, just without a notation in deps.edn. All your suggestions that follow are perfectly fine workarounds. But if I'm calling that function, why should I be robbed of the convenient option to just read a docstring that's already written from a popup, like I would with any other function?

skylize22:02:04

> As for the algorithm, if it has to change at all it will be rewritten from scratch anyway. > As for the algorithm, if it has to change at all, the whole project will be rewritten from scratch in another language anyway. 😛

☝️ 2
skylize22:02:28

Correction to my correction because of a correction?😄 Alex suggested load-file, but corrected himself to load. So I was more right the first time I quoted him: load.

👍 2
pavlosmelissinos22:02:08

> easy-foo can call foo-impl instead of foo, but we've still doubled from 2 to 4 functions, spread across 2 different files with no added value. Well, it seems we're running in circles. I've tried to explain that it's not "with no added value" but obviously failed. 🙂 You have a namespace that contains your contract, everything that a user is allowed to use to interact with your library and one or more namespaces that contain the implementation. These serve different goals by definition. > Maybe different goals. Maybe exactly the same goals. If the docstring for the external user is sufficiently informative, went should I be required to write another docstring for me to benefit from the one I already wrote for the user? Well, I disagree but we've started repeating ourselves so let me close try to wrap this up by saying that potemkin is definitely convenient, so I can see the value if you frame this as a problem that needs fixing, but, in my experience, a little inconvenience (e.g. some code/text duplication) is much better than using magic and potentially breaking go-to-definition in some cases. > If a function is written, and I am calling it how is it any different from another consumer calling it The difference is that you're in control of the changes. Changes in the implementation will not cause a breaking change because you're in control of that (regressions are definitely possible but will either get reverted or fixed). On the other hand, changes in the API will break the user's code. The API is for connecting with the outside world, you don't control where/how it's going to be used, only what the user is allowed to use. > the whole project will be rewritten from scratch in another language anyway I'm curious, you clearly consider this a problem, which languages have you used that offer a good solution to it?

skylize23:02:59

> Well, it seems we're running in circles. I've tried to explain that it's not "with no added value" but obviously failed.  ​ You have a namespace that contains your contract, everything that a user is allowed to use to interact with your library and one or more namespaces that contain the implementation. These serve different goals by definition. > This is where we partially disagree. If your design is already benefiting from defining separating explicit contracts, then the "hoops" are not hoops at all. Everything just works together as you describe. No problem. But if your design is such that the function's own signature serves quite sufficiently as its own contract, and the docs written for the consumer are quite sufficient for self consumption, etc., and this is generally consistent across the project, then putting facades in front of every function just to say the consumer is allowed to use it provides no value, only extra potential pain points. > which languages have you used that offer a good solution to it > JavaScript allows all of the following variants.

export * from "module-name";
export * as name1 from "module-name";
export { name1, /* …, */ nameN } from "module-name";
export { import1 as name1, import2 as name2, /* …, */ nameN } from "module-name";
export { default, /* …, */ } from "module-name";
The one I would typically use, would be
export { foo } from './foo'
which would import foo/index.js and re-export the foo variable exported by that file as its own.

pavlosmelissinos23:02:58

> putting facades in front of every function just to say the consumer is allowed to use it provides no value, only extra potential pain points. This "facade" you mention carries an implicit "DO NOT BREAK THIS" label. So you'll have tests for it to make sure you don't introduce any regressions and as long as you respect those, you can do whatever you want with the implementation. import-vars on the other hand just says "api.a is the same as impl.a". It won't protect you against regressions in impl.a because it's just a pointer. > JavaScript allows all of the following variants. JavaScript is notorious for breaking changes. Don't you think it may have something to do with design decisions like this one? Python supports this too btw, you can also do from x import a (which is kinda like :require x :refer [a]) and a will be visible from other modules (namespaces). Is it good design though? Everything you expose via this technique de facto becomes part of your API, even if you didn't intend it to be. That's very likely to happen and after it happens you can't take it back, so it's definitely bad practice in my book. (Hey, it's gotten quite late here and I assume it's just the two of us, so it might be time to stop burdening #beginners with philosophical debates 😛 Happy to continue in DMs tomorrow though!)

skylize00:02:45

I will delay writing another response, so you don't feel pressured to continue without sleep. 😇 I don't see any reason this needs to move to DM. We're very much still on topic for the original question (even if failing to offer any more meaningful solutions 😟). The original poster seems to be enjoying and participating in the discussion. And at this stage, all the points are being made at a pretty high level, not digging down into dark corners of the language or obscure areas of computer science. Anyone knowledgeable enough to understand the original question should probably be able to keep up well enough at this point if they choose to.

👍 2
Matthew Twomey00:02:28

I am still reading at my leisure so feel free to continue or not continue 🙂. As far as my initial question, borkdude did indeed provide the necessary insight I needed (that I can split my namespace how I like, using in-ns, and represent it rolled-up in a parent namespace when I deem appropriate to do so without using potemkin).

👍 4
skylize22:02:46

> This "facade" you mention carries an implicit "DO NOT BREAK THIS" label. So you'll have tests for it to make sure you don't introduce any regressions and as long as you respect those, you can do whatever you want with the implementation. > If we follow the standards of Clojure culture, then any defn that is documented in the readme also carries an implicit "DO NOT BREAK THIS". I can write tests directly against the function, and as long as I respect those, I can do whatever I want inside the body. > import-vars on the other hand just says "api.a is the same as impl.a". It won't protect you against regressions in impl.a because it's just a pointer. > I don't need it to protect me! I am an adult. I can edit a function without changing the signature, while honoring the behavior promised. I don't need a babysitter function sitting next to me to enforce that. All I need/want is a pointer, so that the shape of the public API is not glued to the private organization of my file system. If at some point, I have needs from the implementation that diverge from the User API, I can abstract the implementation away then. The function becomes the facade: a facade that actually serves a purpose, because the API and implementation no longer agree. > JavaScript is notorious for breaking changes. Don't you think it may have something to do with design decisions like this one? > JS loves breaking changes because they have fully embraced semver, which says "Please make all the breaking changes you want, whenever you feel like it. Just be sure to increase the version number correctly when you do." That and a culture-wide attention deficit. 😉 > Python supports this too btw > Cool > Is it good design though? Everything you expose via this technique de facto becomes part of your API, even if you didn't intend it to be. That's very likely to happen and after it happens you can't take it back, > Some variants are problematic in that way.

export * from './foo'
Just blindly re-exports everything. Asking for trouble, when someone unwittingly exports something for use by another module, and suddenly it's public. (It can still be convenient and reasonably safe if the module containing this call is entirely private.) On the other hand
export { foo, bar, baz } from './foo'
implies that you have vetted these as ready for public consumption. It makes hierarchical file organization very practical and useful, regardless of whether or not your public API should reflect that same hierarchy. You can easily export things up the tree into submodules. From there you can just document all the submodules, make a single entry point to a hierarchy by exporting whole submodules, or pick and choose things from each submodule that should compromise a flat API. The API layer can be literally just a big list of exports, and nothing else. A nice clean overview of everything that will be exposed. And all those things being exported get to actually live where they belong, next to their impls, or, deps, or siblings.

pavlosmelissinos08:02:51

And you don't like (def x "docstring" impl/x) because a) it doesn't preserve metadata and b) when the docstring is the same as the implementation's you don't want to have to repeat it. I hear that. I haven't really considered that a problem because, like I said, a proper API in my opinion is explicit, building it should be like a ceremony and the DRY principle doesn't exactly apply to it. I also want to be able to inspect it in my IDE and see the available symbols and their documentation. Good luck finding tools that support this well. However, you can always request an export feature at http://ask.clojure.org. I'll upvote it even!

👍 2
pavlosmelissinos08:02:33

Perhaps you'd be interested in polylith. In that architecture, you have multiple components and each one has an explicit API. Even if you're just exposing functions, you are expected to take the time and flesh out each item in the API. That's an intentional design decision because, in the case of polylith, if you're a project collaborator you're indeed a user of that component as well, unless you're working on its implementation. That works pretty well if you're a fan of small namespaces.

Daniel Shriki12:02:23

Hi Guys! I’m using next.jdbc & honey.sql to query my sql db. I need to implement a table that contains a JSON column. what is the recommended way of querying records by fields that are on that JSON? tnx 🙏

seancorfield16:02:05

Best to ask in #C1Q164V29 -- and let people know what database you are using.

Donald Pittard14:02:13

Does anyone focus on building non-web-based desktop applications in Clojure? The last time I did desktop programming in Java, Swing was the thing. I've also built something with Electron before. My real job is as a web developer though, so I'm wondering if there is any guidance around building modern desktop applications in Clojure.

delaguardo14:02:21

https://github.com/HumbleUI/HumbleUI the most recent addition in the tools set. very beta but worth looking at if you feel adventurous

6
thanks2 4
lread14:02:59

There is also https://github.com/phronmophobic/membrane, also early days, but very interesting. #membrane

thanks2 4
Hendrik15:02:54

There is also https://github.com/cljfx/cljfx It wraps javafx in a declarative way.

thanks2 2
respatialized15:02:43

one additional note on HumbleUI - while the library itself is indeed as @U04V4KLKC said "very beta but worth looking at if you feel adventurous," it is an interface to https://skia.org/, which is a very mature and stable technology. It's used by Chrome for 2d drawing, so in effect HumbleUI/Skija help tear out many of the intermediate layers between code and pixels that exist in something like Electron.

thanks2 2
respatialized15:02:53

http://github.com/lilactown/humble-starter this starter kit is super helpful for getting a live-reloadable application running with HumbleUI.

👍 4
thanks2 2
Rupert (All Street)16:02:12

If you need something really trivial (quick and dirty) then I think https://github.com/clj-commons/seesaw library probably still works for Swing UI in Clojure. Even if you are building a desktop app you could launch a web server (bound to localhost only) and launch the user's default web browser java.awt.Desktop.getDesktop().browse(new URI(""))

2
phill16:02:38

Swing is quite pleasant to work with in Clojure. I suppose JavaFX was conceived to work around some unfixable deficiencies in Swing. But after clamoring about Swing's peculiarities, pretty much everyone shut up and tiptoed away when they saw the solution. Now I use Swing (when the need arises) and I do not complain about it.

👍 2
Lycheese17:02:53

I have an atom whose contents I would like to clear when not accessed for x minutes. I think the clojure.core.cache TTL cache does not have a mode for resetting the timer on access, right?

delaguardo17:02:59

no but there is caffeine library that might have https://github.com/ben-manes/caffeine

dpsutton18:02:22

what is the behavior of the ttl cache from core.cache? Does the ttl not “extend” on hit? IE, if ttl is 5 seconds, in 5 seconds it will be evicted, regardless of hits? And you want each hit to reset the 5 seconds it has to live?

Lycheese08:02:40

@U04V4KLKC Thanks for the hint. I found https://github.com/ben-manes/caffeine/wiki/Eviction#time-based and expireAfterAccess does exactly what I wanted^^ @U11BV7MTK No, as far as I can tell the TTL cache does not extend on hit. And yes, I want the TTL be reset each time I access it.

dpsutton13:02:37

I suspect it would be easy to copy the source of the ttl cache and make your own. Possible they would take a patch on that as well.

Alexandre EL-KHOURY17:02:41

Hi guys ! Im using metosin/reitit-swagger {:mvn/version "0.5.18"} and I'm having some difficulties to define an optional body-param. Here's how my code looks like :

{:post {:roles #{"employee" "admin"}
             :spec ::upload-request
             :handler post-asset
             :summary "Post an asset"
             :parameters {:path {:company-eid string?}
                          :body {:asset-category string?
                                 :model-subcategory string?
                                 :thumbnail string?
                                 :name string?
                                 :model string?
                                 :gender string? ;;optional
                                 :accessory-subsubcategory  string?  ;;optional
                                 }}
             :swagger {:security [{:Bearer []}]}}}
Thanks 🙏

practicalli-johnny18:02:31

Is there a Clojure function, dynamic var or expression I can evaluate in the REPL to show the class path currently being used by the REPL? I cant seem to find anything that works in a search. Thanks. I know I can see the classpath before or whist running the REPL, just wondered how to see the classpath once the REPL is running (without restarting it)

seancorfield18:02:33

(System/getProperty "java.class.path") ?

seancorfield18:02:24

You'd need to str/split the result on (System/getProperty "path.separator") since it's O/S dependent.

hiredman18:02:08

clojure (and by extension the repl) doesn't "use" the classpath. the jvm uses the classpath to configure one important classloader and boot up. it is easy, and common in some cases (the add-lib branch of tools-deps) to load code into the jvm that isn't present on the classpath, because what really determines things are classloaders

hiredman18:02:21

(I say this just in case you are writing tooling and about to make it depend on finding information on the classpath, which is annoying if you are working in a more complicated classloader environment)

Alex Miller (Clojure team)18:02:04

(and even that model is wrong for apps started with modulepath)

hiredman18:02:53

I try to avoid working blue

practicalli-johnny18:02:53

Thanks @U04V70XH6 that was the property I was looking for... I only use Linux so just hard-coded the separator

(sort-by count (clojure.string/split (System/getProperty "java.class.path") #":"))

practicalli-johnny19:02:54

Or for an o/s independent version

(sort-by 
  count 
  (clojure.string/split  
   (System/getProperty "java.class.path") 
   (re-pattern (System/getProperty "path.separator"))))

Timofey Sitnikov19:02:30

Hello all, trying to clean up some data, but having trouble. How would you go about doing this? Filtering does not work, since criteria can be met in the middle.

tomd19:02:12

(let [coll [[42.5 0 3.1] [42.5 0 3.2] [42.5 0 3.4] [42.5 15177 3.1] [42.9 16104 3.1] [42.9 0 3.2] [42.9 0 3.0]]
      indexes (keep-indexed (fn [i [_x y _z]] (when-not (zero? y) i)) coll)]
  (->> coll
       (drop (dec (first indexes)))
       (drop-last (- (count coll) (last indexes) 2))))

2
Ben Sless19:02:55

drop then take-while or take-until

tomd20:02:12

drop requires an arg to know what to drop, hence the above. take-while takes one too few, and take-until doesn't exist in core, hence the above

Timofey Sitnikov20:02:12

OK, awesome, it took me a minute to understand it, but yes, perfect, it works.

👍 2
Ben Sless06:02:39

take-until is just take-while, in a sense 🙂

tomd07:02:49

Ah I thought you were referring to the take-until https://cljdoc.org/d/org.flatland/useful/0.11.6/api/flatland.useful.seq which takes while, plus one more 🙃

Abhi Saxena21:02:24

Hi Clojurians, I have a json payload for API something like:

{"filters" {"game" ["Phase II"] "getThroughIndication" ["M1" "M2" "M3"]}}
I want to run these attributes i.e. game, getThroughIndication through a Stored Procedure but the variables for Stored Proc are very different i.e. game, getIndices etc. What's the best way to convert the coming keys in JSON to the required attributes by Stored Procedure......

Bob B21:02:29

perhaps something like:

(update-keys {"a" 1 "b" 2} {"a" "c" "b" "d"})
;; => {"c" 1, "d" 2}

Bob B22:02:33

I think update-keys was added to core in 1.11 (based on the var's meta), so if you're on an earlier version it might not be readily available

Abhi Saxena22:02:16

thanks for your response Bob, I am on version 1.9.0 so update-keys would not work for me. Also I am not sure which keys will be available in the payload, it might contain some or none

Bob B22:02:21

the backport looks like it would be pretty trivial, but a shortcut would be using zipmap with (map f (keys m)), where f is the transformation function

Bob B22:02:24

if you happen to have it lying around, the clojure cookbook has a map-keys function written out

Abhi Saxena22:02:49

okay..thanks much for your help

skylize01:02:35

I am always looking for that and update-vals. But I can never remember what they're called and end up writing my own impl again. :melting_face: Hopefully I'll remember this time. 🤞

skylize01:02:03

Would be nice if they showed up on http://ClojureDocs.org

Abhi Saxena05:02:12

(defn map-keys [f m]
  (->> (map (fn [[k v]] [(f k) v]) m)
       (into {})))
this function worked for me. thanks @U013JFLRFS8 and @U90R0EPHA

skylize06:02:52

Looks more-or-less how I usually implement it. Simple and straightforward. 👍