Fork me on GitHub
#clojure
<
2018-12-12
>
ag03:12:43

What's the best way to deal with the problem of recursively fetching something via rest api? so I need to walk folder structure in Box using its api. You can get a folder and its content in a single request, If a folder has 300 sub-folders, you need to send 300 requests. My recursive implementation using pmap and reducers/fold is way too slow. Can someone suggest any good blogposts or whatever? I'm thinking if core.async would be any faster here? Can someone suggest good approach (explain your idea at a high level), e.g. you send first request, then store the results in an atom, then map over that list, etc... etc...

myguidingstar05:12:12

you may have the impression that Pathom is for om.next, but actually it can be used independently

myguidingstar05:12:54

it has support for async operations, too

nha10:12:05

I haven’t used pathom - would be interested in reading how it can be used independently thouhg

aisamu11:12:50

What is the slow part? I'd place my bets on the HTTP calls being the bottleneck, and then it makes little difference (on speed) whether you're using threads, channels or pathom. Just make sure your threads/channels aren't all waiting/parked and you should be near your max throughput.

☝️ 4
Marc O'Morain11:12:06

How long does 1 API call take, and how long does 300 take? Is this “way to slow”?

potetm03:12:34

interestingly enough

potetm03:12:48

I did something very similar to this for a job application once

potetm03:12:31

Well, they required that you keep some amount of state per node in the tree

potetm03:12:39

which I’m not sure that’s a requirement here

potetm03:12:11

but basically, start a go block that creates a bunch of other go block, then waits on results from each

potetm03:12:16

and do that recursively

4
eccentric J03:12:41

Can you start a socket repl server from the clj cli?

potetm04:12:18

that^ isn’t working for me — looking further

potetm04:12:14

clj -J-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"

eccentric J04:12:57

Ah thank you. For some reason I had a tough time trying to track down an example.

emccue05:12:52

I was showing one of the other students at my college clojure and they ran into one of the 1.10 error messages (def vs defn) almost immediately

Alex Miller (Clojure team)13:12:35

If you can recreate, would be curious to see

Alex Miller (Clojure team)13:12:50

And thanks for the feedback

emccue05:12:33

Corrected the mistake quickly and understood the error completely

🎉 36
emccue05:12:03

Solid thumbs up for 1.10 from me

Audrius09:12:49

is it possible to read lazy stream twice? like something from http stream. Or I better is to read everything to a string and use multiple times?

Alex Miller (Clojure team)13:12:10

Reading it consumes it (it is stateful) so you can’t read it twice

caleb.macdonaldblack10:12:32

how do I use named parameters in clojure.java.jdbc? Do I have to repeat the variable in the sqlvec if I want to use it twice?

kirill.salykin10:12:36

I think #sql better place for this question

souenzzo13:12:58

About clojure erros in 1.10: if you put a random macro in middle of some file (example: defn ) then load this file to repl:

Syntax error compiling at (user.clj:1:1).
Can't take value of a macro: #'clojure.core/defn

bronsa13:12:04

what would you expect instead?

souenzzo13:12:35

anything NOT in 1:1

bronsa13:12:32

I'm not sure what you mean

bronsa13:12:17

it's the error location the issue

bronsa13:12:34

I'd say open a ticket in jira

Alex Miller (Clojure team)14:12:38

if you have a repro, would like to see it

Alex Miller (Clojure team)14:12:17

is the user.clj in that namespace really a user.clj being side-loaded by the runtime or is it just a file with a longer namespace like foo/user.clj

bronsa14:12:10

@alexmiller

[~]> cat test.clj
(ns test)

(defn foo [])

defn
[~]> clj
Clojure 1.10.0-master-SNAPSHOT
user=> (require 'test)
Syntax error compiling at (test.clj:1:1).
Can't take value of a macro: #'clojure.core/defn

bronsa14:12:22

he's complaining about the line/col number, I believe the filename is accurate

Alex Miller (Clojure team)14:12:18

I believe it’s accurate, asking whether this is a user.clj being loaded by the RT or if it is just from a normal require (which are different code/error report paths)

👍 4
Alex Miller (Clojure team)14:12:45

but I think that example is enough to answer that

danielneal16:12:16

@alexmiller is it out of the question that clojure might one day allow docstrings on keywords?

danielneal16:12:50

I'm finding myself attaching (sometimes quite subtle) meaning to namespaced keys in our programs, and I get the feeling that sometimes this meaning would be expressed better in a sentence than a name

NoahTheDuke16:12:10

::this-keyword-is-for-function-X-and-it-means-that-we-should-trigger-function-Y-after-it-finishs

😂 12
noisesmith16:12:02

keywords can't carry metadata because they are interned - symbols can have docstrings as they can have metadata

bronsa16:12:46

you could have a separate db for docstrings of namespaced keywords

bronsa16:12:55

like it's done for specs

noisesmith16:12:00

I mean I guess in theory a given keyword could have a docstring, and every instance of that keyword would have the same doc, but that would likely be a big change

bronsa16:12:13

the docstring doesn't have to live in the metadata of the object

mpenet16:12:26

Exactly. I did that (and other questionable stuff) in https://github.com/mpenet/spex

mpenet16:12:01

Private meta data registry for specs. Same approach works for other contexts

Alex Miller (Clojure team)16:12:23

as others stated above, yes it’s out of the question

Alex Miller (Clojure team)16:12:26

independent context-specific registry keyed on keywords would be better

danielneal16:12:07

cool thanks for the answer 🙂 that makes sense

mpenet16:12:13

Guess there could be a Doc protocol for this

bronsa16:12:22

but is it out of the question a centralised registry for namespaced keywords that worked for doc?

mpenet16:12:33

That would allow doc to work

bronsa16:12:38

even better

danielneal16:12:42

😄 my next question @bronsa !

bronsa16:12:59

too slow buddy

Alex Miller (Clojure team)16:12:58

doc already works for keywords

Alex Miller (Clojure team)16:12:06

by looking them up in the spec registry

Alex Miller (Clojure team)16:12:36

user=> (doc :clojure.core.specs.alpha/params+body)
-------------------------
:clojure.core.specs.alpha/params+body
Spec
  (cat :params :clojure.core.specs.alpha/param-list :body (alt :prepost+body (cat :prepost map? :body (+ any?)) :body (* any?)))

bronsa16:12:41

ah yes, good point

Alex Miller (Clojure team)16:12:53

what is missing is the ability to specify doc or other meta for spec keywords

Alex Miller (Clojure team)16:12:00

but we have every intention of adding that

danielneal16:12:01

ah and this is already a ticket, I remember seeing that

Alex Miller (Clojure team)16:12:15

it is the highest voted ticket in jira :)

danielneal16:12:38

democracy.txt

roklenarcic18:12:45

When using Cheshire JSON lib, is there a way to tell it to drop map keys where value is null?

roklenarcic18:12:22

I can walk the structure after and do it myself, but it's not the most performant option.

petterik18:12:00

Was surprised today to find that in 1.10 keyword can only take symbols as values when passed 1 argument, not 2:

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.0-RC5"}}}'
user=> (keyword 'foo)
:foo
user=> (keyword 'foo 'bar)
Execution error (ClassCastException) at user/eval3 (REPL:1).
clojure.lang.Symbol cannot be cast to java.lang.String

petterik18:12:49

Converting a structure of entities with field names to a map with namespaced keywords:

[Person [first-name]] => {:person/first-name}

petterik18:12:52

Wasn't able to find the Jira for the change where keyword now accepts symbols :man-shrugging:

petterik18:12:33

Thinking about it more, there'd have to be checks whether the symbol has namespaces, so I guess this makes sense. I.e. what would happen when passing (keyword 'foo/bar foo/bar)?

petterik18:12:01

Sometimes questions can be answered by just typing them out 🙈

seancorfield18:12:11

(! 577)-> clj
Clojure 1.9.0
user=> (keyword 'foo)
:foo
user=> (keyword 'foo 'bar)
ClassCastException clojure.lang.Symbol cannot be cast to java.lang.String  clojure.core/keyword (core.clj:606)
user=> 
Same behavior in 1.9.

seancorfield18:12:43

I think it's been that way for a long time.

petterik18:12:46

oh? I thought keywords accepting symbols was new? I must be thinking of something else

seancorfield18:12:55

Symbols accepting keywords is new.

👍 4
petterik18:12:33

Ah! It was the other way around

petterik18:12:43

Clojure 1.9.0
user=> (symbol :foo)
ClassCastException clojure.lang.Keyword cannot be cast to java.lang.String  clojure.core/symbol (core.clj:579)

seancorfield18:12:37

Yup. symbol now works on keywords and vars in 1.10.

👍 8
orestis19:12:25

I’m sure I saw some work being done there recently (the name demunging part), but it seems it doesn’t work in this particular case?

Alex Miller (Clojure team)19:12:02

that’s in a different place (error message printing in main)

Alex Miller (Clojure team)19:12:49

is there a specific ticket that he is “waiting” for?

Alex Miller (Clojure team)19:12:39

I am not aware of one, happy to have one or more tickets on clojure.test reporting

orestis19:12:06

Message relayed. I’ll stop now :)

Alex Miller (Clojure team)19:12:14

hard to say “waiting for Clojure.core” is the right phrase when no one has logged or worked on it? there is no resistance to improving this.

orestis19:12:48

Twitter sucks. Sorry for bringing this up in such a way, I could have filtered the message instead of just passing it along.

dpsutton20:12:49

those error messages aren't great. perhaps his patch can land in the actual lib? twitter seems to have worked here?

hiredman20:12:48

I dunno, I am extremely skeptical of that kind of munging

hiredman20:12:57

or demunging or whatever you want to call it

hiredman20:12:08

I mean, great that that person is happy with what they have, but everytime I have used some kind of stracktrace manipulating(printing,formatting,filtering) library that someone else wrote it has driven me nuts

Alex Miller (Clojure team)20:12:10

@orestis no worries, happy to see it as I’m not actively looking at twitter these days

❤️ 12
danielneal20:12:59

I think this is a good way to stay sane :)

cfleming22:12:55

For a long time I’ve had my twitter in notification-only mode, where I only see DMs and mentions. It’s a huge life improvement.

Alex Miller (Clojure team)20:12:40

there are at least 3 ideas bound up in that change above

Alex Miller (Clojure team)20:12:00

1) focusing on root cause in chain - endorse

Alex Miller (Clojure team)20:12:36

2) filtering stack head to remove test infrastructure - endorse (and we’re already doing that - this is just more aggressive)

Alex Miller (Clojure team)20:12:32

3) demunging from Java stack to Clojure function - I’m a bit more ambivalent about this one but not necessarily opposed in this context

Lennart Buit20:12:23

glad that you guys are looking at error reporting!

Alex Miller (Clojure team)20:12:07

but the way to approach all of these is by backing up to principles and goals, like we did with repl error reporting

Alex Miller (Clojure team)20:12:22

and determine the set of changes needed to improve what it does

Alex Miller (Clojure team)20:12:44

so that it satisfies the goals in line with the principles

hiredman20:12:45

filtering out and prettying up of stacktraces is all well and good, until you get an exception where the parts you want are prettied away, then it is infuriating

Alex Miller (Clojure team)20:12:38

eliding the top stack frames that are the test infrastructure itself is generally something I favor, and that is already being done

Lennart Buit20:12:11

cursive does something similar when debugging, just not showing stack frames in clojures internals

Alex Miller (Clojure team)20:12:25

I think there is actually a ticket in the ballpark of part of #2 in https://dev.clojure.org/jira/browse/CLJ-1811

Alex Miller (Clojure team)20:12:58

I’m not a big fan of removing intermediate or bottom stack frames as I find those too often have info I want, but having some kind of a switch or pluggable filter for that kind of thing is a possibility

dpsutton20:12:12

friendly pst and a full pst would be nice

Alex Miller (Clojure team)20:12:38

the thing above is about clojure.test reporting

Alex Miller (Clojure team)20:12:47

similar but different issues in pst

Lennart Buit20:12:30

huh, I find test failures one of the more straight forward errors as it usually says which test case broke on what assertion

alexyakushev20:12:49

Sophisticated environments like CIDER offer just that – ability to toggle "uninteresting"/infrastructural frames. If the tool is on the simpler side (e.g., a bare clj REPL), it's better to err on the side of overtelling, at least for me.

alexyakushev20:12:45

All those "improving" libraries actually don't do any good in the long run. Particularly, because of aviso I now can never tell which end I should start reading the exception from.

NoahTheDuke21:12:08

on websites like rollbar, they shrink/collapse the non-app stacktraces, but let you open them all up. it's very nice to switch between

NoahTheDuke21:12:17

idk how easy that is on the command line

Niki21:12:41

I agree that filtering stacktraces is, well, depends on a situation and prob. shouldn’t be in core But I think no one would argue that datascript.query_v3$resolve_clause.invoke is less readable than datascript.query-v3/resolve-clause. And it seems we can do this sort of transition back to “what was actually called” pretty reliably?

✔️ 12
Niki21:12:02

also two frames (.invoke and .invokeStatic) per clojure call aren’t helping anyone I believe

Niki21:12:56

I can also see there’s some protocol machinery, clojure.lang internals etc

Niki21:12:54

wouldn’t it be nice if all this was translated back to clojure-like names, so you don’t have to be familiar with how Clojure code is compiled to read stack traces?

Niki21:12:29

I guess it’s too late to aim for 1.10?

Niki21:12:41

RC-79-final_final_3

dpsutton21:12:50

.pdf.jar

😂 4
Lennart Buit21:12:07

final-draft-really-final-v3

alexyakushev21:12:13

Wait, didn't some demunging just land into 1.10, or am I imagining things?

Alex Miller (Clojure team)21:12:06

nothing has changed there

Alex Miller (Clojure team)21:12:16

we didn’t touch pst in 1.10

alexyakushev21:12:12

I see that now. I compared to 1.10-0-beta3 which for some reason has (pst) results unmunged. Nevermind 🙂

Alex Miller (Clojure team)21:12:40

I would not expect there to be any difference in any 1.10 release in pst

Alex Miller (Clojure team)21:12:51

we are demunging the location you see in the first line of the exception message in many more cases (the user/mydiv), so that’s different in 1.10

lilactown22:12:39

can meta data being added to everyday Java objects?

noisesmith22:12:14

if they implement the right interface, sure

lilactown22:12:34

but it’s not implemented for Object ?

noisesmith22:12:48

that's the interface you would need

noisesmith22:12:22

(for getting meta - I assume setting it is defined elsewhere)

seancorfield22:12:28

@alexmiller Re: clojure.test -- it hasn't had any love for a long time (only one commit -- for 1.10 -- since December 2015): is that because Cognitect folks don't use it much? Or it's considered "good enough" by them? Or just because no one is really advocating for improving clojure.test?

Alex Miller (Clojure team)22:12:41

Just general lack of attention I think

Alex Miller (Clojure team)22:12:44

But also there is less complaining about it than lots of other things

seancorfield22:12:17

Fair enough. I'm planning to have another run at Expectations at some point soon to get it to a 2.0 gold release and that should give me more insight into clojure.test so I may start picking over some of the open JIRA issues for it. Or adding some new ones, with patches, or at least kicking off discussions... Maybe.

❤️ 8
Marc O'Morain23:12:32

We wrote circleci.test to work around issues with clojure.test. Most acutely, the direct linking in 1.8 broke a lot of our test tooling. We also added the concept of global fixtures that are applied once per run of a test suite. https://github.com/circleci/circleci.test

seancorfield23:12:54

I've seen a few complaints about fixtures not being flexible enough...

Alex Miller (Clojure team)23:12:55

No tickets I’m aware of

seancorfield23:12:32

Too much effort, apparently 😆

rickmoynihan13:12:21

I definitely feel that clojure.test fixtures could be more useful. Being able to run a fixture once for a whole run is definitely one thing. In the past I’ve also found it frustrating that fixtures are configured at a namespace level; either once for the ns or for all deftests in a namespace. Sometimes it would be nice to be able to exclude a few deftests from being within a fixture etc. Perhaps with a metadata flag. Usually though it’s been easy enough to work around by just wrapping the deftest form itself in a with-fixture, so it’s not been something I’ve felt too strongly about. However I have many projects now that implement a similar thing in slightly different ways.

Alex Miller (Clojure team)14:12:07

with clojure.test, the namespace is a meaningful “group” of tests. rather than exclude some, put them in a different namespace.

Alex Miller (Clojure team)14:12:22

but having a per-run fixture is a reasonable idea

Alex Miller (Clojure team)14:12:49

I think having built-in selector support would be useful though, like what lein or test-runner provides

rickmoynihan14:12:59

> with clojure.test, the namespace is a meaningful “group” of tests. That is certainly one way to arrange it. The other which in my experience is better supported with editor tooling is having a foo.bar-test ns for each foo.bar ns under test. Granted the 1-1 mapping is also an artificial source of annoyance; but if you follow that convention editors can jump you to a test definition by convention. Ultimately I think the conventions here are a problem though; and that this sort of thing is probably better being data driven from an edn file or metadata forms which editor tools could read. I think kaocha seems to be taking this approach.

rickmoynihan14:12:08

but mapping test ns’s to sut ns’s can be a problem

Alex Miller (Clojure team)14:12:43

imo, the 1-1 thing is too dogmatic

rickmoynihan14:12:09

I do agree with that. It’s just a shame you can’t have your cake and eat it

Alex Miller (Clojure team)14:12:11

it’s a good general starting point but I find many reasons to deviate from it

Alex Miller (Clojure team)14:12:00

is jump-to-test really that critical? can’t say I ever do that, but maybe I’m an outlier

rickmoynihan14:12:03

As a project scales I definitely feel the urge to diverge from the 1-1 mapping. Though you can partly take that as an indication to restructure your apps namespaces too, so maintaining the 1-1 mapping isn’t a problem. Regarding jump-to-test etc… It’s not just jump-to-test; though I find that very useful; it’s also having the editor create a test ns file for you in the right place according to convention. It can be frustrating when you want to add a small test to have to build directories etc and name a file and then setup the ns requires etc…

rickmoynihan14:12:04

Not a massive problem I grant you; but friction here can lead to some tests not being written.

Alex Miller (Clojure team)14:12:01

I guess what would be helpful for me (in the big picture) is to have a prioritized list of the worst pain points for clojure.test

rickmoynihan14:12:38

Yeah… FWIW we use clojure.test inspite of its flaws; it’s certainly good enough. It’s nice to just have something that’s there - even if it has some rough edges. The above issues are pretty minor really; I feel there are perhaps some deeper issues too though. Writing custom assertions, and sometimes getting useful test failure messages out has been a pain for me in the past. I may well have just been doing it wrong. Also perhaps a bigger issue is integrating clojure.spec generative tests into a clojure.test suite. I’ve written some infrastructure (outside of clojure.test) that does this in some projects - but more native support would be good.

rickmoynihan14:12:04

Where do you think we should assemble such a list for prioritising?

Alex Miller (Clojure team)14:12:05

some shared web location :) google spreadsheet, design wiki (https://dev.clojure.org/display/design/Home), etc. there may even be some prior lists on the design wiki already.

Alex Miller (Clojure team)14:12:29

don’t see any lists there

Marc O'Morain15:12:09

I’m glad to hear that splitting out clojure.test is being considered. I would like to see this taken one step further, and the test runner being split from the assertions library.

Alex Miller (Clojure team)16:12:17

I think if we’re talking about changing what’s “in the box” with clojure.test, the changes need to be non-breaking or additive. if you want to do something substantively different, the only reasonable thing to do is put it in a new library (which many people have already done). So I don’t see how splitting the runner out of clojure.test helps anything. Either incrementally improve what’s there, or make a new thing, and if you’re making a new thing, don’t put it in core, and at that point nothing is blocking you from moving forward, and plenty of people have already done it.

Alex Miller (Clojure team)16:12:49

does that make sense? I’m hashing this out as I type :)

rickmoynihan16:12:40

What is the argument for splitting out the runner? Is it that the runner is more of a common infrastructure concern, and should therefore be closer to core (for tooling support etc?) than the choice of how you express assertions??

lilactown22:12:59

what I’m actually fishing for is in CLJS, how do I add metadata to a ReactElement. It seems that it’s just a bare JS obj, it has no special prototype it inherits from

lilactown22:12:56

but I’m loathe to implement IMeta / IWithMeta for object since I have no idea what that might do

seancorfield22:12:22

(I did a lot of work on Expectations, getting it compatible with clojure.test, so that all the tooling out there would work better with Expectations, but the failure reporting is really basic and, as @tonsky mentioned, stacktraces are pretty horrible when a test does blow up)

cfleming22:12:18

re: stacktrace filtering, Cursive does it using code folding, which means it’s hidden by default but you can undo it if you need to. That’s tooling-dependent obviously, but it works well.

seancorfield22:12:40

There are libraries to "improve" clojure.test failure reporting but they all rely on extending the test reporting machinery as I recall and they can easily step on each other if you accidentally combine them.

Alex Miller (Clojure team)22:12:08

I like expectations and covered it in Clojure Applied, along with clojure.test and test.check

Alex Miller (Clojure team)22:12:57

I find in Cursive’s folding that the first thing I end up doing is unfolding most of it. That is, I find it more irritating than useful.

Alex Miller (Clojure team)22:12:28

But in a couple dimensions I assume I am off the norm

Alex Miller (Clojure team)22:12:18

One thing we have batted around is pulling clojure.test out of core into a lib (like spec, so still available through a dep) to allow it to move faster

🙏 24
Alex Miller (Clojure team)22:12:02

walk and zip are other possible targets like that

simple_smile 4
cfleming22:12:19

@alexmiller Settings-&gt;Languages &amp; Frameworks-&gt;Clojure-&gt;Prettify stacktrace elements using folding &amp; Fold stacktrace lines from outside project

Alex Miller (Clojure team)22:12:53

Thx :). I am so lazy at configuring my tools

Alex Miller (Clojure team)22:12:20

Not sure if those are as useful to split out

cfleming22:12:13

Are any of those 3 likely to receive improvements? Or perhaps better asked as - are there pending improvements that are not being looked at?

Alex Miller (Clojure team)23:12:50

There are things out there for all of them and complete rewrites of both floating around

cfleming22:12:41

clojure.test seems like the most likely candidate since others have been inspired to write e.g. eftest

👆 4
dpsutton22:12:47

Hasn't there been a metadata bug in walk for a while? i'm thinking of ridley and metadata in macro expansion i think

seancorfield23:12:36

clojure.walk finally got a fix for pre walking hash maps, as I recall. Was that 1.10 or in 1.9?

seancorfield23:12:08

postwalk and, yes, 1.10.

timgilbert23:12:46

Say, anybody got a link to a good tutorial on zippers? Ideally Clojure, but any language is cool, I think I'm missing some basic concepts

Alex Miller (Clojure team)23:12:07

Maybe I can list some or all of these on the Clojure survey for possible improvement areas

simple_smile 4
danielglauser00:12:28

I filmed that! 🙂

simple_smile 4
danielglauser00:12:55

Glad folks are finding it useful.

timgilbert23:12:36

Those both look great, thanks guys