Fork me on GitHub
#clojure
<
2019-09-05
>
st3fan01:09:25

What is the proper way to run Jetty in the background in the REPL? I tried run-jetty handler {:port 3000}) but I think that runs in the foreground.

Alex Miller (Clojure team)01:09:38

I think there's another option you can pass

st3fan01:09:35

:daemon? true looks what i want

Alex Miller (Clojure team)01:09:17

Nah, I think it’s join?

ahungry02:09:39

is there a built in facility to reload or refresh an entire ns, or all active ns in a given repl session?

Alex Miller (Clojure team)02:09:45

require has :reload and :reload-all (reloads dependent ns'es too) options

Alex Miller (Clojure team)02:09:38

(require 'foo.bar :reload-all)

didibus02:09:48

I feel like I really don't understand datafy and nav. I thought it would make sense to use it to implement a lazy beanification. So say I call datafy on a Java object, it returns me the bean for it (clojure's bean fn). And then I thought I could nav into a key of the map, which would in turn call bean on the value and return that. But I can't figure it out

didibus02:09:19

But why does nav take a coll a key and a value ?

seancorfield02:09:27

@didibus have you read my blog posts about this, just out of curiosity?

seancorfield02:09:54

I'm happy to chat about this because I've been using it extensively with JDBC stuff...

didibus02:09:02

Maybe I'm also confused about the expected use from a user. Like I'm imaging I would be able to do: (-> a-java-object datafy (nav :field)) or something of the sort

seancorfield02:09:26

By default, datafy doesn't "do" much -- you need to extend the protocol for your specific case.

didibus02:09:58

Right, so I did:

(extend-protocol p/Datafiable
  Object
  (datafy [x]
    (bean x)))

seancorfield02:09:41

(doesn't that override the built-in definition?)

didibus02:09:54

Ya, its just for learning purpose.

didibus02:09:12

Eventually I would have it extend only specific types I want to bean navigate

didibus02:09:18

Say I have:

public class Person {
  public Car car;
}
public class Car {
  public String make;
}

didibus02:09:15

(datafy a-person)
;=>
{:car #object[car 0x234234....]}

didibus02:09:34

Up to here I'm good 😛

didibus02:09:53

But now my thoughts were I could nav into :car. And in turn get the datafied car back

seancorfield02:09:43

If you nav into that you'll get a raw Java object. Then you call datafy again.

seancorfield02:09:38

The process is Thing -> datafy -> data -> nav -> newThing -> datafy -> newData...

didibus02:09:17

Hum, nav can't automatically datafy?

seancorfield02:09:25

No, and it must not.

didibus02:09:50

So here I would need to extend clojure Map so that nav returns the value at key? And then call datafy on that ?

seancorfield02:09:11

Thing and newThing above are "native objects". It's a regular square of operations.

seancorfield02:09:49

A Clojure hash map can be nav'd natively.

didibus02:09:55

I guess I don't understand nav. Like, if nav works over data, why do I need it? I can just use get or anything else to navigate the data

seancorfield02:09:18

Because nav on datafy'd objects will produce new objects

seancorfield02:09:44

If you're in the Clojure world only, datafy is a no-op (identity), and only nav does anything.

didibus02:09:57

In the sense that it preserves meta?

seancorfield02:09:12

Not sure I follow...

seancorfield02:09:05

Example with JDBC. datafy turns a result set into a "result set that can be nav'd"

seancorfield02:09:23

And nav on that produces something that can be datafy'd.

didibus02:09:30

So doesn't my example make sense then? datafy needs to return a Java object that can be nav'd

didibus02:09:43

And this is where I guess I'm not sure how to proceed

didibus02:09:25

I don't understand why nav takes a value ?

didibus02:09:37

Its like a default? In case the thing is non-navigable ?

seancorfield02:09:49

I don't understand your question.

seancorfield02:09:11

It's Thing -> datafy -> data -> nav -> newThing -> datafy -> newData.

seancorfield02:09:22

I don't know how to explain it better than that.

seancorfield02:09:27

It's a square: native objects on the left, Clojure data on the right. datafy takes you from the left to the right. Normal Clojure navigation keeps you on the right. nav takes you from the right to the left.

didibus02:09:56

I think I'm starting to get it. Its probably just my example isn't as good a use case for it.

seancorfield02:09:29

datafy = go right; nav = go diagonally left & down.

didibus02:09:59

So in my example, I would do:

(extend-protocol p/Navigable
  clojure.lang.APersistentMap
  (nav [coll k v]
    (get coll k)))
Where ideally I don't extend APeristentMap, but instead use the new meta-protocol extend so only the Map my datafy return has this nav implementation.

didibus02:09:20

And v is kind of useless ?

seancorfield02:09:46

I've no idea what you're asking here. APersistentMap is already pure data.

didibus02:09:05

So you say datafy returns data

didibus02:09:10

And you call nav on the return of datafy

didibus02:09:23

So I'm confused, isn't nav meant to be called on pure data ?

seancorfield02:09:40

I don't know how to explain this, sorry.

seancorfield02:09:48

It makes complete sense to me.

didibus02:09:58

Haha, that's fine.

seancorfield02:09:17

You have to understand the square with non-data on the left and data on the right.

seancorfield02:09:28

datafy goes from non-data to data.

seancorfield02:09:49

Normal Clojure code (like get) navigates around on the right hand side (all data).

seancorfield02:09:11

nav works a bit like get but takes you back to the left hand side (non-data).

seancorfield02:09:29

If you're talking about Clojure data, none of this matters.

seancorfield02:09:47

data -> datafy -> data is identity

seancorfield02:09:15

so data -> nav is the same as data -> get -- and produces pure data.

seancorfield02:09:41

This only matters/makes sense when you're starting with non-Clojure-data.

valerauko02:09:07

what would be non-clojure data? java objects?

seancorfield02:09:02

@vale It could be anything you want to (lazily) navigate and have a different representation. Have you looked at REBL at all?

seancorfield02:09:47

REBL shows how to turn Vars and namespaces into (Clojure) data that you can then (lazily) nav to interesting new data.

seancorfield02:09:30

In particular, REBL lets you navigate through Vars into what they use and what uses them (dynamically).

seancorfield02:09:57

Also, with namespaces, you have navigate into the items within them etc.

valerauko03:09:04

no i haven't played with rebl yet. i couldn't yet figure out what i'd need it for in the first place

seancorfield03:09:33

In clojure.java.jdbc (experimental) and next.jdbc, you can datafy a result set and get a "navigable" vector of hash maps, that will lazily take you to a new table.

seancorfield03:09:09

@vale Maybe have a look at my YouTube videos about Atom/Chlorine/REBL?

seancorfield03:09:29

REBL is a core part of my workflow, every day now. I love it.

Alex Miller (Clojure team)03:09:46

in general, yes you use datafy/nav with java objects (as clojure data is already data and navigable)

didibus03:09:38

Hum, you can't add meta to APersistentMap ?

seancorfield03:09:14

Non-sequitur 🙂

seancorfield03:09:00

Metadata really has nothing to do with this discussion, IMO.

didibus03:09:17

(extend-protocol p/Datafiable
  Object
  (datafy [o]
    (with-meta (into {} (bean o))
      {`d/nav (fn [coll k v]
                (get coll k))})))

didibus03:09:32

Doesn't work

seancorfield03:09:48

That breaks a lot of stuff, "obviously"

seancorfield03:09:47

I don't know what to say -- you're clearly fundamentally misunderstanding how this whole thing should work 😐

didibus03:09:22

This does though:

(extend-protocol p/Datafiable
  Object
  (datafy [o]
    (bean o)))

(extend-protocol p/Navigable
  clojure.lang.APersistentMap
  (nav [coll k v]
    (get coll k)))

seancorfield03:09:36

But it's wrong

seancorfield03:09:56

You're breaking the non-data/data divide there.

didibus03:09:03

Now you can do (datafy (nav (datafy a-person) :car nil))

seancorfield03:09:12

But that's useless.

didibus03:09:37

Why? I'm taking a Java object: Person, and making it so I can navigate it ?

seancorfield03:09:00

The important aspect here is: non-data -> datafy -> data -> nav -> non-data.

didibus03:09:23

Person --datafy--> data -> nav into car -> Car --datafy--> data

didibus03:09:40

nav returns a Car object

didibus03:09:29

Person -> datafy -> person as data -> nav -> Car -> datafy -> car as data

didibus03:09:13

Am I just really confused?

seancorfield03:09:19

So that works purely because your datafy isn't deep so it leaves native Java objects embedded in the data.

didibus03:09:39

I thought that was the point no? Lazy navigation ?

didibus03:09:51

So you only convert to data when you drill down?

seancorfield03:09:16

Your example is fine, but you're not understanding why.

didibus03:09:56

Ya, I think I need to sit on a it a bit.

didibus03:09:05

Before I really get it

seancorfield03:09:14

It does take a while.

seancorfield03:09:59

I happened to "grok" it quickly, but I've seen a lot of confusion about it since it was shown off at Clojure/conj late last year.

didibus03:09:22

Ya, when searching, I couldn't find any other article than your about it 😛

didibus03:09:36

Anyway, got to run.

didibus03:09:42

But thanks for helping out!

seancorfield03:09:54

I think it's very early days for datafy/`nav` so there's almost no one using it. I think more people need to use REBL to figure this out. That's where this stuff shines.

seancorfield03:09:00

I think REBL is a game-changing breakthrough.. but unless you agree, you probably won't "get" datafy/`nav`...

sogaiu04:09:20

@didibus the following are also from seancorfield and quite helpful: https://github.com/seancorfield/next-jdbc/blob/master/doc/datafy-nav-and-schema.md (specifically the Identifying Foreign Keys section -- i went through this part with a friend using next.jdbc to follow his description in detail [1]) and this video of his demos this foreign key "link" (and more): https://www.youtube.com/watch?v=ZhzMoEz4j1k#t=6m40 [1] https://github.com/sogaiu/datafy-nav-with-next-jdbc -- includes links to other resources 🙂

lread09:09:30

Thanks @UG1C3AD5Z (and @U04V70XH6!) a very helpful summary of REBL resources!

Aklscc05:09:07

Read the docs for spec, I see the :confirm-keys for map-of? But there is no example for it, what's the different between add it and not add? I have done:

;; not add. default `false`
(s/def ::scores (s/map-of int? int?))
(s/conform ::scores {"Sally" 1000, "Joe" 500})
;; => :clojure.spec.alpha/invalid

;; add it. 
(s/def ::scores (s/map-of int? int? :conform-keys true))
(s/conform ::scores {"Sally" 1000, "Joe" 500})
;; => :clojure.spec.alpha/invalid

Alex Miller (Clojure team)11:09:54

In both of these examples, the keys are invalid. The difference is whether the keys are replaced with the conformed values or not in the result. If the key spec is a predicate, you won’t see any difference here - you’d need a key spec that conformed differently like s/or or a regex op

roklenarcic10:09:27

Is there something I can do about these warnings:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000801184840 (file:/Users/roklenarcic/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(org.xml.sax.InputSource,org.xml.sax.HandlerBase)
WARNING: Please consider reporting this to the maintainers of clojure.lang.InjectedInvoker/0x0000000801184840

roklenarcic11:09:48

solved the warning by adding

:jvm-opts ["--add-exports" "java.xml/com.sun.org.apache.xerces.internal.jaxp=ALL-UNNAMED"]

Alex Miller (Clojure team)11:09:09

Would be good to know the source of the reflection - use —illegal-access=debug to learn more

roklenarcic12:09:16

The source is depstar library

roklenarcic12:09:31

WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000801184840 (file:/Users/roklenarcic/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(org.xml.sax.InputSource,org.xml.sax.helpers.DefaultHandler)
	at clojure.lang.Reflector.canAccess(Reflector.java:49)
	at clojure.lang.Reflector.toAccessibleSuperMethod(Reflector.java:84)
	at clojure.lang.Reflector.lambda$invokeInstanceMethod$0(Reflector.java:99)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:101)
	at clojure.xml$startparse_sax.invokeStatic(xml.clj:76)
	at clojure.xml$startparse_sax.invoke(xml.clj:75)
	at clojure.xml$parse.invokeStatic(xml.clj:92)
	at clojure.xml$parse.invoke(xml.clj:78)
	at clojure.xml$parse.invokeStatic(xml.clj:86)
	at clojure.xml$parse.invoke(xml.clj:78)
	at hf.depstar.uberjar$maybe_copy_pom.invokeStatic(uberjar.clj:244)
	at hf.depstar.uberjar$maybe_copy_pom.invoke(uberjar.clj:238)
	at hf.depstar.uberjar$run$fn__244.invoke(uberjar.clj:317)
	at hf.depstar.uberjar$run.invokeStatic(uberjar.clj:309)
	at hf.depstar.uberjar$run.invoke(uberjar.clj:296)
	at hf.depstar.uberjar$uber_main.invokeStatic(uberjar.clj:335)
	at hf.depstar.uberjar$uber_main.invoke(uberjar.clj:328)
	at hf.depstar.jar$_main.invokeStatic(jar.clj:6)
	at hf.depstar.jar$_main.doInvoke(jar.clj:4)

Alex Miller (Clojure team)12:09:32

Prob a good ticket for them then. There are some popular forks of depstar btw, not sure which you’re using

roklenarcic12:09:08

@U04V70XH6 seancorfield/depstar

Alex Miller (Clojure team)12:09:18

Looks like this is use of clojure.xml, which is not recommended these days, should be using org.clojure/data.xml

roklenarcic12:09:13

that looks quite unfinished, also it seems to pull a bunch of development dependencies into the project (piggiback, clojurescript, tools nrepl, test.chec, figwheel-sidecar, binaryage/devtools)

Alex Miller (Clojure team)12:09:37

Those are test deps so not pulled in. It’s more stable than the version implies.

Alex Miller (Clojure team)12:09:49

The pom.xml is the source of truth in that project, not project.clj btw

seancorfield19:09:20

depstar very deliberately uses no external dependencies (so that it can rely on the contents of the classpath when it just excludes itself). That's why I used clojure.xml. Switching to org.clojure/data.xml would complicate stuff quite a bit.

seancorfield19:09:42

Feel free to open an issue on GitHub for me to look at. If I can read the pom.xml file without external dependencies and avoid clojure.xml, that would be the ideal solution for me...

roklenarcic07:09:52

Maybe you could make the jar a java module that declares dependency on java.xml module... although the reflector is in clojure itself so I'm not sure if this would work

roklenarcic07:09:14

I haven't really read up on modules

seancorfield07:09:54

It needs to work on JDK8...

seancorfield02:09:33

seancorfield/depstar "0.3.3" is available -- no longer uses clojure.xml so you won't get "Illegal Reflective Access" warnings on JDK11 any more!

sogaiu10:09:18

i think i get similar messages when i'm using rebl -- would like to know too 🙂

alfredx10:09:52

Every time I see similar issues, my brain’s first reaction would be: Are we using the ‘wrong’ jdk?

Jivago Alves10:09:52

Hey folks, we are thinking about breaking down our glorious monolith into more apps and we were wondering what are the available options. We are open to have monorepos in order to share common code too. Or build a solution with multiple jars, etc. Any thoughts?

reefersleep10:09:56

Is there any idiomatic or example work of extending Transit so that you can send java.time types, like LocalDate, to Clojurescript?

dimovich16:09:22

@U0AQ3HP9U Here is an example read/write handler for Transit: https://github.com/dimovich/linked-transit that itself was based on this code https://github.com/tonsky/datascript-transit

reefersleep08:09:01

Thanks @U051951T6 🙂 To be more precise; I’m working on an application that already uses Java7 types in the backend, such as java.util.Date under #insts, and (I guess) cljs-time types and js native types in the frontend. I’m trying to figure out how to go about migrating to java.time in the backend; the way I see it, I would also need a completely new set of types in the frontend unless I also change all of my existing types in one big-bang move; and if I have both Java7 types and java.time types living side by side in my CLJ backend for a while, how else, other than discerning by the type in the CLJS frontend, is Transit going to know when I’m referring to a type that should be converted to either a Java7 or java.time type when sent to the CLJ backend?

reefersleep08:09:23

So, seeing as java.time is a new set of types, and transit/CLJS/CLJ traditionally and idiomatically rely on #inst which is a Java7 construct, I figured someone else might already have gone through these motions and implemented a set of CLJS types along with transit handlers to match java.time, or at least come up with a strategy for gradually moving an application with existing Java7 types and corresponding CLJS types to java.time.

reefersleep08:09:40

(a strategy that I could piggyback on 😄 )

alfredx23:09:40

Notice that lein compile :all or lein jar does not pick up a wrong-arity mistake in the code, e.g. (double? s "rubbish"). Meaning if those lines of code are not covered by tests, they are shipped and becomes part of you outcome jar and let the runtime catch any exception like that. Are there any tips in handling this situation apart from relying on eastwood?

seancorfield01:09:02

Adopting a solid RDD (REPL-Driven Development) workflow where you eval and test each small piece of code as you write it -- even if you don't create actual test files -- goes a long way to eliminating this sort of mistake.

seancorfield01:09:40

And then there are things like Joker which can check in real time as you're editing code (I use this with Atom all the time and catches nearly all arity errors, as well as unused local bindings and all sorts of other potential code problems).

alfredx01:09:44

This is a individual practice thing, not enforced

seancorfield01:09:14

Those are my tips for handling the situation you asked about 🙂

alfredx01:09:22

IDE prompt when seeing it, yes, cursive is good in that

alfredx01:09:41

But I still see this happening even from experienced devs

alfredx01:09:39

And the issue was caught in deployed env, perhaps prod

seancorfield01:09:18

RDD is a good workflow, regardless, because you get immediate feedback on all your code, and you get into the habit of making small changes and verifying each one. See Stu Halloway's talks (RDD to Chicago Clojure from June 2017, and Running With Scissors to Strange Loop 2018, as I recall) and also Eric Normand's excellent RDD online course (worth the monthly fee).

seancorfield01:09:02

Experienced devs with a bad workflow will have more buggy code than most devs with a really good workflow.

alfredx01:09:40

yes, most devs do repl-dd. In reality, when you are under pressure, urgent to fix a bug, in a rush, things can happen

seancorfield01:09:42

Using Language X's workflow with Clojure isn't enough. no matter how experienced you are.

seancorfield01:09:16

I would argue that most (Clojure) devs do not do RDD. I see all sorts of evidence of that on this Slack...

alfredx01:09:55

Agree, but I would like an automated tooling to catch and stop these kind of obvious mistakes, at least eastwood could help with this kind of stuff, I think

alfredx01:09:39

I would argue that most (Clojure) devs do *not* do RDD, really? :thinking_face:

seancorfield01:09:40

Most Clojure devs still type stuff into the REPL and tend to write whole functions and tests and then run them. None of that is RDD.

seancorfield01:09:41

(and don't get me started on how bad I consider the "watcher"-based workflows! 🙂 )

ag23:09:17

is it possible to slurp a file sitting in a jar in classpath, e.g.: I want to slurp externs.js for cljsjs package. Can someone show me how?

alfredx00:09:40

You can do it like (slurp (io/resource "file_on_classpath"))

ag00:09:05

No, I mean real example. With a file sitting in a jar in ~/.m2. Can you show me an example with a real cljsjs package? How does the path to it looks like?

alfredx00:09:19

If you are talking about JVM classpath, they follow JVM convention, e.g. "com/example/package/file1.json"

alfredx00:09:32

depending on how you package your jar file

alfredx00:09:32

perhaps it should start with "/com", easy to test it out

ag00:09:56

can you throw an example with any clojure library you have in ~/.m2

alfredx00:09:59

and io here is

alfredx00:09:52

Not sure which part you are asking for

alfredx00:09:05

classpath is a runtime thing, not ~/.m2

alfredx00:09:30

What were you referring to when we say classpath

ag00:09:33

ah… I guess that’s my ignorance then. Sorry for the confusion.

ag00:09:00

so then in changes my question. Is it possible to slurp a file from a jar in ~/.m2?

alfredx00:09:04

Why not, then this is just like (slurp "~/.m2/repository/com/example/abc.jar")

alfredx00:09:26

Even though you might better off replace ~ with home dir from System/getProperties

ag00:09:41

aha, but now, I need a specific file in a jar. Do I have to extract it?

alfredx00:09:59

so this is an just file reading, but binary file for slurp, not sure about that

alfredx00:09:32

slurp does not have knowledge of jar

ag00:09:22

So that’s my problem. I need to read an extern file for cljsjs package from Clojure code. I thought, since Clojure knows how to import things from ~/.m2, maybe there is a way to read those files

alfredx00:09:38

jar files are zip files, you can either use JarFile things to read them, or use io.zip.blah to read them

alfredx00:09:07

what do you mean by Clojure nows how to import things from ~/.m2?

ag00:09:29

sorry *knows

ag00:09:18

when I say (require ’some-lib) it knows where to get it. How can I slurp the source of `some-lib?

alfredx00:09:52

The precondition for (require) to work is that the some-lib jar is already on runtime classpath. (though classloader magic can make it dynamic of course).

alfredx00:09:28

source of a lib, it probably depends, if the source is packaged into the jar, you can slurp it via io file.

ag00:09:29

so… it is in the classpath, right? How do I slurp a file that’s in the classpath? It goes back to your first answer. But how do I do it for real? Can you show me using any library that you have listed in project.clj

alfredx00:09:41

if not packaged, you cann’t get it.

alfredx00:09:01

the libraries in your proejct.clj does not exist as jar file on classpath, in stead, they exist like they have been extracted or unzipped

ag00:09:36

but where do they get extracted?

alfredx00:09:42

I feel we don’t have common understanding of classpath

ag00:09:06

Nope, I don’t… I just want to slurp a file

alfredx00:09:12

JVM extracts them, probably in memory

alfredx00:09:13

Sorry, I am lost

ag00:09:17

I guess I’m gonna have to read more about JVM classpath and resources

jjttjj01:09:10

You might know this already, and not sure if this helps, but a jar is essentially a zip archive of the files it contains

jjttjj01:09:24

So do you need to read the file in the jar without unzipping it? Or can you unzip?

jjttjj01:09:03

On my phone but these might help

dominicm04:09:39

There's actually a url registered for inside of jars

dominicm04:09:12

So you can construct a url to your jar & file path and slurp that