Fork me on GitHub
#unrepl
<
2017-12-22
>
dominicm09:12:00

@pesterhazy how much time should unravel spend loading compliment?

dominicm09:12:14

I'm seeing either > 1 minute, or never (I've given up)

cgrand09:12:55

@dominicm in a fully local setup?

dominicm09:12:11

@cgrand What does "fully local" mean in this context?

cgrand09:12:24

server & client on the same machine

dominicm09:12:22

I'm now at 3 minutes.

dominicm09:12:27

[:receive {:origin :aux} [:started-eval {:actions {:interrupt (unrepl.replG__156/interrupt! :session654 6), :background (unrepl.replG__156/background! :session654 6)}} 6]]
[:unknown-command [:started-eval {:actions {:interrupt (unrepl.replG__156/interrupt! :session654 6), :background (unrepl.replG__156/background! :session654 6)}} 6]]
[:receive {:origin :aux} [:exception {:ex #error {:cause "compliment.core", :via [{:type clojure.lang.Compiler$CompilerException, :message "java.lang.ClassNotFoundException: compliment.core, compiling:(unrepl-session:6:63)", :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7010]} {:type java.lang.ClassNotFoundException, :message "compliment.core", :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}], :trace [[java.net.URLClassLoader findClass "URLClassLoader.java" 381] [clojure.lang.DynamicClassLoader findClass "DynamicClassLoader.java" 69] [java.lang.ClassLoader loadClass "ClassLoader.java" 424] [clojure.lang.DynamicClassLoader loadClass "DynamicClassLoader.java" 77] [java.lang.ClassLoader loadClass "ClassLoader.java" 357] [java.lang.Class forName0 "Class.java" -2] [java.lang.Class forName "Class.java" 348] [clojure.lang.RT classForName "RT.java" 2204] [clojure.lang.RT classForNameNonLoading "RT.java" 2217] [clojure.lang.Compiler$HostExpr maybeClass "Compiler.java" 1041] #__1]}, :phase :eval} 6]]
[:unknown-command [:exception {:ex #error {:cause "compliment.core", :via [{:type clojure.lang.Compiler$CompilerException, :message "java.lang.ClassNotFoundException: compliment.core, compiling:(unrepl-session:6:63)", :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7010]} {:type java.lang.ClassNotFoundException, :message "compliment.core", :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}], :trace [[java.net.URLClassLoader findClass "URLClassLoader.java" 381] [clojure.lang.DynamicClassLoader findClass "DynamicClassLoader.java" 69] [java.lang.ClassLoader loadClass "ClassLoader.java" 424] [clojure.lang.DynamicClassLoader loadClass "DynamicClassLoader.java" 77] [java.lang.ClassLoader loadClass "ClassLoader.java" 357] [java.lang.Class forName0 "Class.java" -2] [java.lang.Class forName "Class.java" 348] [clojure.lang.RT classForName "RT.java" 2204] [clojure.lang.RT classForNameNonLoading "RT.java" 2217] [clojure.lang.Compiler$HostExpr maybeClass "Compiler.java" 1041] #__2]}, :phase :eval} 6]]
[:receive {:origin :aux} [:prompt {:file "unrepl-session", :line 7, :column 1, :offset 1082, clojure.core/*warn-on-reflection* false, clojure.core/*ns* #unrepl/ns user}]]
[:unknown-command [:prompt {:file "unrepl-session", :line 7, :column 1, :offset 1082, clojure.core/*warn-on-reflection* false, clojure.core/*ns* #unrepl/ns user}]]
Looks like I need to depend on compliment myself, I expected unravel to be hotloading it.

dominicm10:12:24

Oh, actually, it might be more malignant than only that. I've added compliment as a dependency, and unless I do (require 'compliment.core) first, the completion hangs.

cgrand10:12:40

Hi tried it and I think that unravel doesn’t require compliment.core and hasn’t it on the sideloader path

dominicm10:12:45

@cgrand A bit of an aside, but how do you feel about vendoring depndencies in this context? Should unravel be using unravel.compliment.core, so that compliment 0.3.4 can be used by unravel, and 1.0.0 can be used by clientC?

dominicm10:12:35

Or, perhaps more importantly, so that oldProjectA can use 0.0.1, and unravel can use 0.3.4.

cgrand10:12:31

Yes because this last scenario breaks even in a richhickeyic world.

dominicm10:12:45

The other option, and I'm not sure if you've thought about this, is to use clj-embed: https://github.com/RutledgePaulV/clj-embed which is an independent form of boot's pods. For something like compliment this might not make sense as it's inspecting the classpath & running state. But for other tooling needs (perhaps bundling zprint) it would be a good idea perhaps. I mention this library because I wonder if this is a problem unrepl may want to solve in this way.

dominicm10:12:27

Maybe it should actually be further vendored than just unravel.compliment.core, but instead unravelv0_2_2.compliment.core to allow parallel clients of unravelv0_2_1 also.

dominicm10:12:05

I also question whether it should be scoped to the client's connection altogether, with the vendoring provided by unrepl.

cgrand10:12:03

@dominicm I thought about classloader isolation and even did a quick prototype in the early days of unrepl

cgrand10:12:44

Like you mention, classloader isolation is strong, too strong.

cgrand10:12:55

The “best” option if you set up a second clojure, is to have it communicate in text-based form with the user clojure.

cgrand10:12:40

Basically you may treat it as a low-latency unrepl client.

cgrand10:12:31

Not sure it’s a good idea.

cgrand10:12:36

Classloader isolation has really the problem of being too strong for our purpose — and it doesn’t work in cljs (starting a second JS env in Node?)

dominicm11:12:54

So, vendoring it is. We shall all become professional users of Mr. Anderson!

cgrand11:12:44

With @richiardiandrea we discussed a subtler and better vendoring

dominicm11:12:57

What was the approach considered?

cgrand11:12:25

vendor compliment.core as vnd.sha1_XXXXX.compliment.core

dominicm11:12:20

ah, so it works globally for namespaces which are the same, makes sense.

cgrand11:12:44

;; assuming that (do-vendor vendorable vendored) substitutes all refs to vendored nses in the source of vendorable
;; then computes its hash, then puts the hash in ns in the source and returns the vendor-hashed ns name.

(let [deps-graph XXX]
  (loop [vendored {}]
    (if-some [vendorables (seq (filter (fn [dep] (every? #(or (white-list? %) (vendored %))  (:deps dep)))))]
      (recur (into vendored
               (map (fn [vendorable]
                      [(:name vendorable) (do-vendor vendorable vendored)]))
               vendorables))
      vendored)))

dominicm11:12:35

I wonder if JarJar can do something similar

cgrand11:12:18

can do or could do?

dominicm11:12:42

I suppose either.

cgrand11:12:23

Someone willing to implement that? 😉

cgrand11:12:36

It would make a great xmas present to the community...

dominicm11:12:08

A tool to vendor dependencies you mean?

bozhidar11:12:12

All of this seems to be pretty much what Mr. Anderson does.

cgrand11:12:15

yup, maybe a patch to mranderson

dominicm11:12:35

@bozhidar the hashing rather than a specified prefix is the distinction I think.

bozhidar11:12:37

Vendoring deps was one of the most painful things when building CIDER - something some basic should be easier to do.

bozhidar11:12:51

Yeah, I get the point, but fundamentally it’s the same process.

bozhidar11:12:06

Patching Mr. Anderson to support this would be trivial.

cgrand12:12:40

Just looked at it and it’s not that trivial because of deps ordering, which matters for hashing

cgrand12:12:40

if libA depends on libB, when hash-vendoring, you can’t independently hash them

dominicm12:12:42

@cgrand I was about to ask about that for hashing.

dominicm12:12:17

I guess a version-tied prefix is the simplest way.

cgrand12:12:33

if libC uses libA v1 and libB v1.1 if libD uses libA v1 and libB v1.2 then we should get two different libAs

New To Clojure15:12:14

$ unravel --version
Unravel 0.2.2 (Lumo 1.7.0)

$ lein --version
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM

JVM_OPTS='-Dclojure.server.myrepl={:port,50505,:accept,clojure.core.server/repl}' lein repl

unravel localhost 50505
unravel fails with stacktrace:
$ unravel localhost 50505evalmachine.<anonymous>:22var reader = cljs.reader.push_back_reader.call(null,s); ^TypeError: Cannot read property 'call' of undefined at unravel$lisp$safe_read_string (evalmachine.<anonymous>:22:42) at Transform._transform (evalmachine.<anonymous>:56:60) at Transform._read (_stream_transform.js:186:10) at Transform._write (_stream_transform.js:174:12) at doWrite (_stream_writable.js:385:12) at writeOrBuffer (_stream_writable.js:371:5) at Transform.Writable.write (_stream_writable.js:288:11) at Transform.ondata (_stream_readable.js:642:20) at emitOne (events.js:115:13) at Transform.emit (events.js:210:7)
Any ideas what could be wrong?

New To Clojure15:12:30

$ lumo --version
1.7.0

cgrand15:12:40

@ghsgd2 built from Git or brew?

New To Clojure15:12:49

@cgrand brew install bfontaine/utils/unravel

pesterhazy16:12:29

@ghsgd2 you'll need to use latest master unfortunately because the version on brew is not compatible with lumo 1.7.0+

pesterhazy16:12:00

@dominicm, did you solve the compliment issue?

pesterhazy16:12:34

compliment is only used when you pass --flag compliment

dominicm16:12:51

@pesterhazy There's a bug or 2 there, I found a workaround. Bug 1) compliment.core is used without requiring it first (makes compliment pretty much unusable without knowing to require it first, causes total lockup) Bug 2) unravel doesn't bundle a copy of compliment / it doesn't vendor compliment in order to prevent conflicts with the project.

pesterhazy16:12:52

and that requires compliment to be on the class path

pesterhazy16:12:13

yup both are true (oops!)

pesterhazy16:12:35

could you file an issue?

dominicm16:12:44

Doing for both now, there's a possible 3rd "nice to have" here, handling exceptions that happen during tab completion. (and other aux exceptions)

pesterhazy16:12:54

yeah it's a good point. I wonder how we should catch those exceptions

dominicm16:12:41

@pesterhazy Same way you do for other exceptions? 😄 Displaying them is a little more troubling. Did the code for split screen ever go in? That would be a great place to dump it OR display a message saying "An unexpected exception has occurred, a log has been written to /tmp/unravel_xxx.log" which shows perhaps in the same place documentation would normally show?

pesterhazy16:12:24

the split screen only works for developers (using scripts/debug)

pesterhazy16:12:37

good idea about the log file

pesterhazy16:12:05

the key is to continue with regular operation after the exception occurs, rather than blocking forever

dominicm16:12:46

Yeah, that's super important. Medium level of importance is allowing you to fix bugs which bother people.

dominicm16:12:01

(On this scale anyway)

volrath16:12:29

I'm actually working on this related problem in unrepl.el right now: figuring out the best way to present aux conn exceptions to users... I'm using a popup temp buffer, but would love to hear if you come up with a different UI

cgrand18:12:19

An error log?

New To Clojure16:12:17

@pesterhazy Got it. However, official installation docs say that brew could be used to install unrepl on macOS. And lumo was installed by brew unrepl formulae as a dependency. Looks like brew formulae is broken and needs fixing.

pesterhazy16:12:27

we want to release a new version that fixes that, and brings all the new feature in master as well

pesterhazy16:12:34

haven't found the time lately

New To Clojure16:12:02

@pesterhazy No worries, created the issue as a reminder.

gcast17:12:24

Question, If I wanted to communicate from the browser to a plain socket repl running the unrepl blob, would web sockets be the most appropriate strategy?

gcast17:12:01

I imagine standing up some sort of web socket server to communicate with the client

dominicm19:12:03

With parfix, is there a way to send the current line without hitting "End"? My brain went to Ctrl-Enter

cgrand19:12:03

@gcast why not? The sad part is having to have an intermediate server.

gcast19:12:02

Ok good. I've started down that road. I spent some time trying to see if it was possible to have a direct stream to unrepl, but my research and current understanding suggests browser security may make other approaches problematic

cgrand19:12:36

@dominicm I wanted it to but apparently you can’t get CTRL ENTER in a term app. I don’t remember what got merged. Did you try ^J?

cgrand19:12:53

I’m unhappy with the current behavior and @pesterhazy and I have differing opinion of what a good behavior should be.

cgrand19:12:43

@gcast you want a real browser not a web view in a native app?

gcast19:12:49

well I suppose if the underlying repl can live on a remote server than a native app is just fine as well

cgrand20:12:00

The first protoclient for unrepl was electron-based

gcast21:12:53

really? I'd never heard of electron before but just did some research and it looks pretty nice

gcast21:12:30

how mature is the CLJS environment for electron?

cgrand21:12:18

I don’t remember what I used. I could look it up.

cgrand21:12:17

It worked ok. Too much reloading. I wasn’t familiar with node enough. Otherwise I would have made it expose a socket repl (well two)

gcast21:12:05

I see. this looks quite promising because there is already an electron-based notebook-app for arbitrary jupyter-kernels https://github.com/nteract/nteract

gcast21:12:28

Would simplify the problem to just writing a native clojure backend that support unrepl.

cgrand19:12:01

I believe that sending a http request to a repl would almost work… if you found a way to not have the browser barf on the garbage.

gcast19:12:04

the key is that the compute can happen on an arbitrary networked machine

gcast19:12:05

hmmm, garbage-handling aside, would http-requests be slower for repl communication?

dominicm19:12:06

@cgrand At least in my terminal, Ctrl-J didn't work

pesterhazy19:12:35

It’s not merged yet- you can try the branch though

dominicm19:12:06

Ah, I thought I was cutting edge using master as it is 😄