Fork me on GitHub
#beginners
<
2019-02-14
>
piotr.kurnik08:02:31

Hi, anyone can offer any thoughts on the following problem? I want to execute arbitrary function with a timeout. I want result only if it's before the timeout value, after timeout I want nil.

jumar09:02:12

You could use future and deref with timeout; e.g.

(deref (future (Thread/sleep 10) 999)
                     9
                     nil)
nil

(deref (future (Thread/sleep 10) 999)
                     11
                     nil)
999

jumar09:02:19

You can also use future-cancel to try to cancel the future (might not be possible depending on what it's doing)

piotr.kurnik10:02:23

@ Thanks, will give it a try

alisher.atantayev11:02:32

I've never needed to have a caching in my Clojure program before, so I need some guidance. I have a collection of entities (maps), that are calculated via some expensive functions using data-fetching from the database. Basically, I want to have it in cache as a big map with entity-id as keys and corresponding hash-map as values. I looked up idiomatic ways to handle caching in Clojure, it's suggested to use clojure.core.cache and atoms. It seems to me that none of eviction policies are suitable for my case, so I resorted to use BasicCache. Additional to my "main" cache, implemented as an atom which contains plain entries, I want to have a nested lookup map with same entities grouped in keys for fast access. I want to store this derived lookup map in some atom for future uses, rather than calculating it every time (it's ~ over a million values). I have two questions: 1) Should I use clojure.core.cache's BasicCache? It seems to me that it's just a wrapper over regular map, what benefits do I get using it? 2) Is it ok or idiomatic to use add-watch as a way to keep in sync two atoms? So that my derived-lookup-map in second atom is always relevant and in sync whenever plain data in first atom changes?

cybersapiens9713:02:45

do i have to set site-defaults map key multipart params to true on compojure in order to upload a file? Edit: it seems the answer is no.

hiredman17:02:24

keeping two atoms in sync (sort of depending what you mean by in sync) is bad, just use one atom

shriyansbhatnagar18:02:53

Hi πŸ‘‹ i have just started learning clojure and have a couple of beginner doubts. say i have a vector [1 2 3 4 5] i want to create another vector by combining successive pair of the pervious one [[1 2] [2 3] [3 4] [4 5]] 1. does clojure have or commonly use tuples ? 2. i used partition and it returns a lazy sequence,

(partition 2 1 [1 2 3 4 5])
((1 2) (2 3) (3 4) (4 5)) is there a way to get a vector from it ? 3. since i don't know how to write idiomatic clojure code, my code has a lot of functions that take vector do some computation and returns a list and then some that take a list do something and return a vector and go back and forth which feels awkward (and wonder surely not good for perf ) when to use list vs vector ? is it fine to use mapv a lot ?

dpsutton18:02:52

1) re tuples: data structures are immutable, so vectors are essentially tuples. 2) you can easily achieve this like so (map vector coll (rest coll))

shriyansbhatnagar18:02:32

ahh that makes sense Thank you πŸ™‚

dpsutton18:02:04

or mapv (map vector)

borkdude18:02:37

A lot of Clojure functions returns sequences. As long as you don’t care about it being a vector, list, etc. you should just work with that. When you do need a vector, you can use into, vec or vector to coerce the sequence. An alternative to the sequence abstraction is to use transducers which will not create intermediate sequences.

shriyansbhatnagar18:02:41

okay if i understand correctly i can use sequences for all computation till i need something specific from vector like random access etc, only then i convert to vector

dpsutton18:02:05

> when to ues list vs vector it depends on what you need from your datastructure. if you just need sequential data, then list is fine. if you need random access by index, obviously vector is the way to go. also, think about when these constraints are true. Perhaps you only need access by index after a bunch of processing. then the last step would be to make vectors appropriately, rather than maintaining the vector type while you add, splice, filter, etc your data

borkdude18:02:33

E.g. (filterv odd? (map inc [1 2 3])) or (into [] (comp (map inc) (filter odd?)) [1 2 3]) both return the same thing

shriyansbhatnagar18:02:03

Thanks a lot @dpsutton and @borkdude

rschmukler20:02:05

Hey all. I've run into a bit of an interesting problem. Some time within the last day or two of dev my repl has become extremely slow to start. ie. on the order of minutes or so. I've tried rolling back commits / removing libraries, but so far I can't seem to narrow it down. I do see that one of my cores is pegged at 100%. In the interim I do get some logging from SL4J, but nothing meaningful. Does anyone have ideas about how I would go about debugging this?

dpsutton20:02:35

specific possibly oddball question: have you recently upgraded your java 8 version?

rschmukler20:02:01

Actually, just saw the thread in tools-deps channel

rschmukler20:02:11

Looks promising πŸ™‚

dpsutton20:02:29

yeah that's what i was thinking of.

rschmukler20:02:51

I'm using openjdk 11.0.2 though, so perhaps not?

dpsutton20:02:10

it's the only thread i know to pull on this problem. good luck Β―\(ツ)/Β―

rschmukler20:02:28

Appreciate the idea

alexmiller20:02:54

what kind of repl is it? do some ctrl-\ to see if the stack traces show you anything interesting

rschmukler20:02:36

Just a plain ol' clojure repl? Booting up again to see if ctrl-\ yields anything. I'm using the java-time library mentioned in #tools-deps too

dpsutton20:02:07

@alexmiller i've seen you mention ctrl-\ before. what is this? some kind of interrupt? who "watches" for it? clojure or jvm?

alexmiller20:02:54

yeah, same as a kill -3

alexmiller20:02:59

or you can use jstack

alexmiller20:02:30

or ctrl-break on windows

rschmukler20:02:50

I presume this is standard looking for the main thread? It's surprising how long it runs for.

"main" #1 prio=5 os_prio=0 cpu=106657.69ms elapsed=217.69s tid=0x00007efd6c017000 nid=0x2d50 runnable  [0x00007efd73323000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes([email protected]/Native Method)
	at java.io.FileInputStream.read([email protected]/FileInputStream.java:279)
	at java.io.BufferedInputStream.read1([email protected]/BufferedInputStream.java:290)
	at java.io.BufferedInputStream.read([email protected]/BufferedInputStream.java:351)
	- locked <0x000000060aa47128> (a java.io.BufferedInputStream)
	at sun.nio.cs.StreamDecoder.readBytes([email protected]/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead([email protected]/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read([email protected]/StreamDecoder.java:178)
	- locked <0x000000060acf15d0> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read([email protected]/InputStreamReader.java:185)
	at java.io.BufferedReader.fill([email protected]/BufferedReader.java:161)
	at java.io.BufferedReader.read([email protected]/BufferedReader.java:182)
	- locked <0x000000060acf15d0> (a java.io.InputStreamReader)
	at java.io.LineNumberReader.read([email protected]/LineNumberReader.java:126)
	- locked <0x000000060acf15d0> (a java.io.InputStreamReader)
	at java.io.FilterReader.read([email protected]/FilterReader.java:65)
	at java.io.PushbackReader.read([email protected]/PushbackReader.java:90)
	- locked <0x000000060acf1590> (a java.io.LineNumberReader)
	at clojure.lang.LineNumberingPushbackReader.read(LineNumberingPushbackReader.java:66)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at clojure.main$skip_whitespace.invokeStatic(main.clj:130)
	at clojure.main$repl_read.invokeStatic(main.clj:164)
	at clojure.main$repl_read.invoke(main.clj:152)
	at clojure.main$repl$read_eval_print__9068$fn__9069.invoke(main.clj:410)
	at clojure.main$repl$read_eval_print__9068.invoke(main.clj:409)
	at clojure.main$repl$fn__9077.invoke(main.clj:435)
	at clojure.main$repl.invokeStatic(main.clj:435)
	at clojure.main$repl_opt.invokeStatic(main.clj:499)
	at clojure.main$main.invokeStatic(main.clj:598)
	at clojure.main$main.doInvoke(main.clj:561)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:37)
Beyond that, @alexmiller if it matters, I'm using a pinned branch of tools.deps (w/ add-lib) - I don't know much about how java / clojure builds yet (I've managed to avoid the JVM somehow during my career)

alexmiller21:02:46

by "plain ol' clojure repl" are you using clj, or invoking clojure.main some other way?

rschmukler21:02:56

clojure from the command line

rschmukler21:02:23

(`cider-jack-in` is equally slow)

alexmiller21:02:34

here you're actually in the repl, reading stdin, so this is not startup time

rschmukler21:02:14

Yeah, it's really all before that (ie. the 217s elapsed) it seems

alexmiller21:02:30

you need to do the stack traces earlier I think

rschmukler21:02:55

I didn't realize I could do it before the repl had loaded

rschmukler21:02:51

It looks like it's just actually compiling code?

alexmiller21:02:52

looks like core.async go blocks

alexmiller21:02:18

well the main thing likely to have changed recently is your code :)

rschmukler21:02:58

This project is ~800 lines, so this is a bit surprising... I'll try and revert further back and see if I'm doing something silly.

alexmiller21:02:22

yeah, that seems weird

alexmiller21:02:50

I will note that the reason you see this on startup is because of something in your user.clj

alexmiller21:02:33

if you don't have stuff in user.clj, it shouldn't have to compile anything to get to a repl

rschmukler21:02:22

Well, I'm requiring the project's main for a "reloaded" workflow (via integrant)

rschmukler21:02:44

but it's still surprising as this hasn't been an issue earlier in the project's life-cycle

alexmiller21:02:11

well, might want to think about what you have changed that affects what's loaded on startup

rschmukler21:02:28

Yeah, I'm going to revert to a week ago

alexmiller21:02:18

if you have some big chunks of code in a core.async go macro, that's a giant macro (that is basically a mini analyzer)

rschmukler21:02:45

It's a 1 liner that calls a function, I believe

alexmiller21:02:54

well, that doesn't seem bad

rschmukler21:02:03

It looks like it's pathom

rschmukler21:02:23

but it's surprising to me, because that wasn't an issue earlier - and libraries get cached don't they?

alexmiller21:02:43

they do, but this is compiling your code, and that's not cached

alexmiller21:02:03

(although we have some science experiments around this in the lab :)

rschmukler21:02:57

(ns cryptonic.graph
  "Pathom based query graph for Cryptonic"
  (:require [com.wsscode.pathom.core :as p]
            [com.wsscode.pathom.connect :as pc]
            [duct.logger :as logger]
            [integrant.core :as ig]
            [utopia.integrant.db :as db]
            [cryptonic.store.candle :as store.candle]
            [clojure.spec.alpha :as s]
            [clojure.core.async :refer [<!!]]))

(pc/defresolver hello-world [env input]
  {::pc/input #{}
   ::pc/output [:hello-world]}
  (let [user-name (-> env :ast :params :user-name)]
    {:hello-world (str "Hello " user-name)}))

(def ^:private registry [hello-world])

;; -- Component Boilerplate

(s/def ::graph (s/keys :req-un [::logger/log
                                ::store.candle/candle-store
                                ::db/db]))

(defmethod ig/init-key :cryptonic/graph [_ {:keys [log]}]
  (logger/log log :info ::starting-component)
  (p/parallel-parser
    {::p/env     {::p/reader               [p/map-reader
                                            pc/parallel-reader
                                            pc/open-ident-reader
                                            p/env-placeholder-reader]
                  ::p/placeholder-prefixes #{">"}
                  :log log}
     ::p/mutate  pc/mutate-async
     ::p/plugins [(pc/connect-plugin {::pc/register registry})
                  p/error-handler-plugin
                  p/request-cache-plugin
                  p/trace-plugin]}))

(defmethod ig/pre-init-spec :cryptonic/graph [_]
  ::graph)


(defn query
  [this ctx query]
  (<!! (this ctx query)))


(comment
  (query (user/find-running :cryptonic/graph)
         {}
         '[(:hello-world {:user-name "Ryan"})]))

rschmukler21:02:10

That's the code that causes it to go into minutes of compiling

rschmukler21:02:41

But it definitely didn't used to

rschmukler21:02:40

Confirmed that commenting out the pathom related bits speeds it back up.

rschmukler21:02:52

I'm going to see if requireing pathom takes a long time

dpsutton21:02:10

was pathom a recent addition?

rschmukler21:02:47

No, weeks ago

rschmukler21:02:53

and it worked fine, weeks ago (same version too)

rschmukler21:02:07

Requiring it in repl takes about 7s

rschmukler21:02:13

which isn't fast, but it's not minutes either

rschmukler21:02:15

So, quite interesting....

rschmukler21:02:30

the deps chain is as follows

rschmukler21:02:46

user.clj -> cryptonic.main, has a ig/load-namespaces which attempts to load classes based off of keywords in a config.edn, ultimately resolves to cryptonic.graph which, if it just requires pathom, takes minutes to compile (even if no functions are used from pathom at all)

rschmukler21:02:42

I'm wondering if ig/load-namespaces is somehow causing pathom to be thought of as "my code" instead of library code. That'd also explain why we found the compiler toiling around in async/go macros (despite that I only have one, and it's quite simple)

rschmukler21:02:18

None of this explains why I didn't see it earlier, but at least it's something

hiredman21:02:59

that doesn't make sense

hiredman21:02:12

the compiler doesn't have a concept of "my code"

hiredman21:02:19

(whatever that is)

rschmukler21:02:36

I'm grasping at straws, undoubtedly

hiredman21:02:23

it isn't grasping at straws, it is making up a feature of the compiler that it doesn't have to explain why what you are seeing doesn't match your mental model of what you should be seeing

rschmukler21:02:58

Do you have ideas that fit your mental model given what I'm seeing?

hiredman21:02:22

I haven't entirely been paying attention

hiredman21:02:43

but when you load the clojure code it all passes through the compiler, without distinction

hiredman21:02:05

and before it goes through the compiler it will get macro expanded

alexmiller21:02:18

lot of spec stuff in there too

hiredman21:02:37

so there is nothing weird about seeing the compiler spending time compiling and macro expanding library code

alexmiller21:02:39

in pathom that is, which seems to be pretty macro heavy

rschmukler21:02:57

but if I require pathom via a loaded repl, it takes ~7s

rschmukler21:02:07

Not multiple minutes

alexmiller21:02:29

I suspect it's not pathom itself, it's pathom applied to your code

rschmukler21:02:41

If I comment out all pathom calls, and just require the namespaces, it still takes minutes to launch a repl - I'm not sure how it's being "applied to my code" in that case?

rschmukler21:02:23

(Not trying to be disagreeable or anything here, I just don't understand why requiring the library would cause it - definitely a short coming in my mental model)

alexmiller21:02:59

well, then guess not. I don't have all your context.

hiredman21:02:17

so you have a difference there, in behavior at the repl and in a source file

alexmiller21:02:19

I don't anything about cryptonic but you're declaring defmethods, which change the state of the runtime by installing new polymorphic behaviors

hiredman21:02:47

so the question is, why in the repl does loading a phantom namespace take X time and when requiring it in a source file it takes Y time

alexmiller21:02:49

sorry, not cryptonic, I guess that's integrant

hiredman21:02:05

so in those two contexts something must be different

hiredman21:02:37

how are you launching the repl when requiring pathom when it takes a short amount of time to load vs. a long amount of time?

rschmukler21:02:49

In both cases, I am using clojure

hiredman21:02:16

with the same deps.edn and aliases?

rschmukler21:02:16

in one case I have removed pathom from the requires, and then call (require)

alexmiller21:02:24

as a general idea I would either try reducing the delta between the cases that are "fast" and "slow", or if you have commits, bisect to narrow down a specific change where the perf changed

rschmukler21:02:54

I've got it down such that removing those two lines (the require of the pathom libs) will change it from seconds to minutes

hiredman21:02:25

have you done a verbose require to compare between the two?

rschmukler21:02:37

I'm not familiar with what a verbose require is

hiredman21:02:14

(require 'pathom.core :verbose) if I recall

dpsutton21:02:46

(also, make sure you are starting clojure from the command line, no CIDER in the mix)

rschmukler21:02:54

Yep, no cider

hiredman21:02:12

it will dump a ton of output since pathom loads a ton of stuff

alexmiller21:02:36

or [com.wsscode.pathom.core :as p :verbose true] will work I think

rschmukler21:02:19

@hiredman is the idea to diff the output of the two?

hiredman21:02:40

sure, literally with diff or just by eye

hiredman21:02:19

what the heck is cryptonic?

rschmukler21:02:31

A ML powered crypto bot I'm working on

hiredman21:02:55

and it has something named load-namespaces in it?

alexmiller21:02:06

I assume it's self aware

rschmukler21:02:33

integrant does - which attempts to load namespaces via keys in a map

rschmukler21:02:10

(but, to make sure it wasn't that, I got rid of that call and required the module directly)

hiredman21:02:02

you may also want to launch a slow and a fast repl and check to see if anything in (System/Properties) varies between them

rschmukler22:02:08

So this is promising

rschmukler22:02:27

364 lines of requires when loaded via the file

rschmukler22:02:37

103 lines of requires when loaded via the repl

rschmukler22:02:41

Looks like clojure.tools.analyzer is the majority of it

rschmukler22:02:47

(Which isn't loaded via the repl)

hiredman22:02:06

have you already loaded your project in the repl?

rschmukler22:02:57

Just without pathom

hiredman22:02:13

try loading pathom without loading your project

rschmukler22:02:25

ie. via user.clj?

hiredman22:02:41

I would get rid of user.clj entirely

hiredman22:02:04

too much variation, not enough control

rschmukler22:02:20

how do I "require" pathom from outside the file then?

hiredman22:02:39

what do you mean?

hiredman22:02:47

open a repl and type in (require ...)

rschmukler22:02:58

okay, confirming that that's the experiment

rschmukler22:02:01

Okay, running now

hiredman22:02:01

it looks like the main difference in the verbose loading is one is loading core.async(which loads tools.analyzer) and the other is not, but my guess would be the other is not loading core.async because it was already loaded when loading your code

rschmukler22:02:37

@hiredman right you are, no user.clj is 364 lines, just the same - but fast (ie. not noticeably slower to a human, ~5-7s)

hiredman22:02:42

so in a fresh repl every time, load a namespace from your project(do them all), then load pathom and see how long it takes to load pathom after you've loaded each of your namespaces

hiredman22:02:14

either do them all, or figure out the deps graph between them and do a topo sort

hiredman22:02:48

actually, strike that

hiredman22:02:54

it must be your user.clj

hiredman22:02:22

so bring it back, and go backwards through it comment outlines until it goes from slow to fast

rschmukler22:02:35

@hiredman haven't I achieved that by seeing the project load w/o pathom, (ie. the leaf of the dep tree), and then requiring pathom at the end?

hiredman22:02:21

because you loaded the whole tree and then loaded pathom

hiredman22:02:04

what I was wondering is, is there some minimal subtree that when loaded loading pathom goes from fast to slow

rschmukler22:02:24

What if I tried just requiring pathom from my user.clj?

hiredman22:02:25

but we haven't entirely ruled out some other malarkey in your user.clj

hiredman22:02:01

the point is we want to pull things a part so we can methodically check them in isolation

rschmukler22:02:53

(ns user
  (:require [com.wsscode.pathom.core :as p]
                 [com.wsscode.pathom.connect :as pc]))
Would reproduce it though

rschmukler22:02:02

ie. how do we get more minimal than that?

rschmukler22:02:15

(That does, indeed take minutes to compile / launch a repl on my machine)

hiredman22:02:12

I was under the impression that loading pathom by itself in the repl went fast?

rschmukler22:02:19

That's correct

rschmukler22:02:26

ie. if I have no user.clj

rschmukler22:02:37

and then instead execute the requires, it is ~5 seconds

hiredman22:02:41

what if you have no user.clj and past that ns form in to the repl?

rschmukler22:02:48

Good question

hiredman22:02:27

when you say loading pathom is fast, what are you actually running the repl? requiring those two namespaces?

rschmukler22:02:44

Yes, requiring the two namespaces

rschmukler22:02:50

(or evaluating the ns form)

hiredman22:02:38

do you have another file named user.clj anywhere on the classpath? (in the jars of any of your deps, etc)

rschmukler22:02:06

Is there an easy way to check that?

hiredman22:02:56

maybe remove your use.clj and in the repl check ( "user.clj")

hiredman22:02:35

what if instead of using the ns macro you call require in your user.clj?

rschmukler22:02:02

(ns user)
(require '[com.wsscode.pathom.core])
(require '[com.wsscode.pathom.connect])

hiredman22:02:11

what if you just load one or the other ns?

rschmukler22:02:58

connect depends on core, trying core

hiredman22:02:28

if connect depends on core, you shouldn't need to load it, loading connect will load core

rschmukler22:02:30

(and still running, point is, not 8s)

rschmukler22:02:14

Yep, requiring connect (I was using functions from core in my actual program) takes 1m+

hiredman22:02:42

but clojure -e "(require '[com.wsscode.pathom.connect])" is fast?

rschmukler22:02:18

Slow, actually

rschmukler22:02:21

actually wait

rschmukler22:02:26

my user.clj still exists

rschmukler22:02:29

Other fun facts to this mystery: My openjdk version hasn't changed, and I haven't upgraded pathom. It was working quickly a week ago. I've tried nuking my .cpcache and even my ~/.m2, neither of which did anything

hiredman22:02:40

put a (time ...) around the require in your user.clj and then a (time ...) around it at the repl

rschmukler22:02:16

it's running

rschmukler22:02:57

if you want to see if it's just my machine, I just pushed


rschmukler22:02:06

w/user.clj:

Ξ» clojure
"Elapsed time: 93907.317901 msecs"
Clojure 1.10.0
w/ eval:
Ξ» clojure -e "(time (require '[com.wsscode.pathom.connect]))"
"Elapsed time: 8657.945781 msecs"

hiredman22:02:08

that is something, I can't think of anything that would cause loading code from user.clj and from the repl like that to differ

rschmukler22:02:12

Quite a doozy huh?

rschmukler22:02:09

Does this seem like a bug to you then?

hiredman22:02:09

hard to say

hiredman22:02:58

in theory there could be a macro somewhere that says (when (= &file "user.clj") (Thread/sleep ...))

rschmukler22:02:58

Well, I saw it in other files too at least - unless a macro can see the originating file at the top of a require chain

rschmukler23:02:28

Thank you all for the help and taking the time to investigate this with me. I was able to implement a work-around for now, even though it feels like a bit of a hack.

andy.fingerhut23:02:44

I cloned your repo on a Mac running OSX 10.13.6 JVM 1.8.0_192 and Clojure 1.9.0. It took about 14 seconds both ways.

hiredman23:02:53

I've found it to be slow with zulu 11 jdk and fast with zulu 8 and openjdk-1.8.0.192.b12

hiredman23:02:45

when it is slow it doesn't look like loading one particular file is slow, but there is a sort of uniform slow down loading all the files (when initiated from user.clj)

andy.fingerhut23:02:33

On same physical Mac I mentioned above, in an Ubuntu 16.04 desktop Linux VM, using JVM 1.8.0_191 and Clojure 1.9.0, it took about 2.5 minutes for both ways. The java process has I think the default heap size of 512 Mbytes and fairly quickly got up to that amount of memory and stayed there. Maybe a lot of GC going on?