Fork me on GitHub
#clojure
<
2019-06-27
>
pinkfrog00:06:56

i have this error. can you help me decrypt this:

pinkfrog00:06:02

i don’t know what’s wrong.

andy.fingerhut00:06:37

Did this happen when you started a project that uses the core.unify library, either directly because you explicitly requested it, or if you didn't request it, then probably because you requested a library that in turn depends upon it?

andy.fingerhut00:06:31

My first guess is that somehow your project is attempting to use an older version of core.unify, one that has a small syntax error that has since been fixed in a later version, and you are using a version of Clojure 1.9 or later that is more picky about its syntax checking of some expressions.

andy.fingerhut00:06:08

That guess is based upon seeing the first line of the file you linked: "clojure.lang.Compiler$CompilerException: Syntax error macroexpanding clojure.core/fn at (clojure/core/unify.clj:83:18)." (plus seeing this kind of issue happen before)

andy.fingerhut00:06:42

If you see a syntax error during macroexpanding something like clojure.core/fn, clojure.core/let, clojure.core/ns and a few other things in clojure.core, that is a likely cause.

andy.fingerhut01:06:05

What were you trying to do immediately before that error appeared? e.g. were you running lein repl to start a Leiningen REPL, perhaps?

seancorfield01:06:20

@i I see leinjacker in that stacktrace -- it depends on core.contracts which in turn depends on core.unify -- so you may just need to ensure you're using the latest version of leinjacker (which depends on core.contracts 0.0.6 which uses core.unify 0.5.7 which I believe has the fix Andy is referring to).

seancorfield01:06:16

This page shows that core.unify 0.5.6 (and, presumably, earlier) had a bug that was detected by spec and is fixed in 0.5.7 https://archive.clojure.org/design-wiki/display/design/Errors%2Bfound%2Bwith%2Bcore%2Bspecs.html

pinkfrog01:06:14

i would like to anticipate that the trace should show me from where in my core.clj file that things go wrong.

pinkfrog01:06:39

but the trace now seems only show the point of error occurred in the lib, omitting the stuff of my core.clj.

pinkfrog01:06:50

@andy.fingerhut i was doing lein ring server 3002

andy.fingerhut01:06:00

It is entirely possible, even likely, that the problem occurred before a single line of code in your core.clj file was read.

andy.fingerhut01:06:46

because reading core.clj isn't the first thing that the lein ring server 3002 command does.

andy.fingerhut01:06:19

Also, if you want to make it easier to determine from a stack trace whether your code is involved, or not, you could rename your core.clj file (and the namespace created within it) to something more distinctive, like soul_talk.clj

pinkfrog01:06:07

for now, is the only approach being modifying the verison number of each page to be the latest?

pinkfrog01:06:28

since i don’t know which exact package is the culprit.

pinkfrog01:06:22

lein supports “LATEST”, I will change every libary to that

andy.fingerhut01:06:17

What is the output of lein deps :tree ? That shows a "tree" of libraries, and what depends upon what.

andy.fingerhut01:06:47

You could look through that output for core.unify, then its parent that caused it to be brought in, then its parent, etc. until you get to a project that is in your project.clj file explictly.

andy.fingerhut01:06:19

That might be a more targeted approach rather than updating them all to the latest versions.

andy.fingerhut02:06:51

Also, if you do ever use LATEST, you likely want to get rid of all of those at some point when you want a stable collection of software that has been tested together. Otherwise every time you resolve LATEST again you might get a different version.

pinkfrog02:06:57

yup. now the issue is resolved. i would replace with exact version in productuion code.

Alex Miller (Clojure team)02:06:02

Actually clj will not detect that anything has changed and will just continue using the cached classpath, so using LATEST is somewhat dangerous in that way

Alex Miller (Clojure team)02:06:40

You can use -Sforce to force a recompute

didibus04:06:42

So I've been trying to get this code to OutOfMemory:

(defn do-concurrently2
  [tasks c handler]
  (let [executor (Executors/newFixedThreadPool c)]
    ;; Submit tasks to run concurrently.
    (doseq [task tasks
            :let [handling-task (comp handler task)]]
      (-> executor (.execute handling-task)))
    ;; shutdown executor once all tasks have been processed
    (-> executor .shutdown)
    (-> executor (.awaitTermination Long/MAX_VALUE TimeUnit/DAYS))))

didibus04:06:12

But I can't get it to do so!

didibus04:06:51

From all that I know, since the fixedThreadPool uses an unbounded queue, this should OOM if you were to submit too many tasks

didibus04:06:20

Yet even if I constrain my Xmx to like 100m, and give it a million tasks which all have a 1 second sleep, I don't OOM

andy.fingerhut05:06:28

Is the collection tasks you are passing in a lazy sequence, or fully realized before calling do-concurrently2? If fully realized, that might be most of the memory that needs to be allocated anyway, and the queue of the Executor is a minor amount of memory over that?

andy.fingerhut06:06:48

I was able to get it to throw an OutOfMemory exception with 1 billion entries in the tasks sequence, a lazy sequence, c=4, and all tasks do nothing except (Thread/sleep 1000). I added some more printing to your do-concurrently2, and it showed that even with -Xmx100m, it was still able to enqueue over 700,000 of the tasks before the OOM occurred.

andy.fingerhut06:06:25

After that OOM exception is raised, the executor keeps going, gradually finishing tasks. The JVM didn't exit.

anirudh07:06:00

We are facing problems with our Travis-CI. The build fails as it is not able to pull dependencies from maven

the error is 
`Could not transfer artifact org.clojure:tools.reader:jar:1.2.2 from/to central (): Access denied to:  , ReasonPhrase:Forbidden.)
https://travis-ci.com/gojek/ziggurat/jobs/211003505 Has anyone else faced this problem before?

jumar07:06:15

I saw this yesterday:

Could not transfer artifact org.clojure:clojure:pom:1.5.1 from/to central (): Access denied to: 

jumar07:06:25

It was fixed after triggering rebuild

jumar07:06:30

perhaps just a temporary glitch?

anirudh08:06:25

I have been facing this for 2 days now. Have retriggered the build at least 10 times 😓

jumar08:06:36

Hmm, that's what worked for us yesterday 😮

jumar08:06:00

perhaps ask travis support

anirudh08:06:05

sure, thanks a lot!

gleisonsilva11:06:19

Hello! Can u help me? I'm trying to use incubator strint to have string interpolation and, when I define the string before calling the macro << it doesn't work... how can I workaround that?

(let [a "x"
      b "y"
      qry "select ~{a} from ~{b}"]
  (<< qry))
;; "qry"

jaihindhreddy12:06:17

<< is a macro and it expects a string literal. You gave it the symbol qry which got casted to the string "qry" and because there's nothing to interpolate, that was the output. PS: I would just use str and format in clojure.core for simple string interpolation.

gleisonsilva12:06:58

I was using... but I got at a point where << will be more suitted...

👍 4
Andrew11:06:31

@gleisonsilva

(let [a "x"
     b "y"]
 (<< "select ~{a} from ~{b}"))

gleisonsilva12:06:58

I can't... because I need to build a string for different cases... I mean, the ideal for me is delay the interpolation because a cascade of rules...

Mno12:06:10

you could try converting the sym into a str

Mno12:06:33

(let [a "x"
      b "y"
      qry "select ~{a} from ~{b}"]
  (<< (str qry)))

Mno12:06:31

I don't have access to a repl with strint to test this, sorry.

gleisonsilva12:06:41

i've tried... but... same thing.

gleisonsilva12:06:30

it returns "(str qry)"

Mno12:06:39

(<< "~{qry}")

Mno12:06:12

That should work

Mno12:06:23

oh wait..

gleisonsilva12:06:06

it returns the original string, but without the interpolation done

Mno12:06:38

I'm out of ideas for now.. but I'll keep thinking.

gleisonsilva12:06:46

What I can see from source code is it expect a thing that will be read as #^String reader... but I coudn't figure out how to overcome that..

Mno12:06:00

you might be able to make a macro to eval it before assembling the list, but don't ask me how.. I'm not good with macros

yuhan12:06:04

Yeah the fundamental problem here is that << is a macro, so it works purely based on syntax at macroexpansion time and cannot have access to the local qry binding

yuhan12:06:08

you can get around this by using eval, basically invoking the macro at runtime - but this would take away any performance benefits

yuhan12:06:35

I believe this should work

(let [a   "x"
      b   "y"
      qry "select ~{a} from ~{b}"]
  (eval `(<< ~qry)))

gleisonsilva12:06:22

😞 no... " unable to resolve a in this context"

yuhan12:06:51

oh right.. eval doesn't pick up local vars

yuhan12:06:20

Are you sure you can't use str or format in some way here? Most likely it would be simpler and more idiomatic than macro wrangling

gleisonsilva12:06:53

I was using format, but I got to a point where << will be more simpler e suitted for different "cases" inside this function where I'm working..

gleisonsilva12:06:18

in face of this issue, what I have done so far was duplicating code... instead of def that string and calling the interpolation later, I'm duplicating the code inside the if

yuhan12:06:08

Do you have any sample code to show?

gleisonsilva12:06:26

far from ideal, but I need to continue..

gleisonsilva12:06:20

(if pby
        ;; will generate a list of vectors as [partition-id query]
        (map (fn [[part-key sec]]
               (let [qry (if (.contains opts* :row-count)
                             (<< "select '~{tbl}^~{part-key}' as \"uid\", '~{tbl}' as \"table\",'~{part-key}' as \"partition\", count(1) as \"row_count\" from ~{tbl*} t ~{ftr}")
                             (<< "select ~{uid}, current_date as \"_extract_ts\", t.*, '_' as schemahash from ~{tbl*} t ~{ftr}"))]
                 [part-key (string/trim (str qry (if (empty? ftr) " where " " and " ) sec))]))
             pby)
        (let [qry (if (.contains opts* :row-count)
                      (<< "select '~{tbl}' as \"uid\", '~{tbl}' as \"table\", null as \"partition\", count(1) as \"row_count\" from ~{tbl*} t ~{ftr} group by 1,2,3")
                      (<< "select ~{uid}, current_date as \"_extract_ts\", t.*, '_' as schemahash from ~{tbl*} t ~{ftr}"))]
          (list [nil (string/trim qry)])))))

gleisonsilva12:06:58

if I could def before and interpolate later, I could avoid this second if...

gleisonsilva12:06:18

if (.contains...

gleisonsilva12:06:13

because there is a case inside a case...

yuhan13:06:10

maybe it would be cleaner if you could express the query as a vector of strings and "interpolate" by substituting keys from a map

yuhan13:06:59

(defn kwdsub [mapping xs]
  (apply str
    (for [x xs]
      (if (keyword? x) (mapping x) x))))

(kwdsub {:a "x" :b "y"}
  ["select " :a " from " :b])
;; => "select x from y"

yuhan13:06:38

this way you get to use regular map and sequence operations to build up the query instead of local vars and string manipulations

yuhan13:06:25

I wouldn't be surprised if there is already a SQL library out there that does this in a more sophisticated manner

gleisonsilva13:06:18

it's nice! I'll try! Thank you very much, man!

Andrew13:06:11

<< is for compile time string interpolation. Are you able to use format at run time instead? Or build up strings with str?

Andrew13:06:01

It's a better idea to use an SQL library for this anyway, and use SQL parameters to pass values to the DB

Ben Hammond14:06:21

has any used a Scala library from Clojure? I'm wondering about how best to use http://gatling.io

ghadi14:06:50

there's a clj-gatling somewhere

Ben Hammond14:06:11

yeah I've heard that it's incomplete

Andrew14:06:21

I've used clj-gatling (a while ago now) and it was fine my purposes

Ben Hammond14:06:40

I'll take another look at it

Andrew14:06:10

My recommendation: start using the core.async variant straight away.

Andrew14:06:59

I was testing HTTP endpoints and used the http-kit client (http://www.http-kit.org/) which worked well.

colinkahn18:06:51

I’m using async from clojure.test (actually cljs.test but same principles should apply) and wondering if there’s a way to either add a timeout for that block or if I implement my own timeout calling done, a way that I can check how many assertions were made in that block.

noisesmith18:06:14

sounds like something a fixture should be able to do

colinkahn18:06:42

For checking the number of assertions run?

noisesmith18:06:31

a fixture takes your test as an argument, and can wrap it in whatever context it likes

noisesmith18:06:43

the number of assertions run is tracked by clojure.test via a dynamic var

noisesmith18:06:09

a timeout could be imposed by flipping a switch inside the fixture while running the test async

colinkahn18:06:00

Would I still write the async block in the deftest itself, or wrapping the call to f in the fixture?

noisesmith18:06:51

I think still wrap the deftest body in async - I'm not sure thought

colinkahn18:06:23

Ok cool, yeah I’ll give it shot, thanks!

Graham Seyffert21:06:18

If you have a conversion function that takes e.g. a collection and emits a map, what’s the preferred style for naming those?…

Graham Seyffert21:06:10

We like to name maps like type-of-key->type-of-value, and naming a function like collection->type-of-key->type-of-val seems pretty gross…

Graham Seyffert21:06:09

You could of course do (into {} (map (juxt thing1->thing2 identity)) collection-of-thing-1) I suppose?

Graham Seyffert21:06:35

Or just (zipmap (coll-of-thing-1->coll-of-thing-2 coll-of-thing-1) coll-of-thing-1)

Alan Thompson23:06:19

Best way I've found is to make an abbreviation like key-value-map. Then you can add a prefix like make- or create-, or add a suffix like -generator, -factory, -builder etc.

Alan Thompson23:06:37

I think it also helps to use Plumatic Schema so there is a machine-readable way of specifying and enforcing the shape of the output map.

liath23:06:48

Any one happen to know if there's already something out there like core.async/timeout but with a higher resolution than 10ms?

donaldball00:06:44

I wrote a timer chan that emits ticks using a juc scheduled thread executor for more or less this reason

liath01:06:29

Nice, I think I'll have to scale down to ns but this is definitely a cleaner idea than the nonsense I was planning 😂

noisesmith23:06:39

also, with immutable data structures, the make- prefix is just noise - either it returns one that already existed or makes a new one, either way it's safe to use

noisesmith23:06:44

same goes for the -factory suffix - unless specified otherwise, assume something returns data you can use