Fork me on GitHub
#clojure
<
2017-04-17
>
Alex Miller (Clojure team)00:04:12

Compiling with direct linking means an var invocation makes a static method call instead of derefing the var

Alex Miller (Clojure team)00:04:36

So the var is not used at all in the compiled code

Alex Miller (Clojure team)00:04:02

dynamic and redef vars are not direct linked

Alex Miller (Clojure team)00:04:02

redef vars are not dynamic

Alex Miller (Clojure team)00:04:01

Hard to help without a question

didibus00:04:24

@alexmiller Right, I checked the source. So ^:redef redef just says keep the Var, but its a non dynamic var, and ^:dynamic says keep the Var, but later will make it a dynamic Var. So I assume you can have ^:dynamic and ^:redef together without issue.

Alex Miller (Clojure team)00:04:46

Yes, although there's no reason to

didibus00:04:38

So the symbol lookup will still happen with direct linking? So unmap and intern would still work?

Alex Miller (Clojure team)00:04:27

If I understand your question correctly, no

Alex Miller (Clojure team)00:04:38

The compiled code is a hard coded static method invocation

didibus00:04:09

Oh, hum. I've been diving into the Clojure source. So I got it to understand that symbols are kept in a map. So I was assuming each function call in Clojure would first lookup that symbol in the map, and than deref the Var. And looking at the direct-linking code, it looked like it was just replacing the value of the map-entry of the symbol by the IFn itself. Which would mean that Clojure would still have to lookup in the map to find the function no?

didibus00:04:29

Oh, even without direct-linking the symbols are not looked up?

Alex Miller (Clojure team)00:04:41

Sorry I mean there is no lookup with direct linking

Drew Verlee00:04:16

Hm, so if i had to ask one question, it would be about this: > And you can pull tools out like about right now and do this for yourselves, so the important thing to note is that the Clojure state model is available at the systems level. You do it this way, and the only thing you have to do is put names on your values. What’s a good name for a value? What does he mean by the Clojure state model being available at the systems level? Is he talking about how (for example) Datomic entities are versioned and immutable?

didibus00:04:11

Okay, cool. So direct linking will be completely direct. Which, is great. Thanks. I guess I'll need to loo through the compiler code more so to understand how it does that.

Alex Miller (Clojure team)00:04:51

I believe the general notion is that you need the ability to have values with names, and the ability to safely apply a function that transforms the old value to a new value

Alex Miller (Clojure team)01:04:08

And safely requires CAS, which you can get at the system level with something like Zookeeper, or DyanmoDB, or a db transaction

Alex Miller (Clojure team)01:04:40

(All of which Datomic leverages for different storages)

qqq01:04:47

Has anyone started with a Clojure linter and then gradually added type checking to it?

qqq01:04:14

(and written about it somewhere that I can read)

noisesmith01:04:58

there's only one real linter I know of, and it does no type checking

qqq01:04:45

what surprises me is that sexp is so simple, but the clojure analyzer output is so complex

noisesmith01:04:26

there's also a typing extension for clojure https://github.com/clojure/core.typed

tbaldridge01:04:22

@qqq writing a simple lisp is easy, writing a fast lisp is hard. Clojure does the latter but in turn leans more on the analyzer. And there's even more it could do, at the cost of compilation speed, and compiler complexity.

didibus01:04:29

Does the Clojure compiler actually uses tools.analyzer?

Alex Miller (Clojure team)01:04:29

No, it was written later. ClojureScript uses it though

gonewest81802:04:51

Is there a coverage reporting tool for clojure that reports branch coverage as well as line coverage? It looks like cloverage doesn’t try to understand branches, per se, but it does track which forms are evaluated and which are not.

coltonoscopy03:04:08

is there a more idiomatic way of doing the following? (fairly new to FP and quite new to clojure):

coltonoscopy03:04:33

feels very OO, guessing theres a better way

seancorfield03:04:07

You wouldn't use the getters in Clojure...

seancorfield03:04:21

set-price-per-item would just be an assoc operation.

seancorfield03:04:06

In fact all those setters would just be assoc calls in the client code.

seancorfield03:04:24

Since data is immutable, all that get/set encapsulation is not considered necessary.

seancorfield03:04:09

You'll hear people say "In Clojure, the data is the API."

seancorfield03:04:35

I'd probably only have make-invoice (rename of new-invoice) and calculate-invoice-amount (rename of get-invoice-amount)...

coltonoscopy03:04:40

ah wow that cleaned it up quite a bit already 🙂

tagore03:04:26

Yep- the thing with Java, and OO in general, is that it leads to quite a lot of boilerplate.

seancorfield03:04:47

I wouldn't have the setters -- sends the wrong message: you're not really "setting" anything... you're creating a new invoice data structure with one or more elements having different values.

seancorfield03:04:43

This is the sort of thing that's the biggest mindset when you're coming from an OOP background.

tagore03:04:23

The functional approach requires less code- you have less need for an interface to your data when it can't mutate.

coltonoscopy03:04:55

@tagore you're right, all that boilerplate is one of the reasons ive been trying to make the shift to clojure in the first place 😅

tagore03:04:32

That said, there are reasons to use functions to get at data..

qqq03:04:43

@tbaldridge : (re complexity of clojure analyzer) -- I agree that writing a fast lisp is hard. But I don't see why that implies the syntactical analyser has to be complicated -- I could totally understand if the compiler / JVM bytecode emitter was complicated -- but the analyzer seems like a weird place to be complicated.

coltonoscopy03:04:57

@seancorfield thank you for the tips, the setters def aren't necessary at all since they aren't even used

tagore03:04:37

FP has an analogue to the uniform access principle, and in some cases a bit of boilerplate is worthwhile, either to delay binding, or to make access uniform, IMHO.

coltonoscopy03:04:29

@tagore yeah i can see how that would be particularly applicable to code bases where the underlying implementation for a field (be it direct access or computation) is liable to change

seancorfield03:04:43

Such change is much less common in the real world than a lot of people seem to think, IMO. 😐 Based on doing C++ and Java since '92 (before switching to Clojure around six years ago).

coltonoscopy03:04:37

i see both forms frequently but don't personally deal with change often if at all, you're right

coltonoscopy03:04:54

so much better!

seancorfield03:04:21

You'll get used to writing a lot less code 🙂

tagore03:04:31

Well, I'm known for my love of Perlisisms, and one of my favorites is "Functions delay binding; data structures induce binding. Moral: Structure data late in the programming process."

tagore03:04:02

There's quite a lot of wisdom in that observation, IMHO.

coltonoscopy03:04:21

i'm fond of raymond hettinger's "there has to be a better way"

tagore03:04:43

Well that's one that's likely always true 😉.

tbaldridge03:04:03

@qqq Clojure does quite a number of optimizations, such as inlining forms like clojure.core/+, and turning some calls like instance? into single-bytecode intrinsics. Add to that, that Clojure is statically-bound, and there's a large number of things that the analyzer can do to make code faster.

tagore03:04:07

but @coltonoscopy there are reasons to put things behind abstraction barriers. One thing I'd really strongly recommend to anyone stating with a Lisp is going through at least the first two chapters of SICP.

tagore03:04:28

Things have changed since it was written, and its style is not always what people would consider best these days, but it's woth understanding the approach it outlines.

coltonoscopy03:04:37

am on page 22! 🙂

coltonoscopy03:04:57

i think i see where youre coming from with the perlis quote

coltonoscopy03:04:01

more flexibility that way

tagore03:04:38

Well SICP makes it very explicit, since it works with essentially one data structure.

coltonoscopy03:04:08

i'm gonna go out on a limb and say it's the list 😅

tagore03:04:44

Another Perlisism: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures."

coltonoscopy03:04:52

that one ive heard recently

coltonoscopy03:04:57

i like that one

tagore03:04:09

There is value to uniformity.

tagore03:04:32

That said... the list is not quite enough for real day-to-day programming.

seancorfield03:04:03

The main data structure in Clojure for "objects" is the hash map. Followed by vectors for indexable data. Lists aren't used as much -- certainly not as much as other Lisps.

tagore03:04:24

But, and this is what SICP does a great job of showing, the function is universal.

seancorfield03:04:27

That's partly because Clojure focuses on a sequence abstraction, with concrete collections behind that.

coltonoscopy03:04:13

i have noticed that, i haven't seen lists used much for programming besides in macros. coming from python and lua, hash maps as the workhorse makes sense to me

tagore03:04:25

@seancorfield Yes- and as a practical matter that's a fine decision.

coltonoscopy03:04:00

@didibus agreed, implemented in the latest change, thank you 🙂 @seancorfield recommended as much

seancorfield03:04:16

What I've found, as I've continued to work with Clojure in production, is that I focus more and more on data design, rather than going in function-first.

seancorfield03:04:39

We're using clojure.spec heavily at work and that fits very nicely with a data-first design approach.

tbaldridge03:04:53

@seancorfield +1 to that. And sadly, scheme has only one data structure, makes it hard to get the data design "right"

tagore03:04:41

@seancorfield Yes, but... using clojure.spec is very far to the practical side of the continuum, I think.

seancorfield03:04:09

@tagore not sure what you mean about clojure.spec?

tagore03:04:11

I think it's also very useful to think about programming from a very impractical point.

coltonoscopy03:04:11

first i've seen clojure.spec @seancorfield but i have it open in a new tab and will add it to my reading list, thanks 🙂

tagore03:04:20

For instance SICP introduces almost nothing other than lists and functions in its first hundred pages.

gerred03:04:21

speaking of spec, has a concrete 1.9 timeline come yet?

seancorfield03:04:00

@tagore SICP is a great learning tool but not really a good basis for practical, production code.

tagore03:04:07

In fact, SICP introduces Church numerals (in a footnote) before it introduces quote.

seancorfield03:04:13

The world has moved on a lot over those decades.

tagore03:04:32

@seancorfield I agree, entirely.

seancorfield03:04:40

I think Clojure's real "win" is that it's a very practical language.

tagore03:04:14

@seancorfield That said, I think there are things tht can be learned from SICP that are very valuable even when writing production code.

tagore03:04:04

Though I think it would probably be a mistake to try to compute with Church numerals in production 😉

tbaldridge03:04:35

I've found that the amount of cruft you have to wade through in SICP, kindof offsets the gains.

tbaldridge03:04:11

"production code" often means "how do I SQL?", "How do I generate HTML", and not 2 hours on recursive vs iterative algorithms.

seancorfield03:04:48

I recommend the MIT video lecture series version of SICP as a way to learn about FP. The book is... an interesting piece of reference material.

seancorfield03:04:31

We consider Clojure practical enough to have made it our primary language a few years ago (after it being our secondary language since 2011) and we have over 50K lines of Clojure in production these days.

coltonoscopy03:04:02

im curious how many lines that would be in a language like java or C# 😅

seancorfield03:04:16

My gut says Clojure is typically 2-4x less code than Java but can be up to 10x less in some cases.

tagore03:04:29

Well- I have to write a great deal of javascript in order to make a living and...

tagore03:04:54

My general approach to JS is to treat it as a somewhat broken scheme.

coltonoscopy03:04:15

my understanding is that is what javascript is

coltonoscopy03:04:20

written to look like java 🙂

coltonoscopy03:04:49

or at least, what the creator wanted it to be

tagore03:04:51

I've found that approach to be a good one, to the extent that writing JS can be so 😉.

coltonoscopy03:04:52

obviously its not quite there

coltonoscopy03:04:34

i just can't wait until ES6 is as adopted as ES5 was prior to 2015

tagore03:04:13

@coltonoscopy Yes, but there's so much stuff in JS that you could use...

tagore03:04:11

@coltonoscopy And as for es20xx there is always Babel, or other transpilers.

seancorfield03:04:45

Or there's ClojureScript -- which gives a much better experience than any version of JS 🙂

tagore03:04:10

But I find writing very simple scheme-like JS the best policy, for me at least.

coltonoscopy03:04:24

true @seancorfield 🙂 i fear it will take a while to catch on though, so ES6 is at least a nice refinement of JS for the time being

coltonoscopy03:04:40

@tagore indeed re: babel, but even that is a turn-off for some people

seancorfield03:04:50

I dislike JS enough to avoid it at all costs.

tagore03:04:56

@seancorfield Indeed- I would far prefer clojurescript to JS.

coltonoscopy03:04:58

when we get full browser compatibility and people don't need to futz with transpilation, it'll be a nice day

coltonoscopy03:04:33

@seancorfield yeah JS at least in its ES5 incarnation isnt the greatest lang

coltonoscopy03:04:43

ES6 is def a step up imo though 🙂

tagore03:04:05

@coltonoscopy That's really not a big concerne, IMHO. Compiling to JS is easy enough.

coltonoscopy03:04:27

@tagore it's not a concern for you (or to me) but for many people it is 🙂

seancorfield03:04:30

That's a bit like saying "Scala is a better Java" so you should be OK using it... instead of just going straight to Clojure 🙂

seancorfield03:04:39

(sorry Scala!)

coltonoscopy03:04:05

oh im not saying you should start using it, just that its an improvement and having to deal with it if the situation arises is less painful 🙂

coltonoscopy03:04:12

im around it all the time

tagore03:04:19

@seancorfield I agree- if it were up to me we'd use Clojurescript, and not JS, where I work.

coltonoscopy03:04:19

not necessarily by choice

seancorfield03:04:21

(and, yes, I have used Scala in production so I feel justified in criticizing it)

tagore03:04:53

@seancorfield But it's not the browser support bit that matters. That's easy to fix by compiling.

coltonoscopy03:04:45

its easy to fix by transpiling but i can speak from experience when i say people are turned off by the fact that you have to go through a transpilation bit vs just loading the code into the browser via a script tag like you would do with ES5 🙂

coltonoscopy03:04:09

when i say i speak from experience, i mean i work with plenty of people (mainly more experienced with ES5) who arent a fan

tagore03:04:23

@coltonoscopy Hmm- has been along time since I used a script tag directly 😉.

seancorfield03:04:55

Oh, I'm sure folks who use JS today wouldn't see the value in using a better transpiled language... blub... 🙂

seancorfield04:04:40

This is probably getting all a bit off-topic for #clojure tho' so we should probably move to #other-languages or #off-topic at this point!

coltonoscopy04:04:48

it may just be the work bubble i'm in

tagore04:04:14

@seancorfield I completely agree with you- I wish we'd just use clojurescript at my place of business, but... it would be a hard sell

coltonoscopy04:04:01

react and redux are big now and from what i understand thats a nice functional paradigm shift

coltonoscopy04:04:07

dont have a lot of firsthand exp with em though

seancorfield04:04:09

FWIW, at World Singles we went with JS and React.js rather than ClojureScript because it was easier to build a team to do that on the timeline we had last year.

seancorfield04:04:37

But our front end team like the look of ClojureScript so who knows what our future may hold.

tagore04:04:09

Yeah, Redux is.. well honestly I think the whole concept was lifted from re-frame without attribution, but...

seancorfield04:04:21

(we did a ClojureScript p.o.c. a few years back but the tooling was so rough we struggled to get a good workflow going, so we set it aside for a while)

seancorfield04:04:42

@tagore Redux predates re-frame I believe...

tagore04:04:16

At any rate, redux is a fine framework, but a bit heavy on boilerplate.

tagore04:04:03

It's one of those cases where you really just want a macro.

tagore04:04:58

The thing is, no matter how clever your framework, js is still the underlying language, and it has problems.

tagore04:04:58

For instance, my shop let a bug into production this week, because....

tagore04:04:14

Javascript functions can't distinguish between being called with no arguments and being passed undefined as a single argument.

tagore04:04:00

That's not something that would happen in any reasonable language.

coltonoscopy04:04:42

yeah thats a weird design decision

seancorfield04:04:25

JS, where adding {} and [] produces... c'mon... who knows without trying it out?

coltonoscopy04:04:45

lolol yeah the "what the javascript" vid was great

coltonoscopy04:04:47

and wat man 😄

tagore04:04:00

Yep- the first rule of JS is "be very defensive." The second rule is "you weren't defensive enough."

qqq04:04:05

after all these years, I'm finally starting to get how to use spec / :pre in the body of a function, we do certain things to an argument xyz ; and in doing so, we assume that xyz has certain structure well, the spec should encode this structure, thus assuring that if the arg passes the given spec, the function won't throw exception due to this particular arg

tagore04:04:50

Was it you who mentioned Eifell earlier?

qqq04:04:09

I did mention Eiffel earlier, but there may have been more than one person that mentioend Eiffel earlier.

tagore04:04:23

Well yeah, that's one way of arguing for correctness

qqq04:04:45

it's not sufficient, because this method says NOTHING about post conditions

qqq04:04:59

the only thing it guarantees is "no dynamic type errors"

qqq04:04:07

err, "all tynamic type errors caught at function call"

tagore04:04:26

Very few methods are sufficient- none that I know of 😉

qqq04:04:45

dependent types 🙂

qqq04:04:39

I know about pr-str. Is there a prettyprint-str ?

qqq04:04:47

or is it still (with-out-str (pprint ...)) ?

didibus04:04:47

Specs are equal in power to dependant types

qqq04:04:41

but depndent types are checked at compile time

didibus04:04:16

Yea, true. Spectrum tries to validate specs at compile time. It's worth a look at.

tagore04:04:18

I've proved some very small programs correct.

tagore04:04:56

None were all that useful in isolation.

qqq04:04:10

here's the problem: Idris isn't fully useable yet.

qqq04:04:22

So unless clojure code is easier to analyze than Idris code, I don't see how Spectrum can be practically useful right now.

didibus04:04:24

Also, specs are probably going to allow you to catch some of the bugs dependant types would, while requiring less effort from the programmer.

qqq04:04:30

I don't think the field of dependent types is there yet.

didibus04:04:23

Spectrum is way more practical than Idris

didibus04:04:51

You don't need to spend anymore effort that you'd do if using spec

didibus04:04:29

So if you've got specs, spectrum can already use them to find errors at compile time

tagore04:04:45

Maybe this time it will be different.

qqq04:04:56

@didibus : that'd only be true if spectrum has b etter dependent type checker than idirs

qqq04:04:04

what evidence do you have of that?

didibus04:04:10

You wouldn't be able to fully prove 100% the for all case like in proof languages, that's the trade off

didibus04:04:35

Spectrum can never be as good as Idris, unless Clojure forces everything to be speced, even all Java code.

didibus04:04:08

But in terms of being practical, spectrum can already catch errors that even haskell would not, at compile time

qqq04:04:21

What is one example?

didibus04:04:55

Division by zero

qqq04:04:16

Can you show sample code?

qqq04:04:22

I have no idea what 'division by zero' means without actual code.

didibus04:04:41

Ya, give me 2min

tagore04:04:54

And of course the biggest source of software defects is that... your clients aren't very good at telling you what they want (and largely don't even know what they want,) and you're not very good at finding out what they want.

tagore04:04:29

I'm inclined to think that software is just difficult.

qqq04:04:34

@tagore: you're changing the argument, I want to see @didibus 's example of spectrum beating haskell 🙂

tagore04:04:08

@qqq Youre right about that, I suppose.

tagore04:04:33

Though I'd argue that my point is at least a bit relevant 😉.

qqq04:04:57

@tagore: it's relevant, but it doesn't demonstrate masterful use of spectrum 🙂

didibus05:04:34

@qqq Haskell for

a = 0 :: Int
b = 10 :: Int
c = div b a
main = print c

didibus05:04:44

This will give a runtime error

didibus05:04:50

In haskell

qqq05:04:19

@didibus : sure, this haskell code would be a runtime error; what can spectrum doe?

didibus05:04:23

With spectrum in Clojure

(let [a 0
b 10
c (/ b a)]
(println c))
will be caught at compile time if you spec this properly

qqq05:04:40

how do you spec it properly ?

didibus05:04:58

LiquidHaskell which does dependant type would also be able to catch it at compile time

qqq05:04:01

also, I think, with a few lines of constnat propagation, haskell can catch the above too

didibus05:04:30

Just say that a is int?, b is int? and c is int?

didibus05:04:53

And / is specced as #(not= 0 ret)

didibus05:04:56

@qqq Sorry, just that its specced so that its second argument can not be 0

didibus05:04:34

I actually don't think you need to spec a b c, it'll infer from the literal I think

didibus05:04:05

Anyways, spec can do this because at compile time it can validate literals and specs. So for literals, it'll check if they are valid to the specs they are used in. And it'll validate specs against other specs

didibus05:04:06

Not sure how it does the latter, that's where I think it would be hard to show that one spec is incompatible with another

didibus05:04:27

But in my example, it just associates that a is 0, and when it conforms it with the spec of / which says #(not= 0 secondarg) that fails to conform

didibus05:04:39

So it knows at compile time this won't work.

didibus05:04:49

So, obviously depending how you spec /, how restricted you make it, it could not catch it. But dependent types suffer from this same problem as far as I know, so it's possible to have false positives with dependent types.

didibus05:04:43

Or, it might be better to say refinement types. I don't know enough about Idris and true dependent types. But I think Spectrum could in theory equal LiquidHaskell, given a fully specced clojure code base, and enough work going in spectrum

qqq07:04:45

@didibus: do you use spectrum in practice ?

didibus16:04:54

qqq: I think it's not ready for practical use yet. It's not covering all specs, and I've seen it crash when running on some code base. It's still in alpha. But when it works, it's pretty magical. I'll definitly use it once it has addressed those issues. I think it could be better to think of it as a very good static analyzer, simply because of the nature of Clojure, not everything will be specced. When things are specced though, it has the potential of catching quite a lot of issues, some that even Hakell would miss.

didibus16:04:04

I'd recommend trying it out though, it's pretty simple to use, just one require and you run spectrum/check, done.

qqq07:04:09

spec has map-of // is there also a set-of ?

mpenet08:04:16

qqq: (coll-of ::x :kind set?)

qqq08:04:38

@U050SC7SV: thanks! I keep on forgeting about coll-of :kind arg

fossifoo07:04:28

https://clojure.org/reference/special_forms#def says "Creates and interns or locates a global var with the name of symbol and a namespace of the value of the current namespace (ns)." i don't get the "global" part.

didibus16:04:21

fossifoo: Global means that the var can be accessed globally from everywhere that requires it. It will always be inside a namespace, and there can only be one namespace/symbol pair, so if you intern that same pair again, it would replace the old Var.

didibus16:04:03

I think you're getting confused with the fact that when inside a namespace, Clojure lets you refer to symbols without using a namespace. That's just convenience, under the good, it'll just use the current namespace. There is always a namespace. Even in a repl, it'll take the user namespace if none is specified.

didibus16:04:42

Also, global does not mean can be referred to without having to require it. In the same namespace you don't, they get interned automatically. But in another namespace you could still access that global by simply requiring it: (:require [namespace :refer [symbol]]. You can not do that with a local or a function argument.

fossifoo07:04:23

does "global" in this context mean that there is only ever one var with the location "namespace/symbol"?

fossifoo07:04:50

or is there such a thing as true global vars somehow that don't have a namespace?

fossifoo07:04:54

the later seems to be the case?

mikethompson08:04:52

@tagore @seancorfield re-frame predated Redux by about half a year. But Redux's principle influence was the Elm Architecture. As i remember it, the idea for Redux middleware came indirectly from re-frame.

seancorfield15:04:44

@mikethompson thanks for the correction Mike. I thought that when my team had first tried building stuff with Om and then Reagent, that re-frame hadn't yet appeared (maybe early 2015?) but that Redux existed. So I'm misremembering some parts of that.

jonas08:04:09

@mikethompson before redux there were many flux inspired libraries in JS. I think it’s difficult to sort out who inspired who after the fact

pesterhazy08:04:15

I think Mike might know what he was inspired by

lxsameer08:04:42

hey guys, I'm trying to create a logger component, when i try to fetch the component instance inside the logger's helper functions it's empty, http://dpaste.com/070EMAF

mikethompson08:04:48

@jonas Yes, indeed there was a Cambrian explosion of Flux derivatives. And, because of David Nolen's OM, the idea of immutability was also thrust into the JS zeitgeist. But, from memory. the Redux authors did credit the Elm Architecture for a lot of their inspiration particularly around Action -> Model -> Model (foldp). I was just trying to set the record stright on two points: (1) re-frame was released well before Redux and (2) despite that I'm not sure that re-frame was much of an influence on Redux other than to suggest using middleware to wrap event handlers. At least that's how I remember it.

qqq09:04:39

argh, trying to write this relational db in clojure, without type checking, is killing me

qqq09:04:33

there are so many bugs that a type checker would have caught

qqq09:04:05

maybe what I really need is not clojure, but OCaml/Haskell with sexp syntax 🙂

didibus16:04:06

qqq: Have a look at shen-lang

didibus16:04:31

It's a statically type checked lisp that is fully portable.

didibus16:04:31

Oh, and actually, you could also use core.typed. It'll type check your code no problem. It'll be easier to use then spec I think, because specs tend to define more than what normal type systems require.

matan09:04:17

@qqq why write yet another relational db? anyway is a type checked language not a natural fit for that? of course you could verify types in clojure too, e.g. satisfies?

qqq09:04:41

it's in memory

matan09:04:05

back to tonight's internals discussion for a sec: just making sure: are volatile and static concepts ingrained/implemented at the JVM and the bytecode specification level? clojure docs sporadically mention them as vehicles for its underlying implementation and as an optional way to hinge values in a namespace, respectively. Most docs about them are from the Java world, so it is a little hard to trace how the concepts trace back or rely on the Java implementation of these concepts v.s. clojure accessing that functionality directly from the JVM, v.s. some other interpretation. Does anyone have the exact story on that?

didibus16:04:01

matan: I actually wasn't sure what you meant by Vars are static.

didibus16:04:18

Clojure volatile is a java volatile

didibus16:04:39

There's no difference

didibus16:04:42

None of those are static references though, they're boxed inside an object.

didibus16:04:04

So, yes, volatile is a feature of the JVM, not something custom implemented by Clojure or Java.

didibus16:04:32

But Clojure being implemented in Java, it accesses volatile by using Java. The volatile! function is an interop call to Java

matan17:04:49

@didbus the docs for Vars (or Refs) just say they use a JVM or Java static, that's all, IIRC

noisesmith17:04:51

btw, in general, you can usually expect clojure to directly use the jvm feature for something, with a thin wrapper if additional functionality is needed, but as I can recall currently never actually hiding the underlying things

noisesmith17:04:15

this is in contrast to scala, which has a bunch of language constructs that are not reified on a vm level

matan18:04:50

@noisesmith thanks for these comments, good to learn.

matan09:04:54

@qqq not sure I follow. In-memory and relational do not need to go together. There's already in-memory relational db's. so I wonder what is the motivation?

qqq09:04:06

@matan: have you read the paper I linked?

matan09:04:17

I skimmed it, I read too many papers already

qqq09:04:48

@matan: with all due respect, unless you understand that paper and want the system it proposes, what I'm doing makes no sense

rovanion09:04:33

Is there a list of all of Clojures glabal wars like *warn-on-reflect*?

noisesmith17:04:23

a cool feature is clojure.repl/apropos - clojure.repl will usually be in scope by default in a new repl (in your initial namespace at least)

justin@S: ~$ rlwrap java -jar ~/bin/clojure-1.9.0-alpha15.jar
Clojure 1.9.0-alpha15
(ins)user=> (apropos #"^\*.*\*$") ; find things that are earmuffed
(clojure.core/*agent* clojure.core/*allow-unresolved-vars* clojure.core/*assert* clojure.core/*clojure-version* clojure.core/*command-line-args* clojure.core/*compile-files* clojure.core/*compile-path* clojure.core/*compiler-options* clojure.core/*data-readers* clojure.core/*default-data-reader-fn* clojure.core/*err* clojure.core/*file* clojure.core/*flush-on-newline* clojure.core/*fn-loader* clojure.core/*in* clojure.core/*math-context* clojure.core/*ns* clojure.core/*out* clojure.core/*print-dup* clojure.core/*print-length* clojure.core/*print-level* clojure.core/*print-meta* clojure.core/*print-namespace-maps* clojure.core/*print-readably* clojure.core/*read-eval* clojure.core/*source-path* clojure.core/*suppress-read* clojure.core/*unchecked-math* clojure.core/*use-context-classloader* clojure.core/*verbose-defrecords* clojure.core/*warn-on-reflection* clojure.core.server/*session* clojure.java.browse/*open-url-script* clojure.java.javadoc/*core-java-api* clojure.java.javadoc/*feeling-lucky* clojure.java.javadoc/*feeling-lucky-url* clojure.java.javadoc/*local-javadocs* clojure.java.javadoc/*remote-javadocs* clojure.java.shell/*sh-dir* clojure.java.shell/*sh-env* clojure.pprint/*print-base* clojure.pprint/*print-miser-width* clojure.pprint/*print-pprint-dispatch* clojure.pprint/*print-pretty* clojure.pprint/*print-radix* clojure.pprint/*print-right-margin* clojure.pprint/*print-suppress-namespaces* clojure.spec/*coll-check-limit* clojure.spec/*coll-error-limit* clojure.spec/*compile-asserts* clojure.spec/*explain-out* clojure.spec/*fspec-iterations* clojure.spec/*recursion-limit*)
user=>

noisesmith17:04:26

granted, some are not configs that are for controlling thread-local behaviors, but all the thread-local configs should be in that list

rovanion13:04:23

That's interesting. Thank you!

rovanion09:04:57

The Leiningen docs mention that such a list should exist. But I haven't found it.

kwladyka10:04:04

what is the advantage of transduce over normal function with ->>? Why use it? Do you use it? When? My first impression is: it makes easy things harder without additional advantages, but maybe i miss something.

robert-stuttaford10:04:43

faster performance when you have multiple operations chained together in your ->>

rovanion10:04:10

dysfun answered my question on irc:

(filter #(re-find #"^\*.+\*$" (str %)) (keys (ns-publics 'clojure.core)))

kwladyka11:04:39

so transduce is faster then ->>? How much faster? Who use it in practice and when? Just i am asking to make a decision it is worth to try implement it in solutions or not really.

qqq11:04:29

in spec, when should I use s/cat and when should I use s/tuple ?

didibus16:04:48

qqq: They're equivalent, but s/cat will let you name each element, so it can be conformed into a key/value map, and can help provide more documentation to others.

didibus16:04:14

I often spec my fn args using s/tuple

kwladyka11:04:14

i would say in practice i don’t use it (probably because i work with different type of data)

qqq11:04:36

I feel like the only difference is in s/conform

kwladyka11:04:44

but from this doc explanation for s/tuple Designed for fixed size with known positional "fields" and s/cat Conforms to map with named keys based on the cat tags

qqq11:04:50

with s/cat, you get a map back based on the keywords,

qqq11:04:03

oh, s/cat can support regex

qqq11:04:07

that's the difference

kwladyka11:04:11

it returns output in different way - this is the first thing

kwladyka11:04:18

BTW first time when i read this doc i miss multi-spec in some way… this one could be very important

kwladyka12:04:54

@alexmiller thx, i am going to read it

tianshu12:04:02

any good library for cron-like scheduling?

tianshu12:04:27

I've just tried quartzite, however I can't find a way to set concurrent execution to false.

jstew12:04:25

used to be called cronj. It's not as flexible as quartzite but does get the job done.

tianshu12:04:30

😂 I was using http://hara.io before I tried quartzite, I found it a little buggy, I have both web server and scheduler running, once a http request happen, the scheduler will stop, very strange. but it looks have a upgrade version, I'll have a try

jstew12:04:53

That is strange, considering that the scheduler runs on it's own thread (or did when I tried it a couple of years ago)

tianshu13:04:02

I've just updated version, the problem still exists, looks like it's not stable:disappointed: or I'm using it in a wrong way.

mbjarland14:04:11

hi all, looking for some hints on the fastest possible way to read a file into a byte array. Current attempt (using clj-mmap):

(defn file-to-byte-array ^bytes [file-name]
  (with-open [mapped-file (mmap/get-mmap file-name)]
    (mmap/get-bytes mapped-file 0 (.size mapped-file))))
which for some reason still takes longer than it should (40ms from a hot state for my 40MB sample file. A memory mapped file, if things works as they should, should be more or less instantaneous)

mbjarland14:04:56

not sure if OSX is playing games with me here, I seem to recollect this was very fast on linux

mbjarland14:04:28

also, using the above method as in:

(defn some-fn [file-name]
  (let [data (file-to-byte-array file-name)
...
produces the following java code (decompiled):
Object data = var10000.invoke(var10001);
I thought declaring the return type as ^bytes should be enough for the compiler to figure out we are dealing with a byte array, not Object?

didibus16:04:17

mbjarland: you can set warn-on-reflection to true to have clojure tells you where it is needing to use reflection

didibus16:04:09

I'd also set (set! *unchecked-math* :warn-on-boxed) uncheckedmath to warnonboxed

didibus16:04:27

This will warn you when Clojure is using boxed numbers

mbjarland08:04:49

still learning to pay attention to the newfangled thread thingmajiggies in slack, thanks for the pointers. I have seen those settings documented, but failed to apply them to reality. Will give it a whirl. I would have still thought that defining a return value type hint would be enough, but we'll see what the warnings give me

mbjarland08:04:57

ok so the following code:

(ns clojure-license-plates.typing
  (:require [clj-mmap :as mmap])
  (:gen-class)
  (:import (clj_mmap Mmap)))

(set! *unchecked-math* :warn-on-boxed)
(set! *warn-on-reflection* true)

(defn file-to-byte-array ^bytes [file-name]
  (with-open [mapped-file (^Mmap mmap/get-mmap file-name)]
    (mmap/get-bytes mapped-file 0 (.size mapped-file))))

(defn test-type-warning [file-name]
  (let [data (file-to-byte-array file-name)]
    "foo"))
produces no warnings and still outputs the following decompiled java code for the test-type-warning function:
public static Object invokeStatic(Object file_name) {
    IFn var10000 = (IFn)const__0.getRawRoot();
    Object var10001 = file_name;
    file_name = null;
    Object data = var10000.invoke(var10001);
    return "foo";
  }
it still compiles to Object data = ..., and it does so even if I change the let to let [^bytes data (file...

mbjarland08:04:19

seems very strange to me, if neither the function return type hint or a direct type hint on the let local binding works, then how do we produce a primitive byte array without actually calling byte-array?

didibus00:04:46

@mbjarland There's no reflection though

didibus00:04:10

I don't think having data declared as Object is a problem, it would be later when you used that return value. Try having something use the return and see if that has to resort to reflection.

mbjarland14:04:13

on the upside this is part serious and part a competition between a few languages (mainly c++, kotlin, F#, clojure) with a number of my more ocd friends on solving the following problem (https://gist.github.com/mrange/ea0b2ab04238509dc4fdccf422c76c7a), and so far clojure (with one call-out to java…which is ok, we are on the JVM for a reason) is neck on neck with the fastest solution in c++, have to take into account things like cpu cache locality etc

mbjarland14:04:54

talking with myself here : ) just verified, loading the same file on ubuntu with mmap takes 10ms vs 40ms on OSX with comparable cpu, disk, etc

cddr17:04:14

Asked this in #leiningen but got crickets. Hopefully people won't mind me asking in here. Has anybody adopted rich's advice about ditching semantic versioning yet? Is there an emerging standard for cutting a release with a date string for the version id? Maybe something like "#date-fmt yyMMddHHmmss" in the project.clj which could be replaced at "release time" by running it through the java SimpleDateFormatter?

mobileink18:04:59

cddr: link to that advice?

cddr18:04:02

It was in the talk he gave at the conj this year: https://www.youtube.com/watch?v=oyLBGkS5ICk

didibus03:04:38

But that's far from Rich's vision.

seancorfield17:04:30

I don’t think anyone has figured out the right approach yet in response to that.

seancorfield17:04:11

Part of what he was advocating was never breaking an API across versions — only adding to it. If you want to break the API, you need to change the artifact name instead.

seancorfield17:04:54

So changing from “semantic versions” to dates doesn’t really address anything (beyond just creating a monotonically increasing version number).

weavejester17:04:14

I like the minor/patch part of semantic versioning. I think that’s still relevant. The major version also seems nice for indicating a “large” change, even if it’s not breaking.

weavejester17:04:55

I also wrote https://github.com/weavejester/lein-version-script some years ago for versioning based off commit number.

seancorfield17:04:25

I agree. There are a lot of expectations around version numbers these days so I don’t think they are inherently “bad” in any way. I agree somewhat with rich on not breaking APIs, given that we can’t have more than one version of the same artifact in our projects… but changing the artifact name to reflect breaking API “versions” seems a bit hacky so I don’t know what we can do better there.

seancorfield17:04:44

Except maybe think more about our API in the first place and never actually break it 🙂

cddr17:04:55

The alternative he proposed was to just have a new namespace for wholesale changes. Don't necessarily need a new artifact

mikeb17:04:31

The elm package manager enforces semantic versioning by checking compatibility, perhaps clojure could do something similar with specs across different major/minor versions.

cddr17:04:35

It would be mostly useless given the proportion of clojure projects that have < v1.0

cddr17:04:19

Ah but lein-version-script looks like it would satisfy my requirement. Thanks @weavejester!

shader18:04:58

how do I prompt the user for a password in a way that works on the console or repl?

fabrao18:04:04

Hello all, I have a question about defmulti

(defmulti update-ticket (fn[url user password ticket-info] ???))
(defmethod update-ticket "Incident" [url user password ticket-info] (update-incident url user password ticket-info))
(defmethod update-ticket "Request" [url user password ticket-info] (update-request url user password ticket-info))
and ticket-info has {"ticket-type" "Incident" ...} {"ticket-type" "Request" ...} ?

fabrao18:04:47

is there any way to use defmulti?

fabrao18:04:41

how to use "ticket-type" in deflmulti?

madstap18:04:05

(defmulti update-ticket (fn [_ _ _ {:strs [ticket-type]}] ticket-type))

Matthew Davidson (kingmob)18:04:57

Does anyone have any suggestions on working with unsigned bytes in Clojure (and Java)? I’m trying to do some crypto exercises, and keep butting into issues with a) Clojure defaulting to Longs and b) Java having only signed bytes. I looked at ztellman byte-streams lib, but that’s more about conversion and less about working with the results.

noisesmith18:04:30

@kingmob the useful thing is that if you force unchecked math the basic arithmetic operators and bitwise operations still do the right things

noisesmith18:04:02

that is, java interprets the bits differently, but the bits have the right contents

Matthew Davidson (kingmob)18:04:20

@noisesmith The issue is less with things like bit-xor, but that, when a negative num gets converted to Long, the sign bit changes location

Matthew Davidson (kingmob)18:04:36

(goes from bit 8 to bit 64…)

noisesmith18:04:41

right, you need to do an unchecked bitwise or with (long 0)

noisesmith18:04:40

I used to know the resources describing precisely how to do this, searching and seeing if I can refresh my memory right now…

Matthew Davidson (kingmob)18:04:06

bit-xor already does the right thing. I guess what I’m asking is, what’s the ideal format to represent my data?

noisesmith18:04:54

use (bit-and b 0xff) to get a long

Matthew Davidson (kingmob)18:04:59

E.g., if I use byte[], the bits are ideal, but I lose out on a lot of Clojure’s seq fns, have to use amap and areduce

noisesmith18:04:05

(if you need the unsigned value for math

noisesmith18:04:35

@kingmob there’s vector-of which can hold unboxed bytes

noisesmith18:04:53

> (doc vector-of)
-------------------------
clojure.core/vector-of
([t] [t & elements])
  Creates a new vector of a single primitive type t, where t is one
  of :int :long :float :double :byte :short :char or :boolean. The
  resulting vector complies with the interface of vectors in general,
  but stores the values unboxed internally.

  Optionally takes one or more elements to populate the vector.
nil

Matthew Davidson (kingmob)18:04:10

Ahhh, maybe that’s what I want

noisesmith18:04:41

np - it’s too bad this stuff has to be so complicated on the jvm

Matthew Davidson (kingmob)18:04:13

Even back in 1995, I remember wondering why Java lacked unsigned types…

noisesmith18:04:39

also, you mentioned byte-streams from ztellman before, I think it really will help now that you are thinking about what format you want it to be in

noisesmith18:04:32

also, you can call map / filter / reduce etc. on a native array, clojure knows how to make a seq out of them implicitly

noisesmith18:04:53

just be aware you’ll probably get a seq back (and might need to nudge it into another container after that)

Matthew Davidson (kingmob)18:04:58

I looked it over very carefully, and it seems to be more about converting between different Java types, less about Java <-> Clojure conversion

Matthew Davidson (kingmob)18:04:08

Maybe with a custom transformer…

Matthew Davidson (kingmob)18:04:22

Um, any thoughts on Plumatic’s hiphip?

Matthew Davidson (kingmob)18:04:57

It looked nice, but doesn’t support byte, only int/long/double/float

shader18:04:50

System/console doesn't seem to exist from the repl in cider; any other ways to prompt for a password properly in that context?

Matthew Davidson (kingmob)18:04:24

@noisesmith I know map/reduce can operate on Java native arrays, but if I’m not careful, I’ll get Longs back. Unchecked math will only ignore overflows, right? It’s not the same unsigned math, yes?

noisesmith18:04:26

(hope that’s not a non-sequitor)

noisesmith18:04:25

maybe make a unit test, try a few usages of unchecked-byte followed by the kind of math you are using followed by unchecked-byte again and see what value you get back

noisesmith18:04:33

I think it will work

Matthew Davidson (kingmob)18:04:05

That might help, too. I’m not really doing any math at my level, just a lot of xor-ing

noisesmith18:04:31

aha, in that case (into (vector-of :byte) (map unchecked-byte) coll) should turn any of your results back into a reasonable shapre

noisesmith18:04:10

=> (into (vector-of :byte) (map unchecked-byte) [1 2 3 4 254])
[1 2 3 4 -2]

Matthew Davidson (kingmob)19:04:22

Thx! Idiomatic question: is (into (vector-of:byte) …) preferable to (apply vector-of :byte) ?

noisesmith19:04:43

into lets you use a transducer

noisesmith19:04:53

otherwise you create a lazy-seq that nobody actually needs (via map)

noisesmith19:04:28

you can throw other transducers in there too, if you find them handy

danp19:04:25

Hey all, just a quick question...

danp19:04:32

I am using the REPL to explore a Java SDK, and have started to make a few small namespaces with functions wrapping the Java interop stuff.

danp19:04:19

I've got a "connection" namespace that ultimately defs a connection instance, say conn-instance, that I intend to use. As I'm the only user of the SDK, and will only need a single connection, is this a reasonable approach?

danp19:04:51

I understand that in a larger system I should look at the component library, and I have the alternative of using dynamic vars, but as I'll only ever need the def'ed instance, am I right in thinking these are unnecessary for now?

tbaldridge19:04:06

"I will only need a single one of these" -- famous last words

noisesmith19:04:17

I would avoid putting anything stateful in a def

tbaldridge19:04:46

Seriously though, it's very little extra work to just make a constructor function. I've always come to regret global singletons.

danp19:04:48

@tbaldridge I was thinking that as I was typing it 🙂

danp19:04:35

@noisesmith, what would the easiest alternative be? I'm trying to be fairly nimble in this particular work and not had any experience of dynamic vars nor component (though I've been reading bits and bobs on the latter).

noisesmith20:04:24

I think he already addressed this, but if possible use arguments to functions. Out of top level defs, an atom inside a def, a thread-local dynamic var, or an injected dependency (the main alternatives here), a regular argument to a function is the only one that is easily made into all the others. So it’s the most flexible, and if you stick to that options wherever possible, all the other options are just simple adaptations of it if needed.

danp20:04:42

Understood, thanks for your help.

danp19:04:37

@tbaldridge - so the constructor function, that would be a function wrapping the Java constructor, that I would then def a var for?

danp19:04:10

Letting me create multiple instances if necessary

tbaldridge19:04:27

no, just do (defn connect [conn url] ...) and then pass that result around like a normal value

danp19:04:58

I see, I think I'm probably just being idle and not wanting to type the whole function call when i use it. Thanks for your help 🙂

benny20:04:14

is there a general “rule of thumb” when to use get-in vs thread-first/`->`?

benny20:04:48

i.e.

(-> my-map :person :name :first)
vs
(get-in my-map [:person :name :first])

coltonoscopy20:04:22

not qualified to give advice, since i'm new

coltonoscopy20:04:27

but i much prefer the former

jr20:04:44

depends on the data structure and context

benny20:04:56

specifically with a map

jr20:04:01

(-> my-map :foo 0) won't work for a vector

spei20:04:27

i prefer get-in because it implies that that you are dealing with a map of maps. This condition is not necessarily true when you use the thread first macro

dpsutton20:04:19

get-in allows for a default value when not found. Although you don't necessarily always use it, it means that the instances that do use it would look drastically different from instances that don't use it, leading to drastically different forms for similar behavior

dpsutton20:04:07

ie, you prefer the threading macro but need to use a very orthogonal looking s-exp for similar results

plins20:04:47

is there something similar to javascripts Promise.all to sync async operations/

noisesmith21:04:02

@plins how are you doing async? via core.async or some other method?

noisesmith21:04:55

what arey ou doing in the callbacks?

noisesmith21:04:49

if you create promises, and deliver them in the callbacks, then (do @p1 @p2 @p3) for example won’t return until all those promises have been delivered. But that’s just one approach.

noisesmith21:04:40

or if your promises are a collection (run! deref promises) won’t return until they are all delivered

noisesmith21:04:19

oh - acually I forgot how that library works - in fact you could just start all the requests, and then run http/await on them in order

noisesmith21:04:33

they will continue to run async in another thread, and await will return as each one completes

plins21:04:13

ok thanks! 🙂