Fork me on GitHub
#clojure
<
2016-07-08
>
jtackett01:07:19

Why you run a (future ...) then dereference the future, how do you return that futures thread back to the resource pool?

hiredman01:07:45

it happens for you

hiredman01:07:40

the way executors (which clojure's futures and agents are built on) work is there is a queue of jobs, and a pool of threads, each thread in the pool is running a loop that takes a job of the queue, and then runs it

hiredman01:07:21

so you never "check out" a thread from a threadpool

hiredman01:07:20

so when a future is run, it is job in the queue, and a thread runs it, and then moves on to other things in the queue

hiredman01:07:34

you can look at futures as an abstraction built on top of a work queue and promises

hiredman01:07:42

the future object is like a promise, and the work of the future is put on the job queue, with the addition of delivering to the promise whatever the result of the work is

hiredman01:07:18

where a promise is a mutable write once reference

josh_tackett02:07:44

@hiredman: Thank you for this walk through!

josh_tackett02:07:08

still need to figure out why our system starts timing out when we start flooding the API and full system with requests

hiredman02:07:40

That is going to depend heavily on what you are doing. At the limit, if you are giving a system more work than the hardware is capable of, then it won't work

hiredman02:07:37

If you have reason to believe that the system should be capable of more work, you can start looking for artificial limits

hiredman02:07:10

Timeouts I guess are likely to be the result of work sitting in a queue and not being handled in a timely matter

hiredman02:07:34

If you are using jetty, it handles requests on a threadpool that is capped at 50 threads be default

Alex Miller (Clojure team)02:07:06

futures are handled off an unbounded thread pool, so generally there should not be future tasks sitting on the queue unless they are arriving too fast to be handled. but, there are only so many actual cores, so the scheduler could be not getting to them

dpsutton02:07:18

i'm trying to write a macro for testing to save myself a lot of typing (is (= :data (remove-prefix "rc" :rcdata))) a bunch. I'd to create a toplevel and then pass it an arbitrary number of forms like (with-prefix "rc" (:rcdata => :data) (:datarc => :datarc)) ... I can get the macro almost there except its nested one extra level, ie, ((is (= .....)) so it ends up eval'ing (true true ....) which obviously barfs

bradford02:07:20

Hey fam! quick Q: trying to get a random string generated every time a fn is called, instead of at eval time. I thought 'repeatedly' did that, but I'm still getting the same result every time I call sub-suffix. What am I missing?

dpsutton02:07:29

the prefix tests used to end with a (reverse tests) but this also had the same issue although it didn't reorder the tests

dpsutton02:07:05

oh nevermind. let me keep looking

dpsutton02:07:29

@bradford, its working for me

bradford02:07:00

@dpsutton: It works on the CLI, but if you use it in a program that you eval, then it returns the same answer every time it's called in the code

dpsutton02:07:17

can you show me the code where you use it?

bradford02:07:18

yeah, that puzzled me at first too 🙂

dpsutton02:07:45

you're not confused why sub-suffix is the same every time, right?

bradford02:07:24

I mostly know why (it's evaled at load time), I'm just not sure how to get it to be evaluated every time the fn is invoked, instead

bradford02:07:30

(not quite the right words)

dpsutton02:07:35

i don't see where you call random-str again

dpsutton02:07:42

i only see the one call for sub-suffix

bradford02:07:00

oh, sub-suffix is called about 20 times per input file

dpsutton02:07:01

do you mean to make subsuffix a function?

dpsutton02:07:10

sub-suffix is def'ed not defn'd

akiva03:07:13

It’s the little things that get ya.

bradford03:07:39

@dpsutton: yup, all good now. thx

dpsutton03:07:01

way better to fix a simple issue like that than running into something truly nasty lol

dpsutton04:07:50

In my tests, I'm getting failing tests because clojure seems to not consider nil equal to '()

dpsutton04:07:00

damn, just read up on that. I'm used to nil being identical to '()

seancorfield04:07:57

@dpsutton: You’ll likely find seq or empty? to be what you’re looking for in your tests…

seancorfield04:07:26

Both (seq nil) and (seq ()) produce nil

dpsutton04:07:43

oh sorry. it's comparing two objects

dpsutton04:07:05

so (= {:a nil} {:a () }) ==> false

danielcompton04:07:59

@bradford you could also call (str (gensym "abc"))

bradford05:07:01

@danielcompton: oh, that is awesome. JVM hack lyfe

dpsutton05:07:04

that is also probably superior as that is guaranteed to be unique and not clash, whereas yours is, although a small small chance, not necessarily unique

yogidevbear13:07:47

Hi everyone. I was just wondering what the de facto standard currently is for connecting to MS SQL Server from Clojure?

donaldball13:07:34

That is by far the most commonly used sql database access library

yogidevbear13:07:46

Thanks @donaldball. The project's readme talks about the JDBC driver dependencies. What do I need to do to include these drivers within a project?

ricardo13:07:46

@yogidevbear: I’d also recommend looking at HugSQL, since clojure/java.jdbc is just the lowest layer.

yogidevbear13:07:26

Are there any good articles that I could read through to give me a better understanding of the various options and the pros/cons of each?

donaldball13:07:53

Assuming you’re using leiningen, just add c.j.jdbc and the driver of your choice to your dependency vector

donaldball13:07:41

I prefer honeysql+jdbc, but different projects call for different approaches

mpenet13:07:27

special mention for squee too

mpenet13:07:27

if I had to use jdbc I'd probably use it

ricardo13:07:25

Now @yogidevbear is probably overwhelmed and sorry he asked. 😄

dominicm13:07:06

squee looks pretty neat.

ricardo13:07:24

@yogidevbear: Perhaps start by saying what’s your level of experience and what you’re trying to accomplish. If you’re completely new, look at khorma, HugSQL and HoneySQL, then test first whatever feels the more natural to you.

mpenet13:07:54

dominicm: yeah, protocols all the way down + ireduceinit'ed resultsets is nice

yogidevbear13:07:47

@ricardo: My level in terms of CLJ is n00b. I work mostly in ColdFusion (also a JVM based language). I work with MS SQL Server on a daily basis and have started working with postgres recently. I feel very comfortable writing standard T-SQL queries. Does that help?

yogidevbear13:07:45

I'd like to keep RDBMS based queries as close to the native syntax where possible

ricardo13:07:25

@yogidevbear: If by native you mean SQL, then HugSQL is the way to go.

yogidevbear13:07:25

Yes. I don't really want a 3rd party library where I have to write the queries in a non-standard syntax.

yogidevbear13:07:50

Unless there is a very good de facto offering that is tried and tested

ricardo13:07:21

@yogidevbear: Here you go. http://www.hugsql.org/ The whole philosophy (which it shares with/inherits from YeSQL) is that SQL is a good enough DSL already.

yogidevbear13:07:09

Perfect, thanks. Are there any direct "competitors" to hubsql?

dominicm13:07:21

hugsql is fairly new of itself

dominicm13:07:46

yesql might be considered as one. But hugsql was built adding features to yesql.

ricardo13:07:20

Right. As @yogthos mentions here (http://yogthos.net/posts/2016-02-22-LuminusEmbracingHugSQL.html) it’s just not as actively developed.

yogidevbear13:07:05

Looking at the first example on the docs (`["select * from characters where id = ?", 2]`), does HugSQL deal with parameter sanitisation (eg. preventing SQL injection based attacks)?

donaldball13:07:30

If it wraps clojure.java.jdbc, yes

donaldball13:07:31

I would fwiw aver that honeysql is a well-tested expression of sql as a data structure, but if you don’t need queries to be composible, that may not be relevant

yogthos13:07:51

worth noting that HugSQL does have snippets http://www.hugsql.org/#using-snippets

krisajenkins13:07:23

While YeSQL’s being mentioned, it’s probably a good time to say: If anyone wants to take over the project, please contact me. I’ve been keeping it up to date with the latest Clojure releases, but realistically I’m not going to get time to add any features in the foreseeable.

yogidevbear13:07:05

Is it possible to call stored procs from HubSQL? Would it work in the same sort of way, using EXEC myproc ...?

donaldball13:07:59

(Is it time for a #sql room?)

leontalbot13:07:45

Hello! We want to teach clojure at a meet-up. Some kind of hands-on lab with less friction possible. Do we have an online Clojure Editor? Something like http://clojurerepl.com but a bit more robust (less buggy)

Alex Miller (Clojure team)13:07:06

it’s not perfect, but I have used it successfully for interspersing text and trying things

yogidevbear14:07:45

I think a #sql room would be great 🙂

donaldball14:07:01

I have made that #C1Q164V29 room, we might consider moving discussion there to leave this room for more general questions

cycle33714:07:09

I recommend korma which is a wrapper for java/jdbc

gmcramm14:07:42

Does anyone have any bad experiences with Digital Ocean? I’m looking at moving a project of mine over from Heroku since I’m hitting some memory limits and DO will be cheaper in the long run (albeit more work)

mpenet14:07:27

appart from the occasional hw failure that forces us to re-build a droplet from time to time no. Just make sure you have proper backups and failover scenarios/tools etc (same for any vps provider really, including heroku probably).

mpenet15:07:45

(it only happened a couple of times out of 50'ish droplets)

mpenet15:07:01

if you are in europe, have a look at exoscale

gmcramm15:07:44

Awesome, thanks!

donaldball15:07:24

Has anyone written or seen a map wrapper that maintains a set of all of the entries ever accessed?

donaldball15:07:55

A use case would be to audit usage of a gnarly config blob

manderson16:07:57

I'm peaking behind the curtain to try and determine how deftype is working in order to resolve a problem I'm having and am getting confused... The problem: I'm using deftype to implement a Samza interface for a serializer: Serde in order to add a Nippy serializer. I get an error, however, when I run the Samza job:

java.lang.IllegalStateException: Attempting to call unbound fn: #'taoensso.nippy/thaw
When I decompile the class that deftype generated, I see the code for the Var being created, but it is being done without a root and thus getting Unbound as its root (which throws the above exception when called). So, my troubleshooting has just lead me into more confusion 🙂 I suppose my primary question is, how can I fix my error, but it makes me curious how generated classes by deftype resolve Clojure function implementations under the covers in general. My wild guess is that some other process must bind interned vars with their implementations from the classpath and this isn't happening since my class is being used directly by another framework, but that's just speculation. Any thoughts?

harold16:07:50

Is there a way to determine (without calling) if a method exists for a given dispatch value on a multimethod?

manderson17:07:04

In case anyone is interested, I solved my problem ^^^ using gen-class instead of deftype. Primary differences in decompiled classes were deftype used: (Var)RT.var(String String) and gen-class used Var.internPrivate(String String) for creating the Var. Also, gen-class adds a static block that calls Util.loadWithClass, which I'm assuming is the difference. By analogy, it appears to me that deftype is to jar as gen-class is to uberjar. Anyway, enough learning how the sausage is made for now 🙂

eggsyntax17:07:37

In 1.9, is there a way to disable namespaced-map output (eg for data that's going to be parsed by a process that's unaware of the idiom)? ie to get output like {:foo/bar 1} rather than #:foo{:bar 1}?

eggsyntax17:07:38

It looks like CLJ-1967 addresses this. Anyone have an idea for a good hack in the meantime? http://dev.clojure.org/jira/browse/CLJ-1967

seancorfield17:07:51

Maybe you could redefine the print-method to print maps the old way? (based on this commit that added the namespaced map printing https://github.com/clojure/clojure/commit/6d48ae372a540903173be2974b66b8911371e05d )

eggsyntax17:07:45

Good idea. Or I may just go with string/replace as a purely temporary hack; I've only got one very specific corner case where this needs to happen.

eggsyntax17:07:13

This won't be production code until 1.9.0 hits.

seancorfield17:07:20

We’re on Alpha 7 in production (which doesn’t include the namespaced map printing) and we’re on Alpha 9 in QA right now. So far that change hasn’t broken anything we do.

eggsyntax17:07:48

@seancorfield: I'd probably be down with that, but the team as a whole isn't ready to make the leap. Doesn't matter in this case because what we're initially using our specs for is to generate seed data. I'm wrapping it up today, actually, & I'm quite happy with it. Fully automated datomic-schema-defs -> specs -> seed-data -> transact-to-db. We get auto-generated seed data, which we can push through our system to find bugs, and then when 1.9 hits we'll start using the specs for validation.

eggsyntax17:07:13

I've also set it up so that we can automatically generate either edge-casey seed data, for finding bugs, and/or pretty seed data for eg user testing.

seancorfield17:07:23

Have you considered the backport of clojure.spec (for 1.8)?

eggsyntax17:07:45

Considered doing one? Or did someone already do so?

eggsyntax18:07:26

Oh, I see, tonsky did one. Cool, I missed that announcement.

Jarrod Taylor (Clojure team)18:07:02

Is there a more concise way to accomplish this

(vec (for [row (range 5)] 
       (vec (for [col (range 3)] 
              {:address (str "R" row "-C" col) :other "asd"}))))
without having to wrap the for’s in vec?

jr18:07:25

You can use mapv for accumulating a vector

jr18:07:38

(mapv (fn [row] (mapv (fn [col] {:address (str "R" row "-" "C" col)}) (range 3))) (range 5))

peeja18:07:28

Is there a commonly used way to document function arguments other than just describing them in a docstring? Has anyone built a good structured documentation system for arguments?

peeja18:07:57

I thought I might be able to piggyback on clojure.spec, but it looks a little hairy

peeja18:07:59

I'm basically trying to display nicer documentation (in a web page) than I can with a single docstring (unless i parse that string and expect some kind of standard syntax to document arguments)

seancorfield19:07:28

Depending on what exactly you’re aiming for, you could use arglist metadata to provide more detail on your arguments…?

jstokes19:07:44

something that caught me off guard (yay floating points)… (range 0 10 1) returns a seq of 10 elements, but (range 0 1 0.1) returns a seq of 11 elements

seancorfield19:07:24

@peeja: I don’t think it’s exactly good practice but CongoMongo uses :arglists metadata extensively, e.g., https://github.com/aboekhoff/congomongo/blob/master/src/somnium/congomongo.clj#L195-L210

seancorfield19:07:52

It tends to throw tooling off tho’ (when the arglists metadata isn’t valid argument lists).

peeja19:07:45

@seancorfield: Yeah, something like that. I'm interested in a much longer docstring for each argument, though. Maybe metadata on the arglist symbols? Feels like that might be the most useful data-wise, but would probably look terrible in source.

thomas19:07:11

Hi, I am getting this error:

clojure.lang.ExceptionInfo: java.lang.Exception: Found lib name 'clojure.algo.generic.math-functions' containing period with prefix '[clojure.math.numeric-tower :as math]'.  lib names inside prefix lists must not contain periods, compiling:(gps_tracker/geo_code.cljc:1:1)
and I have no idea what it means and googling hasn’t really helped either really

thomas19:07:16

here is the code:

thomas19:07:46

any idea what I could be doing wrong? TIA!!!

akiva20:07:24

Guessing here but I don’t think you need to put the dependencies inside a vector container.

thomas20:07:31

@akiva: when I don’t do that I get problems with the :clj reader conditional

jr20:07:31

#?@ will splice the requires into the correct form

thomas20:07:31

thanks @jr I just re-read the answer over in #C064BA6G2…. DOH

akiva20:07:05

Yeah, it should look something like #?@(:clj [(:require [clojure.core :as lang])]) according to http://dev.clojure.org/display/design/Reader+Conditionals.

thomas20:07:13

yes that solved it thank you @akiva and @jr !!!