buffering is going to mask any problems with feedback you have

ok, just got a big burst, it looks like I'm not blocking when I publish, the pre/postpublish log messages are all within the same millisecond

it sounds like you are interfacing regular thread using code and core.async, and my bet is you are either using put! to publish or (async/go (>! ...)), when you should be using >!!

Also, go blocks can die without visible feedback or error messages, you could just get backed up as consumers silently fail.

yes, I'm using async/go >!

the question is how you using it

its funny but every example of pubsub I came across did it this way

if you create a go block from a normal thread, you should wait for its result using <!!

I mean, I am kind of guessing here and what are likely to be issues

I still think it is some kind of issue with your topic-fn, but you have assured me it isn't

is identical? guaranteed to be an O(1) time pointer comparison

if the publishers aren't actually blocking, all the symptoms point to a disconnect between pub and sub, the likely reason is the result of topic-fn isn't what you think

so I might do something like replace the topic fn with (constantly 1) make a channel subbed to 1, and verify it is getting all the traffic

@hiredman ok, but I think based on what @noisesmith was saying is that my usage of go (>! ...) could be the prob

literally all of my topics are the same {:transformer-event :ev1 or :ev2 or ev3}

definitely, like I said, "if publishers aren't actually blocking" if they are blocking and you just can't tell, or they are just dying, then that would be your problem

cool, I'm gonna change my publish to use >!! and see how that affects the situation

>!! is really blocking, not for use in go blocks, only for use on real threads

yea I took out the go block

doesn't seem to be making any different behavior

your producer threads are still not blocked?

I have a log message before the publish and another after, and they are always within a millisecond

doesn't look blocked to me

are you sure those aren't spurious log messages coming from threads left running from a previous attempt?

valid question, I'll restart the whole shebang

this is really puzzling to me.

given that I'm not buffering what are the scenarious that would cause events to be dropped?

well, you could think events are being published when they are not, because your producers are just blocking. the topic-fn could be returning something you are not subscribed to so messages get dropped, or your subscribers do get the messages and just don't do anything

givent that I only have 1 publication on one channel there should be no pub log messages that are publishing into nothing

there are however of course several sub channels and loops

are you sure you are subscribing to the same pubsub you are publishing to?

ok, now i'm finally seeing some blocking

I see several pre publish logs, with no coresponding post publis logs

right, that blocking is back pressure being communicated back from the downstream async bits, which are stuck where or going really slow

are the down stream bits re-publishing to the same pubsub?

no thats part of this whole business I've never been too clear on. I have no logic for re-publishing

by downstream bits are you referring to the sub handlers?

yes, things downstream of the pubsub

oh, well my sub handler loops publish on to the sub channels, is that what you mean?

the sub channels are obviously differnt than the pub channel

that could be the source of deadlock

whenever you have a feedback loop (a process that feeds output back in to its input) it is really easy to deadlock

since the sub channels don't have a buffer, publishing to them blocks until someone else consumes from them

depending on what you are doing, adding a buffer might fix it, or might not, because a buffer is fixed in size, and if you feedback more than the size of the buffer, you are deadlocked again

ok let me get this straight, my listener loops take off of the sub channel

the clojure publisher logic puts on to the sub channel based on the topic-fn

all of my sub loopse run forever taking from the sub channels and then calling a handler

I think you're saying if my sub loops get stuck then that could be causing the deadlock, like handler never returning I guess

oh

if you are calling a function that never returns that is another problem

you cannot do that kind of thing from a go block

that would all make sense if the whole thing locked up and never recovered, but that doesn't happen

it will block the threadpool go blocks use

no I don't thnk I'm calling a function that runs forever

I'm not seeing anything like that

if my handlers got stuck the whole system should eventually lock up, and that never happens

or if they run slow, or if they do blocking stuff on the async threadpool

the handlers can be slow compared to the publishers, when there is lots of activity, they all run in their own go blocks

you are ddosing the async threadpool

too much activity for the go blocks that the handlers are in?

I dunno about too much

it sounds like everything is behaving as designed, feedback is slowing your publishers to a rate that matches the consuming

well except for the lost events

that I still don't understand

in this case, the processes reading from the sub channels aren't able to get time on the threadpool to run

are you sure they are lost? or are they just waiting to run in a completely backed up system

you see the thing is, my publishers are bursty, there are short periods of busy activity and long periods of relative calm,

given that the system never locks up, all events should eventually be handled in the calm period, but that isn't what is happening.

aggh, I'm gonna have head home soon, thanks for being a sounding board on this pub/sub stuff

you might want to look at one of the pipeline variants, it may simplify things a lot

I didn't even know about pipelines, something new to learn tomorrow I guess

cool, thanks, have a good evening (or what ever timezone appropriate part of day is appropriate)

@qqq if you didn't already look at the source a few hours ago, yes, identical? is just:

static public boolean identical(Object k1, Object k2){
	return k1 == k2;
}

Trying to flatten a nested map to a 'referenced nested map':

{:a1 {:b1 "val1" :c1 2}
 :a2 {:b1 "val2" :c2 {:d1 3}}}

[{<0> {:a1 <1> :a2 <2>}
  <1> {:b1 "val1" :c1 2}
  <2> {:b1 "val2" :c2 <3>}
  <3> {:d1 3}]

@tjscollins did you get an answer ? is this useful https://github.com/r0man/oauth-clj/blob/master/src/oauth/google.clj

I am somewhat bewildered by core-async terminology. pipeline is for computation (non-blocking all the way), pipeline-async is for "async" operations which seems to be operations which may park the statemachine and pipeline-blocking is used for straight up blocking ops. I am mostly confused about ´pipeline-async`. Am I right in assuming: * (#A1) some operations, say (a/<! (a/timeout 3000)) - would park for approximately 3 seconds - so operations like it (take from a channel) are OK inside of pipeline-async. * (#A2) the reason the af fn in pipeline-async receives a channel onto which the result should be delivered is because I can then easily wrap a typical callback-style async operation by letting the callback put a message onto this result channel ?

@tjscollins this might be better .. I think you need OAuth2 for google and this repo has a google example https://github.com/craygo/clj-oauth2/blob/master/src/clj_oauth2/google.clj

No one has some thoughts / can confirm/deny my assumptions mentioned above ?

@pseud the purpose of the chan passed to af is to allow N results (0 or more) onto to from each call too af

Otherwise it would have to build and return a collection.

@joshjones: (re identical? being pointer equality check) -- nope, haven't read source yet; thanks for checking for me 🙂

Is there a exports.default equivalent (from Node.js) for Clojure? Use case, I have a file called config that loads an EDN file, parse it and store it to a variable called config. Whenever importing it I have to config/config which looks... Weird.

@rinaldi what about refer? (require 'my-app.config :refer [config])

@hmaurer Thanks, that could work. However, I am also exposing two other variables on this file that I later want to reference too. As far as I understand, with your solution, I would only be able to refer to the config variable and nothing else.

@rinaldi you can add any number of variables/function in the vector after :refer

@hmaurer Oh I see, let me try that.

@rinaldi actually I got the syntax a bit wrong. It’s (require '[my-app.config :refer [config]])

@rinaldi another thing to consider is that since good clojure style is to use the namespace prefix, you can use a name that isn’t redundant with the prefix

for example if the ns were environment so it was environment/config and the other things in the ns were environment related but not the config data itself, or if the var were settings, so it’s config/settings and the rest of the ns is about things that are about config but not the specific settings

@noisesmith is using :refer bad clojure style? I am new to clojure so I don’t really know

@hmaurer it’s OK in moderation - but there’s a reason nobody writes ns forms with :use any more

Thanks both @hmaurer and @noisesmith.

Since it only exposes 3 public vars for now, I'm going with :refer

that’s the most I’d want in a client ns period

not just from one ns - from all combined

@noisesmith :use is an order of magnitude worse though, right? it just refers everything

right, but it’s a matter of degree not kind

yep, true

@noisesmith What if I have a library that exposes a bunch of stuff but I just need a few of them. Still the best practice is to go with namespaces instead of using :refer and :only?

use :as

unless you are using the function extremely frequently

But :as would still load everything from that file into memory no?

that has nothing to do with it

all :as does is create an alias in your ns

the whole file is compiled and loaded into permgen regardless of how you refer to it

not using :refer is not about memory usage, it’s about poluting a namespace with bindings that are irrelevant

Gotcha, that's nice to know. Thanks.

it’s about code readaiblity

So I don't really see a use case for anything other than :as (perhaps only :use because can be handy on the REPL)

yup - that’s almost always the right way to do it

yeah tht's the general consensus that's emerged

but there are exceptions eg. if every function in your ns is using >! from core.async

that makes sense to refer

notable exceptions deftest and is

@dpsutton yeah, to me that’s the same exception - every form in my test ns uses deftest once and is multiple times

so it’s ubiquitous enough to drop the prefix

for sure. i'm a big fan of the namespaced functions. common lisp gets difficult to read because of that. also emacs-lisp...

it’s another flavor of the same problem overuse of inheritance causes - “where the hell is this name defined?”

but inheritance tends to get into a worse state because it’s so common to rebind the same names

https://stuartsierra.com/2016/clojure-how-to-ns.html

i quite like these principles

i actually alias clojure.test, mostly to get completion and automatic require in cursive

yeah cursive actually makes :use/:refer even less appealing

oh can it not suggest things that aren't in the current namespace if they don't have an alias?

I'm writting a macro that transform a keyword into a symbol This symbol will be used on a (def ~symbol ...) There is a "default" way to normalize :foo.bar/quoo to a valid symbol?

(symbol (namespace kw) (name kw))?

def can’t take namespaced symbols though

yeah, probably just munge the ns & name into a string, separated by some other string, if the namespace is important

@csm I need a local symbol. Some like foo_bar__quoo

if it needs to be “local” why is it going in a def?

perhaps I misunderstand what you mean by local here, but by the standard clojure definition, def doesn’t and can’t be used to create locals

(my-cool-macro :about.this/keyword [some cool dsl]) macro-magic> (def about_this_keyword :result-of-my-macro)

That addresses the question, that doesn’t do anything with locals as clojure defines them. But given that in order for the macro to work :about.this/keyword needs to be a literal, why not just provide the symbol directly?

@dpsutton no it suggests them fine, it just feels more useful when it narrows it down to a specific alias

ah yeah. easier to browse

CIDER has a browse namespace feature which is really nice

hello, i’m trying to take a sqlvec like

["SELECT * FROM ? WHERE (? = ? AND ? = ?)"
 "Aliens"
 "WritesPoetry"
 "True"
 "TheirPoetryIsGood"
 "False"]

replace ? with '%s' and [...] with (format ...)?

that would certainly do the trick

but sufficient to prevent against malicious input?

definitely not 🙂

therein lies the rub

if athena doesn't have prepared statements then you'll need a function that escapes sql in a manner that's safe for athena

the ? after FROM isn't actually valid jdbc if I recall either

jdbc, or all the jdbc drivers I have used, don't let you parameterize the table name

apparently the .toString method on a prepared statement for some jdbc drivers will give you the sql string

athena also has a jdbc driver, so you could just use that

as it turns out athena does have a jdbc driver

happy days

https://docs.aws.amazon.com/athena/latest/ug/connect-with-jdbc.html

thanks all, much appreciated

what clojure libraries exist for desktop ui's? is there anything like re-frame for clojurefx? not really interested in electron

i found a lot of things that are out of date.... looking for something that's been maintained

@cjhowe I maintain fn-fx, but to push it much further we need users. Not a whole lot of people trying to do JavaFX with Clojure

https://github.com/halgari/fn-fx

That being said it's React for JavaFX and works fairly well for "normal" apps, the rough edges are around things like animation.

hi, just a quick question, how do you do comment blocks? do you just put a string inside (comment)?

@tbaldridge i see... well, i'm a student trying to build a portfolio, so this seems like an interesting area that clojure could improve on... do you know if javafx is popular in the java world?

@joelsanchez #_(this block is commented)

ok, i see, thanks!

@cjhowe there aren't a lot of Java desktop apps, but JavaFX is a really nice platform. It's fast, GPU accelerated, and the API is very uniform. Infact the vast majority of fn-fx is auto-generated code created by the library introspecting on the JavaFX API.

@tbaldridge sounds interesting, i'll check it out! thanks

@joelsanchez "comment block" is a little ambiguous. do you just want a block of comment text (like a multiline comment) or do you want to comment one or more sexp's?

@mbjarland without going too far into it, your filter example is lazy and your transducer example isn't, and you're only taking the first result. so the comparisons are going to be a bit meaningless

@mbjarland are you sure of your measurements? The above benchmark code looks very questionable

also without showing what year-dirs-only-xf is defined as we can only really guess as to what's happening

@dpsutton yes, most of my project is commented like that but it's a little bit of a pain

not sure which editor you're in, but most should have some way to format a long single comment

Cursive

what is the pain point in using the language's comment syntax?

it does not bother me to use it, it bothers me to write ;; all the time when i am writing it

i'll bet cursive will do it for you

but if that's what you prefer go for it

i'll try to activate something in cursive for that, because right now it does not

@mbjarland: ok then yes it looks like it's pretty much because your latter example is lazy and your format example is not

@bfabry was trying to not spam the channel with too much code. Any easy way to keep the composability of transducers but stay lazy and fast?

don't enter newlines? i'll bet the autoformatter takes care of it for you

if the situation/algorithm benefits from laziness my first instinct is to say "don't use transducers there"

doesn't look like it @dpsutton , there's an option to auto-wrap at the 80-columns limit but i don't want that for code, i will ask in the cursive channel

@bfabry is non-laziness inherent for transducers or is it just the way I use them?

I believe so, one of the reasons that they're faster (in situations where laziness isn't a benefit) is because they don't have the overhead of keeping track of unrealised computations

it’s a lot simpler to enumerate the things that are lazy (there’s very few), and sequence is the only built in lazy thing using transducers and I forget the details but saw it explained that sequence was sub-optimal in important was compared to other lazy functions

a transducing function just transforms a single result, so would hardly benefit from laziness, a transducing context can be lazy or not (and sequence is the only lazy built in transducing context)

@bfabry @noisesmith thanks, I will go back and meditate some more on transducers and laziness…

is there a way to access the value of the default ring memory store atom from the repl?

(* 3 (/ 1 3))
=> 1N

is there a design choice which made the return value a BigInt and not an Int?

@michaellindon

user=> (type (/ 1 3))
clojure.lang.Ratio

and Ratio is defined in terms of BigInt https://github.com/clojure/clojure/blob/f572a60262852af68cdb561784a517143a5847cf/src/jvm/clojure/lang/Ratio.java#L23

ah ok, thank you

@joelsanchez

#_(this s-exp will be entirely ignored)

@alex-glv yes, someone told me about that macro, thanks anyway

Didn't realise where my scrollbar was at the time, too early 😉

Any book (or blog) recommendation, specifically for clojure concurrency? I've spent some time today wrangling atoms, agents, add-watch and promises, for managing a work queue. It has been a nice drill. I'm undecided what's the best queue management technique and might suspect clojure.async or even some Java implementation might be more performant/safe.

Maybe someone's written a book I'm not seeing about that?

🙂

@gmercer Thanks. That second did the trick.

one of these two expressions looks wrong to me:

user>  (* 2.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001M 1.0)
2.0
user>  (* 2.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001M 1.0M)
2.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010M

are bigdecimals special in being coerced to a less precise type in * ?

user=> (.doubleValue (BigDecimal. "2.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"))
2.0
user=> 

basically, fixed precision floating point numbers are dirty and their dirtiness will contaminate any math they are used in

1.0 in that expression is a fixed precision float (a jvm double), and that is what causes weirdness

clojure, encoded as a big honking class file, has rules about how you convert types to do math operations, and doubles as a type dominate pretty much everything, meaning if you have doubles in your expression, internally numbers that are not doubles are going to be turned in to doubles when doing math

so long as it's done on purpose i guess