Clojurians Log v2

@cmowforth has joined the channel

@madhuparna has joined the channel

Hi, could you point me some http/2/streaming clojure client?

We are trying to add clojure.spec and metosin/spec-tools to our codebase, and it looks like clojure 1.9 is a prerequisite. We tried upgrading to 1.9-alpha17, but currently facing a bunch of problems related to the upgrade. Our app currently does not start, with errors similar to the followingZ

Caused by: clojure.lang.ExceptionInfo: Call to clojure.core/defn- did not conform to spec:
In: [0] val: clj-tuple/conj-tuple fails spec: :clojure.core.specs.alpha/defn-args at: [:args :name] predicate: simple-symbol?

there's no way to turn off spec checks on macroexpansion

your only option is to upgrade to versions of the libraries that include the fixes you need

hi - is there a way to get all resources on the classpath - whether they be inside of jars of on the filesystem?

@jonpither I’m sure there must be but I don’t know what it is. eg. iirc Spring does a bunch of classpath scanning

@bronsa thanks

Hi! I wrote an article about PostgreSQL to Datomic migration. Hope you’ll find it useful. http://grishaev.me/en/pg-to-datomic

does the describe of a spec always have a similar structure to the data it specifies ? Can you give a counterexample ?

@octo221 I am not overly familiar with clojure specs, but map specs are represented as vectors I think

well, as a map with a :req vector

yes,

(keys :req etc)

I'm trying to think of a case where you describe a nested structure of collections by a flat spec

@xilix has joined the channel

@sl has joined the channel

@jrahme has joined the channel

Double/POSITIVE_INFINITY

or

Number.POSITIVE_INFINITY

!! thanks 🙂

hey guys

what's you're preferred method of checking for nils before assignation ?

@joshkh see also https://dev.clojure.org/jira/browse/CLJ-1074

i just realized I am repeating (:username respone) when I (if-not (nil? (:username response) (:username response) fallback-value))

isn't there a syntactic sugar around this ?

(or maybe-nil-thing default-value)

assuming you also want to exclude false

(assuming that)

ok

(-> good sounds)

@avabinary I end up writing functions like (?apply pred f val & args) for these purposes. If the predicate fails, then pass the value along

but

(if (nil? maybe-nil-thing) default-value maybe-nil-thing)

is clear

if you specifically want to exclude nil and not false, there's if-some

(if-some [u (:username foo)] u default)

there-s also (some->

if-let

which short circuits when nil

@avabinary I usually use some? when checking for nil. https://clojuredocs.org/clojure.core/some_q

I think points for brevity go to (or value fall-back)

but yeah, some? is nice

There is always when-some and if-some as well if you need to bind the value or branch accordingly when nil

@eistre has joined the channel

@soulhacker has joined the channel

@avabinary For your particular case, (:username response fallback-value) will work as you want.

not if username is set but nil

ah, yes

true ! good catch

this might be a dumb question but how do i get org.clojure/algo.graph on lein?

i have [org.clojure/algo.graph "0.1.0"]

the repl complains that it doesn't exist in clojars or maven

i believe the github page https://github.com/clojure/algo.graph is out of date so i looked inside clojure contrib and it seems the namespace is actually clojure.contrib.graph here https://github.com/richhickey/clojure-contrib though the project name is 'algo.graph' from here https://github.com/clojure/algo.graph/blob/master/project.clj

but unfortunately that isn't working for me either

@bcbradley The clojure.contrib stuff was for Clojure 1.2. It was discontinued when Clojure 1.3 came out.

clojure.algo.graph is as recent as it gets in the "modern contrib" libs. Let me check Maven Central for a version.

Confirmed -- no releases have been made to Maven Central. I'll bring it up in the dev channel...

@bcbradley I haven’t done enough graphs work to be sure this is related, but I remembered seeing a recent announcement about https://github.com/Engelberg/ubergraph, perhaps that might have some of what you’re looking for in algo.graph?

i considered loom and ubergraph, but i really don't need to pull in all of that

i really just need a topological sort, preferably one that partitions appropriately into sets

Fair enough.

right now i'm just ripping out algo's source

if sean comes back with any info on it being added to maven

https://gist.github.com/hiredman/075b45eaeb01e4b526ce6f8854685487#file-ur-component-clj-L1-L13

i'll depend on that instead

@bcbradley toposort? https://github.com/stuartsierra/dependency 🎉

thanks for the link i haven't' seen this before

thats not exactly what i wanted but its nice to keep in mind in case i need transitive dependencies

Hi! I have two “beginner” questions on clojure.spec: (1) would it make sense to use clojure.spec to validate input API payloads? (2) in that case, how would you work around the restriction that clojure.spec‘s map validation works by using namespaced keywords for key&spec names?

@moogey has joined the channel

another toposort @bcbradley - can't remember where it was originally homed - https://github.com/employeerepublic/deferst/blob/master/src/deferst/kahn.cljc

Hi. I am getting the follow error when trying to run lein repl with Clojure 1.9.0-alpha17. Could anyone please tell me what’s going on? https://gist.github.com/hmaurer/47dfa107025f51ec71966170c537a578

> Caused by: java.lang.IllegalStateException: Attempting to call unbound fn: #‘clojure.future/ident?

oh - here’s a fun one @bcbradley - stuartsierra/component comes with a toposort (used for figuring out which order to start components in)

oh wait someone mentioned that already

I fixed the issue by reverting to Clojure 1.8.0 and adding [clojure-future-spec "1.9.0-alpha17"] in my dependencies, which is what lacinia specified as a dependency (a library I rely upon). However I would still like to know exactly what the issue was if possible 🙂

at a certain point spec was taken out of the clojure 1.9 alphas so that it could be updated independently of the clojure code

code that was made for a different 1.9 alpha version prior to the split, won’t have that dep

so if you are using the newer alpha17 or later, you need to provide that dep

Oh I see. How are deps resolved by Lein by the way? If a library I rely upon has for dependency a different version of a library that I also use?

you can see the actual versions used via lein deps :tree

Will it just pick the most recent version? Or both?

eariler in the list beats later in the list

oh, so it flattens all dependencies and sub-dependencies in a list, and the first one wins?

also you can use :exclusions to say “don’t use this dep to get this other dep” and avoid versions it asks for

@hmaurer it’s more complicated than that - it’s an ordered tree

lein deps :tree is awesome, thank you 🙂

but it does use the first one it finds

how does the ordered tree behaviour differs from what I was saying, if you don’t mind elaborating?

because it’s not flattened anywhere afaik - it traverses the tree and adds items to a set of deps if you don’t have a version already

oh right, ok, but the behaviour is similar

I guess conceptually that difference doesn’t matter - it’s as if you flattened and took the first you found

Will lein report if dependencies conflict? e.g. if I try to use a version of a dependency that introduced breaking changes, and a library I later require uses an old version of that dependency?

no, lein will not complain, but lein deps :tree will report the conflict and tell you which it picked

great, thank you 🙂 that clarifies some things

you can use lein pedantic (a plugin) to make it bail out with an error https://github.com/xeqi/lein-pedantic

oh wait that’s depricated and lein has the pedantic feature… never mind!

@hmaurer see also lein help deps

Is there a way to create a 'struct' that IS CONSTRUCTABLE ... but is NOT UPDATEABLE. {} does not wok since it's updateable via assoc Is there some way via deftype / defrecord to create something where (1) I have a constructor and (2) it can't be updated via assoc (or if it is updated, it becomes a plain map and is no longer a type / record)

sounds like you want deftype - you can’t assoc or conj etc. a deftype (unless you extend to those protocols of course)

user> (deftype Trial [a b])
user.Trial
user> (def t (Trial. 1 2))
#'user/t
user> t
#object[user.Trial 0x5e0c24f4 "user.Trial@5e0c24f4"]
user> (assoc t :a 2)
ClassCastException user.Trial cannot be cast to clojure.lang.Associative  clojure.lang.RT.assoc (RT.java:792)
user> 

from the deftype page in the clojure docs: deftype provides no functionality not specified by the user, other than a constructor

qqq: {} is not updateable

persistent data structure

sorry, by 'update', I meant create new obj with same 'type' as old obj

I'm concerned with 'invaild objects'

that claim to satissy certain constraints, but in reality do not

due to assoc

@dpsutton: defrecord doesn't does not work, as assoc 'prserves the record type' ; with deftype, how do I read the 'a' and 'b' ?

user> (.a t)
1

> The XY problem is asking about your attempted solution rather than your actual problem

@dpsutton: this was not part of the original request ... but is there a way to have this also work with cljs + optimizations advanced

@ghadi : I agree that my question was poorly formulated.

invalid data is a gating problem -- you validate at the edge of a subsystem, then you operate with the data

what was not part of the original request?

you asked how do you read the fields of the deftype

I wnat field acccess to also work via cljs optimiazationsa dvanced

(.a t) I suspect, will not work as cljs optimizatinos renames the .a

IMHO this is not a data structure problem / impl detail but a program design problem

it would rename your accesses too if it did that - that’s kind of why advanced compilation ever works…

or would it? now I’m unsure

@noisesmith : optimizationsa-dvanced has me so scared I jus use (aget ... "field-name") everywhere

also if you suspect that something is generating invalid data, that is something clojure.spec + instrument is really good at catching

but if you're scared of bad data getting into your data i think you're working on the wrong side

here's the thing; I'm dynamically checking some type constraints

but if I ask that question, ppl tell me to use spec or core.typed 🙂

so what happens if the types don't align?

and who would introduce the bad data

throw an assertion

bad data introduced to my programming bugs

@qqq

(ins)dev:cljs.user=> (deftype Foo [a b])
cljs.user/Foo
(ins)dev:cljs.user=> (aget (Foo. 1 2) "b")
2

@noisesmith : nice; thanks!

well that looks dangerous

that looks like "b" may not track what the fields get renamed to in aggressive compilation

i still don't understand why you can't filter bad data at the edge

@ghadi: the problem is not bad user input

just spitballing: system that isn’t designed with the right edges in place?

the problem is that I, the programmer, introduce bugs

these are bugs that a static type checker would catch at compile time

but in clojure, the next best thing I can do is to catch them at runtime

the input data is fine; it's internal functions that manip the data that violate type constraints

which of these two implementations would you prefer and why?

(defn inversion [nodes edges]
  (transduce
    (mapcat (fn [[k v]] (map #(vector % k) v)))
    (completing (fn [acc [k v]] (update acc k conj v)))
    (zipmap nodes (repeat #{}))
    edges))

(defn inversion [nodes edges]
  (reduce-kv
    (fn [acc k v] (reduce (fn [acc i] (update acc i conj k)) acc v))
    (zipmap nodes (repeat #{}))
    edges))

{:framebuffer-0 #{:texture-0 :texture-1 :program-0 :framebuffer-1}
 :framebuffer-1 #{:framebuffer-2 :texture-2 :texture-3 :program-1}
 :framebuffer-2 #{:texture-0 :program-1}}

{:framebuffer-0 #{}
 :framebuffer-1 #{:framebuffer-0}
 :framebuffer-2 #{:framebuffer-1}
 :texture-0 #{:framebuffer-2 :framebuffer-0}
 :texture-1 #{:framebuffer-0}
 :texture-2 #{:framebuffer-1}
 :texture-3 #{:framebuffer-1}
 :program-0 #{:framebuffer-0}
 :program-1 #{:framebuffer-1 :framebuffer-2}}

@qqq I’d think accidentally calling an accessor and constructor would be a lot less likely than accidentally calling assoc on the wrong object

@noisesmith : I can make the constructor private, then have a function (which calls the constructor) do checks beforehand

@qqq For accessing object properties (as opposed to array elements), consider goog.object/get instead of aget.

@qqq creating custom types that have their own field accessors completely negates the value of generic data access from the clojure std library

your code will essentially be a non-reusable DSL

I firmly believe you are going down the wrong path, and that the advice you're getting about specific implementation is misguided

you should test against invalid / incorrect data (even from your own code). Have you considered writing generative tests for your datastructures? (either from test.check or from spec)

test against bad data while allowing bad data to exist

Lots of things that are common in other languages (e.g. privileged data) are the exact wrong approach in Clojure.

when I say 'generic data access' here is an example of what I mean:

(get-in order [:line-items 2 :product :picture :url])

(-> order
  (get-in [:line-items 2 :product])
  (.-picture)
  (get :url))

You will infect the codebase with specificity

and negate a whole lot of the benefits of "the clojure way"

is it possible to tell if you’re already in a clojure.core.async/go block inside a defmacro?

you could call <!

the code of that is literally throw an error

the go macro totally rewrites everything in it. so if that throws an error then you are not in a go block. otherwise have it take from a channel that has a single value ready to give

super hacky

our devs went a bit bonkers with nested gos

the go macro actually macro expands everything before it does its thing, and the way it macro expands has some differences from the way the compiler does it

so you could try observing differences in the &env map

tries to think of something else that might be helpful there

really, if you are in that place, I would just write everything to assume it is being run from a go block in the async threadpool

it seems like an assert-go macro would be good, or even assert-not-go, but code that changes behavior in/out of go seems like it’s asking for maintenance issues and subtle bugs

@csm what problem are you trying to solve?

it’s stuff like “inject tracking of metrics in async calls”

in retrospect, it probably should have just been a fn instead

And a few mistakes/misunderstandings (like “wrap in a go call to run this in a different thread”)

😞

it only became clear after updating core.async to a newer version, since we started getting method too large errors

So unless you're mixing core.match with core.async you shouldn't hit those problems often.

I mean 65k instructions is fairly large for a method, you might just be able to pull a few functions out the body of the go and reduce the method size a bit.

i'm not able to get that hacky way to work. tried inlining the function, not sure

it might need to be a macro for the async compiler to do its thing?

does anyone know if anyone has tried making futures as efficient as goroutines?

as far as i know futures spin up another thread, but I don't think they are pooled

maybe someone has their own implementation of futures?

i'm only asking because I really dislike core async's approach to concurrency (not that it is worse than most of the alternatives)

after thinking about it, it seems like the urge to “hack how this macro works to handle go” would better be served by “rewrite some stuff as fns, and split things up a little”

@bcbradley futures are pooled, the fundamental thing is that threads are OS level and OS level context switching can’t be as fast as in-process context switches

ah well i'm misinformed about the whole kitnkaboodle then

do you think it would be possible to implement futures using in process context switching rather than os level switching?

then you require either 1) an interrupt and pre-emptive context switch or 2) cooperative multitasking

1 will perform a lot worse - probably as bad as OS task switches, and introduce massive complexity

2 is core.async 😄

Yeah, and futures aren't slow at all, they're just memory heavy

@bcbradley have you looked at agents?

i though agents were pooled, and thats the reason i figured futures weren't

probably shouldn't assume though xd

futures will re-use threads though

well thats reassuring

futures use the agent send-off pool https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Agent.java#L53 https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L6838

i didn't know that, thanks for that

They use a Java Executors.newCachedThreadPool, which means that once created the threads stick around awhile incase they need to be re-used. In my experience "awhile" is something around a minute.

but that's undefined

i don't want to start ranting, but the reason i dislike core async so much is because it is pretty much just imperative programming

i'm not suggesting hardware threads plus semaphore hell is better

Nope, I agree with you 🙂 and I ranted on that subject at some length at Clojure/West

i feel like futures + promises are the cleanest concurrency mechanism anyone has come up with so far

@tbaldridge btw thanks for that talk, I often find myself linking the youtube for people who are clearly confused about how to use core.async properly

its clean, simple-- and doesn't surprise you

But the answer is almost always: move the async, non-deterministic, bits to the edge of your system

from the most recent clojure/west?

rich hickey elaborated on the problems of promises in a strange loop talk

ooh can you link me that?

@bcbradley I’ve still seen suprises eg. someone writes code that creates futures inside doseq - eventually they get a collection big enough that the VM goes nuts

@bcbradley https://www.infoq.com/presentations/clojure-core-async

thanks!

if they were doing channel ops inside a doseq, the reason for the failure would be more clear and the mitigation strategy would be more straightforward

i can pretty much sum up my stance on the matter with one simple comparison

core async and the go-lang tradition of concurrency involves imperative programming on stateful (chans are queues) entities

promises are simply unrealized values

clojure is a language of values, so i would prefer to use promises

and futures allow you to realize something concurrently

seems like a clean separation of responsibilities to me

Nothing stoping you from doing this: (<!! (thread ...))

@bcbradley having worked on a large system which was using futures inside idiomatic fp constructs as a pervasive pattern, give me statefull queues of some kind any day - at least queues give you a straightforward mechanism for flow control and backpressure

you might prefer agents with their implicit queue to core.async of course - but you’ll eventually need a queue of some type if you need concurrency

@bcbradley promises encourage a request/response style of code. This is fairly limited. As @noisesmith mentioned, CSP focuses on data flow, pipelines and queues. It's a "steps of processing" model vs a "call and wait for a response".

At least it should be, people use core.async as a request/response system, but it doesn't work out very well.

i feel as though steps of processing could be implemented clearly with just chains of promises

> people use core.async as a request/response system, but it doesn't work out very well

slowly raises hand and hangs head

i do think backpressure is an issue though

@potetm we've all done it 🙂 no condemnation from me.

is there a neat way to remove an element from a vector at index i

you could use a finger-tree instead of a vector, they are designed for such things https://github.com/clojure/data.finger-tree

thanks for the suggestion, ill take a look 🙂

@bcbradley chains of promises, composed with a promise-monad like https://github.com/funcool/cats/blob/master/src/cats/labs/manifold.clj or https://github.com/funcool/cats/blob/master/src/cats/labs/promise.cljc works brilliantly for us

@mccraigmccraig wow i'll have to put that in a tab and read through it

some sample code chaining promises this way - https://github.com/employeerepublic/er-cassandra/blob/master/src/er_cassandra/model/alia/unique_key.clj#L61

is core.match ok? it’s been on 0.3.0-alpha4 for the last 3 years

It's pretty stable and hasn't needed much maintenance I suspect.

spoke with dnolen at conj last year. the way the decision tree is made is fine in java but removes some optimizations in cljs since it uses exceptions for backtracking.

if i remember correctly, this was what he wanteded to implement a non-alpha release

Versions aren't much of an indicator in Clojure-land, and a contrib library that has gone without updates for a while doesn't indicate it's been abandoned either. Which, yeah, can make it hard to tell the difference between very-stable-ware and abandon-ware 😐

yeah

ok

if the official status is Don’t Worry About It, then all the better

i just really like it and was worried something bad had happened

Someone just pointed out that algo.graph never actually got a release on Maven (and it hasn't had an update in four years).

that’s bad, right?

trying to use clojure 1.9 from emacs with cider, but when I run cider I get

can you run a lein repl in this project without emacs?

@dottedmag has left the channel

this is a known issue with clojure 1.9 and old core.async

upgrade to a newer core.async and it goes away

@urbank ^ forgot to tag

@goran_zic has joined the channel

@michaellindon with specter:

(setval (nthpath 2) NONE [1 2 3 4 5])
;; => [1 2 4 5]

also works with lists

@noisesmith Hm... now to find out which dependency has the wrong core.async version...

you can specify your own core.async version and the deps will have to use it

or use lein deps :tree to see where it comes in

@nathanmarz Isn't that O(n) on the size of the new collection though? Why not just (into [] (filter-nth ...) input)

(that is, if your deps is earlier in the deps list it overrides)

@tbaldridge yea the underlying impl is O(n), but your snippet looks like O(n) as well

oh it is

but the whole idea behind conj and the like is that they are not O(n) on the size of the new collection

so I would flag both of our approaches in a code review for being a possible bottleneck

I don't think you can do better than O(n) for that task on a vector

that's why we suggested finger trees

performance and just being able to do the task elegantly are two separate things

sometimes you need to remove an element from the middle of a vector and it's not a bottleneck

unreasonable to require every operation to be O(1)

even then I'd be tempted to use take + drop to do the same thing

(apply concat ((juxt (partial take 3) (partial drop (inc 3))) [1 2 3 4 5 6]))

or split-at or whatever it's called

core.rrb-vector could be used aswell to keep performance sub linear

nthpath can encapsulate the optimal method

whatever it is

has log subvec & log vector cat

I haven't investigated the optimal implementation for this particular task, but whatever it is it can be encapsulated behind the abstraction

specter is optimal for most of its functionality

@noisesmith Thanks, that worked! What a relief! Nothing more annoying than debugging the dev environment when in a hurry 🙂

on that we'll agree to disagree

what are we disagreeing on?

I think nathan was talking about performance & tim about api?

yeah, I mean the problem is that specter in this case abstracts so much away that I really don't know what it's going to do

you don't know what it will do semantically or performance-wise?

performance wise, there's a underlying assumption in specter that it will do "the right thing" while maintaining the data type. Many times I don't care about the datatype, instead I care about performance.

so our definitions of "optimal" differ

the original questioner asked about removing an element from a vector, which sounds like he cared about maintaining the datatype

perhaps, but sadly I've run into too much code already that falls apart because someone did (nth ...) on a seq.

I really don't understand what you're asserting there

And what does "the same type" mean in the face of PersistentArrayMap? (I'm actually wondering on this one)

MAP-VALS or ALL on a PersistentArrayMap output another PersistentArrayMap

even one that's already above the threshold

which is surprisingly possible

and if I do something that grows it?

you can't grow it with those operations

whatabout ops that you can grow with

they'll convert to PersistentHashMap

maintaining the literal type isn't the goal of specter, rather to maintain a type with the same expected semantics

PersistentHashMap and PersistentArrayMap are implementation details

that specter maintains PersistentArrayMap on ALL and MAP-VALS is because that's the most performance optimal way to do the transformation

@krauser has joined the channel

I've used an array map to maintain ordering for writing out columnar data, a persistenhashmap would make my site more dynamic than i like.

yea there are isolated cases like that where it matters but not for 99% of use cases

for sure

i hate some of the subtleties of the clojure datatypes

like conj being beginning of a vector and end of a seq

or backwards sorry

I also hate some of the side effects of these datatypes, and that's why I've backed off a bit from specter, hiding all these complexities behind a abstraction is nice, but it leaks in performance. How fast is "remove nth"? Well it depends...

Same is true of recursive concat, conj, etc.

what do you mean "side effects of these datatypes"?

bad wording, sorry

"subtleties of these datatypes"

figured. but wanted to know what you meant

ah yeah

actually for most uses of specter it's extremely difficult to outperform it

So yeah, back to the original problem, I think we should educate users of Clojure to say: "What you are trying to do isn't supported natively by the datatype, you can fake it in these ways, but it's going to have performance problems with larger collections".

for most programmers I would say it's impossible because it requires too much internal knowledge of clojure

https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter/navs.cljc#L253

that's 60% faster than next best method for transforming every value of a small map

as for "remove nth", specter is probably not currently optimal but that's only because the work hasn't been put into it

the abstraction can be optimal

optimal given the input data type...that's the catch

how is that a catch?

it can run different code for different data types

remove-nth will always be O(n) on a vector. No way to improve that. However, by educating users as to how the underlying collections work, maybe the'll reach for a different more optimal datatype.

I 100% agree it's the programmer's responsibility to understand the data types they're using and the impacts of that, but that's completely orthogonal to specter

specter lets you manipulate your data way more elegantly, especially compound or recursive data, and in many cases with far better performance

I completely reject characterizing it like some magic library with performance "leaks"

I don't think @tbaldridge was blaming specter for the "leaks" -- but rather, unless you already have a mental model of how the nested datatype looks, you can have a single piece of Specter code that is (1) very fast for certai nstructures and (2) very slow for other structures, because some stuff are O(log n) or O(n) depending on the underlying datastructure

whatever the underlying types are, specter will do the operation in the fastest way

it's the responsibility of the programmer to choose the most appropriate types for their app

criticizing specter for a programmer choosing inappropriate types doesn't make sense

1. I've studied specter a bit, even tried to implement a mini one myself. 2. I don't think I could do a better job myself. 3. I think "leaky" here just means -- as a programmer, you have to keep track of the underlying data structures, i.e. it's "leaky" in that you can't ignore the underlying details; not "leaky" as in space/time leakage.

I think this is the standard definition of "leaky abstraction."

I wouldn't call that leaky

"leaky" more appropriately refers to details that you have to worry about that should be encapsulated

Quoting wikipedia: In software development, a leaky abstraction is an abstraction that exposes details and limitations of its underlying implementation to its users that should ideally be hidden away. But here, Spectre queries (or any other queries for that matter) are 'leaky' in that different data structures have different runtimes for different ops, and the programmer has to keep them in mind, so this isn't really abstracted away from the programmer. [I don't know a way to do this better.] [I think this 'leakiness' problem can not be solved -- i.e. any attempt to build a DSL that allows easy manip of heterogeneous datastructures will have to deal with this[]

Just to be clear, I don't know of a way to improve Specter -- I think it's hit local optima -- and this 'leakiness' is a fundeamtanl problem due to different data structures having different runtimes.

data structures are never a detail that should be hidden away

we're just quibbling over terminology, I think we agree on the underlying principle

To someone who expects specter code to be "what, not how" it is leaky because they have to consider underlying datastructures. To somehow who expects to always keep data structuresin mind, it's not leaky. 🙂

Let's argue over something else,

like ... where can i get a good set of exercises for learning how to write a nanopass compiler 🙂

I'm watching the 2013 clojure conj https://www.youtube.com/watch?v=Os7FE3J-U5Q talk ... and I really want to try this out.

we can agree on that 🙂

@qqq records for the AST, postwalk for the passes, run till fixpoint, about all there is too it

or hashmaps even for the ast, whatever you prefer

or you could get creative and event source a queue of characters and fold over it to generate a projection representing your compiled code

just kidding, don’t do that

depending on your source and target you can do it without an ast

if you say your target is a superset of your source, it is macroexpansion

(or pretty close)

true, but working with order dependant types is unpleasant

may be better with spec, but {:keys [fn-name body]}) is easier than [_ fn-name _ body]

sure

https://github.com/hiredman/qwerty "macroexpands" a lisp in to something like go in parens, then the emitter strips the parens so you can feed it to the go compiler

cool!

I dunno if it rates an exclamation point, it was fun to fiddle with it for a while, then I gave up on it

hmm, and if I write the passes as transducers, can I easily get a monolithic compiler out o fthis?

you can, but you'll quickly find that some passes need to be run more than once, or need to be run before/after other passes

tools.analyzer is a nano-pass compiler imo.

and in different traversal orders

yeah that too.

@bronsa does tools.analyzer.jvm still walk the AST backwards for locals clearing? Maybe I'm mis-remembering, but I thought that was cool the first time I saw it.

yeah

makes the algorithm much simpler than walking forward & collecting usage points

I did it that way just because I couldn't understand the forward algorithm implemented in Compiler.java TBH :)

I'm struggling to wrap my head around clj-oauth. All the examples use twitter, but I can't seem to get it to work with Google. Google and twitter's terminology doesn't seem to be the same, and it's confusing the heck out of me. Are there any examples using google's api I can look at?

I'm having trouble with core.async pub/sub. It seems like there must be something I'm not understanding. I can see that my system sometimes is publishing a lot of events really close together. Say 20 within 2 seconds. There are times that all but one of my subscribe loops (event listeners) aren't doing anything during this burst of activity. Then a while later (a couple of min sometimes) during another burst of publish activity the subscribe loops come to life and grab some data and push on to the subscriber channels.

I started out by not using any buffering on the channels. I'm not really clear when additional buffering makes sense or not

It really seems like some of my published events are being lost

the first thing to do is check that your topic-fn is returning values from the set of things you are subscribing to

when just using (chan) for pub should my source threads block if the subscribers are idle?

yea, I have lots of debug messages that all looks pretty good

how sure are you?

well I currently only have one topic

like, are we talking strings you know are byte for byte the same?

my topic is just a keyword

the next thing to log would be the identity (pr-str prints out the identity hash if I recall) of each thing involved

to make sure you are creating the pub/sub on the same channel you are publishing to, and you are subscribing to the same pub/sub you are publishing to

if the channel you are publishing to isn't being consumed for some reason, your publishes with block if there is no buffer, and will block once the buffer is full

if I recall, a pubsub will consume everything and just ignore messages it has no subscribers for

right, that's the thing, it appears that my publishers never block

so I would double check your topic-fn, make sure it returns what you think it does on the inputs to the channel

when no buffer is supplied to the channel, does that mean it will block after the first put, until the first take?

e.g. if your topic-fn is a keyword, and your messages are maps, calling a keyword on a map that doesn't contain it would just return nil

yes, that is how my events are structured just as maps

so invoke your topic-fn on one of the maps to see what it returns

that's the thing it all works fine when I do it by hand, things get weird once there is lots of simulatneous activity

based on my logging I can see all the topics that are published and those that are received by the listenr loops

how do you know your publishers don't block?

I say that just based on my theads that push out lots of messges saying that they're publishing.... hmm wait you might be on to something

my log message happens right before I push not after, I might be misinterpreting whats going on

push=publish

ok, made some changes to logging and restarting everyting,

so could you help clarify about when is it appropriate to specify a buffer in these sceanrios?

If things lock up under lots of activity, and you aren't buffering, and lots is < thousands, I'd double check for blocking ops in your go blocks.

yea I'm not into thousands of events, generally should be less than 50/sec