core.async question: I want to consume a progress channel returned by cljs.http. is it possible to “consume” a channel by passing a transducer that just returns nil? or will the channel fill up and stop?

a transducer isn't a process, it has to be used somewhere - if you create the channel that cljs.http uses you can put a mapcat channel that drops all input (thus preventing it backing up), but otherwise you need to make something that consumes that channel

so like if I do (chan 10 (map println)), the nils returned by println will effectively prevent anything from going into the channel?

right now if i do that i’m getting tons of weird errors but i can’t figure out where they are coming from

I'd expect the nils to cause an error

(mapcat println) on the other hand I'd expect to work

(consuming all input, never putting a value on the output side though)

is there a way to add quotes to unquoted JSON?

like {test:true} string

to get {"test":true}

String… to parse it later.

@noisesmith i’ll be. that totally works and i have no idea why.

@lee.justin.m mapcat effectively throws away nils

putting nil on a channel (which (map println) would do) is an error

oh i see

clever

thanks!

=> (mapcat println [1 2 3])
1
2
3
()

(that prints the numbers, and returns ())

mapcat is much more general than it looks at first glance - it lets you return 0 or more output items for each item in an input, as a flat list

i think mapcat ends up being the answer to about 40% of the questions in here 🙂

=> (mapcat #(repeat % %) (range 10))
(1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9)

for is better

user=> (for [i (range 10) ii (repeat i i)] ii)
(1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9)
user=> 

but for doesn't have a transducer - it's definitely very powerful though

give me for, and I can easily write all the other seq functions give me all the seq functions but for, and I'd probably implment for incorrectly

just connected cider to a remote repl over ssh for the first time

mind kinda blown right now

it’s a good feel 😄 network REPLs and network editing are my favorite tools ever.

@minikomi it's pretty awesome, last year I spent most of every day working from Denver on a server in the UK. Too much data to pull across the pond, but a REPL over a VPN worked just fine. Cursive was even able to do things like auto-update code on the remote end and keep my local copy and the remote REPL in sync.

I remember thinking once "wait...I'm on a WiFi at a auto shop, connected via a VPN to a server in the UK where I'm then SSH'd into a box in a datacenter running a repl, and I'm seeing no lag at all". Computers are awesome.

wow -- how did cursive achieve that? when you save locally.. it evaluated it on the repl side too?

just being able to C-x C-e expressions locally and have them eval on the server is amazing to me lol

just a hunch - load-string under the hood, or just straight up sending parsed forms via nrepl protocol

@minikomi it has local file tracking, and automatic NS updating. So yeah when it detects that your local file has changed it loads it into the REPL, but it first loads any deps of that NS in the repl.

90% of the time it works every time 😉

It might also have some git integration? IDK, it's worked better than I ever thought it would though

haha

better not to think about it while it works

Just Works (on the other side of the world) (TM)

@noisesmith It uses the nREPL load-file op, which takes the whole file contents. IIRC that creates a file and then uses load-file, perhaps because load-string doesn’t do line numbers correctly? I think that’s right, but it’s been a while.

ahh, cool

The loading just works out which files have changed using the timestamps, calculates the transitive closure of the deps and then sends the files in order (with some trickiness because files and namespaces don’t map 1-1)

I’m planning some more cleverness to ensure that implementations of multimethods and protocols get reloaded too, but that’s still at the pipe dream stage for now.

I need a functioning clojure webapp example in the next 20 min to use as an example for a video course, should I use luminus (i'd really like to get to sleep soon)

Does anybody use toucan here? How can I select multiple rows with IN condition like SELECT * FROM table WHERE id IN (1, 2)

@hawari.rahman17 @camsaul in #toucan is a vrey knowlegable about toucan (we wrote it)

It seems that channel have been quiet for quite some time, I've asked another question there in the past, but there's no one present there to answer.

i suspect they are asleep at the moment

let me look at my examples and see if I can find one os usein IN

@tanzoniteblack is also worth asking

wow, thank you very much @arthur

in general https://github.com/metabase/metabase is the place to look for examples of using toucan properly

Thanks, I'll try to look for it there

I've kinda able to make it work using the query wrapper for honeysql, but I was looking for something more simpler.

her's an example https://github.com/metabase/metabase/blob/master/src/metabase/api/activity.clj#L43

(db/select-ids 'Card,      :id [:in ids])                 
(db/select-ids 'Dashboard, :id [:in ids])                 
(db/select-ids 'Metric,    :id [:in ids], :is_active true)
(db/select-ids 'Pulse,     :id [:in ids])                 
(db/select-ids 'Segment,   :id [:in ids], :is_active true)

@hawari.rahman17 ^

Ah that's just fantastic, thank you very much @arthur

Howdy all! So.. I want to “decorate” entries in a collection with additional data that is calculated by comparing the entry to the previous entry (with the same id). Basically a “stateful map” I guess. Here’s something that works - but looks very clunky: https://gist.github.com/andersenleo/c169b4a5dda59be12e7005f657bd6849 Any advice on how to improve that (maybe there’s a built-in way to handle these cases?). I should say that the “real” implementation will work off a core.async-channel so a normal reduce won’t cut it.

@gnejs how many ids do you expect on the channel?

@schmee +100 - and they will change over time. The id’s represents running tasks in a Mesos cluster.

(since they’ll change over time I’ll probably have use some sort of evicting cache in the final code - otherwise that state will slowly eat more and more memory)

(the elements on the channel represents snapshots of resource-usage for tasks, and the reason I need to do this acrobatic is that the CPU usage needs two samples to be calculated)

if you want to do this streaming on a channel, the only way I can think of is to use a stateful transducer

yeah, the idea is to put that make-running-diff-map into the channel as a transducer… (chan 1 (map (make-running-diff-map)))

@gnejs have you seen https://github.com/cgrand/xforms? there's a reduce transducer that might help

@sundarj ah, yes - I’ve seen that one before but didn’t consider it for this scenario. Thanks for the link - will look into it :thumbsup:

@gnejs @sundarj something like

(sequence
  (comp
    (x/by-key :id
      (x/reductions (fn [{pvalue :value} {:keys [value] :as m}]
                      (cond-> m
                        pvalue (assoc :diff (- value pvalue)))) nil))
    x/vals
    (remove nil?))
  entries)

is there a reason why juxt is not defined for arity 0 ?

it could reasonably return (constantly [])

@leonoel open an enhancement request ticket :)

Thanks @cgrand :thumbsup:

@gnejs another lazy option

(letfn [(rdiff [xs seen]
          (lazy-seq
            ((fn [[{:keys [id value] :as x} :as xs] seen]
               (when-let [s (seq xs)]
                 (cons (assoc x :diff (if (seen id) (- value (seen id)) 0))
                       (rdiff (rest s) (assoc seen id value)))))
             xs seen)))]
  (rdiff entries {}))

Thanks @reborg - In my case I don’t have access to the full seq - I’ll take entry by entry of a core.async channel.

Hi!

Am I correct in assuming that I have to hint the Clojure compiler at variables on which I intend to alert-var-root in order for direct linking not to prevent it from having any effect?

If so: how do I do the equivalent of a :redef hint for vars contained in 3rd-party libraries?

Hello guys

Wait, the variable I'm looking at is already :dynamic hinted. So it's something else I'm running against, please disregard

suppose i have this structure {"country1" {"city1" {:lat 1 :lon 2} "city2" {:lat 3 :lat 4}} "country2" {....} "country3" {.....} .....} what is the efficient solution to query on city ?

query on country is direct

if this hash map big one, what is the best way to query on city without know anything about country

i have tried specter but it's slow here

@abdullahibra If your workload is more read-heavy than write-heavy and cardinality allows it, I'd go for a structure that is both country and city indexed, at the cost of duplicating data. i.e:

{:by-country {:country1 {:city1 :a :city2 :b} :country2 {:city3 :c}}
 :by-city {:city1 {:country :country1 :data :a}
               :city2 {:country :country1 :data :b}
               :city3 {:country :country2 :data :c}}}

@pyr city names may have duplicates

@abdullahibra then in the by-city index, you have to return sets

but can’t you have duplicates even inside one country?

A colleague asks if fireplace is still the go-to solution for VIM integration or if something else needs to be considered

@abdullahibra why do you say specter is slow for this use case?

for that data structure this is how to handle it with specter:

(defn ^:direct-nav city-nav [city] (path MAP-VALS (must city)))
(select (city-nav "city1") data)

@nathanmarz what have i meant here not specter itself but the slowness because the data is very big, not problem with specter

@abdullahibra look at clojure.set/index

@abdullahibra indexing by city

=> (x/into {}
     (comp
       (x/by-key cat)
       (x/for [[country [city data]] %]
         [city [country data]])
       (x/by-key (x/into {})))
      {“italy” {“rome” {:lat 7 :lon 8}} “us” {“rome” {:lat 5 :lon 6} “paris” {:lat 1 :lon 2}}
       “france” {“paris” {:lat 3 :lon 4}}})
{“rome” {“italy” {:lat 7, :lon 8}, “us” {:lat 5, :lon 6}}, “paris” {“us” {:lat 1, :lon 2}, “france” {:lat 3, :lon 4}}}

that's really amazing answers

thanks the most friendly community 🙂

Installing lein on a coworker OSX machine - is there a way to specify to use lein 1.7.1? (for the https issue)

@nha edit ~/bin/lein

at the top there is export LEIN_VERSION="2.7.1" (actual version may vary :-))

there’s also installing lein, then running lein upgrade 2.7.1

ah thanks

(I bet that does about the same thing under the hood, but it seems a tad more elegant)

Thanks 🙂

I’m trying to use Virgil to reload some Java code, but as soon as the code reloads, I can’t def vars in the REPL anymore:

user=> (do (def c 1) (class c))
#<Class@6f5f4ee2 clojure.lang.Var$Unbound>

has anyone had the same issue?

how to get current leiningen profiles?

I tried setting each profile as a system property using profile-specific :jvm-opts but it didn't work

(`:jvm-opts ["-Dprofile=dev"]`)

well there’s lein show-profiles, or if you want to see what your current project definition looks like there’s lein-pprint or https://github.com/greglook/lein-cprint

or did you mean inside the running process?

inside the process, I need to do things depending on the current profile

I’m not sure there’s a way unless you set :eval-in-leiningen true in your project, which would ensure you’re running in the same JVM as lein is.

At which point you can do whatever introspection you want into the leiningen namespaces.

i knew about that option for developing lein plugins but did not think about using it in projects

that could mean...I can use cprop inside project.clj

mindblown

nah it doesn't work 😂 still good to know

I would avoid doing things in code that result in a project that can’t run without leiningen

yeah, that too... : P

it’s nice to be able to wrap up an uberjar and just run that with java, needing leiningen specific constructs to configure the code breaks that

my app uses a default that makes sense during development (environment profile is “develop”) and then the staging and prod processes explicitly set a different value via environment variables

if you control deployment environment that’s a pretty simple solution, if you expect an end user to run a jar you might prefer setting the environment for local dev and defaulting to prod values

basically I wanted to just execute lein uberjar without any env var setup...but still know in the code that this is a prod env

that's why I wanted to get the curr profile

found this https://github.com/athos/lein-with-env-vars

IMHO your compilation process should be totally agnostic of runtime

the same uberjar should be able to run on local dev, staging, or prod

it's not that it is unable to...but in production I want to use a dockerized chrome-headless, and not in dev

could just do it with a config option though...

otherwise you get nonsense where the jar that passed testing on staging isn’t even the same jar that goes on prod which feels a bit mad?

For some reason I'm finding that nippy (https://github.com/ptaoussanis/nippy) is working out of the box freezing/thawing jodatime intervals. Is there a reason I'm missing that this is working? I was expecting to have to do custom freeze/thaw functions

K i will just do it with config

@jjttjj a quick glance shows that it knows how to use anything that implements Serializable https://github.com/ptaoussanis/nippy/blob/master/src/taoensso/nippy/utils.clj#L135

@noisesmith cool thanks!

I’m not sure if every code path allows that, but it at least has some allowance if I read that correctly

I'm coming from Python to Clojure. I'm wondering if there are best practices for helping keep data types and function associated? If I have a few different conventions for shape of map to contain data, is the best practice simply to keep all functions that work on that particular shape of map in a namespace and to do one namespace for each type/convention of map shape?

Maybe this is excessively pedantic, but the best thing is not to write functions so that they are tied to a specific data type. If possible code to interfaces not types. If possible code generally enough that your code works even if a hash-map has unexpected keys or metadata. Avoid specialized code that only works on one specific type as much as you can.

but, for example, using spec you can define specific properties of a piece of data under a given key, and also say that your function takes data in that shape

@jjttjj beware nippy, we’ve been bitten by it several times 😕

if you really want data types with associated functions (sometimes this is actually called for) defrecord defines the two together

@greg316 in what way?

automagically encoding any object with serializable makes it look like it’s working when you implement it, but if your classes have changed at all (say, you upgraded your dependencies) you’ll go to deserialize old data and it will fail in mysterious ways

which eventually boil down to, “oh right, Serializable is a thing”

another time we upgraded nippy from one minor version to another, and could no longer read any of our stored data

finally, if you care about interop with any other languages, you can’t read nippy in anything but clojure, because it’s a custom serialization format

sounds like a bunch of reasons to prefer transit :P

honestly just using EDN and maybe compressing it would be better from an operational cost standpoint

but transit improves on that significantly (performance and reusability)

sorry, I was finishing my “why not nippy” thoughts

we do use transit+json for communication between our front-end and APIs

right, that’s fair

yeah - I’ve used transit for years, over many clojure and library version upgrades and it’s never given us problems

there were errors in cljs with stale js files that showed up as bizarre transit errors, but transit wasn’t the problem

@greg316 thanks, always good to hear opinions from people who’s used a lib in anger

I do have to credit nippy for making me look for better binary formats, and eventually write a CBOR implementation 😄

why was transit invented and not edn encoding/decoding from/to json?

that’s pretty much what transit is

yeah json encoding/decoding is a subset of transit right?

transit provides a much richer (and extensible) set of types than json

and adds caching and other features so it’s also faster, particularly for the cases where you pass maps with the same keys all the time

http://blog.cognitect.com/blog/2014/7/22/transit for many more words :)

another good thing with transit is that unlike edn the serializers / deserializers are a first class argument and not a global compiler state

@jtth as I understand, encoding/decoding to json is for performance

which makes using a specific set of tags for specific types much handier

frustration with global data-readers and extending print-method were part of what led me to write puget 🙂

for example I wrote a library that knows how to preserve and resuscitate atoms via a custom tag, and I don’t have to worry about messing up other people’s stuff by including it

(repost) Has anyone seen "This ResultSet is closed" errors from heroku (or anywhere else)? I am using jetty with with-db-connection. I have one instance that works and other (more recent) instances that fail on updates, all instances are funning identical code. https://github.com/jrootham/voting-server

The particular (failing) path I have put printlns into is updateUser

thanks @noisesmith , other than Protocols, Records etc. would a specific set/depth of keys be considered an interface. Would it be typical to have a (get-config-data db-map) as an interface, or would standard keys in a map be considered an interface? app-db always has :config for config data etc.?

when I mention coding against an interface rather than a data structure I mean the Interface (a vm level construct) that some data structure implements (and you could say that by specifying specific keys you are requiring the clojure.lang.IAssociative interface (this is needed in order to have keys) plus some further specification about the keys found (which spec can describe))

@jrootham the most common reason of connection closed is when you don't fully consume a resultset prior to closing the db connection

(with-whatever [conn db]
...
  (map something result-set))

map is lazy

If you change to mapv or anything eager the problem will go away

In this case the failure is when I am doing an update! My understanding is that update! is not lazy.

maybe gist a specific code example? The maintainer of java.jdbc hangs out here

I did have that problem in other parts of my code. I added doall to fix it. They work now.

Do you think that would help, given that the same code works or fails depending on the heroku instance, and it looks like it fails for recently created heroku instances.

I am thinking this is a heroku specific problem (to be classic: it works on my machine). This may well not be the best forum to ask about this.

It’s been a long day. Is there a better way to say (fn [a b] b) ?

actually now that I think about it, what I really want is more like or

nvm! rubber duck.

just reading the nippy vs transit discussion from earlier. I recently chose nippy over transit for encoding some long-lived data because of this warning on the transit page “while Transit is a great option for transferring data between applications, it should not yet be used for storing data durably over time”. but now I’m worried 😉 I would actually prefer transit, but not sure how much of a risk it would be?

I’ve found that the cognitect team is very conservative about backward compatibility, and I’ve heard that transit is unlikely to break compatibility any time soon

that’s good to know. I believe transit is used for comms in datomic, so I imagine it would be difficult for Cognitect to make a breaking change