This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-02-07
Channels
- # aleph (1)
- # beginners (152)
- # cider (26)
- # clara (2)
- # cljs-dev (13)
- # cljsrn (5)
- # clojure (198)
- # clojure-greece (15)
- # clojure-italy (39)
- # clojure-sanfrancisco (3)
- # clojure-spec (28)
- # clojure-uk (16)
- # clojurescript (52)
- # community-development (15)
- # core-async (26)
- # cursive (42)
- # data-science (28)
- # datomic (19)
- # devops (7)
- # duct (11)
- # emacs (24)
- # fulcro (22)
- # garden (4)
- # leiningen (12)
- # luminus (1)
- # mount (5)
- # off-topic (106)
- # om (5)
- # onyx (10)
- # parinfer (37)
- # re-frame (17)
- # reagent (47)
- # shadow-cljs (36)
- # yada (2)
core.async question: I want to consume a progress channel returned by cljs.http. is it possible to “consume” a channel by passing a transducer that just returns nil? or will the channel fill up and stop?
a transducer isn't a process, it has to be used somewhere - if you create the channel that cljs.http uses you can put a mapcat channel that drops all input (thus preventing it backing up), but otherwise you need to make something that consumes that channel
so like if I do (chan 10 (map println))
, the nils returned by println
will effectively prevent anything from going into the channel?
right now if i do that i’m getting tons of weird errors but i can’t figure out where they are coming from
I'd expect the nils to cause an error
(mapcat println)
on the other hand I'd expect to work
(consuming all input, never putting a value on the output side though)
@noisesmith i’ll be. that totally works and i have no idea why.
@lee.justin.m mapcat effectively throws away nils
putting nil on a channel (which (map println) would do) is an error
=> (mapcat println [1 2 3])
1
2
3
()
(that prints the numbers, and returns ())
mapcat is much more general than it looks at first glance - it lets you return 0 or more output items for each item in an input, as a flat list
=> (mapcat #(repeat % %) (range 10))
(1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9)
user=> (for [i (range 10) ii (repeat i i)] ii)
(1 2 2 3 3 3 4 4 4 4 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9)
user=>
but for doesn't have a transducer - it's definitely very powerful though
give me for, and I can easily write all the other seq functions
give me all the seq functions but for, and I'd probably implment for
incorrectly
@minikomi it's pretty awesome, last year I spent most of every day working from Denver on a server in the UK. Too much data to pull across the pond, but a REPL over a VPN worked just fine. Cursive was even able to do things like auto-update code on the remote end and keep my local copy and the remote REPL in sync.
I remember thinking once "wait...I'm on a WiFi at a auto shop, connected via a VPN to a server in the UK where I'm then SSH'd into a box in a datacenter running a repl, and I'm seeing no lag at all". Computers are awesome.
wow -- how did cursive achieve that? when you save locally.. it evaluated it on the repl side too?
just being able to C-x C-e
expressions locally and have them eval on the server is amazing to me lol
just a hunch - load-string
under the hood, or just straight up sending parsed forms via nrepl protocol
@minikomi it has local file tracking, and automatic NS updating. So yeah when it detects that your local file has changed it loads it into the REPL, but it first loads any deps of that NS in the repl.
90% of the time it works every time 😉
It might also have some git integration? IDK, it's worked better than I ever thought it would though
@noisesmith It uses the nREPL load-file op, which takes the whole file contents. IIRC that creates a file and then uses load-file, perhaps because load-string doesn’t do line numbers correctly? I think that’s right, but it’s been a while.
ahh, cool
The loading just works out which files have changed using the timestamps, calculates the transitive closure of the deps and then sends the files in order (with some trickiness because files and namespaces don’t map 1-1)
I’m planning some more cleverness to ensure that implementations of multimethods and protocols get reloaded too, but that’s still at the pipe dream stage for now.
I need a functioning clojure webapp example in the next 20 min to use as an example for a video course, should I use luminus (i'd really like to get to sleep soon)
Does anybody use toucan
here? How can I select multiple rows with IN
condition like SELECT * FROM table WHERE id IN (1, 2)
@hawari.rahman17 @camsaul in #toucan is a vrey knowlegable about toucan (we wrote it)
It seems that channel have been quiet for quite some time, I've asked another question there in the past, but there's no one present there to answer.
@tanzoniteblack is also worth asking
in general https://github.com/metabase/metabase is the place to look for examples of using toucan properly
I've kinda able to make it work using the query
wrapper for honeysql, but I was looking for something more simpler.
her's an example https://github.com/metabase/metabase/blob/master/src/metabase/api/activity.clj#L43
(db/select-ids 'Card, :id [:in ids])
(db/select-ids 'Dashboard, :id [:in ids])
(db/select-ids 'Metric, :id [:in ids], :is_active true)
(db/select-ids 'Pulse, :id [:in ids])
(db/select-ids 'Segment, :id [:in ids], :is_active true)
Howdy all! So.. I want to “decorate” entries in a collection with additional data that is calculated by comparing the entry to the previous entry (with the same id). Basically a “stateful map” I guess. Here’s something that works - but looks very clunky: https://gist.github.com/andersenleo/c169b4a5dda59be12e7005f657bd6849
Any advice on how to improve that (maybe there’s a built-in way to handle these cases?). I should say that the “real” implementation will work off a core.async-channel so a normal reduce
won’t cut it.
@schmee +100 - and they will change over time. The id’s represents running tasks in a Mesos cluster.
(since they’ll change over time I’ll probably have use some sort of evicting cache in the final code - otherwise that state will slowly eat more and more memory)
(the elements on the channel represents snapshots of resource-usage for tasks, and the reason I need to do this acrobatic is that the CPU usage needs two samples to be calculated)
if you want to do this streaming on a channel, the only way I can think of is to use a stateful transducer
yeah, the idea is to put that make-running-diff-map
into the channel as a transducer… (chan 1 (map (make-running-diff-map)))
@gnejs have you seen https://github.com/cgrand/xforms? there's a reduce
transducer that might help
@sundarj ah, yes - I’ve seen that one before but didn’t consider it for this scenario. Thanks for the link - will look into it :thumbsup:
@gnejs @sundarj something like
(sequence
(comp
(x/by-key :id
(x/reductions (fn [{pvalue :value} {:keys [value] :as m}]
(cond-> m
pvalue (assoc :diff (- value pvalue)))) nil))
x/vals
(remove nil?))
entries)
@gnejs another lazy option
(letfn [(rdiff [xs seen]
(lazy-seq
((fn [[{:keys [id value] :as x} :as xs] seen]
(when-let [s (seq xs)]
(cons (assoc x :diff (if (seen id) (- value (seen id)) 0))
(rdiff (rest s) (assoc seen id value)))))
xs seen)))]
(rdiff entries {}))
Thanks @reborg - In my case I don’t have access to the full seq - I’ll take entry by entry of a core.async channel.
Interesting!
(letfn [(rdiff [c seen]
(lazy-seq
((fn [{:keys [id value] :as x} seen]
(when x
(cons (assoc x :diff (if (seen id) (- value (seen id)) 0))
(rdiff c (assoc seen id value)))))
(<!! c) seen)))]
(rdiff (to-chan entries) {}))
Here's a xducer version just in case:
(def xdiff
(fn [rf]
(let [seen (volatile! {})]
(fn
([] (rf))
([result] (rf result))
([result {:keys [id value] :as x}]
(let [diff (- value (get @seen id value))]
(vswap! seen assoc id value)
(rf result (assoc x :diff diff))))))))
;; (sequence xdiff entries)
Ah, lovely - that’s basically a more efficient version of “my” stateful mapping function (with the volatile!
stuff), right?
Am I correct in assuming that I have to hint the Clojure compiler at variables on which I intend to alert-var-root
in order for direct linking not to prevent it from having any effect?
If so: how do I do the equivalent of a :redef
hint for vars contained in 3rd-party libraries?
Hello guys
Wait, the variable I'm looking at is already :dynamic
hinted. So it's something else I'm running against, please disregard
suppose i have this structure {"country1" {"city1" {:lat 1 :lon 2} "city2" {:lat 3 :lat 4}} "country2" {....} "country3" {.....} .....} what is the efficient solution to query on city ?
query on country is direct
if this hash map big one, what is the best way to query on city without know anything about country
i have tried specter but it's slow here
assuming this:
(def data {:country1 [{:city1 :a}
{:city2 :b}]
:country2 [{:city3 :c}
{:city4 :d}]})
if every city ID is different, then you can do this:
(apply merge (apply concat (vals data)))
result:
{:city1 :a, :city2 :b, :city3 :c, :city4 :d}
if your data is:
(def data {:country1 {:city1 :a
:city2 :b}
:country2 {:city3 :c
:city4 :d}})
just do
(apply merge (vals data))
result
{:city1 :a, :city2 :b, :city3 :c, :city4 :d}
@abdullahibra If your workload is more read-heavy than write-heavy and cardinality allows it, I'd go for a structure that is both country and city indexed, at the cost of duplicating data. i.e:
{:by-country {:country1 {:city1 :a :city2 :b} :country2 {:city3 :c}}
:by-city {:city1 {:country :country1 :data :a}
:city2 {:country :country1 :data :b}
:city3 {:country :country2 :data :c}}}
This means that you guard access against this data structure by ensuring references are kept up to date.@pyr city names may have duplicates
@abdullahibra then in the by-city
index, you have to return sets
A colleague asks if fireplace is still the go-to solution for VIM integration or if something else needs to be considered
@abdullahibra why do you say specter is slow for this use case?
for that data structure this is how to handle it with specter:
(defn ^:direct-nav city-nav [city] (path MAP-VALS (must city)))
(select (city-nav "city1") data)
what is the performance difference of declaring the nav upfront like that and using (select [MAP-VALS (must city)] data)
directly?
extremely small
@nathanmarz what have i meant here not specter itself but the slowness because the data is very big, not problem with specter
@abdullahibra look at clojure.set/index
@abdullahibra indexing by city
=> (x/into {}
(comp
(x/by-key cat)
(x/for [[country [city data]] %]
[city [country data]])
(x/by-key (x/into {})))
{“italy” {“rome” {:lat 7 :lon 8}} “us” {“rome” {:lat 5 :lon 6} “paris” {:lat 1 :lon 2}}
“france” {“paris” {:lat 3 :lon 4}}})
{“rome” {“italy” {:lat 7, :lon 8}, “us” {:lat 5, :lon 6}}, “paris” {“us” {:lat 1, :lon 2}, “france” {:lat 3, :lon 4}}}
that's really amazing answers
thanks the most friendly community 🙂
Installing lein on a coworker OSX machine - is there a way to specify to use lein 1.7.1? (for the https issue)
there’s also installing lein, then running lein upgrade 2.7.1
(I bet that does about the same thing under the hood, but it seems a tad more elegant)
I’m trying to use Virgil to reload some Java code, but as soon as the code reloads, I can’t def vars in the REPL anymore:
user=> (do (def c 1) (class c))
#<Class@6f5f4ee2 clojure.lang.Var$Unbound>
how to get current leiningen profiles?
I tried setting each profile as a system property using profile-specific :jvm-opts
but it didn't work
(`:jvm-opts ["-Dprofile=dev"]`)
well there’s lein show-profiles
, or if you want to see what your current project definition looks like there’s lein-pprint or https://github.com/greglook/lein-cprint
inside the process, I need to do things depending on the current profile
I’m not sure there’s a way unless you set :eval-in-leiningen true
in your project, which would ensure you’re running in the same JVM as lein is.
At which point you can do whatever introspection you want into the leiningen namespaces.
i knew about that option for developing lein plugins but did not think about using it in projects
that could mean...I can use cprop inside project.clj
mindblown
nah it doesn't work 😂 still good to know
I would avoid doing things in code that result in a project that can’t run without leiningen
yeah, that too... : P
it’s nice to be able to wrap up an uberjar and just run that with java, needing leiningen specific constructs to configure the code breaks that
my app uses a default that makes sense during development (environment profile is “develop”) and then the staging and prod processes explicitly set a different value via environment variables
if you control deployment environment that’s a pretty simple solution, if you expect an end user to run a jar you might prefer setting the environment for local dev and defaulting to prod values
basically I wanted to just execute lein uberjar
without any env var setup...but still know in the code that this is a prod env
that's why I wanted to get the curr profile
IMHO your compilation process should be totally agnostic of runtime
the same uberjar should be able to run on local dev, staging, or prod
it's not that it is unable to...but in production I want to use a dockerized chrome-headless, and not in dev
could just do it with a config option though...
otherwise you get nonsense where the jar that passed testing on staging isn’t even the same jar that goes on prod which feels a bit mad?
For some reason I'm finding that nippy (https://github.com/ptaoussanis/nippy) is working out of the box freezing/thawing jodatime intervals. Is there a reason I'm missing that this is working? I was expecting to have to do custom freeze/thaw functions
K i will just do it with config
@jjttjj a quick glance shows that it knows how to use anything that implements Serializable https://github.com/ptaoussanis/nippy/blob/master/src/taoensso/nippy/utils.clj#L135
@noisesmith cool thanks!
I’m not sure if every code path allows that, but it at least has some allowance if I read that correctly
I'm coming from Python to Clojure. I'm wondering if there are best practices for helping keep data types and function associated? If I have a few different conventions for shape of map to contain data, is the best practice simply to keep all functions that work on that particular shape of map in a namespace and to do one namespace for each type/convention of map shape?
Maybe this is excessively pedantic, but the best thing is not to write functions so that they are tied to a specific data type. If possible code to interfaces not types. If possible code generally enough that your code works even if a hash-map has unexpected keys or metadata. Avoid specialized code that only works on one specific type as much as you can.
but, for example, using spec you can define specific properties of a piece of data under a given key, and also say that your function takes data in that shape
if you really want data types with associated functions (sometimes this is actually called for) defrecord defines the two together
automagically encoding any object with serializable makes it look like it’s working when you implement it, but if your classes have changed at all (say, you upgraded your dependencies) you’ll go to deserialize old data and it will fail in mysterious ways
another time we upgraded nippy from one minor version to another, and could no longer read any of our stored data
finally, if you care about interop with any other languages, you can’t read nippy in anything but clojure, because it’s a custom serialization format
sounds like a bunch of reasons to prefer transit :P
honestly just using EDN and maybe compressing it would be better from an operational cost standpoint
but transit improves on that significantly (performance and reusability)
right, that’s fair
yeah - I’ve used transit for years, over many clojure and library version upgrades and it’s never given us problems
there were errors in cljs with stale js files that showed up as bizarre transit errors, but transit wasn’t the problem
I do have to credit nippy for making me look for better binary formats, and eventually write a CBOR implementation 😄
transit provides a much richer (and extensible) set of types than json
and adds caching and other features so it’s also faster, particularly for the cases where you pass maps with the same keys all the time
http://blog.cognitect.com/blog/2014/7/22/transit for many more words :)
another good thing with transit is that unlike edn the serializers / deserializers are a first class argument and not a global compiler state
which makes using a specific set of tags for specific types much handier
frustration with global data-readers
and extending print-method
were part of what led me to write puget 🙂
for example I wrote a library that knows how to preserve and resuscitate atoms via a custom tag, and I don’t have to worry about messing up other people’s stuff by including it
(repost) Has anyone seen "This ResultSet is closed" errors from heroku (or anywhere else)? I am using jetty with with-db-connection. I have one instance that works and other (more recent) instances that fail on updates, all instances are funning identical code. https://github.com/jrootham/voting-server
The particular (failing) path I have put printlns into is updateUser
thanks @noisesmith , other than Protocols, Records etc. would a specific set/depth of keys be considered an interface. Would it be typical to have a (get-config-data db-map) as an interface, or would standard keys in a map be considered an interface? app-db always has :config for config data etc.?
when I mention coding against an interface rather than a data structure I mean the Interface (a vm level construct) that some data structure implements (and you could say that by specifying specific keys you are requiring the clojure.lang.IAssociative interface (this is needed in order to have keys) plus some further specification about the keys found (which spec can describe))
@jrootham the most common reason of connection closed is when you don't fully consume a resultset prior to closing the db connection
In this case the failure is when I am doing an update! My understanding is that update! is not lazy.
I did have that problem in other parts of my code. I added doall to fix it. They work now.
Do you think that would help, given that the same code works or fails depending on the heroku instance, and it looks like it fails for recently created heroku instances.
I am thinking this is a heroku specific problem (to be classic: it works on my machine). This may well not be the best forum to ask about this.
It’s been a long day. Is there a better way to say (fn [a b] b) ?
actually now that I think about it, what I really want is more like or
nvm! rubber duck.
just reading the nippy vs transit discussion from earlier. I recently chose nippy over transit for encoding some long-lived data because of this warning on the transit page “while Transit is a great option for transferring data between applications, it should not yet be used for storing data durably over time”. but now I’m worried 😉 I would actually prefer transit, but not sure how much of a risk it would be?
I’ve found that the cognitect team is very conservative about backward compatibility, and I’ve heard that transit is unlikely to break compatibility any time soon
that’s good to know. I believe transit is used for comms in datomic, so I imagine it would be difficult for Cognitect to make a breaking change
I think this link confirms it’s Transit: https://docs.datomic.com/cloud/whatis/accessing.html