This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-07-14
Channels
- # announcements (10)
- # babashka (14)
- # beginners (51)
- # calva (2)
- # cider (28)
- # cljs-dev (1)
- # cljsrn (3)
- # clojure (185)
- # clojure-dev (15)
- # clojure-europe (6)
- # clojure-italy (2)
- # clojure-nl (6)
- # clojure-uk (45)
- # clojurescript (4)
- # conjure (12)
- # datomic (37)
- # duct (37)
- # figwheel-main (15)
- # helix (23)
- # interceptors (1)
- # jobs (1)
- # jobs-discuss (11)
- # jvm (5)
- # luminus (3)
- # malli (15)
- # off-topic (69)
- # pathom (8)
- # pedestal (2)
- # re-frame (7)
- # reagent (58)
- # reitit (33)
- # remote-jobs (1)
- # rum (14)
- # sci (7)
- # shadow-cljs (15)
- # spacemacs (9)
- # testing (19)
- # timbre (3)
- # tools-deps (14)
Trying to understand why Heroku deploys started to fail:
remote: Download complete and in download only mode
remote: -----> Installing Clojure 1.10.0.411 CLI tools
remote: ! Push rejected, failed to compile Clojure (Leiningen 2) app.
remote:
remote: ! Push failed
curl
curl: (60) SSL certificate problem: certificate has expired
More details here:
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
curl -k to skip ssl errors
yeah, I just don't want to fork/update the official Heroku Clojure buildpack. I'll just wait then
It expired 38 minutes ago :)
subscribing for updates 😄
thx! still percolating through cloudfront so not everyone may see it yet
Hello folks! Have you tried to order a map into a specific order? I would like that a map stays organized by a sequence of keys, but i'm not finding a answer to it
Sorted by some function
(def test-order [:key :other-key :one-more-key])
#'user/test-order
user=> (sorted-map-by test-order {:other-key 1 :one-more-key 2 :key 12})
Execution error (ClassCastException) at user/eval2005 (REPL:1).
clojure.lang.PersistentVector cannot be cast to java.util.Comparator
As you said it yourself, it has to be a function. So, turn it into a function.
(def test-order (into {} (map-indexed (fn [i v] [v i])) test-order))
.
But for the pattern that you seem to require, it's common to just store the map and the vector together, and iterate along the vector to ensure the order.
Sorry, i do not got
I have a predefined instertion order
Well, my statement above converts your vector into a map. Which, by its nature, is a function. Which you can use as a key for sorted-map-by
.
user=> (def my-pattern [:a :b :c :d])
#'user/my-pattern
user=> (def order (into {} (map-indexed (fn [i v] [v i])) my-pattern))
#'user/order
user=> order
{:a 0, :b 1, :c 2, :d 3}
user=> (sorted-map-by order {:b 6 :d 8 :a 5 :c 7})
Execution error (ClassCastException) at user/eval2007 (REPL:1).
clojure.lang.PersistentArrayMap cannot be cast to java.util.Comparator
Is there something that i could be missing?Right, sorry - you have to also wrap order
to make a comparator out of it:
(defn order-comparator [a b]
(compare (order a) (order b)))
Apart from that, sorted-map-by
accepts multiple arguments that represent key-val pairs. So instead of the last line in your code block, you have to use this:
(sorted-map-by order :b 6 :d 8 :a 5 :c 7)
the keyval pairs could be in a vector or something?
user=> (doc sorted-map-by)
-------------------------
clojure.core/sorted-map-by
([comparator & keyvals])
keyval => key val
Returns a new sorted map with supplied mappings, using the supplied
comparator. If any keys are equal, they are handled as if by
repeated uses of assoc.
@ramonp.rios what do you expect to happen when new keys will be assoc'ed to your sorted-map?
It will run inside a (map)
(map (fn [row]
(-> row
clojure.walk/stringify-keys)))
I'm getting a list of maps, into this map function, i want to sort the map into a predefined order before get this data and put into a excel file
if the purpose is to output a list of csv columns, have you considered returning a sorted vector of tuples?
an ordered-map is useful when you need both O(1) random access to a key-value and seq'ing in a specific order
but that's an aside; if you are getting a map and just want to sort it by an arbitrary order AND return a map, you can just sort-it first and then return an array-map
(which keeps insertion order)
relying on array-map is tricky - calling conj will sometimes return a hash-map instead
^ damn, I was hoping if you used array-map
directly clojure wouldn't change it with a bigger input
so, this approach makes more sense to me than a custom comparator:
(let [cols [:a :b :c :d]]
(mapv #(zipmap cols (map % cols))
[{:a 1 :d 2 :b 3 :c 4}
{:a 200 :c 201 :b 202 :d 203}]))
but you would need to replace zipmap
with something more convoluted that keeps insertion-order guarantees; or a tuple; or https://github.com/frankiesardo/linked
Thanks, let me try it
That solution that was posted earlier with making a comparator does actually do what you want, but as a potential future reader of the code, I would question why you're passing a sorted-map downstream if the primary semantics you want to convey are ordered tuples for generating a CSV-like structure. (Emphasis on: that's how I'm understanding the intention, since you didn't confirm/deny.)
Thank you folk
another approach is to use a normal hash-map, and instead of using seq
(or the implcit seq call from map / filter / into etc.), use a function that returns the k/v pairs in sorted order
@U05476190 I do not know if you are familiar with this library, but it might address some of your desires: https://github.com/clj-commons/ordered
Thank you @U0CMVHBL2
@ramonp.rios : user=> (into {} (sort {:b 1 :a 2 :c 3})) {:a 2, 😛 1, :c 3}
not saure if that's what you wanted specifgically, or even if it makes sense to do so, but there. i'm a bit of a beginner myself
@bitpadawan into does not preserve order above a fixed number of elements, it auto-promotes to an unordered implementation
(cmd)user=> (into {} (map #(vector % %)) (range 8)); ordered
{0 0, 1 1, 2 2, 3 3, 4 4, 5 5, 6 6, 7 7}
(ins)user=> (into {} (map #(vector % %)) (range 9)); unordered
{0 0, 7 7, 1 1, 4 4, 6 6, 3 3, 2 2, 5 5, 8 8}
the fact that 8 is the switch over is an implementation detail, and not a promise made by clojure
using conj
/ assoc
with array-map
has the same issue
You should not assume that any operation on a map preserves seq order
For Clojure's built in map implementations, agreed. https://github.com/clj-commons/ordered implements maps and sets with more promises, for some extra cost.
Even for those, there are operations like merge, clojure.set/union etc. that are not promised to return the same concrete type of map/set that they are given, or at least you don't know which concrete type they will return unless you look at their implementation.
Just switch between testing on Clojure 1.5.1 and the latest version on alternate test runs 🙂
maybe a test library could use alter-var-root
in an opt in manner (via fixture...) to augment the array-map code to include a randomization step (or perhaps all maps, and the hash-map would just impose its order on the input)
There is a proposed patch for Clojure that is only correct if array-map
does not create key/value pairs in the same order as given in its args.
So at least some Clojure core team developers think randomizing the seq order of array-map
is an invalid implementation of array-map
.
Ghadi may have been referring to hash maps in particular.
The proposed patch on this issue, IIRC: https://clojure.atlassian.net/browse/CLJ-2469
Which reminds me .... I may be responsible for making array-map
taking O(n^2) time ...
array-map is a constructor function specifically to create a map with a specific seq order so I do not include that in my prior statement as a "map operation"
(and subsequently any map ops you do on that map may result in a different seq order)
I once speculated that there should be a testing jvm that intentionally added delays and changed execution orderings across threads to test concurrent programs
I bet there's some overlap between that and what Aphyr's jepsen does
though jepsen works on the OS layer, and intentionally does the things that are most likely to reveal bugs
has anyone created a paid or OSS FaaS-like offering for Clojure, similar to datomic cloud (but without datomic necessarily)?
either "failure as a service" or "functions as a service" maybe?
while I'm obviously biased in favor of datomic ions and the thinking behind them, I think deploying web apps in the cloud as functions is a perfect use case for Clojure - ring is a pretty clean map in / map out kind of function (fudging the streaming aspect).
cloud functions are mostly not that interesting without access to cloud data of course, which is the whole point of datomic ions (which do not actually have to talk to datomic btw)
An option now for Lambdas is to use RDS via https://aws.amazon.com/blogs/aws/amazon-rds-proxy-now-generally-available, which can be much cheaper than Datomic Cloud.
the function part is easy. taking care of workflow and all the service plumbing is the pain part of it (which datomic does, at no small effort)
> FaaS is a Data-Shipping Architecture. This is perhaps the biggest architectural shortcoming of FaaS platforms and their APIs. Serverless functions are run on isolated VMs, separate from data. In addition, serverless functions are short-lived and non-addressable, so their capacity to cache state internally to service repeated re-quests is limited. Hence FaaS routinely “ships data to code” rather than “shipping code to data.” This is a recurring architectural anti-pattern among system designers, which database aficionados seem to need to point out each generation.
the one thing that I disliked about ions, similar to lambda, is that it at the end of the day felt a lot like programming at a distance
ions suffer from this less than lambda, but having to construct a local dev env that matches the env of the deployed code is smelling worse and worse to me as I get more experience all things cloud computing
Clojure to me feels like it’s ripe for enabling a truly cloud-first experience, where you remotely connect to a private cluster and develop within that, promoting changes as needed
Have you seen darklang @lilactown ? It sounds similar to what you’re talking about. (not Clojure obviously)
I have 😄 IMO the paradigm is very much the future. It’s unfortunate that darklang is proprietary, makes it difficult for me to use
I feel like Ions is 3/4 of the way there 😄 but obviously there’s hard problems to solve w.r.t. version control/change management, other operational issues with developing in the cloud
The IDE stuff is interesting but probably not worth the time. The “deployless” aspect is truly magical. I think it would be possible to do with Clojure but quite hard.
I still can't accept a solution that I can't entirely run on my laptop. If your api routes have to go through aws, how do you work if the internet goes down?
Internets go down for days. If you travel and want to hack on a personal thing, you're stuck.
Fundamentally, I don’t think there’s anything that prevents a setup like darklang from running on your computer in an offline fashion. Darklang is (was? They may have open sourced recently) closed source so that’d be the main blocker.
I guess worst case you go back to what we’re doing now, e.g. run everything in docker
I think we were talking about something new, similar to what darklang does. Darklang is, afaik, online only though.
what do you think about the new dev-local?
the new "divert" stuff in particular
you can import cloud to a dev-local system (https://docs.datomic.com/cloud/dev-local.html#import-cloud), then divert all client calls in your app to the local system instead
and import has a lot of functionality to it - can import subsets of data and do incremental catch-up
I’m curious if there’s any exploration with ions to support what I said above - e.g. rather than running a dev-local system, moving all of development into the cloud
I believe there may have been some experimentation on that but I was not involved
what are the benefit of moving more development to the cloud? I've been trying to do the opposite and make it easier to develop more locally
the big efficiency boosts are local for dev and cloud for administration
I'm definitely on board with deploying to the cloud. not sure I'd want to run my own server infrastructure
right - but if the infrastructure is well designed (big if, I know) it's something you can drop into your local seamlessly for the most important things
having a good architecture team can do wonders here, because a good decision has amplified rippling effects (as does a bad decision...)
totally agree. my preferred setup is local dev and deploy to cloud
I'm so frequently bothered by the little mismatches - like the fact that we put the jvm inside an OS inside a docker image inside an OS on a machine managed by a cluster manager - so many layers of virtualization of virtualization
the resource requirements just blow up eventually
i've been doing a lot of hobby projects during covid and it's been nice to just upload a jar to digital ocean and run java -jar my.jar
inside a daemonized emacs
ideally we can write code that's transparent under each case and get best of both worlds (transducers can actually help here - over a generator locally and a data store on the cloud)
also if your services don't map cleanly to a transducer over a data store / generator, they might not be properly factored
like, first-order is the fact that replicating the environment locally is a pain. a lot of our services require this or that aws/az/whatever service, which can differ in behavior locally vs. cloud, and also at scale
we rely heavily on resources abstracted behind protocols when running locally, but it’s like… what’s the point? I could just run the code I mean to in prod
that’s just labor saving, though. the follow-on benefits are more enticing, e.g. being able to load test my changes. like radical idea: in order to test my changes, I join my dev cluster with the prod system and route 10% of traffic to mine
my experience has been that systems that are hard to set up locally are usually harder to reason about generally. if you have to spin up this amalgam to get anything to work, then it makes dev and test harder.
lots of our operation issues end up being differences between cloud providers, or some other aberrant behavior that only shows up when in the cloud env and receiving a certain amount of load
@lilactown maybe it's just that I've been forced to work with bad implementations of that, but my primary experience is that once my feature requires communicating with another service, a few seconds to run something in a repl turns into 15 minutes to build and deploy to an euat container
so my biggest win is abstracting the rest of the cluster away from my service, so I can go back to that fast repl turnover
oh no, it would be far, far worse to require build + deploy cycles to see relevant changes
if I had been the architect I wouldn't
The problem with diverting production traffic to dev is: - security - security - setting up local state so it makes sense
right - I can't divert a prod system for a financial system to use my dev env
With GDPR, I'd recommend you not do that in any system that processes personal data. "Oh, you have personal data on everyone's machine? What's your procedure for processing a deletion request? Yeah, here's a fine"
Just to be clear, PII != personal data. Personal data is broader. IP addresses are included
I guess you’d have to solve some problems first: • how do you operate a production system without exposing personal info? • how do we extend those same practices to “dev” envs while allowing speed of development?
yeah, so on to the second problem! although it might inform the solution to handling it in prod
like, if I can REPL into prod… does that lead to violations of GDPR in certain situations?
Yeah, in compliance systems you have to either disable that kind of access, or you add logging. Compliance loves logging.
Usually you can go a long way if you can explain when, by whom, and why data was accessed.
Although GDPR probably isn't as keen. You'd need a policy saying you don't export personal data to dev machines or something. You don't need to make things technically impossible, you just need a policy against them and reasonable measures (logging)
yeah. in a GDPR / CCPA world, it seems like moving development to a remote cluster could enable you to be a bit more gung-ho, since as long as data doesn’t touch dev machines it’s easy to track access to a bastion and, worst case, to wipe a VPC
the tricky thing is if eg. I call tap>
to debug some service function, then I connect via repl and attach some function via add-tap
that exposes PII onto my repl scrollback - is that serious? does it need to be prevented? even just logging the access only goes so far unless your compliance people are skilled with clojure
The compliance people tried to take away admin access from our laptops when this was raised. Then they could control that there was a 30 day deletion policy on any logs, which would cover this.
when I worked at a company that handled health records, we had a logging framework that would redact PII/PHI when in prod
yeah - fintech has another layer - not only is there restriction (from our board more than legislation here), but there's a financial incentive for bad actors to work around it
if a big part of your threat profile includes bad actors working internally, then yeah I don’t know how to solve that yet lol
I don't know if that's big factor as much as worst case... but it exists unlike in healthcare
I wonder if you could have a record that auto-redacts fields when printed / toStringed