Fork me on GitHub
#clojure
<
2020-07-14
>
nick12:07:37

Trying to understand why Heroku deploys started to fail:

remote:        Download complete and in download only mode
remote: -----> Installing Clojure 1.10.0.411 CLI tools
remote:  !     Push rejected, failed to compile Clojure (Leiningen 2) app.
remote: 
remote:  !     Push failed

nick12:07:46

curl 
curl: (60) SSL certificate problem: certificate has expired
More details here: 

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Alex Miller (Clojure team)12:07:20

That looks bad, I will check that cert

👍 6
🙏 3
Adam Arlett12:07:25

curl -k to skip ssl errors

nick12:07:12

yeah, I just don't want to fork/update the official Heroku Clojure buildpack. I'll just wait then

Alex Miller (Clojure team)12:07:31

It expired 38 minutes ago :)

martinklepsch12:07:55

subscribing for updates 😄

nick12:07:46

@alexmiller Looks like fixed. Thank you very much!

6
Alex Miller (Clojure team)12:07:06

thx! still percolating through cloudfront so not everyone may see it yet

Ramon Rios15:07:01

Hello folks! Have you tried to order a map into a specific order? I would like that a map stays organized by a sequence of keys, but i'm not finding a answer to it

p-himik15:07:51

You mean insertion order? Or so that keys are sorted by some function?

Ramon Rios15:07:07

Sorted by some function

p-himik15:07:37

clojure.core/sorted-map-by.

Ramon Rios15:07:03

(def test-order [:key :other-key :one-more-key])
#'user/test-order
user=> (sorted-map-by test-order {:other-key 1 :one-more-key 2 :key 12})
Execution error (ClassCastException) at user/eval2005 (REPL:1).
clojure.lang.PersistentVector cannot be cast to java.util.Comparator

p-himik15:07:47

As you said it yourself, it has to be a function. So, turn it into a function. (def test-order (into {} (map-indexed (fn [i v] [v i])) test-order)).

p-himik15:07:08

(maps are functions)

p-himik15:07:57

But for the pattern that you seem to require, it's common to just store the map and the vector together, and iterate along the vector to ensure the order.

Ramon Rios15:07:36

Sorry, i do not got

Ramon Rios15:07:42

I have a predefined instertion order

p-himik15:07:23

Well, my statement above converts your vector into a map. Which, by its nature, is a function. Which you can use as a key for sorted-map-by.

Ramon Rios16:07:45

user=> (def my-pattern [:a :b :c :d])
#'user/my-pattern
user=> (def order (into {} (map-indexed (fn [i v] [v i])) my-pattern))
#'user/order
user=> order
{:a 0, :b 1, :c 2, :d 3}
user=> (sorted-map-by order {:b 6 :d 8 :a 5 :c 7})
Execution error (ClassCastException) at user/eval2007 (REPL:1).
clojure.lang.PersistentArrayMap cannot be cast to java.util.Comparator
Is there something that i could be missing?

p-himik16:07:00

Right, sorry - you have to also wrap order to make a comparator out of it:

(defn order-comparator [a b]
  (compare (order a) (order b)))
Apart from that, sorted-map-by accepts multiple arguments that represent key-val pairs. So instead of the last line in your code block, you have to use this:
(sorted-map-by order :b 6 :d 8 :a 5 :c 7)

Ramon Rios16:07:35

the keyval pairs could be in a vector or something?

pithyless16:07:24

user=> (doc sorted-map-by)
-------------------------
clojure.core/sorted-map-by
([comparator & keyvals])
  keyval => key val
  Returns a new sorted map with supplied mappings, using the supplied
  comparator.  If any keys are equal, they are handled as if by
  repeated uses of assoc.

p-himik16:07:44

> the keyval pairs could be in a vector or something? Use apply for that.

p-himik16:07:56

Of into if you already have a map.

pithyless16:07:37

@ramonp.rios what do you expect to happen when new keys will be assoc'ed to your sorted-map?

pithyless16:07:27

(or is this a one-time sort?)

Ramon Rios16:07:57

It will run inside a (map)

Ramon Rios16:07:41

(map (fn [row]
                                      (-> row
                                          
                                          clojure.walk/stringify-keys)))

Ramon Rios16:07:08

I'm getting a list of maps, into this map function, i want to sort the map into a predefined order before get this data and put into a excel file

pithyless16:07:35

if the purpose is to output a list of csv columns, have you considered returning a sorted vector of tuples?

pithyless16:07:34

an ordered-map is useful when you need both O(1) random access to a key-value and seq'ing in a specific order

pithyless16:07:29

but that's an aside; if you are getting a map and just want to sort it by an arbitrary order AND return a map, you can just sort-it first and then return an array-map (which keeps insertion order)

noisesmith16:07:59

relying on array-map is tricky - calling conj will sometimes return a hash-map instead

pithyless16:07:12

^ damn, I was hoping if you used array-map directly clojure wouldn't change it with a bigger input

pithyless16:07:53

so, this approach makes more sense to me than a custom comparator:

(let [cols [:a :b :c :d]]
  (mapv #(zipmap cols (map % cols))
        [{:a 1 :d 2 :b 3 :c 4}
         {:a 200 :c 201 :b 202 :d 203}]))

pithyless16:07:23

but you would need to replace zipmap with something more convoluted that keeps insertion-order guarantees; or a tuple; or https://github.com/frankiesardo/linked

Ramon Rios16:07:32

Thanks, let me try it

pithyless16:07:44

(no reason that's a mapv and not map)

pithyless16:07:47

That solution that was posted earlier with making a comparator does actually do what you want, but as a potential future reader of the code, I would question why you're passing a sorted-map downstream if the primary semantics you want to convey are ordered tuples for generating a CSV-like structure. (Emphasis on: that's how I'm understanding the intention, since you didn't confirm/deny.)

Ramon Rios16:07:03

Thank you folk

noisesmith16:07:37

another approach is to use a normal hash-map, and instead of using seq (or the implcit seq call from map / filter / into etc.), use a function that returns the k/v pairs in sorted order

andy.fingerhut17:07:32

@U05476190 I do not know if you are familiar with this library, but it might address some of your desires: https://github.com/clj-commons/ordered

emccue00:07:03

LinkedHashMap as well

emccue00:07:23

i mean, its mutable and all but if you pinky swear not to it is there

bitpadawan17:07:56

@ramonp.rios : user=> (into {} (sort {:b 1 :a 2 :c 3})) {:a 2, 😛 1, :c 3}

bitpadawan17:07:28

not saure if that's what you wanted specifgically, or even if it makes sense to do so, but there. i'm a bit of a beginner myself

noisesmith17:07:05

@bitpadawan into does not preserve order above a fixed number of elements, it auto-promotes to an unordered implementation

noisesmith17:07:52

@bitpadawan

(cmd)user=> (into {} (map #(vector % %)) (range 8)); ordered
{0 0, 1 1, 2 2, 3 3, 4 4, 5 5, 6 6, 7 7}
(ins)user=> (into {} (map #(vector % %)) (range 9)); unordered
{0 0, 7 7, 1 1, 4 4, 6 6, 3 3, 2 2, 5 5, 8 8}

noisesmith17:07:36

the fact that 8 is the switch over is an implementation detail, and not a promise made by clojure

noisesmith17:07:17

using conj / assoc with array-map has the same issue

Alex Miller (Clojure team)18:07:01

You should not assume that any operation on a map preserves seq order

6
andy.fingerhut18:07:52

For Clojure's built in map implementations, agreed. https://github.com/clj-commons/ordered implements maps and sets with more promises, for some extra cost.

ghadi18:07:42

sometimes I wish clojure randomized map seq order

6
andy.fingerhut18:07:16

Even for those, there are operations like merge, clojure.set/union etc. that are not promised to return the same concrete type of map/set that they are given, or at least you don't know which concrete type they will return unless you look at their implementation.

andy.fingerhut18:07:53

Just switch between testing on Clojure 1.5.1 and the latest version on alternate test runs 🙂

noisesmith18:07:59

maybe a test library could use alter-var-root in an opt in manner (via fixture...) to augment the array-map code to include a randomization step (or perhaps all maps, and the hash-map would just impose its order on the input)

andy.fingerhut18:07:23

There is a proposed patch for Clojure that is only correct if array-map does not create key/value pairs in the same order as given in its args.

andy.fingerhut18:07:03

So at least some Clojure core team developers think randomizing the seq order of array-map is an invalid implementation of array-map .

andy.fingerhut18:07:23

Ghadi may have been referring to hash maps in particular.

andy.fingerhut18:07:15

Which reminds me .... I may be responsible for making array-map taking O(n^2) time ...

Alex Miller (Clojure team)18:07:08

array-map is a constructor function specifically to create a map with a specific seq order so I do not include that in my prior statement as a "map operation"

Alex Miller (Clojure team)18:07:57

(and subsequently any map ops you do on that map may result in a different seq order)

ghadi19:07:27

my thoughts were idle dreaming

ghadi19:07:48

randomized iteration would save a lot of time answering questions about map order

💯 3
Alex Miller (Clojure team)19:07:28

I once speculated that there should be a testing jvm that intentionally added delays and changed execution orderings across threads to test concurrent programs

noisesmith20:07:01

I bet there's some overlap between that and what Aphyr's jepsen does

noisesmith20:07:23

though jepsen works on the OS layer, and intentionally does the things that are most likely to reveal bugs

lilactown19:07:52

has anyone created a paid or OSS FaaS-like offering for Clojure, similar to datomic cloud (but without datomic necessarily)?

kenny19:07:38

I think they mostly stopped on that after Datomic Ions were released.

noisesmith19:07:02

either "failure as a service" or "functions as a service" maybe?

Alex Miller (Clojure team)19:07:39

while I'm obviously biased in favor of datomic ions and the thinking behind them, I think deploying web apps in the cloud as functions is a perfect use case for Clojure - ring is a pretty clean map in / map out kind of function (fudging the streaming aspect).

Alex Miller (Clojure team)19:07:09

cloud functions are mostly not that interesting without access to cloud data of course, which is the whole point of datomic ions (which do not actually have to talk to datomic btw)

lilactown19:07:54

yeah. I liked a lot of the development workflow aspects of ions.

lilactown19:07:16

running a datomic cloud cluster was expensive for my use case, though

lilactown19:07:29

(dumb hobby project)

kenny19:07:01

An option now for Lambdas is to use RDS via https://aws.amazon.com/blogs/aws/amazon-rds-proxy-now-generally-available, which can be much cheaper than Datomic Cloud.

Alex Miller (Clojure team)19:07:09

the function part is easy. taking care of workflow and all the service plumbing is the pain part of it (which datomic does, at no small effort)

ghadi19:07:48

bringing the code next to the data is a critical piece that Ions do

ghadi19:07:58

Lambda in general are far away from the data

lilactown19:07:24

yeah, that’s one aspect I do not like about lambda and a lot of other generic FaaS

ghadi19:07:37

> FaaS is a Data-Shipping Architecture. This is perhaps the biggest architectural shortcoming of FaaS platforms and their APIs. Serverless functions are run on isolated VMs, separate from data. In addition, serverless functions are short-lived and non-addressable, so their capacity to cache state internally to service repeated re-quests is limited. Hence FaaS routinely “ships data to code” rather than “shipping code to data.” This is a recurring architectural anti-pattern among system designers, which database aficionados seem to need to point out each generation.

ghadi19:07:45

sorry, formatting got messed up

lilactown19:07:48

the workflow is garbage for lambda IMO.

ghadi19:07:01

Datomic Ions let you ship the code to the data

💯 3
ghadi20:07:04

that short paper is very well worth the read for aspiring system designers

ghadi20:07:16

> Serverless Computing: One Step Forward, Two Steps Back

lilactown20:07:19

the one thing that I disliked about ions, similar to lambda, is that it at the end of the day felt a lot like programming at a distance

lilactown20:07:20

ions suffer from this less than lambda, but having to construct a local dev env that matches the env of the deployed code is smelling worse and worse to me as I get more experience all things cloud computing

lilactown20:07:13

Clojure to me feels like it’s ripe for enabling a truly cloud-first experience, where you remotely connect to a private cluster and develop within that, promoting changes as needed

kenny20:07:19

Have you seen darklang @lilactown ? It sounds similar to what you’re talking about. (not Clojure obviously)

kenny20:07:50

I’ve thought a lot about something similar for Clojure.

lilactown20:07:04

I have 😄 IMO the paradigm is very much the future. It’s unfortunate that darklang is proprietary, makes it difficult for me to use

✔️ 3
lilactown20:07:19

yeah, I think it would be beautiful to have something Clojure-native

kenny20:07:39

I agree! I haven’t seen anyone tackle it yet though

lilactown20:07:52

I feel like Ions is 3/4 of the way there 😄 but obviously there’s hard problems to solve w.r.t. version control/change management, other operational issues with developing in the cloud

kenny20:07:03

The IDE stuff is interesting but probably not worth the time. The “deployless” aspect is truly magical. I think it would be possible to do with Clojure but quite hard.

dominicm20:07:00

I still can't accept a solution that I can't entirely run on my laptop. If your api routes have to go through aws, how do you work if the internet goes down?

kenny20:07:44

If “the internet” goes down or your personal internet?

kenny20:07:02

If the former, you have problems. If the latter, take a coffee break?

dominicm20:07:12

Internets go down for days. If you travel and want to hack on a personal thing, you're stuck.

kenny20:07:19

Fundamentally, I don’t think there’s anything that prevents a setup like darklang from running on your computer in an offline fashion. Darklang is (was? They may have open sourced recently) closed source so that’d be the main blocker.

dominicm20:07:29

But yeah I meant the latter.

lilactown20:07:37

I guess worst case you go back to what we’re doing now, e.g. run everything in docker

kenny20:07:20

Yeah, something like that.

dominicm20:07:20

AWS API gateway doesn't run in docker

kenny20:07:48

Don’t couple to propriety services.

kenny20:07:37

Some of that stuff could be a shortcut upfront ofc

dominicm20:07:16

In the context of datomic ions and lambda, that's the only way to web with faas

kenny20:07:05

I think we were talking about something new, similar to what darklang does. Darklang is, afaik, online only though.

lilactown20:07:39

moving code close to data feels like a crucial and necessary first step

Alex Miller (Clojure team)20:07:50

what do you think about the new dev-local?

lilactown20:07:08

I saw the announcement but haven’t used it! it does sound very useful

Alex Miller (Clojure team)20:07:45

the new "divert" stuff in particular

lilactown20:07:40

I haven’t looked at “divert” yet

Alex Miller (Clojure team)20:07:35

you can import cloud to a dev-local system (https://docs.datomic.com/cloud/dev-local.html#import-cloud), then divert all client calls in your app to the local system instead

Alex Miller (Clojure team)20:07:03

and import has a lot of functionality to it - can import subsets of data and do incremental catch-up

lilactown20:07:25

I’m curious if there’s any exploration with ions to support what I said above - e.g. rather than running a dev-local system, moving all of development into the cloud

lilactown20:07:43

right now the biggest hurdle seems like change management/version control

Joe Lane20:07:19

It's certainly an area worth exploring.

3
lilactown20:07:05

of course that would def be even more expensive for my hobby projects lmao

lilactown20:07:20

but feels like something I really want to push us towards at work

Alex Miller (Clojure team)20:07:44

I believe there may have been some experimentation on that but I was not involved

phronmophobic20:07:52

what are the benefit of moving more development to the cloud? I've been trying to do the opposite and make it easier to develop more locally

👍 3
lilactown20:07:16

it’s the second-order things that it unlocks which I see as being valuable

noisesmith20:07:33

the big efficiency boosts are local for dev and cloud for administration

phronmophobic20:07:21

I'm definitely on board with deploying to the cloud. not sure I'd want to run my own server infrastructure

noisesmith20:07:20

right - but if the infrastructure is well designed (big if, I know) it's something you can drop into your local seamlessly for the most important things

💯 3
noisesmith20:07:52

having a good architecture team can do wonders here, because a good decision has amplified rippling effects (as does a bad decision...)

phronmophobic20:07:20

totally agree. my preferred setup is local dev and deploy to cloud

noisesmith20:07:48

I'm so frequently bothered by the little mismatches - like the fact that we put the jvm inside an OS inside a docker image inside an OS on a machine managed by a cluster manager - so many layers of virtualization of virtualization

noisesmith20:07:00

the resource requirements just blow up eventually

phronmophobic20:07:01

i've been doing a lot of hobby projects during covid and it's been nice to just upload a jar to digital ocean and run java -jar my.jar inside a daemonized emacs

noisesmith20:07:17

ideally we can write code that's transparent under each case and get best of both worlds (transducers can actually help here - over a generator locally and a data store on the cloud)

👆 3
👍 3
noisesmith20:07:05

also if your services don't map cleanly to a transducer over a data store / generator, they might not be properly factored

lilactown20:07:17

like, first-order is the fact that replicating the environment locally is a pain. a lot of our services require this or that aws/az/whatever service, which can differ in behavior locally vs. cloud, and also at scale

lilactown20:07:08

we rely heavily on resources abstracted behind protocols when running locally, but it’s like… what’s the point? I could just run the code I mean to in prod

lilactown20:07:46

that’s just labor saving, though. the follow-on benefits are more enticing, e.g. being able to load test my changes. like radical idea: in order to test my changes, I join my dev cluster with the prod system and route 10% of traffic to mine

phronmophobic20:07:46

my experience has been that systems that are hard to set up locally are usually harder to reason about generally. if you have to spin up this amalgam to get anything to work, then it makes dev and test harder.

☝️ 3
lilactown20:07:08

lots of our operation issues end up being differences between cloud providers, or some other aberrant behavior that only shows up when in the cloud env and receiving a certain amount of load

noisesmith20:07:36

@lilactown maybe it's just that I've been forced to work with bad implementations of that, but my primary experience is that once my feature requires communicating with another service, a few seconds to run something in a repl turns into 15 minutes to build and deploy to an euat container

noisesmith20:07:02

so my biggest win is abstracting the rest of the cluster away from my service, so I can go back to that fast repl turnover

lilactown20:07:18

oh no, it would be far, far worse to require build + deploy cycles to see relevant changes

☝️ 3
lilactown20:07:24

but we don’t do that in Clojure, do we?

lilactown20:07:40

I have a REPL open rn that I haven’t restarted in 10 days

noisesmith20:07:44

if I had been the architect I wouldn't

lilactown20:07:55

it could just be living in some VPC instead of my MBP 😄

dominicm20:07:58

The problem with diverting production traffic to dev is: - security - security - setting up local state so it makes sense

😆 9
noisesmith20:07:36

right - I can't divert a prod system for a financial system to use my dev env

noisesmith20:07:46

and if I could, I'd advise you no longer use that system

😬 3
dominicm20:07:50

With GDPR, I'd recommend you not do that in any system that processes personal data. "Oh, you have personal data on everyone's machine? What's your procedure for processing a deletion request? Yeah, here's a fine"

lilactown20:07:20

sure, I’m not suggesting running PII through your macbook

dominicm20:07:25

Just to be clear, PII != personal data. Personal data is broader. IP addresses are included

dominicm20:07:37

Common misconception about gdpr

lilactown20:07:43

I guess you’d have to solve some problems first: • how do you operate a production system without exposing personal info? • how do we extend those same practices to “dev” envs while allowing speed of development?

dominicm20:07:19

Production is a solved problem. Unless you have further constraints.

lilactown20:07:30

yeah, so on to the second problem! although it might inform the solution to handling it in prod

lilactown20:07:59

like, if I can REPL into prod… does that lead to violations of GDPR in certain situations?

lilactown20:07:42

(probably) so we would have to figure out how to handle that use case

dominicm20:07:25

Yeah, in compliance systems you have to either disable that kind of access, or you add logging. Compliance loves logging.

emilaasa20:07:07

Usually you can go a long way if you can explain when, by whom, and why data was accessed.

dominicm20:07:45

Although GDPR probably isn't as keen. You'd need a policy saying you don't export personal data to dev machines or something. You don't need to make things technically impossible, you just need a policy against them and reasonable measures (logging)

lilactown20:07:37

yeah. in a GDPR / CCPA world, it seems like moving development to a remote cluster could enable you to be a bit more gung-ho, since as long as data doesn’t touch dev machines it’s easy to track access to a bastion and, worst case, to wipe a VPC

lilactown20:07:36

i’m obviously not an expert here so ¯\(ツ)

noisesmith20:07:08

the tricky thing is if eg. I call tap> to debug some service function, then I connect via repl and attach some function via add-tap that exposes PII onto my repl scrollback - is that serious? does it need to be prevented? even just logging the access only goes so far unless your compliance people are skilled with clojure

dominicm21:07:55

The compliance people tried to take away admin access from our laptops when this was raised. Then they could control that there was a 30 day deletion policy on any logs, which would cover this.

lilactown20:07:32

yeah, that’s another problem.

lilactown20:07:31

when I worked at a company that handled health records, we had a logging framework that would redact PII/PHI when in prod

lilactown20:07:19

it hashed user ids so we could tie support tickets to logs at least

noisesmith20:07:19

yeah - fintech has another layer - not only is there restriction (from our board more than legislation here), but there's a financial incentive for bad actors to work around it

lilactown20:07:58

if a big part of your threat profile includes bad actors working internally, then yeah I don’t know how to solve that yet lol

noisesmith20:07:29

I don't know if that's big factor as much as worst case... but it exists unlike in healthcare

lilactown20:07:38

I wonder if you could have a record that auto-redacts fields when printed / toStringed

dominicm21:07:00

You can override print