Fork me on GitHub
#beginners
<
2019-05-09
>
Chris Reyes03:05:56

Hello world, I’m working on a small app that needs to save data to long term storage. For my production version, I plan to use postgres as the backing storage, but I’d like to abstract the specific backing away from the app. (For example if I want to test with a text file in local development). For the sake of simplicity, let’s say that the application only cares about Loading a specific Foo, Saving a Foo, and Listing which Foos exist. I started writing a “persistence” protocol which has those methods, thinking that I could then write a couple of things that implement the methods in that protocol. But now I’m second guessing myself and wondering if I’m secretly just doing OOP. Am I on the right track here? Or is there a more idiomatic way to do this in Clojure?

valtteri04:05:43

Your protocol abstraction sounds like idiomatic Clojure to me and in general it IS a good idea to use such abstractions. However what comes to prod vs local environment my personal preference is to try to keep them as 1:1 as possible. For example it’s very easy to spin up a postgres with Docker for local testing so why bother with files when you can test “the real thing” locally end-to-end.

valtteri05:05:36

My reasoning goes something like this: eventually you need to ensure both that a) your app works in prod and b) your app does what it should. You should try to find the sweet spot between the implementation and tests that covers both aspects with minimum amount of overhead.

valtteri05:05:16

And from there we get to the point that if you are running 1:1 same stack in dev and prod, do you need the abstraction layer anymore?

valtteri05:05:28

There are many aspects to consider. 🙂 Difficult to provide the “right answer”.

seancorfield05:05:19

I would not advise going down the protocol path for this, to be honest.

seancorfield05:05:15

In any real world app you're likely to end up with a large number of methods in your protocol. And you'll probably end up with methods that have a number of conditional arguments so your queries can be built conditionally.

seancorfield05:05:13

You'll end up creating a completely artificial abstraction -- purely for the theoretical benefits of testing with a mock for the persistence. Lots more code to maintain and a very brittle abstraction as your code evolves.

seancorfield05:05:51

As @valtteri suggested, I'd simply create a local DB instance via Docker. It's the simplest way to ensure you're testing the same thing as you will run in production. If you test against a protocol-based mock that does something like text file storage, and then you run it in production with PostgreSQL then you've never tested your production code paths.

seancorfield05:05:51

If your SQL is all very standard, you could theoretically test against an in-memory database. However, as the maintainer of the primary JDBC library for Clojure, I can tell you all sorts of ways that PostgreSQL, MySQL, SQL Server, H2 in-memory, SQLite, Derby, HSQLDB etc are all very, very different (unfortunately). @chrisreyes

valtteri05:05:23

Heh, I also initially went down the protocol path (in case of db) but ended up refactoring to good old’ functions when I realised I had created a redundant abstraction. In general I think that abstraction IS usually a good and powerful thing but unnecessary abstraction is just… unnecessary. 🙂

Chris Reyes13:05:28

Thanks for all the great advice!

seancorfield05:05:46

In all the apps I've ever built that interact with a database, conditionally built queries are unavoidable and you end up with SQL leaking out into your application somewhere. Frankly, I've just learned to "love" SQL 🙂

seancorfield05:05:35

One of my colleagues gave a talk about HoneySQL some years back at Clojure/West. We use that to help compose query fragments conditionally in a number of places. Abstracting that away behind some artificial function layer just isn't worth the effort.

seancorfield05:05:37

Protocols can be great -- if you're going to have multiple implementations in your production code.

👍 4
Chris Reyes13:05:18

Thanks for all the great advice!

dangercoder06:05:24

let's say I have some code in my app in the namespace: app.db.user.settings. I have my tests in my test-directory. Would It be most idiomatic to mirror the application structure in my test-catalog? I am using leiningen.

seancorfield06:05:51

It is normal to mirror the namespace structure and append -test to the last segment.

seancorfield06:05:07

So for src/app/db/user/settings.clj you'd have test/app/db/user/settings_test.clj

seancorfield06:05:52

That's why when you create a new project with lein, it creates src/myapp/core.clj and test/myapp/core_test.clj

dangercoder06:05:30

It helps, its just like in other jvm-languages. Thank you @seancorfield! ✌️

gklijs11:05:03

What would be the easiest/best way to go from {a: [1 2] b: [4 5]} {a: [3 4] c: [8 9]} to {a: [1 2 3 4] b: [4 5] c: [8 9]} so basically conjoin on all the keys.

lispyclouds11:05:32

@gklijs (merge-with concat {:a [1 2] :b [4 5]} {:a [3 4] :c [8 9]}) should be one way

gklijs11:05:48

Thanks, ended with (merge-with into {:a [1 2] :b [4 5]} {:a [3 4] :c [8 9]}) cause of keeping vector's and no need to be lazy

4
tabidots12:05:59

Is there a way to log Criterium outputs somehow? Specifically, I have implemented several integer factorizing algorithms and I want to evaluate their performance on randomly generated sets of numbers meeting certain criteria. I want to save everything to a CSV so I can plot the data. What is the easiest way for me to go about this?

delaguardo12:05:25

quickest way to do that - wrap calls for criterium functions in with-out-str. after that you can save this report somewhere

tabidots12:05:58

Thanks. So then I’d parse the output text file manually to create a CSV? Trying to think how I’d organize this so I can keep track of what inputs went with what outputs

tabidots12:05:02

Also, does this mean there’s no hands-off way to do it? I was hoping to have these tests run while I sleep 😅

delaguardo12:05:46

there is main function that prints report for every metric - https://github.com/hugoduncan/criterium/blob/develop/src/criterium/core.clj#L870 You can try to override print or format to get each metric individually but this will be super fragile solution. Parsing the report string looks like a best option for you. I can recommend to try instapars to do so. At least it will be a bit more stable than handcrafted override for core functions)

delaguardo13:05:51

forget that I just sad) there is a functions that returns report in machine readable form - benchmark and quick-benchmark

4
tabidots13:05:34

That looks great! I see it doesn’t save the inputs though? So instead of supplying an empty map as the second arg, I should supply a map populated with the quoted input?

delaguardo13:05:01

second argument are an options for underling functions. defaults are here - https://github.com/hugoduncan/criterium/blob/develop/src/criterium/core.clj#L83-L99

tabidots13:05:10

Great, thank you. Sorry I’m on phone atm so a bit hard to skim through code but I will take a look later 👍:skin-tone-2:

dumrat13:05:31

Hi guys, a simple problem: I just modified a project.clj, I want to reload automata.core/init on figwheel reload. So I followed the instructions (I think). The relevant part in project.clj:

:cljsbuild
  {:builds
   [{:id           "dev"
     :source-paths ["src/cljs"]
     :figwheel     {:on-jsload "automata.core/init"}
     :compiler   ...
and core.cljs
(ns automata.core
  (:require [automata.config :as config]
            [automata.automata :as auto]))

(defn ^:export init []
  (enable-console-print!)
  (println "core/init")
  (auto/run))
But I don't see this being reloaded after file is changed. The console shows that the file was reloaded, but the init function doesn't seem to get called. Any pointers?

john14:05:30

Are you calling init somewhere? Like from your html file?

john14:05:55

ah, on-jsload

john14:05:56

are there any other namespaces at play in your project?

john14:05:12

Does :main have "automata.core"?

dumrat15:05:22

@john: Gave up and created a fresh project. Working now. Thanks for replies.

ben14:05:49

Has anyone come across the following pattern before? • I have 2+ projects with a single (small) data file that I don’t want replicated • I want this data file to be versioned • I create a data-only library which has the file in resources • And add it as a dependency to my other projects • My projects can now all access the data file with io/resources as if it was in its own resources dir? Conceptually this makes sense to me; although I cannot work how exactly how the last step is enabled.

john15:05:42

I've seen projects pulled in as deps simply for their data resources

ben15:05:20

But how does it link the resources? Surely this isn’t the default behaviour?

Alex Whitt15:05:46

I believe that is the default behavior actually, @ben606. I don't know the specifics myself, other than that they get bundled in with the .jar and are on the classpath. I believe you can also put them under subdirectories and access them by the full path. https://en.wikipedia.org/wiki/Resource_(Java)

ben15:05:10

🤯 thanks, Alex

seancorfield16:05:57

@ben606 It's "just" about the classpath -- and typically that has src and resources on it. clj only has src by default. You can specify any folders to go on the classpath and they'll get rolled into the running system or the JAR file if you're building one.

seancorfield17:05:24

lein puts quite a few folders on the classpath by default. In dev mode, that includes dev-resources, resources, src, target/default/classes, and test.

ben08:05:49

Good to know! Thank you, Sean!

lepistane18:05:31

How do i save a file using cljs? I am using cljs-ajax but i am not getting the window popup to save the file are there any examples online?

Kari Marttila18:05:44

Sorry for the long post. 🙂 I'm rewriting one of my earlier Clojure learning projects using e.g. deps.edn (earlier using Leiningen), and using mount for storing application state. At the same time I would like to rewrite the mechanism that I'm using for choosing dynamically domain logic depending which environment the app is running (single-node, AWS or Azure). I would really appreciate if some Clojure guru takes a look at this: https://github.com/karimarttila/clojure/tree/master/clj-ring-cljs-reagent-demo/simple-server/src/simpleserver/domaindb In domain_factory.clj I have several multimethods which dispatch regarding the environment and they return a defrecord defined in either domain_single_node.clj, domain_dynamodb.clj (AWS) or domain_table_storage (Azure). All these defrecords implement the same service interface defined in domain_service_interface.clj. Then in web server I'm able to: (def domain-svc (ss-domain-factory/create-domain)) and then use that service interface so that the web server doesn't need to know whether it's actually getting data from local single-node implementation, from AWS or from Azure. The reason I'm asking this is that I'm not at all sure if this solution is idiomatic Clojure solution. I can now make a more idiomatic Clojure solution if I just figure out how to do it or someone provides me some hints. If someone says the solution is idiomatic Clojure solution for that purpose - I'm happy with that information as well. 🙂

Kari Marttila19:05:55

Really? Thanks! I guess I leave that part as it is then.

john19:05:45

You can find plenty of differences of opinion on how to go about doing those things. I prefer to prototype things out as plain maps and functions and only move to types/records and protocols if/when necessary. But since you're already working with component, it probably makes sense to start with records.

john18:05:58

You may need to fire it from a user initiated action, or it may not run the save dialog

lepistane18:05:08

@john thank you but i dont think solves my problem. I will give little bit more context currently i am using html form with multiple input checkbox fields and it downloads file that i checked - regular html form behavior now i would like to do that with cljs or any lib that does this

john18:05:07

downloads to one's download directory? Or downloads them as data into the browser's memory? @lepistane

lepistane18:05:50

download to directory

john18:05:10

I believe you can only accomplish that by going through a "save as" dialog. Were you wanting to avoid that?

lepistane18:05:51

i am ok with having that dialog i just don't want to use html forms to make request

manutter5118:05:11

A co-worker recently found [cljsjs/filesaverjs “1.3.3-0”]

👍 4
john18:05:46

Make the request to download the file to the browser from the server? Or some user request that might initiate a save action?

john18:05:54

And what is the file type?

manutter5118:05:16

He used it to build a cljs-based mechanism for downloading files directly from an ajax call

lepistane18:05:07

file type - zip user chooses which files he wants to download (checkbox) click save - wait for zip

john18:05:37

Getting the file from the server and then getting that file saved to the downloads folder are two separate concerns, IMO. Ahh

john18:05:49

Are you able to get the zip file downloaded from the server and into a js/File object?

john18:05:11

I'd work on first getting that confirmed first. Then you should be able to leverage the above code or filesaverjs to save it.

john18:05:05

@lepistane for cljs-ajax, it looks like you may need to look into :response-format... you probably need :type to be :blob... more info here https://github.com/JulianBirch/cljs-ajax/blob/master/docs/formats.md

john18:05:45

You should also be able to brute force it with JS docs

Daouda19:05:45

hey folks, what is the difference between seqand coll?

Mno19:05:44

are you asking about the data types (sequences vs collections) or the methods themselves?

Daouda19:05:32

(def
 ^{:arglists '(^clojure.lang.ISeq [coll])
   :doc "Returns a seq on the collection. If the collection is
    empty, returns nil.  (seq nil) returns nil. seq also works on
    Strings, native Java arrays (of reference types) and any objects
    that implement Iterable. Note that seqs cache values, thus seq
    should not be used on any Iterable whose iterator repeatedly
    returns the same mutable object."
   :tag clojure.lang.ISeq
   :added "1.0"
   :static true}
 seq (fn ^:static seq ^clojure.lang.ISeq [coll] (. clojure.lang.RT (seq coll))))

Mno19:05:10

that’s the definition of the method called “seq”

Mno19:05:34

the method seq takes a collection as the parameter

Mno19:05:43

that entire article is a great read for understanding clojure data structures.

Daouda19:05:49

Can you tell me the relevant difference between sequences vs collections please?

Mno19:05:35

collections are pretty much anything that can be “collected” and sequences have an order so you can call first and rest on sequences.

Mno19:05:46

I’m pretty new myself so that’s all I really know

john19:05:16

Aye, I think you could consider a collection to be a superset of sequences. Sequences are ordered/indexical collections, with a light-weight linked-list first/rest interface, like @UGFL22X0Q said

lilactown19:05:48

seq in Clojure is like an interface or protocol

lilactown19:05:24

you can operate on any data structure that implements the sequence interface(s) using the sequence functions

lilactown19:05:33

like map/filter/etc.

john19:05:52

hmm, I guess saying "ordered" is questionable, since you can seq a map, which isn't ordered.

lilactown19:05:26

right. most common clojure data isn’t a sequence either, they’re seqable

lilactown19:05:38

as in, they can be turned into a sequence

john19:05:48

But the output to seq results in a sequence that is not exactly un-ordered :thinking_face:

lilactown19:05:22

a sequence is an object that acts like a view on a collection

lilactown19:05:02

so you can call first on it, rest on it and it will return the appropriate thing given the underlying collection

lilactown19:05:17

this enables you to map over vectors, maps, lists and sets

lilactown19:05:46

most collections are seqable, which means you can call (seq collection) and it will return a sequence over the collection you passed in

john19:05:22

I think Clojure's balance between indexical sequential collections and associative collections is instructive too. A lot of data manipulation seems to go into translating between ordered sequential and associative domains and Clojure tends to focus on those.

Daouda20:05:36

sorry guys, i wasn’t on my computer. Thank you all for the answers, helped a lot 😄

Daouda21:05:54

So vectors, lists, hash-map, setsare all collections

Daouda21:05:23

but only vectors and lists are sequences?

lilactown21:05:42

none of them are sequences

Daouda21:05:54

hahahahahaha

lilactown21:05:02

user=> (seq? [])
false

lilactown21:05:19

but they are sequencable

lilactown21:05:32

user=> (seqable? [])
true

john21:05:35

All of them are sequencable

lilactown21:05:22

what I’m trying to say is: a sequence is an object that provides a view on a collection

john21:05:23

calling seq on a set or map though will not have a guaranteed order

Daouda21:05:36

so why would I use seq for?

Daouda21:05:27

I mean a use case for seq

lilactown21:05:41

usually you don’t have to call seq yourself

john21:05:56

1. you want a thing as a sequence. Like for a map. 2. you want to fully realize a lazy thing

lilactown21:05:20

the sequence operations like map call seq on it’s collection argument

lilactown21:05:03

right… the other thing is that sequences are lazy

lilactown21:05:56

I think this page does a decent job of explaining it (better than me) https://clojure.org/reference/sequences

Idan Melamed19:05:38

Hi starting with Clojure here... I'm looking at the source for reverse and it says "(reduce1 conj () coll)". is reduce1 a typo or is there another reduce function?

john19:05:42

reduce1 is used internally to build up the language. Not intended for public use.

👍 4
vlad_poh19:05:28

howdy! anyone know how i can see the list of resource folders http://clojure.java.io/resource looks through

seancorfield19:05:31

@kbosompem That would be anything on the classpath.

seancorfield19:05:10

lein classpath | tr ':' '\n' | sort is a good way to see that with Leiningen on macOS/Linux.

vlad_poh19:05:11

I uberjar'ed my service but it won't use the config file

seancorfield19:05:12

When you have an uberjar, the classpath is just what's inside the uberjar.

seancorfield19:05:41

If your config file is outside your uberjar, io/resource won't see it.

👍 4
vlad_poh19:05:07

in that case do i do a slurp

Nathan Knox19:05:57

@kbosompem Thank you for giving me new hackerspeak. Making uberjar a verb is good, but "doing a slurp" is one of the best things I've heard in a while.

👍 4
seancorfield20:05:48

Clojure has slurp and spit functions 🙂

👍 4
Nathan Knox20:05:53

Oh my god, really?! That's epic.

Eric Ervin20:05:38

I once showed Ms Ladyfriend some Clojure and she was put off by those function names.

😂 4
seancorfield20:05:34

I certainly raised an eyebrow when I first encountered them... but we're also very used to paredit terminology of slurping and barfing...

Eric Ervin20:05:25

Maybe playing into computer dude stereotypes .

Nathan Knox20:05:38

@U9U0G3Q8Y maybe, but only the best ones. 💩

Eric Ervin20:05:04

Show us your bits

v3ga20:05:37

What would you is the preferred way for organizing your projects? Component, Mount or Integrant. I’m familiar with the first two and I’ve just started looking over integrant. From the examples I’ve seen it feels like there’s a lot more wiring that we have to take on. I saw people wrapping individual handlers in web services. Is that a must or would you just wrap your routes as a whole? (def routes ….)

Alex Whitt21:05:57

@decim: +1 for Integrant, although I'm not a web developer so my opinion may not mean much. I use it to manage state in a very different kind of application, and I've found it to be really powerful. Also, @weavejester is a fantastic author and maintainer.

hiredman21:05:54

I like component, because I like the isolation and flexibility it gives me, and as a result I like to either not def routes as a global, or use a routing library that lets me define routes separate from handlers

hiredman21:05:37

(if you use compojure and def your routes as a global you then end up having to pass your whole system to all the routes which defeats the isolation and flexibility)

lilactown21:05:43

yes -1 for compojure. +1 for reitit

hiredman21:05:57

there are some newer less used routing libraries like bidi (or maybe reitit) which let you define routes separately from handlers, so you can def you routes as a global, and have your handlers setup via componets

Mno21:05:13

I like compojure… 😞

Ivan Koz21:05:24

reitit and bidi are the way to go

v3ga21:05:43

@hiredman hmm ok thats something to think about. i’ve only passed routes as a global. i didn’t think of that. i’ve been toying around with pedestal and looking at reitit. Something tells me I may have more pleasure going with compojure though.

hiredman21:05:00

compojure is ok, but it combines routing and handling in an opaque way which can be annoying

lilactown21:05:43

I like reitit’s error messages and matching better

v3ga21:05:46

reitit seems nice

hiredman21:05:58

because compojure just has functions you can't query to ask it for the handler for a given route, or if a route is valid, or if a route exists for handler

hiredman21:05:06

all you can do is invoke it on a request

hiredman21:05:47

but you can use compojure with component, I would just try and stay away from def'ing the routes as a global

v3ga21:05:31

hmm, i may stick with pedestal but i’ll have to make the shift from using a global. I just haven’t seen that in the examples, to my knowledge.

hiredman21:05:34

even if you do insist on def the routes globally, you can still use component, we have a few projects at work that do that, I just prefer a more tightly scoped style

hiredman21:05:18

there is a routes macro that works just like the defroutes macro, but without the def

hiredman21:05:54

that is for compojure, not sure about pedestal

v3ga21:05:07

well i guess i’ll stick to component then.

seancorfield21:05:08

I'll +1 pretty much everything @hiredman said. Although we use Compojure in nearly all our apps, it does have some significant downsides. It is "easy" rather than "simple". We use Bidi in one app and I'm ambivalent about that. I may look at reitit at some point.

👍 8