Fork me on GitHub
#clojure
<
2018-02-01
>
gzmask05:02:17

Hey folks, can we have a clojure 1.9 library inside a clojure 1.8 project?

andy.fingerhut06:02:02

What makes it "a clojure 1.9 library"?

andy.fingerhut06:02:36

If it uses features of Clojure 1.9 that were not in Clojure 1.8, then it might be a bit tricky.

seancorfield07:02:18

Via Boot pods maybe?

hawari08:02:24

Hi guys, so I'm trying to create a web service for work. Basically what this service does is that it processes request from external clients and when doing so, it communicates with other services. Some request can have as much as 5 services to communicate with in order to be processed. There are things I'd like to get advice for, since I'm new to clojure and don't have "production grade" experience with it. How would you guys reuse HTTP connection? clj-http has a connection manager, but I'm not sure on how to use it properly, since it state that it needs to be shut down after use. Is the resource (I don't know, maybe socket or file descriptor) is forever going to be unrecycled even if the java jar process has stopped? I apologize if this sounds trivial, as my understanding of Java is pretty shallow.

noisesmith15:02:05

no resource is unrecycled outside the jar process - it uses a thread pool, and an explicit shutdown of java will close that pool. IIRC it will also be shut down if you explicitly call (shutdown-agents) but it should be easy to find out for yourself

noisesmith15:02:33

there's no mainstream non-broken OS where you can have a process fully exit and deny the usage of a socket to other processes

hawari03:02:28

Alright, thanks @noisesmith, I was afraid I need to setup some mechanism to gracefully close all resources when the process is signaled to stop.

qqq11:02:55

(defn foo [expr]
      (case expr
        'a 1
        'b 2))
this doesn't work as it's an 'duplicate quote'

qqq11:02:06

is there a way to match on clojure symbols ?

rauh11:02:39

@qqq Just omit the quote

qqq11:02:30

@bronsa @tbaldridge: (re discussion weeks ago on jvm / bytecode) Here's a minimal basic "dsl -> bytecode" supporting 'variable names' instead of [:iload const]

(do "* gen-func" 

    (defn vks->kv [m]
      (into {} (for [[v ks] m, k ks]
                 [k v])))

    (def jvm-field-size (vks->kv {2 [:long]
                                  1 [:int :float :farr :iarr]}))
    
    (do "** vars->locs"
        (defn vars->locs [vars]
          (second (reduce (fn [[offset mp] [nm kw]] 
                            [(+ offset (get jvm-field-size kw))
                             (assoc mp nm [kw offset])]) 
                          [1 {}] vars)))

        (vars->locs '[[a :long] [b :int] [c :float] [d :long] [e :farr]]))

    (defn cexpr->bc [vars] 
      (fn me [expr]
        (cond
          (number? expr) [:ldc2 expr]
          (symbol? expr) (let [[kw offset] (get vars expr)]
                           [(case kw
                              :long :lload)
                            offset]) 
          (seq? expr)    (case (first expr)
                           do      (vec (map me (rest expr)))
                           l+      [(vec (map me (rest expr))) [:ladd]]
                           l*      [(vec (map me (rest expr))) [:lmul]] 
                           lreturn [:lreturn]
                           ))))

    (defn cfn->bc [[nm inputs ret locals & body]]
      {:name (str nm)
       :desc (conj (vec (map second inputs)) ret)
       :emit ((cexpr->bc (vars->locs (concat inputs locals)))
              (cons 'do body))})

    (defn lst->bc [lst]
      {:name    "my.dyn.TestClass"
       :fields  []
       :methods (mapv cfn->bc lst)}
      )

    (defn gen-obj [lst]
      (let [co (ic/visit (lst->bc lst))
            obj (ic/new-instance co)] 
        obj))

    (def code '[(test0 [(a :long) (b :long)] :long
                       []
                       (l+ (l* 2 a) b)
                       (lreturn))])

    (.test0 (gen-obj code) 2 3)
    
    )

gfredericks12:02:11

is there yet a clojure reader (tools.reader?) that can read cljc conditionals unevaluated?

qqq12:02:50

I believe tools.reader can do that

qqq12:02:40

someone helped me find the option flag sometime ago, but unfortunately, I can't find it right now

qqq12:02:51

but there exists an option for you to specify :clj, :cljs, or unevaluated

bronsa13:02:35

:read-cond :preserve

bronsa13:02:57

works on both tools.reader and clojure’s reader FWIW

bronsa13:02:15

(`(read {:read-cond :preserve} in)`)

gfredericks13:02:41

okay cool thanks

pfeodrippe13:02:38

Why would a function like this throw

Caused by java.lang.IllegalStateException
   Attempting to call unbound fn:
   #'my-project.utils/build-event-stream
(defn build-event-stream
  [{:keys [builder input-topic output-topic check-fn prepare-fn schema schema-registry]}]
  (->
   (.stream builder input-topic)
   (kf/map-values avro->clj)
   (kf/group-by-key)
   (kf/aggregate check-fn
                 {}
                 (Materialized/with (Serdes/String) (EdnSerde.)))
   (kf/to-stream)
   (kf/filter event?)
   (kf/map-values prepare-fn)
   (kf/map-values #(map->avro schema %))
   (kf/peek #(println %1 " - " %2))
   (.to output-topic (Produced/with (Serdes/String) (build-avro-serde
                                                     schema-registry)))))

gfredericks13:02:22

is it trying to call build-event-stream somewhere that's not visible here? look at the stack trace to see where it was calling from

pfeodrippe13:02:59

I'm calling like this

(u/build-event-stream {:builder builder-frequency-events
                       :input-topic new-electric-data-topic
                       :output-topic frequency-events-topic
                       :check-fn check-frequency-event
                       :prepare-fn (u/build-prepare-function
                                    [:read-at
                                     :customer
                                     :freq-t
                                     :freq-a
                                     :freq-b
                                     :freq-c]
                                    [:freq])
                       :schema sch/frequency-event-schema
                       :schema-registry schema-registry-url})

pfeodrippe13:02:49

All of these are map values are well defined

pfeodrippe13:02:55

Even when I call like this

(u/build-event-stream {:a 3})

pfeodrippe13:02:10

It gives me the same exception

pfeodrippe13:02:06

If I load the build-event-stream namespace buffer manually with cider, it works, but in a new restart, it doesn't

pfeodrippe14:02:13

@gfredericks Working now, was related to some imports... java at the way, thanks!

grav18:02:58

I have a program that fetches some data from an SQL server and does some transformation on it. What’s the best way of testing both the querying and the transformation? Currently I have some clojure.test tests and a fixture like (use-fixtures :once fetch-data-and-store-in-an-atom) and some deftests that use the atom. I’d then execute it with something like DB_PASSWORD=1234 lein test.

alexk19:02:43

unit tests aren’t unit tests if they rely on an external system; consider mocking out the database calls and purely testing your code without involving a db

alexk19:02:05

But ignore that if you are past that decision and really want to know how to involve the db

joelsanchez19:02:44

assuming there's a function that fetches the data from the database, mock it with with-redefs and have it return data that resembles real data

joelsanchez19:02:20

if you want to access the db that's not an unit test but it is an integration test, so...valuable but not the same

tanzoniteblack19:02:44

we use postgres in production, and for our unit tests we make use of https://github.com/opentable/otj-pg-embedded which downloads & runs a real postgres server on the machine running the tests. Just have to set up the tables in the initialization. Don't know if something similar might exist for SQL server?

seancorfield19:02:06

FWIW, we use Docker locally to run MySQL, Redis, MongoDB (and a custom search engine) to support our test suites -- with DB migration scripts to tear down / set up test data. But @alex438 is right that it's not really a "unit test" if you're involving a DB in your test.

seancorfield19:02:39

In some of our tests, we have a protocol covering the storage operations that are required and then we mock the whole thing for tests. That works for a specific API that is implemented underneath as a set of queries and updates, but doesn't support arbitrary queries. And you potentially still need some level of integration testing or UAT to ensure it all works with a real database.

grav20:02:12

@alex438 Yes, they definitely are not just unit tests. They are supposed to test data transformations on the real data.

alexk20:02:39

What about downloading a sample of the real data and storing it in a test resources directory in your project, that your unit tests could then load when mocking out the db connection?

grav20:02:45

@alex438 Yeah, so have a separate process fetch the data and then run the unit tests. Sounds like a good idea.

grzm20:02:56

@seancorfield I’m seeing some funkiness with boot-tools-deps and Metosin’s alt-test. I’m digging in a bit, but before I get too far in, are you aware of any issues?

seancorfield20:02:32

@grzm Can you be a little bit more specific please? 🙂

seancorfield20:02:20

(in particular, boot-tools-deps assumes you are using deps.edn files instead of setting :dependencies in build.boot, and that you are using aliases for management of dependencies and classpath stuff, rather than leaning too heavily on :scope in dependencies)

grzm21:02:30

@seancorfield I’m seeing errors like “java.lang.RuntimeException: Unable to resolve symbol: deftest in this context”,

seancorfield21:02:05

In other words, the use case for boot-tools-deps (that I need to explain better in the README!) is that you have a project that "runs" fine with clj first -- and you use boot-tools-deps to add more Boot functionality on top of that.

grzm21:02:15

One of the things I liked about boot-tools-deps was that I could keep boot-related dependencies in build.boot and out of deps.edn. That doesn’t seem to be the case anymore.

seancorfield21:02:25

@grzm You'll need to share a lot more detail about your setup and exactly what you're trying to do... Feel free to do that in either #tools-deps or #boot depending on which part you think you're having problems with.

grzm21:02:36

oh, sure. I’ll put up a repo.

seancorfield21:02:29

That would be perfect, thank you!

bhauman21:02:41

a preview of a readline lib I'm working on

schmee22:02:58

dang, that looks great! 😮

qqq22:02:12

how do I get one deftest to call another deftest, i.e. I'm trying to setu pa hierachy of tests, and have one deftest call all the 'smaller' tessts

qqq22:02:16

(ct/deftest test-jvm-misc 
      ;; ireturn
      (ct/is (= 42 (.go (rbc [:int] [[:ldc 42] [:ireturn]]))))
      ;; iload
      (ct/is (= 2 (.go (rbc [:int :int] [[:iload 1] [:ireturn]]) 2)))
      ;; goto
      (ct/is (= 20 (.go (rbc [:int] [[:ldc 20] [:goto :END] [:ldc 30] [:iadd]
                                     [:mark :END] [:ireturn]])))))

    (ct/deftest test-jvm-arith 
      (ct/is (= 50 (run-arith :iadd 30 20)))
      (ct/is (= 10 (run-arith :isub 30 20)))
      (ct/is (= 600 (run-arith :imul 30 20)))
      (ct/is (= 1 (run-arith :idiv 39 20)))
      (ct/is (= 19 (run-arith :irem 39 20))))


    (ct/deftest test-jvm
      ;; how do I call the other two tests
      ;; i.e. test-jvm-misc and test-jvm-arith 
      )

tanzoniteblack23:02:02

@qqq deftest just generates a function with the name you give it, so you can just call (test-jvm-arith) and it will work. However, you probably don't want to do this in a test suite, since clojure test runner looks for functions with the testing metadata and runs them, so when you run all the tests, you would run test-jvm-misc, test-jvm-arith, and then run both of these again when test-jvm is run again.

tanzoniteblack23:02:44

The is macro doesn't have to be inside a deftest, so you might consider just making test-jvm-misc & test-jvm-arith normal functions, which are called inside test-jvm, so the tests only get run once but are still organized in a way you're happy with?

qqq23:02:56

okay, so the answer is two parts: 1. the solution is

(deftest test-jvm
  (test-jvm-misc)
  (test-jvm-arith))
2. I don't want to do the above since (test-jvm-misc) will be called twice (atleast)

tanzoniteblack23:02:45

Yep! But like I said in the second response, you can just make test-jvm-arith & test-jvm-misc normal functions, then they'll only be tested once when test-jvm is run with the test runner

qqq23:02:47

okay, so the 'solution' is:

(def test-jvm-arith
  (ct/is ...)
  (ct/is ...))

(def test-jvm-misc
  (ct/is ...)
  (ct/is ...))

(deftest test-jvm
  (test-jvm-arith)
  (test-jvm-misc))

tanzoniteblack23:02:16

@qqq instead of def test-jvm-arith do defn test-jvm-arith (and same for test-jvm-misc)

tanzoniteblack23:02:44

of you do it as def, it will run the is macro when you compile the code 🙂

qqq23:02:02

oh right, good call, def -> defn

tatyana23:02:17

I'm using leiningen version 2.5.3 and I originally installed it using brew. One of my projects is asking me to upgrade to version 2.7.1. I tried 'lein upgrade' but it gave me an error:

lein upgrade
This version of Leiningen was installed with a package manager, but
the upgrade task is intended for manual installs. Please use your
package manager's built-in upgrade facilities instead.

tatyana23:02:11

I then tried to run brew install leiningen but it responded with:

Updating Homebrew...
==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
  

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Error: Failure while executing: git config --local --replace-all homebrew.analyticsmessage true
Error: leiningen 2.5.3 is already installed
To upgrade to 2.8.1, run `brew upgrade leiningen`

tatyana23:02:24

but I don't need 2.8.1, I need to be on 2.7.1

tatyana23:02:26

How do I do this?

JJ23:02:33

why not brew update as the message says?

JJ23:02:21

download leiningen from their website and put the lein script in your path

JJ23:02:34

delete the brew one first

tatyana23:02:07

how do I delete the existing version?

noisesmith23:02:32

lein is a shell script on your path, plus a cache under $HOME/.lein/

noisesmith23:02:28

you can tell lein to change versions without deleting or uninstalling it with lein upgrade or lein upgrade specific.version (can be older or newer than current, it doesn't care)

tanzoniteblack23:02:25

@tlomberg brew uninstall leiningen should delete the existing one installed through brew; then install the lein shell script as specified on https://github.com/technomancy/leiningen#installation (and then you can follow the directions that @noisesmith gave you)

noisesmith23:02:43

@tanzoniteblack what I was trying to get at was that even if brew was totally broken and unfixable, it's sufficient to either delete a directory plus a file, or tell leiningen to change versions - but yeah definitely better to use brew as intended if using brew

noisesmith23:02:49

@tanzoniteblack @tlomberg re-reading, you don't need to uninstall the brew leiningen to get 2.7.1, it suffices to install 2.8.1 and then run lein upgrade 2.7.1