Fork me on GitHub
#clojure
<
2019-08-15
>
Cora04:08:06

is there a fast embedded database for clojure that persists to disk?

Cora04:08:30

derby maybe...

dpsutton05:08:16

@corasaurus-hex what do you mean by "for clojure"?

Cora05:08:16

well, I mostly mean simple to use from clojure and embedded in the runtime. I suppose there's h2 as well

Cora05:08:24

I considered rocksdb too, even though it's through jni. or sqlite

valerauko05:08:10

sqlite is "embedded" too isn't it

Cora05:08:24

sure, definitely

kah0ona06:08:36

What is the idiomatic way of doing a select-keys like operation to retain only the keys with a certain namespace? ie. (select-keys my-map "mynamespace")

andy.fingerhut06:08:38

Many libraries contain a filter-keys or similar function, including medley: http://weavejester.github.io/medley/medley.core.html#var-filter-keys that could be used to achieve that effect (and many others).

andy.fingerhut07:08:58

That plus calling the fn namespace on the keys to extract the namespace of the keyword, and comparing it to the desired namespace name as a string.

kah0ona08:08:40

thanks Andy

sogaiu08:08:35

is print-throwable the only thing that leads pr-str to have literal newlines in results?

mrchance13:08:12

Hey, is there an easy way to start a Clojure repl from a gradle project? I'm probably being blind, but all the things i found lead to long dead things, like Clojuresque

mrchance13:08:51

Cool, will try that, thanks

deadghost14:08:22

Is there a nice way to do something before running all tests(setup) and do another thing after the tests(cleanup)? I know use-fixtures can do this once per fn or once per ns.

alexmiller14:08:21

not in clojure.test

deadghost14:08:49

There's a global-fixtures thing. Not sure if circleci still uses clojure.

dpsutton14:08:38

thought i heard their front-end is moving away from cljs but i assume their backend is still clojure heavy

deadghost14:08:06

Yeah that's my impression as well

deadghost15:08:17

Looks like it uses clojure 1.8 and lein. Might not play nice with 1.9 and deps.edn

deadghost15:08:40

I happen to be looking at that right now

deadghost15:08:50

I don't see anything global-fixtures related

dpsutton15:08:03

oh yeah. it's on an old version of lein and that version is annoyingly prevented from upgrading. that wasn't fun

dpsutton15:08:20

- run:
          name: upgrade Lein to 2.9.0
          command: |
            wget -q 
            chmod a+x lein
            sudo mv lein /usr/local/bin

dpsutton15:08:45

(as opposed to yes | lein upgrade 2.9.0 or whatever)

ghadi15:08:47

@deadghost this is weird to say, but you can always write ordinary clojure code to setup fixtures, then run tests and tear them down

ghadi15:08:14

Clojure accepts a script file as input, can do arbitrary things in it

ghadi15:08:54

or fork the cognitect-labs/test-runner and add a fixture to it (it's 120LOC)

deadghost15:08:28

@ghadi like $ clj -m ns-to-setup-run-teardown-tests?

ghadi15:08:52

or clj dostuff.clj

ghadi15:08:04

can reuse all the test running machinery

deadghost15:08:25

yeah that can work

deadghost15:08:55

feels offbeat but I don't see why not

alexmiller15:08:46

happy to consider extensions to test-runner

deadghost15:08:17

@alexmiller sure, let me get a basic thing working before I veer off into test-runner

Suni Masuno18:08:33

Ok, this is kinda a weird question... More workflow than anything. I have an old repo made by someone(s) who left the company I got handed today. It has a cond in it that examines a network request for a version number than runs different versions of the same function based on the version... ie

(condp = (:verison request)
    1 (fun-v1 args)
    2 (fun-v2 args)
    3 (fun-v3 args)
    4 (fun-v4 args)
    5 (fun-v5 args)
    6 (fun-v6 args))
But only one system can call this, and it only calls v6. Anything else will error out the other systems. So I want to remove the code form the old versions. So far so simple. The problem... v1-v5 are often calling the same code, and often calling files with the same name in different folders, or functions with the same name in different files. Each version is effectively a different author from a different time in history, and they sometimes reused code and sometimes didn't. So here's my real question... How do you go about removing those functions that are depended upon by v1-5 without effecting those depended upon by v6? Can I generate some sort of dependency graph or something?

andy.fingerhut18:08:35

Not sure if there is some subtle problem with this idea, but you could start by removing the calls to fun-v1 thru fun-v5, then remove all other code except for funv6 and the call to it. Try to compile the code, seeing a flurry of errors about undefined functions. Add those back. Repeat until no such errors. Basically, the idea is "start from empty, add what is needed to compile, stop when it compiles".

andy.fingerhut18:08:07

The compiler is your checker for missing functions.

noisesmith18:08:24

#_ can be very helpful here

noisesmith18:08:46

use #_ to take out subforms, see if a fresh repl blows up, bring back the definitions needed, etc.

Suni Masuno18:08:16

That's a neato idea, I can even limit myself to #_ just the things that vX that I just removed is using

Suni Masuno18:08:34

There's still effectively some manual tree traversal, but not a tone

andy.fingerhut18:08:18

A different question to think on -- how did you reach the conclusion that no requests are arriving other than version 6?

Suni Masuno18:08:56

a LOT of digging. But the big thing is call control. This system will reject anything that doesn't have the id secret that only one other system actually has. And I have access to that other system's code and it is hard coded to v6. Asking some management it apparently once did more, but that functionality all got cut out a while ago.

Suni Masuno18:08:45

And watching logs, and checking the file write outs that this thing makes (which have the version attached) show yeeeears since any v other than the latest was used.

Suni Masuno18:08:16

Though I'll admit, final validation will come when I roll the system out without those endpoints and we see if anyone comes crawling outta the woodwork. I'll be damned impressed if they do, but I'll be ready with a rollback just in case.

ghadi18:08:51

you can also log metrics about the various v < 6 calls, and then retire the ones that have been 0 for a long time

andy.fingerhut19:08:48

wow. years of logs is pretty impressive evidence.

noisesmith19:08:23

@suni_masuno backing up to what you literally asked for in the first question, lein yagni will tell you which functions in your codebase don't have callers

🤯 2
noisesmith19:08:56

if you know which cases you can comment out / remove the yagni plugin will let you know which other code was "orphaned" and likely also safe to remove

noisesmith19:08:33

sadly it is dumb about multimethods and recoords

Suni Masuno19:08:08

This code was, mostly, the first clojure work for the people who've worked on it, so I think I'll avoid multimethods, though I can't be 100%. Still, yagni should help a TON! Thanks!

Suni Masuno20:08:49

Using yagni, got a question. What are the most common root functions/bindings that should be listed in a yagni init

noisesmith20:08:48

usually for a lib you put all your API functions in there, for an app just -main

bwstearns20:08:45

So I was just setting up my clojure env on a new machine and Sublime seems to have completely forgotten how to indent clojure for me. Has anyone else had this happen?

jrychter20:08:52

People using core.async, how do you implement error handling and completion status? Use a single channel with a convention/protocol on top, use multiple channels? For context: let's say I'm accessing a database asynchronously, and I will produce a stream of values, but somebody also needs to keep the connection open while I'm reading, and sometimes there will be errors/exceptions. Or, simpler context: I get handed a CompletableFuture. I know about manifold, but I'm trying to see if I can avoid pulling it in along with all its dependencies.

csm23:08:11

We usually put a Throwable object on the channel, and some throwing-take utilities from https://github.com/fullcontact/full.async

csm23:08:22

OR, we’ve been passing anomalies on channels (https://github.com/cognitect-labs/anomalies). This is somewhat “newer” for us, and I like it better

jrychter01:08:49

@UFQT3VCF8 Hmm. So you use a single channel for both data and exceptions.

jrychter01:08:44

Thanks for both pointers, these libraries are very interesting. I feel like we are missing something that CompletableFuture packages together: status of the computation (if it's been completed or not), possible exceptions/errors, and the (possible) result.

jrychter01:08:49

But — if you place anomalies on your channels, then you have to detect them somehow when reading from channels. Which probably means that your channels have a protocol (you encapsulate "normal" values), right?

csm02:08:10

yes, you do need to check if the value you get off a channel is an anomaly, if you use anomalies. I think it’s preferable for its simplicity, though (also, the domain of error types in anomalies is tiny, compared to the galaxy of different exception types you can encounter)

csm02:08:47

and as far as parity to what you get from CompletableFuture, I think manifold is the best option (we use that too). core.async is really best for doing CSP, not for “generic” async programming, whatever “generic” means

csm02:08:31

and, the best way to distinguish anomaly or not is just (clojure.spec.alpha/valid? :cognitect.anomalies/anomaly value)

jrychter16:08:20

Thanks for all your advice. The pointers are really helpful, though full.async does too much for my taste, I'd rather see a smaller library which is only about handling exceptions. As for manifold, I'm still considering it, but it is a large and complex library with multiple dependencies.

sogaiu22:08:49

@jrychter fwiw, there is also a #core-async 🙂

Benny kach23:08:06

Is there a reason to use transducers over reducers other than that with small collections or small operations reducers might have overhead (due concurrency) ?

noisesmith23:08:20

reducers are still a way of consuming a collection, while transducers are agnostic to data source

noisesmith23:08:36

(eg. you can't attach a reducer to a core.async channel)

Benny kach23:08:27

How can I attach a transducer to core.async channel? What does that mean?

noisesmith23:08:58

it's an optional argument to chan after the buffer, and it means that all values that come through the chan are processed by the transducer

noisesmith23:08:31

eg. core.async used to have >map for mapping on channel values, that's gone now and instead we use (chan N (map f))

noisesmith23:08:19

it does the same thing, without having to rewrite map for every context of processing data (same with filter, partition-all etc.)

noisesmith23:08:39

so reducers are a strategy for parallel computation, transducers are a way to describe modular units of data processing abstracted from the source of the data

noisesmith23:08:42

I assume if reducers were invented more recently, they would have used transducers