Fork me on GitHub
#clojure
<
2020-01-08
>
hieu.luu07:01:23

I'm having some difficulties grasping Manifold streams & Aleph networking. As I understand, you have sinks and sources, which basically means "something-to-put-into" and "something-to-get-data-from". Say I want two different clojure program to communicate via a TCP port (many clients <-> one server). If I use manifold streams and aleph tcp, do I need to take care of: 1. Ordering of data sent/received? 2. Integrity of data sent/received? I get that a lot of stuffs has been abstracted away so I just need to have a stream from the source (client) then I send it into an aleph tcp-client, then on the server I (may or may not?) get the data that was sent. What I don't get is: do I have to take care of ensuring if data was sent fully and in the right order (1&2), what if in the middle of a client sending data, another sends in data and the server gets mixed up between the two (1&2).

hieu.luu08:01:13

In the examples, Gloss is used to ensure a protocol is agreed between clients and servers (e.g. what comes first and what comes last). But if data is fragmented into tcp packets, do I need to ensure protocol is neatly divided into tcp packets, or does it get taken care of already?

hieu.luu08:01:13

finally, what/how can one google/learn from documentation about what a certain java stream objects can be converted into in clojure? This is where I usually stumble because of juggling between Java Documentation and (pure?) Clojure data definition --- what's the right way to look into a returned data and know what it is, if it's java or not, ... from a repl point of view ?

andy.fingerhut08:01:55

I don't know much about Gloss, manifold, or aleph, but I do know that TCP is a byte stream service, so you should never need to worry about how bytes are broken up into packets, or about the ordering of bytes within a single TCP connection (they are guaranteed to be delivered to the receiving application in the same order they were sent). If any of these libraries handle automatically reconnecting a TCP connection that becomes broken, there might be ordering issues between different TCP connections that are yours to deal with, i.e. I know that TCP has no ordering guarantees between different connections.

hieu.luu08:01:03

Thanks andy. I infer from what you mean about "single TCP connection" meaning (by design of TCP) two different TCP connections would (have to) be distinguished if any tcp library is to be taken seriously (i.e. not aleph but if X purports to be a TCP library, it should by definition handle what you said and this)

andy.fingerhut08:01:00

Some application level messaging libraries try to "hide" some details of when TCP connections are established, broken, and perhaps automatically reconnected for you, under the layer of the API. If so, whether there are ordering guarantees across connections breaking and being reconnected is something you would need to find out about from that library's documentation. TCP by itself cannot do anything to guarantee that.

hieu.luu08:01:23

alright. I understand what you mean. 😀 it's the asynchronous tcp that makes my head hurts a bit. I'll read more into the library itself

pez08:01:41

I find this discussion on if/when/how to use comp being very enlightening: https://clojureverse.org/t/to-comp-or-not-to-comp/ I also find that I can read it several times and find new pieces to the, yet tiny, Clojure and functional puzzle that is slowly assembling in my head.

plexus09:01:53

@pez I'm starting to regret having instigated that discussion.

pez13:01:36

Haha. Well, I'm happy you did!

Santiago09:01:24

Has anyone used clojure to verify if an email address is valid and exists on a server?

robert-stuttaford09:01:22

The only sure way is to send it email and receive data via a new http request - e.g. a verification link. it's also the simplest and most secure.

dominicm11:01:43

There is another way

dominicm11:01:18

Weavejesters library of validator predicates does something funky that sounds like what you want.

dominicm11:01:44

It doesn't guarantee who the owner is, but I think it gives you a good idea that it exists.

Santiago12:01:25

yeah, I don’t want a lot of info about the email or the person behind it, just “does it exist on the server?”

Santiago12:01:05

I’ll look into it, thanks @U09LZR36F, also @U0509NKGK for chiming in 🙂

Santiago12:01:47

yes, the library has a predicate valid-email-domain? that takes a step in the direction I need, nice 😄

lukasz14:01:06

there are also services like clearbit or http://kickbox.io

dominicm16:01:15

I'm confused about what kickbox is actually selling. One could interpret their marketing to suggest that they will email everyone in my mailing list and ask if they are real. I'm sure they aren't, but still.

lukasz16:01:30

@U09LZR36F you can use their free API to verify if the email address is valid, disposable (mailinator etc) or coming from a free account (gmail, yahoo). If you just want to know that the email address can receive messages - then your only reliable solution is the verification link

Santiago09:01:48

I've seen bash code using nslookup and telnet

Akash13:01:04

I am experiencing a problem with with-redefs in my clojure test. Here is my setup namespace a -> requires namespace b and has a function foo namespace b -> requires namespace c namespace c -> has a function bar Test namespace a-test I want to test function foo but I want to redef bar Here is my implementation

(ns app.a-test
  (:require [app.a :as a]))

(deftest foo-test
  (testing "retuns something"
    (with-redefs [a.b.c/bar (constantly true)]
      (is (= true (foo "test"))))))
However, here still the actual bar is called! What am I missing here. Please advise

Drew Verlee14:01:59

I wasn't even aware that a.b.c was valid way to access a ns.. Why not just require the c namespace directly e.g (:require [app.c :as c])

Drew Verlee14:01:45

I'm not sure thats the issue, as i'm also not familuare with with-redefs.

dpsutton14:01:57

wow. i assumed that was just standard prefix for an actually imported namespace

dpsutton14:01:08

i'm surprised that's not an error

dpsutton14:01:03

well for me something like that is definitely an error

dpsutton14:01:27

(with-redefs [a.b/bar (constantly true)] (+ 1 1)) throws when a.b/bar can't be resolved

Akash14:01:31

The problem is even requiring c won’t work here. I tried that as well. It still calls the bar method.

Akash14:01:30

I’m not sure why

dpsutton14:01:35

is there any async, multiple threads stuff going on here?

p-himik14:01:22

I'm not that familiar with with-redefs but it feels like the issue can be in the app.a namespace. Can you give the code where foo is defined and bar is used in there?

Akash14:01:03

(ns app.a
  (:require [app.b :as b]))

(defn foo [user provider]
  (let [params {:user user :provider provider}]
    (b/sample params)))


(ns app.b
  (:require [vault.core :as c]))

...
...
(defn sample [params]
  (c/read-secret (client) params))

Akash14:01:25

I want to test foo and mock the read-secret` function here.

Akash14:01:21

@U11BV7MTK There are no async functions. However there is a defmethod in it. Not sure if that causes any problem

noisesmith16:01:41

when you say "bar method" do you mean that bar is defined via defmethod?

Akash07:01:00

Yes, you are right @U051SS2EU

Akash07:01:17

Won’t that work?

p-himik08:01:43

Interested in it myself. I can't find why that wouldn't work - defmulti uses defonce internally, which still uses def.

noisesmith18:01:40

minimal repro that what @USFNA0VPH wants does work - I suspect there's something else going on

Clojure 1.10.1
(cmd)user=> (ns c)
nil
(cmd)c=> (defmulti bar :type)
#'c/bar
(cmd)c=> (ns b (:require c))
nil
(cmd)b=> (defmethod c/bar :x [m] (:y m))
#object[clojure.lang.MultiFn 0x5b43fbf6 "clojure.lang.MultiFn@5b43fbf6"]
(cmd)b=> (defn foo [x] [(c/bar x)])
#'b/foo
(cmd)b=> (ns a (:require [b] [c]))
nil
(cmd)a=> (with-redefs [c/bar (constantly true)] (b/foo {:type :x :y 42}))
[true]

Akash08:01:45

I scratched my head a lot to understand what’s going on here. Couldn’t figure out why It was not working. Finally ended up using the mock client from vault for testing instead of with-redefs

jjttjj16:01:59

Anyone using https://github.com/ptaoussanis/sente ever have the desire to make a request on the client with a callback that expects multiple responses? ie to call ?reply-fn multiple times on the server? This is not currently supported (send messages with callbacks only support a single reply). I realize I can replicate this by implementing my own callback id machinery, but I sort of wish I could piggyback of sente’s internal calback ids. Is there a good way to deal with this that Im missing?

p-himik16:01:36

I have something like that - for some operations, I need to report progress. And yeah, I've done pretty much what you describe, and the server calls a special RPC called :on-progress or something like that.

jjttjj16:01:39

Good to know. I suppose the main reason not to have this functionality offered directly by sente is that it would then need to expose a means to unregister a callback manually and it doesn't want to support that additional complexity?

p-himik17:01:17

You should ask Peter Taoussanis, not me. :) Maybe it's something that could be implemented in a way that would benefit all, I don't know. But right now it seems that there can be too many corner cases where each project would require something special.

Ben Grabow16:01:59

I'm working with Redis in my application, and I'd like to have an in-process mock of my Redis DB for development and testing purposes. Right now I have a protocol with my common DB access fns, and two places I reify the protocol: one with the real Redis impl and one with a clojure map in an atom as my "db". I'd like to be able to easily inspect the whole map, but since it's part of the reify closure I'm not sure how to easily get at it. Is there a better way to do this?

Ben Grabow16:01:50

I thought about using defrecord but since the fields are immutable I think that excludes using my map in a mutable atom?

noisesmith16:01:03

you can store an atom inside a field in a defrecord, and then implement the protocols via updating / accessing that field

lukasz16:01:17

I had exactly the same problem in the past - and it was not feasible to create a protocol which implements all possible redis functions. So I wrote a Redis component, which works a bit more like the redis CLI/protocol: https://github.com/nomnom-insights/nomnom.omega-red

noisesmith16:01:28

it's slightly brittle as you'd need a wrapper around the default constructors to ensure an atom is created, but after that it would just work

lukasz16:01:48

With it creating a mock is pretty simple: you create a record which implements the Omega Red protocol and store the state in an atom

Ben Grabow16:01:43

Omega Red looks a little heavy for what I'm looking for at the immediate moment but I'll definitely check it out for future work, thanks!

Ben Grabow16:01:26

@U051SS2EU I control enough of the calling code that I can just pass in (atom {}) to the constructor each time if I need to. Will that work?

lukasz16:01:43

Sure thing, I meant it more as a suggested approach - we use Redis quite heavily in different forms so it made sense to create a Component for it

noisesmith16:01:51

that works too, yeah - simple demo coming up

Ben Grabow17:01:27

(defrecord MockRedis [db] IRedis (blah... Later... (MockRedis. (atom {}))

noisesmith17:01:00

(ins)user=> (defprotocol Db (store [this k v]) (access [this k]))
Db
(ins)user=> (defrecord StubDb [storage] Db (store [this k v] (swap! storage assoc k v) v) (access [this k] (get @storage k)))
user.StubDb
(ins)user=> (def db (->StubDb (atom {})))
#'user/db
(ins)user=> (access db :some-key)
nil
(ins)user=> (store db :some-key 42)
42
(cmd)user=> (access db :some-key)
42

noisesmith17:01:15

@UANMXF34G just fyi it's pretty much always better to use the auto-defined constructor ->Foo rather than the interop constructor Foo. (but they do end up doing the same thing)

Ben Grabow17:01:00

Any specific gotchas when using Foo. or is it just a style thing?

noisesmith17:01:35

the function version is smarter about reflection, and less likely to hit some gotchas when using the record from another ns

Ben Grabow17:01:00

hmm, okay good to know, thanks

noisesmith17:01:02

to use Foo. you need to both require and import , ->Foo only needs require

Ben Grabow17:01:29

Yeah based on my reading so far I figured I'd have to tackle the import issue at some point so it would be nice to avoid needing that.

noisesmith17:01:01

also the auto-generated map->Foo is great when you have more than ~2 fields defined

Ben Grabow19:01:19

Is there a trick to getting this working when my protocol is defined in a different namespace than the defrecord is defined in?

Ben Grabow19:01:52

I get Unable to resolve symbol for the protocol method name when I call the method on a record instance.

noisesmith19:01:47

the protocol method belongs to the ns that defines the protocol

noisesmith19:01:05

so use that ns, not the record's ns, when invoking

Ben Grabow19:01:31

gotcha, I think that's working

Harold17:01:50

clj -A:graph is rad! Very helpful in reasoning about dep trees. On a related note, is it possible to get the clojure/`clj` cli tools to list 'dependency differences' and 'dependency conflicts'? • Some libs we're using are in disagreement about the best version of Jackson.

p-himik17:01:32

What's -A:graph? :graph is just an alias somewhere in one of your deps.edn. What does it specify?

mpcjanssen17:01:15

Maybe walmartlabs/vizdeps

mpcjanssen17:01:29

it can specifically only show conflicts and the origin

Harold18:01:45

@U0E2P47B7 😉 - that's the functionality I'm trying to recover in deps.edn land.

Alex Miller (Clojure team)18:01:41

I've been thinking about it. There are some challenges. The trace mode stuff for tdg does actually show some of this but it's not a very efficient way to consume that info.

Harold18:01:45

I'd love to be able to produce reports like this one automatically: https://github.com/aws/aws-sdk-java/issues/1108

seancorfield18:01:03

@hhausman -Strace will help you there.

seancorfield18:01:30

It produces a trace.edn file listing every dependency clojure looked at, which one it chose, and why.

Alex Miller (Clojure team)18:01:47

The raw info is in there but I've been thinking about ways to improve it. It's on my list of things to look at the next time I take a cycle through this stuff

Harold18:01:43

@seancorfield - that is nice, thank you. @alexmiller - for sure, super-grateful for your efforts so far. I can imagine tools that consume trace.edn and produce actionable insights.

Alex Miller (Clojure team)18:01:06

tools.deps.graph does, for example :)

simple_smile 4
Alex Miller (Clojure team)18:01:37

if you do something like clj -A:graph -t -o trace you will get a series of numbered images trace100.png, ...

Alex Miller (Clojure team)18:01:55

if you step through those, that's kind of a visual depiction of what it's trying at each step, what it chose, etc

Alex Miller (Clojure team)18:01:28

I've been thinking about how to collapse all of that info into one graph

Alex Miller (Clojure team)18:01:18

they might end up looking like that graph you posted in the thread

👍 4
Harold18:01:21

The little red lines that vizdeps had were appreciated in this regard. The classic "Possibly confusing dependencies found:" output from lein deps :tree was never as user-friendly as I'd like, but was another approach.

Alex Miller (Clojure team)18:01:19

make sure you try the new --size option too :)

Harold18:01:44

Ha! On our team I'm known for unzipping uberjars and doing ncdu ... 😛

p-himik19:01:14

Oh... TIL there's better life than constantly doing du -hd1 by hand.

Harold19:01:37

@U2FRKM4TW - then it was a good day.

bmo 4
zane18:01:55

Is there a benchmarking analogue to clojure.test? Perhaps one that makes use of Criterium? I'm looking to define suites of benchmarks, generate reports, etc.

butterguns19:01:38

This feels like a long shot, but I'd love clojure.tools.logging not to print the entire namespaced keyword. Like printing :f.b.b.q/customer instead of :foo.bar.baz.quux/customer. Now that I'm using spec, my logs are really dense with namespaces I don't really care about

andy.fingerhut19:01:23

You can change the data that your code sends to clojure.tools.logging calls...

andy.fingerhut19:01:29

alternately, make your own locally modified version of clojure.tools.logging that makes replacements in data sent to it, but changing what you send to it seems better in the sense that the replacements you want are specific to your application, it seems.

butterguns19:01:52

I just want to pass a map to a logging call, without having to transform it into a new map with different (un-namespaced) keys

didibus20:01:48

I think there's a flag to have it be using namespaced maps?

didibus20:01:08

It makes it a bit shorter, but won't strip out the namespace fully

butterguns21:01:07

Would love to know what that flag is, I couldn't find it on Google 🙂

andy.fingerhut21:01:05

Default behavior is this in Clojure 1.10.x:

user=> {:foo/bar 1 :foo/baz 2}
#:foo{:bar 1, :baz 2}

andy.fingerhut21:01:18

But it only prints that way if every key in the map has the same qualifier/namespace.

andy.fingerhut22:01:41

Whether you pass a modified map, or something changes the way the map is printed after you make the logging call, someone has to write that code, which I don't believe exists today. What if you have two namespaces/qualifiers that would become identical after abbreviation?

butterguns22:01:18

Indeed, it's not a perfect solution. It's really just to reduce log verbosity. It can be hard for a human to parse it visually when there's a lot of "meaningless" cruft in there.

didibus00:01:30

It defaults to false

didibus00:01:41

There's a chance making it true would also affect tools.logging

didibus00:01:14

@U0CMVHBL2 The repl binds it to true, but otherwise it defaults to false.

butterguns16:01:43

This is excellent. I think I see the problem though - if you have keys from different namespaces in the map, it falls back to printing the full namespace: Desired

user=> {:a.b.c/bar 1 :a.b.d/baz 2}
#:a.b{:c/bar 1, :d/baz 2}
Actual
user=> {:a.b.c/bar 1 :a.b.d/baz 2}
{:a.b.c/bar 1, :a.b.d/baz 2}
Oh well ¯\(ツ)

andy.fingerhut17:01:25

That is a feature, not a problem, for the people that wrote that code 🙂

andy.fingerhut17:01:58

They wanted something that could be read back in and know it was equal to what was printed. They did not have your use case in mind, because your use case loses information.

butterguns18:01:05

I'm not making any normative statements here

butterguns18:01:40

I was just curious if anything out there existed that matched my individual requirements. No big deal if not

didibus07:01:14

You could try extending print-method for map types

didibus07:01:30

(defmethod print-method clojure.lang.IPersistentMap [v ^http://java.io.Writer w] (.write w "{") (reduce-kv (fn [_ k v] (.write w (str (if (qualified-keyword? k) (str ":" (name k) " " v) (str k " " v)))) (.write w ", ")) {} v) (.write w "}"))

didibus07:01:38

something like that

didibus07:01:41

But that's gonna take effect everywhere, not just when logging

didibus07:01:24

Otherwise your best bet is to wrap the logging functions or have a util fn that strips out the qualifiers that you call on the maps before passing them to tools.logging

kaosko21:01:42

easy? one for you quys this time. I'm in the middle of clojurifying a big, rather serious, enterprisy java software. clj conventions seems to be not to use fully qualified package names (i.e. com.mycompany....). I can (re)name packages to whatever I want before I give it for wider consumption but afterwards, it will be a challenge. is there a reason not to use fully qualified package names?

bfabry21:01:34

you should leave the fully qualified packages you have

bfabry21:01:53

they're not as popular in the clj community, but that's an anti-pattern not a pattern. it's been mentioned a few times that generally we should

👍 8
kaosko21:01:27

yeah, that's kind of my reading on it, thanks. anybody else with other data points why I shouldn't keep em?

noisesmith22:01:47

I think it's fundamentally a conflict between common lisp aesthetics vs. java pragmatics

noisesmith22:01:28

a good compromise is to use a unique enough project name that it won't conflict, and otherwise company (if unique enough) or com.company

noisesmith22:01:50

related: foo.core is a weird convention, and arguably you should never have com.company.foo.core (and many people would even suggest avoiding foo.core and instead having a more meaningful multi-part namespace like company.foo)

👍 8
vemv22:01:13

The thing with core is that it can mean anything, which can lead to conflated namespaces if you split it in api, main etc, things can be more self-evident for consumers and maintainers alike

dvingo22:01:48

i have an error from AWS cloudwatch which is a string containing EDN. All the strings inside this EDN data structure are invalid clojure strings because they are backslash escaped. I'm wondering if anyone has a clever solution how to parse this as edn in order to pretty print it. Right now I'm just using some shell scripts to do it

andy.fingerhut22:01:38

Could it be that you can call read-string on it and it would give back Clojure data?

andy.fingerhut22:01:13

Shoot, never mind. If the "outer quotes" have backslashes in the text you have, read-string will not do it.

sparkofreason22:01:37

Question on tagged literals and metadata: Some tagged forms will have line/col metadata available, but anything that isn't IMeta will not. Is there any other way of accessing that info in the context of a data reader?

andy.fingerhut22:01:58

Clojure's built-in reader and the contrib library tools.reader have that limitation, I believe.

andy.fingerhut22:01:35

There are other libraries for reading Clojure data/code (perhaps with limitations of what they support -- not sure about tagged literal support) that return custom data structures that are not just the Clojure data alone. I don't recall all of them right now, but one is https://github.com/cgrand/sjacket

andy.fingerhut22:01:58

I have not used one for any serious use case to give feedback comparing any of them.

sparkofreason22:01:38

Interesting, thanks.

dvingo22:01:43

a sample is :

"{:msg \"error response triggered\"}"

andy.fingerhut22:01:40

If you call read-string on that one time, you get back a Clojure string. If you call read-string on that string, it would give you a Clojure map.

dvingo22:01:59

awesome! that did work

dpsutton22:01:25

user=> (clojure.edn/read-string "{:msg \"error response triggered\"}")
{:msg "error response triggered"}
user=>

bfabry22:01:42

this is calling the read functions twice technically 🙂

bfabry22:01:25

first in the Read from the REPL, then in the Eval

dpsutton22:01:37

am i missing something?

dvingo22:01:43

nope that's it

andy.fingerhut22:01:17

I think the argument to read-string that you showed in your sample was in a file, not already in the code in question.

andy.fingerhut22:01:33

but I could be misinterpreting the situation.

Ivan Koz23:01:11

https://gist.github.com/nxtk/7c646697240fff2833d104211c8f36c1 guys would you be kind to review my lazy iterative hanoi tower implementation, especially if i can do something about if odd-n? branch to make it more compact\readable algorithm explanation https://cs.stackexchange.com/a/96669

noisesmith23:01:22

using letfn just to make helpers private is not really idiomatic, for starters

Ivan Koz23:01:51

when to use letfn then?