Sorry on vacation. When I get back happy to chat but not really interested in private discussion about stuff like this. Against the point really. But feel free to reach out when I return.

i think the best thing about clojurescript is that they actively use #cljs-dev and you can follow along with the enthusiasm that the developers have to their field. you can follow along with how issues are approached and you can see how priorities are set. i think this is drastically different from the way clj development happens

I'm a noob to clojure but am working through books and examples and I am trying out the whole proxy, gen-class, interface, defrecord, deftype, reify thing fight now and it appears to me that you can use proxy to override Java object methods OR implement interface methods on an anonymous object but NOT BOTH. Is this correct?

@funkrider If you're new to Clojure, you'll probably find #beginners helpful for "getting started" with any particular feature. As for proxy, according to the docs, you can supply both a class name and interfaces -- but only one class (as many interfaces as you want).

@funkrider As a silly example:

user=> (def p (proxy [java.io.InputStream clojure.lang.IFn] [] (read [] 123) (invoke [] "World!")))
#'user/p 
user=> (.read p)
123 
user=> (p)
"World!" 

(the latter is just shorthand for (.invoke p))

thanks sean! I get your example but why wouldn't this work? (defprotocol prot (foo [this])) (def p (proxy [java.io.InputStream prot] [] (read [] 123) (foo [] "World!")))

@funkrider My first reaction is "a protocol is not an interface"...

(you get clojure.lang.Var cannot be cast to java.lang.Class, right?)

omg (palmface) interface, protocol not the same thing... Thanks sean 😉 And yes thats the error I get

You can reify protocols and interfaces (but not classes)

boot.user=> (def p (reify prot (foo [this] "Hello") clojure.lang.IFn (invoke [this] "World!")))
#'boot.user/p
boot.user=> (foo p)
"Hello"
boot.user=> (p)
"World!"

BTW @funkrider you can use triple-backtick to format blocks of code (feel free to ask in #slack-help for hints & tips on using Slack).

OK so - is there a way to extend a java class, such that the class has some new private state and either option A) a new method on the class instance that has access to the state or option B) a new clojure protocol can be created and dispatched based on the extended class type that has access to this new private state within an instance of the class? Essentially I am working through a bunch of interview type questions in Clojure to compare use against say Java and the question goes like this: Create a new Stack class that implements push, pop and max methods and all operate in O(1) time. It is relatively easy in Java but in Clojure I am really having a hard time getting it done!

and gen-class doesn't seem like the "Clojure" way especially as it ends up being tied to a specific clojure version?

Thanks sean for the tip on formatting too!

for testing purposes I'd like to use with-redefs to replace a function with one suitable for testing. however, the project uses direct linking, so I will have to define the function that can be redef'd with ^:redef -- is the best way to get this directive out for production to elide-meta when i compile? or is there a much better way to do this?

preferably, a way to avoid direct linking when i'm running tests so i can avoid ^:redef altogether?

I'm writing a webapp where both server and client uses clj/cljs. I'm currently serializing/unserializing using str and edn/read-string. Is transit a drop in for this? Due to websockets, it wants / givbes me strings. However, transit seems to want to control it's own streams.

it’s not a drop-in, and you’re right, transit wants streams

but it’s not a pain in the ass

I’ve written this like 3 times, now, I should probably put it on clojars: https://gist.github.com/mattly/217eb6f26cb5d728a6cc88b4d6b926bb

new to clojure here trying to make heads and tails of documentation and tutorials etc and a bit lost. Hoping to test clojure for data science purposes. Using the following projects: boot, huri, clojure-csv, semantic-csv, neaderthal, kixi, bayadera in a gorilla repl with emacs (spacemacs) as my editor. 1) I want to know if anyone can point me to good resources to get more comfortable with clojure in data science? 2) bayadera is struggling to install: I use [uncomplicate/bayadera "0.1.0-SNAPSHOT"] ,but get could not find artifact error, can anyone give me some advice/help with this?

(->> [1 2 3 4 5 6 7 8 9 10]
     (map #(do (println %) %))
     (take-while #(> 3 %)))

Swap the map and the take-while?

@jeff.terrell the result of the map is important for the take-while predicate

@martinklepsch https://stackoverflow.com/questions/3407876/how-do-i-avoid-clojures-chunking-behavior-for-lazy-seqs-that-i-want-to-short-ci

@martinklepsch the short answer is, don't do it. Mixing I/O with lazy sequences is not a good idea. What are you trying to actually do?

You can "de-chunk" the sequence, but that's all a workaround that usually indicates you don't need to have this in a lazy sequence to begin with

@rauh @joshjones thanks, interesting points. I want to execute a potentially side effecting function f for each element in a sequence and abort if (f x) returns an “error” (error represented by a special keyword)

@martinklepsch I'd use transducers instead of dechunking

reduce + reduced is a good choice

;; condition to terminate is when input is 3, in this example
(reduce (fn [_ x]
          (if (= 3 x)
           (reduced x)
           (do (println "value: " x) x)))
        (range 20))

@joshjones yeah, thats a good suggestion, thanks

@pjones Welcome to Clojure! It lo ks like the author has not released that library yet -- I can't find it on http://clojars.org -- so you'd have to clone the repo and build it yourself. Since you're new, #beginners will be a helpful channel for you. Also #boot since you're using that.

There’s also #data-science which might be a good channel to hang around in.

Is there a "best practice" for creating non-global core-async pipelines? As an example, suppose I have a trivial async computation pipeline that looks like this: `(def in-chan (chan)) (def out-chan (chan)) (pipeline 16 out-chan (map (juxt identity (partial * 2.0))) in-chan) (go-loop [] (prn (<! out-chan)) (recur)) (defn inject [query] (go (>! in-chan query)))`

it feels like a bad practice to pus put that right at the ns level.

(def in-chan (chan))
(def out-chan (chan))
(pipeline 16 out-chan (map (juxt identity (partial * 2.0))) in-chan)

(go-loop []
  (prn (<! out-chan))
  (recur))

(defn inject [query]
  (go (>! in-chan query)))

sorry, bad formatting first time

option B could be:

(def in-chan (chan))
(def out-chan (chan))
(pipeline 16 out-chan (map (juxt identity (partial * 2.0))) in-chan)

(defn launch []
  (go-loop []
    (prn (<! out-chan))
    (recur))
)
(defn inject [query]
  (go (>! in-chan query)))

you decide to initialize when ready, but the pipeline sits at the ns level. Seems better.

In order to make inject work in-chan needs to be in scope, so unless the whole thing is wrapped in a lambda and inject were returned as a fn I don't see how to encapsulate it all.

for long text you may consider the #core-async channel fyi you still have global stuff, unnecessarily i think -- no reason you can't create the chans in a function along with setting up the pipeline, and pass those channels to functions which do something with them

ah, didn't even know #core-async existed. I'll pop over there.

I would opt for simple first; if your solution works as it is, I would stick with it. Otherwise you’re complecting things by creating “context” objects for no real benefit.

YAGNI or whatever. 🙂

I would avoid globals and opt for arguments

launching the pipeline belongs in launch... as well as construction of the channels

you have an infinite loop in the consumption function @markbastian . Need to check (when-some [v (<! chan)] (prn v) (recur))

when the channel is closed it will return nil repeatedly and infinitely.

cool, thanks for the tip.

(defn sink [ch f]
  (go-loop []
    (when-some [v (<! ch)]
      (f v)
      (recur))))

(sink ch prn)

@tbaldridge knows that function ^

hey i want to ask you about experience with grouping logs. For example in sentry there is fingerprint conception https://docs.sentry.io/learn/rollups/#customize-grouping-with-fingerprints . It works like ['foo' 'bar'] is grouping with others logs with the same values, but not with only ['foo']. Do you use it? What method do you have to group logs?

@markbastian I agree with @ghadi. Taking channels as arguments (instead of storing them in vars) is pretty essential to writing compose-able software.

Thanks for the tips. I figured that was the case, but I haven't seen any core-async resources online that explain the "right" way to use them. This is really helpful.

you're welcome mark -- generally you'll follow principles similar to what you'd use in any situation, i.e., prefer local state / params to global state

How do I get all the methods that are part of a protocol?

And hello everyone 🙂

I would like to create local bindings for all functions of a protocol against a certain instance of a record. So I want (to build) something that allows me to write (local-my-fn 123) instead of (protocol/my-fn impl/my-record-instance 123) for all my-fn in protocol/MyProtocol.

@seancorfield thnx, will definitely check that out 🙂

@chrisjd thnx, will add

On a different topic: I have a config.json, where I define which implementation shall be used for certain parts of my application. I start these parts using "mount", with (defstate my-part :start (condp = (:part config) "impl1" (impl1/->Part) "impl2" (impl2/->Part)). How would I do that without hardcoding all the implementations? Could I somehow make the different implementations for each Part register themselves? I played a bit with top-level forms in the different part's namespaces, which would update an atom somewhere, but that does not work, since the the code where the-atom-is-updated and where the-atom-is-used-for-lookup is not in the correct order.

@urzds a protocol is a map that contains its implementation

(defprotocol P (m1 [x y]) (m2 [a b]))
=> P
P
=>
{:on user.P,
 :on-interface user.P,
 :sigs {:m1 {:name m1, :arglists ([x y]), :doc nil}, :m2 {:name m2, :arglists ([a b]), :doc nil}},
 :var #'user/P,
 :method-map {:m2 :m2, :m1 :m1},
 :method-builders {#'user/m1 #object[user$eval2043$fn__2044 0x7087e497 "user$eval2043$fn__2044@7087e497"],
                   #'user/m2 #object[user$eval2043$fn__2057 0xbbcec2d "user$eval2043$fn__2057@bbcec2d"]}}

none of this is “public” api (you won’t find it in any docs), but it’s also not something I would expect to change

@alexmiller: So I could get all methods belonging to that interface by iterating over the :method-map? And with that list, I would write a macro, that generates the defines the local names / bindings?

yep

:sigs might be better

@alexmiller Thanks!

Hi guys, what’s the “modern” way of building http apis these days? I’ve tried liberator and yada, but want to get more exposure

tried compojure also

Yada and liberator are both fine. pedestal is also a good option. It's all a matter of tradeoffs. Liberator and yada have a resource abstraction but the first binds you to the ring api, the later to Netty. Pedestal has solid async support but you need to implement a lot of the http resource semantics yourself.

I’d like something with swagger, component/mount/etc and validation out of the box. nothing wrong with ring or netty imho, as long as I get an uberjar

I’m a fan of compojure-api

the swagger and component integration has been great; we had to do a lot of finagling to support async responses, though

I would recommend looking for (or writing) a language designed to be sandboxed like that

clojure isn't, so it will take a lot of work to do it

basically you'll need what amounts to a custom compiler

the clojure compiler will emit code that uses reflection at runtime when the compiler doesn't know the classes involved, so not only do you need to control compilation, but you need to be able to do your checks at runtime

ah, hmm

I presented a REPL for our Java services, and the team was uncomfortable with exposing dynamic code execution even in our pre-prod environment

I'm trying to find a good way to secure a REPL without requiring too much work

it's already behind HTTP security using drawbridge

@baritonehands Consider copying the socket REPL code in clojure.core.server and adding a TLS socket listener to it, and then maybe wrap clojure.main/repl with a simple auth mechanism (PSK)

nREPL is a whole lot of baggage

if its for trusted users, I wouldnt care about sandboxing - not that different from having access to prod db, otherwise I wouldn’t expose a repl. too many possibilities 😄

I built a system, where I can select which implementation of a Protocol to use based on a config file (using mount). Then I built multimethods to "register" these different implementations to certain config strings. Now the problem is that Clojure apparently never evaluates the namespaces where I define those multimethod implementations and hence I cannot load the Protocol implementations. Is there a better way to do such "component registration and selection at runtime" or is there a way to force Clojure to evaluate certain namespaces, even if they are not required from another namespace?

@mping Unfortunately that argument didn't work

@ghadi Has to be HTTP, so I don't have to expose any new ports

The problem with requiring the namespaces from the "core" namespace is that there would be cyclic dependencies, since those require the "core" namespace again.

@urzds Seems like you could require things dynamically, if you have the string of the full namespace

that was for reloading a namespace of an existing symbol, like my-ns/foo

@urzds this is pretty common, the problem is your multimethod/protocol definitions shouldn't be in your core namespace

they go in some other namespace

you multimethods/protocol defnitions are like interface definitions

@baritonehands That works:

(mount/defstate my-protocol
 :start (do
  (require (symbol (str "my-ns.my-protocol." (:my-protocol config))))
  (my-protocol/start config)))

do not do that, just structure your project well

I will, once I figure out how.

you defprotocols and defmulti's don't belong in your core namespace

your

hey, i am looking library like https://github.com/ptaoussanis/timbre but for business logs purpose. Unfortunately i can’t use timbre, because i use it for developer logs purpose. But i needs more or less the same logic for business logs purpose. Do you recommend something? Just i want push to something some data and add appenders to do something with this data.

@urzds create new namespace, typically ending with protocols or spi, and stick defmultis and defprotocls in there

then the implementation requires that, and core requires that

(and core requires the implementations)

@hiredman: They are in my-ns.my-protocol. The defrecords and defmethods are in my-ns.my-protocol.my-impl namespaces. The problem appears when I try to create the state from the config using (mount/defstate ... :start ...). I would like the ...my-impls to register themselves, so I don't have to hardcode the list somewhere.

"the problem" being what I described above: The implementations of the multimethod not being know, since the namespace was never required anywhere.

@hiredman: Before I tried the multimethod approach to selecting implementations based on a config file, I had exactly the core -> impl -> protocol setup, but ran into some issues: my-ns.core requires my-ns.protocol1.impl1 so it can parse / execute the config file and create my-ns.core/protocol1-impl (using either my-ns.protocol1.impl1 or my-ns.protocol1.impl2). my-ns.protocol1.impl1 requires my-ns.protocol2 so it can use that, but it also requires my-ns.core, so it can access the selected implementation at my-ns.core/protocol2-impl. Thus there was a cyclic dependency. I could not break that cycle by moving (mount/defstate protocol2-impl ...) into my-ns.protocol2, because that would also have created a cycle between my-ns.protocol2 and my-ns.protocol2.implX. So I tried to break that cycle using the "everyone registers themselves" approach, which also breaks, because now my-ns.core requires nothing but my-ns.protocol1 and my-ns.protocol2, and hence no implementation is ever registered / defmethod is never called.

@kwladyka Could you not just use a different config with timbre for business logging? See the GitHub page you linked under “Configuration”; you could easily wrap the log* / logf* macros in a “business logging” ns, for instance.

@kwladyka I think it depends on what your data looks like, what you mean by doing something with it, and when you want it done. You could use different log levels and scrape the data later, or you might actually need to use message passing (core.async, manifold, even kafka).

@baritonehands What are the use-cases of that java services REPL?

@yonatanel Just like a clojure REPL, I wanted to provide a programmatic interface to poke at anything, kinda like a remote debugger

you can grab stuff out of the IoC container, query the DB, really anything

@baritonehands We run a bare Socket REPL in one of our production web app servers. It starts up on an unusual-but-known-to-us port and it is only accessible via the loopback address via telnet directly on that server -- the port is not open, even within the server's DMZ. It's proven extremely useful for debugging over the years (it used to be an nREPL server but we switched to the built-in Socket Server a while back).

Given that once you're physically on the server, you could start a REPL from the command line and run any code you wanted (from inside the JARs deployed there), having the process-embedded REPL accessible via telnet 127.0.0.1 <port> doesn't seem any more problematic. What sort of objections have your colleagues raised?

privilege escalation?

@seancorfield Main objection was with devs running a REPL. It exposes more access than they currently have. SREs already have such access so it could be open to them. There was more concern over an accidental outage with good intentions than malicious use.

@bfabry Exactly, and much more terse explanation

Do devs have shell access to the production servers? I'm guessing not, in your case.

We do not

nor preproduction

The entire infrastructure was not designed with a REPL in mind. I think it would be possible to secure it, but may take some broad sweeping changes to the fundamental assumptions.

Right now I'm trying to find a way to get most of the features without the priviledge escalation

I agree that REPL access is pretty equivalent to full, unfettered shell access so if you don't have the latter, you shouldn't be given the former.

Given the power of the REPL, I'm not sure how you could ensure restricted functionality via a REPL-like connection. You'd easily have to write your own DSL and parser and interpreter I think...

...because restricting functionality would mean "not running Clojure code" -- it would have to be some DSL equivalent to a subset you wanted to allow.

clojurebot instance with a replacement for the irc ui?

I think a repl running under a user with only the privileges you want users to have would work?

we do have restricted access clojure sandboxes

clojurebot still allows you to run a lot of raw code tho'...

it does, yes

but if strangers on irc can use it…

@bfabry Not sure what you mean?

so like a separate clojure process that happens to be on the same machine?

yeah, and then regular unix permissions for access control

@noisesmith To be useful for debugging production issues you'd want SELECT access to any DBs, I'm sure, so the scope is pretty large. A separate Clojure process configured with only readonly DB credentials might be an option I guess.

@noisesmith I was thinking an actual REPL, pretty locked down in its own service, then a special interface for things it can do inside the other service

so you'd get a repl, data manipulation, utility functions

but only be allowed to do specific opt-in things in the other system

if so, clojail could be a place to start looking

but it sounds tricky to get right

You'd have to restrict which Java classes it could create, and which clojure namespaces were off-limits (clojure.java.shell, http://clojure.java.io, etc).

https://github.com/Raynes/clojail

@seancorfield For restricting Java access, a simple @REPL annotation could do

i would love to hear a follow up on how this turns out a month after it's put into place

I'd love to have a follow up

hopefully I don't get fired over this 😓

the utility of a repl is its generallity, once you start restricting what you can in it, it becomes less general

i echo @seancorfield 's recommendation for using a local socket REPL if possible

once you need to explicitlly allow everything you might as well just slap an api on it and access from a local repl

Alright, I'm off

I will play around with these ideas

thanks for the suggestions everyone

I love the clojurebot idea for this. Then these production investigations happen in public and knowledge gets shared with the team. Is this something you do on your team @noisesmith?

We have a socket repl, I thought of the bot as an example of solving, or nearly solving, the repl with restrictions issue.

When I need/want a sub-ns for spec, is there any particular reason not to just do:

(create-ns 'headmarc.rua.dkim)
(alias 'dkim 'headmarc.rua.dkim)
(create-ns 'headmarc.rua.spf)
(alias 'dkim 'headmarc.rua.spf)

lots of people do that, the only downside I'm aware of is lack of cursive support

why create the namespaces at all?

create-ns creates a namespace for holding code

spec namespaces are keyword namespaces

boot.user=> (alias 'foo 'foo.bar)
java.lang.Exception: No namespace: foo.bar found

the only place where code and keyword namespaces intersect is the '::' syntax, which is a tempting short cut, but a bad idea outside of playing at the repl

@hiredman why is it a bad idea? It’s pretty convenient when you have a bunch of large maps to type in.

lvh: because the names will change whenever you move the code around

the whole idea is stable names

(anyway yes that’s why; I understand how kw namespaces differ from regular namespaces)

yeah, type :dkim/foo then use your editors find and replace to change it to the real thing once you have typed all the maps

I would go further and recommend not using keyword namespaces that match a code namespace's name

spec names are like table.column names in a sql database, you don't name those after the code that reads or writes it

disagree with this. lots of keyword namespaces are good fits to be the same as a code namespace, though obviously not all. in the cases where it's not I still don't see a downside to having a real ns object for it and getting an alias

@lvh You can put the create call inline:

(alias 'dkim (create-ns 'headmarc.rua.dkim))
(alias 'spf (create-ns 'headmarc.rua.spf))

As for specs, I think there are cases where having them per @hiredman's suggestion is the right way and other cases where having them tied to actual code namespaces is the right way. It depends what they're a spec of and also what their intended scope is (public API, internal subsystem, opaque within a namespace etc).

@noisesmith how does the clojurebot solve the REPL with restrictions?

@baritonehands it provides a sandboxed repl where users have restricted permission - but others have brought up the complications that come into play if you need to access data or apis (things like the db for example) that the clojurebot isolation does not address

It seems it doesn't use clojail, how does the isolation work?

According to the Clojail readme, clojurebot does use it.

this seems to be the meat of it https://github.com/hiredman/clojurebot/blob/master/clojurebot-eval/src/clojurebot/sandbox.clj

which clojurebot?

@seancorfield I thought only lazybot used clojail

Ah, maybe I misread the README then...

,(clojure-version)

(isn't that how to invoke the bot here?)

I guess not...

the bot in slack, however it is invoked, is not that clojurebot

as far as I know, unless someone else is running and instance and wired it to slack

Anyways, @baritonehands You'd have to write some sort of pseudo-REPL that used Clojail to do the actual evaluation part -- so you couldn't just leverage the standard Socket Server REPL and Clojail together easily I don't think?

So it's /clj (some form)

The integration seems to be owned by @samflores and points to so I'm not sure which bot that is.

Oops!

I'll take that elsewhere...

OK, so whatever bot is attached to /clj does use Clojail...

Having it respond "clojurebot" when it's not @hiredman "clojurebot" is confusing 🙂

have to namespace the nicks

I miss IRC...

(Know that I will repeat this every few months)

IRC is still there! @devn