Fork me on GitHub
#clojure
<
2020-04-20
>
kwladyka08:04:10

(str "labels/templates/" label-template ".html") What is the safest and simplest way to make the path to the file while source of the label-template is untrusted?

danboykis11:04:41

@kwladyka do you know all the template names that label-template is supposed to refer to beforehand? if you do, i'd list all the know templates in directory labels/templates/ and turn it into a set. whenever you get a new label-template i'd check to make sure the untrusted label-template is contained in the trusted set before applying it

kwladyka11:04:45

@danboykis oh this is interesting solution. Thank you. But no way to make path and be sure it is valid for this directory? Hmm probably I can evaluate this path somehow to full path and check prefix after all as an alternative solution.

danboykis11:04:43

@kwladyka you can also have map of {`label-template` /path/to/template.html} and do a look up that way too

danboykis11:04:22

i think you can do fullpath checks, but it seems like a harder problem where you have to worry about thing like ".." and "." in your path and to make sure they're validated correctly

kwladyka11:04:41

I think there is a way to evaluate this .. and . to full path

kwladyka11:04:53

But I don’t have time now to check this 🙂

danboykis11:04:24

right i am sure it's possible, but IMO there are less moving parts that can fail in a static lookup process

kwladyka11:04:13

unless templates are not constant ;)

vemv12:04:31

in face of ambiguous classpaths (e.g. a clj ns is contained in two classpath entries), will one entry be picked deterministically? e.g. for the classpath string a:b, will a always be picked becase it appears first?

vemv12:04:06

note that this is more of a jvm question than a clj one... I'm not asking about how clj code gets loaded

hindol12:04:24

When you build the uberjar, which one takes precedence?

vemv12:04:54

I'm not building an uberjar... I'm asking about the semantics of the classpath string

hindol12:04:52

I was thinking the uberjar will apply the same precedence rule and you can figure out that way. But your answer is here (last paragraph): https://docs.oracle.com/javase/7/docs/technotes/tools/windows/classpath.html

Multiple specifications
To find class files in the directory C:\java\MyClasses as well as classes in C:\java\OtherClasses, you would set the class path to:

C:> java -classpath C:\java\MyClasses;C:\java\OtherClasses ...
    
Note that the two paths are separated by a semicolon.

Specification order
The order in which you specify multiple class path entries is important. The Java interpreter will look for classes in the directories in the order they appear in the class path variable. In the example above, the Java interpreter will first look for a needed class in the directory C:\java\MyClasses. Only if it doesn't find a class with the proper name in that directory will the interpreter look in the C:\java\OtherClasses directory.

vemv12:04:32

Got it. Yeah I had made some empirical observations but was seeking for some more certainty. Specification order in your link settles it. Thanks!

👍 4
solf15:04:35

Any recommendation for a date extractor library written in clojure or java? I found https://duckling.wit.ai/ but it's been deprecated (and crashes on the file I tested it with)

Nir Rubinstein15:04:24

java.time ns can pretty much do everything you need

solf16:04:50

You mean this? It's parsing functions seems fairly basic, I'm looking more for something that can extract all the dates (that might be formatted in different ways) from a text

Nir Rubinstein16:04:20

No, I mean basic java.time ns This: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String- Can accept multiple patterns like in this example:

DateTimeFormatter formatter = DateTimeFormatter.ofPattern("[yyyy-MM-dd][dd/MM/yyyy][MM-dd-yyyy]");
A full readthrough can be seen here: https://www.foreach.be/blog/java-8-date-and-time?lang=nl

hindol16:04:00

When using component, what is the pattern to pass dependency to normal functions (behaviours)? I understood the start/stop flow, but cannot figure out how to receive the component as the first parameter to other functions.

noisesmith16:04:46

typically your normal function is called via some call chain that starts with a top level component, eg. a main component that starts your app

nbardiuk16:04:29

we propagate dependencies through function calls. start function has access to all dependencies and it passes them as arguments down the call stack

Chris O’Donnell16:04:03

Here's an example app using component if it's helpful: https://github.com/seancorfield/usermanager-example

hindol16:04:51

So, I have a web server and the execution starts with an HTTP call.

hindol16:04:07

@U0DUNNKT2 Checking the link. Thanks!

👍 4
hindol16:04:33

Still cannot completely wrap my head around this. So, for a Pedestal ReST API, do I create an interceptor with pre-bound database and then inject to every route? (Currently database is just an in memory map, stored as a key in the systems map)

noisesmith17:04:45

I haven't used pedestal, but with ring, the http server would be a component, and a middleware would inject the db into the request context before the handlers are invoked

noisesmith17:04:53

(or at least that's how I did it)

hindol17:04:07

How do you define the handler that injects the DB? And if possible, some examples. I think I almost get it.

noisesmith17:04:18

with ring the function would be nested higher order: (fn [db] (fn [handler] (fn [rq] (handler (assoc rq :db db)))))

noisesmith17:04:41

so calling with db, returns a function, calling that with handler returns a new handler, calling that with request calls handler but passes in db

noisesmith17:04:04

I have heard pedestal can do things more elegantly, but I can't say how

hindol17:04:43

And you pass in the db and thus create the middleware in the "start" of the main app component? Where do you store this initialized middleware?

noisesmith17:04:13

you don't store it, it returns your handler at app startup

noisesmith17:04:52

there is no "initialized middleware" - the middleware takes a db and returns a function on handler, passing the handler to that f returns a new handler, that you use in place of the old one

noisesmith17:04:05

in the big picture, the goal here is precisely to offload "initialization" so that it only needs to be done to components, everything else just uses the data the initialized component passes in

noisesmith17:04:35

higher order functions and pedestal interceptors are two ways to integrate this sort of flow

hindol17:04:25

Still not very clear but I am taking too much of your time. I will look around the web some more. Thank you so much!

dpsutton18:04:31

i've enjoyed this in the past if you haven't seen it yet https://cb.codes/a-tutorial-of-stuart-sierras-component-for-clojure/

dominicm16:04:11

Someone once posted a clojure gui you could hook up for development for things like showing tables and performing actions on your application. Not REBL. You'd configure a miniature view for doing admin tasks. Does anyone know what that is?

💡 4
dominicm16:04:11

Neither of those. It was quite a while ago. Predates reveal by a long way.

noisesmith17:04:33

clojure.inspector is very old

dominicm18:04:38

It was third party. Younger than inspector.

dominicm18:04:09

You'd define things like a button to toggle something, or an input & button to set a value, stuff like that.

dominicm18:04:22

Or a function to fill a table.

nick02:04:12

@U09LZR36F if you ever find it please post it in this thread. Sounds interesting

cjsauer17:04:00

I remember hearing about a new clojure core function in the works that implements general paging/incremental loading of (remote) sequences. Does this ring a bell with anyone?

😍 4
ghadi17:04:16

@cjsauer that's the one

cjsauer17:04:28

Reading the patch. I really like this approach. I’m working on a project right now that this pattern keeps popping up. Looking forward to 1.11 metal

hindol17:04:27

When's 1.11 coming?

ghadi18:04:41

@cjsauer if you use it please send some feedback

👍 4
cjsauer18:04:26

Is there an async variant? I’m currently set up using go-loop from core.async with non-blocking HTTP requests, so iteration would need to be adapted in some way.

ghadi18:04:15

@cjsauer no promises, but yes we're considering an async variant that returns a channel of the iterated values

kenny19:04:29

I meant to tell you but totally forgot. We have copied the code for iteration and it's been great! We also wrote an async one, like the one you describe, and mostly use it.

ghadi18:04:47

with configurable "lookahead" buffer of pages fetched

ghadi18:04:58

so instead of [fetch page] [consume] [fetch page] [consume]...

ghadi18:04:29

you can:

[fetch page] [fetch page] [fetch page]
            [consume] [consume] [consume]

ghadi18:04:14

potentially more concurrent processing than traditional iterators

cjsauer18:04:33

Makes sense, cool! Is there a patch in the works for that already? Would love to see the code.

dpsutton19:04:43

anyone remember user-friendly UUIDs? I think @ghadi mentioned them once but its been a few years. got a friend who i think could use that

jrychter11:04:59

I use https://github.com/tonsky/compact-uuids which qualifies as kind-of user-friendly. I'm very happy with the results.

joefromct19:04:56

Hi, does anyone know of the correct way to escape special characters in a regex pattern being created from a string? I'm looking for something analagous to python's re.escape() and don't think i've found the best way quite yet.. i'd hate to re-invent the wheel.

noisesmith19:04:50

in a non-regex string literal, "\\" becomes "\", that's the main escape that clojure strings accept or recoognize

noisesmith19:04:23

ins)user=> (re-matches (re-pattern "foo\\.bar") "foo.bar")
"foo.bar"
(cmd)user=> (re-matches (re-pattern "foo\\.bar") "foo_bar")
nil

👍 4
noisesmith19:04:28

there's also the equivalent, but much more verbose \Q / \E pair

user=> (re-matches (re-pattern "foo\\Q.\\Ebar") "foo_bar")
nil

noisesmith19:04:29

that can be used on a full string - which might be what you want?

user=> (re-matches (re-pattern "\\Qfoo.bar\\E") "foo_bar")
nil

noisesmith19:04:20

I'd make a helper function that does (str "\\Q" x "\\E")

joefromct19:04:53

yeah ok that helps thank you. I think what i was missing was the extra stuff python does in that re.escape module... they have this stuff:

_special_chars_map = {i: '\\' + chr(i) for i in b'()[]{}?*+-|^$\\.&~# \t\n\r\v\f'}

joefromct19:04:07

just looking to port some python stuff to clojure. thanks.

dpsutton19:04:01

finally found it. they are called "proquint" i believe

👍 8
Nico20:04:27

I'm remembering I found a CLI tool that would render clojure templates and do other clojure site-generation tasks at native speeds with graal but I cannot find it again

Nico20:04:34

does anybody know what it was called?

Nico20:04:34

I found it (bootleg), dw

ghadi21:04:23

@dpsutton I've never seen Proquints but BIP-0039 is a pretty nice document https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki

ghadi21:04:23

proquints seem do have a disadvantage that they're not real words

ghadi21:04:23

so pronouncing those is weird

ghadi21:04:53

BIP-0039 splits a byte array into 11 bit segments and looks those up in a vector of 2048 words (2^11)

ghadi21:04:23

I had a clojure impl somewhere

nikolavojicic23:04:40

Is parallel transduce still a thing?

reborg06:04:46

I took this approach (which is opinionated especially for stateful transducers) to the problem https://github.com/reborg/parallel#ptransduce feel free to hit me with questions.

👍 4
jrychter08:04:05

I wrote a fair bit of code parallelized using core.async pipelines, and I was quite happy with the results. Using pipelines is slightly more complex , but not by much.

reborg09:04:50

Agree, you should use pipelines or reducers/fold whenever possible. However, the library above investigates an unification semantic for parallel stateless and stateful transducers (you can’t use stateful transducers with fold or core.async pipelines). We can then argue if that stateful parallel semantic is sound or not, but at least that’s an option.

jrychter11:04:07

Ok, that's intriguing — I thought stateful transducers didn't make any sense with parallel pipelines. In my case, I split my flows into multiple "pipes", where some stages are parallel pipelines, and some are channels with transformations (this is where stateful transducers sit).

reborg11:04:25

maybe “any sense” no, but “some sense” depending on context and semantic, perhaps. It’s definitely different from the sequential sense.

Alex Miller (Clojure team)23:04:35

Has not been implemented so I think my response there is still the state

👍 4