Fork me on GitHub
#clojure
<
2019-12-11
>
datran04:12:23

Does anybody have any thoughts about relative merits of core.async, manifold, and promesa for asynchronous code?

datran04:12:56

I'm just learning about the topic and I'm wondering what the tradeoffs between the three are

Balaji Sivaramgari05:12:12

Hi Team, I was trying to setup the Clojure code analysis with Leiningen as build tool and Sonarqube as code analysis tool. I am getting the below error while executing the stage

10:53:36  05:23:35.933 ERROR: Error during SonarQube Scanner execution
10:53:36  org.sonar.api.utils.command.CommandException: .IOException: Cannot run program "lein": error=2, No such file or directory
10:53:36  	at org.sonar.api.utils.command.CommandExecutor.execute(CommandExecutor.java:102)
10:53:36  	at org.sonar.plugins.clojure.sensors.CommandRunner.run(CommandRunner.java:36)
10:53:36  	at org.sonar.plugins.clojure.sensors.CommandRunner.run(CommandRunner.java:50)

Balaji Sivaramgari05:12:35

We are executing this from Jenkinsfile

Balaji Sivaramgari05:12:58

And I am able to print the Leiningen version inside the workspace

Balaji Sivaramgari05:12:51

10:53:19  + ./lein -v
10:53:19  which: no lein in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin)
10:53:23  Warning: implicit hook found: lein-junit.plugin/hooks 
10:53:23  Hooks are deprecated and will be removed in a future version.
10:53:23  Leiningen 2.9.1 on Java 1.8.0_232 OpenJDK 64-Bit Server VM

seancorfield05:12:28

@balaji.sivaramgari Two possibilities come to mind: 1) the PATH for the process does not actually have lein on it, despite the shell that you are running having it or 2) the process does not have sufficient memory allocated to it to run Leiningen.

seancorfield05:12:24

Leiningen itself spawns two JVMs: one to run the initial command and then one to run the Clojure process that kicks off.

seancorfield05:12:16

(when I was first trying to run Leiningen on early builds of WSL on Windows, it would fail with error=2 because it wasn't allocating process memory correctly)

Balaji Sivaramgari05:12:13

sh "curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein > /tmp/lein" sh "chmod a+x /tmp/lein" sh 'echo "export PATH=$PATH:/tmp/lein" >> ~/.bashrc' sh "source ~/.bashrc" sh "echo $PATH" sh "/tmp/lein do cloverage" sh "/tmp/lein -v" this is the code snippet I used inside the jenkinsfile

noisesmith17:12:51

the PATH should contain the folder the executable is in, not the executable itself

noisesmith17:12:05

by calling /tmp/lein you are ignoring the path and directly executing a file

noisesmith17:12:18

if your path was correct you could just run lein

Balaji Sivaramgari08:12:21

@U051SS2EU I have tried as below. But still it is not accepting the lein sh "curl https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein > /tmp/lein" sh "chmod a+x /tmp/lein" sh 'echo "export PATH=$PATH:/tmp" >> ~/.bash_profile' sh "source ~/.bash_profile" sh ". ~/.bash_profile" sh "echo $PATH" sh "cat ~/.bash_profile"

Balaji Sivaramgari08:12:03

14:21:35  + cat /home/jenkins/.bash_profile
14:21:35  export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin:/tmp

Balaji Sivaramgari08:12:28

14:21:38  + lein do cloverage
14:21:38  /home/jenkins/workspace/Conflux_Clojure_Test@tmp/durable-cc1e0c9b/script.sh: line 1: lein: command not found

Balaji Sivaramgari08:12:03

Please suggest, thanks in advance

Balaji Sivaramgari09:12:51

@U051SS2EU this is happening in my local windows machine.

Balaji Sivaramgari09:12:33

15:21:27.477 ERROR: Error during SonarQube Scanner execution org.sonar.api.utils.command.CommandException: http://java.io.IOException: Cannot run program "lein": CreateProcess error=2, The system cannot find the file specified at org.sonar.api.utils.command.CommandExecutor.execute(CommandExecutor.java:102) Caused by: http://java.io.IOException: Cannot run program "lein": CreateProcess error=2, The system cannot find the file specified at java.base/java.lang.ProcessBuilder.start(Unknown Source) C:\Users\sbalaji\Desktop\conflux>lein -v Warning: implicit hook found: lein-junit.plugin/hooks Hooks are deprecated and will be removed in a future version. Leiningen 2.9.1 on Java 1.8.0_231 Java HotSpot(TM) Client VM

noisesmith18:12:21

when you run sh "source ~/.bash_profile" that creates a new shell, makes it load the profile, and then it exits

noisesmith18:12:26

it doesn't effect your shell

Balaji Sivaramgari11:12:10

@U051SS2EU, yes we tried to add multiple shell commands in a single shell. Now it is working fine. We are able to push the metrics to sonarqube server.

Balaji Sivaramgari11:12:21

Thanks for your time.

🍻 4
noisesmith18:12:39

glad I could help

Balaji Sivaramgari05:12:03

11:05:07  + curl 
11:05:07    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
11:05:07                                   Dload  Upload   Total   Spent    Left  Speed
11:05:08  
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 12537  100 12537    0     0  44772      0 --:--:-- --:--:-- --:--:-- 44935
11:05:08  [Pipeline] sh
11:05:08  + chmod a+x /tmp/lein
11:05:08  [Pipeline] sh
11:05:08  + echo 'export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin:/tmp/lein'
11:05:09  [Pipeline] sh
11:05:09  + source /home/jenkins/.bashrc
11:05:09  ++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin:/tmp/lein
11:05:09  ++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin:/tmp/lein
11:05:09  [Pipeline] sh
11:05:09  + echo /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin
11:05:09  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/alloy-jdk8-3.4.2/bin:/opt/docker:/opt/kubernetes:/opt/stedolan:/opt/garethr:/opt/go/bin:/opt/go/home/bin
11:05:09  [Pipeline] sh
11:05:10  + /tmp/lein do cloverage

Balaji Sivaramgari05:12:19

this is the output in the jenkins build log

kulminaator11:12:34

you should pick your databases accordingly to the problem you are solving , not have a favorite one 🙂

👍 4
josh_tackett15:12:42

Anyone else deal with Connection refused: connect issue when running chromedriver and etaoin?

vemv16:12:10

there's #etaoin (and its GH issue tracker of course)

josh_tackett16:12:48

@U45T93RA6 ya i checked there to see if this happened to anyone else, no solutino yet

Prometheus16:12:19

Is it not possible to use “str” to produce a querystring in jdbc? I’ve had issues with it, but only when using “str” with only elements of type string.

hiredman16:12:51

It is possible, but you should be using jdbc's built in templating

Alex Miller (Clojure team)16:12:40

using str to make SQL strings is a path that can lead to sql injection attacks

8
Prometheus17:12:00

@hiredman @alexmiller I was using a combination, the prepared statement was only modified by me, just tried some magic with pre-processing the statement, before using the standard prepared statement.

Prometheus17:12:16

and then injecting the user values with “?”

Alex Miller (Clojure team)17:12:05

just beware of Little Bobby Tables https://xkcd.com/327/

👍 12
😂 12
Prometheus17:12:21

might just be poor design on my part 😅 @alexmiller

Prometheus17:12:09

luckily the user values are handled correctly

sogaiu17:12:32

is (read-string "`1") being 1 and not (quote 1) an optimization? (i see the lines for it in LispReader.java, am curious about reasoning.)

Alex Miller (Clojure team)17:12:44

What use would the quote be?

Alex Miller (Clojure team)17:12:39

I would say no, it’s not an optimization, it’s an important part of the definition of Clojure

Alex Miller (Clojure team)17:12:34

The reader takes strings and returns Clojure data. The number 1 (a Java Long) is the data representation of integers.

sogaiu17:12:39

i have been confused trying to understand quote and syntax-quote -- in many ways they are similar, but (read-string "'1") gives (quote 1)

Alex Miller (Clojure team)17:12:31

‘ is a special syntax that expands to the special form (quote <val>)

Alex Miller (Clojure team)17:12:37

The quote special form, when evaluated, returns it’s value without evaluating it

sogaiu17:12:04

thanks for trying -- i am still failing to escape my confusion -- i will keep reading the source and experimenting

Alex Miller (Clojure team)17:12:06

Why is the quoted return from reading ‘1 confusing to you?

weavejester17:12:28

Perhaps it’s that ’1 expands to (quote 1), while `1 doesn’t expand to, for example, (syntax-quote 1)?

kwladyka17:12:45

I have a huge data structure with maps and vector of maps. Everything nested. I have to find path to the key :foo . It is so big I can’t figure out manually how to read this key. I have to read many keys like that. So there is {:a {:b [{:c 1} [{:d 1}]}} etc. imagine it much more deeper and nested. I need to find a way how to read :foo from it. So I have to figure out whole path to it: Like :a :b map of vector key :c in each map in the vector and deeper and deeper. Any tricks how can I find the path?

kwladyka10:12:06

Hi, thanks. Do you have specific function in mind?

p-himik11:12:34

Depends on what you mean by "find path to the key". It can mean either of: 1. Knowing at what "layer" :foo keys are used (like [:a :b *all-vectors* :c :foo], navigate to all of them and get all of the values 2. Not knowing where :foo keys are used at all, find all the paths to them.

p-himik11:12:51

If it's #1, then it's definitely select. #2 would be a bit more complicated but it's still possible. Although the end result probably wouldn't be much shorter than if you'd use raw Clojure with no libraries. About 10 lines or so.

kwladyka12:12:10

well I need 2. -> 1. 🙂

kwladyka12:12:24

I have to find where it and alter read as 1.

kwladyka12:12:31

1 is easier part of the job here

kwladyka12:12:03

I was trying to write 2. to find my this :foo but it is quite complex

kwladyka12:12:22

more than it looks

p-himik12:12:07

So do you need only the values of :foo ? Or both paths to the keys and the values?

kwladyka12:12:26

I need path to :foo to write code which read :foo from this structure. It is just so complex I can’t figure out the path manually

kwladyka12:12:49

But I got an idea… just now

kwladyka12:12:56

I will try to convert it into YAML

kwladyka12:12:09

then I can easy read YAML path from the Visual Studio Code

kwladyka12:12:15

I have to try this 🙂

kwladyka12:12:21

ech unfortunately this convert doesn’t work for this data :/

p-himik12:12:19

YAML is not a great idea. 🙂 Even by itself, the format is quite bad. I'm still not sure what you want. Suppose you already have this magic function f - what are the inputs and what is the desired output? Can you give a small example just so I'm sure I understand?

p-himik12:12:22

A small example based on specter of something that seems to be similar to what you want:

(select [:a MAP-VALS ALL :b]
        {:a {:x [{:b 1}
                 {:b 2}]
             :y [{:b 3}]}})
=> [1 2 3]

kwladyka12:12:45

I have :foo somewhere if very complex structure. I have to figure out the path to :football:

kwladyka12:12:02

Imagine structure as complex and deep you can’t manually figure it out

kwladyka12:12:54

So you have to figure it out to write code to read this later like (get-in [:a :fsda :fsda :fhuifader :fsdfgyuasdf :sdfafasdsd]) and more and more and pass through vectors of maps

p-himik12:12:19

Is :foo at a fixed place? Or are these keys placed at multiple different locations throughout the structure?

kwladyka12:12:32

yes, :foo is fixed place

kwladyka12:12:56

and I have to find a path to this place

kwladyka12:12:15

doing it manually take me hours

p-himik12:12:32

OK, so you can easily use get-in once you figure out where it's at. The only task is to find this single needle in a haystack.

p-himik12:12:21

And do you use only maps and vectors?

kwladyka12:12:47

well some kind of get-in with map because :foo can be multiple times, but always in the same path (considering N maps in vector)

kwladyka12:12:58

probably list also

kwladyka12:12:17

I am trying to convert it to JSON now, but not so easy

kwladyka12:12:55

hmm actually the path for JSON for this file doesn’t work in editor

p-himik12:12:18

> can be multiple times, but always in the same path (considering N maps in vector) Well, that's different paths. Vectors are exactly like maps from 0-based indices to the values for our purpose. OK, so lists as well. Anything else? If :foo is in the fist map within a vector, can we safely assume that the rest of the maps have this key as well?

kwladyka12:12:23

I think I can find random one :foo path and I will know everything what I need

kwladyka12:12:01

it is just harder, than it looks ;)

p-himik13:12:25

(def data {:a {:b {:c [{:x 1}
                       {:foo 2}
                       {:yy 7}]}}})

(defn find-first-key-path [d tk]
  (letfn [(get-path [k v]
            (when-some [p (find-first-key-path v tk)]
              (into [k] p)))]
    (cond
      (map? d) (if (contains? d tk)
                 [tk]
                 (first (keep (fn [[k v]] (get-path k v)) d)))
      (sequential? d) (first (keep-indexed get-path d)))))

(find-first-key-path data :foo)

p-himik13:12:28

Seems to work.

kwladyka07:12:19

(find-first-key-path @atom-response :request_type)
=> nil

kwladyka07:12:39

thank, unfortunately it doesn’t work for my case

p-himik07:12:57

Maybe there are other data types, like sets?

p-himik07:12:17

Sets that contain maps that contain :request_type .

p-himik07:12:48

If so, finding a path to the key is not possible without returning the whole map from a set since sets are not indexable.

p-himik07:12:42

Other possibilities: 1. The keyword is not used as a key of a map. It maybe a first item in a two-item list in a vector or something like that. Or just a keyword in a vector. 2. The keyword is just not there. 🙂

p-himik07:12:47

I don't think there are any other ways for the find-first-key-path function to fail to find something.

kwladyka07:12:06

I think there are no sets

p-himik07:12:30

Can you serialize the data and send me the file?

kwladyka07:12:54

I can’t 😞

kwladyka07:12:18

not my data

p-himik07:12:05

OK. And is it really so huge that you can't possibly pretty-print it, find the key with the regular text search, and go back from there?

p-himik07:12:33

And are you sure the keyword is there? And that it is used as a key in a map?

kwladyka07:12:59

I can find it in editor when cmd+f

p-himik07:12:21

Ah, another possibility - if the map with this key is itself used as a key, the function above will not find it.

kwladyka07:12:25

So silly issue 🙂

p-himik07:12:45

Wait, so you can open the data in an editor. Why can't you not pretty-print the data then?

kwladyka07:12:48

I didn’t understand last one

p-himik07:12:02

{{:request-type 1} 2}

kwladyka07:12:09

I can write data to the file

p-himik07:12:24

Uhmm. Then don't.

kwladyka07:12:24

and open it in editor

kwladyka07:12:53

but even if pretty print in REPL I can’t find it manually

kwladyka07:12:59

it is too complex structure

kwladyka07:12:17

also I found yesterday key can be in a few different places with a different meaning

p-himik07:12:17

Can you not create a separate file with the same data? How come?

kwladyka07:12:27

I mean things like :request_type is common name

kwladyka07:12:57

not sure what you mean

p-himik07:12:53

Ah, sorry, I read one of the previous messages wrong - I thought you said that you can't write data to the file.

p-himik07:12:01

1. Load the data in REPL 2. Pretty-print it, but not into REPL itself but to a string 3. Write that string to a regular text file (or just write to a file right away and skip the string altogether) 4. Open the file in some editor/viewer 5. Find the required key with a regular text search 6. Go from the bottom to the top and from the right to the left while gathering all elements of the path to the required key

p-himik07:12:53

In order for this to be insurmountable for manual work, either the key has to be thousands of levels deep, or the data has to be multiple GBs in size.

p-himik07:12:44

Yet another way to handle it - try to convert it to JSON. The keywords will of course become strings. When the conversion fails, understand why and fix it - e.g. turn all numeric keys into strings, turn all sets into vectors, remove all custom types that can't have any content, turn all custom types that can into maps etc.

p-himik07:12:33

You will end up with a regular JSON file. When you highlight the required key, IDEA (and I'm sure other IDEs/editors as well) will show you the breadcrumbs to it, which is exactly what you need.

kwladyka07:12:49

well it is not a thousands, maybe max 10, but still I can’t do it manually so far

p-himik07:12:50

How large is the data?

kwladyka07:12:05

yeah, converting it to JSON in the one way. I was trying, but give up, because of magic errors during it 🙂

kwladyka07:12:45

hmm it depends. Let’s say 15 MB

kwladyka07:12:57

as a minimum one to search in

kwladyka07:12:47

maybe less, I don’t remember. I have to check once again.

p-himik07:12:43

OK, I have no idea how you can't find it then. 15MBs with 10 level nesting is nothing. You said you cannot send the data. But can you show it on your screen? If you want, we can hop on a call and you can share your screen so that I could see what you're doing and what prevents you from finding the path to the key.

kwladyka07:12:07

there is a lot of mess, because of values itself. If I could see only keys without values hmm

kwladyka07:12:34

Others developers have this issue too 🙂

p-himik07:12:37

You cannot, because the key is in one of the values.

kwladyka07:12:13

Hmm maybe we can try screen share, but now I am a little in rush. I can do it +6 hours from now or something like that.

p-himik07:12:24

Works for me.

kwladyka07:12:04

> You cannot, because the key is in one of the values. I mean by remove last leaf in the tree, so the values.

kwladyka07:12:17

Thank you, that is silly and interesting issue. I appreciate your engage into the topic. Of course I don’t want to take too much your time 🙂

p-himik07:12:33

It won't achieve much, since the leaves can be only of primitive data types. "Vector of keywords" is not a leaf.

p-himik07:12:03

Don't worry about my time - I find joy in programming outside of my work.

sogaiu17:12:40

in syntaxQuote (LispReader.java), some of the cases essentially end up as (quote form) -- so superficially, quote and syntax-quote appear to have similar behavior. for some reason, keywords, symbols, numbers, and strings don't. yet those 4 things when wrapped in (quote ...) and passed to read-string, do end up with (quote ...) around them. that seems somehow odd to me.

Alex Miller (Clojure team)17:12:55

I would flip your sense here and see syntax quote as the weird case

sogaiu17:12:20

that was my initial impression -- which was why i asked whether it was an optimization 🙂

Alex Miller (Clojure team)17:12:43

Well you didn’t ask in the context of syntax quote

sogaiu17:12:08

i think i did type a backtick -- did i get that wrong?

Alex Miller (Clojure team)17:12:17

Those are all things that evaluate to themselves

noisesmith17:12:35

except symbols (I almost said the same thing :D )

Alex Miller (Clojure team)18:12:25

So the quote is unnecessary in those cases and could be seen as an optimization

Alex Miller (Clojure team)18:12:01

The symbol is more about the semantics of syntax quote

sogaiu18:12:02

just to be clear then, apart from the symbol then (still thinking about that one), the syntaxQuote behavior for strings, keywords, and numbers might be seen as an optimization then?

Alex Miller (Clojure team)18:12:55

Well I think if it was quoted, you’d be unhappy looking at the results too (lots of unnecessary quoted stuff) even if it eval’ed the same

Alex Miller (Clojure team)18:12:29

I probably spend an unhealthy amt of time looking at macro expansions though

sogaiu18:12:57

i don't disagree that verbosity would increase -- it was just surprising as i was trying to study some macro expansion things

sogaiu18:12:37

in any case, thanks all for the help 🙂

sogaiu18:12:05

(on a side note, i tried compiling a custom version of clojure with the quote being thrown in and it appears to have passed all tests)

didibus18:12:43

I think the truth is it's just like that. Maybe it was an optimization, maybe it was an accident. But in any case, it doesn't seem to matter much.

didibus18:12:55

I think it could also be that syntax-quote isn't about quoting things, but about fully qualifying them and performing templating over the input

didibus18:12:15

So things that don't fully qualify aren't manipulated

sogaiu18:12:45

it appears there's a bit more to the story according to syntaxQuote's source -- e.g. autogen stuff is handled there as apparently some interop things (i.e. related to the period character).