Fork me on GitHub
#beginners
<
2023-02-22
>
quan xing02:02:05

vs code start clojure from calva show error message: org.apache.http.conn.ConnectTimeoutException: Connect to [] failed: connect timed out

quan xing02:02:15

I can't visit the http://repo.clojars.org , Are you normal?

seancorfield02:02:15

Seems to be working again now -- maybe it was just a transient failure?

seancorfield02:02:56

That IP looks like Fastly in Tokyo so it could be a regional outage if you're still having trouble...

seancorfield02:02:22

(Fastly hosts the CDN in front of Clojars I believe)

quan xing02:02:56

I use the VPN from the China

Timofey Sitnikov04:02:24

I am attempting to combine two collections of map, where each of the maps are combined based on the :msn, so the collections look like this:

(def c1 [{:msn "1465", :vol 14.592, :acc 0.9695, :R_e 2043.47}
         {:msn "1466", :vol 14.441, :acc 0.9595, :R_e 2059.35}
         {:msn "1502", :vol 14.497, :acc 0.9631, :R_e 2111.35}
         {:msn "1516", :vol 14.512, :acc 0.9641, :R_e 2066.06}
         {:msn "1460", :vol 14.850, :acc 0.9866, :R_e 2163.54}
         {:msn "1451", :vol 14.358, :acc 0.9539, :R_e 2060.79}
         {:msn "1433", :vol 14.503, :acc 0.9636, :R_e 2056.02}
         {:msn "1422", :vol 14.559, :acc 0.9673, :R_e 2039.06}])
                               ;; ----------
(def c2 [{:msn "1465", :accuracy -3.077, :period 0.22}
         {:msn "1466", :accuracy -4.064, :period 0.20}
         {:msn "1516", :accuracy -3.621, :period 0.27}
         {:msn "1460", :accuracy -1.321, :period 0.25}
         {:msn "1502", :accuracy -3.676, :period 0.18}
         {:msn "1451", :accuracy -4.621, :period 0.23}
         {:msn "1433", :accuracy -3.639, :period 0.16}
         {:msn "1422", :accuracy -3.276, :period 0.15}])
                           
; my attemp, but it is ugly
(map (fn [y] (merge y (first (filter (fn [x] (= (:msn x) (:msn y))) c2)))) c1)
I got the brute force approach to work, but it seems like there may be a function or an external library that can do that already, any suggetions?

seancorfield04:02:55

With transforms like this, it's often better to switch one or more structures to a more suitable form first...

seancorfield04:02:41

...in this case, you want to create an index on :msn for one structure and then use that as a lookup when merging into the other.

seancorfield04:02:48

(group-by :msn c2) would give you a map from the :msn values in c2 to vectors of matching elements (which you know will only have one entry here).

seancorfield04:02:39

Hmm, yeah, I tend to forget clojure.set has a lot of useful stuff in this area.

phronmophobic04:02:27

I should probably use clojure.set/index more often.

seancorfield04:02:20

user=> (clojure.pprint/pprint (clojure.set/join (set c1) (set c2)))
#{{:msn "1422",
   :vol 14.559,
   :acc 0.9673,
   :R_e 2039.06,
   :accuracy -3.276,
   :period 0.15}
  {:msn "1433",
   :vol 14.503,
   :acc 0.9636,
   :R_e 2056.02,
   :accuracy -3.639,
   :period 0.16}
  {:msn "1516",
   :vol 14.512,
   :acc 0.9641,
   :R_e 2066.06,
   :accuracy -3.621,
   :period 0.27}
  {:msn "1465",
   :vol 14.592,
   :acc 0.9695,
   :R_e 2043.47,
   :accuracy -3.077,
   :period 0.22}
...

seancorfield04:02:13

(you lose ordering but you could easily sort-by to get that back)

hiredman05:02:01

clojure.set also has a join function

hiredman05:02:34

(pretty brute force too)

hiredman05:02:22

Whoops, saw group by and index and missed that join was mentioned

skylize05:02:24

Looks like the collections are of equal length in the same order (based on msn). If you can guarantee that is always true, this should work.

(map merge c1 c2)

skylize05:02:37

If you can't guarantee order you could always just sort them both on :msn before calling that on the sorted colls.

hiredman05:02:06

https://en.m.wikipedia.org/wiki/Sort-merge_join is really what you want, more optimal on pre-sorted inputs than set/join

skylize05:02:15

Reading that, it sounds to me like a long-winded way of saying to sort the lists on :msn and then step through the sequence, merging each pair together. So, basically do exactly what I just said? :thinking_face:

hiredman06:02:12

Well knowing the name makes it easier to search for, and searching this slack turned up all the other times I've linked the article, and some of those have threads with potentially useful example code

👍 2
phronmophobic06:02:59

> sort the lists on :msn and then step through the sequence, merging each pair together. It sounds like that assumes that every map on each side of the join has exactly one match for the join, which may not be the case (the sort-merge join accounts for this).

dgb2310:02:16

To me the indexing has to be worth it down the line. It also implies overhead. So N should be large to begin with. If you can do the indexing right at the moment when you construct the data sets it's likely beneficial. Otherwise you could maybe also index the data sets while you merge them: reduce over both collections, put record into index if ID seen for the first time, merge otherwise.

Timofey Sitnikov13:02:30

OK, awesome, sorting and combining seems to be the right option.

oly14:02:26

I have some code that is inserting into 3 different databases, I am noticing the latency is quite slow as it has to wait for one db before using the next, so I was wondering if i could use async so it does not block, doing a little bit of googling I am not finding much on this approach so any advice? are there other ways existing libraries or approaches I can use ?

rolt14:02:27

using futures you could simply use pvalues if you want more control, handling failure, etc there are libraries

oly15:02:51

I may use promises directly just thought I would see what other have done, what libraries are there I only really found this one https://github.com/metosin/porsas which says experimental, search for databases in the context of clojure promises and async did not return much could be a topic for someone to blog about 🙂

rolt15:02:41

you may be focussing on the keyword "async" too much. In the vast majority of cases, futures are enough and not the bottleneck. You'd usually drown your db before it becomes a problem. And the landscape may vary once java fibers are globally available

rolt15:02:32

still it's interesting from an exploratory point of view

oly16:02:10

yeah maybe often half the struggle getting the correct search terms 🙂

oly16:02:12

yeah going down this path, just figured it was worth checking what the options are in case there are certain considerations which may have been handled in libs or anything like that.

popeye14:02:57

what is the command to generate clojure project with deps.edn ? I am trying below command, but not getting generated

clojure -X:new :template app :name my-clojure-project :exec-fn server/main

ander14:02:14

I've really been enjoying neil for bootstrapping / managing deps.edn projects https://blog.michielborkent.nl/new-clojure-project-quickstart.html

1
popeye15:02:53

@U7ERLH6JX , will below command to downloads deps ?

clojure -Sdeps 

lispyclouds15:02:32

clojure -P should be better

practicalli-johnny15:02:29

deps-new and clj-new are two common community projects to create projects from templates clj-new can use a wide variety of existing project templates deps-new provides Clojure CLI specific templates (and has a simpler approach when creating your own templates) https://practical.li/clojure/clojure-cli/projects/create-from-template/

skylize15:02:38

There is not currently any command built into Clojure CLI Tools for generating a new project. It must be done by hand or use a 3rd party tool. The command in your question looks like how you would call to https://github.com/seancorfield/deps-new after it is installed. > will below command to downloads deps ? The -P flag

clojure -P
will "prepare deps", which includes downloading the deps listed in deps.edn to the cache. If you want to "install" a new dep, https://github.com/babashka/neil offers that feature. It also uses https://github.com/seancorfield/deps-new under the hood for generating new projects, so you would not need to call directly to deps-new in that case.

popeye18:02:44

thanks you all for your inputs, I am going through the tread one by one

popeye18:02:33

@U05254DQM I got below error with the command specified in the doc

clojure -T:project/create :template app :name practicalli/playground
WARNING: Specified aliases are undeclared and are not being used: [:project/create]
No function found on command line or in :exec-fn

skylize18:02:15

To follow that tutorial, you to need to define aliases in your User deps.edn file. For some reason that code is hidden, but you can show it by clicking on “Clojure Aliases for creating projects from templates“ just above the YouTube video.

practicalli-johnny20:02:15

I've updated the page to hopefully make it cleare that either Practicalli Clojure CLI Config is required or the example aliases Thanks for the feedback

👍 2
lspector23:02:26

FWIW from my perspective as a teacher it would be really sweet if an absolutely minimal project could be created with the CLI tools alone. Believe it or not, "It must be done by hand or use a 3rd party tool" can be a non-trivial impediment to some newcomers. By minimal I mean just a directory containing a deps.edn that contains {}, and also containing a src folder that contains a single subfolder with the same name as the top-level folder, that in turn contains a single file (might as well be core.clj) that just contains an appropriate minimal ns expression. For some newcomers who don't have experience working at the command line, and may not know the "helpful" things their OS or editor might do with hidden extensions or text encoding, it can be a pretty tedious process to get to the point of working in a minimal project. I have talked with @U0ETXRFEW about adding something that creates such a project within #CBE668G4R, since Calva can work with deps.edn projects without even installing the CLI tools, which is fabulous. But it can't currently create them. Installation of anything can be a pretty big hassle in a class setting, with students using different machines and OSes and having different backgrounds (many of which might include just a single CS course, or less). If I can easily get them over the hump of installing Java and the CLI tools, it would be really great if that gave them all they needed to begin working with code in a project.

skylize02:02:50

> FWIW from my perspective as a teacher it would be really sweet if an absolutely minimal project could be created with the CLI tools alone. - @U06BV1HCH From previous conversations with Alex, this is at least "on the map". There are a lot of open questions about what a project generated by the official CLI tool should look like.

❤️ 2
lspector02:02:23

Do you know if any/many of those open questions arise for projects as minimal as what I've suggested? I can imagine lots of questions once you get even slightly more complex, but something like I've suggested at least gets you started (e.g., #CBE668G4R can open and deal with it) without committing one to much of anything.

skylize02:02:44

Keep in mind, though, the fact the Clojure team is interested does not guarantee anything will happen. From the same conversations, Alex also seemed quite positive about the idea of simply relegating such things to tools like Neil. Third party tools bring a benefit of much more flexibility than official tooling could/would. So, honestly, I think the only serious pain point is as it is already is onboarding beginners.

skylize02:02:51

@U064X3EF3 question for you from @U06BV1HCH ^

Alex Miller (Clojure team)02:02:52

please please, put any requests or problems on https://ask.clojure.org - I am in way too many threads here and elsewhere that I am seriously overloaded. putting it on ask means that I will (eventually) sweep it into tickets. it also means others can find it via search or similar question prompts, comment on it, and vote on it, which is what we use for prioritization

Alex Miller (Clojure team)02:02:32

and yes, I think having means for initial project creation is in scope

skylize02:02:52

So, just in case that ^ wasn't clear @U06BV1HCH, it is definitely being hammocked. If you need more specifics than that, or want to offer feedback or ideas about it, open a question on Ask Clojure.

Alex Miller (Clojure team)03:02:24

even if you don't want to do those things, it's still helpful to log it

lspector16:03:34

Just to leave a note of it here, I posted a request at ask.clojure, and @U064X3EF3 logged it at https://clojure.atlassian.net/browse/TDEPS-242

metal 1
stopa20:02:48

Hey team, noob question: I have a list of items, which have a few different ways to identify them:

[{::id 1, ::label "foo" ...}, {::id 2, ::label "bar"}]
Both id and label are unique to the item. I want to create a few helpers, to get by item or by id. I'm thinking of doing something like this:
(defn get-context [conn app-id]
  (let [items (get-by-app-id conn app-id)
        id->item (ucoll/->map ::id items)
        label->item (ucoll/->map ::label items)]
    {:id->item id->item
     :label->item label->item}))

(defn by-id [{:keys [id->attr]} id]
  (id->attr id))

(defn by-label [{:keys [label->attr]} id]
  (label->attr id))
Some things I don't like about this: • An over-optimization, but this does actualize the maps before I ever access them. Maybe they should be lazy. Perhaps I should add a delay • I don't like the name context But can't think of a better one And generally am thinking: I wonder if there's some higher abstraction I should use here.

hiredman21:02:50

clojure.set/index

❤️ 2
skylize23:02:29

How about some?

(defn some-kv [k v coll] 
  (some (fn [m]
          (when (= v (k m)) m)) 
        coll))

(let [data [{::id 1 ::label "foo"}
            {::id 2 ::label "bar"}]]
  {:a (some-kv ::id 2 data)
   :b (some-kv ::label "foo" data)})
; =>
{:a {:user/id 2
     :user/label "bar" }
 :b {:user/id 1
     :user/label "foo" }}

phill01:02:38

Why not make two more maps: id->item and label->item? To make such maps, as @U0NCTKEV8 pointed out, clojure.set/index could be useful. But in your case the ids and labels are both unique, so you could do even better. You can follow the "source" link in the API documentation to see how clojure.set/index works.

daveliepmann07:02:47

I would usually define additional lookup maps: (def foo->x (into {} (map (juxt :foo identity) xs))) or (def bar->x (zipmap (map :bar xs) xs))

stantheman14:02:56

What about a filter? Or (for with a :when if you have humongous list.