Fork me on GitHub
#beginners
<
2021-06-29
>
hiredman00:06:33

I would start by writing a for expression that destructs the input, and generates paths and values, then just reduce with assoc-in

hiredman00:06:45

Deconstructs

hiredman00:06:33

(reduce
 (fn [m v]
   (assoc-in m (pop v) (peek v)))
 {}
 (for [[a-or-b m] {:AAA {:prop1 ["A"] :prop2 ["a"]}
                   :BBB {:prop1 ["B"] :prop2 ["b"]}}
       [p1-or-p2 m] m]
   [p1-or-p2 a-or-b m]))

lsenjov00:06:57

@grzegorz.rynkowski In your output, are you sure you want the nested structure to be a vector, or should it be a map?

greg11:06:45

oh, yes, you are right, map is better there

hiredman00:06:07

ah, sorry, misread the request output

lsenjov00:06:23

That was a question for him, actually. My bad

hiredman00:06:40

for vectors the same, just more annoying to construct, but always going through for to tear things apart

👍 3
sova-soars-the-sora05:06:44

hi how can i know if something is set up for lazy evaluation (versus total realization) ? i assume some functions are lazy while others are not... is there a good resource I can use to find out which is which?

sova-soars-the-sora05:06:51

phrased differently, if i have a large dataset, how can i ensure i'm consuming one input at a time from the dataset, as opposed to lining it up for total eval

phronmophobic06:06:15

It's hard to give a generic answer since the input data format, the output data format, the type of data processing, and the hardware specs of the target computer(s) can all influence the design.

hiredman06:06:37

The only thing lazy is lazy is lazy seqs, anything not lazy seqs in -> lazy seq out is not lazy

2
hiredman06:06:33

And lazy seqs may not be strictly lazy, they may sometimes chunk up operations to improve efficiency

phronmophobic06:06:15

It's hard to give a generic answer since the input data format, the output data format, the type of data processing, and the hardware specs of the target computer(s) can all influence the design.

quan xing09:06:06

I run clj -X hello/run reprot error :Execution error (FileNotFoundException) at clojure.run.exec/requiring-resolve' (exec.clj:31). Could not locate hello__init.class, hello.clj or hello.cljc on classpath. I'm going to follow the code here https://clojure.org/guides/deps_and_cli

delaguardo09:06:43

did you create src/hello.clj as stated on this page?

quan xing10:06:41

im os is win10

delaguardo10:06:59

looks like you running clj -X hello/run from src directory but you should run it from hello-world

delaguardo10:06:41

clj command expects to have deps.edn in the same directory where it is invoked

quan xing10:06:18

yes. you'are right. good job. thank you very much!

quan xing10:06:56

I have another question. I Include a local jar on disk. and I use (:require [libs/pdftools :as pdf]), when I press C-c C-k compile file report error:

delaguardo10:06:53

require expects a namespace not a file

delaguardo10:06:54

I don’t know what is inside of pdftools to suggest what to put into require.

quan xing10:06:37

pdftools is a local disk jar file

quan xing10:06:38

I want to use a local jar file from disk to my project,

delaguardo10:06:23

sure, but the content of that jar should be either compiled java classes or clj source files. depending of type of the content you should use either :require to “require” clojure namespace or :import to load compiled java classes

delaguardo10:06:01

in your last screenshot you have wrong argument for second :require statement (btw. this is an antipattern in organizing clojure namespaces, normally you should have single :require in one ns declaration)

delaguardo10:06:52

could you show what is inside of that jar? jar -tf path/to/file.jar

quan xing10:06:14

This is my create a process pdf file tools in java

delaguardo10:06:44

ok, so there are some java .class files then you should add into your ns something like this

(ns 
  ...
  (:import (org.pansome.pdf.common PdfUtils)
           (org.pansome.pdf.common FontUtils)
           ...))
that means you are “importing” java class to make it accessable as PdfUtils , FontUtils etc.

quan xing10:06:07

is this right?

quan xing10:06:55

in deps.edn file add depend file

quan xing11:06:15

run error 😭

delaguardo11:06:28

(org.pansome.pdf.common PdfUtils) note the difference - in my form there is no dot between common and PdfUtils

quan xing11:06:56

Thanks, It's run.

quan xing11:06:39

clj -X hello/run is run. but in Emacs C-c C-k is error.

delaguardo11:06:06

emacs? looks like screenshots are taken in intellijidea

practicalli-john12:06:02

@U04V4KLKC Emacs 27 with tabs and LSP activated makes it look a lot more like Intellij...

practicalli-john12:06:14

@U025AG2H55F there is an #emacs channel for help. You might want to say what command you are running to help understand what you are doing in Emacs. The key binding could be doing anything...

practicalli-john12:06:09

Generally you need to evaluate things in order, so ensure the namespace has been evaluated in Emacs with Cider, which makes the functions from the required libraries available to the rest of the code.

quan xing13:06:07

How can I import dependencies from mvn. report error like this

quan xing06:06:50

Clojure is amazing but Its hard to learner than other programming language!:hugging_face:

practicalli-john07:07:40

Is it hard because you are more familiar with other programming languages? Many language are based on the concepts from C, meaning they have obvious similar concepts. Clojure is a variation of Lisp, so prior knowledge of lisp makes Clojure trivial to learn. The issues raised in this thread are related to the environment rather than the syntax of the language itself (which is very small).

sheluchin14:06:22

I'm trying to use this thing: https://clojars.org/cljsjs/rangy-textrange Steps I've taken: - added cljsjs/rangy-textrange {:mvn/version "1.3.0-1"} to deps.edn under :deps - added [cljsjs.rangy-textrange :as rangy] to my ui.cljs under :require - restarted my compiler and confirmed that rangy gets installed:

shadow-cljs - starting via "clojure"
Downloading: cljsjs/rangy-textrange/1.3.0-1/rangy-textrange-1.3.0-1.pom from 
Downloading: org/clojure/clojure/maven-metadata.xml from 
Downloading: org/clojure/clojure/maven-metadata.xml from 
Downloading: cljsjs/rangy-core/1.3.0-1/rangy-core-1.3.0-1.pom from 
Downloading: cljsjs/rangy-core/1.3.0-1/rangy-core-1.3.0-1.jar from 
Downloading: cljsjs/rangy-textrange/1.3.0-1/rangy-textrange-1.3.0-1.jar from 
but now when I go to compile it, I get the following error:
The required namespace "cljsjs.rangy-textrange" is not available, it was required by "sheluchin/ui.cljs".
Can anyone explain what I'm doing wrong?

Fredrik15:06:09

According to https://github.com/cljsjs/packages/wiki/Using-Packages#a-quick-javascript-interoperability-refresher, you shouldn't use :as rangy . Simply require cljsjs.rangy-textrang , then use js/some-fn-from-rangy to call into it.

Fredrik15:06:28

I'm not 100% sure your problem is related to this, because I don't get the same error. However, when I do as the link suggest, everything seems to work fine. EDIT: Nevermind, I didn't catch you were using shadow-cljs 🙂

dpsutton15:06:49

shadow doesn't work with cljsjs packages, but with the native npm package itself

bnstvn16:06:58

I’m trying to process a 2mb xml with xml-seq parsed with clojure.data.xml/parse with 1Gb heap — I’m getting

Syntax error (OutOfMemoryError) ,,,
Java heap space
with higher heap it works and it is fairly fast. it boils down to the below — any obvious thing I’m doing wrong?
(->> (xml-seq response)
       (filter #(some? (seq (:content %))))
       (keep (fn [e]
               (when-let [id (get-in e [:attrs :id])]
                 [id e])))
       (into {})))

bnstvn16:06:38

from the OOM heap dump:

bnstvn16:06:22

allocation profile with async profiler:

delaguardo16:06:50

you could refactor ->> to transducer equivalent to avoid creation of intermediate collections

(into {}
      (comp (filter #(some? (seq (:content %))))
            (keep (fn [e]
               (when-let [id (get-in e [:attrs :id])]
                 [id e]))))
      (xml-seq response))

Apple16:06:44

i was gonna say the filter and keep can be fit into one keep, right?

Apple16:06:03

on top of transducer

delaguardo16:06:57

the semantic is different, not sure if it is possible to replace with single operation

bnstvn16:06:55

thanks, trying this out — but what it the actual issue here? is this somewhat expected?

Apple16:06:27

(into {} (comp (keep (fn [e] (if (some? (seq (:content e))) (when-let [id (get-in e [:attrs :id])] [id e]))))) (xml-seq response))

alexmiller16:06:22

every seq op is essentially a new chain of seq cells over the input (this is mitigated somewhat by chunking)

Apple16:06:48

if that holds transducer is not needed.

alexmiller16:06:36

transducers remove all intermediate seq cells and produce only the output collection, which is particularly good if you are going to hold the entire output in memory

alexmiller16:06:45

this definitely seems like a case where transducers would reduce memory footprint

Apple16:06:47

2mb is not that big probably something else applies here

alexmiller16:06:31

the in-memory object representation of that 2 mb is probably a lot bigger (and then there are multiple seq wrappers around each node)

Apple16:06:29

other aspects: 1) some? is not needed 2) (get-in e [:content :attrs :id]) instead of (get-in e [:attrs :id]), right?

bnstvn16:06:56

well seems a good opportunity to learn transducers.. in the meanwhile, with clj-memory-meter, the retained size of the whole xml is:

(mm/measure (xml/parse-str (slurp "/Users/bani/tmp/loadgroup.xml")))
=> "3.8 MiB"

alexmiller16:06:31

is that fully realized or lazy?

alexmiller16:06:01

I guess that's realized

bnstvn16:06:19

(mm/measure (doall (xml-seq (xml/parse-str (slurp "/Users/bani/tmp/loadgroup.xml")))))
=> "26.6 MiB"

alexmiller16:06:35

well, might be something else going on then

bnstvn17:06:43

(seems so, i failed to create a minimal example yet..)

alexmiller17:06:16

tools like yourkit can take heap snapshots and diff heap snapshots which I find to be pretty instructive in understanding leaks

alexmiller17:06:36

they also have a "path to root" which can tell you why something is being held

bnstvn09:07:39

[turned out i actually fixed this issue and encountered a similar one with a bigger xml (30mb) — apologies — that didn’t even parsed with 512mb. in that case, i only needed to count some elements, and eventually went with plain java stax route. thanks for the help!]

sova-soars-the-sora16:06:44

favorite lib for assertions / testing ?

greg16:06:24

for testing data structures https://github.com/nubank/matcher-combinators is really nice

4
19:06:48

I’m finishing my read of the Brave Clojure and would like to know where to go from here. I started a small project putting the things I learned to work and would like to know more about the following topics: • Server-side clojure, or how to make a rest api with it • Deploying clojure, specially on Heroku. Any help or recommended readings would be nice, thanks!

jeffy19:06:02

I found this documentation helpful to deploy a very simple app to heroku. https://devcenter.heroku.com/articles/getting-started-with-clojure?singlepage=true

practicalli-john21:06:30

https://practical.li/clojure-webapps/ may have some interesting pieces for you. Also take a look at metosin/reitit as a data oriented alternative to compojure for routing

jamesleonis02:06:54

The Luminus framework is a good start as well. Great docs. It's by the same author as Web Dev book mentioned by @U01EFUL1A8M https://luminusweb.com/docs/guestbook

Michaël Salihi08:06:41

Here is an example of a backend REST API whose contains all Heroku deployment config, etc. Maybe it can help: https://github.com/prestancedesign/todo-backend-reitit