Fork me on GitHub
#clojure
<
2020-03-08
>
Aviv Kotek08:03:13

Hey guys! what's your thoughts on reading SICP today, is it still relevant / will improve a current-clojure-with-no-lisp-knowledge programmer?

p-himik10:03:10

Even if we knew you current level of LISP knowledge, it would be really hard to tell how useful the book would be to you since it contains many things apart from just some Scheme programming (assuming you're talking about the edition that used Scheme and not the new one that uses Python). My advice - just read through the ToC, skim chapters that you think might be especially interesting, and decide based on that.

Aviv Kotek13:03:16

I know about the course itself, just curious if it's worth doing it in 2020,

agigao13:03:27

Well, then we should ask the following question - what’s the initial goal?

Aviv Kotek13:03:40

Improve my general coding skillz, my philosophy on software 😉 or are the topics still relevant for a production-programmar in 2020?

👍 4
p-himik13:03:20

The topics there are generic CS, they will always be relevant.

👍 4
hindol13:03:45

There is an excellent online edition, freely available here: https://sarabander.github.io/sicp/ Just started reading. I was blown away by the chapter on recursion. Why not just try it?

Aviv Kotek15:03:50

@UJRDALZA5 that was not the Q

hindol15:03:04

I just wanted to add the link to the excellent online edition (not mine). I am reading it from there.

🚀 4
agigao16:03:31

Well, then I do wholeheartedly recommend and even urge you to take that course, follow and complete it. Lectures do provide all details bit by bit in a digestible manner.

🚀 4
zilti18:03:56

"the new one that uses Python"? Heresy!

pinkfrog11:03:37

(def a (make-array String 2))

pinkfrog11:03:47

how to convert a to type Object[] instead of String[] ?

Cameron11:03:47

One way off the top of my head is

(into-array Object a)
;; Example 
(def a (into-array Object [2 "4" "8" 5]))
#'boot.user/a
boot.user> (type a)
[Ljava.lang.Object;

pinkfrog11:03:21

that becomes two arrays. I want type cast on array a.

p-himik16:03:18

String[] is an instance of Object[].

p-himik16:03:24

Why do you need to cast it?

Cameron11:03:22

and if for some reason you want to make the array outright as an object array

(object-array size-or-seq)
;; Example 
(object-array [1 2 3 4]) 
;; Or, to initialize as nils, still as [Ljava.lang.Object;
(object-array 5) 
;;=> [nil nil nil nil nil] 

pez16:03:24

Dear Clojurians: I'm working with getting Calva Paredit do the right thing when forms are reader tagged. I learnt yesterday that readers be reading, so have now tried to accomodate for that. Can some of you people who are in the know look at this and see if it seems to be correct? A summary of what is going on in the GIF: 1. I am painting the reader tags with the same rainbow color as any list they are tagging. 2. The session starts with expanding the selection a few times from the nil form. 3. Then I move past the #foo-0 tagged list and back again. 4. Then I move down into to the #foo-0 tagged list. 5. I drag the 1 forward through all its siblings in the list. And back again. 6. Then I drag the #foo-3 quadruple tagged function call forward past its siblings, and back again. 7. Then I drag it down the [1 2 3] vector, and then out the end of it. 8. Then I drag it backward to its original place. 9. The last thing I do is to demonstrate that I don't need to place the cursor at the very start/end of a form for Calva Paredit to understand which form is the ”current” form. (Matters only a little to this question, but anyway). Regarding what happens when something is dragged pass the ignore markers. I have chosen to let those be ”forms”, so that things can be hidden from the reader by dragging things in front of them. Main question here: Does it look like I have understood how reader tags work in Clojure? If I misunderstand it, Calva will suffer. 😃

sogaiu22:03:33

btw, i'm having a much easier time understanding by opening the animated gif in the gimp and using Filters > Animation > Playback feature to step through frame-by-frame. perhaps there are easier ways, but this seems to make it doable here.

❤️ 4
sogaiu22:03:04

in step 5, you mention dragging through siblings. i found it odd that the 1 descends into the discard / ignore forms while it doesn't descend into the tag forms.

sogaiu22:03:32

similar comment for step 6.

sogaiu22:03:11

step 7 mentions dragging down the [1 2 3] vector and out the end, but i only see it enter at the left most point before exiting (none of the other intermediate inner positions of the vector appear visited). may be that's what was meant -- was expecting to see it move through the vector step by step.

pez10:03:44

Thanks! As for the step 5 and 6 comments, it will be a design decision. Ignore markers are special and stack in a different way (if I have understood it correctly). Currently in Calva, ignore markers are considered forms that you can move around freely.

pez11:03:24

Step 7. Very observant of you! In Paredit, moving down into a form can be done from in front or from behind of it. The cursor will enter either at the beginning, or at the end of the list, respectively. Moving up can be done backwards, placing the cursor in front of the list, or forwards, moving to behind the list, and it doesn't matter where in the list the cursor is when the movement command is issued. In step 7 I drag the form down from the front of the vector and then drag up forwards. So it is expected that you do not see the form be dragged forwards passed the siblings inside the vector.

sogaiu11:03:20

thanks for the explanation

zara17:03:08

What is the fastest way to convert a deep nested List<Map<Keyword, ...>> from java to Persistent clojure structs?

zara17:03:01

I am using PersistentVector.adopt for Lists and PersitentHashMap.create for Maps. But the PersitentHashMap.create takes too much time.

jumpnbrownweasel18:03:48

I don't know a way to do that more efficiently, but I was wondering if you'd considered not converting the maps at all, since many Clojure functions work on plain old Java maps.

ghadi18:03:48

those are not public functions

zara18:03:15

I tried this first, but i am using this in every db query. So it is hard to garantee that no mutation will happen.

zara18:03:51

@U050ECB92 they are public static.

ghadi18:03:20

i'm aware - they're not considered public parts of clojure's API

jumpnbrownweasel18:03:37

Have you tried (into {} java-map)?

ghadi18:03:59

please post the shape of the datastructure you need to convert

zara18:03:30

It is a deep nested List<Map<Keyword, Object> Object can be List<Map<Keyword, Object>> or Map<Keyword, Object>

zara18:03:28

into does the same thing but is slower.

jumpnbrownweasel19:03:24

You may have already done this, but if it were me I would try to find a way to avoid the copy completely. It sounds like this is a high throughput code path, so copying is undesirable if there is any way to avoid it. If you describe where the Java maps come from and how they're used, maybe someone here will have some ideas.

zara19:03:55

I am doing a lib to pull entities from mysql. So I build a single query with left joins for nested entities and group the nested entities in java. The result from this queries will be used everywhere in the application.

zara19:03:20

Doing some profiling I see that the performance problem comes from trying to iterate through the hashmap. Maybe I should use some other structure...

jumpnbrownweasel19:03:57

Yes, that's an idea. A couple other thoughts, not sure they'll help: • You could build the map in Clojure rather than Java, although of course this may be a big change. • Instead of converting to a Clojure map just to get immutability, you could wrap it using java.util.Collections/unmodifiableMap .

zara19:03:48

1. I tried to build in clojure first, but I was not able to produce a code with good performance. 2. In the application i will have something like (assoc entity :d 1) and it will be a UnmodifiableHashMap so it will throw an exception. I think I really need to have a PersitentHashMap in the end.

jumpnbrownweasel19:03:43

I can't think of any other ideas, except perhaps to look at your Clojure code for building the map to see if it could be improved.

zara19:03:42

I will try to implement it without using HashMaps and see how it goes. If the copy stills take too long I will post the code here. Thanks for your help.

jumpnbrownweasel19:03:31

OK. But there would be no copy, right, if you build it as a persistent map?

zara19:03:34

But everything else is much slower.

zara19:03:20

Maybe I have to think in a better way to implement it using mutable structures.

jumpnbrownweasel19:03:38

Are you using a Java library that returns Java maps, is that part of the problem?

zara19:03:28

No, my data source is ResultSet from jdbc.

zara19:03:57

I let clojure.jdbc read the columns correctly and implement a result-set-fn to convert from rows to the nested entities.

jumpnbrownweasel19:03:41

It would be interesting to look at the Clojure code for building the map, and try to figure out why it is slower. But it is up to you to decide if that approach is worth the time it will take for you.

zara20:03:59

I will put in a public repo so you can take a look.

zara23:03:08

@UBRMX7MT7, I found a solution with a good enough performance. I am using TransientMap and TransientVectors instead of HashMap and ArrayList. At the end a just call the method persistent of the transients recursively. I guess that a good solution in clojure would involve using transients through the whole process too.

jumpnbrownweasel23:03:24

So you're using transients while building the map (and it is a Clojure map all the way through), or you just use transients at the end when you convert from Java maps to Clojures maps?

zara23:03:19

while building the map

jumpnbrownweasel23:03:39

That's really cool.

zara23:03:51

I wrap the transient in a object so I can keep track of the modified transient without loosing the structure of the tree that is been built.

jumpnbrownweasel23:03:53

That's a good pattern to remember

zara23:03:27

public class IdentList{

    public HashSet<Long> ids;
    public ITransientVector ret;
    public IPersistentCollection vec;
    public boolean persisted;

    public IdentList()
    {
        ids = new HashSet<Long>();
        ret = EMPTY.asTransient();
        persisted = false;
    }

    public void addWithId(Long id, IdentMap e) {
        if(!ids.contains(id)){
            ids.add(id);
            ret.conj(e);
        }
    }

    public IPersistentCollection persistent()
    {
        if(!persisted) {
            for (int i = 0; i < ret.count(); i++) {
                ret.assocN(i, ((IdentMap) ret.valAt(i)).persistent());
            }
            persisted = true;
            vec = ret.persistent();
        }

        return vec;
    }
}
public class IdentMap{
    public ArrayList<Keyword> hasOneList;
    public ArrayList<Keyword> hasManyList;
    public ITransientMap ret;
    public IPersistentMap map;
    public boolean persisted;

    public IdentMap()
    {
        hasOneList = new ArrayList<Keyword>();
        hasManyList = new ArrayList<Keyword>();
        ret = EMPTY.asTransient();
        persisted = false;
    }

    public Object get(Object key)
    {
        return ret.valAt(key);
    }

    public void addField(Keyword key, Object obj)
    {
        ret = ret.assoc(key, obj);
    }

    public void putObject(Keyword key, IdentMap val)
    {
        hasOneList.add(key);
        addField(key, val);
    }

    public void putList(Keyword key, IdentList val)
    {
        hasManyList.add(key);
        addField(key, val);
    }

    public IPersistentMap persistent()
    {
        if (!persisted) {
            for (Keyword key : hasOneList) {
                Object o = ret.valAt(key);
                if (o instanceof IdentMap) {
                    ret.assoc(key, ((IdentMap) o).persistent());
                }
            }

            for (Keyword key : hasManyList) {
                Object o = ret.valAt(key);
                if (o instanceof IdentList) {
                    ret.assoc(key, ((IdentList) o).persistent());
                }
            }

            map = ret.persistent();
            persisted = true;
        }

        return map;
    }
}

jumpnbrownweasel23:03:39

Thanks, I get it. It's really encouraging that transients improved performance for a case like this.

jumpnbrownweasel23:03:56

Don't forget to use the return value of ret.assocN and ret.assoc (two places).

zara23:03:21

humm, good catch. thanks.

zara00:03:40

Most of the time now is spent associng in the transients, but still the overall performance is better because there is no copy.

jumpnbrownweasel00:03:10

Interesting. How large are these structures in total, 100's of lists/maps, 1000's?

jumpnbrownweasel00:03:08

Maybe not worth maintaining ids (in IdentList ) and just use the transient vector instead? Just a thought.

jumpnbrownweasel00:03:59

Oh, never mind, I see you're checking for set existence.

jumpnbrownweasel00:03:15

Although it might be possible to use a transient/persistent set instead of the vector, if you don't care about the order of elements.

zara15:03:16

It can get to 1000s. I think it will not be the most common case, 100s is more likelly. I need to keep the order without duplicating elements.

quadron18:03:55

does it make sense to use spec to "pattern match" and search for data? as a "getter" for example? are there any examples out there?

quadron18:03:32

let's say I want an expression to return all the data points in a collection that satisfy a certain spec

markaddleman22:03:36

Spec by itself isn't a good fit for this. Spec + clojure.walk might work though I've never tried it. #meander is a great tool for this kind of use case.

4
dadair22:03:37

Is there a way to force *warn-on-reflection* true for an entire deps.edn project repl session? Eg., not having to pepper set! over several files?

jumpnbrownweasel22:03:11

This is a common question and I haven't done it but here are my notes on what others have said:

delaguardo:
    You can use --init-opt for that.
    clj -e "(set! *warn-on-reflection* true)"... 
  alex:
    You can use :jvm-opts ["-Dclojure.compile.warn-on-reflection=true"] in an
    alias, but can't currently do that always

👍 12
💯 4
dadair22:03:00

Thanks! I’ll give that a try

borkdude13:03:55

I tend to just copy paste this line on the top of every namespace that does interop, just to not be dependent on tooling to do that for me. It's a bit more tedious, but works 😉

dadair15:03:15

The problem is more finding all the namespaces that do interop! I suppose a regex should do it

dadair22:03:06

I’m using cider if that impacts things