Ok I'm back with another url-encoder type of question... the thing is, the built in function from ring replaces special characters and spaces with %20 and so on... this makes the urls look really ugly. Websites like quora and medium will take a blog post title and have a url "that-is-like-this". So I wonder, can I just write a clojure function that would take a blog title (a string), and ignore all special characters? I know I can use replace to change the spaces to "-" but how do I ignore the special characters, or take only numbers and letters?

If you want a valid URL, then you need things like %20, yes? If you want to avoid those, then you need to find other replacements that work well enough for your use case. However, if your use case is "turn an arbitrary Unicode string into a valid URL", I doubt you can avoid the %20 kind of sequences in the result.

I don't think I need %20 to have a valid url... Medium is a popular website and their url's just turn spaces into "-". Maybe I'm not sure what you mean?

For example if someone has a blog post "my #1 blog's title" then medium has a url to the blog post "/my-1-blogs-title". However url encoders have the output "my%20%231%20blog%27s%20title". I want to figure out how to get the same url that Medium has? It ignores special characters

Then you can look for any documents that explain what Medium uses for their translation of strings to URLs. One question you may or may not need to worry about is "if symbols like # and ' are simply removed, what should happen when two different titles produce the same URL?"

Oh, I see. Yes it appears they have some characters that come after the title, which must be to take care of that problem

There very well might be some widely used function that Medium uses for that title-to-URL translation, but it might also be some custom thing they wrote themselves.

Thanks, that's helpful

For the sake of trying to tie this in with clojure... Any thoughts on how I can take a string and get rid of special characters in clojure?

I'm thinking (replace "my $string" "$" "")

But how would this cover all the special characters? Is there a code I can use?

If you enumerate all characters you want to keep, that is probably a shorter list than all characters you want to remove, but not sure whether you consider most of the Unicode character set as something you want to remove, or something you want to keep. It is a huge character set.

I see... so if I want to just keep certain characters from a string... I would probably use a different function than replace, right?

What about doing something like:

(apply str                                                                                       
       (remove #{\# \%} "the#quick%brown")) => "thequickbrown"

Thanks, chase. I think that's similar to what I'm looking for. Is there a way I can remove all special characters without listing every one I can think of? (Or I can probably look up which characters are not allowed in URLs)

Look at the docs for (Java) regular expressions: it has escaped characters that represent entire classes of Unicode characters.

In particular this page: https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html. and things like \p{Lower} and others listed there

Here's a function from our code that attempts to check whether something is a vaguely valid URI:

(defn invalid-uri?
  "Given a URI, return true if it contains characters that we don't believe
  should be in a valid URI. For now, what we allow in a URI are:

  * - . ( ) ' / _
  * Any Unicode alphanumeric p{L} letter p{M} combining mark p{N} numeric

  Anything else, we'll assume it's a bad URI and we'll give a 404 for it.

  As of WS-10501, we also use this to validate affiliate IDs."
  [uri]
  (try
    (->> uri
         (java.net.URLDecoder/decode)
         (decode-location)
         (re-find #"[^-\.()'/_\p{L}\p{M}\p{N}]")
         (boolean))
    (catch Throwable _
      ;; any URI that won't decode is almost certainly invalid!
      true)))

This Java doc page for URL encoder mentions explicit lists of characters that it does not transform: https://docs.oracle.com/javase/7/docs/api/java/net/URLEncoder.html

This uses a negative match [^...] followed by things we do allow.

(don't worry about the decode-location function in the middle -- ignore that, it's specific to some URI parsing we do -- the important part is the re-find call @bradj4333)

Cool thanks! This is really helpful @andy.fingerhut @seancorfield

I still have the question though if you don't mind 😅 Let's say I come up with a list of characters that I want to keep from a string - is there an easy way to just take certain characters from a string in clojure? (for example if I have a string with numbers and letters, and only want to take the numbers - or have alphanumeric mixed with special characters and only want to take alphanumeric)

I see there's ways to remove & replace - should I just focus on using those for my needs?

At this point I kind of want to try writing my own url-translation function for practice 🙂

Well the opposite of remove is filter if that is what you are asking.

(apply str                                                                                       
       (filter #{\# \%} "the#quick%brown")) ;; "#%"

But it sounds like you want to be exploring regex es more.

I'm still in a "I can do regex but it takes a lot of googling and it's write only (as in I won't understand my own solution 6 weeks from now" phase

Ahh ok thanks @chase-lambert Yea, I am VERY intimidated by regex at this point! I'll try filter first, but if I can't do that then I'll look more into regex (which I need to learn better at some point anyway 🙂 )

dev=> (clojure.string/replace "the quick brown fox jumped over the lazy dog" #"[remove]" "")
"th quick bwn fx jupd  th lazy dg"
dev=> 

#"[...]" is regex-speak for "the set of characters inside the brackets". Then you just replace those matched substrings with the empty string. Matched characters begone! @bradj4333

dev=> (clojure.string/replace "the quick brown fox jumped over the lazy dog" #"[^remove]" "")
"eroomeovereo"
dev=> 

An example closer to what you're asking about: only keeping alphanumeric characters from a URI:

dev=> (clojure.string/replace "" #"[^a-zA-Z0-9]" "")
"httpsappslackcomclientT03RZGPFRC053AK3F9cdnfallback1"
dev=> 

Oohhhhh cool thanks @seancorfield! Something like #"[^a-zA-Z0-9]" is exactly what I was looking for. I wasn't sure how to represent all that so succinctly. Thanks a lot!

(def t {:a "a" :b "b" :c "c})
((juxt :a :b) t)
=> ["a" "b"]

(let [ks (conj [:a] :b)] ((apply juxt ks) {:a "a" :b "b" :c "c"})) -> ["a" "b"]

((apply juxt [:a :b]) t) =>["a" "b"] 👌

fwiw, i think the use of apply and juxt is great, but people might find this more obvious at first glance if i understnad your input and output spec right: (-> {:a "1" :b "2" :c "3"} (select-keys [:a :b]) vals);; => ("1" "2"). though juxt might be faster.

Vals on select keys won't necessarily return the keys in the order expected here

I suppose i would hope the order didn't matter given it came from a hashmap. But yea.

Juxt is how you can do a positional selection though

You could do it with map as well: (map m ks) (I still prefer juxt though)

(defprotocol DatetimeToStrProtocol
  (date-str [date-time])

  (date-add-min [date-time min])

  (str-date [s pattern]))

(extend-protocol DatetimeToStrProtocol
  ;; LocalDateTime to string
  java.time.LocalDateTime
  (date-str
    [date-time]
    (.format
     (java.time.format.DateTimeFormatter/ofPattern "yyyy-MM-dd HH:mm:ss")
     date-time))
  (date-add-min
    [date-time min]
    (.plusMinutes date-time min))
  (str-date
    "string to date 
    format : yyyy-MM-dd HH:mm:ss"
    [s p]
    (.parse
     (java.time.format.DateTimeFormatter/ofPattern p)
     s))

  ; Date to string
  java.util.Date
  (date-str [date-time]
    (.format
     (java.text.SimpleDateFormat. "yyyy-MM-dd HH:mm:ss")
     date-time))

  (str-date
    "string to date 
    format : yyyy-MM-dd HH:mm:ss"
    [s p]
    (.parse
     (java.text.SimpleDateFormat. p)
     s)))

; Evaluating file: date.clj
; Syntax error (IllegalArgumentException) compiling at (src/xing/util/date.clj:12:1).
; Don't know how to create ISeq from: java.lang.Character
; Evaluation of file date.clj failed: class clojure.lang.Compiler$CompilerException

What would I need to run on the terminal to get a clojure process to start with 3 aliases selected and then to start a socket repl?

I’ve tried a few variants on this clj -A:testing:test:app-clj -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}" but with no luck

how can I use IN(?,?,?) in next.jdbc with mysql. I wrote code like this:

(defn query-task
  [task-id-coll call-times]
  (db/query db-con-spec ["SELECT * FROM task_phone_record 
                          where task_id = (?)
                          and call_times = ? 
                          " (int-array task-id-coll) call-times]))

(query-task [123 124] 1)

sql log:
next.jdbc/execute! [SELECT * FROM task_phone_record 
                          where task_id = (?)
                          and call_times = ? 
                           #object[[I 0x4f60ec51 [I@4f60ec51] 1]

Is there a nicer way to handle different failure conditions than nested ifs?

(if (s/valid? ...)
    (if (s/valid? ...)
      ;; more nested ifs
      (throw (ex-info "Invalid ..."
                      {::error-id :validation
                       :errors "..."})))
    (throw (ex-info "Invalid ..."
                    {::error-id :validation
                     :errors "..."})))

Is it weird that I always feel I need an extra pair of parenthesis for anonymous functions 😅

(defn abracadabra
  "Find the number of c's in abracadabra"
  [input]
  (count (filter #(= \c %) input))
  

where would they go? I’ll say you get used to it and I don’t see it but interested to see where your your instinct says they go

I start typing something similar to -

#( (= \c %...

it makes a lot of sense, the lack of extra parens there trips a lot of people up

people try things like #(1) to return 1 sometimes, which of course doesn't work

… and (constantly 1) just seems so verbose!

More than verbose, it's just hard to remember the name of the function (at least for me)

constantly is great https://gist.github.com/hiredman/5644dd40f2621b0a783a3231ea29ff1a#file-yield-clj-L30

I get tripped up in repeat and repeatedly every now and then

Hey. How can I serialize a map entry differently from the rest of it? I have my app state in an atom and I have a datascript db inside my app state under :ds-db. Currently I’m calling (update app-state :ds-db ds/serializable) and then calling transit write. Can I write a function that can serialize my app state the “right” way without hard coding or passing in the path to the datascript db. Thanks!