Fork me on GitHub
#clojure-uk
<
2018-08-30
>
agile_geek07:08:27

Bore da pawb welsh_flag

alexlynham08:08:06

@agile_geek what do you think burned you out about it?

agile_geek08:08:51

There were other factors I will not discuss in open channels.

alexlynham08:08:23

there's an element of the idioms thing that I def sympathise with

alexlynham08:08:19

did pro clojure on and off for 3+ years and continually felt dumb or like I didn't get it because of diff coding styles etc... but because it's a simple language in terms of its core you can at least read it and mentally parse it quite fast

alexlynham08:08:43

but stuff like use of protocols or records or OO constructs still throws me all the time in other peoples' clojure (for example)

alexlynham08:08:02

there's a bunch of other examples of stuff I don't do & others do (& vice versa)

mccraigmccraig09:08:27

E_DYNLANG is the thing that bums me out most... comprehending the correct shape of deeply nested structures and errors coming from changes in shape not being consistently propagated across the codebase

mccraigmccraig09:08:08

tests help, particularly with regressions, but static checking would help a lot more

👍 4
alexlynham09:08:45

I think plumatic/schema etc are good tools for that at an interface between systems level

👍 4
alexlynham09:08:04

and debugging data structures imo is def less painful than debugging objects

alexlynham09:08:15

but yeah there's some things that are just hard

mccraigmccraig09:08:45

schema helps for sure - it's too little, too late though, and while i think i would still choose clojure again were i starting this project tomorrow (trivial JVM lib integration is what swings it) it does mean that clojure is definitely not my ideal language. were there another plausible, functional, persistent-ds, jvm+js targeted lisp with decent static checking i would prefer that

alexlynham09:08:41

that's quite a specific wish list 🙂

alexlynham09:08:05

but I know what you mean

alexlynham09:08:19

however all things considered it's still an impressively well thought-out lang

alexlynham09:08:27

credit where credit is due

mccraigmccraig10:08:47

absolutely - i'm only down on clojure because it's so close to ideal... most other langs i don't care about enough to criticize

👍 4
alexlynham10:08:36

I know what you mean 🙂

maleghast08:08:17

Morning All!

👋 8
maleghast08:08:05

Quick question - anyone in here done the whole HTTP Upload -> S3 without__ a temp file on the disk?

maleghast08:08:28

I think that it can be done using Amazonica, but I am struggling to find a crib / example to work from... (My Google-Fu has been WEAK of late)

danm12:08:43

(defn build-s3-write-request [s3-bucket s3-key content-bytes]
  {:bucket-name s3-bucket
   :key s3-key
   :input-stream (ByteArrayInputStream. content-bytes)
   :metadata {:content-length (count content-bytes)
              :content-type "binary/octet-stream"}})

(defn write-s3 [content]
  (let [req (build-s3-write-request "bucket" "key" (nippy/freeze content))]
    (s3/put-object req))

danm12:08:20

Of course if you're not serialising using nippy you might need a bit more in there before you can just wrap it in a ByteArrayInputStream, but it shouldn't be too much

danm12:08:30

To get it back we have

(defn read-input-stream [input-stream]
  (with-open [outputStream (ByteArrayOutputStream.)]
    (io/copy input-stream outputStream)
    (.toByteArray outputStream)))

(defn get-s3 [bucket key]
  (-> (s3/get-object s3-bucket s3-key)
    (:input-stream)
    (read-input-stream)
    (nippy/thaw)))

danm12:08:41

If you want the definitions for the namespaces I can give 'em, but it should be reasonably clear 🙂

maleghast12:08:22

What the heck is nippy?

maleghast12:08:30

(I know that I could Google it 😉 )

danm12:08:10

It's a serialisation/deserialisation library by taoensso (also did the timbre logging lib and I'm sure some others)

danm12:08:21

What we were working with there was a map full of a load of java-time instances (Hurrah for no more joda-time and crappy Java time stuff now we're Java 8+), which aren't immediately serialisable to JSON with cheshire, and we didn't want to store as strings and keep having to parse and write them

danm12:08:53

So we just serialise the whole map to write to S3, and deserialise it on read from there

danm12:08:17

It's something we only do periodically anyway

danm12:08:32

If you had something that was just JSON serialisable or whatever, I'm not sure what changes you'd need to make. The content-type would probably want to be application/json, but I think you'd still need to provide/read an input-stream, so you'd need to convert the map to a string and then get the string into a bytestream somehow (The Java classes might Just Do It if they're given a string, I can't recall)

alexlynham08:08:23

but the code is in a place I can't see it any more

alexlynham08:08:34

there's a way to do it with amazonica iirc

firthh08:08:35

I did as well some time ago, but I don’t have the code either 😞

firthh08:08:12

Whatever you want to upload needs to go into that InputStream

alexlynham08:08:13

that looks familiar

firthh08:08:56

I don’t remember what the amazonica interface into that looks like though

alexlynham09:08:06

nah me neither

alexlynham09:08:27

is amazonica the one that's super dense & basically the tests are the docs?

mccraigmccraig09:08:10

@maleghast i went with the tempfile for s3 uploads - iirc i had to implement less to get it to work without blocking my threads

guy10:08:08

Morning people!

guy10:08:11

I love coffee 😄

3Jane10:08:22

well, if Clojure runs on Java, it is appropriate for Clojurians to run on coffee

😂 16
otfrom10:08:46

works for me. 🙂

guy10:08:18

So do any of you kind folks go on certification courses or things like that?

Conor10:08:36

They tend to be fairly useless in my experience

guy10:08:58

Yeah i am feeling the same thing, but just want to know if anyone has found a useful one

Conor10:08:42

Perhaps I've not been on the right ones

Conor10:08:41

I would not make a hiring decision based on someone's certification or lack thereof

guy10:08:54

its more for personal development rather than getting a job

Conor10:08:07

Just read a book, yo

👍 4
guy10:08:49

I do like reading books and will continue to do so 😄

guy10:08:08

if that makes sense?

3Jane10:08:46

in that case I think it depends on what your learning style is more than anything.

3Jane10:08:14

and on what your accountability model is (some people need external accountability like a gym buddy or trainer, some can do it themselves, some can only do it if they’re NOT supposed to do it)

3Jane10:08:56

I remember reading an article about why MOOCs have not turned out to revolutionise education: they have a huge dropout rate, something like 90%

firthh10:08:04

I did a 2 day Clojure course with Stuart Sierra before I’d written a single line of Clojure. I found it extremely useful. If you’re not doing it for the certification I think there are really good courses out there

yogidevbear20:08:26

Hi Hugo. Was this an in-person course or something online?

firthh08:08:31

It was in person. Back in Australia before a functional programming conference. I have no idea if he still does them

yogidevbear10:08:38

That must have been pretty awesome. I'm not jealous in the slightest 😉

yogidevbear10:08:15

Do you remember what sort of things he covered?

firthh13:08:18

Haha it was. Right before I started on my first Clojure gig 😉

firthh13:08:11

I was the language from the complete basics. I think we made it as far as macros in two days

firthh10:08:22

Though none that come to the top of my head

alexlynham10:08:16

I think I'd never look for certifications on a CV really

alexlynham10:08:35

the only time I'd think "okay, that's good" is an AWS cert for a platform engineer

alexlynham10:08:48

just because for some reason platform folks are v variable in quality

alexlynham10:08:59

(well, no more than software engineers)

alexlynham10:08:11

but maybe it's that the pool is smaller

3Jane10:08:49

(and in terms of personal experience, I’ve done internal company trainings in a giant corp, and certification with external company (inviqa iirc) and also (human) language courses, and in each case I found that I self-teach faster and more effectively, but benefit hugely from dense reference-like resources. Thus, I agree with “read a book”, if that includes “and play/write your own test code at the same time”)

👍 8
firthh10:08:07

Some of the workshops that get put on with conferences have looked interesting, though I’ve never been to any. For example - https://craft-conf.com/2018/workshops

3Jane10:08:37

thanks for that! found the book the main interesting workshop was based on 😄

firthh11:08:16

Amazing. Glad I could help :)

3Jane10:08:10

and certs are very industry-specific; apparently if you want to do security they’re worth the money. Cisco used to be worth it as well, I don’t know about these days.

guy10:08:35

ok thanks all!

guy12:08:15

welcome!

guy12:08:17

back 😄

maleghast12:08:49

Thanks All, specifically @carr0t @mccraigmccraig @alex.lynham @firthh and @jasonbell for S3-based input... I had to do a conference call and then I was head-first in some research and then lunch and finally I remembered that I asked a question in here...

maleghast12:08:43

@mccraigmccraig - When you went with tempfile on the disk in order to no block any threads, did you have a separate app / worker pushing the files to S3?

danm12:08:56

I;ve not found any training courses I've done to be hugely useful, but I think I've possibly not done them early enough

👌 4
danm12:08:40

Normally I seem to pick something up, we work or I experiment with it for a few months on and off, then I/someone goes on a training course to see if we're missing everything/if we're doing it right

maleghast12:08:42

I've had little bits of training here and there that I found very helpful, but I know what you mean @carr0t

danm12:08:10

Which means often you're sat there for 95% of the course wishing they'd get onto more meaty stuff

danm12:08:42

And yet at the same time there's other people there who are new to the (whatever), who are asking "Why are we doing X? What does Y mean?", and it often isn't covered so well in the course

mccraigmccraig12:08:36

@maleghast just looked it up - yes, we're using a manifold.deferred/future to do the transfer, which uses a separate thread

danm12:08:56

Even the Cisco courses I did, and sysadmin cert ones, were "enter these commands", but no real explanation of what the commands were for and why you were entering them. Which was fine for me, as I'd been doing net/sys adminning for a while, but meant that I didn't get as much out of it as I would have liked, and ended up helping other people understand what it all meant

maleghast12:08:26

@mccraigmccraig - Thanks, that's a paradigm I grok - I have a similar approach to doing database inserts off the back of async API calls.

mccraigmccraig12:08:23

it's not my favourite @maleghast - it's one of a very few places in our app where we aren't properly non-blocking, but doing it properly would be too much work for something that doesn't happen that often, so a thread it is

otfrom12:08:57

training courses are often good (I've helped teach some and learned clojure from Stu Halloway many moons ago, and having him to ask questions was really useful)

otfrom12:08:31

certification is a different thing. Sometimes even the "useless" ones are handy as they can be used as a filter by some bits of the org that aren't as clueful, but can help you get into a team that does have a clue

otfrom12:08:39

(I'm thinking things like scrum certs)

otfrom12:08:00

some of the scrum trainers will actually give you some good info in addition to the bit you need for the certification

maleghast12:08:01

Yeah, I found doing the Juxt Advanced Clojure with @malcolmsparks and @jonpither last year was really helpful.

maleghast12:08:05

I was 2 years from having done any "Clojure for money" and it really helped me get back into the swing of things, and pick up on some new (to me) ideas.

otfrom12:08:37

one of my questions to Stu H (after being frustrated by common lisp, emacs lisp and scheme) was "will map and reduce be the way to actually write production code that is suitably performant" and his answer was "yes"

✔️ 12
otfrom12:08:01

I think reducers, transducers and core.async show that to be true

otfrom12:08:11

and I wouldn't have been able to get that from a book

maleghast12:08:39

Yeah, that is the sort of thing that you need from "the Horse's mouth"...

otfrom13:08:49

what do you think horseshoes are?

otfrom13:08:01

you don't get to leave now @thomas

otfrom13:08:12

only in the end

thomas13:08:46

in that case I don't know I'm 'fraid.

thomas13:08:29

ooohhhhh..... parens!

alexlynham13:08:34

is a horseshoe just a (partial reduce +)?

otfrom13:08:41

obviously they need to turn around to close the statements

otfrom13:08:13

@alex.lynham that use of partial really hurts me given the arities + takes. 😉

alexlynham13:08:37

haha I partially (see what I did there) wanted to troll

otfrom13:08:50

uauahghgauahghgughaghhg

alexlynham13:08:19

the 'joke' also I guess wouldn't work with (reduce + <horse?>)

alexlynham13:08:32

I overreached with this one I think

alexlynham13:08:36

it was tenuous at best

thomas13:08:58

quite an achievement... do we finally talk about Clojure... do we somehow manage to get back to other things in no time.

3Jane13:08:13

there’s a reason they call it a stable release.

😱 8
😍 4
thomas13:08:45

:drum_with_drumsticks:

3Jane13:08:59

¯\(ツ)

alexlynham13:08:51

wait hang on, I was thinking of sigma, not omega

alexlynham13:08:08

I don't have a lousy syntax joke about the end of a set

thomas13:08:05

so... game, set and match to @lady3janepl and me?

guy13:08:17

solid jokes folks 10/10

alexlynham13:08:04

@thomas I hang my head in shame

🔔 4
guy13:08:38

the bell of shame

😱 4
alexlynham13:08:50

i know it well

😂 4
jasonbell13:08:15

@guy <<the bell of shame>> Now what have i done?

🔔 12
😱 8
otfrom15:08:43

if you have some reference data in a file that you use, what is your preferred way to make it available in your application?

otfrom15:08:01

say a csv file that you turn into a lookup table that gets used by lots of other functions

guy15:08:53

how long have you got to complete the task?

guy15:08:29

or maybe i read that wrong

guy15:08:53

Do you mean, where do i store the csv? like sticking the csv in the resources folder?

mccraigmccraig15:08:43

@otfrom depends on how it's going to get used - all yapster's bulk imports get turned into records in a bunch of append-only import tables, basically a direct representation of the import source data in a common scannable format

mccraigmccraig15:08:25

after that those import tables get joined with the tables from the live database and used to generate changes

otfrom15:08:34

mostly I'm trying to avoid doing it in a def as if I get a problem with the file, it will blow up during compilation

otfrom15:08:38

the sizes are very small

otfrom15:08:09

so no need to put it into a database, and again it is a batch process so I only need it in memory for the script to run

otfrom15:08:48

that was my thought, but wanted to get other opinions

otfrom15:08:07

turn the calls from foo into (foo), but have the function memoized

otfrom15:08:16

so that it wasn't reloading every time

otfrom15:08:55

but at least then if it goes bang it happens at run time rather than compile time (and my exceptions would be easier to read and the ns would actually load)

otfrom15:08:04

@dominicm not used delay before

mccraigmccraig15:08:14

delay is more appropriate than memoize here i think

otfrom15:08:16

that looks pretty good actually

mccraigmccraig15:08:25

either way don't do it at compile time

otfrom15:08:52

makes startup a pain too (when the files get biggish or the processing long)

otfrom15:08:31

future is tempting too

otfrom15:08:07

ah, future attempts to do things immediately, delay waits until it is dereferenced

otfrom15:08:10

delay is what I want

danm15:08:11

Interesting. Not come across delay before

otfrom16:08:46

mmm... tried combining some futures with some JNI and that... didn't go well.

otfrom16:08:56

I do like being able to core dump the jvm tho 😄

otfrom16:08:52

pretty sure the JNI is trying to be "clever" about how it manages file resources and not doing it very well

guy16:08:32

Hi folks

guy16:08:44

I’ve probably asked this before

guy16:08:00

but why does (conj nil nil) produce (nil)

mccraigmccraig16:08:48

@guy 'cos you are adding a nil value to an empty list, giving you a list with one nil element

guy16:08:00

so nil is an empty list?

otfrom16:08:47

@mccraigmccraig the resource handling looks to be pretty buried in the library. It is doing some kind of reference counting. I'm just gonna run it single threaded

guy16:08:47

cos (seq nil) is nil ye? so it counts as empty?

otfrom16:08:47

(conj nil 1)
(1)

otfrom16:08:09

conjing on to a nil gives you a list with the 2nd arg in it

otfrom16:08:21

the first nil is the collection you are conjing onto

guy16:08:29

but how is a nil a collection

otfrom16:08:44

nil isn't a collection

otfrom16:08:50

from the docs

Added in 1.0
  conj[oin]. Returns a new collection with the xs
    'added'. (conj nil item) returns (item).  The 'addition' may
    happen at different 'places' depending on the concrete type.

guy16:08:00

oh right

mccraigmccraig16:08:08

@guy 'cos it's always been that way since way before clojure was a twinkle in rh's eye

guy16:08:18

ok thanks!

otfrom16:08:33

more nil punning sugar I think

guy17:08:13

what is nil punning sorry

guy17:08:35

is that like returning nil when something doesn’t return what you want it to?

otfrom17:08:56

static public IPersistentCollection conj(IPersistentCollection coll, Object x){
	if(coll == null)
		return new PersistentList(x);
	return coll.cons(x);
}

3Jane17:08:39

I think this is a Lispism

3Jane17:08:57

(in that nil is the same as an empty list *in Lisp)

3Jane17:08:08

it’s not that it’s a collection, it’s literally defined as an equivalence

3Jane17:08:28

oh, that’s what it’s called 😄

guy17:08:41

oh great thanks @otfrom

otfrom17:08:47

@lady3janepl except that an empty list doesn't equal nil

(= nil '())
false

3Jane17:08:33

(but yeah)

minimal17:08:41

it’s seqable? and (seq nil) returns nil because they made it that way

3Jane17:08:22

(one interesting thing I found is that you can extend-protocol to nil)

3Jane17:08:07

(but then if you have a more than one whatever-type implements the protocol, it’s not possible to fluently cast between nil and “empty type X”. I’m guessing that’s why there’s no equivalency in Clojure - at the base level things just aren’t lists anymore?)

sundarj17:08:11

nil represents an abstract concept of nothing. what happens if you conj 1 to nothing? well, conj returns a collection, so it must be a collection containing 1, hence (1). similarly if you do (map inc nil); map returns a sequence - what's the sequence equivalent of nothing? an empty sequence, hence ()

sundarj17:08:49

nil not being the same as () is one of the differences Clojure has from other lisps, documented here: https://clojure.org/reference/lisps

guy18:08:48

thanks for this!

sundarj18:08:07

np 🙂

❤️ 4
3Jane17:08:09

interesting it still chooses list as the type to convert nil to 🙂

3Jane17:08:44

(by which I mean: if you conj to a vector, you’ll get a vector; if you conj to nil, you get a list)

mccraigmccraig17:08:24

nil-punning is the devil's work

truestory 4