Fork me on GitHub
#clojure
<
2016-02-06
>
jonahbenton00:02:50

right- they dispatch a function to the same thread pool; however, my understanding- not tested but easily testable-is that the function that is executed in the agent case is transaction aware, while the function that is executed in the future case is not.

alexisgallagher00:02:29

Ah. Good to hear my hunch wasn't completely nuts...

jonahbenton00:02:41

the idea is that the state encapsulated by an agent can participate in transactions

jonahbenton00:02:54

there is no state associated with a future

jonahbenton00:02:22

very interesting situation, though, thanks for bringing it up

alexisgallagher00:02:42

It's not as esoteric as it might sound. Basically, I'm working with SQS (amazon's queue service).

alexisgallagher00:02:03

After you take a message off the queue, you need to tell the queueu to delete the message to confirm that you handled it successfully.

alexisgallagher00:02:05

So I take a message.

alexisgallagher00:02:12

I update local state in a transaction.

alexisgallagher00:02:35

But SQS does not guarantee in order delivery.

alexisgallagher00:02:57

So I explicitly do not want to receive out-of-order deliveries. I will fail to ACK those, and then they will timeout and get redelivered later.

alexisgallagher00:02:47

So when I update local state, I check if the message is in order. If it is, local state changes in a transaction and then I want to delete the message in SQS (that's my side-effect).

alexisgallagher00:02:05

If the message is out of order, then I don't change local state, and I don't want to delete the message.

jonahbenton00:02:09

yes, got it, very nice.

alexisgallagher00:02:29

So I want this side-effect (ACKing the message) only when the transaction succeeds and when the local state actually changes.

alexisgallagher00:02:28

But to define a transaction I need to switch from an atom to an agent to get dosync. And then I need to introduce this spurious agent just to get send-off. So I suspect I'm making this harder than necessary.

jonahbenton00:02:17

well, these semantics- distributed consensus- are very difficult to maintain. e.g., in the case you're describing, the delete at SQS initiated by your agent fn may fail, which would result in redelivery of an otherwise in-order message.

jonahbenton00:02:16

it just depends how much it matters

alexisgallagher00:02:36

Yes, I haven't even opened the mental can of worms of what if the delete-message fails to deliver. In truth this system has low concurrecny requirements, so I'm going to call it "correct enough" if I handle out-of-order correctly.

alexisgallagher00:02:51

Would be more fun to work on something that justified perfect correctness.

jonahbenton00:02:07

have you read the raft paper?

alexisgallagher00:02:12

Looks like fun. Maybe I can find a peice of work that justified digging into it. simple_smile

cddr02:02:42

Is there anything easier than apache.commons.compress for creating tar.gz from clojure?

jonahbenton02:02:34

hey @cddr if you have them as command line utilities you can do something like this: http://blog.jayfields.com/2012/10/clojure-lein-tar.html

cddr02:02:45

Yeah that might be a bit easier at the risk of being less portable

alexisgallagher02:02:02

@jonahbenton: (btw, realized I was being an idiot re the agent and send-off. Can just return a value from the dosync form and conditionally use a future or whatever. No need to kickoff the side effect inside the transaction.)

jonahbenton02:02:45

hey @alexisgallagher: hmm...do you have multiple threads processing these SQS messages, or just one thread?

alexisgallagher02:02:27

I'll need to support multiple threads eventually

jonahbenton03:02:25

just a suggestion- if throughput matters it will be best to minimize the rate of redeliveries, so it may be more efficient and simpler overall to let one thread service SQS, do your own reordering of received messages, and then use send-off to both process the reordered messages and to ack SQS.

jonahbenton03:02:01

but the dosync + agent approach sounded ok- the problem with just returning a value from dosync is that unless the value that's being returned is a ref participating in the transaction and indicating success or failure, you don't know whether the transaction committed

alexisgallagher03:02:41

@jonahbenton: good point. I hadn't thought of that. An exception could stop the transaction. But what else?

cddr04:02:25

Hey folks. I've been working on some tooling for using samza from clojure. Would be interested to hear what you think. https://github.com/cddr/samza-config/

jonahbenton04:02:27

well...that's it, an exception or deliberate abort. the implementation works extremely hard to deliver on transactional semantics. which comes of course with tradeoffs- operations that involve transactions are orders of magnitude more expensive than those that don't. you're in effect asking to incur an extreme cost in order to get a specific set of semantics and behaviors. so- if the problem really requires transactional semantics, then the expensive tool being utilized to deliver on it should be employed in the way it was designed. if those semantics are not that important- which means- if the problem has other solutions that don't require transactional updates in a concurrent context- then transactions aren't needed

kul06:02:33

is it reliabled to destructure records like this (let [{:keys [a b]} (MyRecord. a b)) ?

kul06:02:04

i have seen weird behaviour like sometimes it works with :keys and somtimes with :strs

jethroksy06:02:54

@kul: i think it is

kul06:02:08

i am not sure

kul06:02:15

let me see if i can reproduce

kul06:02:49

(defrecord Foo [a b])

kul06:02:05

(let [r (map->Foo {:a 1 😛 2}), {:keys [a b]} r, f1 [a b], {:strs [a b]} r, f2 [a b]] [f1 f2])

kul06:02:12

(let [r (map->Foo {"a" 1 "b" 2}), {:keys [a b]} r, f1 [a b], {:strs [a b]} r, f2 [a b]] [f1 f2])

kul06:02:33

that eomji replace with :b

kul06:02:38

eh got it

jethroksy06:02:14

boot.user=> (def x (map->Foo {"a" 1 "b" 2}))
#'boot.user/x
boot.user=> x
<#C053K90BR>.user.Foo{:a nil, :b nil, "a" 1, "b" 2}

kul06:02:39

makes sense

jethroksy06:02:57

in general I try to avoid using strs as keys anw

Leonid18:02:35

Hi! Can anybody help with ztellman's gloss? When using header frame is it possible to pass parsed frame down to codec that is chosen to be next?

jonahbenton19:02:06

hey @von_zeppelin can you say a little more about what exactly you're looking to do?

Leonid19:02:24

@jonahbenton: I want to parse messages of a protocol in which every message starts with a predefined header, the header contains message type and current transaction id. With gloss/header it is easy to organize selection of the next codec based on type read from msg header, but I also want to pass tx id to the next codec, for example. Is it possible to do?

jonahbenton19:02:33

hey @von_zeppelin: i was considering gloss for a project, wound up not using it, so with the caveat that i don't have my hands that dirty with it- isn't it the case that when you provide gloss with a header for use in decoding data, you're providing it with instructions to decode both the header and the body data, so you'll get both data structures?

jonahbenton19:02:17

am not sure i'm following what you mean by "pass tx id to the next codec"?

Leonid19:02:25

@jonahbenton: What I meant is the following: we parsed some initial amount of bytes to decide what codec to use next, the next codec will start from the remaining bytes chunk, but initial amount of bytes also contained some useful info that I want to pass to the next codec, not only the remaining bytes

Leonid19:02:48

@jonahbenton: Thanks for your answer anyway. What lib did you choose in the end?

jonahbenton20:02:58

@von_zeppelin: header will pass the decoded header data to the function you provide, and that function is expected to return the codec to use with the remaining bytes. If you need to have the structure of the next codec be dependent on the header data- that looks straightforward, just extract the data you need and then generate the structure to provide to compile-file. However, if you need to have the next codec return a data structure that includes both data from the part of the stream it is responsible for decoding, AND data from the preceding block- to my eyes there doesn't seem to be a way to do that. Is that what you found?

Leonid20:02:00

@jonahbenton: Yes, the latter case

jonahbenton20:02:59

@von_zeppelin: we wound up rolling our own, we were dealing with an ancient bank communication protocol that we had to both read and write and my partner found some oddities that obviated any benefits from gloss. my recollection is that we had come into it with the understanding that our responses would precisely match the format of the data we received; however, we discovered that there were more exceptions than adherents to this

Leonid20:02:28

@jonahbenton: OK, thank you for help!

Leonid20:02:55

Yeah, it seems I'll need it simple_smile

jonahbenton20:02:20

just curious- why do you need to annotate the next frame with extra data? can't your data consumer do that?

Leonid20:02:58

At the end I want to have a parsed record that will also have as a field the current transaction id read from header

jonahbenton20:02:07

gotcha- and you don't want to reduce over the plain data structures gloss would hand back?

mheld22:02:09

man lighttable is super buggy

mheld22:02:41

it’s almost a shame that it’s so easy to use

mheld22:02:57

are there other IDEs that y’all like?

mheld22:02:08

emacs, lighttable, cursive, etc

seriousbug22:02:30

I'm using Emacs with Cider, clj-refactor, paredit and rainbow-delimiters. It is pretty nice if you are comfortable with Emacs

meow22:02:09

#C050AN6QW

mheld22:02:26

hah thanks

profil23:02:10

Anyone got ideas for how to update a static ring response on interval? I want to return a simple response containing some data, and every 3rd hour I want this data to be "updated" or recomputed