Fork me on GitHub
#clojure
<
2018-01-07
>
ivana00:01:29

Hello. Can anyone tell how to get wright behaviour from this: (let [fs (cycle '(inc dec))] ((second fs) 10)) i can make it work, but eval is disabled on 4Clojure, so i want a solution without it (let [fs (cycle '(inc dec))] (eval (list (second fs) 10)))

noisesmith00:01:20

the elements in '(inc dec) are not the functions inc and dec, they are just symbols

noisesmith00:01:44

can you use [inc dec] instead?

ivana00:01:53

my shame, i try to refact code

ivana00:01:22

i want a lazy-seq of funcs

noisesmith00:01:25

if not, you can look up the real function? eg. {'inc inc 'dec dec} - you can use that to look up the function from the symbol

noisesmith00:01:38

@ivana then (cycle [inc dec])

noisesmith00:01:49

that is a lazy-seq, and it's the actual functions instead of the symbols

ivana00:01:01

ok, thanks, i see my mistake

ivana00:01:14

But its very strange for me, that when i create a function fn [x & fs] and want a honest real lazy stream of real functions, i can NOT get it from (cycle fs) but only this way (cycle (into [] fs))

noisesmith00:01:40

@ivana it's because 'inc is not inc

noisesmith00:01:57

quoting creates symbols, symbols that are not resolved, and therefore don't get looked up

noisesmith00:01:20

'(inc dec) is the same as ['inc 'dec]

noisesmith00:01:35

you can just use (cycle fs) if fs is already a collection

ivana00:01:55

which quoting in & fs ?

ivana00:01:24

(cycle fs) doesnt work

noisesmith00:01:26

so the thing you showd above was not the actual problem, and you aren't using quote

noisesmith00:01:03

(cycle fs) and (cycle (into [] fs)) are effectively the same

noisesmith00:01:42

you don't need eval, and the code should just work

noisesmith00:01:01

if eval fixed it, then somewhere, some code was quoting symbols

ivana00:01:13

something strange... i hope it is the same, and 4Clojure test shows it. but in my repl i had a mistake. i check one more time.

noisesmith00:01:40

maybe you were quoting in your repl, that would explain it

noisesmith00:01:09

the common mistake is that people see something like '(1 2 3) to make a list of numbers, and get the false conclusion that '(x) is the same as (list x) or [x]

ivana00:01:41

my shame again, i can't reproduce that strange behaviour in repl now

qqq00:01:34

is there any library for translating a highly restricted set of clj/cljs into ultra fast wasm ?

qqq01:01:14

this is ugly: I'm writing some lcojure code which is going to, at run time, generate a *.c file, call gcc on it, run it, read back the data (via output written to file) are there any clojure libraries that may be of use with this unholy mess ?

noisesmith01:01:02

there's code in pixie lang which does this - it's how it implements some interop stuff

noisesmith01:01:16

you probably can't reuse the code, but you'd probably get some good ideas from it

Tom H.01:01:17

May also be worth investigating https://github.com/carp-lang/Carp

Tom H.01:01:46

A clojure-like lisp that compiles to C

lilactown03:01:01

is there any magic to using deps.edn in a lein project?

lilactown03:01:27

do I just put my deps in deps.edn, skip the lein coordinates and call it a day?

seancorfield04:01:52

@lilactown You will need a Leiningen plugin that reads deps.edn.

seancorfield04:01:35

deps.edn is for tools.deps and the clj script. Leiningen (and Boot) do not read it natively. Leiningen has a plugin that reads the file. Boot has a task that uses tools.deps to read the full series of deps.edn files (the system file, your user file, and the project file).

vemv04:01:38

in case that plugin doesn't exist yet, you can run arbitrary clojure code (but without external deps) within project.clj, using unquote syntax

vemv04:01:06

I guess you can read deps.edn with slurp, do a couple transformations, and embed it into :dependencies

qqq05:01:02

I'm using Java 8. I'm trying to call : https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#createTempDirectory-java.lang.String-java.nio.file.attribute.FileAttribute...- I'm using: (java.nio.file.Files/createTempDirectory "/tmp") I am getting error 'no matching method: createTempDirectory`

qqq05:01:45

apparently the correct invocation is:

(java.nio.file.Files/createTempDirectory
 (java.nio.file.Paths/get (java.net.URI. "file:///tmp/"))
 "some-prefix" (into-array java.nio.file.attribute.FileAttribute []))

noisesmith05:01:57

right, varargs always requires an array arg coming from clojure, even if it's empty

qqq05:01:45

and then, the first arg needs to be a path, not a string, but to construct a path, we need to construct a URI first, and to construct a URI, we need to prepend "file://"

noisesmith05:01:24

paths/get works with a string, but it's varargs 😄

noisesmith05:01:30

or non varargs with a URI

noisesmith05:01:29

+user=> (java.nio.file.Paths/get "/tmp" (into-array String []))
#object[sun.nio.fs.UnixPath 0x5cbe877d "/tmp"]

noisesmith05:01:22

if trying to conserve characters

+user=> (java.nio.file.Paths/get "/" (into-array ["tmp"]))
#object[sun.nio.fs.UnixPath 0xc35172e "/tmp"]

qqq07:01:46

(let [f #(list % 1 (do (println "foo") 23))] 
    (doseq [i (range 5)]
      (f i)))
the println executes 5 times

qqq07:01:11

is there a way to cache the args in a # %, or do I need a let binding first ?

robert-stuttaford08:01:34

curious - i guess with https://clojure.org/news/2018/01/05/git-deps and multiple concurrent project development (e.g. library project(s) + app project) one would point at local repos and local hashes? could one use “HEAD” as a hash, thereby always using the latest code?

dominicm08:01:50

@robert-stuttaford there's a :local type already 🙂

robert-stuttaford09:01:46

doh, of course there is! 🙈 thanks @dominicm

carkh12:01:55

i may have misunderstood this "facilitates parallel development of sibling libraries" in the announcement of git-deps. Don't we still need to restart the repl when we use a newer version of a library ? Are there reliable tools to update the :rev key and load the library in the current environment ?

qqq12:01:50

1. Int is a 32bit int. 2. Float is a 32bit float. 3. IntTensor is a {::shape (vector of Int) ::data (vector of Int, of size (reduce * shape))} 4. FloatTensor is a {::shape (vector of Int) :: data (vector of Float, of size (reducd * shape))} 5. i need to store a List of arbitrary length, where each element is either Int, Float, IntTensor, or FloatTensor 6. I need to this to be easily readable in both Clojure and C (so I'm not using read-string or edn, I'm laying stuff out directly in bytes.) 7. Everything is 32bit. Machine is x86 (little endian). 8. Any suggestions / advice? (If not, I'll be doing something adhoc, but am wondering if there is a 'standard' way to do this.)

noisesmith16:01:22

if it needs to be usable from C, use byte-arrays of the specific lengths needed, and on the java side you can create ByteBuffer objects to read the data back, on the C side you can treat it as a struct

noisesmith16:01:36

there's a library for this stuff too, ztellman/gloss

noisesmith16:01:57

but the ByteBuffer API isn't so hard to use - tell it where the data starts, and what type to turn the bytes from that index into https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

qqq20:01:52

@noisesmith: yeah, I'm going to just use ByteBuffer; thanks!

qqq13:01:29

==== are there builtins for convedrting ints/floats <-> byte-array ink clojur ? from googling, there seems to be a bunch of custom code but no builtin

noisesmith16:01:45

ByteBuffer does this, as mentioned in a previous thread

noisesmith16:01:53

+user=> (def bb (java.nio.ByteBuffer/allocate 64))
#'user/bb
+user=> (seq (.array bb))
(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
+user=> (.putFloat bb 0 666.0)
#object[java.nio.HeapByteBuffer 0x2dc995f4 "java.nio.HeapByteBuffer[pos=0 lim=64 cap=64]"]
+user=> (seq (.array bb))
(68 38 -128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
+user=> (.putFloat bb 4 420.0)
#object[java.nio.HeapByteBuffer 0x2dc995f4 "java.nio.HeapByteBuffer[pos=0 lim=64 cap=64]"]
+user=> (seq (.array bb))
(68 38 -128 0 67 -46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
+user=> (.getFloat bb 0)
666.0
+user=> (.getFloat bb 4)
420.0

pasi15:01:57

hm, I guess there's no better channel to ask Dali questions.. is there a way to rotate a group of objects? I'm using place/align to create "reusable objects" (i.e. pretty much using them as a grouping method). Is there a mechanism for rotating the entire layout object?

lspector16:01:11

anyone know a super simple way to run a clojure terminal session in the (paid) cloud? (please lmk if there's a more appropriate forum). What I mean is that I'd like to do something that gets me a command line prompt (could be in a local terminal or in a browser or whatever), lets me do a git clone and then a lein run > out, disconnect if I want, check in and kill (or not) and eventually download out and shut the thing down, and pay. Is there a simple way to do this?

lspector16:01:14

maybe important to note that the code I want to run does no communication with the outside world -- just a lot of computing and dumping results to out

noisesmith16:01:42

if you know what your deps are ahead of time, you can make an uberjar and then you don't need lein, all you would need is your uberjar and a jvm

noisesmith16:01:00

I think that expands your options and simplifies things a bit

lspector16:01:13

thanks @noisesmith -- yes, the deps are small and stable... suppose I do make an uberjar, then what's the next step?

noisesmith16:01:30

@lspector the uberjar can be run directly with a jvm - that's the only other thing you need

noisesmith16:01:12

with aws lambda, you can even skip the part where you set up a server, and just let lambda figure out the rest - just send a request from somewhere (that's the model) and collect the output

noisesmith16:01:35

(after uploading the jar to s3 and doing some config)

noisesmith16:01:01

thinking about it more, yeah lambda is probably the lowest complexity thing for that

noisesmith16:01:25

though heroku can run from a git repo - that's also an option (they have leiningen on their end)

lspector16:01:25

unfortunately this is all still pretty mysterious to me... suppose I want to do lambda (or heroku?), is there command I can issue or website that i can go to and say "here is my uberjar, please run it and let me get the output, and let me monitor and kill it as it's running if I want, and bill me"?

noisesmith16:01:25

lambda is closer to that, but both are oriented to request / response - assuming a request will be coming from somewhere else

noisesmith16:01:09

@lspector there's a lambda walkthrough - it's not too complicated https://aws.amazon.com/blogs/compute/clojure/

lspector17:01:25

ah... took a quick look and see that "not too complicated" is observer-relative 🙂

noisesmith17:01:45

it's lein uberjar plus one weird command line

noisesmith17:01:03

once the upload is done, it bills by the millisecond, which is cool

noisesmith17:01:03

a much more manual option is to go to eg. digitalocean (there are a lot of options, that's just one), create a server, install java, then run your uberjar with java

noisesmith17:01:10

it sounds like that's all you need

lspector17:01:50

huh... i believe you! but fwiw there's a lot in the lambda thing i don't understand, like what those arguments to that weird command line are, or what the :methods thing in the ns is, or what POJO is, or how AWS would know who i am and how to bill me (and how much?). I will look into all of this -- thanks!

lspector17:01:22

also just context: i have access to a lot of servers on which i do something like i'm asking for here using just ssh and scp, but of course these have java and leiningen installed... and sometimes when the demand is high i just want a few more nodes, and i'm hoping that i can pay somebody to do something similar on their hardware...

noisesmith17:01:25

your authentication sets that up, it's stored in a dot file the aws command uses

noisesmith17:01:31

err, the billing that is

noisesmith17:01:34

if you already have that set up, you could look for a place that has short term servers...

lspector17:01:44

so i guess step 1 on the aws path is to set up aws, which will involve something about authentication, and presumably I can do by following directions at https://aws.amazon.com?

lspector17:01:22

okay, will look into that, thanks

lspector17:01:04

but your comment about "short term servers" also sounds like it might be better... is there someone who offers those pre-setup for running Clojure or just Java jobs?

noisesmith17:01:22

good question

mpenet18:01:37

I wonder how expensive it would be to run a containerized clj app in gke. It can prolly vary a lot depending on the setup

mpenet18:01:04

For one off playing stuff it might just be free

jgh19:01:26

ive been messing with docker + clojure for a bit now because i’m building a site with that stack. I’m also building out a clojure docker api to help me automate orchestration and stuff a bit (also as a bit of a learning exercise to get to know how it all works better).

jgh19:01:04

It’s pretty easy to get a clojure app containerized and running, and docker-machine + docker compose makes it easy to provision and run a server + clojure container

jgh19:01:30

personally I’m going with docker swarm for orchestration/cluster management because it all fits pretty nicely together

jgh19:01:53

maybe i’ll switch to kubernetes some day, but for now im just learning the docker ecosystem

noisesmith19:01:20

in my experience making an uberjar, putting the uberjar, and running it via a jvm are pretty simple - would docker be simpler than that?

mpenet20:01:49

It s simple once the jvm is installed, the service demonized, logging/metrics etc setup for the service etc... containers allow to "package" it all nicely

jgh19:01:42

with digital-ocean + docker-machine i can spin up a $5/mo server and deploy a clojure service in a couple mins

jgh19:01:24

umm i think it would be about the same…there’s a higher up-front cost I think but once you have your compose/stack file and you’re able to automate building your images it’s really easy to replicate across servers

jgh19:01:55

the nice thing it’s for is as you guys were mentioning short term servers

jgh19:01:25

spinning up and tearing down additional machines is trivial, with the caveat that startup for e.g. digitalocean is about 50 seconds

jgh19:01:35

some providers may be faster or slower

jgh19:01:02

i just like digitalocean cause it has a nice api and is independent 🙂

jgh19:01:22

once you have a warm server it’s just a second or so to deploy a container (depending on whether the image is cached and whatnot of course)…some people have managed to get container startup to like ~300ms and serve a single web request on it…which is obviously absurd but goes to show how quickly it’s possible to startup new instances

mpenet20:01:40

Kubernetes is quite easy to use too, as long as you rely on somebody else (google/amazon) for the cluster.

mpenet20:01:10

Or minikube for playing

mpenet20:01:49

We re prolly slightly #off-topic tho

jgh20:01:19

there’s #docker

pasi20:01:12

drawing a few objects on a pdf document appears to be harder than I imagined -.-

valtteri20:01:38

@lspector What inputs do you provide to your code and where are you storing the results? If single computation task takes less than 5min (there’s a hard limit for single execution) I’d probably use lambda to run the code and make the code output the results into some persistent storage (db service, s3 object storage.. or whatever suits your data best). AWS provides also traditional virtual servers, they’re called “EC2 instances”. You need an AWS account in order to use Lambda or provision servers. If you choose to try out AWS there’s #aws and #aws-lambda Heroku was also mentioned, but imo it’s more oriented towards building web-apps than running background jobs. Well.. there’s concept of “worker instances” for running backround jobs… but you’d need to figure out when/how to start the workers and how you’d provide the jobs for your workers and where to store the outputs.

pasi20:01:55

I guess I could give it a try at some point. Right now I'm struggling with clj-pdf and Dali. I'm close but rotation is giving me a headache

valtteri20:01:17

Yep, why I love url-to-pdf is that it’s often easier to create HTML of your layout and let Chrome Canary deal with the PITA part of making the PDF.

pasi20:01:50

I think I solved my problem just now 😛

pasi20:01:02

typing my complaint here made me realize what I was doing wrong, heh

pasi20:01:15

I mean, it's not the cleanest approach but hey. It's a pdf file. If it works, it works.

jgh20:01:58

there’s hyper.sh which is for FaaS type stuff as well, I’m not sure if they have the same 5-min limit like lambda does or not

jgh20:01:14

on their website the example is wordpress + mysql so I guess it’s not really time limited

jgh20:01:21

but it does bill per second

lspector20:01:39

@valtteri: my code takes no inputs except from its own source code (and normally from command line arguments to lein run, but I could bake those into the source if necessary). The results all go to standard out, which I pipe to a text file. Runtimes are usually much more than 5min -- usually a couple of hours to a couple of days. Occasionally I'll kill something after a few minutes, but usually I'll want them to run much longer. It's rare that I know in advance how long I'll want to let it run, so it's important that I can look at the output and kill it if I want to.

noisesmith20:01:22

@lspector oh with run times like that you don't want lambda, I didn't realize they were that long

lspector20:01:56

@noisesmith ah thanks -- i suppose that's because of pricing? i hadn't figured that out yet

noisesmith20:01:24

they also have time outs, it's not meant for long jobs

noisesmith20:01:02

and yeah, it's priced to optimize short usage times

lspector20:01:34

ah too bad -- not sure where to turn in that case

valtteri20:01:10

@lspector in that case I think traditional virtual servers are probably the simplest choice for you. Especially if your workflow is pretty much manual, meaning that it’s ok that you start the server and launch and monitor your code execution by yourself.

lspector20:01:49

@valtteri that sounds good -- do you have a pointer to getting a traditional virtual server of this sort, ideally set up to run clojure/java out of the box?

jgh20:01:23

hyper.sh seems like it might be a simple option for you, you’d just have to containerize your application

valtteri20:01:23

One moment, I’ll check if I can quickly find something. Let’s continue on 1-1 chat so we don’t pollute this channel with #off-topic 🙂 Or actually.. It’s not off-topic since we’re figuring out how to run easily Clojure in the cloud.

jgh21:01:53

So here’s an example of a Dockerfile for an image I use for my services. In my case I have the services broken up into their own apps, and I simply copy all the apps into one services image and when running the container I run the appropriate app, in your case you’d just have the one: https://gist.github.com/jgh-/9fc46bdfc1bf8413990573ee04d6af10

jgh21:01:45

and then when running the container you give it the command java -jar /opt/hunting/hunting.rules.jar -c /etc/hunting.clj .. and that’s it.

jgh21:01:50

now with a single app in the image you can use the RUN directive in the dockerfile (with that command) and it’ll run that instead of having to manually do it

jgh21:01:31

so you’d basically make the image, push it to docker hub, and then pull it via hyper.sh and run it for however long it needs to run then shut it down when you’re done and you’re not dealing with an idle server.

valtteri21:01:19

@lspector There seems to be Java 1.7 pre-installed on out-of-box EC2-instances on AWS. If you uberjar your code it should be quite simple? Internet is full of instructions how to create AWS-account and how to launch EC2 instances. https://www.youtube.com/watch?v=Xs0g_ZEv2bw

valtteri21:01:50

@jgh I think Docker is awesome and I use it for many things at work.. But it’s not maybe the easiest to begin with. I find my self quite often fiddling with Docker stuff as a side-track. + there’s some mental overhead in learning the concepts.

jgh21:01:05

of course, it’s a little more up-front cost for payoff down the road

jgh21:01:13

sure beats fiddling with saltstack though 😉

ghadi21:01:39

I was going to add to the earlier discussion about easiest set up for a repl in the cloud: don't use lein. Use the server repl built into clojure 1.8+

ghadi21:01:20

lein can be very slow to startup, clojure socket REPL in concert with the new official Clojure launchers ( clj ) is great. https://clojure.org/reference/repl_and_main#_launching_a_socket_server https://clojure.org/guides/deps_and_cli

ghadi21:01:47

(Super straightforward with kubernetes / docker too)

lspector21:01:36

@valtteri cool -- it sounds like EC2 instances may be best for me.... although there's a lot in that video so far (5 min in) that's quite mysterious to me... and seemingly not necessary? anyway, i'll watch through the end and google up some more

lspector21:01:47

@jgh i've gathered that Docker is a good way to go for lots of stuff, but i'm not up on it yet and so i'm looking first for things that don't require new concepts/skills

jgh21:01:01

fair enough 🙂

valtteri21:01:04

@lspector That was just the first result in Youtube, didn’t validate the contents. 🙂 And just to clarify: AWS is just one provider of virtual servers. There are many others. I happen to work with AWS services and therefore it’s my first recommendation.

jgh21:01:51

digitalocean’s dashboard is a bit more approachable imo, but a virtual server is a virtual server..once you get to that [insert your favorite linux distro] command line it doesnt matter much after that

lspector21:01:02

@ghadi nice to know, although fwiw startup time isn't an issue for my project (unless it takes many minutes)

lspector21:01:02

@jgh do you happen to know if a digitalocean virtual server comes with Java set up?

jgh21:01:31

that’s a good question

jgh21:01:01

it doesnt look like it, at least through the web interface…it’s possible the api has other images

jgh21:01:46

you can create a droplet (their version of EC2) with a snapshot or backup of a server you’ve created before though.

jgh21:01:13

so you could create an ubuntu image and install java and whatever other dependencies, then create a snapshot of that server and use the snapshot for future instances if you intend to teardown/recreate

lspector21:01:38

thanks all -- so much i don't understand here and from googling "how to run a clojure program on ec2"... all very interesting-looking, but bunches of learning curves and cascades of new terminology and steps that I'm not sure how to do

Chris O’Donnell21:01:05

@lspector if you're willing to use digital ocean, you should be all set to run a command line application after completing step 1 of https://www.digitalocean.com/community/tutorials/how-to-deploy-a-clojure-web-application-on-ubuntu-14-04

jgh21:01:23

there should be a #cloud channel or something like that

valtteri21:01:28

@lspector if you know Linux and ssh that’s pretty much all you need. There’s lot’s of confusing terminology around cloud technologies, but basically starting and running virtual servers is same as firing up any Linux box. You provision hardware and start the server using your browser and then you’re given an ip-address where you can ssh into.

jgh21:01:56

^^ exactly

Chris O’Donnell21:01:36

if that's not something you're comfortable with, I believe there's a link to a pretty step-by-step guide for provisioning and getting access to a digital ocean machine at the top of the guide I pasted above

lspector21:01:28

i don't actually know much about linux, at least in terms of admining or installing or configuring things...

lspector21:01:45

i use a cluster that runs some kind of linux, but i don't admin it, and i just ssh in and run programs, and do simple unix commands (lots of grep!), etc.

valtteri21:01:53

@lspector I think you have sufficient amount of knowledge to start a virtual server and run your jar there. We just need to find you good and clear instructions. Or someone to guide you through the initial steps.

lspector21:01:15

and i run programs on large numbers of cluster nodes using some scripts that others have written... but what i'm aiming for here is just what i can do by sshing into a single server, moving some code there (scp or git), running it (`lein run` or java with an uberjar), and getting the output back to a local machine (scp or whatever)

lspector22:01:37

@codonnell the digital ocean page you linked (and the places it links like https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-14-04) look promising, but i'll have to spend more time with them to be sure if i understand. Would i have to do all of that every time i want to run a program in the cloud?

jgh22:01:32

i made a #cloud channel so we can move this there 🙂