Fork me on GitHub
#aleph
<
2019-03-05
>
ahmed1hsn05:03:46

How do I approach network programming in Clojure as beginner?

niklas.collin09:03:02

Seriously speaking, you need to be more specific than that

ahmed1hsn09:03:03

Trying to implement a server to host multimedia like images, videos. I'm a beginner.

niklas.collin12:03:12

Ok. And apparently you are planning to use Aleph. Well, the easiest solution is to just return an InputStream object to your media file to the browser

niklas.collin12:03:40

This allows streaming the file from filesystem to the browser without you having to first load it into server memory

niklas.collin12:03:50

Which would be disastrous especially with videos and multiple concurrent users

niklas.collin12:03:26

Plus would make it way slower to start the download for the client since server would have to first load the whole file. By returning a stream client will start to receive the file immediately

niklas.collin12:03:09

Now when implementing a true video media server then you'd have to take into account also jumping around in the video, this is more about video codecs and formats and how to deal with those

niklas.collin12:03:46

But basically the basic idea is the same there: you find the correct position in the video file, then create an InputStream to that file starting at that specific point in time and send that to the client

niklas.collin12:03:35

But it gets more complex since you cannot just start streaming from a random byte in a video file: in most codecs there is a concept called "keyframe" which is a point in time where you can rewind into and start streaming from there

niklas.collin12:03:15

Basically a time segment between two keyframes is a "full video file" so normal video file is actually a sequence of these

niklas.collin12:03:39

And this is the actual video part of the file, audio is a story on itself

niklas.collin12:03:06

But anyway, you get the point. This is a way too complex topic to cover in detail here but the very basics of it is very simple when it comes to your original question

niklas.collin12:03:53

With images however it is very simple, client always needs the whole image. Just send it in whole to the client as a stream

niklas.collin12:03:12

Remember to set proper Content-Type header so browser knows what kind of data it is it is receiving

niklas.collin12:03:53

All this has nothing to do with Aleph itself really although Aleph can be used as a server for this

niklas.collin12:03:35

Hope this helps

ahmed1hsn12:03:44

Thanks alot 🙂

kachayev13:03:35

@ Please, check this example I’ve implemented recently: static files server on top of Aleph https://github.com/kachayev/nasus

kachayev13:03:26

Other than that… the question is really about details: number of clients, size of files and frequency of changes, geo distribution, goals/plans, hardware/network in terms of bandwidth etc etc etc

kachayev13:03:31

If you need to implement backend for video- or sound streaming…. that’s an entirely different dimension and different problems

rahul08032712:03:12

Hello, I have a TarArchiveInputStream object and I want to make it available as a download over http. I tried the response on the handler like this

{:status  200
 :headers {"Content-Type"        "archive/tar"
           "Content-Disposition" "inline; filename=download.tar"}
 :body    (get-stream)}
I am able to trigger a download with the correct file names and content types, but the response is of 0 bytes. What am I doing wrong?

niklas.collin13:03:23

You need to provide also Content-Length header

niklas.collin13:03:09

Aleph cannot deduce that automatically from stream (in order to do that it would first have to read the whole stream which would make the whole stream pretty pointless)

niklas.collin13:03:29

Thus you need to retrieve that information from somewhere else,

niklas.collin13:03:37

If it is in filesystem, retrieve that data from filesystem

niklas.collin13:03:51

if it is in database, record the content-length to database when you store it as a blob

niklas.collin13:03:28

Also if your file is in the filesystem you can just pass a File object to aleph and it understands what you want to do

niklas.collin13:03:50

Aleph will get the file size from filesystem and will send the FIle as a stream to the client

niklas.collin13:03:09

However, if you are calculating the tar file on fly from something else then you need to solve your problem separately

kachayev13:03:56

@niklas.collin in fact… it’s better not to provide content-length header. In such a case the chance that it would work correctly is actually higher 🙂 If body is represented as an InputStream, Aleph should invoke send-streaming-body that automatically sets “Transfer-Encoding: chunked” header when content length is not provided manually https://github.com/ztellman/aleph/blob/626bec56dea84ddc5930f8bd9852fc1a236e23f7/src/aleph/http/core.clj#L260 (usually you’re not able to tell the size of the stream upfront anyways)

kachayev13:03:04

@rahul080327 Do you have any debug information about packets sent? (either from browser or wireshark or similar) So we can check raw request/responses?

rahul08032713:03:33

@kachayev So to give some context im building a clojure system over a lib i created. It exposes a path in a docker container as a TarArchiveInputStream. https://github.com/lispyclouds/clj-docker-client/blob/master/src/clj_docker_client/core.clj#L318 This is beacuse the Spotify docker lib im using does it. I want to expose this over HTTP so that a client can download the stuff from the container

rahul08032713:03:41

This is how i call my lib:

(defn get-stream-of
  "Fetches the TarStream of a path from a container."
  [path id]
  (docker/stream-path e/conn id path))

rahul08032713:03:34

This is my aleph handler:

(defn stream-artifact
  [name group number artifact]
  (d/let-flow [pipeline (u/name-of name group)
               id       (a/artifact-container-of pipeline number)
               path     (a/get-artifact-path-of pipeline artifact)
               stream   (a/get-stream-of path id)]
    (if (f/failed? stream)
      (res/bad-request {:message "No such artifact."})
      {:status  200
       :headers {"Content-Type"        "archive/tar"
                 "Content-Disposition" (format "inline; filename=%s.tar"
                                               artifact)}
       :body    stream})))

rahul08032713:03:17

im able to list all files in if i do it in the REPL. just that i cant send it over http

kachayev13:03:35

If you want the browser to start downloading, you need attachment not inline, right?

rahul08032713:03:45

tried that too

kachayev13:03:19

inline means that you want this to be shown on the page, which is impossible. Yeah, I known this is not the root of the problem, just noticed

rahul08032713:03:44

Just tried a curl, totally empty response

kachayev13:03:35

Could you please show the result of curl -vvv ...?

rahul08032713:03:14

$ curl -vvv 
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 7777 (#0)
> GET /api/pipeline/test/test/1/artifact/source HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: archive/tar
< Content-Disposition: attachement; filename=source.tar
< Server: Aleph/0.4.4
< Connection: Keep-Alive
< Date: Tue, 05 Mar 2019 13:40:48 GMT
< transfer-encoding: chunked
<
• Connection #0 to host localhost left intact

rahul08032713:03:20

Dunno if it may help, im using compojure-api for the routing

kachayev13:03:17

Any chance to upgrade at least to 0.4.6?

kachayev13:03:41

Let’s try, this version is ancient 😉

rahul08032713:03:04

it is at 0.4.6

kachayev13:03:16

< Server: Aleph/0.4.4

kachayev13:03:46

transitive deps?

kachayev13:03:03

you’re cURL response indicates another version

rahul08032713:03:03

not sure where to put it.

rahul08032713:03:28

yeah ive noticed it too. saw it with plain standalone aleph projects too

rahul08032713:03:01

but afaik im using 0.4.6

rahul08032713:03:58

ls ~/.m2/repository/aleph/aleph/
0.4.6

rahul08032713:03:35

im not sure if its an aleph problem or something else, have a lot of moving parts in my code. can try it without the compojure-api routing

kachayev13:03:23

I doubt it’s about routing

kachayev13:03:05

Nevertheless, I will try to reproduce locally with just a single tar stream to see how it works

rahul08032713:03:43

Thanks a lot for the help! 😄 you have all the repos there too. lemme know if you need more info

kachayev14:03:22

@rahul080327 another question, you said you’re using TarArchiveInputStream but the filename is *.tar… what do you want to archive? read file and return as a tar archive? or read an archive and return as unzipped file?

rahul08032714:03:01

I wanna return the file itself. a single download, as the path may contain multiple files

rahul08032714:03:31

the docker lib returns a tar stream of the path. i want to return an archive of that path over http

kachayev14:03:37

if they’re reading directory to create a tar… shouldn’t that return TarArchiveOutputStream?

rahul08032714:03:55

should i convert to an output stream? i was under the impression aleph needs an input stream to send stuff over http

kachayev14:03:05

no, but you don’t need to wrap it in TarArchiveInputStream either

kachayev14:03:02

You need TarArchiveInputStream *only* in case you want to get raw files (decompressed)

rahul08032714:03:38

ah okay. But will this cause an issue for Aleph? TarArchiveInputStream is an InputStream too, so i thought it would be okay

kachayev14:03:06

Tried with FileInputStream, - works fine

kachayev14:03:21

Try this, let me know if it works

rahul08032714:03:25

@kachayev But these are the constructors of FileInputStream FileInputStream(File file) FileInputStream(FileDescriptor fdObj) FileInputStream(String name) and i have a InputStream

kachayev14:03:49

I’ve just showed that as an example

kachayev14:03:04

You can set :body to your InputStream

rahul08032714:03:34

okay without the wrapping

kachayev14:03:45

Yep, exactly

rahul08032714:03:11

Ah yesss, it works!!

rahul08032714:03:20

Thanks a ton! 😄 😄

kachayev14:03:41

Sure! Let me know if you have more questions. Glad to help

rahul08032714:03:15

will do. great learnings and kudos for having aleph around!