Discussion:
large file uploads part by part?
(too old to reply)
M. Aaron Bossert
2014-05-08 03:27:23 UTC
Permalink
All,

I have a working app with the following route:

post '/raw_upload' => sub {
my $self = shift;
my $file = $self->req->upload('upload');
if($file) {

$file->move_to($dataDir.$self->session('user').'/raw/'.$file->filename);
$self->rendered(200);
}
else {
$self->rendered(500);
}


};

Where I am falling flat is with larger files. I am not sure exactly what
the threshold is, but certainly, a 100MB file is fine (I upped the max
message size to 2GB), but a 900MB file does not show up. It just silently
fails with no visible signs of errors. I checked for an inactivity
timeout...which was not the issue (anymore)...and there are no error
messages anywhere I can see (client or server side). Any ideas as to what
is happening? With smaller files, the file just shows up in the
appropriate directory...with a large file, it "acts" as if it is
transferring...but nothing ever shows up.

The only thing I could even think of was that the Asset is perhaps
switching from a memory to a file one? Even so, I am not sure why that
would matter. My server is pretty beefy (512GB of RAM, 24Cores) and I
would not expect a 1GB file to cause so much as a hiccup on the memory side
of the house...but what do I know?

Any suggestions would be appreciated...and if appropriate, would it be
better to copy/move the file as it comes in, rather than as a whole? Also,
as a side note...I ended up setting the inactivity timeout as an
environment variable (globally)...which I don't really want to do...but
when I use: Mojo::IOLoop->stream($self->tx->connection)->timeout(6000)
within my route (right after "my $self=shift;")...it does not have any
effect..any thoughts on that would be appreciated as well...
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Tobias Oetiker
2014-05-08 06:03:19 UTC
Permalink
Hi Aron,
Post by M. Aaron Bossert
All,
post '/raw_upload' => sub {
my $self = shift;
my $file = $self->req->upload('upload');
if($file) {
$file->move_to($dataDir.$self->session('user').'/raw/'.$file->filename);
$self->rendered(200);
}
else {
$self->rendered(500);
}
};
Where I am falling flat is with larger files. I am not sure exactly what
the threshold is, but certainly, a 100MB file is fine (I upped the max
message size to 2GB), but a 900MB file does not show up. It just silently
fails with no visible signs of errors. I checked for an inactivity
timeout...which was not the issue (anymore)...and there are no error
messages anywhere I can see (client or server side). Any ideas as to what
is happening? With smaller files, the file just shows up in the
appropriate directory...with a large file, it "acts" as if it is
transferring...but nothing ever shows up.
The only thing I could even think of was that the Asset is perhaps
switching from a memory to a file one? Even so, I am not sure why that
would matter. My server is pretty beefy (512GB of RAM, 24Cores) and I
would not expect a 1GB file to cause so much as a hiccup on the memory side
of the house...but what do I know?
Any suggestions would be appreciated...and if appropriate, would it be
better to copy/move the file as it comes in, rather than as a whole? Also,
as a side note...I ended up setting the inactivity timeout as an
environment variable (globally)...which I don't really want to do...but
when I use: Mojo::IOLoop->stream($self->tx->connection)->timeout(6000)
within my route (right after "my $self=shift;")...it does not have any
effect..any thoughts on that would be appreciated as well...
I have only been fighting with this ... it all seemed rather
difficult to reproduce probably also connectd to some browser
issues ...

I ended up using http://github.com/23/resumable.js

here is the program I wrote around it:

https://github.com/oetiker/upload-service

cheers
tobi
--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch ***@oetiker.ch +41 62 775 9902
sri
2014-05-08 12:39:40 UTC
Permalink
I had no problem uploading a 2GB file with

MOJO_MAX_MESSAGE_SIZE=3221225472 and the example app from the documentation.


http://mojolicio.us/perldoc/Mojolicious/Lite#File_uploads

--

sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-05-08 12:42:42 UTC
Permalink
I had no problem uploading a 2GB file...
Tested with OSX 10.9.2, Mojolicious 4.97, Perl 5.18.2.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-05-08 15:17:00 UTC
Permalink
Post by sri
Tested with OSX 10.9.2, Mojolicious 4.97, Perl 5.18.2.
And Chrome 34 on the client side.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
M. Aaron Bossert
2014-05-08 21:24:18 UTC
Permalink
Really weird. I see the file uploading in the /tmp directory on my server
and then it just stops. I can't discern if there is a size issue because
the upload seems to stop at different sizes. I don't believe that it is a
timeout as that would show up in my logs (as it did before I upped it). I
am really stumped. I don't think this is a showstopper just yet...but I
really need to ensure that I can reliably upload files, regardless of size.

Consequently, I know it might be out of scope for "just" mojolicious as the
code would be done mostly client-side...but has anyone messed around with
paralellized uploads for larger stuff? As in, break a file up into chunks
if bigger than N bytes and send in chunks at the same time (configurable
based on available bandwidth, perhaps)?
Post by sri
Tested with OSX 10.9.2, Mojolicious 4.97, Perl 5.18.2.
And Chrome 34 on the client side.
--
sebastian
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2014-05-08 21:28:36 UTC
Permalink
You asked for any input... Are you running Mojo behind a proxy like nginx
as is so common? You'll want to ensure that the proxy's file upload limit
matches what you intend for Mojo.
Post by M. Aaron Bossert
Really weird. I see the file uploading in the /tmp directory on my
server and then it just stops. I can't discern if there is a size issue
because the upload seems to stop at different sizes. I don't believe that
it is a timeout as that would show up in my logs (as it did before I upped
it). I am really stumped. I don't think this is a showstopper just
yet...but I really need to ensure that I can reliably upload files,
regardless of size.
Post by M. Aaron Bossert
Consequently, I know it might be out of scope for "just" mojolicious as
the code would be done mostly client-side...but has anyone messed around
with paralellized uploads for larger stuff? As in, break a file up into
chunks if bigger than N bytes and send in chunks at the same time
(configurable based on available bandwidth, perhaps)?
Post by M. Aaron Bossert
Post by sri
Post by sri
Tested with OSX 10.9.2, Mojolicious 4.97, Perl 5.18.2.
And Chrome 34 on the client side.
--
sebastian
--
You received this message because you are subscribed to the Google
Groups "Mojolicious" group.
Post by M. Aaron Bossert
Post by sri
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
Post by M. Aaron Bossert
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
M. Aaron Bossert
2014-05-08 23:41:45 UTC
Permalink
Nope...running with morbo and then hypnotoad when I get it to production...

Aaron
You asked for any input... Are you running Mojo behind a proxy like nginx as is so common? You'll want to ensure that the proxy's file upload limit matches what you intend for Mojo.
Really weird. I see the file uploading in the /tmp directory on my server and then it just stops. I can't discern if there is a size issue because the upload seems to stop at different sizes. I don't believe that it is a timeout as that would show up in my logs (as it did before I upped it). I am really stumped. I don't think this is a showstopper just yet...but I really need to ensure that I can reliably upload files, regardless of size.
Consequently, I know it might be out of scope for "just" mojolicious as the code would be done mostly client-side...but has anyone messed around with paralellized uploads for larger stuff? As in, break a file up into chunks if bigger than N bytes and send in chunks at the same time (configurable based on available bandwidth, perhaps)?
Post by sri
Post by sri
Tested with OSX 10.9.2, Mojolicious 4.97, Perl 5.18.2.
And Chrome 34 on the client side.
--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...