Discussion:
Mojo::UserAgent and large files
(too old to reply)
a***@gmail.com
2018-07-30 16:16:02 UTC
Permalink
Hello,

On a "back-end" server, I have an application responsible to manage large
files (some Gigabytes).
On a "front-end" server, I want to allow the end-users to download files
stored on the back-end, but without making the back-end server directly
accessible from all Internet.
On the front-end server.

On the front-end, I tried the following piece of code :

use Mojolicious::Lite;
use Mojo::UserAgent;


get '/dl' => sub {
my $c = shift;
my $ua = Mojo::UserAgent->new;
my $tx = $ua->build_tx(GET => 'https://BACKEND/my_file.zip');
$tx->res->content->unsubscribe('read')->on(read => sub {
my ($content, $bytes) = @_;
$c->write($bytes);
});
$tx = $ua->start($tx);
};

app->start;

The problem is that the whole file seems to be downloaded and stored in
memory, before being sent. I would like to be able to write a chunk of data
just after receiving it from the back-end server, without waiting the end
of the whole file download. I thought that redefining the "read" event
would do the job but something is probably missing. Does anyone have any
suggestions?

Alex
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Bernier
2018-08-03 13:28:11 UTC
Permalink
Hello,

If soemone has some hints to help me to solve the problem described below, it woule be very apreciated ;)

Alex
Post by a***@gmail.com
Hello,
On a "back-end" server, I have an application responsible to manage
large files (some Gigabytes).
On a "front-end" server, I want to allow the end-users to download
files stored on the back-end, but without making the back-end server
directly accessible from all Internet.
On the front-end server.
use Mojolicious::Lite;
use Mojo::UserAgent;
get '/dl' => sub {
my $c = shift;
my $ua = Mojo::UserAgent->new;
my $tx = $ua->build_tx(GET => 'https://BACKEND/my_file.zip');
$tx->res->content->unsubscribe('read')->on(read => sub {
$c->write($bytes);
});
$tx = $ua->start($tx);
};
app->start;
The problem is that the whole file seems to be downloaded and stored in
memory, before being sent. I would like to be able to write a chunk of
data just after receiving it from the back-end server, without waiting
the end of the whole file download. I thought that redefining the
"read" event would do the job but something is probably missing. Does
anyone have any suggestions?
Alex
--
You received this message because you are subscribed to the Google
Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at [3]https://groups.google.com/group/mojolicious.
For more options, visit [4]https://groups.google.com/d/optout.
Références
3. https://groups.google.com/group/mojolicious
4. https://groups.google.com/d/optout
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-06 16:43:39 UTC
Permalink
Post by a***@gmail.com
On a "back-end" server, I have an application responsible to manage large
files (some Gigabytes).
On a "front-end" server, I want to allow the end-users to download files
stored on the back-end, but without making the back-end server directly
accessible from all Internet.
On the front-end server.
use Mojolicious::Lite;
use Mojo::UserAgent;
get '/dl' => sub {
my $c = shift;
my $ua = Mojo::UserAgent->new;
my $tx = $ua->build_tx(GET => 'https://BACKEND/my_file.zip');
$tx->res->content->unsubscribe('read')->on(read => sub {
$c->write($bytes);
});
$tx = $ua->start($tx);
};
app->start;
Yea, something like that should work. We've been meaning to add a more user
friendly API, but never got around to it. Right now i'm afraid it's quite
tricky to get
right, but possible.
Post by a***@gmail.com
The problem is that the whole file seems to be downloaded and stored in
memory, before being sent. I would like to be able to write a chunk of data
just after receiving it from the back-end server, without waiting the end
of the whole file download. I thought that redefining the "read" event
would do the job but something is probably missing.
That doesn't sound right. If you unsubscribed from the read event there is
nothing
left to store the message content anywhere.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-06 16:45:44 UTC
Permalink
Post by a***@gmail.com
$tx = $ua->start($tx);
Oh right, this is a blocking call, so you're just spinning in-place
appending chunks to the
write buffer but never actually letting the event loop write to the socket.
You need to do it
async.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
a***@gmail.com
2018-08-07 09:10:50 UTC
Permalink
Post by a***@gmail.com
$tx = $ua->start($tx);
Oh right, this is a blocking call, so you're just spinning in-place
appending chunks to the
write buffer but never actually letting the event loop write to the
socket. You need to do it
async.
I tried to use "start_p" or "get_p" but I get a "Premature connection
close" error. I don't understand why. Here is the code of my sample
Mojolicious controller :

package Foo2::Controller::Example;
use Mojo::Base 'Mojolicious::Controller';
use Mojo::UserAgent;
use Mojo::Promise;


sub welcome {
my $c = shift;
my $ua = Mojo::UserAgent->new;
$ua->get_p("https://www.mojolicious.org")->then(sub {
my $tx2 = shift;
say $tx2->result->body;
})->catch(sub {
my $err = shift;
warn "Connection error: $err";
})->wait;
$c->render_later;
}

When I run the same "get_p" code in a command-line script (without
Mojlicious), I do not have this erro rmessage.

Alex
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Bernier
2018-08-07 08:57:33 UTC
Permalink
Post by a***@gmail.com
$tx = $ua->start($tx);
Oh right, this is a blocking call, so you're just spinning in-place
appending chunks to the
write buffer but never actually letting the event loop write to the
socket. You need to do it
async.
I tried to use "start_p" or "get_p" but I get a "Premature connection close" error. I don't understand why. Here is the code of my sample Mojolicious controller :

package Foo2::Controller::Example;
use Mojo::Base 'Mojolicious::Controller';
use Mojo::UserAgent;
use Mojo::Promise;


sub welcome {
my $c = shift;
my $ua = Mojo::UserAgent->new;
$ua->get_p("https://www.mojolicious.org")->then(sub {
my $tx2 = shift;
say $tx2->result->body;
})->catch(sub {
my $err = shift;
warn "Connection error: $err";
})->wait;
$c->render_later;
}

Whzn I run the same "get_p" code in a script I run in command-line (without Mojlicious), I do not have the erro rmessage.

Alex
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-07 12:20:47 UTC
Permalink
Google is starting to suspect you are spamming the list and flags all your
messages. That forces us moderators to manually approve everything
you write. So please try to avoid making too many typos and posting
similar messages multiple times.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Heiko Jansen
2018-08-17 17:22:50 UTC
Permalink
Hmm, there's no access to the controller ($c) and no reference to it kept
in the fullfillment or rejection callbacks.
I'd guess it is therefore garbage collected once you leave "welcome" and
the client connection thus closed immediately.

hth,
heiko
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-08-17 17:39:35 UTC
Permalink
You don't need to keep a reference to the controller but to the
transaction. See
https://mojolicious.org/perldoc/Mojolicious/Guides/FAQ#What-does-Transaction-already-destroyed-mean
Post by Heiko Jansen
Hmm, there's no access to the controller ($c) and no reference to it kept
in the fullfillment or rejection callbacks.
I'd guess it is therefore garbage collected once you leave "welcome" and
the client connection thus closed immediately.
hth,
heiko
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-08-17 17:41:11 UTC
Permalink
Though the other problem in this case is that you are not keeping the
Mojo::UserAgent object alive because it's scoped inside the action:
https://mojolicious.org/perldoc/Mojolicious/Guides/FAQ#What-does-Premature-connection-close-mean
Post by Dan Book
You don't need to keep a reference to the controller but to the
transaction. See
https://mojolicious.org/perldoc/Mojolicious/Guides/FAQ#What-does-Transaction-already-destroyed-mean
Post by Heiko Jansen
Hmm, there's no access to the controller ($c) and no reference to it kept
in the fullfillment or rejection callbacks.
I'd guess it is therefore garbage collected once you leave "welcome" and
the client connection thus closed immediately.
hth,
heiko
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...