[Mojolicious] How to get the result of a finished Minion job back to the app

[Per Carlson]

Hi.

I'm thinking of using Minion as a job queue in an existing Mojolicious
application. The concept of handing off a job to Minion is straight
forward, but how to get the result back?

Here's a small (not working) example application:

#!/usr/bin/env perl

use Mojolicious::Lite;
use Net::Ping;

plugin Minion => { File => "minion.db" };

app->minion->add_task(ping => sub {
my ($job, $host) = @_;

my $p = Net::Ping->new;
$p->port_number(22);

if ($p->ping($host)) {
$job->finish("Host $host is alive");
} else {
$job->fail("Host $host doesn't respond");
}
return;
});

get '/ping/(:host)' => sub {
my $c = shift;

my $jobid = $c->minion->enqueue(
ping => [ $c->param('host') ]
);
$c->render_later;

# Register event handlers to get the result/error
# from Minion
my $job = $c->minion->job($jobid);

$job->on(error => sub {
my ($job, $err) = @_;
$c->render(status => 500, text => $err);
});

$job->on(finish => sub {
my ($job, $result) = @_;
$c->render(text => $result);
});
};

app->start;

​The above doesn't work at all. I suspect that's because the worker is
running in another process than the app, and doesn't share the same event
scopes.

Is there another (working) way to retrieve the result in the same route​?
--
Pelle

Research is what I'm doing when I don't know what I'm doing.
- Wernher von Braun

[Glen]

The main intent of minion is to execute long running, blocking tasks that
would take longer than the life cycle of a request, so return values aren't
a feature.

Think of it as a fire and forget in terms of the initiating request, and
you can check the status of it in subsequent requests.

Post by Per Carlson
Hi.
I'm thinking of using Minion as a job queue in an existing Mojolicious
application. The concept of handing off a job to Minion is straight
forward, but how to get the result back?
#!/usr/bin/env perl
use Mojolicious::Lite;
use Net::Ping;
plugin Minion => { File => "minion.db" };
app->minion->add_task(ping => sub {
my $p = Net::Ping->new;
$p->port_number(22);
if ($p->ping($host)) {
$job->finish("Host $host is alive");
} else {
$job->fail("Host $host doesn't respond");
}
return;
});
get '/ping/(:host)' => sub {
my $c = shift;
my $jobid = $c->minion->enqueue(
ping => [ $c->param('host') ]
);
$c->render_later;
# Register event handlers to get the result/error
# from Minion
my $job = $c->minion->job($jobid);
$job->on(error => sub {
$c->render(status => 500, text => $err);
});
$job->on(finish => sub {
$c->render(text => $result);
});
};
app->start;
​The above doesn't work at all. I suspect that's because the worker is
running in another process than the app, and doesn't share the same event
scopes.
Is there another (working) way to retrieve the result in the same route​?
--
Pelle
Research is what I'm doing when I don't know what I'm doing.
- Wernher von Braun
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[sri]

Post by Glen
The main intent of minion is to execute long running, blocking tasks that
would take longer than the life cycle of a request, so return values
aren't a feature.

They actually have become a (minor) feature recently, since it was
convenient to turn the old error field into a generic result field. But we
do indeed not support async notifications for job state transitions, for
the reason you mentioned.

--
sebastian

[Per Carlson]

Hi Glen and Sebastian.

Thanks for clarifying this in text and POD.

I'm more or less exploring the possibilities of Minion. If it were possible
to get async notifications back, this would be a magnificent way to bridge
the sync and async worlds!

For most of my intended uses of Minion, it's perfectly fine to enqueue a
job, keep track of the jobid and return from the route. But there are also
some shorter blocking operations that I was curious if it could handle.

Post by Glen
The main intent of minion is to execute long running, blocking tasks that

Post by Glen
would take longer than the life cycle of a request, so return values
aren't a feature.

They actually have become a (minor) feature recently, since it was
convenient to turn the old error field into a generic result field. But we
do indeed not support async notifications for job state transitions, for
the reason you mentioned.
--
sebastian
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

--
Pelle

Research is what I'm doing when I don't know what I'm doing.
- Wernher von Braun

[sri]

Post by Per Carlson
I'm more or less exploring the possibilities of Minion. If it were
possible to get async notifications back, this would be a magnificent way
to bridge the sync and async worlds!

It used to be supported in the beginning, but didn't fit in very well.


https://github.com/kraih/minion/commit/2399f7b94be5a2e611b17df2c6b8e3b5eba040cb#diff-b30d9d6fe78ad2d38daed2446ba66d92L321

For most backends it would have to be done with a recurring timer trying to
pull the result in regular intervals anyway. Which you can of course
implement yourself as a helper in just a few lines too.

--
sebastian

[John]

Post by sri
For most backends it would have to be done with a recurring timer
trying to pull the result in regular intervals anyway. Which you can
of course implement yourself as a helper in just a few lines too.

Maybe I did this in an especially ugly way. With Websockets I want real
time updates to client:

https://github.com/john-/rtl_fm-scanner/blob/audio_manager/bin/rtl_fm-scanner#L180

Ideally it would be nice for the route to receive a message when
something interesting happens (determined by logic associated with the
long running process).

John

[sri]

Post by sri
For most backends it would have to be done with a recurring timer trying
to pull the result in regular intervals anyway.

This is actually the case for both backends included with Minion, DBM::Deep
has no notification mechanism, and PostgreSQL is not good enough at high
concurrency yet. You'd need a specialized backend for push notifications,
like Redis.

--
sebastian

[John]

Post by Per Carlson
I'm more or less exploring the possibilities of Minion. If it were
possible to get async notifications back, this would be a
magnificent way to bridge the sync and async worlds!
It used to be supported in the beginning, but didn't fit in very well.
https://github.com/kraih/minion/commit/2399f7b94be5a2e611b17df2c6b8e3b5eba040cb#diff-b30d9d6fe78ad2d38daed2446ba66d92L321
For most backends it would have to be done with a recurring timer
trying to pull the result in regular intervals anyway. Which you can
of course implement yourself as a helper in just a few lines too.

The first two mojolicious applications I have written now have the same
pattern: Web Socket client that should get instant notification when
something managed by that same application generates events of interest
to the client.

It would seem some kind of publish/subscribe type thing would be used
here. Can a "third party" module be grafted in or is there some
fundamental reason why these kinds of applications aren't cleanly
implemented?

John

[sri]

Post by John
It would seem some kind of publish/subscribe type thing would be used
here. Can a "third party" module be grafted in or is there some
fundamental reason why these kinds of applications aren't cleanly
implemented?

Of course, in fact i'm considering the addition of a much easier to use
pub/sub system to Mojo::Pg for example.

https://gist.github.com/anonymous/9d6b6d9f049a8ce12931

--
sebastian

[sri]

Post by sri
https://gist.github.com/anonymous/9d6b6d9f049a8ce12931

There's a pubsub branch for Mojo::Pg too now.

https://github.com/kraih/mojo-pg/compare/pubsub

I wouldn't hold my breath though, as long as reconnects are this fragile
there is not much of a chance for it to get merged.

--
sebastian

[sri]

Post by Per Carlson
​The above doesn't work at all. I suspect that's because the worker is
running in another process than the app, and doesn't share the same event
scopes.

I've updated the event descriptions to make the behavior more obvious.


https://github.com/kraih/minion/commit/3f10630d154b7b7c914719900c9758d4591a36fc

--
sebastian