Discussion:
Need advice with Mojo::IOLoop::Delay and segmentation faults
(too old to reply)
Scott
2014-12-09 21:08:39 UTC
Permalink
I maintain sites that use email validation services through a http api. I
use the Mojolicious delay non-blocking code. In the last day or so, I have
had my delay processes hanging resulting in 502 errors (hypnotoad reverse
proxy with nginx) and at time the http service returning numerous blank
records. I blamed them at first, but when I switched to non-blocking code,
it worked fine. I boiled down my code to a command line program:

Mojo::IOLoop::Delay->new->steps(
sub {
my $delay = shift;
$ua->get( $bv . $_ => $delay->begin ) for @emails;
},
sub {
my $delay = shift;
for my $tx ( @_ ) {
my $res = $tx->res->json;
say Dumper $res;
}
}
)->wait;

At times it works great, and at other times I get errors:

Segmentation fault (core dumped)

and

*** Error in `perl': double free or corruption (out): 0x00007f9c40000910 ***
Aborted (core dumped)


Some of you may know exactly what the problem is, but I can only guess. Is
this something to do with Perl itself? For a workstation and server I use
Ubuntu 14.04 and it is happening on both. I update as updates happen. I
noticed in the apt log that libc6, libc6-dev updated a few days back. Since
I use perlbrew, do I need to recompile my perl after certain packages are
updated? Currentlty, I am using the most up-to-date Mojolicious. I read the
changes before updating and test out on my workstation first, but I have
not had a Perl problem before and I am not sure I am managing things
correctly.

Thanks for any advice,

Scott K.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-09 21:55:28 UTC
Permalink
Is this something to do with Perl itself?
That would be my first guess, but it could also be a bug in an XS module.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-09 23:14:28 UTC
Permalink
Thanks Sebastian for the quick response.

I have recompiled perl with perlbrew (5.20.0 > 5.20.1) on a fully
up-to-date Kubuntu 14.04 and nothing has changed, same errors as before.
Sam on the server which is perl 8.18.2. The entire script is:

#!/usr/bin/env perl

use Modern::Perl '2013';
use Mojo::UserAgent;
use Data::Dumper;
use Mojo::IOLoop::Delay;

my $bv = '...';
my $ua = Mojo::UserAgent->new;
my @emails = ( ... );

for my $email ( @emails ) {
my $tx = $ua->get( $bv . $email );
my $res = $tx->res->json;
say Dumper $res;
}

Mojo::IOLoop::Delay->new->steps(
sub {
my $delay = shift;
$ua->get( $bv . $_ => $delay->begin ) for @emails;
},
sub {
my $delay = shift;
for my $tx ( @_ ) {
my $res = $tx->res->json;
say Dumper $res;
}
}
)->wait;

I do have EV installed.

I have several sites using the non-blocking ua code and until a few days
ago it was fine. I use the mojo ua to send emails through another http api
and still use mango (only until end of January and with a freeze on the
MongoDB updates) on one site in non-blocking fashion with no problems.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Efros
2014-12-09 23:54:24 UTC
Permalink
Hi!

Please add error-checking code and see is it change anything.
Also, I recommend you do add something like "warn 1;", "warn 2;", etc.
between most lines to find out on which line perl crash.
Post by Scott
my $tx = $ua->get( $bv . $email );
warn($tx->error->{message}), next if $tx->error;
Post by Scott
my $res = $tx->res->json;
say Dumper $res;
}
Mojo::IOLoop::Delay->new->steps(
sub {
my $delay = shift;
},
sub {
my $delay = shift;
warn($tx->error->{message}), next if $tx->error;
Post by Scott
my $res = $tx->res->json;
say Dumper $res;
}
}
)->wait;
I do have EV installed.
Cool, but then please add 'use EV;' at beginning of your script, just to
make sure it works.
--
WBR, Alex.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-10 16:33:59 UTC
Permalink
I have tracked down my issue to Net::DNS::Native. If removed, then the
non-blocking script works. I played around with installing older
Mojolicious versions and the script started intermittently segment faulting
at version 5.59 with Net::DNS::Native installed. Per the docs, the module
needs threads and will not install if the environment cannot support the
module, but it installs without message with cpanm and pass tests when
installed manually on up-to-date Ubuntu 14.04. I have removed it from my
workstations and server and all is OK.

Scott K.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 16:40:09 UTC
Permalink
Post by Scott
I have tracked down my issue to Net::DNS::Native. If removed, then the
non-blocking script works.
Please open an issue for Net::DNS::Native, this needs to be tracked down.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Efros
2014-12-10 16:49:20 UTC
Permalink
Hi!
Post by sri
Post by Scott
I have tracked down my issue to Net::DNS::Native. If removed, then the
non-blocking script works.
Please open an issue for Net::DNS::Native, this needs to be tracked down.
To be honest, when I noticed this module in Mojo changelog it sounds like
very bad idea… perl, XS, threads… really? May I ask you why do you choose
it over, for example, pure-perl, fast and reliable AnyEvent::DNS?
Mojo is usually used with EV (at least by everyone who bothers about async
DNS), so AnyEvent will work just fine.
--
WBR, Alex.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-10 17:35:47 UTC
Permalink
I have opened up an issue on rt.cpan.org. Very much a novice at it, hope it
is done correctly.

Scott K.
Post by Scott
I have tracked down my issue to Net::DNS::Native. If removed, then the
Post by Scott
non-blocking script works.
Please open an issue for Net::DNS::Native, this needs to be tracked down.
--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-10 16:40:06 UTC
Permalink
Should also mention that my perl (5.18.2, 5.20.0) is not compiled with
threads. Might of been assumed, but should of mentioned.

Scott K.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 16:43:52 UTC
Permalink
Post by Scott
Should also mention that my perl (5.18.2, 5.20.0) is not compiled with
threads. Might of been assumed, but should of mentioned.
Just to make sure, the Net::DNS::Native version is 0.13, right?

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-10 17:02:39 UTC
Permalink
Yes, Net::DNS::Native 0.13.
Post by Scott
Should also mention that my perl (5.18.2, 5.20.0) is not compiled with
Post by Scott
threads. Might of been assumed, but should of mentioned.
Just to make sure, the Net::DNS::Native version is 0.13, right?
--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 17:23:11 UTC
Permalink
Post by Scott
Yes, Net::DNS::Native 0.13.
Too bad, i would hate having to remove support for non-blocking name
resolution. Please keep us updated on the issue.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 17:38:05 UTC
Permalink
Post by sri
Please keep us updated on the issue.
In the meantime, it would be really nice if you could try again without
extra threads. (by applying this patch)

https://gist.github.com/anonymous/5a8b49b2749d77cd716b

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Scott
2014-12-10 18:43:17 UTC
Permalink
Sorry, did not seem to help.
Post by sri
Please keep us updated on the issue.
In the meantime, it would be really nice if you could try again without
extra threads. (by applying this patch)
https://gist.github.com/anonymous/5a8b49b2749d77cd716b
--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Oleg
2014-12-10 18:52:03 UTC
Permalink
Can you connect to #mojo channel at irc.perl.org and ping me to resolve
this faster?
Post by Scott
Sorry, did not seem to help.
Post by sri
Please keep us updated on the issue.
In the meantime, it would be really nice if you could try again without
extra threads. (by applying this patch)
https://gist.github.com/anonymous/5a8b49b2749d77cd716b
--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 20:13:17 UTC
Permalink
Post by Scott
Sorry, did not seem to help.
Could you please help Oleg with tracking this down?

https://rt.cpan.org/Public/Bug/Display.html?id=100834#txn-1441004

It's a little unfortunate that every Net::DNS::Native bug has such severe
consequences.
If this can't get resolved within the next few days, i think we only have
two options.

a) We remove support for non-blocking name resolution, until
Net::DNS::Native had more time to iron out the kinks.
b) We add proper unit tests for non-blocking name resolution, and make
MOJO_NO_NDN default to 1. (This would depend on volunteers stepping up to
design portable tests)

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-10 20:15:55 UTC
Permalink
Post by sri
a) We remove support for non-blocking name resolution, until
Net::DNS::Native had more time to iron out the kinks.
I guess we could also add a public API for pluggable resolvers, but this
would once again depend on volunteers coming up with a nice design.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-11 14:51:24 UTC
Permalink
Oleg has found the problem, and i'm afraid it doesn't look good.

https://github.com/kraih/mojo/issues/716

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-11 16:26:34 UTC
Permalink
Post by sri
https://github.com/kraih/mojo/issues/716
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fkraih%2Fmojo%2Fissues%2F716&sa=D&sntz=1&usg=AFQjCNG4EEikwVaLMTtv1vfnmOJLGaKn9g>
I'm afraid this has now resulted in the removal of non-blocking name
resolution support. :(

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-11 19:54:08 UTC
Permalink
Post by sri
I'm afraid this has now resulted in the removal of non-blocking name
resolution support. :(
We do have a prototype for a pluggable resolver API though, but currently
no volunteers to clean it up.

https://github.com/kraih/mojo/compare/pluggable_resolver

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Efros
2014-12-12 01:01:27 UTC
Permalink
Hi!
Post by sri
We do have a prototype for a pluggable resolver API though, but currently
no volunteers to clean it up.
https://github.com/kraih/mojo/compare/pluggable_resolver
It makes sense to implement second plugin using AnyEvent::DNS to see how
it fit into resolver API.

Also, I think it probably doesn't make sense to use resolver in case user
already provided IP (v4/v6) in $address.

And I'm not sure about special handling of 'localhost':
- IP for localhost is usually defined in /etc/hosts, and it might be IPv6
one or be like 127.0.0.2
- in my experience it's very annoying when library see 'localhost' and
make some "smart" decision based on this - like when DBD::MySQL (or
libmysqlclient.so, not sure) switch to using UNIX sockets instead of TCP
- Net::DNS::Native is better than AnyEvent::DNS because it also support
host names defined in /etc/hosts, append search domain defined in
/etc/resolv.conf, etc. - but we can partially compensate for this by
switching to using (blocking) system resolver for host names which
doesn't contain dot/domain… this will also solve issue with non-standard
address for 'localhost', and it won't be actually blocking in most
cases, but it probably should be implemented in plugin for
AnyEvent::DNS, not in resolver API
--
WBR, Alex.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-12 01:10:11 UTC
Permalink
Post by sri
We do have a prototype for a pluggable resolver API though, but currently
no volunteers to clean it up.
https://github.com/kraih/mojo/compare/pluggable_resolver
Sadly i've made a mistake, and this version is off the table again, since
it only allows to change the resolver globally. And we need to be able to
allow event loop specific resolvers like UV::getaddrinfo or
IO::Async::Resolver.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2014-12-13 02:13:39 UTC
Permalink
We are going to try again with Net::DNS::Native 0.14, if you encounter any
new segfaults with Mojolicious 5.69 that disappear with MOJO_NO_NDN=1,
please let us know.


https://github.com/kraih/mojo/commit/ce0b9082ac0bad4a907d938f47e65813cc053623

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Loading...