Discussion:
Mojo::Log and UTF-8
(too old to reply)
Alex Povolotsky
2018-10-06 10:35:48 UTC
Permalink
Hello

How do I send UTF-8 constants to Mojo::Log? It forcefully encodes
everything so UTF-8 goes with double encoding and unreadable. Trying to
decode UTF-8 cat

$log->info(decode('UTF-8', 'тест лПга'));

results in

Use of uninitialized value $_[0] in join or string at
/usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.

Looks like something is wrong...
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Povolotsky
2018-10-06 14:15:05 UTC
Permalink
Quite interesting. The simple script

=== cut mojolog ===

#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "ПрПстП прОМт\n";
$log->info('тест лПга');
=== cut mojolog ===

yields fine results on all my servers but one.

On it, I'm getting

% perl mojolog
ПрПстП прОМт
[Sat Oct 6 15:22:43 2018] [info] ÑᅵеÑᅵÑᅵ лПга

The second line is clearly wrong and seems to be badly encoded. Same
version of OS, Perl, Mojolicious, same set of environment variables. What
else could influence unicode processing?

суббПта, 6 Пктября 2018 г., 13:35:48 UTC+3 пПльзПватель Alex Povolotsky
Post by Alex Povolotsky
Hello
How do I send UTF-8 constants to Mojo::Log? It forcefully encodes
everything so UTF-8 goes with double encoding and unreadable. Trying to
decode UTF-8 cat
$log->info(decode('UTF-8', 'тест лПга'));
results in
Use of uninitialized value $_[0] in join or string at
/usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.
Looks like something is wrong...
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-10-06 17:29:51 UTC
Permalink
The script is not quite correct. I suspect if you add "use warnings" you
will see a "Wide character in print" warning, because you are not encoding
your unicode characters to UTF-8 before printing them. However, on the
server where you are seeing the double encoding, perhaps you have a
PERL5OPT environment variable which is setting something like "-CSAD" and
thus encoding all output handles by default (this should not be globally
set, because it can mess with modules that don't expect it like this).

-Dan
Post by Alex Povolotsky
Quite interesting. The simple script
=== cut mojolog ===
#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "ПрПстП прОМт\n";
$log->info('тест лПга');
=== cut mojolog ===
yields fine results on all my servers but one.
On it, I'm getting
% perl mojolog
ПрПстП прОМт
[Sat Oct 6 15:22:43 2018] [info] ÑᅵеÑᅵÑᅵ лПга
The second line is clearly wrong and seems to be badly encoded. Same
version of OS, Perl, Mojolicious, same set of environment variables. What
else could influence unicode processing?
суббПта, 6 Пктября 2018 г., 13:35:48 UTC+3 пПльзПватель Alex Povolotsky
Post by Alex Povolotsky
Hello
How do I send UTF-8 constants to Mojo::Log? It forcefully encodes
everything so UTF-8 goes with double encoding and unreadable. Trying to
decode UTF-8 cat
$log->info(decode('UTF-8', 'тест лПга'));
results in
Use of uninitialized value $_[0] in join or string at
/usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.
Looks like something is wrong...
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Alex Povolotsky
2018-10-10 15:53:27 UTC
Permalink
So I should not binmode STDERR, ':utf8' if I'm using Mojo::Log and always
explicitly encode() data?
Post by Dan Book
The script is not quite correct. I suspect if you add "use warnings" you
will see a "Wide character in print" warning, because you are not encoding
your unicode characters to UTF-8 before printing them. However, on the
server where you are seeing the double encoding, perhaps you have a
PERL5OPT environment variable which is setting something like "-CSAD" and
thus encoding all output handles by default (this should not be globally
set, because it can mess with modules that don't expect it like this).
-Dan
Post by Alex Povolotsky
Quite interesting. The simple script
=== cut mojolog ===
#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "ПрПстП прОМт\n";
$log->info('тест лПга');
=== cut mojolog ===
yields fine results on all my servers but one.
On it, I'm getting
% perl mojolog
ПрПстП прОМт
[Sat Oct 6 15:22:43 2018] [info] ÑᅵеÑᅵÑᅵ лПга
The second line is clearly wrong and seems to be badly encoded. Same
version of OS, Perl, Mojolicious, same set of environment variables. What
else could influence unicode processing?
суббПта, 6 Пктября 2018 г., 13:35:48 UTC+3 пПльзПватель Alex Povolotsky
Post by Alex Povolotsky
Hello
How do I send UTF-8 constants to Mojo::Log? It forcefully encodes
everything so UTF-8 goes with double encoding and unreadable. Trying to
decode UTF-8 cat
$log->info(decode('UTF-8', 'тест лПга'));
results in
Use of uninitialized value $_[0] in join or string at
/usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.
Looks like something is wrong...
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-10-10 16:07:50 UTC
Permalink
Mojo::Log already encodes data by default. Anything you send to stderr
outside of that should be encoded if necessary.

-Dan
Post by Alex Povolotsky
So I should not binmode STDERR, ':utf8' if I'm using Mojo::Log and always
explicitly encode() data?
Post by Dan Book
The script is not quite correct. I suspect if you add "use warnings" you
will see a "Wide character in print" warning, because you are not encoding
your unicode characters to UTF-8 before printing them. However, on the
server where you are seeing the double encoding, perhaps you have a
PERL5OPT environment variable which is setting something like "-CSAD" and
thus encoding all output handles by default (this should not be globally
set, because it can mess with modules that don't expect it like this).
-Dan
Post by Alex Povolotsky
Quite interesting. The simple script
=== cut mojolog ===
#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "ПрПстП прОМт\n";
$log->info('тест лПга');
=== cut mojolog ===
yields fine results on all my servers but one.
On it, I'm getting
% perl mojolog
ПрПстП прОМт
[Sat Oct 6 15:22:43 2018] [info] ÑᅵеÑᅵÑᅵ лПга
The second line is clearly wrong and seems to be badly encoded. Same
version of OS, Perl, Mojolicious, same set of environment variables. What
else could influence unicode processing?
суббПта, 6 Пктября 2018 г., 13:35:48 UTC+3 пПльзПватель Alex Povolotsky
Post by Alex Povolotsky
Hello
How do I send UTF-8 constants to Mojo::Log? It forcefully encodes
everything so UTF-8 goes with double encoding and unreadable. Trying to
decode UTF-8 cat
$log->info(decode('UTF-8', 'тест лПга'));
results in
Use of uninitialized value $_[0] in join or string at
/usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.
Looks like something is wrong...
--
You received this message because you are subscribed to the Google
Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...