Discussion:
"mojo get" fails on some HTTPS website
(too old to reply)
jo8532
2018-08-06 15:37:33 UTC
Permalink
Hi,

I get no response for this website:

$ mojo get https://www.justhappy.fr/
$

I'm able to get the content of this webpage with other user-agents (wget,
telnet-ssl, Firefox, perl script using IO::Socket::SSL, ...). When I try to
use Mojo::UserAgent in my own script, I get a 200 response code, but the
content is empty.

Mojo works fine with other HTTPS websites:

$ mojo get https://www.google.com/ | wc -c
10803

Is this a bug in Mojolicious? Any help appreciated.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-06 15:54:05 UTC
Permalink
Post by jo8532
Is this a bug in Mojolicious? Any help appreciated.
Yes, that is very likely a bug in out HTTP message framing code. We
should expect a message body in this case, but do not.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-06 16:25:23 UTC
Permalink
Afraid i don't have time to fix this at the moment. So we'll need a
volunteer.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-06 16:50:08 UTC
Permalink
If you open a GitHub issue for this i'll flag it as a confirmed bug.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-07 08:49:24 UTC
Permalink
It looks like the problem is that the response does not include a
Content-Length or Transfer-Encoding header. So Mojolicious defaults to
expecting a HTTP/1.1 response on a keep-alive connection without a body,
when it should expect a body terminated by a connection close.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-07 09:16:46 UTC
Permalink
I think all of this is related to Mojo::Content::expect_close. If that
defaults to 1 the site can be opened. We need better code to detect which
HTTP responses are supposed to be empty.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-08-07 09:32:37 UTC
Permalink
http://paste.opensuse.org/58336341

This patch actually makes the site work. But of course it also breaks
existing tests and
looks like it might not be exactly compatible with the HTTP spec, so we
can't just apply
it. Unfortunately it doesn't look like i will have time to properly look
into it anytime soon,
so i can only give you a few pointers.

But given the fact that nobody has even opened a GitHub issue yet, i
suppose this is
not exactly a high priority bug.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Tekki
2018-08-07 06:48:41 UTC
Permalink
I've found a second site that doesn't work: https://gl.ch
What they have in common and what is different from other websites is that
the response headers are not written in lower case.
justhappy.fr:


1. Access-Control-Allow-Origin:
*
2. Cache-Control:
max-age=0, private, must-revalidate
3. Content-Encoding:
gzip
4. Content-Type:
text/html; charset=utf-8
5. Date:
Tue, 07 Aug 2018 06:38:04 GMT
6. ETag:
W/"a43244322c99f68a2d2aea1b7e491de9"
7. Set-Cookie:
_simplebo_tool_session=dGZYU3k5d2VVSS9IejQzRjBlTjk2NzZjZEhpSGRvWjFoMmlpZWRtYWJsdEF2RWpTbjYxazlRZ0ZENVlaaTVvTWNpNmtFd1R1QmFPUEFWR1A1NnJERFc3KzhUbUpDeVdzbSs3L0pXWHJwN0ZSZ1l5Y0N1b3N2QXJRTEJlRkVaVHl3dTRVYWdINjBxYVpOYjh2OFhmeUVBPT0tLTdxUHlSS0NzWTN5Z2EydVdqRmo4L1E9PQ%3D%3D--fb247d72a1e3b42cf7f940100584403db220eb73;
path=/; HttpOnly
8. Vary:
Accept-Encoding
9. Via:
1.1 alproxy
10. X-Content-Type-Options:
nosniff
11. X-Request-Id:
71265db3-9b5d-46c4-807e-1629fa89fcbd
12. X-ServedBy:
simplebo
13. X-XSS-Protection:
1; mode=block


gl.ch:


1. Connection:
close
2. Content-Encoding:
gzip
3. Content-Language:
de-CH
4. Content-Type:
text/html;charset=ISO-8859-1
5. Date:
Tue, 07 Aug 2018 06:40:22 GMT
6. Transfer-Encoding:
chunked
7. Vary:
Accept-Encoding
8. X-Powered-By:
ASP.NET


No idea if this is related to the problem.

Tekki
Post by jo8532
Hi,
$ mojo get https://www.justhappy.fr/
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dmitry L.
2018-08-07 08:12:56 UTC
Permalink
It is problem with website, but not with mojo.

Try:
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'
Post by jo8532
Hi,
$ mojo get https://www.justhappy.fr/
$
I'm able to get the content of this webpage with other user-agents (wget, telnet-ssl, Firefox, perl script using IO::Socket::SSL, ...). When I try to use Mojo::UserAgent in my own script, I get a 200 response code, but the content is empty.
$ mojo get https://www.google.com/ | wc -c
10803
Is this a bug in Mojolicious? Any help appreciated.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
//wbr, Dmitry L.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Tekki
2018-08-07 13:52:58 UTC
Permalink
Why a problem with the website? Try:
curl -v 'https://www.google.com/' -H 'Accept-Encoding: gzip'
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip' | gunzip -c
Post by Dmitry L.
It is problem with website, but not with mojo.
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dmitry L.
2018-08-07 14:51:56 UTC
Permalink
There are no content-length header
Post by Tekki
curl -v 'https://www.google.com/' -H 'Accept-Encoding: gzip'
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip' | gunzip -c
Post by Dmitry L.
It is problem with website, but not with mojo.
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Tekki
2018-08-08 06:18:56 UTC
Permalink
Correct, the same for https://gl.ch
Post by Dmitry L.
There are no content-length header
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
jo8532
2018-08-09 08:58:23 UTC
Permalink
Thank you to everyone who tried to help. Thank you sri for publishing a fix.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...