[Mojolicious] "mojo get" fails on some HTTPS website

[jo8532]

Hi,

I get no response for this website:

$ mojo get https://www.justhappy.fr/
$

I'm able to get the content of this webpage with other user-agents (wget,
telnet-ssl, Firefox, perl script using IO::Socket::SSL, ...). When I try to
use Mojo::UserAgent in my own script, I get a 200 response code, but the
content is empty.

Mojo works fine with other HTTPS websites:

$ mojo get https://www.google.com/ | wc -c
10803

Is this a bug in Mojolicious? Any help appreciated.

[sri]

Post by jo8532
Is this a bug in Mojolicious? Any help appreciated.

Yes, that is very likely a bug in out HTTP message framing code. We
should expect a message body in this case, but do not.

--
sebastian

[sri]

Afraid i don't have time to fix this at the moment. So we'll need a
volunteer.

--
sebastian

[sri]

If you open a GitHub issue for this i'll flag it as a confirmed bug.

--
sebastian

[sri]

It looks like the problem is that the response does not include a
Content-Length or Transfer-Encoding header. So Mojolicious defaults to
expecting a HTTP/1.1 response on a keep-alive connection without a body,
when it should expect a body terminated by a connection close.

--
sebastian

[sri]

I think all of this is related to Mojo::Content::expect_close. If that
defaults to 1 the site can be opened. We need better code to detect which
HTTP responses are supposed to be empty.

--
sebastian

[sri]

http://paste.opensuse.org/58336341

This patch actually makes the site work. But of course it also breaks
existing tests and
looks like it might not be exactly compatible with the HTTP spec, so we
can't just apply
it. Unfortunately it doesn't look like i will have time to properly look
into it anytime soon,
so i can only give you a few pointers.

But given the fact that nobody has even opened a GitHub issue yet, i
suppose this is
not exactly a high priority bug.

--
sebastian

[Tekki]

I've found a second site that doesn't work: https://gl.ch
What they have in common and what is different from other websites is that
the response headers are not written in lower case.
justhappy.fr:


1. Access-Control-Allow-Origin:
*
2. Cache-Control:
max-age=0, private, must-revalidate
3. Content-Encoding:
gzip
4. Content-Type:
text/html; charset=utf-8
5. Date:
Tue, 07 Aug 2018 06:38:04 GMT
6. ETag:
W/"a43244322c99f68a2d2aea1b7e491de9"
7. Set-Cookie:
_simplebo_tool_session=dGZYU3k5d2VVSS9IejQzRjBlTjk2NzZjZEhpSGRvWjFoMmlpZWRtYWJsdEF2RWpTbjYxazlRZ0ZENVlaaTVvTWNpNmtFd1R1QmFPUEFWR1A1NnJERFc3KzhUbUpDeVdzbSs3L0pXWHJwN0ZSZ1l5Y0N1b3N2QXJRTEJlRkVaVHl3dTRVYWdINjBxYVpOYjh2OFhmeUVBPT0tLTdxUHlSS0NzWTN5Z2EydVdqRmo4L1E9PQ%3D%3D--fb247d72a1e3b42cf7f940100584403db220eb73;
path=/; HttpOnly
8. Vary:
Accept-Encoding
9. Via:
1.1 alproxy
10. X-Content-Type-Options:
nosniff
11. X-Request-Id:
71265db3-9b5d-46c4-807e-1629fa89fcbd
12. X-ServedBy:
simplebo
13. X-XSS-Protection:
1; mode=block


gl.ch:


1. Connection:
close
2. Content-Encoding:
gzip
3. Content-Language:
de-CH
4. Content-Type:
text/html;charset=ISO-8859-1
5. Date:
Tue, 07 Aug 2018 06:40:22 GMT
6. Transfer-Encoding:
chunked
7. Vary:
Accept-Encoding
8. X-Powered-By:
ASP.NET


No idea if this is related to the problem.

Tekki

Post by jo8532
Hi,
$ mojo get https://www.justhappy.fr/

[Dmitry L.]

It is problem with website, but not with mojo.

Try:
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'

Post by jo8532
Hi,
$ mojo get https://www.justhappy.fr/
$
I'm able to get the content of this webpage with other user-agents (wget, telnet-ssl, Firefox, perl script using IO::Socket::SSL, ...). When I try to use Mojo::UserAgent in my own script, I get a 200 response code, but the content is empty.
$ mojo get https://www.google.com/ | wc -c
10803
Is this a bug in Mojolicious? Any help appreciated.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Tekki]

Why a problem with the website? Try:
curl -v 'https://www.google.com/' -H 'Accept-Encoding: gzip'
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip' | gunzip -c

Post by Dmitry L.
It is problem with website, but not with mojo.
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'

[Dmitry L.]

There are no content-length header

Post by Tekki
curl -v 'https://www.google.com/' -H 'Accept-Encoding: gzip'
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip' | gunzip -c

Post by Dmitry L.
It is problem with website, but not with mojo.
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'
--

You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Tekki]

Correct, the same for https://gl.ch

Post by Dmitry L.
There are no content-length header

[jo8532]

Thank you to everyone who tried to help. Thank you sri for publishing a fix.