Change the Data Request Form code (all 3 versions) so that it URL encodes the query before using it.

Description

Currently Tomcat-8.5.31 ships configured to reject URL's that are not correctly URL encoded to the HTTP-1.1 standard. This is a part of a larger security push industry wide and we need to get out client code (the Data Request Forms) into compliance by updating our Data Request Form generation code so that the Form URL encodes the URL before an attempt is made to dereference it. This should not be applied to the text in the Data URL field in the form as it will make it impossible to read. Rather, we do the encoding as we send the request.

Here is the original message that brought this to my attention:

Hi,
Some of our users are reporting that some requests to our opendap are returning a 400 to them. For example this link returns a 400 error:
https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:359]

We consistently get a 400 error on that link but if you remove the XDim parameter the link works. Additionally, the response seems to be slightly different. For example with curl we simply receive a 400 HTTP code with no data while with Chrome browser we get a 400 code along with message:

Since this is a live system with a lot of requests it’s hard to associate error messages from tomcat with a particular request but I think when we make this request we receive this error message from tomcat:

I partially suspect that the log is unrelated to this error.

We’re running olfs version 1.17.0, libdap and bes version 3.19.1-1.
The olfs container is based off of the tomcat:8-jre8 image. The exact tomcat version is 8.5.31.0. The base OS is Debian 9.4.
The bes container is based off of centos:7 image. The bes image is identical to https://github.com/OPENDAP/hyrax-docker/blob/master/hyrax-1.14.0/besd/Dockerfile

Any help?

Navid

Environment

We’re running olfs version 1.17.0, libdap and bes version 3.19.1-1.

The olfs container is based off of the tomcat:8-jre8 image. The exact tomcat version is 8.5.31.0. The base OS is Debian 9.4.

The bes container is based off of centos:7 image. The bes image is identical to https://github.com/OPENDAP/hyrax-docker/blob/master/hyrax-1.14.0/besd/Dockerfile

Activity

Show:
Nathan Potter
May 25, 2018, 10:24 PM
Edited

From https://tomcat.apache.org/tomcat-8.5-doc/config/http.html I see that Tomcat is now strictly enforcing HTTP-1.1 encoding standards for URLs.

There are three configuration parameters for the Tomcat Connector that allow some remediation for users that have operational systems.

From https://tomcat.apache.org/tomcat-8.5-doc/config/http.html

rejectIllegalHeaderName
If an HTTP request is received that contains an illegal header name (i.e. the header name is not a token) this setting determines if the request will be rejected with a 400 response (true) or if the illegal header be ignored (false). The default value is false which will cause the request to be processed but the illegal header will be ignored.

relaxedPathChars
The HTTP/1.1 specification requires that certain characters are %nn encoded when used in URI paths. Unfortunately, many user agents including all the major browsers are not compliant with this specification and use these characters in unencoded form. To prevent Tomcat rejecting such requests, this attribute may be used to specify the additional characters to allow. If not specified, no addtional characters will be allowed. The value may be any combination of the following characters: " < > [ \ ] ^ ` &#123; | &#125; . Any other characters present in the value will be ignored.

relaxedQueryChars
The HTTP/1.1 specification requires that certain characters are %nn encoded when used in URI query strings. Unfortunately, many user agents including all the major browsers are not compliant with this specification and use these characters in unencoded form. To prevent Tomcat rejecting such requests, this attribute may be used to specify the additional characters to allow. If not specified, no addtional characters will be allowed. The value may be any combination of the following characters: " < > [ \ ] ^ ` &#123; | &#125; ". Any other characters present in the value will be ignored.

Hopefully we can get this into the next release.

James Gallagher
May 25, 2018, 11:12 PM

Hmm. I guess we should hack the form (ifh) and probably talk to the folks at Unidata...

Done

Assignee

Nathan Potter

Reporter

Nathan Potter

Labels

Time remaining

0m

Epic Link

Components

Priority

Medium
Configure