Uploaded image for project: 'Hyrax Data Server'
  1. HYRAX-767

Change the Data Request Form code (all 3 versions) so that it URL encodes the query before using it.

    Details

    • Type: Bug
    • Status: Done (View workflow)
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: bes, olfs
    • Labels:
    • Environment:

      Description

      Currently Tomcat-8.5.31 ships configured to reject URL's that are not correctly URL encoded to the HTTP-1.1 standard. This is a part of a larger security push industry wide and we need to get out client code (the Data Request Forms) into compliance by updating our Data Request Form generation code so that the Form URL encodes the URL before an attempt is made to dereference it. This should not be applied to the text in the Data URL field in the form as it will make it impossible to read. Rather, we do the encoding as we send the request.

      Here is the original message that brought this to my attention:

      Hi,
      Some of our users are reporting that some requests to our opendap are returning a 400 to them. For example this link returns a 400 error:
      https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:359]

      We consistently get a 400 error on that link but if you remove the XDim parameter the link works. Additionally, the response seems to be slightly different. For example with curl we simply receive a 400 HTTP code with no data while with Chrome browser we get a 400 code along with message:

      Error {
          code = 400;
          message = "libdap exception building response: error_code = 1005: Failed to get values as ascii: Constraint expression parse error: syntax error";
      }
       

      Since this is a live system with a lot of requests it’s hard to associate error messages from tomcat with a particular request but I think when we make this request we receive this error message from tomcat:

      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | 24-May-2018 21:55:17.313 INFO [http-nio-8080-exec-4] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    |  Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    |  java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:479)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:687)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.lang.Thread.run(Thread.java:748)
       

      I partially suspect that the log is unrelated to this error.

      We’re running olfs version 1.17.0, libdap and bes version 3.19.1-1.
      The olfs container is based off of the tomcat:8-jre8 image. The exact tomcat version is 8.5.31.0. The base OS is Debian 9.4.
      The bes container is based off of centos:7 image. The bes image is identical to https://github.com/OPENDAP/hyrax-docker/blob/master/hyrax-1.14.0/besd/Dockerfile

      Any help?

      Navid

        Attachments

          Activity

            People

            • Assignee:
              ndp Nathan Potter
              Reporter:
              ndp Nathan Potter
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour
                1h