Properly parse field containing '+' char

Question

I'm running into a strange situation which I reproduced in https://github.com/lgueye/uri-parameters-behavior

Since we migrated to spring-boot 2 (spring framework 5) when requesting one of our backends in GET method we ran into the following situation: all fields with a + char were altered to (whitespace) char when they reached the backend

The following values are altered:

• +412386789 (phone number) into ** 412386789**
• 2019-03-22T17:18:39.621+02:00 (java8 ZonedDateTime) into 2019-03-22T17:18:39.621 02:00 (resulting in a org.springframework.validation.BindException

I've spent quite some time on stackoverflow (https://github.com/spring-projects/spring-framework/issues/14464#issuecomment-453397378) and github (https://github.com/spring-projects/spring-framework/issues/21577)

I've implemented both a mockMvc unit test and an integration test

The unit test behaves properly The integration test fails (like our production)

Can anyone point help me fix this issue ? My goal is obviously to make the integration test pass.

Thank you for your help.

Louis

Answer 1

After some fight with this issue I finally got it to work the way we expect it in our company.

The offending component is not spring-boot but rather UriComponentsBuilder

My initial failing test looks like this:

    @Test
public void get_should_properly_convert_query_parameters() {
    // Given
    final String device = UUID.randomUUID().toString();
    final String id = UUID.randomUUID().toString();
    final String phoneNumber = "+225697845";
    final String value = "foo#bar@quizz+foo-bazz//quir.";
    final Instant now = Instant.now();
    final ZonedDateTime timestamp = ZonedDateTime.ofInstant(now, ZoneId.of("+02:00"));

    final MultiValueMap<String, String> params = new LinkedMultiValueMap<>();
    params.add("id", id);
    params.add("device", device);
    params.add("phoneNumber", phoneNumber);
    params.add("timestamp", timestamp.toString());
    params.add("value", value);

    final URI uri = UriComponentsBuilder.fromHttpUrl("http://localhost").port(port).path("/events").queryParams(params).build().toUri();
    final Event expected = Event.builder().device(device).id(id).phoneNumber(phoneNumber).value(value).timestamp(timestamp).build();

    // When
    final Event actual = restTemplate.exchange(uri, HttpMethod.GET, new HttpEntity<>(new HttpHeaders()), Event.class).getBody();

    // Then
    assertEquals(expected, actual);
}

The working version looks like this:

    @Test
public void get_should_properly_convert_query_parameters() {
    // Given
    final String device = UUID.randomUUID().toString();
    final String id = UUID.randomUUID().toString();
    final String phoneNumber = "+225697845";
    final String value = "foo#bar@quizz+foo-bazz//quir.";
    final Instant now = Instant.now();
    final ZonedDateTime timestamp = ZonedDateTime.ofInstant(now, ZoneId.of("+02:00"));
    final Map<String, String> params = new HashMap<>();
    params.put("id", id);
    params.put("device", device);
    params.put("phoneNumber", phoneNumber);
    params.put("timestamp", timestamp.toString());
    params.put("value", value);
    final MultiValueMap<String, String> paramTemplates = new LinkedMultiValueMap<>();
    paramTemplates.add("id", "{id}");
    paramTemplates.add("device", "{device}");
    paramTemplates.add("phoneNumber", "{phoneNumber}");
    paramTemplates.add("timestamp", "{timestamp}");
    paramTemplates.add("value", "{value}");

    final URI uri = UriComponentsBuilder.fromHttpUrl("http://localhost").port(port).path("/events").queryParams(paramTemplates).encode().buildAndExpand(params).toUri();
    final Event expected = Event.builder().device(device).id(id).phoneNumber(phoneNumber).value(value).timestamp(ZonedDateTime.ofInstant(now, ZoneId.of("UTC"))).build();

    // When
    final Event actual = restTemplate.exchange(uri, HttpMethod.GET, new HttpEntity<>(new HttpHeaders()), Event.class).getBody();

    // Then
    assertEquals(expected, actual);
}

Note 4 required differences:

• MultiValueMap param templates is required
• Map param value is required
• encode is required
• buildAndExpand with param values is required

I'm a bit sad because all this is quite error prone and crumbersome (specially the Map/MultiValueMap part). I would gladly have them generated from a java bean.

This has a big impact on our solution but I'm afraid we won't have a choice. We'll settle for this solution for now.

Hope this helps others facing this issue.

Best,

Louis

Answer 2

The whole misalignment comes from the fact that there's a non-standard practice how to encode/decode space into "+".

Arguably space can(is being) encoded into "+" or "%20".

For example Google does this to the search strings:

https://www.google.com/search?q=test+my+space+delimited+entry

rfc1866, section-8.2.2 states that the query part of a GET request should be encoded in 'application/x-www-form-urlencoded'.

The default encoding for all forms is `application/x-www-form-
urlencoded'. A form data set is represented in this media type as
follows:

1. The form field names and values are escaped: space characters are replaced by '+'.

On the other hand rfc3986 states that spaces in URLs have to be encoded using "%20".

This basically means there's a different standards to encode spaces, depending on where they are in the URI syntax components.

     foo://example.com:8042/over/there?name=ferret#nose
     \_/   \______________/\_________/ \_________/ \__/
      |           |            |            |        |
   scheme     authority       path        query   fragment
      |   _____________________|__
     / \ /                        \
     urn:example:animal:ferret:nose

Based on these remarks, we can state that in GET http calls in URIs:

• spaces before "?" needs to be encoded to "%20"
• spaces after "?" in the query parameters needs to be encoded to "+"
• which means "+" signs needs to be encoded to "%2B" in query parameters

Spring implementation is following the rfc specifications, so that's why when you send "+412386789" in the query parameters, the "+" sign is interpreted as whitespace char and it gets to the backend as " 412386789".

Looking at:

final URI uri = UriComponentsBuilder.fromHttpUrl("http://localhost")
                                    .port(port)
                                    .path("/events")
                                    .queryParams(params)
                                    .build()
                                    .toUri();

You will find that:

"foo#bar@quizz+foo-bazz//quir." is encoded to "foo%23bar@quizz+foo-bazz//quir." which conforms to the specification (rfc3986).

So if you want the "+" char in your query params to not be interpreted as space, you need to encode it to "%2B".

The parameters you're sending to backend should look like:

   params.add("id", id);
   params.add("device", device);
   params.add("phoneNumber", "%2B225697845");
   params.add("timestamp", "2019-03-25T15%3A09%3A44.703088%2B02%3A00");
   params.add("value", "foo%23bar%40quizz%2Bfoo-bazz%2F%2Fquir.");

In order to do that you can use UrlEncoder when passing the parameters to the map. Beware of UriComponentsBuilder double encoding your stuff!

You can achieve correct URL with:

final MultiValueMap<String, String> params = new LinkedMultiValueMap<>();
params.add("id", id);
params.add("device", device);
String uft8Charset = StandardCharsets.UTF_8.toString();
params.add("phoneNumber", URLEncoder.encode(phoneNumber, uft8Charset));
params.add("timestamp", URLEncoder.encode(timestamp.toString(), uft8Charset));
params.add("value", URLEncoder.encode(value, uft8Charset));

final URI uri = UriComponentsBuilder.fromHttpUrl("http://localhost")
                                    .port(port)
                                    .path("/events")
                                    .queryParams(params)
                                    .build(true)
                                    .toUri();

Note that passing "true" to the build() method turns off the encoding, so this means the scheme, host etc. from the URI parts won't be encoded properly by UriComponentsBuilder.