How to handle � in spring boot with rest client

Question

These are my Data classes

@JsonIgnoreProperties
data class Tri4Data(
    @JsonProperty("Status")
    val status: String,
    @JsonProperty("data")
    val data: Tri4JsonData
)
@JsonIgnoreProperties
data class Tri4JsonData(
    @JsonProperty("children")
    val children: List<Tri4Children>
)
data class Tri4Children(
    @JsonProperty("ID")
    val id: Int,
    @JsonProperty("Code")
    val code: String,
    @JsonProperty("Description")
    val description: String?,
    @JsonProperty("Gender")
    val gender: String?,
    @JsonProperty("Initials")
    val initials: String?,
    @JsonProperty("Surname")
    val surname: String?,
    @JsonProperty("Pracno")
    val pracNum: String?
   )

and this is my api request

restTemplate.exchange(url, HttpMethod.GET, httpEntity, Tri4Data::class.java).body

this is the error i am receiving and i assume its from this response in the API call:

org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: 
Invalid UTF-8 start byte 0x86

response

        {
            "ID": 27203,
            "Code": "G73.1",
            "Description": "Lambert-Eaton syndrome (C00-D48�) "
        },

Answer 1

Could you print response headers?

When I'm trying with latest Spring Boot app (3.2.3), I only get this behavior when the server explicitly returns header 'Content-type', 'application/json; charset=utf-8' (but then returns this UTF-8 invalid byte). If the header is not present, I'm getting different errors. So I assume that the server you're reaching to communicates that Content-Type is UTF-8 but proceeds to return invalid byte as part of its response.

The 0x86 seems to be coming from windows-1252 (or similar) and represents †.

Maybe, the server you're communicating with is "well-behaved" and if you specify that you accept UTF-8, it would return it in UTF-8. You can achieve that by something like:

HttpHeaders headers = new HttpHeaders();
headers.set("Accept-Encoding", "application/json;charset=UTF-8");
HttpEntity<...> httpEntity = new HttpEntity<>(headers);
// and use this in restTemplate.exchange

If you're integrating with legacy (so it can't be changed) and ill-behaved server, then another non-standard possibility would be to implement and register custom HttpMessageConverter that would accept UTF-8 encoding, but internally treat it as windows-1252. (see this question for context)

That being said, as pointed out in the comment above - RFC-8259 states, that JSON should be encoded as UTF-8. This JSON does not conform to that, as it contains string field that is not valid UTF-8 encoded string.

But there is room for (mis)interpretation depending on the closed ecosystem definition:

JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8