Elasticsearch 8.3 configuration in Spring Boot

Question

I have been trying to implement Elastic Search 8.3 with spring boot.

i am getting the below exception:

Cannot convert value of type 'co.elastic.clients.elasticsearch.ElasticsearchClient' to required type 'org.springframework.data.elasticsearch.core.ElasticsearchOperations'

Below is my configuration for elastic search:

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder.HttpClientConfigCallback;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.repository.config.EnableElasticsearchRepositories;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientTransport;
@Configuration
@EnableElasticsearchRepositories
public class ElasticConfig {

    @Bean
    public RestClient client() {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", "test321"));

        HttpHost host = new HttpHost("localhost", 9200);

        RestClient restClient = RestClient.builder(host)
                .setHttpClientConfigCallback(new HttpClientConfigCallback() {

                    @Override
                    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                        // TODO Auto-generated method stub
                        httpClientBuilder.disableAuthCaching();
                        return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                    }
                }).build();
        return restClient;
    }

    @Bean
    public ElasticsearchClient elasticsearchTemplate() {
        ElasticsearchTransport transport = new RestClientTransport(client(), new JacksonJsonpMapper());
        return new ElasticsearchClient(transport);
    } 
    
}

Below is POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.7.2</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>aisha</groupId>
    <artifactId>courseapp</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>war</packaging>
    <name>courseapp</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
        <guava.version>20.0</guava.version>
        <geoip2.version>2.15.0</geoip2.version>
        <uap-java.version>1.4.0</uap-java.version>

    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-websocket</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.tomcat.embed</groupId>
            <artifactId>tomcat-embed-jasper</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>jstl</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-validation</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>${guava.version}</version>
        </dependency>
        <dependency>
            <groupId>com.maxmind.geoip2</groupId>
            <artifactId>geoip2</artifactId>
            <version>${geoip2.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
        </dependency>
        
        
         <dependency>
            <groupId>com.github.ua-parser</groupId>
            <artifactId>uap-java</artifactId>
            <version>${uap-java.version}</version>
        </dependency>
        <dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-elasticsearch</artifactId>
        </dependency>
        
          <dependency>
      <groupId>co.elastic.clients</groupId>
      <artifactId>elasticsearch-java</artifactId>
      <version>8.3.3</version>
    </dependency>

    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
    </dependency>
     <dependency>
      <groupId>jakarta.json</groupId>
      <artifactId>jakarta.json-api</artifactId>
      <version>2.0.1</version>
    </dependency>
   

<!-- https://mvnrepository.com/artifact/com.github.javafaker/javafaker -->
<dependency>
    <groupId>com.github.javafaker</groupId>
    <artifactId>javafaker</artifactId>
    <version>1.0.2</version>
</dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Below is test class which call simple query for index product :

@SpringBootTest

public class ElkSpringTest {

     @Autowired
        private ElasticsearchRestTemplate elasticConfig;
    
    
    
    @Test
    void contextLoads() {
        SearchResponse<ELKProduct> search;

        try {
            search = elasticConfig.search(s -> s.index("product").size(10), ELKProduct.class);
            System.out.println(search.hits().total().value());

            for (Hit<ELKProduct> hit : search.hits().hits()) {
                System.out.println(hit.source().getId());
            }

            System.out.println("******ENding *****");
        } catch (ElasticsearchException | IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        
        
        
    }
}

Since elastic is upgraded to latest version 8.3. I am not able to set up elastic search in spring boot. I usually work with ElasticSearch repository to store the data in my indexs and query it using ElasticsearchOperations.

Answer 1

spring-data-elasticsearch(version 5.0) is now compatible with ES 8.5.0

It looks like the RestHighLevelClient is deprecated as of this version.

Check this documentation for implementing the client configuration.

Answer 2

Current spring-data-elasticsearch(version 4.4) not much compatible with es version 8.3 (>7.17.6).

ES RestHighLevelClient can setApiCompatibilityMode(true) to enables compatibility mode that allows HLRC 7.17 to work with Elasticsearch 8.x. But the compatibility is not very good.

Answer 3

On the date of this answer, I am currently able to config Java API client (provided officially by Elasticsearch) to works with ElasticSearch 8.3.x

Version of dependencies used (if it helps):

• org.springframework.boot:spring-boot-starter:2.7.13
• org.springframework.data:spring-data-elasticsearch:4.4.13
• co.elastic.clients:elasticsearch-java:7.17.9 (depended by the above one).

Your ElasticConfig (with the annotation @Configuration) class just needs to extend ElasticsearchConfiguration abstract class.

Something similar to this

@Configuration
public class ElasticConfig extends ElasticsearchConfiguration {
    @Override
    public ClientConfiguration clientConfiguration() {
        try {

            // TODO: Should be on the dev profile to run this on self-signed TLS server.
            final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
            credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(this.esUsername, this.esPassword));

            SSLContextBuilder sslBuilder = SSLContexts.custom()
                    .loadTrustMaterial(null,
                            (x509Certificates, s) -> true
                    );
            final SSLContext sslContext = sslBuilder.build();

            return ClientConfiguration.builder()
                    .connectedTo(esUris)
                    .usingSsl(sslContext, NoopHostnameVerifier.INSTANCE)
                    .withConnectTimeout(10000)
                    .withBasicAuth(esUsername, esPassword)
                    .build();
        } catch (Exception e) {
            // throwing properly
            return null;
        }

    }
}

The abstract class has (pre)-declared enough Bean for our job. We only need to declare some Operations or Repository to do the job.

Adapt your esUris, esUsername, esPassword with yours or eliminate them if you're using an unsecure local ES server.