I'm exploring all the features of Spring Cloud Dataflow for our project. Currently I'm doing all development locally(No docker, Cloudfoundry & Kubernates) Currently I have created 2 Spring Batch Jobs using Spring Boot 3 & Spring Batch 5.
- Job 1 - Create a CSV file
- Job 2 - Read the CSV and insert data to DB
Since the current version of SCDF(2.10.3) doesn't support Spring Boot3, I have used 2.11.0-RC1 version for now as its only for local development.
I have created a Spring Boot application with version 2.7.14 (Same as the one used by SCDF 2.11.0-RC1). And Using own DB, I had successfully started the server on a port 7070 (instead of default 9393). Similarly I have started Skipper server as well(on a different port - 7500). And the 2 jobs were registered and the task was executed manually and it was successful.
Now I tried a composite task. But it does not run. Here is my server code.
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.14</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.demo.test</groupId>
<artifactId>scdfdemo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>scdfdemo</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-dataflow-server</artifactId>
<version>2.11.0-RC1</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dataflow</artifactId>
<version>2.11.0-RC1</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dataflow-ui</artifactId>
<version>3.4.0-RC1</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-skipper</artifactId>
<version>2.11.0-RC1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
The only Java code
package com.demo.test.scdfdemo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration;
import org.springframework.cloud.dataflow.server.EnableDataFlowServer;
import org.springframework.cloud.deployer.spi.cloudfoundry.CloudFoundryDeployerAutoConfiguration;
@EnableDataFlowServer
@SpringBootApplication(exclude = { CloudFoundryDeployerAutoConfiguration.class, SecurityAutoConfiguration.class,
ManagementWebSecurityAutoConfiguration.class})
public class ScdfdemoApplication {
public static void main(String[] args) {
SpringApplication.run(ScdfdemoApplication.class, args);
}
}
application.properties
server.port=7070
spring.main.cloud-platform=NONE
spring.profiles.active=cloud
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localhost:5432/scdfdemo
spring.datasource.username=postgres
spring.datasource.password=<password>
spring.sql.init.continue-on-error=false
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect
### SCDF Configurations
spring.cloud.dataflow.features.schedules-enabled=true
spring.cloud.dataflow.features.streams-enabled=true
spring.cloud.dataflow.features.tasks-enabled=true
spring.cloud.skipper.client.server-uri=http://localhost:7500/api
#spring.cloud.dataflow.task.composedtaskrunner.uri=http://localhost:7070
In the UI, I registered 2 apps and ran successfully. When I create a composite task(name :invoice-composed) with 2 tasks added, and if I open the task list it shows the composite job and the 2 sub jobs without any issues. And when I open the launch screen of composed task it shows UI properly without any error. But if we check log, it shows below exception
[2m2023-09-15 12:47:53.044[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-3][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:47:53.073[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-5][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:47:59.031[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-8][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:47:59.176[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-1][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:47:59.294[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-3][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:47:59.533[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-9][0;39m [36mApplicationConfigurationMetadataResolver[0;39m [2m:[0;39m Failed to retrieve properties for resource org.springframework.cloud:spring-cloud-dataflow-composed-task-runner:jar:2.11.0-RC1 because of UnknownHostException: repo.maven.apache.org
Why is it trying to connect to remote maven repo? I'm behind a proxy. So it wont get connected. Can we point to local repo directory? If so how?
Since there was no error in log, I launched the composite task and it showed error. Got below NullPointerException in logs.
[2m2023-09-15 12:49:13.556[0;39m [33m WARN[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-4][0;39m [36mo.s.c.d.a.t.AggregateExecutionSupport [0;39m [2m:[0;39m Cannot find AppRegistration for invoice-composed
[2m2023-09-15 12:49:13.580[0;39m [31mERROR[0;39m [35m3004[0;39m [2m---[0;39m [2m[nio-7070-exec-4][0;39m [36mo.s.c.d.s.c.RestControllerAdvice [0;39m [2m:[0;39m Caught exception while handling a request
java.lang.NullPointerException: Cannot invoke "String.contains(java.lang.CharSequence)" because the return value of "org.springframework.cloud.dataflow.core.dsl.TaskApp.getLabel()" is null
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionInfoService.composedTaskChildNames(DefaultTaskExecutionInfoService.java:256) ~[spring-cloud-dataflow-server-core-2.11.0-RC1.jar:na]
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService.executeTask(DefaultTaskExecutionService.java:418) ~[spring-cloud-dataflow-server-core-2.11.0-RC1.jar:na]
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$FastClassBySpringCGLIB$$422cda43.invoke(<generated>) ~[spring-cloud-dataflow-server-core-2.11.0-RC1.jar:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.29.jar:5.3.29]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.29.jar:5.3.29]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.29.jar:5.3.29]
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$EnhancerBySpringCGLIB$$9bd23d3b.executeTask(<generated>) ~[spring-cloud-dataflow-server-core-2.11.0-RC1.jar:na]
at org.springframework.cloud.dataflow.server.controller.TaskExecutionController.launchBoot3(TaskExecutionController.java:246) ~[spring-cloud-dataflow-server-core-2.11.0-RC1.jar:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.29.jar:5.3.29]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.29.jar:5.3.29]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.29.jar:5.3.29]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) ~[tomcat-embed-core-9.0.78.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.29.jar:5.3.29]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.78.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.29.jar:5.3.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.29.jar:5.3.29]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.29.jar:5.3.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.29.jar:5.3.29]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.29.jar:5.3.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.7.14.jar:2.7.14]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.29.jar:5.3.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.29.jar:5.3.29]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.29.jar:5.3.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:768) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.78.jar:9.0.78]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
I even tried to add composite task runner dependency in pom
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dataflow-composed-task-runner</artifactId>
<version>2.11.0-RC1</version>
</dependency>
But then the server itself doesn't start, throwing ClassNotFoundException.
Exception in thread "main" java.lang.IllegalArgumentException: Cannot instantiate interface org.springframework.context.ApplicationListener : org.springframework.cloud.dataflow.composedtaskrunner.support.CfServiceBindingPropertySourceInitializer
at org.springframework.boot.SpringApplication.createSpringFactoriesInstances(SpringApplication.java:449)
at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:431)
at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:424)
at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:267)
at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:245)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
at com.demo.test.scdfdemo.ScdfdemoApplication.main(ScdfdemoApplication.java:16)
Caused by: java.lang.ClassNotFoundException: org.springframework.cloud.dataflow.composedtaskrunner.support.CfServiceBindingPropertySourceInitializer
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
at org.springframework.boot.SpringApplication.createSpringFactoriesInstances(SpringApplication.java:442)
... 7 more
I Googled, but could not find which extra dependency is required here. Can someone help?