Missing instruction about accessing historical data in the migration guide

[Jacopo47]

Bug description
Scenario: We're upgrading out solution from Spring boot 2.7.x to 3.0.x passing so from Spring Batch: 4.3.x to 5.0.1
That means that our current application is already in production and it leverage in a JobRepository that is on an Oracle Database and it's filled by job executed in the past on top of version 4.3.x

Upgrading to version 5.0.1 we're now facing this exception while trying to access to job parameters' stored in the repository by v. 4.3.x :

java.lang.ClassNotFoundException: STRING
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao$2.processRow(JdbcJobExecutionDao.java:424)
	at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1688)
	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:723)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:773)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:789)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao.getJobParameters(JdbcJobExecutionDao.java:440)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao$JobExecutionRowMapper.mapRow(JdbcJobExecutionDao.java:469)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao$JobExecutionRowMapper.mapRow(JdbcJobExecutionDao.java:451)
	at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:94)
	at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:61)
	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:723)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815)
	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao.findJobExecutions(JdbcJobExecutionDao.java:172)
	at org.springframework.batch.core.explore.support.SimpleJobExplorer.getJobExecutions(SimpleJobExplorer.java:90)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:218)

I'm digging a bit into it in order to collect more details.
Right now what I can share is that this is the code that throws the exception:

Class<?> parameterType = null;
try {
	parameterType = Class.forName(rs.getString("PARAMETER_TYPE"));
}
catch (ClassNotFoundException e) {
	throw new RuntimeException(e);
}

And I think that it's due to the fact that: rs.getString("PARAMETER_TYPE") will return STRING if the record in BATCH_JOB_EXECUTION_PARAMS was add by a version <5.x.x. Instead if the record is add by a version >= 5.x.x this will work because on the DB the PARAMETER_TYPE will be: java.lang.String
(right now we're facing this issue only for STRING but if this supposition is confirmed other types: DATE, LONG, will face the same issue I guess)

Environment
Spring Batch version: from 4.3.x to 5.0.1
Java version: 17
Oracle 19 and Oracle 21

Steps to reproduce

1. Run a job on top of Spring Batch 4.3.x with a long-term repository (like on Oracle)
2. Access though Java, on top of Spring Batch 5.0.1, to the previous job's parameters that are stored in the long-term repository.

Expected behavior
Library should be able to access the data stored in a long-term repository by and older version

Minimal Complete Reproducible example
Unzip file: spring-batch-4352-mre.zip

1. Run an oracle DB instance with a TEST schema with Spring batch's tables at v. 4.3.x :

 docker run --name test -p 1521:1521 -e ORACLE_PASSWORD="iamapassword" -v `pwd`/init_db:/container-entrypoint-initdb.d gvenzl/oracle-xe:18.4.0-full

2. Execute job with Spring batch 4.3.x: cd batch-processing4.3.x/batch-processing ; ./gradlew clean bootRun

3. Execute job with Spring batch 5.0.x: cd ../../batch-processing5.0.x/batch-processing ; ./gradlew clean bootRun

This step at point 3. will fail due to: java.lang.ClassNotFoundException: STRING

[fmbenhassine]

Thank you for raising this. v4 and v5 are not compatible, and once the migration is done, historical data of v4 should not be accessed with v5 (which is what your sample is doing). The only case I see is restarting a failed job instance that was started with v4, and then attempted again with v5, but this should not be done.

We might need to clarify this in the migration guide, but users should ensure all failed v4 job instances are either restarted to success or abandoned before the migration. Does this clarify your concern?

[Jacopo47]

Hi @fmbenhassine ,
thank you for the clarification!
I agree with you that it would be great if this will be specified in the migration guide too.

Just I would highlight that restarting a failed job is not the only scenario in which "historical data" is accessed.
For example in the MRE that I shared I was trying to navigate the data through the JobExplorer:

explorer.findJobInstancesByJobName("test", 0, 10)
  .stream()
  .map(explorer::getJobExecutions)
  .flatMap(Collection::stream)
  .count();

What I mean is that there are APIs (like JobExplorer) that allows to navigate the data and those functionalities will be prevented by this breaking change.

Hoping to help you in the analysis let me share a bit about our scenario.
We have jobs that runs periodically and each jobs takes in consideration a time range that it's the DELTA. So candidate records that were modified since the latest considered records.
We're storing this information as job parameters.

So we define a job: EXTRACTION that expects two parameters: FROM and TO.

First run:
EXTRACTION (instance: 1) runs with FROM: 2023-01-01 , TO: 2023-01-31

Second run:
Before launch the job, through a JobExplorer we're looking for the latest job's execution in order to retrieve the TO param.
and then a new instance is run like:
EXTRACTION (instance: 2) runs with FROM: 2023-01-31 , TO: 2023-02-28

I don't know if how we're handle this scenario is a best practice but my aim was to share a scenario in which this issues impacts but it's not relate to restarting a failed job.

[fmbenhassine]

By once the migration is done, historical data of v4 should not be accessed with v5 (which is what your sample is doing), I was referring to that snippet. explorer.findJobInstancesByJobName("test", 0, 10) grabs the last 10 job instances, which could be a mix of v4 and v5.

Apologies for the confusion, but we definitely need to add this in the migration guide. In your time-based sample, you should calculate the last delta using v4, and once that done, you can migrate to v5 so that the next range is exclusively v5.

I don't think there is anything we can do here for the code, so I am going to turn this into a documentation issue and update the migration guide accordingly.

[Jacopo47]

Clear! Thank you for your support and the clarification!

[fmbenhassine]

I updated the migration guide with a new section about the implications of the change related to job parameters handling on historical meta-data access:

https://github.com/spring-projects/spring-batch/wiki/Spring-Batch-5.0-Migration-Guide#historical-data-access-implications.

Thanks again for raising this!