[Bug]: did not validate against our published schema #1072
Comments
|
please give the complete running output |
It was found that it was caused by this SQL. I forced to remove is null in the code and there was no problem importing it to S3.select * from test_his where (id is not null) AND id IS NULL SingleTableSplitUtil public static List<Configuration> splitSingleTable(Configuration configuration, int adviceNum)
{
List<Configuration> pluginParams = new ArrayList<>();
List<String> rangeList;
String splitPkName = configuration.getString(Key.SPLIT_PK);
String column = configuration.getString(Key.COLUMN);
String table = configuration.getString(Key.TABLE);
String where = configuration.getString(Key.WHERE, null);
boolean hasWhere = StringUtils.isNotBlank(where);
if (dataBaseType == DataBaseType.Oracle) {
rangeList = genSplitSqlForOracle(splitPkName, table, where, configuration, adviceNum);
// warn: mysql etc to be added...
}
else {
Pair<Object, Object> minMaxPK = getPkRange(configuration);
if (null == minMaxPK) {
throw AddaxException.asAddaxException(DBUtilErrorCode.ILLEGAL_SPLIT_PK,
"Primary key-based table splitting failed. The key type ONLY supports integer and string.");
}
configuration.set(Key.QUERY_SQL, buildQuerySql(column, table, where));
if (null == minMaxPK.getLeft() || null == minMaxPK.getRight()) {
// 切分后获取到的start/end 有 Null 的情况
pluginParams.add(configuration);
return pluginParams;
}
boolean isStringType = Constant.PK_TYPE_STRING.equals(configuration.getString(Constant.PK_TYPE));
boolean isLongType = Constant.PK_TYPE_LONG.equals(configuration.getString(Constant.PK_TYPE));
if (isStringType) {
rangeList = splitStringPk(configuration, table, where, minMaxPK.getLeft().toString(), minMaxPK.getRight().toString(),
adviceNum, splitPkName);
}
else if (isLongType) {
rangeList = RdbmsRangeSplitWrap.splitAndWrap(new BigInteger(minMaxPK.getLeft().toString()),
new BigInteger(minMaxPK.getRight().toString()), adviceNum, splitPkName);
}
else {
throw AddaxException.asAddaxException(DBUtilErrorCode.ILLEGAL_SPLIT_PK,
"the splitPk[" + splitPkName + "] type is unsupported, it only support int and string");
}
}
String tempQuerySql;
List<String> allQuerySql = new ArrayList<>();
if (!rangeList.isEmpty()) {
for (String range : rangeList) {
Configuration tempConfig = configuration.clone();
tempQuerySql = buildQuerySql(column, table, where) + (hasWhere ? " and " : " where ") + range;
allQuerySql.add(tempQuerySql);
tempConfig.set(Key.QUERY_SQL, tempQuerySql);
pluginParams.add(tempConfig);
}
}
else {
//pluginParams.add(configuration); // this is wrong for new & old split
Configuration tempConfig = configuration.clone();
tempQuerySql = buildQuerySql(column, table, where)
+ (hasWhere ? " AND " : " wHERE ")
+ String.format(" %s IS NOT NULL", splitPkName);
allQuerySql.add(tempQuerySql);
tempConfig.set(Key.QUERY_SQL, tempQuerySql);
pluginParams.add(tempConfig);
}
// deal pk is null
//Configuration tempConfig = configuration.clone();
// tempQuerySql = buildQuerySql(column, table, where)
// + (hasWhere ? " AND " : " WHERE ")
// + String.format(" %s IS NULL", splitPkName);
//allQuerySql.add(tempQuerySql);
LOG.info("After split(), allQuerySql=[\n{}\n].", StringUtils.join(allQuerySql, "\n"));
//tempConfig.set(Key.QUERY_SQL, tempQuerySql);
// pluginParams.add(tempConfig);
return pluginParams;
} |
No where conditionThere will be is null without the where condition, and the same error will be reported. I see that the code will still go this way when where is empty. |
|
Assuming your table has null values in the splitting field(splitPK), how should these null records be handled? Without specifying a WHERE clause, the expectation is to transfer the same number of records as in the source table. However, null records don't meet S3 upload requirements. This is a user-configurable scenario, such as specifying a WHERE clause, rather than simply ignoring these records. In summary, if you want to exclude records with null values, you should configure the WHERE clause instead of hardcoding in source code. |
|
Thanks for the answer. I have another question. If other columns are empty, \N,null will be written, which is equivalent to writing two columns. How can I solve this problem? |
What happened?
A bug happened!
Version
4.1.5 (Default)
OS Type
Linux (Default)
Java JDK Version
OpenJDK 1.8.0
Relevant log output
The text was updated successfully, but these errors were encountered: