BigQuery Java API: NPE on com.google.cloud.bigquery.StandardTableDefinition.fromPb(StandardTableDefinition.java:298)

[chetanmeda]

Environment details

1. Specify the API at the beginning of the title (for example, "BigQuery: ...") : BigQuery
   General, Core, and Other are also allowed as types
2. OS type and version: Any OS
3. Java version: 1.8
4. google-cloud-java version(s):
   com.google.cloud
   google-cloud-bigquery
   1.110.0

Steps to reproduce

NOT ABLE TO REPRODUCE

Code example

Page<Table> tables = bigquery.listTables(datasetId, BigQuery.TableListOption.pageSize(1));
Iterable<Table> table_iterator = tables.iterateAll();
for (Table table : table_iterator){
if (table.getDefinition().getType().toString().equalsIgnoreCase("TABLE")) {
                                           System.out.println("table");
                                        }
}

Stack trace

Caused by: java.lang.NullPointerException: Null pointer - Got unexpected time partitioning {"field":"XXXXX"} in project XXXXXX in dataset XXXXX in table XXXXX java.lang.NullPointerException: Name is null
        at com.google.cloud.bigquery.StandardTableDefinition.fromPb(StandardTableDefinition.java:298)
        at com.google.cloud.bigquery.TableDefinition.fromPb(TableDefinition.java:151)
        at com.google.cloud.bigquery.TableInfo$BuilderImpl.<init>(TableInfo.java:188)
        at com.google.cloud.bigquery.Table.fromPb(Table.java:624)
        at com.google.cloud.bigquery.BigQueryImpl$21.apply(BigQueryImpl.java:847)
        at com.google.cloud.bigquery.BigQueryImpl$21.apply(BigQueryImpl.java:844)
        at com.google.common.collect.Iterators$6.transform(Iterators.java:786)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
        at com.google.cloud.PageImpl$PageIterator.computeNext(PageImpl.java:72)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)

Any additional information below

This issue occurs on a few tables in Prod which I am not able to replicate in Dev.
The rpc call has the partition type as null but the TimePartitioning object in the response pb is not null which is leading to an NPE at this point.

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/StandardTableDefinition.java

Line 276 in b539f95

builder.setTimePartitioning(TimePartitioning.fromPb(tablePb.getTimePartitioning()));

[pmakani]

@chetanmeda I tested it many times but still not able to reproduce that error,
does this happening with every partition table or specific table ? if yes, could you please provide response pb ?

[chetanmeda]

@pmakani A table created with the API google-api-services-bigquery does not define partition type as an Enum. And the TimePartitioning Type need not be "DAY" and can be null as well.

Try creating a table with the below code.

Table tablePb = new Table();

    `Table tablePb = new Table();

    tablePb.setType("TABLE");


    TableSchema tableSchemaPb = new TableSchema();
    TableFieldSchema tableFieldSchema1 = new TableFieldSchema();
    tableFieldSchema1.setName("stringField");
    tableFieldSchema1.setType("STRING");
    TableFieldSchema tableFieldSchema2 = new TableFieldSchema();
    tableFieldSchema2.setName("booleanField");
    tableFieldSchema2.setType("BOOL");
    TableFieldSchema tableFieldSchema3 = new TableFieldSchema();
    tableFieldSchema3.setName("timestampField");
    tableFieldSchema3.setType("TIMESTAMP");
    List<TableFieldSchema> tableFieldSchemaList= new ArrayList<TableFieldSchema>();
    tableFieldSchemaList.add(tableFieldSchema1);
    tableFieldSchemaList.add(tableFieldSchema2);
    tableFieldSchemaList.add(tableFieldSchema3);
    tableSchemaPb.setFields(tableFieldSchemaList);
    tablePb.setSchema(tableSchemaPb);
    TimePartitioning timePartitioning = new TimePartitioning();
    timePartitioning.setField("timestampField");
    timePartitioning.setType(null);

    tablePb.setTimePartitioning(timePartitioning);

    TableReference tableReference = new TableReference();
    tableReference.setProjectId("bigquerty-poc");
    tableReference.setDatasetId("some_data_set");
    tableReference.setTableId("try_1_part");
    tablePb.setTableReference(tableReference);



     try {

         bigquery
                 .tables()
                 .insert(tableReference.getProjectId(), tableReference.getDatasetId(), tablePb)
                 .execute();
     }
     catch (Exception e){
         e.printStackTrace();
     }

}`

bigquery above is an instance of com.google.api.services.bigquery.Bigquery.

Table created using code like above ( timePartitioning.setType(null) ) is a valid partitioned table and works perfectly. But the new API has Timepartitioning.Type as Enum which causes the exception.

As per the below Type is a enum

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/TimePartitioning.java

Line 43 in b539f95

public enum Type {

So we have to change the below line

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/TimePartitioning.java

Line 145 in b539f95

return newBuilder(Type.valueOf(partitioningPb.getType()))

to
newBuilder(Type.valueOf(firstNonNull(partitioningPb.getType(),Type.DAY))

[shollyman]

So, this is an odd behavior, and the backend is allowing nulls but probably should not. I've filed an internal issue (155198865) to see if this can be corrected or documented in the backend.

The important thing to note here is that the behavior (setting a null partitioning type) is undefined, and shouldn't be used. We can consider allowing it through the manual library, but I'd like to confirm what the intent is internally (document or restrict) for unset types before we address this in this library.

[chetanmeda]

@shollyman The backend should allow nulls because setting TimePartitioning type as "DAY" is not mandatory requirement. So In the above table creation code, omitting
timePartitioning.setType(null);
will cause the same issue. So it was more of a non-mandatory internally assigned parameter being made mandatory on the new library. It is always a "DAY" internally. So it need not be set explicitly.

The new library can expect it to be mandatorily assigned, but cannot always expect a value to be returned for TimePartitioning.Type because most of the tables created using java library did not need it. More of what the client java library is expecting as mandatory behaviour should become non-mandatory behaviour. This issue is causing the new client library to be unusable for us and we have to revert to using google-api-services-bigquery directly.

Can this change for TimePartitioning.Type to be able to accept null Types while reading a Table be done so that we can continue using google-cloud-bigquery.

[spauldurai]

@pmakani - Thanks for the PR , Let me know it will get merged and when will we get the latest jar out if it

[spauldurai]

@stephaniewang526 - Any idea when we can expect the latest jar ? https://repo.maven.apache.org/maven2/com/google/cloud/google-cloud-bigquery/1.111.1/

[stephaniewang526]

It should be release on Monday, 5/4.

[spauldurai]

@stephaniewang526 - Thanks for the update

[spauldurai]

@stephaniewang526 - I could see latest jars https://repo.maven.apache.org/maven2/com/google/cloud/google-cloud-bigquery/1.111.3/

[stephaniewang526]

https://repo1.maven.org/maven2/com/google/cloud/google-cloud-bigquery/ please find the latest jar, 1.111.3 here.