Skip to content

examples use WriteOptions incorrectly #79

@nickvecelliooo

Description

@nickvecelliooo

Specifications

  • Client Version: influxdb3-python==0.3.6
  • InfluxDB Version: 3.0 serverless
  • Platform: mac

Code sample to reproduce problem

Any file in examples that provides the following code

wco = write_client_options(success_callback=callback.success,
                          error_callback=callback.error,
                          retry_callback=callback.retry,
                          WriteOptions=write_options 
                        )

with InfluxDBClient3.InfluxDBClient3(token=token,
                                     host=url,
                                     org=org,
                                     database=database, enable_gzip=True, write_client_options=wco) as _client:

Examples:
https://github.com/InfluxCommunity/influxdb3-python/blob/main/Examples/batching_example.py
https://github.com/InfluxCommunity/influxdb3-python/blob/main/Examples/pokemon-trainer/write-batching.py
https://github.com/InfluxCommunity/influxdb3-python/blob/main/Examples/pokemon-trainer/write-batching-flight-calloptions.py
https://github.com/InfluxCommunity/influxdb3-python/blob/main/Examples/batching_example.py

The easiest way to observe this is by modifying the provided callback to measure the amount of data that is being sent:

class BatchingCallback(object):

    def __init__(self):
        self.lines = 0

    def success(self, conf, data: str):
        lines = len(data.decode().split('\n'))
        self.lines += lines
        print(f"Written batch: {conf}; {lines} lines; {self.lines} total")

    def error(self, conf, data: str, exception: InfluxDBError):
        print(f"Cannot write batch: {conf}, data: {data} due: {exception}")

    def retry(self, conf, data: str, exception: InfluxDBError):
        print(f"Retryable error occurs for batch: {conf}, data: {data} retry: {exception}")

Expected behavior

Documented examples of batch sizing should send the batch sizes as expected.

Actual behavior

batch_size defaults back to 1000

Additional info

This appears to be a result of passing in keyword arguments in the form of **self._write_client_options, where the WriteAPI would expect write_options instead of the provided key of WriteOptions.

This can be seen in the output from the callback.
This is with the following code in place:

write_options = WriteOptions(batch_size=6_000,
                                        flush_interval=10_000,
                                        jitter_interval=2_000,
                                        retry_interval=5_000,
                                        max_retries=5,
                                        max_retry_delay=30_000,
                                        exponential_base=2)

wco = write_client_options(success_callback=callback.success,
                          error_callback=callback.error,
                          retry_callback=callback.retry,
                          WriteOptions=write_options 
                        )
$ python3 code/testing_client.py
Written batch: ('mig_test', '', 's'); 1000 lines; 1000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 2000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 3000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 4000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 5000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 6000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 7000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 8000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 9000 total
Written batch: ('mig_test', '', 's'); 1000 lines; 10000 total

Compared to the following:

write_options = WriteOptions(batch_size=6_000,
                                        flush_interval=10_000,
                                        jitter_interval=2_000,
                                        retry_interval=5_000,
                                        max_retries=5,
                                        max_retry_delay=30_000,
                                        exponential_base=2)

wco = write_client_options(success_callback=callback.success,
                          error_callback=callback.error,
                          retry_callback=callback.retry,
                          write_options=write_options 
                        )
$ python3 code/testing_client.py
Written batch: ('mig_test', '', 's'); 6000 lines; 6000 total
Written batch: ('mig_test', '', 's'); 4000 lines; 10000 total

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions