ARROW-10121: [C++] Fix emission of new dictionaries in IPC writer

[pitrou]

When a dictionary changes from the previous batch, it is emitted again in the IPC stream.
If this happens when writing the IPC file format, an error is returned.

[pitrou]

@wesm This is a first draft. It is functional (i.e. fixes the underlying issue) but doesn't try to detect deltas.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-10121

[wesm]

I assume this will populate last_dictionaries_ entries even when the length of the dictionary is zero? I recall there was some discussion about empty dictionaries (and whether they need to be written at all) at the start of a stream

[pitrou]

Ah, we can skip zero-length dictionaries explicitly, but won't it confuse the IPC reader?

[wesm]

I'm not sure if skipped dictionaries will work, but they are permitted by the format, so it should probably be investigated in a follow up issue

[pitrou]

I still need to add tests for this (I'll do that once #8309 is merged). Detecting and writing delta dictionaries is not implemented, but that's a feature, not a bug, so it may also wait for another PR.
(also, it will have to be disabled by default, for compatibility with other IPC readers)

[pitrou]

@wesm Do you want to review this again? Otherwise perhaps @bkietz ?

[wesm]

I can take a final look this morning.

[wesm]

I'm not sure if skipped dictionaries will work, but they are permitted by the format, so it should probably be investigated in a follow up issue