ARROW-3408: [C++] Add CSV option to automatically attempt dict encoding

[pitrou]

This is tied to type inference, and only triggers on string or binary columns.
Each chunk is dict-encoded up to a certain cardinality, after which the whole
column falls back on plain encoding.

[pitrou]

Quick performance rundown:

• dict encoding is ~30% slower in single-thread mode than regular encoding
• dict encoding is as fast in multi-thread mode as regular encoding

Also, if auto dict encoding is enabled but max cardinality is reached in the first chunks, performance doesn't suffer

[github-actions]

https://issues.apache.org/jira/browse/ARROW-3408

[pitrou]

@wesm

[wesm]

This seems good to me, will be pretty valuable as far as reducing memory use.

Some high level questions (and I haven't read closely enough to determine the answers already)

• Are different converted chunks allowed to yield different dictionaries? Overall I would say there's little benefit to going to extra effort to have a "global" dictionary. This might be made configurable also
• Is the max cardinality enforced on a per-chunk basis or globally? How does this impact parallel conversions? In a worst case scenario you could imagine chunks having smallish dictionaries but the global dictionary may be larger than the threshold

[wesm]

In the future this could certainly be const T& to save on some typing below

[wesm]

Nice code-deduping in this file

[wesm]

Would it be possible to indicate to use dictionary encoding explicitly for a column (would probably have to be forced to yield int32 indices)? Can be follow up work

[pitrou]

Probably as a followup JIRA, since the configuration mechanism must be discussed.

[pitrou]

Are different converted chunks allowed to yield different dictionaries?

Yes. This is mandatory for parallel processing, actually.
(we could reconcile them at the end, though it might as well be a manual step by the user)

Is the max cardinality enforced on a per-chunk basis or globally?

On a per-chunk basis.

[pitrou]

Rebased and comments addressed.

[wesm]

On a per-chunk basis.

What will happen if one chunk overflows the limit but others do not? Will the dictionary-encoded chunks be casted to dense?

[pitrou]

Will the dictionary-encoded chunks be casted to dense?

Yes, all of them.
(concretely, they are not cast, conversion is done again)

[wesm]

Got it, thanks, wasn't sure if there were tests specifically about this but I trust you =)

+1