Fix the IOStats computation

[Thejas-bhat]

The newly introduced stats:
num_bytes_read_at_query_time - computes the bytes read from disk while query
num_bytes_indexed_after_analysis track the bytes read - computes the bytes written to disk while indexing

are mainly used to track the disk utilisation for the current index. Currently, the num_bytes_indexed_after_analysis (now changing to num_bytes_written_at_index_time) considered only the total bytes of the tokens after the analysis of a field's content. However, a user can further store a field, enable doc values to be stored, enable location information of a term to be stored or even include the field's content in _all field. All these options incur additional cost in terms of disk utilisation and have to be considered in the stats. The PRs #119 and #125 and the current one aim to achieve these changes.

[abhinavdangeti]

I would recommend against adding this flag in scorch_segment_api.
Instead make it a scorch config option and apply this only while -

• Estimating usage within bleve
• Reporting usage reported by zap again from bleve

Leave zap's computation on always.

[abhinavdangeti]

Unit test TestBytesWritten is flaky in the results it expects -

[11:56:02] AD: ~/Documents/go/src/github.com/blevesearch/bleve $ go test -run=TestBytesWritten -race
PASS
ok  	github.com/blevesearch/bleve/v2	2.217s
[11:56:18] AD: ~/Documents/go/src/github.com/blevesearch/bleve $ go test -run=TestBytesWritten
--- FAIL: TestBytesWritten (0.49s)
    index_test.go:314: expected bytes written is 55471, got 92127
FAIL
exit status 1
FAIL	github.com/blevesearch/bleve/v2	0.762s
[11:56:24] AD: ~/Documents/go/src/github.com/blevesearch/bleve $ go test -run=TestBytesWritten
--- FAIL: TestBytesWritten (0.83s)
    index_test.go:338: expected bytes written is 62347, got 117818
FAIL
exit status 1
FAIL	github.com/blevesearch/bleve/v2	1.068s
[11:56:27] AD: ~/Documents/go/src/github.com/blevesearch/bleve $ go test -run=TestBytesWritten
--- FAIL: TestBytesWritten (1.08s)
    index_test.go:363: expected bytes written is 102602, got 164949
FAIL
exit status 1
FAIL	github.com/blevesearch/bleve/v2	1.322s
[11:56:31] AD: ~/Documents/go/src/github.com/blevesearch/bleve $ go test -run=TestBytesWritten
PASS
ok  	github.com/blevesearch/bleve/v2	1.326s

[Thejas-bhat]

Turns out,TestBytesWritten behaviour was flaky mainly because of a memory leak issue in zapx which has been addressed in zapx PR #125

[sreekanth-cb]

clean up of this path is missing?
Also, should we need to defer it as we could try it right after the index close? (doesn't matter much)
It could also be written like a table-driven test since the variations across run are only a few values.

[Thejas-bhat]

Done

[abhinavdangeti]

Insert a new line at the end of a method.

[abhinavdangeti]

Ditto.