minor refactoring in MicroBatches class to prep for Spark3 streaming change

[SreeramGarlapati]

1. StreamingOffset stores file index as long - so changing MicroBatches.startFileIndex from int to long.
2. renaming MicroBatches.isStarting to MicroBatches.scanAllFiles - as isStarting is when it is used, in terms of the callers state - whereas, scanAllFiles is the actual what MicroBatches.generate should do.

this is part of effort #2611 & is copied from #2272.

[rdsr]

Generally looks to me, but do we know why we are not checking this in as part of #2272 ?

[SreeramGarlapati]

Generally looks to me, but do we know why we are not checking this in as part of #2272 ?

@rdsr - #2272 is spark2. Its a matter of time @XuQianJin-Stars - resolves the comments and pushes it. I am trying to push spark3 connector - for SalesForce needs & needing this change.

[RussellSpitzer]

I'm ok with merging this in, but let's try to wrap future patches in more logical and testable chunks.

[SreeramGarlapati]

I'm ok with merging this in, but let's try to wrap future patches in more logical and testable chunks.

ack. - will keep this in mind @RussellSpitzer.

[RussellSpitzer]

Thanks @SreeramGarlapati for making progress here, but like I mentioned please try to make changes a bit more grouped in the future and testable :)