Prevent SimpleDirectoryReader from excessive memory consumption

[AstraBert]

Description

With this PR, we prevent SimpleDirectoryReader from loading all the files within a list and then limiting their number (which causes memory exhaustion in resource-intense edge use cases), by implementing the limitation within the loop that add files to the all_files list.

Fixes this Huntr issue

[logan-markewich]

Does this actually fix the issue? I think we still load all file-refs into memory 🤔

Maybe a better fix is something like

for _ref in os.glob(...):
  if len(all_files) > limit:
    break
  # Manually check the ref...
  ...

[logan-markewich]

tbh it would be nice if we could pass the limit directly into glob 🤔 But I didn't see a limit in their docs

[AstraBert]

We could use something as:

import os 

counter = 0
for root, dirs, files in self.fs.walk():
    for file in files:
        counter += 1
        if counter > limit:
            break
        refs.append(os.path.join(root, file))

[logan-markewich]

ah fs.walk is a nice approach yes

[logan-markewich]

Since these two codepaths are the same, we can reduce code duplication by just change the args I think?

depth = None if self.recursive else 1

(I'm actually also hesitant to pass in None to begin with (we should have some upper bound like 1000 ?)