Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSError: too many files open on linux #332

Closed
Tracked by #339
goodboy opened this issue Jun 6, 2022 · 1 comment
Closed
Tracked by #339

OSError: too many files open on linux #332

goodboy opened this issue Jun 6, 2022 · 1 comment
Labels
bug guille broke it prolly data-layer real-time and historical data processing and storage

Comments

@goodboy
Copy link
Contributor

goodboy commented Jun 6, 2022

It's odd because i've seen this from 2 diff sources in the #302 history but not sure either are legit.

Main source of issue seems to be that we're loading boat loads of data and one of 2 sys level calls trigger this:

  • socket re-connection of some sort to the FSP engine inside the trio streams apis
  • our ShmArray's usage of stdlib's SharedMemory causes it when attaching (not creating) to an existing buffer repeatedly (again something the fsp engine will do on task respawn-resyncs when a new history frame is loaded.

The next time someone sees this please put in the whole traceback. I've stuck some preventative code in #302 but it didn't seem to do much for me iirc.

Another thing we cam try for the shm case is a busy loop around the non-create call:

key = 'blah'
 shm = SharedMemory(
                name=key,
                create=True,
            )
 while True:
    shm = SharedMemory(
                name=key,
                create=False,
            )

and see if that triggers it, in which case it's likely a lower level bug we can report?

@goodboy goodboy added bug guille broke it prolly data-layer real-time and historical data processing and storage labels Jun 6, 2022
@goodboy goodboy mentioned this issue Jun 6, 2022
7 tasks
goodboy added a commit that referenced this issue Jul 19, 2022
Minimize calling `.data._shmarray.attach_shm_array()` as much as is
possible to avoid the crash from #332. This is the suggested hack from
issue #359.

Resolves #359
goodboy added a commit that referenced this issue Jul 19, 2022
Minimize calling `.data._shmarray.attach_shm_array()` as much as is
possible to avoid the crash from #332. This is the suggested hack from
issue #359.

Resolves #359
@goodboy
Copy link
Contributor Author

goodboy commented May 26, 2023

Seems to be resolved, haven't seen it in ages so closing.

@goodboy goodboy closed this as completed May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug guille broke it prolly data-layer real-time and historical data processing and storage
Projects
None yet
Development

No branches or pull requests

1 participant