Importing yf fails if ~/.cache directory doesn't exist

[blackary]

Describe bug

Even though yf has a .set_tz_cache_location, simply importing yfinance fail if ~/.cache directory doesn't exist

Simply importing yf should not have side-effects like creating a database, and especially should not fail before there is a chance to overwrite the cache directory.

This is particularly problematic when it's not possible to create the .cache directory as the app is being deployed to a 3rd-party system.

Simple code that reproduces your problem

# Use a non-existing directory for ease of testing
import appdirs as ad

ad.user_cache_dir = lambda *args: ".cache"

# If the ad.user_cache_dir directory doesn't exist, this fails
import yfinance as yf

# This line is never reached
yf.set_tz_cache_location(".cache")

Debug log

python test.py
Traceback (most recent call last):
  File "/private/tmp/repos/yf/test.py", line 5, in <module>
    import yfinance as yf  # noqa
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/__init__.py", line 23, in <module>
    from .ticker import Ticker
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/ticker.py", line 29, in <module>
    from .base import TickerBase
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/base.py", line 38, in <module>
    from . import utils
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/utils.py", line 991, in <module>
    class _KV(_peewee.Model):
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/utils.py", line 995, in _KV
    class Meta:
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/utils.py", line 996, in Meta
    database = _DBManager.get_database()
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/utils.py", line 953, in get_database
    cls._initialise()
  File "/private/tmp/repos/yf/.direnv/python-3.10/lib/python3.10/site-packages/yfinance/utils.py", line 972, in _initialise
    _os.mkdir(cls._cache_dir)
FileNotFoundError: [Errno 2] No such file or directory: '.cache/py-yfinance'

Bad data proof

No response

yfinance version

0.2.30

Python version

3.10.3

Operating system

OSX

[ValueRaider]

Can you submit a fix? #1084. I only ask because you might discover other bugs specific to your situation.

[blackary]

I'll try to submit a fix when I get some time, but it might be a little while

[rickturner2001]

@blackary if toggling caching when appropriate is a fair solution for you guys, then I can take over

for reference, what I am suggesting is to check whether we are allowed to write in the root directory of the current file system, if that's the case then we can instantiate the database, else, we don't and we prevent further calls and references to _KVSTORE

[blackary]

@rickturner2001 That sounds like a great solution to me!

[wowo]

if you want to use yfinance on GCP Cloud Functions, here's solution for above problem:

import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"

import yfinance as yf

[labibk00]

if you want to use yfinance on GCP Cloud Functions, here's solution for above problem:

import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"

import yfinance as yf

I've just tried your suggestion but now I'm getting the error locally using firebase emulator:
FileNotFoundError: [WinError 3] The system cannot find the path specified: '/tmp\\py-yfinance'

[wowo]

if you want to use yfinance on GCP Cloud Functions, here's solution for above problem:

import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"

import yfinance as yf

I've just tried your suggestion but now I'm getting the error locally using firebase emulator: FileNotFoundError: [WinError 3] The system cannot find the path specified: '/tmp\\py-yfinance'

Probably due to fact you're on Windows.

[labibk00]

Probably due to fact you're on Windows.

...Yup lol. Did the whole thing in WSL Ubuntu, worked like a charm. Cheers!

In hindsight I should've realised right away given the error had a mix of forward and backward slashes lol.

[danielvinter]

Was having this problem too, though my application is only using quantstats, where yfinance is imported by quantstats.

Adding

import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"

At the top of my program fixed it, it now works on AWS Lambda again.

[ValueRaider]

Can people try branch #1705? I can release a beta via PIP if helps.

[genelll]

Was having this problem too, though my application is only using quantstats, where yfinance is imported by quantstats.

Adding

import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"

At the top of my program fixed it, it now works on AWS Lambda again.

Hi. I have the same issue on AWS Lambda, but your solution did not work for me. Is there anything else you are also doing to make this work? Without using appdirs I get this:

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/home/sbx_user1051/.cache/py-yfinance'

Using appdirs I get this:

Response
{
"errorMessage": "near "WITHOUT": syntax error",
"errorType": "OperationalError",
"requestId": "1966b8ad-c5f7-4822-acb4-22691fc1b85a",
"stackTrace": [
" File "/var/task/lambda_function.py", line 8, in lambda_handler\n hist = btcusd.history(period="1wk")\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/utils.py", line 114, in wrapper\n result = func(*args, **kwargs)\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/base.py", line 188, in history\n tz = self._get_ticker_tz(proxy, timeout)\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/base.py", line 1644, in _get_ticker_tz\n cache = utils.get_tz_cache()\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/utils.py", line 1040, in get_tz_cache\n return _TzCacheManager.get_tz()\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/utils.py", line 932, in get_tz\n cls._initialise()\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/utils.py", line 938, in _initialise\n cls._tz_cache = _TzCache()\n",
" File "/opt/python/lib/python3.11/site-packages/yfinance/utils.py", line 1004, in init\n db.create_tables([_KV])\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 3440, in create_tables\n model.create_table(**options)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 6924, in create_table\n cls._schema.create_all(safe, **options)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 6026, in create_all\n self.create_table(safe, **table_options)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 5877, in create_table\n self.database.execute(self._create_table(safe=safe, **options))\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 3259, in execute\n return self.execute_sql(sql, params)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 3249, in execute_sql\n with exception_wrapper:\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 3019, in exit\n reraise(new_type, new_type(exc_value, *exc_args), traceback)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 192, in reraise\n raise value.with_traceback(tb)\n",
" File "/opt/python/lib/python3.11/site-packages/peewee.py", line 3251, in execute_sql\n cursor.execute(sql, params or ())\n"
]
}

My full Lambda:

import pandas as pd
import appdirs as ad
ad.user_cache_dir = lambda *args: "/tmp"
import yfinance as yf

def lambda_handler(event, context):
btcusd = yf.Ticker("BTC-USD")
hist = btcusd.history(period="1wk")
print(hist)

---

[rickturner2001]

The implementation in the fix/tx-cache-init branch catches this error and would enable you to work on lambda, for now see #1708. I suspect it might be merged to main really soon, but for now, there's no way to catch it because of how the caching is initialized.

[ValueRaider]

I've released a beta to fix issue - should be fine but means I can rush out a release. Install with:
pip install --upgrade --pre yfinance

[elgalu]

I think this can be closed, re-tested (Ubuntu 22 + conda) and the issue is solved on "yfinance==0.2.31"