Models meta data cache fails to save and the app is DOWN!!!

[Shay12tg]

If the model meta data cache fails to save, the app crashes... WTF?? :(
Over time it happens a lot and for me it's an issue... please help me find a solution for a stable app.

I use memcache to save metadata cache with the provided adapter ( Phalcon\Mvc\Model\Metadata\Memcache ).
I installed a simple local memcached instance.
it works.... until it went to production and it started failing.
So I added a memcached health check (connect and save random data) to the health check file. but no luck... still getting these errors about once a week and the app goes down for a few minutes/hours.

The app should not fail in this case (imo). it should catch the error, throw a warning and fallback to memory or something...

To recreate, use a memcached metadata adapter with no server.. it's about the same..
The app doesn't have to fail if the cache fails...

Error from cloudwatch:

{
    "code": 0,
    "message": "Failed storing data in memcached",
    "file": "phalcon/cache/backend/memcache.zep",
    "line": 241,
    "trace": "
#0 [internal function]: Phalcon\\Cache\\Backend\\Memcache->save('meta-files-File...', Array)
#1 [internal function]: Phalcon\\Mvc\\Model\\MetaData\\Memcache->write('meta-files-File...', Array)
#2 [internal function]: Phalcon\\Mvc\\Model\\MetaData->_initialize(Object(Files), 'files-Files', 'Files', '')
#3 [internal function]: Phalcon\\Mvc\\Model\\MetaData->writeMetaDataIndex(Object(Files), 10, Array)
#4 [internal function]: Phalcon\\Mvc\\Model\\MetaData->setAutomaticCreateAttributes(Object(Files), Array)
#5 [internal function]: Phalcon\\Mvc\\Model->skipAttributesOnCreate(Array)
#6 /var/www/api/app/models/Files.php(46): Phalcon\\Mvc\\Model->skipAttributes(Array)
#7 [internal function]: Files->initialize()
#8 [internal function]: Phalcon\\Mvc\\Model\\Manager->initialize(Object(Files))
#9 [internal function]: Phalcon\\Mvc\\Model->__construct()
#10 [internal function]: Phalcon\\Mvc\\Model::_invokeFinder('findFirstByID', Array)
#11 /var/www/api/app/controllers/FilesController.php(284): Phalcon\\Mvc\\Model::__callStatic('findFirstByID', Array)
#12 [internal function]: FilesController->previewAction('1352')
#13 [internal function]: Phalcon\\Dispatcher->callActionMethod(Object(FilesController), 'previewAction', Array)\n#14 [internal function]: Phalcon\\Dispatcher->dispatch()
#15 /var/www/api/index.php(44): Phalcon\\Mvc\\Application->handle()
#16 {main}
"
}

Details

• Phalcon version: 3.4.0
• PHP Version: 7.2.7
• Operating System: Ubuntu 16.04 x64
• Installation type: package manager
• Zephir version (if any): 0.10.10-d1b4cc68d9
• Server: Apache
• Other related info (Database, table schema): MySQL..

[sergeyklay]

@Shay12tg We'll need a script to reproduce. As small as possible please.

[Shay12tg]

@sergeyklay thanks for the quick response.
I get a different error on production (fails to save, not connect) but I can't reproduce it.
However if it fails to connect it still produces a fatal error. warning is enough imo. cache doesn't have to break the app.

<?php

$di = new Phalcon\DI\FactoryDefault();

$di->set('db', function() use ($config) {
	return new Phalcon\Db\Adapter\Pdo\Mysql([
		'host' => 'localhost',
		'dbname' => 'test',
		'username'	=> 'root',
		'password'	=> ''
	]);
});

$di->set('modelsMetadata', function(){
	return new Phalcon\Mvc\Model\Metadata\Memcache([
		'port' => 1337
	]);
});

class Files extends \Phalcon\Mvc\Model
{
	public $id;
	public $path;
	public $code;

	public function initialize()
	{
		$this->skipAttributes(['code']);
	}

	public function getSource()
	{
		return 'Files';
	}
}

echo Files::findFirstById(1) ? 'found' : 'notfound';

[Shay12tg]

Without DB:

<?php

$di = new Phalcon\DI\FactoryDefault();

$di->set('modelsMetadata', function(){
	return new Phalcon\Mvc\Model\Metadata\Memcache([
		'port' => 1337
	]);
});
$data = $di->get('modelsMetadata')->read('a');

This case should fail, but the case above should not. the app can continue without the cache or fallback to memory

[Tanami]

The app should not fail in this case (imo). it should catch the error, throw a warning and fallback to memory or something...

I think what you mean is "I need to write some code to check if the cache backend is up and if not fall back to a different cache method".

[Jurigag]

To be honest in our production application we also have this problem with meta data memcache adapter, sometimes we even have some weird things written to metadata. It happens more often if we have enabled persistent connection. On other hand we use apc/apcu on fpm(don't ask me why) and it works perfectly fine.

[niden]

Relates to #13439

[niden]

The Memcache adapter has been deprecated so this is no longer relevant (at least for this particular use case).

However I have added an exception to the MetaData::write() so that if the data cannot be stored in any of the available adapters for v4, an exception will be thrown to inform the user.

[niden]

The relevant commit is here:

https://github.com/niden/cphalcon/commit/7e6be9bb86e4eb7eb6e90a3f8054230e58e6dccc

[niden]

Reopening this after discussion with @CameronHall

If the metadata cache dies it should not kill the whole application.

• Introduce a method to throw or not throw an exception on write fail something like

$metadata->doNotRaiseExceptionOnWriteFailOtherwiseCameronWillThrowAFit(true);

that will not throw an exception if the write fails but rather issue a warning in the logs for the devops team.

[sergeyklay]

@niden Consider using globals:

cphalcon/config.json

Lines 35 to 112 in 9fae862

	"globals": {
	"db.escape_identifiers": {
	"type": "bool",
	"default": true
	},
	"db.force_casting": {
	"type": "bool",
	"default": false
	},
	"orm.parser_cache": {
	"type": "hash",
	"default": "NULL"
	},
	"orm.ast_cache": {
	"type": "hash",
	"default": "NULL"
	},
	"orm.cache_level": {
	"type": "int",
	"default": 3
	},
	"orm.unique_cache_id": {
	"type": "int",
	"default": 3
	},
	"orm.events": {
	"type": "bool",
	"default": true
	},
	"orm.virtual_foreign_keys": {
	"type": "bool",
	"default": true
	},
	"orm.column_renaming": {
	"type": "bool",
	"default": true
	},
	"orm.not_null_validations": {
	"type": "bool",
	"default": true
	},
	"orm.exception_on_failed_save": {
	"type": "bool",
	"default": false
	},
	"orm.enable_literals": {
	"type": "bool",
	"default": true
	},
	"orm.late_state_binding": {
	"type": "bool",
	"default": false
	},
	"orm.enable_implicit_joins": {
	"type": "bool",
	"default": true
	},
	"orm.cast_on_hydrate": {
	"type": "bool",
	"default": false
	},
	"orm.ignore_unknown_columns": {
	"type": "bool",
	"default": false
	},
	"orm.update_snapshot_on_save": {
	"type": "bool",
	"default": true
	},
	"orm.disable_assign_setters": {
	"type": "bool",
	"default": false
	},
	"orm.case_insensitive_column_map": {
	"type": "bool",
	"default": false
	}
	},

For more see: https://docs.zephir-lang.com/0.11/en/globals

[Jurigag]

I agree, better use global.

Also in some cases - metadata can't be saved because it can be actually too large - The maximum size of a value you can store in memcached is 1 megabyte there are cases that for some kind of huge model it can be bigger.

[niden]

Resolved #14112