Segfault when detaching after deleting shadow on Classic

[Noremos]

The easiest way to reproduce it is fbtest

{
'id': 'bug',
'qmid': None,
'title': 'Deleting a shadow',
'description': '',
'min_versions': '2.6',
'versions': [
{
 'firebird_version': '3.0',
 'platform': 'All',
 'page_size': '4096',
 'test_type': 'python',
 'test_script': '''
import os

Shadow1Path = os.path.join(context['temp_directory'], 'DDL_Shadow.shd')

script="""
CREATE SHADOW 1 MANUAL '%s';
commit;

DROP SHADOW 1;
""" %(Shadow1Path)
runProgram('isql', [dsn, '-user', user_name, '-password', user_password, '-q'], script)

 ''',
 'expected_stdout': """ 
 """,
 'expected_stderr': """ 
 """
}
]
}

The stack:

Jrd::Attachment::createPool(Jrd::Attachment * const this) (src\jrd\Attachment.cpp:150)
CMP_compile(Jrd::thread_db * tdbb, const UCHAR * blr, ULONG blrLength, bool internalFlag, ULONG dbginfoLength, const UCHAR * dbginfo) (src\jrd\cmp.cpp:155)
CMP_compile_request(Jrd::thread_db * tdbb, const UCHAR * blr, ULONG blrLength, bool internalFlag) (src\jrd\cmp.cpp:216)
Jrd::AutoRequest::compile(Jrd::AutoRequest * const this, Jrd::thread_db * tdbb, const UCHAR * blr, ULONG blrLength) (src\jrd\exe_proto.h:171)
MET_get_shadow_files(Jrd::thread_db * tdbb, bool delete_files) (..\Debug\jrd\met.cpp:6539)
SDW_get_shadows(Jrd::thread_db * tdbb) (src\jrd\sdw.cpp:584)
CCH_fetch_lock(Jrd::thread_db * tdbb, Jrd::WIN * window, int lock_type, int wait, SCHAR page_type) (src\jrd\cch.cpp:869)
CCH_fetch(Jrd::thread_db * tdbb, Jrd::WIN * window, int lock_type, SCHAR page_type, int wait, const bool read_shadow) (src\jrd\cch.cpp:802)
CCH_FETCH(Jrd::thread_db * tdbb, Jrd::win * window, USHORT lock_type, SCHAR page_type) (src\jrd\cch_proto.h:84)
TRA_update_counters(Jrd::thread_db * tdbb, Jrd::Database * dbb) (src\jrd\tra.cpp:944)
JRD_shutdown_database(Jrd::Database * dbb, const unsigned int flags) (src\jrd\jrd.cpp:8155)
purge_attachment(Jrd::thread_db * tdbb, Jrd::StableAttachmentPart * sAtt, unsigned int flags) (src\jrd\jrd.cpp:8628)
Jrd::JAttachment::freeEngineData(Jrd::JAttachment * const this, Firebird::CheckStatusWrapper * user_status, bool forceFree) (src\jrd\jrd.cpp:3477)
Jrd::JAttachment::internalDetach(Jrd::JAttachment * const this, Firebird::CheckStatusWrapper * user_status) (src\jrd\jrd.cpp:3414)
Jrd::JAttachment::detach(Jrd::JAttachment * const this, Firebird::CheckStatusWrapper * user_status) (src\jrd\jrd.cpp:3426)
Firebird::IAttachmentBaseImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher(Firebird::IAttachment * self, Firebird::IStatus * status) (src\include\firebird\IdlFbInterfaces.h:13379)
Firebird::IAttachment::detach<Firebird::CheckStatusWrapper>(Firebird::IAttachment * const this, Firebird::CheckStatusWrapper * status) (src\include\firebird\IdlFbInterfaces.h:2852)
Why::YAttachment::<lambda()>::operator()(void) const(const Why::YAttachment::<lambda()> * const __closure) (src\yvalve\why.cpp:6029)
std::_Function_handler<void(), Why::YAttachment::detach(Firebird::CheckStatusWrapper*)::<lambda()> >::_M_invoke(const std::_Any_data &)(const std::_Any_data & __functor) (\usr\include\c++\8\bits\std_function.h:297)
std::function<void ()>::operator()() const(const std::function<void()> * const this) (\usr\include\c++\8\bits\std_function.h:687)
Why::done<Why::YAttachment>(Firebird::CheckStatusWrapper*, Why::YEntry<Why::YAttachment>&, Why::YAttachment*, std::function<void ()>, std::function<void ()>)(Firebird::CheckStatusWrapper * status, Why::YEntry<Why::YAttachment> & entry, Why::YAttachment * y, std::function<void()> newClose, std::function<void()> oldClose) (src\yvalve\why.cpp:1355)
Why::YAttachment::detach(Why::YAttachment * const this, Firebird::CheckStatusWrapper * status) (src\yvalve\why.cpp:6028)
Firebird::IAttachmentBaseImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher(Firebird::IAttachment * self, Firebird::IStatus * status) (src\include\firebird\IdlFbInterfaces.h:13379)
Firebird::IAttachment::detach<Firebird::CheckStatusWrapper>(Firebird::IAttachment * const this, Firebird::CheckStatusWrapper * status) (src\include\firebird\IdlFbInterfaces.h:2852)
rem_port::end_database(rem_port * const this, PACKET * sendL) (src\remote\server\server.cpp:3347)
process_packet(rem_port * port, PACKET * sendL, PACKET * receive, rem_port ** result) (src\remote\server\server.cpp:5244)
loopThread() (src\remote\server\server.cpp:7138)
(anonymous namespace)::ThreadArgs::run((anonymous namespace)::ThreadArgs * const this) (src\common\ThreadStart.cpp:78)
(anonymous namespace)::threadStart(void * arg) (src\common\ThreadStart.cpp:94)
libpthread.so.0!start_thread (Unknown Source:0)
libc.so.6!clone (Unknown Source:0)

[AlexPeshkoff]

What FB version?

[Noremos]

Firebird 3.0
But probably all versions are affected

[Noremos]

I'm sorry, the stack related to a different version. Here is the stack for Firebird 3.0:

libEngine12.so!Jrd::Attachment::createPool(Jrd::Attachment * const this) (src\jrd\Attachment.cpp:115)
libEngine12.so!CMP_compile2(Jrd::thread_db * tdbb, const UCHAR * blr, ULONG blr_length, bool internal_flag, ULONG dbginfo_length, const UCHAR * dbginfo) (src\jrd\cmp.cpp:167)
libEngine12.so!Jrd::AutoRequest::compile(Jrd::AutoRequest * const this, Jrd::thread_db * tdbb, const UCHAR * blr, ULONG blrLength) (src\jrd\exe_proto.h:170)
libEngine12.so!MET_get_shadow_files(Jrd::thread_db * tdbb, bool delete_files) (temp\Debug\jrd\met.cpp:5987)
libEngine12.so!SDW_get_shadows(Jrd::thread_db * tdbb) (src\jrd\sdw.cpp:584)
libEngine12.so!CCH_fetch_lock(Jrd::thread_db * tdbb, Jrd::WIN * window, int lock_type, int wait, SCHAR page_type) (src\jrd\cch.cpp:797)
libEngine12.so!CCH_fetch(Jrd::thread_db * tdbb, Jrd::WIN * window, int lock_type, SCHAR page_type, int wait, const bool read_shadow) (src\jrd\cch.cpp:730)
libEngine12.so!CCH_FETCH(Jrd::thread_db * tdbb, Jrd::win * window, USHORT lock_type, SCHAR page_type) (src\jrd\cch_proto.h:84)
libEngine12.so!TRA_update_counters(Jrd::thread_db * tdbb, Jrd::Database * dbb) (src\jrd\tra.cpp:832)
libEngine12.so!JRD_shutdown_database(Jrd::Database * dbb, const unsigned int flags) (src\jrd\jrd.cpp:6711)
libEngine12.so!purge_attachment(Jrd::thread_db * tdbb, Jrd::StableAttachmentPart * sAtt, unsigned int flags) (src\jrd\jrd.cpp:7162)
libEngine12.so!Jrd::JAttachment::freeEngineData(Jrd::JAttachment * const this, Firebird::CheckStatusWrapper * user_status, bool forceFree) (src\jrd\jrd.cpp:2953)
libEngine12.so!Jrd::JAttachment::detach(Jrd::JAttachment * const this, Firebird::CheckStatusWrapper * user_status) (src\jrd\jrd.cpp:2913)
libEngine12.so!Firebird::IAttachmentBaseImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher(Firebird::IAttachment * self, Firebird::IStatus * status) (src\include\firebird\IdlFbInterfaces.h:9081)
libfbclient.so.2!Firebird::IAttachment::detach<Firebird::CheckStatusWrapper>(Firebird::IAttachment * const this, Firebird::CheckStatusWrapper * status) (src\include\firebird\IdlFbInterfaces.h:1964)
libfbclient.so.2!Why::YAttachment::detach(Why::YAttachment * const this, Firebird::CheckStatusWrapper * status) (src\yvalve\why.cpp:5464)
libfbclient.so.2!Firebird::IAttachmentBaseImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher(Firebird::IAttachment * self, Firebird::IStatus * status) (src\include\firebird\IdlFbInterfaces.h:9081)
Firebird::IAttachment::detach<Firebird::CheckStatusWrapper>(Firebird::IAttachment * const this, Firebird::CheckStatusWrapper * status) (src\include\firebird\IdlFbInterfaces.h:1964)
rem_port::end_database(rem_port * const this, PACKET * sendL) (src\remote\server\server.cpp:3075)
process_packet(rem_port * port, PACKET * sendL, PACKET * receive, rem_port ** result) (src\remote\server\server.cpp:4481)
loopThread() (src\remote\server\server.cpp:6032)
(anonymous namespace)::ThreadArgs::run((anonymous namespace)::ThreadArgs * const this) (src\common\ThreadStart.cpp:78)
(anonymous namespace)::threadStart(void * arg) (src\common\ThreadStart.cpp:94)
libpthread.so.0!start_thread (Unknown Source:0)
libc.so.6!clone (Unknown Source:0)](url)

[AlexPeshkoff]

Did not reproduce.
I've tried with current B3_0_Release, both debug & release builds. No segfaults.

[Noremos]

Did not reproduce. I've tried with current B3_0_Release, both debug & release builds. No segfaults.

Weird, I get this in the log on Classic with the test.

localhost.localdomain	Tue Mar 28 10:38:44 2023
	REMOTE INTERFACE/gds__detach: Unsuccesful detach from database.
	Uncommitted work may have been lost.
	Error reading data from the connection.

As I see it, the problem is that fbt transaction disconnects at the same time as the commit of the isql transaction.
The Lock(tdbb, key_length, LCK_shadow, dbb, blocking_ast_shadowing) sets the flag DBB_get_shadows, and when the fbt transaction enters SDW_get_shadows, which requires an attachment (the fbt attachment has already been deleted at this moment)

2023-03-28T10:09:27.4490 (1124653:0x7fffe0dfad50) ATTACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_3, SYSDBA:NONE, NONE, TCPv6:::1/51452)
	/usr/bin/python2.7:1124591

2023-03-28T10:09:27.6340 (1124664:0x7fffe0dfad50) ATTACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_4, SYSDBA:NONE, NONE, <internal>)

2023-03-28T10:09:27.6580 (1124664:0x7fffe0dfad50) DETACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_4, SYSDBA:NONE, NONE, <internal>)

2023-03-28T10:09:27.7430 (1124664:0x7fffe0dfad50) ATTACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_5, SYSDBA:NONE, NONE, TCPv6:::1/51460)
	isql:1124661

2023-03-28T10:09:37.0380 (1124664:0x7fffe0dfad50) DETACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_5, SYSDBA:NONE, NONE, TCPv6:::1/51460)
	isql:1124661

2023-03-28T10:09:37.0960 (1124653:0x7fffe0dfad50) DETACH_DATABASE
	fbt-repository/tmp/test.fdb (ATT_3, SYSDBA:NONE, NONE, TCPv6:::1/51452)
	/usr/bin/python2.7:1124591

----------- Segfault -----------

[AlexPeshkoff]

OK, when we deal with races it's not guranteed to be reproduced on different HW. Will have to use long way to fix.
Please send me core dump of a failure and FB binaries you've used (reference like 3.0.10 official release is also OK).

[Noremos]

HEAD
commit 7fb0a59 (HEAD -> B3_0_Release, origin/B3_0_Release)
Author: Dmitry Yemanov [email protected]
Date: Tue Mar 28 07:57:31 2023 +0300

Fixed #7517: Successful compiling of procedure with wrong PLAN(s) used by some of its statement(s)

Dumbs: https://drive.google.com/drive/folders/1Elfi20dq5XmJNoD7ByhNecLOI1YBGRpE?usp=sharing
core_a_flag is a dump at the moment when fbt connection accepts the flag;
core_b_commit is a dump at the moment when isql connection commits (and sends the flag);
core_a_crash is a dump at the moment when fbt connection segfaults.

[AlexPeshkoff]

May be I was not enough explicit - I need binaries, src is not enough to analyze core dump.

[Noremos]

Sorry, missed that one. I've uploaded the binaries at the Disk (the same link)

[AlexPeshkoff]

shadowsShutdown.patch.gz
Please try with this patch.

[Noremos]

The fix has helped. Thanks