Skip to content

Disable array support for the COM variant wrapper classes when built-in COM is disabled. #55756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eerhardt merged 10 commits into from
Jul 18, 2021
Merged

Disable array support for the COM variant wrapper classes when built-in COM is disabled. #55756

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5985f54
Disable array support for the COM variant wrapper classes when built-…
jkoritzinsky Jul 15, 2021
69d7d61
Fix config call.
jkoritzinsky Jul 15, 2021
10d3b3d
Fix one more location of wrapper class usage.
jkoritzinsky Jul 15, 2021
6f51a07
Fix method name.
jkoritzinsky Jul 15, 2021
e0e219f
Add trimming test.
jkoritzinsky Jul 15, 2021
19bca14
Fix AV in IsArrayOfWrappers
elinor-fung Jul 16, 2021
801d9c8
Fix trimming test
elinor-fung Jul 16, 2021
8cf985b
Apply suggestions from code review
elinor-fung Jul 16, 2021
ade2649
Set Trim=true
elinor-fung Jul 16, 2021
b1d7638
Add licence header
elinor-fung Jul 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 33 additions & 17 deletions src/coreclr/vm/olevariant.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,20 +329,24 @@ VARTYPE OleVariant::GetVarTypeForTypeHandle(TypeHandle type)
#endif

#ifdef FEATURE_COMINTEROP
if (CoreLibBinder::IsClass(pMT, CLASS__DISPATCH_WRAPPER))
return VT_DISPATCH;
if (CoreLibBinder::IsClass(pMT, CLASS__UNKNOWN_WRAPPER))
return VT_UNKNOWN;
if (CoreLibBinder::IsClass(pMT, CLASS__ERROR_WRAPPER))
return VT_ERROR;
if (CoreLibBinder::IsClass(pMT, CLASS__CURRENCY_WRAPPER))
return VT_CY;
if (CoreLibBinder::IsClass(pMT, CLASS__BSTR_WRAPPER))
return VT_BSTR;
// The wrapper types are only available when built-in COM is supported.
if (EEConfig::IsBuiltInCOMSupported())
{
if (CoreLibBinder::IsClass(pMT, CLASS__DISPATCH_WRAPPER))
return VT_DISPATCH;
if (CoreLibBinder::IsClass(pMT, CLASS__UNKNOWN_WRAPPER))
return VT_UNKNOWN;
if (CoreLibBinder::IsClass(pMT, CLASS__ERROR_WRAPPER))
Comment thread
elinor-fung marked this conversation as resolved.
return VT_ERROR;
if (CoreLibBinder::IsClass(pMT, CLASS__CURRENCY_WRAPPER))
return VT_CY;
if (CoreLibBinder::IsClass(pMT, CLASS__BSTR_WRAPPER))
return VT_BSTR;

// VariantWrappers cannot be stored in VARIANT's.
if (CoreLibBinder::IsClass(pMT, CLASS__VARIANT_WRAPPER))
COMPlusThrow(kArgumentException, IDS_EE_COM_UNSUPPORTED_SIG);
// VariantWrappers cannot be stored in VARIANT's.
if (CoreLibBinder::IsClass(pMT, CLASS__VARIANT_WRAPPER))
COMPlusThrow(kArgumentException, IDS_EE_COM_UNSUPPORTED_SIG);
}
#endif // FEATURE_COMINTEROP

@LakshanF LakshanF Jul 15, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a more surgical fix than I was thinking and thanks for doing this!

I assume the code under the other #FEATURE_COMINTEROP in this method will not cause a problem?

@jkoritzinsky jkoritzinsky Jul 15, 2021

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other ones won't cause problems.

We'll always have SafeHandle and CriticalHandle (which arrays of are always unsupported in interop).

And then the rest is based on RCW/CCW support having been used to create a wrapper, which is already blocked previously.

if (pMT->IsEnum())
Expand Down Expand Up @@ -4857,22 +4861,33 @@ BOOL OleVariant::IsArrayOfWrappers(BASEARRAYREF *pArray, BOOL *pbOfInterfaceWrap
}
CONTRACTL_END;

if (!EEConfig::IsBuiltInComSupported())
{
return FALSE;
}

TypeHandle hndElemType = (*pArray)->GetArrayElementTypeHandle();

if (!hndElemType.IsTypeDesc())
{
if (hndElemType == TypeHandle(CoreLibBinder::GetClass(CLASS__DISPATCH_WRAPPER)) ||
hndElemType == TypeHandle(CoreLibBinder::GetClass(CLASS__UNKNOWN_WRAPPER)))
{
*pbOfInterfaceWrappers = TRUE;
if (pbOfInterfaceWrappers)
{
*pbOfInterfaceWrappers = TRUE;
}
return TRUE;
}

if (hndElemType == TypeHandle(CoreLibBinder::GetClass(CLASS__ERROR_WRAPPER)) ||
hndElemType == TypeHandle(CoreLibBinder::GetClass(CLASS__CURRENCY_WRAPPER)) ||
hndElemType == TypeHandle(CoreLibBinder::GetClass(CLASS__BSTR_WRAPPER)))
{
*pbOfInterfaceWrappers = FALSE;
if (pbOfInterfaceWrappers)
{
*pbOfInterfaceWrappers = FALSE;
}
return TRUE;
}
}
Expand All @@ -4889,6 +4904,7 @@ BASEARRAYREF OleVariant::ExtractWrappedObjectsFromArray(BASEARRAYREF *pArray)
GC_TRIGGERS;
MODE_COOPERATIVE;
PRECONDITION(CheckPointer(pArray));
PRECONDITION(IsArrayOfWrappers(pArray, NULL));
}
CONTRACTL_END;

Expand Down Expand Up @@ -5022,6 +5038,7 @@ TypeHandle OleVariant::GetWrappedArrayElementType(BASEARRAYREF *pArray)
GC_TRIGGERS;
MODE_COOPERATIVE;
PRECONDITION(CheckPointer(pArray));
PRECONDITION(IsArrayOfWrappers(pArray, NULL));
}
CONTRACTL_END;

Expand Down Expand Up @@ -5067,8 +5084,7 @@ TypeHandle OleVariant::GetArrayElementTypeWrapperAware(BASEARRAYREF *pArray)
}
CONTRACTL_END;

BOOL bArrayOfInterfaceWrappers;
if (IsArrayOfWrappers(pArray, &bArrayOfInterfaceWrappers))
if (IsArrayOfWrappers(pArray, nullptr))
{
return GetWrappedArrayElementType(pArray);
}
Expand Down