vCheck hangs after vCenter Uprade to 7.0U3c

[VM-Master]

Hi guys,

we updated our vCenter last weekend from 7.0.2 18455184 to latest version 7.0.3 19234570.
Now I could see that plugin "33 VM CPU Usage" will not finished anymore. If I disabled these plugin all other enabled plugins runs like before. Our vCenter has approx. 500 VMs. It's a vCenter for our VMware Horizon VDIs.

Any ideas why this check doesn't work anymore or maybe need a lot of more time?

[roydebets]

Hi,

We experience the same thing, but not with al vCenters on 7.0U3c.
It started with one vCenter which did not complete the vCheck, but today a 2nd vCenter does not complete the vCheck as well.

Also, it seems that it's not always the same plugin which takes too much time to complete. This seems random and even multiple plugins take long completion time.

If anyone has a solution, please share!

[roydebets]

I found that for a specific vCenter the vCheck used to complete in 5 minutes and now takes 67 minutes.
Also, it lists the plugins that take longer than 60 seconds to complete.

09 Created or cloned VMs.ps1 1285.52
96 VMWare Replication Check.ps1 1189.46
77 Lost access to volume.ps1 655.85
01 General Information.ps1 629.19

Just collecting general info already takes more than 10 minutes.
Our vChecks do not complete since we run the via scheduled task and it is set to abort of the task runs for more than 1 hour.
I will extend that time now, but still strange that it takes more than 10x longer to complete.

[VM-Master]

I don't have plugin "96" and "09" enabled in my environment.
Plugin 01 took on vCenter 7 U1 100seconds, now on 7U3c it takes 210 seconds.

Plugin "77" seems to have an issue in my environment...
++++++++++++++++++++++++++++++++++++
[03:39:51] ..start calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89]
PS>TerminatingError(Get-VIEvent): "Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again."
Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the
argument collection contains a null value. Supply a collection that does not contain any null values and then try the
command again.
At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83

• ... .AddDays(-$LostAccessVolumeAge) -MaxSamples $MaxSampleVIEvent | Where ...

•                                             ~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBindingValidationException
  • FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetEvent
    Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument
    is null, empty, or an element of the argument collection contains a null
    value. Supply a collection that does not contain any null values and then try
    the command again.
    At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83
• ... .AddDays(-$LostAccessVolumeAge) -MaxSamples $MaxSampleVIEvent | Where ...

•                                             ~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBinding
    ValidationException
  • FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutom
    ation.ViCore.Cmdlets.Commands.GetEvent

[03:39:51] ..finished calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89]
++++++++++++++++++++++++++++++++++++

[roydebets]

Digging into it deeper I see various plugins having error messages about "Get-VIEvent". When checking the events on the vCenter manually, there are just no events to display. We have now raised a ticket at VMware for this. We have upgraded various vCenters to 7u3c, but only 3 of them have issues. Van: VM-Master ***@***.***> Beantwoorden - Aan: alanrenouf/vCheck-vSphere ***@***.***> Datum: woensdag 16 maart 2022 om 12:04 Aan: alanrenouf/vCheck-vSphere ***@***.***> CC: Roy Debets ***@***.***>, Comment ***@***.***> Onderwerp: Re: [alanrenouf/vCheck-vSphere] vCheck hangs after vCenter Uprade to 7.0U3c (Issue #743) I don't have plugin "96" and "09" enabled in my environment. Plugin 01 took on vCenter 7 U1 100seconds, now on 7U3c it takes 210 seconds. Plugin "77" seems to have an issue in my environment... ++++++++++++++++++++++++++++++++++++ [03:39:51] ..start calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] PS>TerminatingError(Get-VIEvent): "Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again." Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again. At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83 * ... .AddDays(-$LostAccessVolumeAge) -MaxSamples $MaxSampleVIEvent | Where ... · ~~~~~~~~~~~~~~~~~ · * CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBindingValidationException * FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetEvent Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again. At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83 * ... .AddDays(-$LostAccessVolumeAge) -MaxSamples $MaxSampleVIEvent | Where ... · ~~~~~~~~~~~~~~~~~ · * CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBinding ValidationException * FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutom ation.ViCore.Cmdlets.Commands.GetEvent [03:39:51] ..finished calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] ++++++++++++++++++++++++++++++++++++ — Reply to this email directly, view it on GitHub<#743 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUXNEXXLLTDBZ6PDVW2RFGLVAG55DANCNFSM5QCREXYA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[TheGabeMan]

Same here.
Measure-Command{ Get-VIEvent -MaxSamples 100 -Start (get-date).AddDays(-1) }
Eventually I then get an error:
Get-VIEvent : 23-3-2022 09:59:10 Get-VIEvent The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
Always after 10 minutes.

But we're also seeing impact on backups wanting to create snapshots and time out waiting. Other processes also have issues, like gathering licenses via Cloud Usage Meter.

[TheGabeMan]

Solved by performing two actions:

• set tasks and events history in vCenter to 1 day, reboot (in two days you can set it back to 25 again)
• VMware Support gave instructions how to "vacuum" the postgres database.

[VM-Master]

Hi,
yes, the DB optimizations seems to be helpful --> https://kb.vmware.com/s/article/87837?lang=en_US
It's officially recommended for everyone who upgraded the VCSA from 7.0U2 to U3.

[TheGabeMan]

Case can probably be closed now.