[Obsolete] FabricObserver 3.1.18, ClusterObserver 2.1.11

FabricObserver 3.1.18, ClusterObserver 2.1.11 - SFPkgs with Microsoft-signed binaries. Microsoft-signed Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

• Concurrent process monitoring on capable CPU configurations (>= 4 logical processors) which greatly improves AppObserver's performance when monitoring several services. FabricSystemObserver and ContainerObserver also support this feature. This feature is per-observer configurable. See ApplicationManifest.xml and the specific observer sections to enable this feature for capable hardware. Note that if you are not running FO on a multi-core machine, then enabling this feature doesn't do anything (the monitoring and reporting loops will run sequentially).
• Added Thread count metric for AppObserver and FabricSystemObserver.
• ApplicationInsights Telemetry format change: Numeric data will be held in customMeasurements collection. You will need to modify your existing queries, which assume all data is present in customDimensions collection. This is no longer the case. There are two collections that will be available: customDimensions which contains string,string values and customMeasurements which contains string,double values. For example: given EphemeralPorts telemetry data for application fabric:/foo, customDimensions will contain the app name, node name, etc... customMeasurements will contain the metric name (EphemeralPorts) and corresponding value (say, 4200).
• Bug fixes (see Issues) and code improvements.

ClusterObserver Changes

• Bug fixes.

[Obsolete] FabricObserver 3.1.17

FabricObserver 3.1.17 - SFPkgs with Microsoft-signed binaries. Microsoft-signed Nupkgs will be located in the nuget.org gallery.

FabricObserver Changes

• ContainerObserver joins the mix, promoted to a built-in observer. This observer monitors CPU and Memory use of Service Fabric containerized applications (docker). It's a slimmed down AppObserver, but for containers. See Observers Readme for more information.
• Many (not all) ObserverManager configuration settings moved into ApplicationManifest.xml as overridable parameters (so, you can change OM settings with versionless application parameter upgrades and not have to redeploy FO when you want to make a change to an overridable OM setting).
• Monitoring and performance improvements for AppObserver, FabricSystemObserver and NodeObserver.
• Finally, this version ships with an updated internal operational telemetry impl for Non-PII data sharing with Microsoft (opt out).

FabricObserver operational data is transmitted to Microsoft and contains only basic information about FabricObserver (enabled observers, built-in (non-plugin) observer warnings or errors detected), the type of cluster (SFRP, for example), the type of OS (Windows or Linux).

This information is only used by the Service Fabric team and will be stored (data retention) for no more than 90 days.

We are only interested in the following information:

• Is FO is working and healthy? - If FO crashes with an unhandled exception that can be caught, related error information will be sent to us (this will include the offending FO stack).
• What is the number of enabled observers?
• Are there any FO plugins running?
• Is FO finding issues (generating health events)? This data is represented in the total number of Warnings/Errors an observer finds in an 8 hour window.
• This telemetry is sent every 8 hours and internal error/warning counters are reset after each telemetry transmission.

This agnostic information is very helpful to your friends on the Service Fabric team working on FO.

If you do not want to share this information, simply disable it in ApplicationManifest.xml by setting ObserverManagerEnableOperationalTelemetry to false, e.g.,

<Parameter Name="ObserverManagerEnableOperationalTelemetry" DefaultValue="false" />

Here is a full example of exactly what is sent in one of these telemetry events, in this case, from an SFRP cluster:

{
    "EventName": "OperationalEvent",
    "TaskName": "FabricObserver",
    "EventRunInterval": "08:00:00",
    "ClusterId": "50bf5602-1611-459c-aed2-45b960e9eb16",
    "ClusterType": "SFRP",
    "NodeNameHash": "1672329571",
    "FOVersion": "3.1.17",
    "HasPlugins": "False",
    "UpTime": "00:00:27.2535830",
    "Timestamp": "2021-09-12T00:51:42.8588118Z",
    "OS": "Windows",
    "EnabledObserverCount": 5,
    "AppObserverTotalMonitoredApps": 4,
    "AppObserverTotalMonitoredServiceProcesses": 6,
    "AppObserverErrorDetections": 0,
    "AppObserverWarningDetections": 1,
    "CertificateObserverErrorDetections": 0,
    "CertificateObserverWarningDetections": 0,
    "DiskObserverErrorDetections": 0,
    "DiskObserverWarningDetections": 0,
    "NodeObserverErrorDetections": 2,
    "NodeObserverWarningDetections": 0,
    "OSObserverErrorDetections": 0,
    "OSObserverWarningDetections": 0
  }

Please see FabricObserver Operational Telemetry for more information.

[Obsolete] FabricObserver 3.1.16

FabricObserver 3.1.16 - SFPkgs with Microsoft-signed binaries. Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

This release adds improvements to AppObserver's process dumping feature (Windows only) and introduces a new related observer, AzureStorageUploadObserver. Please see the documentation for details. These changes include new settings in Settings.xml and their overrides in ApplicationManifest.xml, so you will need to update both of these files (merge in your settings with these new ones).

• AzureStorageUploadObserver provides periodic, configurable, Windows-only (for now) support for uploading dmp files (compressed to zip files) to your specified Azure Storage account (blob). AppObserver's dump-on-error-threshold-breach is a Windows-only feature today, so there is no need to enable AzureStorageUploadObserver for Linux deployments. It’s capabilities will expand over time to support different scenarios (e.g., uploading FO logs, etc..).

• Design fix: process creation decrease by AppObserver and NodeObserver and less stalking of FO by Windows Defender on Windows Server 2019 as a consequence of these changes. For small sku VMs, you will see much less CPU usage by MsMpEng.exe when FabricObserver runs. This is a Windows only issue that is now fixed. For Linux, these changes are also useful since they will decrease FabricObserver's overall CPU usage in that environment as well (which is already low). Note that the MsMpEng.exe problem for FO 3.1.1.5 only exists on Windows Server 2019 Datacenter images and on VM skus with low numbers of cores. That said, please upgrade to 3.1.16 as soon as you can.

• AppObserver will now put FO into Warning state if you supply malformed JSON configuration or missing objects in said configuration. This setting is configurable: You can put the app into Warning or Error or do nothing. AppObserver will also now put FO into Warning if you supply configuration targets for applications that run service processes at a higher privilege than FabricObserver.

• REMOVED: ObserverBase.WriteToLogWithLevel and ObserverBase.HealthReporter.ReportFabricObserverServiceHealth functions. Use ObserverLogger.LogWarning instead in your plugins. Both of the removed functions just call the FO logger API anyway. They are unnecessary abstractions.

• Code enhancements and other bug fixes, like adding support for old-style-only AppObserver JSON configuration, which was semi-broken in 3.1.15.

[Obsolete] FabricObserver 3.1.15, ClusterObserver 2.1.10

FabricObserver 3.1.15, ClusterObserver 2.1.10 - SFPkgs with Microsoft-signed binaries. Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

This release adds support for service process tree monitoring by AppObserver. This adds accuracy to the monitoring results when your service processes launch child processes that consume resources. Even if your children launch children who launch children who launch children, AppObserver will monitor the descendants and apply their resource usage to the parent service. You will be able to track descendant behavior via telemetry as the data is captured in a new type that all current telemetry provider impls support. Please see Observers.md for more information about this feature.

• Any child process (and descendants at max depth = 4) launched by a service process that is being monitored by AppObserver will also be monitored and its resource usage will be added to the parent's for use in threshold violation checks for an observed (configured) metric.

• New configuration settings for AppObserver:

Settings.xml (where the Parameters are defined):

<Section Name="AppObserverConfiguration">
   ...
    <Parameter Name="EnableChildProcessMonitoring" Value="" MustOverride="true" />
    <Parameter Name="MaxChildProcTelemetryDataCount" Value="" MustOverride="true"/>
</Section>

ApplicationManifest.xml (where you set the overridable (required) parameter values):

<!-- AppObserver -->
    ...
    <!-- Process family tree monitoring. -->
    <Parameter Name="AppObserverEnableChildProcessMonitoring" DefaultValue="true" />
    <Parameter Name="AppObserverMaxChildProcTelemetryDataCount" DefaultValue="5" />

• Two new related types added to FO: ChildProcessTelemetryData and ChildProcessInfo.

• Added support for new child process monitoring data in ETW, AppInsights and LogAnalytics telemetry provider impls.

• Fixed minor bug in AppObserver monitor duration support.

• Updated TelemetryData member types: Value is double (was object). ProcessId is int (was string). ReplicaId is long (was string).

ClusterObserver Changes

• Updated TelemetryData to match FO 3.1.15's TelemetryData.
• Updated ApplicationInsights impl to support updated TelemetryData.

[Obsolete] FabricObserver 3.1.14

FabricObserver 3.1.14 SFPkgs with Microsoft-signed binaries. FO Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

• Bug fix in FO plugin loader: Native plugin dependencies will no longer crash loader (and thus FO).

• Added support for individual plugin folders in the Plugins directory (each plugin and its dependencies can live in a unique plugin folder (see below)).

Plugins and Dependencies

• You MUST place all plugin dependency libraries in the same folder as your plugin dll.
• Plugins (and their dependencies) can live in child folders in the Plugins directory, which will keep things cleaner for folks with multiple plugins.

The Plugins folder/file structure MUST be:

E.g.,

• Config/Data/Plugins/MyPlugin/MyPlugin.dll (required), MyPlugin.pdb (optional), [ALL of MyPlugin.dll's private dependencies] (required)

OR

• Config/Data/Plugins/MyPlugin.dll (required), MyPlugin.pdb(optional), [ALL of MyPlugin.dll's private dependencies] (required).

A private plugin dependency is any file (typically a dll) that you reference in your plugin project. So, any of the runtime binaries in the nuget packages you install into your plugin project(s), for example.

If you do not follow these guidelines, then your plugin will not work due to dependency resolution failure. So, please follow this advice!

[Obsolete] FabricObserver 3.1.13

FabricObserver 3.1.13 SFPkgs with Microsoft-signed binaries. FO Nupkgs will be located in the nuget.org gallery.

FabricObserver Changes

• New EventSource (ES) implementation and ES moved into Extensibility library. This is to better support folks who build plugins that need to capture FO ETW events (with EventListener). No public API changes (you still call ObserverLogger.LogETW to generate ES traces).
• Performance tuning.
• No bug fixes.

FabricObserver 3.1.12 [Obsolete]

FabricObserver 3.1.12 SFPkgs with Microsoft-signed binaries. FO Nupkgs will be located in the nuget.org gallery.

FabricObserver Changes

• This is a performance-based release with improvements in FO workingset, which was already relatively small (default configuration).
• Code-changes (refactoring across all observers that monitor resources to support better memory usage control).
• Bug fix (regression) in OSObserver - MachineTelemetryData.HotFixes will no longer contain hyperlinks.

Please report back if you notice improvements in memory (workingset).

[Obsolete] FabricObserver 3.1.11, ClusterObserver 2.1.9

FabricObserver 3.1.11, ClusterObserver 2.1.9 - SFPkgs with Microsoft-signed binaries. FO/CO Nupkgs are located in the nuget.org gallery.

Target SF runtime for SFPKGs and nupkgs = 7.2.x.

Note: FO source code has been tested on SF 8.0.516.9590 as well, so you can update all related packages if you are running 8.0.516.9590 with no code changes. These pre-built binaries, as stated above, however, target 7.2.x. This means they will function properly on any version greater than or equal to 7.2.

This is a minor release with one bug fix and several minor code improvements across FO and CO. Note there are member name changes in MachineTelemetryData class which is a type used by one Observer, OSObserver. The machine info output (in SFX, Telemetry and ETW) has also been slightly modified for Linux environments (for example, if some property does not exist on Linux, then it won't just show up as an empty string (like HotFixes, which today are only related to Windows Update patches).

FabricObserver Changes

• Bug fix in OSObserver
• Name changes in MachineTelemetryData
• Minor code improvements.

ClusterObserver Changes

• Code improvements.

[Obsolete] FabricObserver 3.1.10, ClusterObserver 2.1.8, FabricObserverWebApi 2.0.2

FabricObserver 3.1.10, ClusterObserver 2.1.8, FabricObserverWebApi 2.0.2 - SFPkgs with Microsoft-signed binaries. FO/CO Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

• Added error stream capture to WindowsInfoProvider port monitoring code to log transient netstat failures.
• Lots of non-critical code maintenance and some refactoring.
• Some optimizations. (Note that over half of FO working set is held in native memory - not much can be done about that.)

ClusterObserver Changes

• Added Repair Job monitoring (with on/off configuration).
• Minor refactoring and general code maintenance.
• Telemetry data updates.

FabricObserverWebApi Changes

• Bug fixes and code maintenance.

[Obsolete] FabricObserver 3.1.9, ClusterObserver 2.1.7

FabricObserver 3.1.9, ClusterObserver 2.1.7- SFPkgs with Microsoft-signed binaries. Nupkgs are located in the nuget.org gallery.

FabricObserver Changes

• Fixed minor EventSource logger bug.
• Added retry logic and logging to Windows port monitoring code.
• Added result set paging support in AppObserver's DeployedApplication query logic.
• Added retry logic to AppObserver's FabricClient calls.
• Added best effort code to automatically fix malformed targetApp values supplied in AppObserver configuration.
• Updated ApplicationInsights telemetry provider impl.
• Updated CsvLogger file management logic.
• Removed counter warm up calls in Windows Provider impls. Moved to callers.
• Code maintenance.

ClusterObserver Changes

• Fixed important bug in Application health query processor.
• ApplicationInsights TelemetryProvider impl updates.
• Improvements in service close code.