Monitoring VMware vSphere with C#

To monitor a vSphere environment successfully usage of the "Web Services" SOAP API is still required. Until vSphere 7.0 the REST API did not provide any access to performance metrics, and the new vStats endpoint which looks to provide this is still in preview. In addition properties available to the SOAP API which are useful in monitoring the general health of objects are still missing from the REST API. The only alternative to the vSphere SOAP API is to use a vRealize Operations Manager instance which has a REST API that provides access to performance metrics, although I have not tested this personally.

A vSphere Web Services Client

We will be looking at how to create a basic client in C#/.NET to query the vSphere Web Services SOAP API with the goal of doing some simple monitoring. There are full libraries in other languages that VMware support, but for C# there is just the SDK which only has code samples. Given that the SOAP API has quite a steep learning curve and the SDK samples can be confusing the simple client we will create should be easier to understand.

Development Environment

Visual Studio 2019 Community and LINQPad 5 are required. Examples are provided in LINQPad which can be run interactively and used to easily test changes. There are two SDK downloads that will be used, the vSphere Management SDK and vSAN Management SDK.

Firstly clone the git repository with the example code we will work with. Then run through the steps below.

Download the vSphere SDK and load the Visual Studio solution at SDK\vsphere-ws\dotnet\cs\samples\Samples2012.sln.
Build the VMware.Binding.WsTrust project which implements the WS-Trust protocol for the VMware SSO authentication.
Copy the three DLL files that are created by the build to the lib folder in the git repo cloned previously.
Download the vSAN Management SDK and install Web Services Enhancements (WSE) 3.0 for Microsoft .NET (requires .NET 3.5).
Review vSAN SDK README file and install Python if it is not already available.
Run the following command to build the DLL (with the file paths adjusted as necessary) and copy this file to the lib folder.

python builder.py ^
"C:\Path\To\vsan-sdk-cs\bindings\wsdl\vsan.wsdl" ^
"C:\Path\To\vsan-sdk-cs\bindings\wsdl\vsanService.wsdl" ^
-w "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\wsdl.exe" ^
-s "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\sgen.exe" ^
-c "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" ^
-v "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\SvcUtil.exe" ^
-m "C:\Program Files (x86)\Microsoft WSE\v3.0\Microsoft.Web.Services3.dll"

The lib folder in the cloned repo should now contain STSService.dll, Vim25Service.dll, VMware.Binding.WsTrust.dll and VsanhealthService.dll.

Testing the Client

To get hands on right away we can run vSphereWebService.linq from the extras folder, this contains the Connection class that creates our simple client application.

Below is an example from the file of how to initialise a connection and retrieve the host and cluster objects. The ConnectionConfig class takes user/pass credentials and the URL of the vCenter or ESXi web service endpoint to connect to.

void Main()
{
    var conn = new Connection(
        new ConnectionConfig("[email protected]", "password", "https://192.168.0.1/sdk")
    );

    ObjectContent[] about = conn.RetreieveObjectProperties(conn.ServiceInstance, new string[] { "content.about.fullName" });
    about.Dump("Some properties can be retrieved without authentication.");

    // create an authenticated connection
    conn.Connect();

    // retrieve the name of the managed objects in two of the views created during the connection setup
    ObjectContent[] hosts = conn.RetrieveViewProperties(conn.HostView, "HostSystem", new string[] { "name" });
    ObjectContent[] clusters = conn.RetrieveViewProperties(conn.ClusterView, "ClusterComputeResource", new string[] { "name" });
    hosts.Dump("The name of all the HostSystem managed objects (Hosts).");
    clusters.Dump("The name of all the ClusterComputeResource managed objects (Clusters).");

    conn.Disconnect();
}

The Connection Class

The Connection class will create an authenticated connection and has some helper methods which make querying properties on managed objects easier. The heavy lifting is done by the VMware.Binding.WSTrust (which implements the WS-Security protocol for VMware SSO) and STSService (Security Token Service) libraries. These are provided by VMware in the SDK and are based on the Windows Communication Foundation (WCF) framework.

While setting up a connection a set of Views are created, these provide an inventory of the vSphere environment for the main managed object types. The ConnectionConfig class is simply a way to pass around the details for a specific vCenter server or ESXi host. More properties could be added if multiple connections were being managed.

Finally there are a couple of custom exceptions (NotAuthenticatedException and ObjectDeletedException) to handle authentication and session time out errors.

Web Services API Overview

Now we have some code that will connect to a vCenter server or ESXi host understanding how to retrieve objects and properties is the next task. The vSphere environment is represented by the API as a hierarchy of objects (see figure 1) which form an inventory of all the resources that are visible from the standard vCenter web interface. The vSphere Web Services SDK Programming Guide and the vSphere Web Services API documentation both have more information about the object hierarchy.

Figure 1. Inventory/object hierarchy example.

Along with the inventory heirarchy the Managed Object Browser and Property Collector are both key to understanding the API, so we will take a look at those in more detail next.

Managed Objector Browser

The Managed Object Browser provides a way to interactively navigate the inventory of a vCenter server or standalone ESXi host, and is the best way to understand how objects and properties are related. Since vSphere 6 it is disabled by default, but after it is enabled simply add /mob to the vCenter URL to access it, for example https://192.168.1.1/mob.

The root of the inventory is displayed when first loading the browser, which is the SerivceInstance managed object created automatically by the server. Through the content property all the managed objects that allow a client to interact with the various functions of the API can be reached.

Additionally, the rootFolder property references the first folder in the inventory tree, and can be used to browse through the other resources.

Property Collector

The Property Collector is a managed object type that can retrieve the properties of other managed objects. Basic usage allows the value of a specific property to be read, and this is what we will use to monitor the state of the various vSphere resources.

Retrieval of properties can be limited to a single object, or the collector can traverse the inventory hierarchy and retrieve the properties of multiple objects. This works by providing an object that will be starting point for the property collector to begin traversal, and then building PropertyFilterSpec and TraversalSpec objects to configure the collector.

The best way to understand it will be to run the LINQPad examples provided next and review a couple of helpful links. The first is a full development guide which is probably the best explanation out there for how the vSphere Web Services API works in general (despite being over 10 years old). The second is a video that does a reasonable job of explaining how the property collector works specifically.

Property Collector Examples

The extras folder in the git repository has LINQPad examples that show usage of the property collector in a few scenarios. To run these, first follow the steps below.

Build the "vSphere WS Client" project from the solution in the repository.
Copy the vSphereWsClient.dll library that is compiled to the lib folder.

This provides the Connection class shown in the first LINQPad example as a library file which the property collector examples use.

There are three example files:

vSphereWebService-Example1.linq shows selecting a property on a single object.
vSphereWebService-Example2.linq shows use of a "TraversalSpec" to walk the inventory tree.
vSphereWebService-Example3.linq a couple of more practical examples using a "TraversalSpec".

View and Search Methods

To avoid having to use complex Property Collector invocations every time objects need to be retrieved the ViewManager and SearchIndex managed objects can be used to simplify the process. A Container View can retrieve a specific set of objects which can be used with the Property Collector to easily read their properties. In our example code views are created for Clusters, Hosts, Datastores and VMs through the CreateContainerView method, and are queried using the RetrieveViewProperties method.

The SearchIndex can query the inventory for specific objects based on the type of search method used, and will return a list of managed object references that match. Search is not suited to retrieving all the objects of a specific type like a view can, so it is not used in the example code.

A Monitoring Application

The monitoring application is a simple command line program which takes a few input parameters and reads the state of properties and counters on objects in the vSphere environment. As we are looking to monitor the basics in this example only key metrics will be retrieved, but there are of course many other ways to leverage the API for monitoring.

Usage

An executable named MonitoringCLI.exe will be created by building the "vSphere Monitoring CLI" project. This can be run with the following options:

Usage:
  MonitoringCLI --username <username> --password <password> --url <url> [--vsan --vsanfullsummary]

Options:
  --username         The user account for the vCenter server or ESXi host.
  --password         The password for the chosen user account.
  --url              The URL of the vCenter server or ESXi host (https://192.168.1.1/sdk).
  --vsan             Include vSAN health summary metrics.
  --vsanfullsummary  Include all vSAN health summary properties.

The expected output should be similar to the below.

CLUSTERS
  Name: Cluster
  Alarm Status: green

DATASTORES
  Name: Media
  Alarm Status: green
  Free Space %: 62

  Name: Virtual Machines
  Alarm Status: yellow
  Free Space %: 80

HOSTS
  Name: 10.66.30.12
  Alarm Status: yellow
  Status: connected
  CPU Usage %: 7
  Memory Usage %: 56

  Name: 10.66.30.11
  Alarm Status: yellow
  Status: connected
  CPU Usage %: 15
  Memory Usage %: 90

Press ESC to exit or any key to refresh.
--------------------------------------------------------------------------------

Details

The project contains two source files Program.cs and Metrics.cs. Program.cs is obviously the entry point for execution, there is some basic parsing for the command line parameters and then it enters a loop to start polling the API. Metrics.cs defines some classes to create a simple interface for reading the "metrics" from a managed object.

You will notice that setting up the connection takes a long time (around 90 seconds), this is reported to be an issue with XML serialization overhead (the serialization step was not pre-generated for the library files included in the SDK). However when I checked the Vim25Service.dll in SDK\vsphere-ws\dotnet\cs\samples\lib it did look to already have the optimization applied.

There are more details in an open GitHub issue, with a link to the process for building the C# library from the WSDL file at SDK\vsphere-ws\wsdl\vim25. I have not tried the full process myself as I was often testing using LINQPad, which is only slow for the first execution of a query, as the serialization step is held in memory. However there is a comment on the GitHub issue which suggests that re-compiling from the WSDL file can work.

For a monitoring application running as a background service slow connection setup may not be a serious problem as it is likely the same session will be active for days or weeks.