Automatically Assign Availability Set Names to VMs with Powershell

This blog-post is about using System Center Virtual Machine Manager (SCVMM) Availability Sets to spread similar VM’s to different Hyper-V Hosts to increase reliability both when using Failover Clustering, and when using stand-alone Hyper-V hosts.

First of all, what is Availability Sets?
In SCVMM 2012 SP1, Microsoft added Availability Sets. Failover Cluster Manager users are probably familiar with AntiAffinityClassNames, and Availability Sets are a very similar concept. This allow the user to specify a set of VMs which they would prefer to keep on separate hosts, and the Intelligent Placement engine works hard to make sure that all our features respect that preference.

Attempting to place multiple VMs with the same Availability Set onto a single host will generate a placement warning, meaning that the host will be prioritized last in the placement dialog

  • When placing a VM with an Availability Set into a cloud placement or as part of a service will avoid hosts with another VM from the same Availability Set, and warn the user if that was the only choice.
  • Dynamic Optimization will never move 2 VMs from the same Availability Set onto the same host. It will also actively attempt to separate any VMs with the same Availability Set that are on the same host.
  • Power Optimization will never power off a host that would lead to 2 VMs with the same Availability Set sharing a host.
  • Putting a host in maintenance mode will attempt to spread VMs with the same availability set to different target hosts.
  • If your VMs are highly available and hosted on a Hyper-V failover cluster, VMM will create AntiAffinityClassNames on the VMs with an Availability Set, so that even during cluster failover, SCVMM opt to failover to different hosts, if possible.

You can manually create ASCVMM2vailability Sets through SCVMM by selecting Properties on a VM.
Just click Create to make a new Name and assign it to the VM’s you want to keep on separate Hosts. When a Availability Set is not assigned to a VM any longer, the Availability Set will be deleted automatically, thus cleaning up the list for you.

For example, for your SQL Server Cluster, you may want to create a Availability Set name called SQL and assign it to your SQL Server Nodes. Easy!
Also, if you are using Service Templates, you can opt in to automatically create Availability Set names for your services.

Though I like to control things like SCVMM1that automatically. Depending on your naming convention for your Virtual Servers, this might or might not be possible for you.
In our case we have a strict naming policy to name servers with:
PREFIX FUNCTION NUMBER as seen in this picture:

Which makes it very easy for me to define that all servers called CLAZSQ* are similar and should be kept on different servers.

But, if all servers were called SRV0001-SRV9999 it would not be possible to utilize the ServerName for setting Availability Set names, and you would have to query the CMDB for info first.

Also, in our environment we have multiple Tenants, who could each have servers called DomainController01 and DomainController02. So just having a availability set called DomainController, would not be enough. I have to make it DomainController_TenantName or something similar.

I wrote this quick and short Powershell script to automatically assign a Availability Set to all VM’s. It will remove Numbers from the VM Name, and use the VMName + UserID (Tenant Subscription id) as the Availability Set Name. Clean, simple and easy, just schedule it to run regularly, or even make a SMA Job to trigger when a VM is created through AzurePack.

And then trigger an Host Cluster Optimization of all Clusters in the Environment if you don’t want to wait for the normal one.

 

 

Microsoft Fabric (datacenter and private cloud) related Hotfixes

Here is the list of Hotfixes I’m deploying in our production environment and that I deploy regularly at customers. Those production environments are a Fabric (Private Cloud) running Hyper-V, Storage Spaces, SOFS, ADFS, Domain Controllers, Azure Pack, System Center, SQL Servers, and more, yes everything you need in a Fabric. Though not Exchange, Lync or Sharepoint etc. So this list might not be complete for your system.
And as always, use your own judgement which hotfixes you would like to deploy in your environment or not. Hotfixes are not tested as much as ServicePacks used to be, and Update Rollups are, so it’s possible there are problems with them.

My philosophy is that I like to have everything updated and reduce the risk of having a problem. The number of times I have had issues with a hotfix are, as far as I can remember one (1), including the several years I worked at Microsoft Premier support and were assisting customers with problems and now and then provided a hotfix for an issue. So I rather install hotfixes I know of and are relevant to reduce the risk of hitting a real problem than wait for that issue to actually happen and then find a hotfix or open a case with Microsoft.

A hotfix included all previous fixes for that module too, so when troubleshooting a problem, it’s common that Microsoft Support asks you to install hotfix X, Y and Z to get the components involved in the problem to the latest revision. Thus, it might look like some of the KB Articles and hotfixes below does not apply to you, or you don’t have that problem in your environment. But if it’s related to Cluster, Hyper-V or any other component that you do use, it might be wise to install it anyway as it could fix 10 other problems that you are not aware of.

I’ll always import the updates directly into WSUS and deploy them, so I can use approval rules and see reporting of which updates has been installed where. Here is a good guide for how to do it; http://www.thirdtier.net/2013/03/how-to-manually-add-a-hotfix-to-wsus/

There is as far as I know (and I’ve also asked Premier Support) no way to script the import of updates into WSUS directly from Windows Catalog. You will have to manually use a Web Browser to import them. Click, Click, Click, wait, Click, Click….

The list is ordered by release date so the latest hotfixes are at the top. And looking at a fresh Fabric deployment, it looks like most hotfixes older than 10/14/2014 has been superseded, except for KB2965733 which was still needed by a couple of servers in this new fresh environment. But things might be different for you. wsus1

I’ve also written a Powershell Script for SCVMM to create Baselines and import all Updates and Hotfixes there. So it’s easy to use compliance scans and use remediation from SCVMM to keep the Fabric updated.
You can find it here; https://gallery.technet.microsoft.com/scriptcenter/SCVMM-Automatic-Baseline-8779597b

It’s not that easy to find new hotfixes or to know which ones are mandatory. Luckily, there is a blogpost to help you out. I’ve collected all sources from Microsoft product teams, where they list the hotfixes they recommend.
You can find the lists here: http://www.isolation.se/list-of-resources-to-find-hotfixes-and-updates-for-windows-server-2012-r2/ 

 

Anyway, here is the long list of fixes for possible problems in your environments. Updated: 7/22/2015

Hyper-V cluster unnecessarily recovers the virtual machine resources in Windows Server 2012 R2

http://support.microsoft.com/kb/3072380   Released: 7/14/2015

Virtual machines that host on Windows Server 2012 R2 may crash or restart unexpectedly
http://support.microsoft.com/kb/3068445   Released: 7/14/2015

Added 07/22/2015    “0xc0000017” error when you restart a UEFI-based computer in Windows
https://support.microsoft.com/kb/3072381   Released: 7/13/2015

Interrupts to the Intelligent Platform Management Interface driver are missed in Windows Server 2012 R2
http://support.microsoft.com/kb/3061460   Released: 6/9/2015

Unexpected ASP.Net application shutdown after many App_Data file changes occur on a server that is running Windows Server 2012 R2
http://support.microsoft.com/kb/3052480   Released: 6/9/2015

Update adds support for compound ID claims in AD FS tokens in Windows Server 2012 R2
http://support.microsoft.com/kb/3052122   Released: 6/9/2015

Update to improve the backup of Hyper-V Integrated components in Hyper-V Server 2012 R2
http://support.microsoft.com/kb/3063283   Released: 6/9/2015

Stop error code 0xD1, 0x139, or 0x3B and random crashes in Windows Server 2012 R2
http://support.microsoft.com/kb/3055343   Released: 5/12/2015

Backup application that calls the VSS service becomes unresponsive when the DFSR service is running in Windows
http://support.microsoft.com/kb/3054249   Released: 5/12/2015

Resolution of external DNS records on a Windows Server 2012 R2 Hyper-V guest cluster fails through a Hyper-V Network Virtualization Gateway
http://support.microsoft.com/kb/3049448   Released: 5/12/2015

Shared Hyper-V virtual disk is inaccessible when it’s located in Storage Spaces on a Windows Server 2012 R2-based computer
http://support.microsoft.com/kb/3025091   Released: 5/12/2015

“The URL cannot be resolved” error in DirectAccess and routing failure on HNV gateway cluster in Windows Server 2012 R2
http://support.microsoft.com/kb/3047280   Released: 5/12/2015

Hyper-V host crashes and has errors when you perform a VM live migration in Windows 8.1 and Windows Server 2012 R2
http://support.microsoft.com/kb/3031598   Released: 4/14/2015

Hotfix enables AD FS token replay protection for Web Application Proxy authentication tokens in Windows Server 2012 R2
http://support.microsoft.com/kb/3042121   Released: 4/14/2015

“HTTP 400 – Bad Request” error when you open a shared mailbox through WAP in Windows Server 2012 R2
http://support.microsoft.com/kb/3042127   Released: 4/14/2015

Files cannot be copied when drive redirection is enabled in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/3042841   Released: 4/14/2015

“STATUS_PURGE_FAILED” error when you perform VM replications by using SCVMM in Windows Server 2012 R2
http://support.microsoft.com/kb/3044457   Released: 4/14/2015

You cannot upgrade Hyper-V integration components or back up Windows virtual machines
http://support.microsoft.com/kb/3046826   Released: 4/14/2015

RDP session becomes unresponsive when you connect to a Windows Server 2012 R2-based computer
http://support.microsoft.com/kb/3047296   Released: 4/14/2015

“Your computer can’t connect to the remote computer” error because RD Gateway service freezes in Windows Server 2012 R2
http://support.microsoft.com/kb/3042843   Released: 4/14/2015

A SQL Server that is running in a Hyper-V virtual machine takes a long time to restore a database to a dynamic VHD
http://support.microsoft.com/kb/2970653   Released: 3/10/2015

DNS server does not try the second forwarder and other DNS improvements in Windows Server 2012 R2
http://support.microsoft.com/kb/3038024   Released: 3/10/2015

“0x000000D1” Stop error when you fail over a cluster group in Windows Server 2012 or Windows Server 2012 R2
http://support.microsoft.com/kb/3036614   Released: 3/10/2015

Hotfix for update password feature so that users are not required to use registered device in Windows Server 2012 R2
http://support.microsoft.com/kb/3035025   Released: 3/10/2015

AD FS cannot process SAML response in Windows Server 2012 R2
http://support.microsoft.com/kb/3033917   Released: 3/10/2015

Added 7/18/2015    “0x0000003B” or “0x0000007E” Stop error on a Windows-based computer that has 4K sector disks
https://support.microsoft.com/kb/3027108  Released: 2/10/2015

Custom values for various MPIO timers in Windows Server 2012 R2 may not be honored
http://support.microsoft.com/kb/3027115   Released: 2/10/2015

System may freeze if a reserved disk is mounted accidentally in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/3027110   Released: 2/10/2015

RemoteApp window is too large or too small when you use RDP to run a RemoteApp application in Windows Server 2012 R2
http://support.microsoft.com/kb/3026738   Released: 2/10/2015

Operation fails when you try to save an Office file through Web Application Proxy in Windows Server 2012 R2
http://support.microsoft.com/kb/3025080   Released: 2/10/2015

You are not prompted for username again when you use an incorrect username to log on to Windows Server 2012 R2
http://support.microsoft.com/kb/3025078   Released: 2/10/2015

Hotfix to avoid a deadlock situation on a CSV file system volume on Windows Server 2012 R2
http://support.microsoft.com/kb/3022333   Released: 2/10/2015

You are prompted for authentication when you run a web application in Windows Server 2012 R2 AD FS
http://support.microsoft.com/kb/3020813   Released: 2/10/2015

Time-out failures after initial deployment of Device Registration service in Windows Server 2012 R2
http://support.microsoft.com/kb/3020773   Released: 2/10/2015

You are prompted for a username and password two times when you access Windows Server 2012 R2 AD FS server from intranet
http://support.microsoft.com/kb/3018886   Released: 2/10/2015

Cluster fixes for deadlock and resource time-out issues in Windows Server 2012 R2 Update 1
http://support.microsoft.com/kb/3023894   Released: 2/10/2015

RDS License Manager shows no issued free or temporary client access licenses in Windows Server 2012 R2
http://support.microsoft.com/kb/3013108   Released: 12/9/2014

iSCSI SAN server that’s running Windows Server 2012 R2 restarts unexpectedly on a high-speed network
http://support.microsoft.com/kb/3000123   Released: 11/11/2014

TRIM and UNMAP activities for thin provisioning on one volume block all activities on other volumes
http://support.microsoft.com/kb/2996802   Released: 11/11/2014

SMBv1 named pipe requests do not time out when the remote server hangs in Windows 7, Windows Server 2008, Windows 8.1, and Windows Server 2012 R2
http://support.microsoft.com/kb/2995054   Released: 10/14/2014

SMB 3.0 Transparent Failover feature does not work after you disconnect a drive cable in Windows
http://support.microsoft.com/kb/2991247   Released: 10/14/2014

WTSQuerySessionInformation API function always returns zero bytes for WTSIncomingBytes and WTSOutgoingBytes
http://support.microsoft.com/kb/2981330   Released: 10/14/2014

A network printer is deleted unexpectedly in Windows
http://support.microsoft.com/kb/2967077   Released: 8/12/2014

“0x00000018” Stop error when volumes are mounted in Windows Server 2012 R2 or Windows Server 2012
http://support.microsoft.com/kb/2973052   Released: 8/12/2014

Updates to improve the compatibility of Azure RemoteApp in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/2977219   Released: 8/12/2014

Error 58 when an application calls BackupRead function to back up files that are shared by using SMB in Windows
http://support.microsoft.com/kb/2973055   Released: 7/8/2014

2965733 The guest cluster is not available to service users after failover in a Hyper-V Network Virtualization environment
https://support.microsoft.com/kb/2965733   Released: 6/10/2014

NFS version 4.1 and version 3 work unexpectedly in Windows Server 2012 R2 or Windows Server 2012
http://support.microsoft.com/kb/2934249   Released: 4/8/2014

CSV snapshot file is corrupted when you create some files on the live volume in Windows
http://support.microsoft.com/kb/2929869   Released: 4/8/2014

On-demand virus scan freezes in Windows
http://support.microsoft.com/kb/2904100   Released: 3/11/2014

Windows Server 2012 R2 or Windows 8.1 crashes when virtual volumes are exposed to hyper-v virtual machines
http://support.microsoft.com/kb/2925766   Released: 2/11/2014

iSCSI Target stops responding to requests in Windows Server 2012 R2
http://support.microsoft.com/kb/2919740   Released: 2/11/2014

Memory and deadlock issues for the RD Virtualization Host and RD Connection Broker role services in Windows 8.1
http://support.microsoft.com/kb/2908810   Released: 2/11/2014

Hotfix improves storage enclosure management for Storage Spaces in Windows 8.1 and Windows Server 2012 R2
http://support.microsoft.com/kb/2913766   Released: 1/14/2014

OffloadWrite is doing PrepareForCriticalIo for the whole VHD in a Windows Server 2012 or Windows Server 2012 R2 Hyper-V host
http://support.microsoft.com/kb/2913695   Released: 1/14/2014

 

Set MPIO Policy via PowerShell for Storage Spaces

Here is a small script to set the MPIO Policy via Powershell according to Microsofts Best Practices for Storage Spaces as seen here https://technet.microsoft.com/library/0923b851-eb0a-48ee-bfcb-d584363be668

It will set the Global MPIO policy to Least Block and then change the MPIO Policy for all SSD’s to Round Robin. Though, it’s possible that mpclaim.exe will use a different DiskID from what Powershell/Device Manager is using.
So the script has a built-in feature to adjust the DiskId if needed, though you have to verify and set the value manually before running the script! 

 

List of resources to find Hotfixes and updates for Windows Server 2012 R2

When you use the more advanced features of Windows and System Center components, you tend to run into issues and undocumented features once in a while. Here are a couple of useful links to websites which lists the recommended hotfixes that should be installed before contacting Microsoft support. And I’m usually deploying them in our own live environment as soon as they are released, to hopefully not run into the issues or sometimes find out potential problems with the fixes before recommending them to our customers.

Updated: 2015-08-23 with additional collections (DFS).

Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters
https://support.microsoft.com/en-us/kb/2920151

List of currently available hotfixes for the File Services technologies in Windows Server 2012 and in Windows Server 2012 R2
https://support.microsoft.com/en-us/kb/2899011

Recommended hotfixes, updates, and known solutions for Windows Server 2012 and Windows Server 2012 R2 Hyper-V Network Virtualization (HNV) environments
https://support.microsoft.com/en-us/kb/2974503

Updates for Active Directory Federation Services (AD FS)
https://technet.microsoft.com/en-us/library/mt126278.aspx

Available Updates for Remote Desktop Services in Windows Server 2012 R2
https://support.microsoft.com/en-us/kb/2933664

Recommended hotfixes and updates for Windows Server 2012 DirectAccess and Windows Server 2012 R2 DirectAccess
https://support.microsoft.com/en-us/kb/2883952

Hyper-V: Update List for Windows Server 2012 R2 – kept up to date by the community (MVPs mostly).
http://social.technet.microsoft.com/wiki/contents/articles/20885.hyper-v-update-list-for-windows-server-2012-r2.aspx?PageIndex=2

List of Public Microsoft Support Knowledge Base (KB) Articles for System Center 2012 R2 Virtual Machine Manager (VMM 2012 R2)
http://social.technet.microsoft.com/wiki/contents/articles/22074.list-of-public-microsoft-support-knowledge-base-kb-articles-for-system-center-2012-r2-virtual-machine-manager-vmm-2012-r2.aspx

List of currently available hotfixes for Distributed File System (DFS) technologies in Windows Server 2012 and Windows Server 2012 R2
https://support2.microsoft.com/kb/2951262/en-us?sd=rss&spid=17383

Remember that you can import Hotfixes directly into WSUS and deploy them that way, just like any other update.

And this is a RSS Feed that lists the Latest KB Articles for Windows Server 2012 R2
https://support2.microsoft.com/common/rss.aspx?rssid=17383
I’ve got the RSS feed imported into Outlook, so I’ll easily see when there is a new one and will hopefully remember the issue when/if I run into it in the future.

 

Scale Out File Server – Latest Hotfixes of June 4, 2015

When I had contact with Microsoft Premier Support about a SOFS (Scale Out File Server) problem they provided me with this list of hotfixes, to get all components up to the latest version.

Ensure the SOFS nodes are updated with *ALL* the Cluster recommended updates:

Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters  http://support.microsoft.com/kb/2920151

Plus 4 additional ones below:
3023894                Cluster fixes for deadlock and resource time-out issues in Windows Server 2012 R2 Update 1  http://support.microsoft.com/kb/3023894/EN-US

3027115                Custom values for various MPIO timers in Windows Server 2012 R2 may not be honored  http://support.microsoft.com/kb/3027115/EN-US

3022333                Hotfix to avoid a deadlock situation on a CSV file system volume on Windows Server 2012 R2  http://support.microsoft.com/kb/3022333/EN-US

2970653     A SQL Server that is running in a Hyper-V virtual machine takes a long time to restore a database to a dynamic VHD  http://support.microsoft.com/kb/2970653/EN-US

All theses hotfixes can be imported to WSUS via Microsoft Update for easy distribution.
In addition to SOFS, I’ve also deployed all the cluster hotfixes to my Hyper-V Clusters.

The Interactive Services Detection service terminated with the following error: Incorrect function.

This morning I noticed that one of the Hyper-V Hosts at a customer were logging this error regularly in the system Eventlog;

The full detailed entry:

It looks like the events are happening every  30 minutes, and at the same time as Windows is for some (so far) unknown reason doing a reinstall of a lot of MSI packages, and the above Interactive Service is triggered at the same time as it’s reinstalling the DHCPExt.msi

I can so far unfortunately not find anything that’s logging why Windows is reconfiguring all MSI Packages on the server every 30 minutes.

It does look like it’s the DHCP Server extension that’s causing the Interactive Service errors, as they always happen at the same time. Though, the DHCP Server extension shouldn’t be reconfiguring in the first place.

We always enable the Reliability History on all servers whireliabilitych can be handy at times to see when a problem begun happening.
Check this Out!

It looks like the problem started on April 28 at 8:42 PM.

As the Reliability History tool is disabled by default, I’ll make another blogpost showing how you can enable this feature for all your servers.

Weventloghen I wanted to see what had happened around April 28th. I noticed that was the oldest entries in the Application log. When the log has become full, it has removed the oldest entries according to the settings.

So I don’t think I’ll get any more details that way, and it does look like this problem has gone on for quite some time.

I’ll just reinstall the Hyper-V Host as it’s done in a few minutes compared to spending hours trying to fix the problem.
AND… I’ll create a Group Policy that will increase the Eventlog Size to x10 the default. So the next time something like this happens, I’ll have information to dig deeper.

Updated 2015-05-19 09:08:

After doing some more digging, it seems according to this KB Article (KB974524 : Event log message indicates that the Windows Installer reconfigured all installed applications) that this problem can happen if one of the following is true:

  • You have a group policy with a WMIFilter that queries Win32_Product class.
  • You have an application installed on the machine that queries Win32_Product class.

As the problem is not happening every 90-120 minute which would be true if it was GPO Triggered, I would say it’s an application that uses the Win32_Product class. And after doing some digging, it turns out it’s a known problem with VMM which will be fixed in UR7. Or hopefully earlier with a hotfix.

Updated 2015-05-19 10:12:

Wow, I got a hotfix for the issue within 15 minutes after contacting the VMM Team.
I’ve just installed it in our test environment and will later install it in the customers production environment.

Unfortunately I don’t have a KB or Hotfix ID for this, but if you contact Premier Support I think you can mention that you need a hotfix for Engine.Adhc.Operations.dll which gives support for RegKey: UpdateDHCPExtension
That info should make them able to find the correct hotfix.

Unable to Connect to VMM in AzurePack after UR install

After upgrading to Update Release 6 (UR6) we got the same issue as seen in earlier UR’s. It’s not possible to connect to VMM in AzurePack so you can’t see your Virtual Machines, Clouds or Networks.

It turned out that when UR6 gets applied to SPF, the bindings are once again messed up. To fix this, just logon to the server hosting SPF and in IIS check the bindings as seen here;

SPF1

The SPF Website is not running and you can see two Bindings.
In my case, one has a certificate and the other doesn’t. So I just remove the binding without a certificate. Then start the Website and everything is working as expected again.

In earlier UR’s I’ve also seen how there is no bindings at all listed here. So you may have to create some binding then.

 

The request size exceeded the configured MaxEnvelopeSize quota

Today when I was updating our AzurePack WebSites Servers, I got an error which prevented the upgrade of most of the WebSite Roles like these;
Management Servers, Publishing Servers, Front End Servers and all the Web Workers. Yes, everyone except the Web Sites Controller.
Resulting in some unexpected downtime.  Luckily, all that was affected was this blogsite.

The error message I got was;
The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota.
And I could also see that the files being copied to c:\windows\temp (WebFarmAgent.msi) were broken.

I also had an error “Failed to copy role artifacts to agent” in the logfile seen on Windows Azure Pack Websites Controller.

First of all, I ran this command in an Elevated Command prompt on the server hosting the Controller Role;
C:\Windows\system32>winrm g winrm/config

winrm1

And then the same command on one of the failing servers;
C:\Windows\system32>winrm g winrm/config

winrm2

Notice the difference in MaxEnvelopeSizekb between the servers. One of the other servers had MaxEnvelopeSizeKB set to 700.

I don’t know why it’s different between the servers or what has suddenly changed it, my guess it’s some Windows Update patch. Though it’s the same patches being installed on all the servers, and I’ve seen three different values. Wicked.
So by using the same value on all the servers I got the setup to work. And as you can see, this blog site is now also running. YAY!

I chose to set the value to the same as on the Controller Server which is the one trying to run the commands and copy the files to the other servers.
winrm set winrm/config @{MaxEnvelopeSizekb=”8192″}
It will now take 5-60 minutes for all update and repair jobs to complete.

I couldn’t find any Group Policy object to use to set that value as a default value on all AzurePack WebSites servers. So I’ve got to come up with another longterm solution. Maybe doing it with Desire State Configuration (DSC) or via Configuration Manager?

Update AD-Users with new Phone-number and Pager via Powershell

Had a quick question from a customer about how one can automatically update the phone number and pager of a lot of AD users. The customer was changing switchboard and had to add 1 number in front of the current number.  Adding it in the middle of the string is also possible, but slightly more complicated as you have to split the string.

This is possible to do in a few different ways, but I chose the quickest way for me, via Powershell.

End Result:

aduser