May 2014 – A Geeks World

List and Remove Corrupted files reported by Data Deduplication with Powershell

I’ve been copying 7TB of data in about 100.000 files from an old fileserver to the new one, but I just noticed that some of the files are corrupted! Gahhh…

Chkdsk found some issues, but didn’t solve the problem. As this server is running Windows Server 2012 R2 with Data Deduplication I decided to have a look at that.

Yeah, unfortunately a lot of corrupted files with EventID 12800

Log Name:      Microsoft-Windows-Deduplication/Scrubbing
Source:        Microsoft-Windows-Deduplication
Date:          5/16/2014 11:11:10 AM
Event ID:      12800
Task Category: Data Deduplication Scrubbing Task
Level:         Error
Keywords:      Reporting
User:          SYSTEM
Description:
Data Deduplication service detected corruption in "Drive:\PATH\FILE.Name". The corruption cannot be repaired.

Log Name: Microsoft-Windows-Deduplication/Scrubbing

Source: Microsoft-Windows-Deduplication

Date: 5/16/2014 11:11:10 AM

Event ID: 12800

Task Category: Data Deduplication Scrubbing Task

Level: Error

Keywords: Reporting

User: SYSTEM

Description:

Data Deduplication service detected corruption in "Drive:\PATH\FILE.Name". The corruption cannot be repaired.

So Data Deduplication is reporting a lot of corrupted files, and this error message didn’t really make me any happier.

Log Name:      Microsoft-Windows-Deduplication/Operational
Source:        Microsoft-Windows-Deduplication
Date:          5/21/2014 8:00:16 PM
Event ID:      4105
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Description:
Data Deduplication error: Unexpected error.


Operation:
   Add a merged target chunk store container.
   Reconciling duplicate chunks in the chunk store.
   Running the deduplication job.

Context:
   File name: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream\002b0000.00000002.ccc
   File name: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream\002b0000.00000001.ccc
   Chunk store: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream
   Volume name: E: (\\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\)

Log Name: Microsoft-Windows-Deduplication/Operational

Source: Microsoft-Windows-Deduplication

Date: 5/21/2014 8:00:16 PM

Event ID: 4105

Task Category: None

Level: Error

Keywords:

User: SYSTEM

Description:

Data Deduplication error: Unexpected error.

Operation:

Add a merged target chunk store container.

Reconciling duplicate chunks in the chunk store.

Running the deduplication job.

Context:

File name: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream\002b0000.00000002.ccc

File name: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream\002b0000.00000001.ccc

Chunk store: \\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\System Volume Information\Dedup\ChunkStore\{D2110CC9-B0D2-4F6A-B391-0680AE13C77B}.ddp\Stream

Volume name: E: (\\?\Volume{d44b1e9d-922f-46b7-965b-05d184cc6b74}\)

Hopefully this quick and dirty powershell script that I just wrote can help you too.
As I still had the working fileserver with working files available, I decided to just delete all corrupted files with this script.

$corrupted = Get-WinEvent -LogName Microsoft-Windows-Deduplication/Scrubbing | where ID -EQ "12800" 

$msg = $corrupted | select Message 
$file = $msg -replace ". The corruption cannot be repaired.}",""
$files = $file -replace "@{Message=Data Deduplication service detected corruption in ",""

foreach ($file in $files) {
  Get-Item ($file.Remove(0,1) -replace ".$") | Remove-Item
}

$corrupted = Get-WinEvent -LogName Microsoft-Windows-Deduplication/Scrubbing | where ID -EQ "12800"

$msg = $corrupted | select Message

$file = $msg -replace ". The corruption cannot be repaired.}",""

$files = $file -replace "@{Message=Data Deduplication service detected corruption in ",""

foreach ($file in $files) {

Get-Item ($file.Remove(0,1) -replace ".$") | Remove-Item

}

And then ran a robocopy script to recopy everything (it will skip any files that already exists making it a quite fast process).
robocopy /mir /copyall /r:1 /w:1 \\source\path \\destination\path

Updated 2014-05-22 16:22: Added a full delete and copy script, which is a bit better written;

$corrupted = Get-WinEvent -LogName Microsoft-Windows-Deduplication/Scrubbing | where ID -EQ "12800" | Sort-Object -Property Message

$sourcedir="\\server\share$\"

foreach ($entry in $corrupted) {
#  Write-Output $entry
   $file = "$((($entry | select -ExpandProperty Properties)[4]).value)$((($entry | select -ExpandProperty Properties)[5]).value)$((($entry | select -ExpandProperty Properties)[6]).value)" 
   write-output "Deleting file: $file" 
   Get-Item $file -ErrorAction SilentlyContinue | Remove-Item -Force
   write-output "Copying file: $file" 
   Copy-Item "$($sourcedir)$((($entry | select -ExpandProperty Properties)[5]).value)$((($entry | select -ExpandProperty Properties)[6]).value)" -Destination $file -Force 
}

$corrupted = Get-WinEvent -LogName Microsoft-Windows-Deduplication/Scrubbing | where ID -EQ "12800" | Sort-Object -Property Message

$sourcedir="\\server\share$\"

foreach ($entry in $corrupted) {

# Write-Output $entry

$file = "$((($entry | select -ExpandProperty Properties)[4]).value)$((($entry | select -ExpandProperty Properties)[5]).value)$((($entry | select -ExpandProperty Properties)[6]).value)"

write-output "Deleting file: $file"

Get-Item $file -ErrorAction SilentlyContinue | Remove-Item -Force

write-output "Copying file: $file"

Copy-Item "$($sourcedir)$((($entry | select -ExpandProperty Properties)[5]).value)$((($entry | select -ExpandProperty Properties)[6]).value)" -Destination $file -Force

}

Azure Pack: Failed to load virtual machine templates for subscription …

Problem: When a user login to Azure Pack, they sometimes get the error message: “Failed to load virtual machine templates for subscription <subscription ID>”.
And if the user tries to deploy a a Virtual Machine, there is no templates to choose from.

Cause: I’m not sure what the real cause is, but it seems to be a bug where Azure Pack forgets that information. The template information is there, it’s just Azure Pack that does not read it.

Workaround: Until this is solved by Microsoft in a hotfix or next updated you will have to handle this by yourself.
You can as an Administrator touch the Plans so they are re-synced and it will immediately start working again. Or you can schedule a powershell script to run regularly, touching the plans.

Here is the powershell command I’ve setup for our environment.

# Fill in values
# Address to your ADFS Server 
$adfsAddress = 'https://adfs.server.com'

# Address to Admin Sites AdminAPI
$adminUri = 'https://admin.server.com:30004'

# Username to connect to the server 
$username = 'domain\username'

# Use this command once to generate the passwords file
# read-host -assecurestring | convertfrom-securestring | out-file C:\path\to\passwords\file\password.txt 
$password = get-content C:\path\to\passwords\file\password.txt | convertto-securestring

$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $username,$password

$token = Get-AdfsToken -adfsAddress $adfsAddress -credential $credentials 
$plans = Get-MgmtSvcPlan -AdminUri $adminUri -Token $token -DisableCertificateValidation:$true 

# Loop though all plans and sync them 
foreach ($id in $plans) { 
  write-host Syncing $id.DisplayName
  Sync-MgmtSvcPlan -AdminUri $adminUri -Token $token -DisableCertificateValidation:$true -PlanId $id.id
  }

# Fill in values

# Address to your ADFS Server

$adfsAddress = 'https://adfs.server.com'

# Address to Admin Sites AdminAPI

$adminUri = 'https://admin.server.com:30004'

# Username to connect to the server

$username = 'domain\username'

# Use this command once to generate the passwords file

# read-host -assecurestring | convertfrom-securestring | out-file C:\path\to\passwords\file\password.txt

$password = get-content C:\path\to\passwords\file\password.txt | convertto-securestring

$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $username,$password

$token = Get-AdfsToken -adfsAddress $adfsAddress -credential $credentials

$plans = Get-MgmtSvcPlan -AdminUri $adminUri -Token $token -DisableCertificateValidation:$true

# Loop though all plans and sync them

foreach ($id in $plans) {

write-host Syncing $id.DisplayName

Sync-MgmtSvcPlan -AdminUri $adminUri -Token $token -DisableCertificateValidation:$true -PlanId $id.id

}

First of all, notice that it’s using a file for the password, to make this automatic.
Use this command once to creat the password.txt file

read-host -assecurestring | convertfrom-securestring | out-file C:\path\to\passwords\file\password.txt

1	read-host -assecurestring \| convertfrom-securestring \| out-file C:\path\to\passwords\file\password.txt

And it’s using the Get-AdfsToken function to get the a ADFS Token from our ADFS Server (more info: Get-AdfsToken Function), but you can modify the above script to use a normal Windows Token too if you rather want to use the Windows Authentication site than ADFS. Then use this command (replace line 17 in the script above with this line):

$Token = Get-MgmtSvcToken –Type Windows –AuthenticationSite https://yourauthenticationsite:30072 – ClientRealm http://azureservices/AdminSite -User $Credentials -DisableCertificateValidation

1	$Token = Get-MgmtSvcToken –Type Windows –AuthenticationSite https://yourauthenticationsite:30072 – ClientRealm http://azureservices/AdminSite -User $Credentials -DisableCertificateValidation

Get-MgmtSvcToken to get a ADFS Token is broken

Last week, I spent hours trying to get Get-MgmtSvcToken to get a Admin Token from our ADFS server without succeeding.

Get-MgmtSvcToken

Creates an identity token.

Syntax
Parameter Set: Default
Get-MgmtSvcToken [-Type] TokenType {Adfs | Membership | MembershipAdfs | Windows | WindowsAdfs} [-AuthenticationSite] Uri [-ClientRealm] Uri [-AdfsAddress Uri ] [-AdfsRealm Uri ] [-DisableCertificateValidation] [-User PSCredential ] [ CommonParameters ]

				1
2

						Parameter Set: Default
Get-MgmtSvcToken [-Type] TokenType {Adfs | Membership | MembershipAdfs | Windows | WindowsAdfs} [-AuthenticationSite] Uri [-ClientRealm] Uri [-AdfsAddress Uri ] [-AdfsRealm Uri ] [-DisableCertificateValidation] [-User PSCredential ] [ CommonParameters ]

I tried every possible combination with both “-type WindowsADFS” and “-type ADFS” in combination with various URL’s that should have worked, but didn’t.

With the help of @vNiklas and @_marcvaneijk on Twitter, I was pointed to TechNet where there is a documented bug/error/problem with the Get-mgmtSvcToken command.
By writing this blog, I hope someone will find it through a search and save themselves some time as that TechNet article never showed up when I was searching.

Technet Article: Why can’t I get a token with the Get-MgmtSvcToken cmdlet?

And the solution is to use your own function instead like this;

function Get-AdfsToken([string]$adfsAddress, [PSCredential]$credential)
{
    $clientRealm = 'http://azureservices/AdminSite'
    $allowSelfSignCertificates = $true

    Add-Type -AssemblyName 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
    Add-Type -AssemblyName 'System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

    $identityProviderEndpoint = New-Object -TypeName System.ServiceModel.EndpointAddress -ArgumentList ($adfsAddress + '/adfs/services/trust/13/usernamemixed')
    $identityProviderBinding = New-Object -TypeName System.ServiceModel.WS2007HttpBinding -ArgumentList ([System.ServiceModel.SecurityMode]::TransportWithMessageCredential)
    $identityProviderBinding.Security.Message.EstablishSecurityContext = $false
    $identityProviderBinding.Security.Message.ClientCredentialType = 'UserName'
    $identityProviderBinding.Security.Transport.ClientCredentialType = 'None'

    $trustChannelFactory = New-Object -TypeName System.ServiceModel.Security.WSTrustChannelFactory -ArgumentList $identityProviderBinding, $identityProviderEndpoint
    $trustChannelFactory.TrustVersion = [System.ServiceModel.Security.TrustVersion]::WSTrust13

    if ($allowSelfSignCertificates)
    {
        $certificateAuthentication = New-Object -TypeName System.ServiceModel.Security.X509ServiceCertificateAuthentication
        $certificateAuthentication.CertificateValidationMode = 'None'
        $trustChannelFactory.Credentials.ServiceCertificate.SslCertificateAuthentication = $certificateAuthentication
    }

    $ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($credential.Password)
    $password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($ptr)
    [System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($ptr)

    $trustChannelFactory.Credentials.SupportInteractive = $false
    $trustChannelFactory.Credentials.UserName.UserName = $credential.UserName
    $trustChannelFactory.Credentials.UserName.Password = $password #$credential.Password

    $rst = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityToken -ArgumentList ([System.IdentityModel.Protocols.WSTrust.RequestTypes]::Issue)
    $rst.AppliesTo = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.EndpointReference -ArgumentList $clientRealm
    $rst.TokenType = 'urn:ietf:params:oauth:token-type:jwt'
    $rst.KeyType = [System.IdentityModel.Protocols.WSTrust.KeyTypes]::Bearer

    $rstr = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityTokenResponse

    $channel = $trustChannelFactory.CreateChannel()
    $token = $channel.Issue($rst, [ref] $rstr)

    $tokenString = ([System.IdentityModel.Tokens.GenericXmlSecurityToken]$token).TokenXml.InnerText;
    $result = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($tokenString))
    return $result
}

# Fill in values
$adfsAddress = 'https://adfshost'
$username = 'domain\username'
$password = 'password'
$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username,$securePassword

$token = Get-AdfsToken -adfsAddress $adfsAddress -credential $credential 
$token

function Get-AdfsToken([string]$adfsAddress, [PSCredential]$credential)

{

$clientRealm = 'http://azureservices/AdminSite'

$allowSelfSignCertificates = $true

Add-Type -AssemblyName 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

Add-Type -AssemblyName 'System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

$identityProviderEndpoint = New-Object -TypeName System.ServiceModel.EndpointAddress -ArgumentList ($adfsAddress + '/adfs/services/trust/13/usernamemixed')

$identityProviderBinding = New-Object -TypeName System.ServiceModel.WS2007HttpBinding -ArgumentList ([System.ServiceModel.SecurityMode]::TransportWithMessageCredential)

$identityProviderBinding.Security.Message.EstablishSecurityContext = $false

$identityProviderBinding.Security.Message.ClientCredentialType = 'UserName'

$identityProviderBinding.Security.Transport.ClientCredentialType = 'None'

$trustChannelFactory = New-Object -TypeName System.ServiceModel.Security.WSTrustChannelFactory -ArgumentList $identityProviderBinding, $identityProviderEndpoint

$trustChannelFactory.TrustVersion = [System.ServiceModel.Security.TrustVersion]::WSTrust13

if ($allowSelfSignCertificates)

{

$certificateAuthentication = New-Object -TypeName System.ServiceModel.Security.X509ServiceCertificateAuthentication

$certificateAuthentication.CertificateValidationMode = 'None'

$trustChannelFactory.Credentials.ServiceCertificate.SslCertificateAuthentication = $certificateAuthentication

}

$ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($credential.Password)

$password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($ptr)

[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($ptr)

$trustChannelFactory.Credentials.SupportInteractive = $false

$trustChannelFactory.Credentials.UserName.UserName = $credential.UserName

$trustChannelFactory.Credentials.UserName.Password = $password #$credential.Password

$rst = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityToken -ArgumentList ([System.IdentityModel.Protocols.WSTrust.RequestTypes]::Issue)

$rst.AppliesTo = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.EndpointReference -ArgumentList $clientRealm

$rst.TokenType = 'urn:ietf:params:oauth:token-type:jwt'

$rst.KeyType = [System.IdentityModel.Protocols.WSTrust.KeyTypes]::Bearer

$rstr = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityTokenResponse

$channel = $trustChannelFactory.CreateChannel()

$token = $channel.Issue($rst, [ref] $rstr)

$tokenString = ([System.IdentityModel.Tokens.GenericXmlSecurityToken]$token).TokenXml.InnerText;

$result = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($tokenString))

return $result

}

# Fill in values

$adfsAddress = 'https://adfshost'

$username = 'domain\username'

$password = 'password'

$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force

$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username,$securePassword

$token = Get-AdfsToken -adfsAddress $adfsAddress -credential $credential

$token

Moving your WordPress site running in Microsoft Azure to Azure Pack

Drum Roll please ….

I’ve now moved my blog from Microsoft Azure to our own Private Cloud running on Windows Azure Pack! Hurray!

It was a really smooth transition, involving a couple of very easy steps to maintain all blogposts, media and comments.

Provision a new WordPress Blog in Azure Pack.
Login to WordPress to finish the installation.
Upgrade WordPress (to the same version as your old wordpress site).
Add the same Themes as on your old site.
Install WP Clone plugin (free) on both the old and the new wordpress site.
On the old blog, use the WP Clone plugin to make a Backup. When it’s done, you will get a URL, save that.
On the new blog, use the WP Clone plugin to import the Backup from the URL you got in the previous step.
In the new site, update your sites URL’s. In my case, I chanced isolation.labcenteronline.com to www.isolation.se.
Update your DNS points, to point to your private cloud url (in my case, isolation.labcenteronline.com).
Done!