Powershell Needful Things put that in your pipeline

14Oct/091

Collecting Exchange database white space from the event log using .NET

Posted by Jean Louw

A recent comment from a reader, prompted me to do some updates and bug fixes to my Exchange 2007 audit script. As a part of this process, I decided to add the white space count into the mailbox store check.

I discovered an extremely helpful post, as usual, from Shay Levy, which pointed me in the right direction.

Although this function does get exactly what I needed, I did however want to search for the white space by mailbox store name, in order to get the value, as each mailbox store was passed during the script processing.

I changed my script to use .NET instead of WMI for event logs so I decided to continue using this method for the white space as well.
The basic script to collect the white space sizes from the event log using .NET is as follows:

$now = Get-Date
$colMailboxStores = Get-MailboxDatabase -Server SERVER -Status | Sort-Object Name
$spaceLog=[System.Diagnostics.EventLog]::GetEventLogs('SERVER') | where {($_.LogDisplayName -eq "Application")}

foreach ($objMailboxStore in $colMailboxStores)
{
Write-Host "..Getting database white space for" $objMailboxStore.Name
$store = @{Name="store";Expression={$_.ReplacementStrings[1]}}
$freeMB = @{Name="freeMB";Expression={[int]$_.ReplacementStrings[0]}}
$whiteSpace = @()
$whiteSpace += $spaceLog.entries | where {($_.TimeWritten -ge $now.AddDays(-1))} | where {($_.EventID -eq "1221")} | where {($_.ReplacementStrings[1] -match $objMailboxStore.Name)} | select $store,$freeMB -last 1
$whiteSpace.freeMB
}

This method is very slow, as it has to dredge through the entire event log for every database. It’s really not a problem if you have a small number of databases, but in a large environment like ours, with multiple mailbox servers, this could take ages to complete.

It was was painful during testing to wait for the above script to complete and I really felt that the speed of this process should be increased, so instead I came up with the following solution:

$now = Get-Date
$spaceLog=[System.Diagnostics.EventLog]::GetEventLogs('SERVER') | where {($_.LogDisplayName -eq "Application")}
$db = @{Name="database";Expression={$_.ReplacementStrings[1]}}
$freeMB = @{Name="MB";Expression={[int]$_.ReplacementStrings[0]}}
$whiteSpace = $spaceLog.entries | where {($_.TimeWritten -ge $now.AddDays(-1))} | where {($_.EventID -eq "1221")} | select $db,$freeMB

($whitespace | where {$_.database -match $objMailboxStore.Name} | select -last 1).mb

The code above will collect all of the Event ID 1221’s for the last day and store them in a variable with the customised place holders from the expressions.
This happens once per server only and any subsequent searches can be performed against the variable instead.

The select statement at the end, also selects the last item in the list to ensure that you also look at the latest event for each database. This literally reduces the runtime of the script by a factor equal to the amount of databases on your server.

I will be posting an update to the Exchange 2007 audit script soon, so stay tuned.

6Oct/091

Measure the SMTP roundtrip time to an external email address

Posted by Jean Louw

In an attempt to be more proactive about Internet email delays, whether caused by our systems, or those of our ISP, I have written a script which tests the roundtrip time on SMTP mail.
The basic idea behind the script is to send a message with a GUID, and wait for the return of that specific message. When that message returns, it measures the roundtrip time, and logs the result to disk. If the message is not returned within 30 mins, it will send you a warning message informing you of the problem.

Finally, the script creates a nice JPG with the results up to the last run.

Setting up and using this script is a little more complex than usual as it combines different technologies and resources to achieve its goal, which is to measure the roundtrip time on an actual SMTP message.
To start off, the script sends a message using a standard .NET relay. On Powershell V2 you could use send-mailmessage instead. At this point, the message is time stamped in the subject, with the current date and time. The message is also marked with a distinguishable word “SMTPPing” for the reply rule, and a random GUID, which aids in recognising the message when it returns.

I use Gmail, as my “auto-reply” robot, as I am fairly certain that their infrastructure is robust and pretty stable. If you choose to use Gmail, you will need to setup a filter, which automatically forwards all mail with the word “SMTPPing” back to your email address, and then deletes it from Gmail.

Once you are sure that the auto reply is working, you can configure the script with your SMTP email addresses and relay host.

The return messages are collected from an Outlook mailbox using MAPI. You need to customise the script for the Outlook profile it needs to logon to. More details regarding this can be found as comments inside the script.

Outlook does not like strangers poking around in your stuff, so it will constantly warn you about this. To get around this problem, and also be a little selective about what you allow, you can download an awesome free tool from MapiLab called Advanced Outlook Security.

Lastly, the script needs Excel installed, in order for the chart creation and export to JPG.

I am not sure why, but I am having problems currently closing Excel. Although I issue the command to close the application, it sometimes remains running, so lookout for excel.exe in process monitor.

As usual, your comments and suggestions are always welcome.

If you like this script and use it in production, please help me keep this project free by considering a small donation.





The script can be downloaded from here:

30Jun/096

Cleanup unused Exchange 2007 mailboxes

Posted by Jean Louw

I often use my orphaned home directory cleanup script at work, to recover unused space from our file and print clusters. So my manager recently suggested that I do something similar for Exchange. Knowing that the orphan folder cleanup utility is still my responsibility as the administrators are not too comfortable with running scripts, I decided to give this utility a nice GUI.
To generate the code for the forms, I used SAPIEN PrimalForms. What beautiful tool. Very short learning curve, and very, very powerful. When the form loads, it will get a list of all the Exchange mailbox servers using get-mailboxserver.
This excludes Exchange 2003 servers as get-mailboxstatistics does not work with legacy mailboxes. I may develop a solution for that later. The three query buttons (Disabled, Hidden, Stale) will perform the following actions respectively: Disabled – Find mailboxes linked to disabled AD accounts Hidden – Find mailboxes hidden from the address book.  Stale – Find mailboxes linked to accounts which have not logged on in the last 3 months.
This search may take a little time to complete and this button is not supported against Exchange 2003 servers. These queries will populate the listbox with the names of the mailboxes. Besides the “Export List” button, the Action buttons at the bottom will action only selected items.
You can select items using SHIFT or CTRL. Export List will create a text file containing your search results. Export PST will grant the current user Full Mailbox with Send As and Receive As permission, and then export the mailbox to the path specified. Rename will change the display name based on the query performed. For mailboxes found with the “Disabled” button the display name will be prefixed with “DISABLED-MBXCleaner-“, for “Hidden” with “HIDDEN-MBXCleaner-“ and so forth. Users previously renamed will be excluded from subsequent searches. The “Disable” action will remove Exchange Attributes without deleting the AD account. The mailbox will be removed when the retention time expires. Delete will remove the mailbox and AD account completely.
I have not had a chance to test the Delete button as I would need to submit a change control request before using the utility in our live environment. All of the Action buttons are set to –whatif mode by default. The “Go Hot” checkbox will activate the heavy hitters (Export PST; Rename; Disable and Delete) so you can safely test each button first before taking any action. The “Reserved” button, currently, does nothing. I plan to allow this button to read or build a custom search for users, either by Name or other criteria.
 
WARNING: This is a dangerous utility, and can wreck your Exchange system if you are not careful. Please test this in your test environment first, and adhere to your change control procedures before using this utility in the live environment. I take absolutely no responsibility for any damage caused by using this tool. The utility requires the Exchange Management shell, and if launched from a Vista / Windows 7 needs to be “Run as Administrator”

Full information and the download location for this script can be found here:
http://www.powershellneedfulthings.com/?page_id=293

12Jun/090

Whats going on here?

Posted by Jean Louw

As part of the Exchange audit scripts, I recently changed the 2007 version of the script to use .NET to collect the event logs instead of WMI. Virtu-Al made an interesting suggestion, which was to say, which of these methods are quicker at collecting the logs. So in order to do this I needed to setup a race.
This race would basically involve the two methods of retrieval collecting a large list of events from a selected server. The basic command to accomplish this is as follows: For WMI one would simply use:
$wmi = Get-WmiObject -computer SERVER1 Win32_NTLogEvent  
Using .NET, it retrieves the actual Event Logs, so the entries have to be enumerated with a quick bit of code:
System.Diagnostics.EventLog]::GetEventLogs('SERVER1') ForEach ($eventLog in $eventLogs){ $dotNet += ($eventLog.entries)} 
In both cases, where SERVER1 is the name of the remote server you need to collect the events from. Now, in order to make sure that there is no cheating, I would have to count how many objects are returned by each method. This could be done by simply saving the collection to a variable and counting the total. So in this scenario, .NET would return approximately 56000 items and WMI would return less. About 500+ less every time. From here I went down a crazy path of checking date and time formats etc. and in the end, I came to the conclusion that it had to be the security log. Entries were being written into the Security Log so quickly, that by the time the 2nd script is run, the number of entries have changed, or I remembered that you needed special permissions to read certain Security Log entries. Or so I thought. So I decided to exclude the Security log from my collection. This was easy enough, but still the totals were inconsistent. In an effort to try and eliminate where the problem could be, I decided to include only one log at a time, starting with the Application Log. Here is the script used to collect the Application from a remote server using WMI:
$d1 = get-date

$wmiDate = [System.Management.ManagementDateTimeConverter]::ToDmtfDateTime([DateTime]::Now.AddDays(-1))
$WMI = Get-WmiObject -computer SERVER1 -query ("Select * from Win32_NTLogEvent Where Logfile = 'Application' and TimeWritten >='" + $WmiDate + "'")

$wmiCount = ($WMI).Count

$wmiDT = [System.Management.ManagementDateTimeConverter]::ToDateTime($wmiDate)
Write-Host From Date $wmiDT
Write-Host Total $wmiCount
$d2 = Get-Date
$d2 - $d1
WMI Script results: From Date 09/06/2009 01:28:49 PM Total 317 Here is the script used to collect the same event log entries from the same server, using .NET instead:
$d1 = get-date
$dotNetDate = ([DateTime]::Now.AddDays(-1))
$eventLogs=[System.Diagnostics.EventLog]::GetEventLogs('SERVER1') | where {$_.LogDisplayName -eq "Application"}
ForEach ($eventLog in $eventLogs ){

$dotNet += ($eventLog.entries) | where {($_.TimeWritten -ge $dotNetDate)}
}

$dotnetCount = ($dotNet).count

Write-Host From Date $dotNetDate
Write-Host Total $dotnetCount
$d2 = Get-Date
$d2 - $d1
.NET Script Results:
From Date 09/06/2009 01:28:49 PM Total 650
This was still very confusing so, to see exactly at which record the problem is, I had both scripts display the record number of the first and last record in each respective collection, by adding the following to each script: For the .NET script:
$dotNet | Select-Object -First 1 $dotNet | Select-Object -Last 1 For the WMI script: $WMI | Select-Object RecordNumber, TimeWritten, Type, SourceName, EventCode -First 1 $WMI | Select-Object RecordNumber, TimeWritten, Type, SourceName, EventCode -Last 1 
Now I could see that, at least they were starting at the same record, but for some odd reason, WMI was quitting before the job was done. .NET record results:
Index Time Type Source EventID ----- ---- ---- ------ ------- 51 Jun 09 14:55 Warn MSExchange Availa... 4004 705 Jun 10 14:51 Warn MSExchange Active... 1008 WMI Results: RecordNumber TimeWritten Type SourceName ------------ ----------- ---- ---------- 353 20090610012624.00000... Warning MSExchange ActiveSync RecordNumber TimeWritten Type SourceName ------------ ----------- ---- ---------- 51 20090609145522.00000... Warning MSExchange Availability
To make sure this problem wasn’t specific to the current server I started collecting logs from other servers, to record the results. I also did an add-member on the WMI script to convert the time and date back for easier reading. With the following string:
ForEach-Object { Add-Member -inputobject $_ -Name myTime -MemberType NoteProperty -Value ([System.Management.ManagementDateTimeConverter]::ToDateTime($_.TimeWritten)) -Force -PassThru} 
Over a number of servers this still made no difference, WMI still did not return all the results. This seems to be a problem specific to the Application and Security Log, and could well be related to the WMI impersonation or authentication which will be available in version 2.
This I have not had time to investigate. I decided to re-write the WMI script to collect all results and then filter out the unwanted events with “where-object”. At this point I also changed the selected log to the system event log, as someone cleared the application logs on the selected servers.
This worked great for most of the servers and finally I was getting similar results from both scripts. I did however find, that servers with large numbers of events generate a WMI Quota Violation, which seems to imply that there are too many items in the list, which is yet another blow to WMI.
This could also explain the incomplete results from previous attempts. The Quota Violation is a known problem and there is a resolution for it posted here: http://support.microsoft.com/kb/828653. To get around this problem, I changed the script again, to use the WMI query. So now that we were getting results, it was time to start testing the speed of each method.
I decided to test the speed against 3 different servers, and increment the number of records retrieved until I could not collect anymore, or up to a maximum of 240 days worth of events.
I decided to also give each method and average read time over 3 attempts.
Here are some of the results:

As the amout of days, or number of records increase, the read speed of WMI starts decreasing.

In summary, WMI scales nicely when using a WMI query directly in the Get-WMIObject command. It does however loose speed as the number of records to retrieve start increasing.
It has to be mentioned, that WMI slows down to a crawl, if all records are retrieved and the result is filtered with “where-object”.
Although WMI is faster with less records, I am going base all my event log queries on .NET for now, as WMI proved to be inconsistent and erroneous in what retrieves, or atleast in my testing it did.
I hope that this problem is related to impersonation, and that it is resolved in Powershell v2. The final scripts I used to retrieve the information can be downloaded from here: