Configuring HP ILO Settings and TLS Certificates With Powershell

I've been configuring HP ILOs lately. And of course, the cardinal rule in I.T. is that if you're going to do something more than once, then you must start automating it.  And of course, if you want to automate something, then you fire up Powershell.

Luckily, HP is playing ball with the HP Scripting Tools for Windows Powershell. The cmdlets are not half bad, either. Essentially, what I needed to do was configure a bunch of ILOs, including renaming them, setting some IPv6 settings, and putting valid SSL/TLS certificates on them.

First, let's save the ILOs address (or hostname,)  username and password for future use:

[String]$Device   = '10.1.2.3'
[String]$Username = 'Admin'
[String]$Password = 'P@ssword'

Next, let's turn off IPv6 SLAAC and DHCPv6 (for ILO 3s, firmware ~1.28 or so, and above):

Set-HPiLOIPv6NetworkSetting -Server $Device `
                            -Username $Username `
                            -Password $Password `
                            -AddressAutoCfg Disable

Set-HPiLOIPv6NetworkSetting -Server $Device `
                            -Username $Username `
                            -Password $Password `
                            -DHCPv6Stateless Disable

Set-HPiLOIPv6NetworkSetting -Server $Device `
                            -Username $Username `
                            -Password $Password `
                            -DHCPv6Stateful Disable

Next I wanted to set the FQDN to what I wanted it to be... it was important that I turned DHCP off first, because the ILO wanted to set the domain name using DHCP and thus locked it from being edited, even though no DHCP server was actually on the network:

Set-HPiLOServerName -Server $Device `
                    -Username $Username `
                    -Password $Password `
                    -ServerName 'server1-ilo.contoso.com'

Now I wanted to put a valid SSL/TLS certificate on the ILO. So, I needed to first generate a Certificate Signing Request (CSR) on the ILO:

Get-HPiLOCertificateSigningRequest -Server $Device `
                                   -Username $Username `
                                   -Password $Password

IP                          : 10.1.2.3
HOSTNAME                    : server1-ilo.contoso.com
STATUS_TYPE                 : OK
STATUS_MESSAGE              : OK
CERTIFICATE_SIGNING_REQUEST : -----BEGIN CERTIFICATE REQUEST-----
                              a1b2c3d4e5A0B1C2D3F4E5
                              3ba43+/evnokaDvzG9nbs3
                              a1b2c3d4e5A0B1C2D3F4E=
                              -----END CERTIFICATE REQUEST-----

Nice... now copy and paste the entire CSR text block, including the -----BEGIN and END------ bits, and submit that your certificate authority.  Then, the administrator of the certificate authority has to approve the request.

This is the one piece where automation breaks down, in my opinion, and some manual intervention is necessary. This is not a technical limitation, though... it's by design.  The idea is that the entire basis of SSL/TLS public key cryptography is that it's based on trust.  And that trust has to come from other sources such as the Certificate Authority administrator phoning the requestor and verifying that it was actually them making the request, or getting some additional HR info, or whatever.  If, at the end of the day, there was no extraordinary measure taken to really verify the requestor's identity, then you can't really trust these certificates.

Anyway, once the CA has signed your CSR, you just need to import the signed certificate back into the ILO:

Import-HPiLOCertificate -Server $Device `
                        -Username $Username `
                        -Password $Password `
                        -Certificate (Get-Content C:\mycert.cer -Raw) 

Assuming no errors were returned, then you're done and your HP ILO will now reboot, and when it comes back up, will be using a valid SSL certificate.

Also, HP ILOs cannot read certificates if they are using PKCS #1 v2.1 format. Add that to the huge pile of devices that cannot read an X509 standard that came out in 2003.

Testing Authenticated NTP Configurations

It's been a long time since I posted, I know. I've been busy both with actual work, and also working on a personal project that involves getting way better at old-school C programming. I've been following Casey Muratori and his Handmade Hero series for a couple of months now, and I've been really inspired to write more C.

More on what I've been working on as it develops.

Also, my friend and co-conspirator Wesley sent me a gift for accomplishing the "Serverfault 10k Challenge" in 2014:

This majestic creature embodies a never-ending thirst for knowledge and symbolizes the triumphs and tribulations of the sysadmin.  It's not a Microsoft MVP award...

... it's better.

Alright so the topic of today's post: Authenticated NTP.  NTP is one of the very oldest protocols on the internet, it has had very few vulnerabilities reported over its 30+ year lifespan, and is ubiquitous in virtually every computer network on the planet. (Because most computers are awful at keeping time.) There are many different versions of it and spinoffs from the reference implementation. People tend to find NTP a boring protocol, but it's one of my favorite internet protocols.  Most people just point their router at pool.ntp.org and never think about NTP again.

Until the day comes that you want to enable authentication in your NTP architecture. Authentication is the mechanism that allows a message recipient to verify that the response came from the intended sender and wasn't tampered with in transit. Within a Windows Active Directory domain, we already have this. Domain members use the Kerberos session keys they already have to create authenticated Windows Time messages with domain controllers.  But that means if you're not a member of the domain, you can't participate.

The administrator of an NTP server who wishes to send you authenticated NTP messages will probably send you an ntp.keys file, or at least a password for you to use. An example ntp.keys file looks like this:

# ntpkey_MD5key_ntp.domain.com.2825130701
# Thu Jan 15 20:00:01 2015
 1 MD5  JzF&f})0ocK1{H9	# MD5 key
 2 MD5  Dv(0v@W8vJ8%#*2	# MD5 key
 3 MD5  N(BzeyvYx$qzs5]	# MD5 key
 4 MD5  TVd2*DXtu-mewLs	# MD5 key
 5 MD5  F9UTa)8AQ9O9561	# MD5 key
 6 MD5  F9}{%$d9vs3Dpxb	# MD5 key
 7 MD5  D]Z*OOr56ukpiD6	# MD5 key
 8 MD5  TTr$OIR9+f74J28	# MD5 key
 9 MD5  EC3F9Zr%-3190&0	# MD5 key
10 MD5  Ndi5+]F^3x3Gdeb	# MD5 key
11 MD5  S+27&8(ba30qM@5	# MD5 key
12 MD5  CnO8)=CyG)QBj]}	# MD5 key
13 MD5  Em62oK!RXhw#y9_	# MD5 key
14 MD5  K-l(^UE@&T(Zj5B	# MD5 key
15 MD5  Gcff1nJb(CuF$*!	# MD5 key
16 MD5  W-*5^xbp3@v8br)	# MD5 key

There aren't any tools that ship with Windows that will help you test this stuff.  The Windows implementation of NTP ("Windows Time") is good enough to keep Windows working within the context of an AD domain, but it's not a full-featured reference NTP implementation.  So, I downloaded the Windows port of NTP from ntp.org. You can use the ntpdate program to query an NTP server using the shared secrets in your ntp.keys file to verify that authentication is successful:

C:\Users\Ryan>ntpdate.exe -b -d -k C:\Users\Ryan\ntp.keys -a 1 ntp.domain.com
...
transmit(70.144.88.104)
receive(70.144.88.104)
receive: authentication passed
transmit(70.144.88.104)
receive(70.144.88.104)
receive: authentication passed
...
server 70.144.88.104, port 123
stratum 2, precision -20, leap 00, trust 000
refid [70.144.88.104], delay 0.03090, dispersion 0.00066
transmitted 4, in filter 4
...
Authenticated NTP: Check. I'll probably write more about this topic in the future, but I have to perform some experiments first.

One Way of Exporting Nicer CSVs with Powershell

One of the ever-present conundrums in working with computers is that data that looks good and easily readable to a human, and data that is easy and efficient for a computer to process, are never the same.

In Powershell, you see this "immutable rule" manifest itself in that, despite all the various Format-* cmdlets available to you, some data will just never look good in the console.  And if it looks good in the console, chances are you've mangled the objects so that they've become useless for further processing over the pipeline.  This is essentially one of the Powershell "Gotcha's" espoused by Don Jones, a term that he refers to as "Format Right."  The principal is that if you are going to format your Powershell output with a Format-* cmdlet, you should always do so at the end of the statement (e.g., on the right side.)  The formatting should be the last thing you do in an expression, and you should never try to pass something that has been formatted over the pipeline.

CSV files, in my opinion, are a kind of happy medium, because they are somewhat easy for humans to read (especially if the human has an application like Microsoft Excel or some such,) and CSV files are also relatively easy for computers to read and process.  Therefore, CSVs are a popular format for transporting data and feeding it to computers, while still being legible to humans.

When you use Export-Csv to write a bunch of objects out to a CSV file:

# Get Active Directory groups, their members, and memberships:
Get-ADGroup -Filter * -SearchBase 'CN=Users,DC=domain,DC=local' -Properties Members,MemberOf | `
    Select Name, Members, MemberOf | `
    Export-Csv -NoTypeInformation -Path C:\Users\ryan\Desktop\test.csv 

And those objects contain arrays or lists as properties, you'll get something like this in your CSV file:

"Name","Members","MemberOf"
"MyGroup","Microsoft.ActiveDirectory.Management.ADPropertyValueCollection","Microsoft.ActiveDirectory.Management.ADPropertyValueCollection"

Uh... that is not useful at all.  What's happened is that instead of outputting the contents of the Active Directory group members and memberOf attributes, which are collections/arrays, Powershell has instead output only the names of the .NET types of those collections.

What we need is a way to expand those lists so that they'll go nicely into a CSV file.  So I usually do something like the script excerpt below.  This is just one possible way of doing it; I by no means claim that it's the best way or the only way.

#Get all the AD groups:
$Groups = Get-ADGroup -Filter * -SearchBase 'OU=MyOU,DC=domain,DC=com' -Properties Members,MemberOf

#Create/initialize an empty collection that will contain a collection of objects:
$CSVReadyGroups = @()

#Iterate through each one of the groups:
Foreach ($Group In $Groups)
{
    #Create a new object to hold our "CSV-Ready" version of the group:
    $CSVReadyGroup = New-Object System.Object #Should probably be a PSObject
    #Add some properties to the object.
    $CSVReadyGroup | Add-Member -Type NoteProperty -Name 'Name'     -Value  $Group.Name
    $CSVReadyGroup | Add-Member -Type NoteProperty -Name 'Members'  -Value  $Null
    $CSVReadyGroup | Add-Member -Type NoteProperty -Name 'MemberOf' -Value  $Null

    # If the group has any members, then run the code inside these brackets:
    If ($Group.Members)
    {
        # Poor-man's serialization.
        # We are going to convert the array into a string, with NewLine characters 
        # separating each group member. Could also be more concise just to cast
        # as [String] and do  ($Group.Members -Join [Environment]::NewLine)

        $MembersString = $Null
        Foreach ($GroupMember In $Group.Members)
        {
            $MembersString += $GroupMember + [Environment]::NewLine
        }
        #Trim the one extra newline on the end:
        $MembersString = $MembersString.TrimEnd([Environment]::NewLine)
        #Add to our "CSV-Ready" group object:
        $CSVReadyGroup.Members = $MembersString
    }

    # If the group is a member of any other groups, 
    # then do what we just did for the Members:
    If ($Group.MemberOf)
    {
        $MemberOfString = $Null
        Foreach ($Membership In $Group.MemberOf)
        {
            $MemberOfString += $Membership + [Environment]::NewLine
        }
        $MemberOfString = $MemberOfString.TrimEnd([Environment]::NewLine)
        $CSVReadyGroup.MemberOf = $MemberOfString
    }

    #Add the object we've created to the collection:
    $CSVReadyGroups += $CSVReadyGroup
}

#Output our collection:
$CSVReadyGroups | Export-Csv -NoTypeInformation -Path C:\Users\ryan\Desktop\test.csv

Now you will have a CSV file that has readable arrays in it, that looks good when you open it with an application such as Excel.

Have You Been Pwned by CVE-2014-6324/MS14-068?

In case you haven't heard, there is a critical [Windows implementation of] Kerberos bug that you need to be updating, right now.

More information on the vulnerability can be found here.

In the "Detection Guidance" section of the above blog post, you will see that you can detect if the vulnerability has been exploited on an unpatched machine by analyzing the Security event logs. Specifically, looking at Event ID 4624 logon events, and taking note that the "Security ID" and "Account Name" fields in that event description match.  If they don't, chances are high that you have been a victim of a privilege escalation attack.

I whipped up a detection script to check all the domain controllers:

#Requires -Module ActiveDirectory
Set-StrictMode -Version Latest
Get-Job | Remove-Job -Force
[String]$DomainName = $(Get-ADDomain).Name
$DCs = $(Get-ADDomain).ReplicaDirectoryServers

:NextDC Foreach ($DC In $DCs)
{
    Start-Job -ScriptBlock {
        Param($DC)
        [Int]$PotentialMS14068s = 0
        Write-Output "Fetching Security event log from $DC."
        Try
        {
            $Events = Get-EventLog -LogName Security -InstanceId 4624 -ComputerName $DC -ErrorAction Stop
        }
        Catch
        {
            Write-Error "An error occurred while reading event log from $DC.`r`n$($_.Exception.Message)"
        }

        :NextEvent Foreach ($Event In $Events)
        {            
            $MessageLines  = $Event.Message -Split [Environment]::NewLine
            
            [String]$SecurityID    = [String]::Empty
            [String]$AccountName   = [String]::Empty
            [String]$AccountDomain = [String]::Empty

            # Server 2012 Format
            If ($MessageLines[13].Trim() -Like 'Security ID:*')
            {
                $SecurityID    = ($MessageLines[13].Trim() -Split ':')[1].Trim()
                $AccountName   = ($MessageLines[14].Trim() -Split ':')[1].Trim()
                $AccountDomain = ($MessageLines[15].Trim() -Split ':')[1].Trim() 
            }
            
            # Server 2008 R2 Format
            If ($MessageLines[11].Trim() -Like 'Security ID:*')
            {
                $SecurityID    = ($MessageLines[11].Trim() -Split ':')[1].Trim()
                $AccountName   = ($MessageLines[12].Trim() -Split ':')[1].Trim()
                $AccountDomain = ($MessageLines[13].Trim() -Split ':')[1].Trim()
            }

            If (($SecurityID -EQ [String]::Empty) -OR ($AccountName -EQ [String]::Empty) -OR ($AccountDomain -EQ [String]::Empty))
            {
                Write-Error "Event log message format unrecognized on $DC!"
                $Event | Format-List
                Break NextEvent
            }

            If ($AccountDomain -Like $DomainName -And $SecurityID -NotLike 'S-1-5-18')
            {
                $SID = New-Object System.Security.Principal.SecurityIdentifier($SecurityID)
                $Username = $SID.Translate([System.Security.Principal.NTAccount])        
                If ($Username -Like '*\*')
                {
                    $Username = ($Username -Split '\\')[-1]
                }
                If ($Username -Like '*@*')
                {
                    $Username = ($Username -Split '@')[0]
                }
                If ($Username -NE $AccountName)
                {
                    $Event | Format-List
                    $PotentialMS14068s++
                }        
            }
        }
        Write-Output "Finished with $DC. $PotentialMS14068s interesting events found."
    } -ArgumentList $DC
}

While ($(Get-Job -State Running).Count -GT 0)
{
    Get-Job -State Completed | Receive-Job   
    Start-Sleep -Seconds 10 
}

The script uses Powershell jobs to achieve some parallelism, because if you have more than one or two domain controllers in your environment, this quickly becomes a Herculean, time-consuming task.  The script will display potential security event log events that may indicate exploits currently being used in your environment.

Windows Server Technical Preview: Soft Reboot

Windows Server Technical Preview Desktop

Microsoft's technical preview of the next version of Windows Server has been out for a month or two now.  (Go download it, what are you waiting for?)  Is it Windows Server 10?  Server 2015?  I suppose they could try to get away from naming it altogether and just call it "Windows Server," to signify that they only plan on evolving the platform incrementally from now on, rather than using the traditional punctuated equilibrium of boxed product releases that we're used to... that's for Microsoft to know and for us to find out.

 

One of the fun things about public CTP releases is that the thorough documentation always comes last... so we download these tech previews, and we see all these new features, and a lot of them are not well documented, if they're documented at all.  And that makes them great blog fodder.  So let us begin a journey through these poorly-documented features, starting with a new feature called Soft Restart.

The promise of this simple feature is to allow server administrators to restart or reboot the Windows operating system on a physical computer, without having to wait through the long and annoying process of the machine's POST, initializing RAID controllers, out-of-band management devices, network adapters, etc.  On some physical server hardware, this process can take several minutes just to come back from a reboot.

However, on a virtual machine, this feature is not likely to save you much time, however, since the virtualized/synthetic/emulated devices on VMs don't typically have those long initialization procedures anyway.

You can install the Soft Restart feature via the GUI:

Soft Restart via GUI

Or by my preferred method, Powershell:

PS C:\> Install-WindowsFeature Soft-Restart -Restart

Installing the feature requires a reboot.

So, one thing you'll notice in the new Windows Server that we did not have before, is a new parameter - /soft - for the shutdown.exe program:

shutdown.exe soft restart

However, I find it interesting that this new parameter exists, and works, with or without the new Soft Restart feature actually being installed!

There is also a new Powershell equivalent: Restart-Computer -Soft. This also appears to work regardless of whether the Soft Restart feature is actually installed or not... but this may be because I only have the tech preview on virtual machines right now.  It could be a different story if I were playing on physical hardware.