DNS over HTTP

I was discussing with some fellow IT admins, the topic of blocking certain websites so that employees or students couldn't access them from the work or school network.  This is a pretty common topic for IT in most workplaces.  However, I personally don't want to be involved in it.  I realize that at some places, like schools for instance, filtering of some websites may be a legal or policy requirement.  But at the workplace, if an employee wants to waste company time on espn.com, that is an issue for HR and management to take up with that employee.  And again in my opinion, it's not about how much time an employee spends on ESPN or Reddit either, but simply whether that employee delivers satisfactory results.  I don't want to handle a people problem with a technical solution.  I don't want to be the IT guy that derives secret pleasure from blocking everyone from looking up their fantasy football scores.  (Or whatever it is people do on espn.com.)  I could spend my entire career until I retire working on a web proxy, blocking each and every new porn site that pops up.  If there's one thing the internet has taught me, it's that there will always be an infinite number of new porn sites.

On the other extreme of black listing, someone then suggested white listing.  Specifically, implementing "DNS white listing" in their environment for the purpose of restricting what internet sites users were allowed to access to only a handful of internet sites.  Well that is a terrible idea.  The only proper way of doing this in my opinion is to use a real web proxy, such as ISA or TMG or Squid.  But I could not help but imagine how I might implement such a system, and then how I might go about circumventing it from the perspective of a user.

OK, well for my first half-baked idea, I can imagine standing up a DNS server, disabling recursion/forwarders on that DNS server, and putting my "white list" of records on that DNS server.  Then, by way of firewall, block all port 53 access to any other IP except my special DNS server.  Congratulations, you just made your users miserable, and have done almost nothing to actually improve the security of your network or prevent people from accessing other sites.  Now the users just have to find another way of acquiring IP addresses for sites that aren't on your white list.

Well how do I get name resolution back if I can't use my DNS server?  I have an idea... DNS over HTTP!

The guys at StatDNS have already thought about this.  And what's awesome, is that they've created a web API for resolving names to IPs over HTTP.  Here's what I did in 5 minutes of Powershell:

PS C:\> Function Get-ARecordOverHTTP([string]$Query) { $($($(Invoke-WebRequest http://api.statdns.com/$Query/a).Content | ConvertFrom-Json).Answer).rdata }

PS C:\> Get-ARecordOverHTTP google.com
173.194.70.101
173.194.70.100
173.194.70.138
173.194.70.102
173.194.70.139
173.194.70.113

PS C:\> Get-ARecordOverHTTP myotherpcisacloud.com
168.61.52.184

Simple as that. How cool is Powershell, seriously?  One line to create a function that accepts a name and returns a list of IPs by interacting with an internet web service.  Pretty awesome if you ask me.

As long as you have port 80 open to StatDNS, you have internet name resolution.  Now, to wrap this into a .NET-based Windows service...

Why Does It Say <Unknown Contact> When Viewing Network Share Permissions?

I only get to work with Active Directory trusts every so often. I think multi-domain forests seem to be falling out of fashion. At least for those of us who've heard of federation. Regardless, here's an interesting issue I ran into the other day:

So I have this domain, contoso.com.  In contoso.com, there is a one-way forest trust established with fabrikam.com, such that contoso.com trusts fabrikam.com.  This way, users in fabrikam.com can access resources in contoso.com.

On pc1.contoso.com, I create a network file share. I add permissions for my own user account, contoso\ryan, to it.  Then, I add permissions for fabrikam\steve to access the file share also.  The operation was successful and the file share appears to be set correctly.  However, when I go back and view the permissions for the file share again, it takes a very long time, as if it were waiting on something to time out, and then eventually, this is what I see:

File Share Permissions

So why is fabrikam\steve showing up as <Unknown Contact> when viewed from the contoso.com domain?  What we have here, is a SID translation failure.  But why?  First, a little background.  Here is what Microsoft says about users in your forest who are members of another forest:

"When a trust is established between a domain in a forest and a domain outside of that forest, security principals from the external domain can access resources in the internal domain. Active Directory creates a foreign security principal object in the internal domain to represent each security principal from the trusted external domain. These foreign security principals can become members of domain local groups in the internal domain. Directory objects for foreign security principals are created by Active Directory and should not be manually modified. You can view foreign security principal objects from Active Directory Users and Computers by enabling advanced features." [ Source ]

So if you go look in the ForeignSecurityPrincipals container in contoso.com, you'll see an object that represents the user account of fabrikam\steve, but his friendly name or samAccountName is not part of that record. It's just a SID.  When we pull up a permissions file dialog box like the one above, Windows attempts a SID to name translation... but it fails.  There's a little bit of technical documentation on how SID translation occurs:

"LSA on the computer that the call is sent to (using the LSA RPC interface) will resolve the SIDs it can map and send on the remaining unresolved SIDs to a domain controller in the primary domain. The domain controller will resolve additional SIDs to account names from the local database, including SIDs found in SidHistory on a global catalog.

If SIDs cannot be resolved there, the domain controller will send remaining SIDs to domain controllers in a trusted domain where the domain part of the SID matches the trust information." [ Source ]

There are many functions regarding SID lookups, and I don't know exactly which ones are used at each location, but the general concept is the same and you can see how this procedure could take a while to time out. And when it fails, you see <Unknown Contact>.  Or maybe just the unresolved SID of security principal.  Depends on which version of Windows you're using and exactly which dialog box you're looking at.

The reason it happens in our scenario is because of the one-way trust. Contoso.com cannot call upon fabrikam to translate SIDs from its forest, because fabrikam does not trust contoso.

To fix it, we could allow anonymous SID translation in fabrikam... but for many that is an unacceptable security risk.  Or we could make the trust two-way.  Or, if you're unable to do either of those things, you could at least create a security group in contoso, add the individuals from fabrikam to that group, and just assign the group to the network share ACL.  The functionality would be the same but at least you wouldn't have to look at "<Unknown Contact>" every time you opened that dialog box.

Mystery solved.

For more information on this, see the ServerFault question that I answered here, as well as the much better Ask the Directory Services team blog post here.

Excuse Me Sir, You Got Your Rally In My Powershell

If you work in the tech industry, especially in the area of software development, you might use or at least be familiar with Rally. It's a web-based project management tool that follows the principles of the Agile development lifecycle.  I've even seen Rally used to organize and track projects that were not actually software development projects.  I've seen this because I do not specifically work in the area of software development, yet I use Rally anyway.

I don't really care much for its web interface though. In my opinion, it often takes way more clicking around in the GUI than it should. Sometimes the GUI isn't intuitive and you end up making mistakes like deleting an entire User Story when you meant to just delete a single Task.

So I started thinking to myself.  "Self," I said. "I wonder if Rally has a REST API? If they did, I could interface with it in Powershell and automate a lot of simple tasks."

Well lo and behold they sure do.  Let's see how freaking simple this is with Powershell:

#
$Rally = New-WebServiceProxy https://rally1.rallydev.com/slm/webservice/1.41/meta/132645/rally.wsdl -Credential (Get-Credential) -Namespace Rally

And just like that, we're connected to Rally.  To verify, you could check the currently logged on user:

#
$Rally.getCurrentUser()

Now, let's pull some data from Rally. Let's retrieve, say, my 20 most recently-created Tasks in Rally:

#
PS C:\> $Query = $Rally.Query($null, "Task", "(Owner.Name = `"ryan@domain.com`")", "CreationDate desc", $True, 1, 20)
PS C:\> $Query.Results.Name

Write some code
Write some more code
Fix this defect
Fix that defect
Fix all the defects
Save the company from bankruptcy
Reconcile marriage
Catch the guy who keeps running the copier out of paper and not refilling it
etc.

Playing with REST services and web services is so delightfully easy with Powershell.


Update 4/10/2017

A reader emailed me recently about this post and informed that instead of using New-WebServiceProxy, using Invoke-RestMethod works better:

#
$url = "https://rally1.rallydev.com/slm/webservice/v2.0/task/?query=(Owner = ""ryan@domain.com"")&fetch=true&start=1&pagesize=20&order=CreationDate desc"

$Rally = Invoke-RestMethod -Uri $url -Credential (Get-Credential) 
$Query = $Rally.QueryResult
$Query.Results.Name

Examining Internet Explorer Tracking Protection Lists (with Powershell)

I know that Firefox and Chrome are still the only browsers that most people will ever even consider using, but ever since I started using Windows 8, I've not yet really felt a compelling reason to stop using the built-in IE10. It's fast, it passes Acid tests, and has a nice "Developer Mode" built-in where you can change IE and rendering versions on the fly (hit F12 on your keyboard.) That, and Compatibility Mode, which I think is really under-rated in a corporate environment, where employees are forced to use antiquated corporate websites that were built for IE5.

About the only thing I miss from Chrome is my sweet, sweet AdBlock. Does IE have anything that can compare? Well, it has Tracking Protection Lists, which is a good start:

IE Tracking Protection Lists

IE Tracking Protection Lists

The Personalized List kinda' sucks because you only have two choices - either block everything that IE detects as a Javascript-type thing that pulls info from other domains and is frequently seen in similar pages, or, you have to wait for IE to detect the script 10 times or so, then you have to go back in and manually choose to block it. 

Personalized Tracking List

Of course Google is going to be the number one offender here, since that's how they make money is by shoving ads in your face around every corner and making you want to buy stuff.  Honestly, I sympathize with online advertisers to a certain point, because I believe the internet would not have as much rich, free content as it does if internet advertisers were unable to make money by providing free services.  I mean, Google doesn't keep Youtube up just because they love paying those network bandwidth bills so much. But, the ads can quickly get simply too obnoxius, and we users need a way to filter all that junk that's flashed before our eyes.  Especially the Google ones that break Internet Explorer's back button.  It's Google's code that decides to open 4 instances of some doubleclick.net resource in a way that it causes you to have to hit the Back button in your browser 5 times to get back to where you were. From legalinsurrection.com

The flip-side of the coin is that if you block too much, certain websites will stop working properly. So you need to find a happy medium. Queue Tracking Lists.  Just like AdBlock, Tracking Lists are designed to block just the web content that has been decided is more harmful or annoying than good.  That way we can keep using our Gmail, Youtube, StackExchange, etc., without interruption, while still cutting out a good chunk of the ads.

You can download new Tracking Protection Lists online, right from within the tracking lists dialog box. Just click the "Get A Tracking Protection List Online..." link. It will take you to a website.  Notice in the screenshot above, that I have already added the "EasyList" TPL, from the same people that do AdBlock.  Also notice that the Tracking Protection List is simply an HTTP URI to a *.tpl text file.  And you know what that means... that means we can play with it from Powershell!

Say you wanted to examine a TPL and see if it contained a certain domain name. Well first, let's download the TPL and store it to a variable:

PS C:\> $list = Invoke-WebRequest http://easylist-msie.adblockplus.org/easylist.tpl

Now the content of the tracking list will be in $list.Content. These are the web resources that the list will block.  You can go here to see the syntax of TPLs. Warning: There will be distasteful words in this list... as the list is designed to block distasteful content.

Alright, so what if we want to know whether this TPL will block content from the domain streamcloud.eu. First, let's break $list.Content up into lines by splitting it on newlines:

PS C:\> $list.Content.Count
1
PS C:\> $Content = $list.Content.Split("`r`n")
PS C:\> $Content.Count
10203

After splitting the content element of the web request on newlines, we can see that the TPL contains 10,203 lines. I first thought to split on [String]::NewLine, but that did not yield correct results. (Dat character encoding!) Now, keeping in mind that lines that start with a # are comments, let's see if we can find entries that contain streamcloud.eu:

PS C:\> foreach($_ in $Content) { If(!$_.StartsWith('#') -and $_.Contains("streamcloud.eu")) { $_ } }
-d streamcloud.eu /deliver.php
+d streamcloud.eu

So from this output, it appears that we are allowing streamcloud.eu, but we are specifically blocking any document named deliver.php coming from streamcloud.eu.  This could have also been written as:

PS C:\> foreach($_ in $Content) { If(!$_.StartsWith('#') -and $_ -match "streamcloud.eu") { $_ } }

But a lot of times I just naturally prefer the C# parlance. The good thing about Powershell is that you're free to mix and match.

You can certainly elaborate on the concept I've started on above, and I hope you will.  Until next time!

Encrypt-File and Decrypt-File using X.509 Certificates

I like encrypting stuff.  Here are two Powershell functions I whipped up that can encrypt and decrypt a file using a valid key pair from an X.509 certificate. The encryption uses the public key, but only the corresponding private key can decrypt the data.

Encrypt-File:

#
<#
.SYNOPSIS
This Powershell function encrypts a file using a given X.509 certificate public key.
.DESCRIPTION
This Powershell function encrypts a file using a given X.509 certificate public key.
This function accepts as inputs a file to encrypt and a certificate with which to encrypt it.
This function saves the encrypted file as *.encrypted. The file can only be decrypted with the private key of the certificate that was used to encrypt it.
You must use a certificate that can be used for encryption, and not something like a code signing certificate.
.PARAMETER FileToEncrypt
Must be a System.IO.FileInfo object. $(Get-ChildItem C:\file.txt) will work.
.PARAMETER Cert
Must be a System.Security.Cryptography.X509Certificates.X509Certificate2 object. $(Get-ChildItem Cert:\CurrentUser\My\9554F368FEA619A655A1D49408FC13C3E0D60E11) will work. The public key of the certificate is used for encryption.
.EXAMPLE
PS C:\> . .\Encrypt-File.ps1
PS C:\> Encrypt-File $File $Cert
.EXAMPLE
PS C:\> . .\Encrypt-File.ps1
PS C:\> Encrypt-File $(Get-ChildItem C:\foo.txt) $(Get-ChildItem Cert:\CurrentUser\My\THUMBPRINT)
.INPUTS
Encrypt-File <System.IO.FileInfo> <System.Security.Cryptography.X509Certificates.X509Certificate2>
.OUTPUTS
A file named $FileName.encrypted
.NOTES
Written by Ryan Ries - ryan@myotherpcisacloud.com
.LINK
http://msdn.microsoft.com/en-us/library/system.security.cryptography.x509certificates.x509certificate2.aspx
#>

Function Encrypt-File
{
	Param([Parameter(mandatory=$true)][System.IO.FileInfo]$FileToEncrypt,
		  [Parameter(mandatory=$true)][System.Security.Cryptography.X509Certificates.X509Certificate2]$Cert)

	Try { [System.Reflection.Assembly]::LoadWithPartialName("System.Security.Cryptography") }
	Catch { Write-Error "Could not load required assembly."; Return }	
	
	$AesProvider                = New-Object System.Security.Cryptography.AesManaged
	$AesProvider.KeySize        = 256
	$AesProvider.BlockSize      = 128
	$AesProvider.Mode           = [System.Security.Cryptography.CipherMode]::CBC
	$KeyFormatter               = New-Object System.Security.Cryptography.RSAPKCS1KeyExchangeFormatter($Cert.PublicKey.Key)
	[Byte[]]$KeyEncrypted       = $KeyFormatter.CreateKeyExchange($AesProvider.Key, $AesProvider.GetType())
	[Byte[]]$LenKey             = $Null
	[Byte[]]$LenIV              = $Null
	[Int]$LKey                  = $KeyEncrypted.Length
	$LenKey                     = [System.BitConverter]::GetBytes($LKey)
	[Int]$LIV                   = $AesProvider.IV.Length
	$LenIV                      = [System.BitConverter]::GetBytes($LIV)
	$FileStreamWriter           
	Try { $FileStreamWriter = New-Object System.IO.FileStream("$($FileToEncrypt.FullName)`.encrypted", [System.IO.FileMode]::Create) }
	Catch { Write-Error "Unable to open output file for writing."; Return }
	$FileStreamWriter.Write($LenKey,         0, 4)
	$FileStreamWriter.Write($LenIV,          0, 4)
	$FileStreamWriter.Write($KeyEncrypted,   0, $LKey)
	$FileStreamWriter.Write($AesProvider.IV, 0, $LIV)
	$Transform                  = $AesProvider.CreateEncryptor()
	$CryptoStream               = New-Object System.Security.Cryptography.CryptoStream($FileStreamWriter, $Transform, [System.Security.Cryptography.CryptoStreamMode]::Write)
	[Int]$Count                 = 0
	[Int]$Offset                = 0
	[Int]$BlockSizeBytes        = $AesProvider.BlockSize / 8
	[Byte[]]$Data               = New-Object Byte[] $BlockSizeBytes
	[Int]$BytesRead             = 0
	Try { $FileStreamReader     = New-Object System.IO.FileStream("$($FileToEncrypt.FullName)", [System.IO.FileMode]::Open)	}
	Catch { Write-Error "Unable to open input file for reading."; Return }
	Do
	{
		$Count   = $FileStreamReader.Read($Data, 0, $BlockSizeBytes)
		$Offset += $Count
		$CryptoStream.Write($Data, 0, $Count)
		$BytesRead += $BlockSizeBytes
	}
	While ($Count -gt 0)
	
	$CryptoStream.FlushFinalBlock()
	$CryptoStream.Close()
	$FileStreamReader.Close()
	$FileStreamWriter.Close()
}

Decrypt-File:

#
<#
.SYNOPSIS
This Powershell function decrypts a file using a given X.509 certificate private key.
.DESCRIPTION
This Powershell function decrypts a file using a given X.509 certificate private key.
This function accepts as inputs a file to decrypt and a certificate with which to decrypt it.
The file can only be decrypted with the private key of the certificate that was used to encrypt it.
.PARAMETER FileToDecrypt
Must be a System.IO.FileInfo object. $(Get-ChildItem C:\file.txt) will work.
.PARAMETER Cert
Must be a System.Security.Cryptography.X509Certificates.X509Certificate2 object. $(Get-ChildItem Cert:\CurrentUser\My\9554F368FEA619A655A1D49408FC13C3E0D60E11) will work. The public key of the certificate is used for encryption. The private key is used for decryption.
.EXAMPLE
PS C:\> . .\Decrypt-File.ps1
PS C:\> Decrypt-File $File $Cert
.EXAMPLE
PS C:\> . .\Decrypt-File.ps1
PS C:\> Decrypt-File $(Get-ChildItem C:\foo.txt) $(Get-ChildItem Cert:\CurrentUser\My\THUMBPRINT)
.INPUTS
Decrypt-File <System.IO.FileInfo> <System.Security.Cryptography.X509Certificates.X509Certificate2>
.OUTPUTS
An unencrypted file.
.NOTES
Written by Ryan Ries - ryan@myotherpcisacloud.com
.LINK
http://msdn.microsoft.com/en-us/library/system.security.cryptography.x509certificates.x509certificate2.aspx
#>

Function Decrypt-File
{
	Param([Parameter(mandatory=$true)][System.IO.FileInfo]$FileToDecrypt,
		  [Parameter(mandatory=$true)][System.Security.Cryptography.X509Certificates.X509Certificate2]$Cert)

	Try { [System.Reflection.Assembly]::LoadWithPartialName("System.Security.Cryptography") }
	Catch { Write-Error "Could not load required assembly."; Return }
	
	$AesProvider                = New-Object System.Security.Cryptography.AesManaged
	$AesProvider.KeySize        = 256
	$AesProvider.BlockSize      = 128
	$AesProvider.Mode           = [System.Security.Cryptography.CipherMode]::CBC
	[Byte[]]$LenKey             = New-Object Byte[] 4
	[Byte[]]$LenIV              = New-Object Byte[] 4
	If($FileToDecrypt.Name.Split(".")[-1] -ne "encrypted")
	{
		Write-Error "The file to decrypt must be named *.encrypted."
		Return
	}
	If($Cert.HasPrivateKey -eq $False -or $Cert.PrivateKey -eq $null)
	{
		Write-Error "The supplied certificate does not contain a private key, or it could not be accessed."
		Return
	}
	Try { $FileStreamReader = New-Object System.IO.FileStream("$($FileToDecrypt.FullName)", [System.IO.FileMode]::Open)	}
	Catch
	{ 
		Write-Error "Unable to open input file for reading."		
		Return 
	}	
	$FileStreamReader.Seek(0, [System.IO.SeekOrigin]::Begin)         | Out-Null
	$FileStreamReader.Seek(0, [System.IO.SeekOrigin]::Begin)         | Out-Null
	$FileStreamReader.Read($LenKey, 0, 3)                            | Out-Null
	$FileStreamReader.Seek(4, [System.IO.SeekOrigin]::Begin)         | Out-Null
	$FileStreamReader.Read($LenIV,  0, 3)                            | Out-Null
	[Int]$LKey            = [System.BitConverter]::ToInt32($LenKey, 0)
	[Int]$LIV             = [System.BitConverter]::ToInt32($LenIV,  0)
	[Int]$StartC          = $LKey + $LIV + 8
	[Int]$LenC            = [Int]$FileStreamReader.Length - $StartC
	[Byte[]]$KeyEncrypted = New-Object Byte[] $LKey
	[Byte[]]$IV           = New-Object Byte[] $LIV
	$FileStreamReader.Seek(8, [System.IO.SeekOrigin]::Begin)         | Out-Null
	$FileStreamReader.Read($KeyEncrypted, 0, $LKey)                  | Out-Null
	$FileStreamReader.Seek(8 + $LKey, [System.IO.SeekOrigin]::Begin) | Out-Null
	$FileStreamReader.Read($IV, 0, $LIV)                             | Out-Null
	[Byte[]]$KeyDecrypted = $Cert.PrivateKey.Decrypt($KeyEncrypted, $false)
	$Transform = $AesProvider.CreateDecryptor($KeyDecrypted, $IV)
	Try	{ $FileStreamWriter = New-Object System.IO.FileStream("$($FileToDecrypt.Directory)\$($FileToDecrypt.Name.Replace(".encrypted",$null))", [System.IO.FileMode]::Create) }
	Catch 
	{ 
		Write-Error "Unable to open output file for writing.`n$($_.Message)"
		$FileStreamReader.Close()
		Return
	}
	[Int]$Count  = 0
	[Int]$Offset = 0
	[Int]$BlockSizeBytes = $AesProvider.BlockSize / 8
	[Byte[]]$Data = New-Object Byte[] $BlockSizeBytes
	$CryptoStream = New-Object System.Security.Cryptography.CryptoStream($FileStreamWriter, $Transform, [System.Security.Cryptography.CryptoStreamMode]::Write)
	Do
	{
		$Count   = $FileStreamReader.Read($Data, 0, $BlockSizeBytes)
		$Offset += $Count
		$CryptoStream.Write($Data, 0, $Count)
	}
	While ($Count -gt 0)
	$CryptoStream.FlushFinalBlock()
	$CryptoStream.Close()
	$FileStreamWriter.Close()
	$FileStreamReader.Close()
}