Loading

Wednesday, February 8, 2017

Disable TLS on Server 2012 R2 in the Registry with PowerShell

It's common knowledge that TLS is preferred over SSL because it provides better security, and because an industry-wide push to stop using SSL, use HTTPS exclusively (instead of HTTP), and increase security overall has been underway for a while. But it's also important to use the latest version of TLS. Fortunately Windows Server 2012 R2 supports all three current versions of TLS, 1.0, 1.1 and 1.2. But, what if your server requires the disabling of lower versions, like 1.0 or even 1.1? Sure, there are various resources on the Internet from .reg files to both paid and free utilities, but since I often work in environments restricting the use of such methods, and since I like to use the simplest native method possible I have a set of commands I run in PowerShell to both disable TLS 1.0 and 1.1, and explicitly create keys and enable TLS 1.2 (which aren't already in the registry for some reason).

Note: although this was written specifically for Server 2012 R2 these commands work on Server 2008 R2 as well.

After installing the latest version of PowerShell on new servers one of the next things I do is run the set of commands below. First though, we'll take a look at the current security (SCHANNEL) protocols on a new 2012 R2 server with:

Get-ChildItem -Path HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols -Recurse
View SCHANNEL Registry settings in PowerShell on Server 2012 R2.
Here is the set of commands I run to disable TLS 1.0 and 1.1 and explicitly enable TLS 1.2 on Windows Server 2012 R2:

#2012 R2 - Disable TLS 1.0, 1.1, enable 1.2 
$TLSProto = "TLS 1.0" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "TLS 1.0" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 0 –Type DWORD 

$TLSProto = "TLS 1.1" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "$TLSProto" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name DisabledByDefault –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name DisabledByDefault –Value 1 –Type DWORD 

$TLSProto = "TLS 1.2" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "$TLSProto" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name DisabledByDefault –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name DisabledByDefault –Value 0 –Type DWORD

Then run this again to verify the settings:

Get-ChildItem -Path HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols -Recurse

And, don't forget to reboot for the changes to take effect.


Thursday, January 19, 2017

Shrinking EBS Windows Boot Volume

After migrating my physical server to AWS recently I needed to shrink the boot volume a bit. The original server's drive was ~1TB, so that's the size my boot EBS volume was after the migration, but since I only have about 125GB of used space I wanted to reduce the overall volume size to about 150GB. Not surprisingly AWS doesn't provide a native way to do this so I had to get creative. I found most of the steps on the AWS Developer Forum and have adapted them to my needs, along with adding a few. And just like with the physical server-to-cloud migration we'll do here what many say can't be done....

Step 1 - Create Snapshot of Volume
Using the AWS Console or AWS CLI create a snapshot of the volume you want to reduce, or an AMI of the instance. This will protect you in case something goes off the rails, making it quick and easy to recover.
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Snapshot of my root volume."
Step 2 - Shrink Volume
On the server in Disk Management, right-click the volume and select Shrink Volume. Select the desired size and let it run. Depending on a variety of factors this could take a while (several minutes to an hour or so) so be patient.

Step 3 - Stop Server and Detach Volume
When the volume shrink completes, stop (power off) the server. Preferably from within Windows, or use the AWS console or AWS CLI to do so. Then, detach the volume from the Windows server:
aws ec2 detach-volume --volume-id vol-1234567890abcdef0

Step 4 - Start Ubuntu EC2 Instance, Attach Volumes
Select the appropriate Ubuntu AMI (version and region) and launch it. Either through the web console or AWS CLI.
aws ec2 run instances --image-id-<AMI ID> --key-name <key> --count 1 --instance-type m1.large --security-group-ids sg-<SecGroup> --placement AvailabilityZone=<AZ>
Create a new EBS volume the size you want, making it at least the same size or larger than the shrunk volume size. Attach the volume you want to clone to the Ubuntu instance and choose a mount point. For this document we will use "sdo". "o" for Original (Note "sdo" in the AWS interface gets remapped to "xvdo" in ubuntu). Attach the new volume you want to clone to to the Ubuntu instance and choose a mount point. We will use "sdn". "n" for New. (Note "sdn" in the AWS interface gets remapped to "xvdn" in Ubuntu).
aws ec2 create-volume --size 150 --region <region%gt; --availability-zone <AZ> --volume-type gp2
aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-01474ef662b89480 --device /dev/sdo
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b89480 --device /dev/sdn
Step 5 - Connect to Ubuntu, Run Commands
Connect to the Ubuntu instance and elevate yourself to SU with sudo su.

View and Save partition information for the "original" disk:
fdisk -l -u /dev/xvdo
Setup partitions for the "new" disk:
fdisk /dev/xvdn
At the prompt (Command (m for help):
"n" to create a new partition
"p" to make it a primary partition
Select a partition number (match the original setup)
Enter the first sector (match the original setup)
Enter the last sector (for first partition match the original, for a second, use the default last sector)
"t" to set the partition type to 7 (HPFS/NTFS/exFAT) on all partitions
Repeat the above process for all needed partitions
"a" to set the boot flag on partition 1
"p" to review the setup
"w" to write changes to disk and exit

Run the following to verify settings on both "old" and "new" drives:
fdisk -l -u /dev/xvdo
fdisk -l -u /dev/xvdn

Copy the MBR (Master Boot Record). The MBR is on the first sector of the disk, and is split into three parts: Boot Code (446 bytes), Partition Table (64 bytes), Boot Code Signature = 55aa (2 bytes). We only want the boot code, and will copy it with the the "dd" command to do a direct bit copy from disk to disk:
dd if=/dev/xvdo of=/dev/xvdn bs=446 count=1
Clone the NTFS file system one partition at a time (/dev/xvdo1, /dev/svdo2):
ntfsclone --overwrite /dev/xvdn1 /dev/xvdo1
ntfsclone --overwrite /dev/xvdn2 /dev/xvdo2
Repeat for all partitions, monitoring progress:


Step 6 - Detach from Ubuntu, Attach to Windows
Detach both volumes from Ubuntu instance and attach new volume to Windows instance as device /dev/sda1:
aws ec2 detach-volume --volume-id vol-1234567890abcdef0
aws ec2 detach-volume --volume-id vol-1234567890abcdef1
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b8948a --device /dev/sda1
Step 7 - Verify and Cleanup
Start the Windows instance:
aws ec2 start-instances --instance-ids i-1234567890abcdef0
Note: it may take several minutes to start the instance, so don't be impatient or alarmed..... Once the instance starts, logon to the Windows instance, then run chkdsk to validate the drive and correct any errors:
chkdsk c: /f
Terminate Ubuntu instance:
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0
Finally, as good measure make an AMI of your instance or Snapshot of the volume.

Tuesday, January 17, 2017

Migrating Physical Windows Servers to AWS AMI - EC2

Doing What They Say Can't be Done
I've had to do this task a few times but because they were separated by a significant amount of time both the methods changed slightly and my memory of the exact steps faded. These are the new-and-improved way to convert a bare metal Windows server (I'm doing 2008 R2, but it will work with 2012 R2, etc.) into an EC2 AMI. It took me several days and multiple attempts (mostly due to the time it took to copy the 120 GB image to AWS) and some trial and error, but if you follow these steps you should have success moving your server to the cloud.

Although various commercial tools exist to perform cloud migrations I used all free tools for this physical-to-cloud migration.

Review these prerequisites for VM Import to help ensure success. This AWS document is a good reference for other necessities and steps as well. According to AWS a physical server-to-AMI conversion cannot be done, but we'll trick them a bit by converting to a Hyper-V VM from physical, then to an AMI, finally launching an EC2 instance.

Step 1 - Prepare Your Server
Prior to migration you should do a little house keeping to minimize potential issues and decrease the overall time the migration will take. Some suggestions are first, clean up your drive(s) by removing any unnecessary files and directories, this will make the virtual disk smaller and reduce time to copy files to AWS. Next, make sure at least one NIC has DHCP enabled (one of the things that will cause your import to fail). I also took the opportunity to make sure all apps and patches were up-to-date as well. I chose not to remove my server from the AD domain at this point - only after a successful import of the server into EC2.

Step 2 - Create VHD (Virtual Hard Disk) from Physical Server
This is simple with the free Sysinternals tool Disk2vhd. Download and run it. Select the volume(s) you want to turn into a VHD and the name of the destination VHD file (by default it uses the NetBIOS name of the server). Make sure to uncheck "use Vhdx" option as AWS will only allow you to import a VHD file and not a VHDX file. It is recommended that you save the VHD file to a drive other than one you are imaging, but since I didn't have another drive at the time I wasn't able to do that and the conversion worked fine. The server I am working on currently is named Alymere, so you'll see that name throughout.

Disk2vhd - convert server to virtual hard disk.
Step 3 - Import VHD to Hyper-V
Use Hyper-V Manager to import the VHD exported in the previous step. I had another server (at another location) which I was able to install Hyper-V on to perform these steps, but I suppose you could do this on the origin server if it's the only one you have. Maybe I'll try it and update later..... Start your newly imported VM to make sure it boots and works as a VM, then shut it down. One critical step is to remove any CD or DVD drives from the VM as these too will cause the EC2/AMI import to fail.

Step 4 - Export VM to VHD
Again, using Hyper-V Manager export the VM. This will create a few directories and possibly additional files, but the only one you need is the new VHD file - in my case this is Alymere.vhd (although it's the same name as the physical to virtual VHD file it is a different file).

Step 5 - Break up the VHD
If you've ever tried to copy a large file over the Internet you know it can be fraught with problems. So, for multiple reasons I used 7-zip to break the file into 100MB chunks. I did it as 7z format with a compression level of "normal." Although it took about four hours for the compression I saved about eight times that much time when uploading to AWS. My ~120GB VHD file compressed to 41GB of zipped files.

Step 6 - Copy/Upload File(s) to EBS
Since I will have to extract my hundreds of 100MB files back to the original VHD I copied them to an EBS volume on one of my existing EC2 servers over a VPN connected to my VPC using robocopy. One of the reasons for breaking the original file into many smaller ones is that if there's a problem with the copy (as is common over the Internet) I won't lose much progress - yes this can be done by using the /z (restart) switch with robocopy - which I would highly recommend, but I've had better experiencing breaking large files into smaller ones. Another reason I did this was that the office where the server resides has terrible upload speeds, so I copied the files to an external drive and had my business partner take it to another office (I'm in a different state). It still took 2-3 days to copy the files from both locations to AWS, but was considerably faster doing it the way we did - copying zipped files from two locations to EC2 simultaneously.

Step 7 - Reassemble VHD
Once the files were all copied to my EBS volume on my EC2 server I used 7-zip to extract the files to the original VHD file. As mentioned previously this whole process (zip, copy, unzip) took several days, but using the methods described I feel it was the most efficient way possible given the circumstances. If you have low upload bandwidth or huge files it may make sense to use the AWS Import/Export service, which I've used with great success previously.

Step 8 - Copy/Upload to S3
In order to use AWS's VM import/export the VHD file(s) have to reside in S3. Some tools (like my beloved CloudBerry) cannot copy files of this size (120 GB), so I used the AWS CLI. Namely, aws s3 cp:

aws s3 cp E:\Temp\Alymere\ALYMERE.VHD s3://<bucket name>/Alymere/

The AWS CLI displays copy progress, including speed.

aws s3 cp - Progress and completion.
Step 9 - Import VHD to AMI
This step requires a little extra work. Follow these steps to create an AMI role and policy necessary to access S3, along with the necessary .json files tailored to your needs. With that done run the command (adapted for your use):

aws ec2 import-image --description "Alymere" --disk-containers file://C:\AWS\JSON\containers.json

Depending on the size of the VM this will take from a few minutes to an hour or more. My 120GB VHD took about 90-120 minutes. Executing the export-image command will produce various output including a task ID, which can be used to check the progress of the import:

aws ec2 describe-import-image-tasks --import-task-ids import-ami-<task ID>

I ran this several times and could see it progressing.


Upon completion the message, "AWS Windows completed..." is displayed, along with the AMI ID.

Step 10 - Launch EC2 Instance
Finally, an EC2 instance can be launched from this newly created AMI.

aws ec2 run-instances --image-id ami-<task ID>


Step 11 - Post Launch
At this point I logged onto my newly created VM to verify all my applications and data were intact. Since they were I removed my "old" physical server from my domain and joined this one. Success!

Troubleshooting VM Import/Export
A variety of issues can cause problems, i.e. failure. I would suggest reading the prerequisites for VM Import and "Troubleshooting VM Import/Export" AWS pages before beginning to make sure you can avoid issues, then be able to troubleshoot if necessary.

Good luck and happy importing!

Saturday, December 17, 2016

HowTo: Install AWS CLI on Both Windows and Linux

Several years ago before the unified AWS CLI was released I wrote about installing the EC2 command line tools. Now it's time to update that.

It goes without saying that over the decade Amazon has been providing cloud services their interfaces and tools have matured greatly (along with their overall offerings). Early on Amazon didn't even have web console and we had to rely on a disparate offering of command line tools for managing S3, EC2, ELB, etc. Finally, in the fall of 2013 Amazon released the AWS CLI, which is a unified set of command line tools that works similarly across platforms (Windows, Linux, Mac) and across AWS services. It's definitely nice to use the same (or nearly the same) syntax on both Windows and Linux machines to perform various functions.

Installing AWS CLI on Linux (Ubuntu)
The easiest way I've found to install and update the AWS CLI on Ubuntu (and Debian-based linux distros) is using apt, with this simple one-liner:

sudo apt install awscli -y

Once installed or updated check the version with:

aws --version


Installing AWS CLI on Windows
Assuming you are installing on Windows 2008 R2 or later, we'll leverage the native capability of PowerShell to use Invoke-WebRequest (AKA wget) to retrieve the latest installer. I'm also assuming you have a directory c:\temp. If not, create it or use a directory of your choosing. Note: these steps can be used to update the AWS CLI to the latest version as well. If you aren't running at least PS v3 you can update PowerShell to get this functionality. And, you'll have to restart PowerShell if you are doing a new install of the AWS CLI to access them.

To download the latest version of the installer in PowerShell run:

wget -uri https://s3.amazonaws.com/aws-cli/AWSCLI64.msi -OutFile c:\temp\AWSCLI64.msi

Launch the installer (I'm using the /passive switch to accept defaults and not have to answer any questions):

c:\temp\AWSCLI64.msi /passive

In under a minute I am able to download and execute the installer, which works both for first time installs and updating to the latest version. Just like on Linux the version can be checked with (make sure the installer completed before running):

aws --version


Setup AWS CLI
Whether you are using the AWS CLI on Windows or Linux you can run the aws configure command to enter the default values; AWS Access Key, Secret, default region and default output type. Then you can run AWS commands to your heart's content.


Test AWS CLI
Once the CLI is installed and configured you can run various commands to test connectivity to AWS services.

aws s3 ls s3:// (list all S3 buckets)


aws ec2 describe-instances




Friday, September 16, 2016

Install PowerShell v5 on Windows Server 2008 R2 or 2012 R2

Prior to updating to PowerShell version 5 make sure to be running the latest version of .NET Framework (.NET 4.6.1 as of this writing). See Checking .NET Versions.

Check Current Version of PowerShell
Run either $Host.Version or $PSVersionTable.PSVersion

Install latest version of .NET.

PowerShell 5 is part of Windows Management Framework (WMF) version 5 and can be downloaded from Microsoft (2008 R2 or 2012 R2). Select the correct download for your version of Windows Server and run the installer. After completion verify the version.



Wednesday, September 14, 2016

Install .NET 4.6.1 on Windows Server 2008 R2 or 2012 R2

Start by downloading .NET 4.6.1 from Microsoft. Launch the installer - I typically do this by running the command
c:\temp\NDP461-KB3102436-x86-x64-AllOS-ENU.exe
from an elevated PowerShell. Accept the licensing agreement and wait....

Check Version(s) of .NET Installed
From an elevated PowerShell run the following set of commands:

Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP' -recurse |

Get-ItemProperty -name Version,Release -EA 0 |

Where { $_.PSChildName -match '^(?!S)\p{L}'} |

Select PSChildName, Version, Release, @{

  name="Product"

  expression={

      switch($_.Release) {

        378389 { [Version]"4.5" }

        378675 { [Version]"4.5.1" }

        378758 { [Version]"4.5.1" }

        379893 { [Version]"4.5.2" }

        393295 { [Version]"4.6" }

        393297 { [Version]"4.6" }

        394254 { [Version]"4.6.1" }

        394271 { [Version]"4.6.1" }

      }

    }

}


The result will look something like this.


Thursday, May 14, 2015

Use PowerShell to Save Excel Worksheet as CSV

I'm working on a PowerShell script to launch some virtual servers on VMWare based on input from a CSV file. To create the CSV file I'm using Excel, and I have more than one worksheet in the Excel workbook. Since it's a bit of a hassle to use Excel's GUI to save a given worksheet as a CSV, especially when this has to be done more than once, I set out to find a way to do this programatically. At first I was indifferent on using either a macro within Excel, or a PowerShell method. I decided on PowerShell because I'm already using a PS script to launch the servers, so I decided I would write a block of code to open my spreadsheet and create a CSV from the desired worksheet.

Excel spreadsheet file.
In my quest to figure this out I ran across a lot of pieces, but not a complete solution, and as is often the case on the web I couldn't find explanations of some of the solutions. That's what prompted me to save this all here. I'll step through some of the logic, then post the script in its entirety at the bottom. The best resource I found, although not complete (for my needs), was this great post from LazyWinAdmin.

First thing is to set some variables specific to my environment, the path and name to both my input (xlsx) and output (csv) files, and the name of the tab or worksheet in the spreadsheet.
$Sheet = "CreateVM-CA"
$xls = "C:\Temp\PC\PCTest.xlsx"
$csv = "C:\Temp\PC\$Sheet.csv"

NOTE: Make sure you save your Excel file as this script reads the file from disk & not what's in an open workbook.

Next we will open Excel, the workbook, and the specific worksheet we want to export. I'm also using "Excel.Visible = $False" to prevent the Excel GUI from opening, and "Excel.DisplayAlerts = $False" to prevent messages such as being prompted to overwrite the file if it already exists (see below).
$objExcel = New-Object -ComObject Excel.Application
$objExcel.Visible = $False
$objExcel.DisplayAlerts = $False
$WorkBook = $objExcel.Workbooks.Open($xls)
$WorkSheet = $WorkBook.sheets.item("$Sheet")

Prompted to replace existing file (when not using DisplayAlerts=$False)
A useful note (again thanks to LazyWinAdmin) is that we can verify excel has opened, and we are working with the desired workbook using the following, which lists the available worksheets within the opened workbook.
$WorkBook.sheets | Select-Object -Property Name

Listing Excel worksheets with PowerShell.
Saving the file. When saving the file we will, of course specify that it is a CSV, but also the file type (CSV) is designated with the number 6. For some reason you can't pass the number 6 directly in the SaveAs command, so we are using the variable xlCSV. In my discovery I ran across a list of several file types, but cannot find that now. It's out there somewhere....
$xlCSV = 6
$WorkBook.SaveAs($csv,$xlCSV)

Closing/quitting Excel. Initially I was using just the first command to quit Excel. However, I discovered that it remained open and subsequent executions of my script would fail to write the file because the open Excel process had the file locked. So after a little research and testing I found the second command which kills the process, and prevents the CSV file from being locked.
$objExcel.quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($objExcel)

This little script works beautifully now and does just what I need by opening a specific worksheet from an Excel workbook, then saving it as a CSV file. Here's the entire script.
$Sheet = "CreateVM-CA"
$xls = "C:\Temp\PC\PCTest.xlsx"
$csv = "C:\Temp\PC\$Sheet.csv"
$objExcel = New-Object -ComObject Excel.Application
$objExcel.Visible = $False
$objExcel.DisplayAlerts = $False
$WorkBook = $objExcel.Workbooks.Open($xls)
$WorkSheet = $WorkBook.sheets.item("$Sheet")
$xlCSV = 6
$WorkBook.SaveAs($csv,$xlCSV)
$objExcel.quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($objExcel)


Output CSV file.

Sunday, April 19, 2015

History of Internet Addressing

Volumes have been written about the Internet and its protocols so what I have here is nothing new or ground-breaking. It is, however, a good overview of the history of IP addressing, related Request For Comments (RFC) and other links.

I've been working with the Internet Protocols for a while, including data center builds, network operations and teaching various Cisco (CCNA), Microsoft (MCSE), Novell (CNE), etc. certification and training courses over the years. And although I do know a fair bit about the Internet and IP in particular I actually learned a few new things recently. Or I should say while doing a little research for a new project I had some thoughts gel in my mind which led me to record this information. This will be a useful resource for me, my future students, and hopefully you as well.

If you'd like to cut to the chase scroll down to the summary/timeline at the end.

Origins of The Internet
Host-to-host communications via packet-switched networks was in its infancy in the late 1960's with the Advanced Research Projects Agency Network (ARPANET), from which grew TCP/IP, AKA "The Internet Protocol Suite", with IP and IP addressing at its core. Two key pioneers of the Internet and the godfathers of IP, Bob Kahn and Vint Cerf, had to go through a lot of experimentation and work before finally coming up with what we know today as IPv4. First we'll discuss the IP versions prior to 4, then subsequent enhancements of IPv4 and finally the next generation of IP. The previous versions of IP where part of the initial experimentation. An example of one of these earlier versions is IPv2 as documented in Draft Internetwork Protocol Description Version 2. This and the other preceding versions are listed in table 1 along with the dates they were published.

Table 1
Version  Date
0        March 1977
1        January 1978
2        February 1978 version A
3        February 1978 version B
4        August 1979 version 4

IPv4
In January of 1980 Jon Postel, who was editor and/or writer of many RFCs, published RFC 760, "DOD STANDARD, INTERNET PROTOCOL," the first IPv4 RFC. At this point it is important to note no concept of address classes, subnetting, CIDR, etc. were defined at this point, therefore they didn't exist. All of those things came along later to enhance the Internet Protocol addresses. So in this original definition IPv4 used the first 8 bits as the network address, and the last 24 bits as the "host" portion of the address, which was termed at the time Source or Destination Local Address (see RFC 760, pages 14 and 22 for exact wording).

In a 2011 interview, Vint Cerf, on of the Internet Founding Fathers said of IPv4, "I thought it was an experiment and I thought that 4.3 billion [IPv4 addresses] would be enough to do an experiment." He said he never thought, "this experiment wouldn't end." So for an "experiment" IPv4 is doing pretty damn good! Albeit with a few enhancements over the next several years.

Looking back to the 1970's computers hadn't yet reached the mainstream so it wasn't imagined by the Pioneers that within a few short years we would see an explosion of networks and devices connecting to the Internet. This is clearly evident in RFC 762 titled "Assigned Numbers," which lists 33 "networks" out of a possible 254 (0 and 255 being reserved). Again, IP address classes hadn't yet been defined.... This RFC was published in January 1980, but it also lists when the 0-3 Assigned IP versions were initially defined, giving us a little more insight into the timeline of IP address versions.

From this we can see that IP addressing began to be defined around March 1977 with IPv0. Then by August 1979 IPv4, the main version on which the Internet was built and is still widely in use today, was defined. Now, over 35 years on, the Internet runs mainly with the "experimental" IP (v4) addressing created by Vint Cerf and Bob Kahn.

Classful IPv4 Addresses
Fairly quickly it was apparent to these Founding Fathers that the intial 8/24 (nnnnnnnn.hhhhhhhh.hhhhhhhh.hhhhhhhh) addressing scheme initially defined by IPv4 needed an enhancement. In September 1981 RFC 791 defined IP address classes in order to "provide for flexibility in assigning address to networks and allow for the large number of small to intermediate sized networks the interpretation of the address field is coded to specify a small number of networks with a large number of host, a moderate number of networks with a moderate number of hosts, and a large number of networks with a small number of hosts." Since this was just an enhancement or further definition of the use of the 32 bit IPv4 address scheme it wasn't necessary to increment the version.

This enhancement to IPv4 is known as classful IP addressing. Table 2 provides a quick view of the classes A, B and C as a reference. Much more about these classes, etc. are available elsewhere on the Internet.

Table 2
Class Leading Bits Network Bits Host Bits Number of networks Addresses per network Start address End address
A 0 8 24 128 (2^7)  16,777,216 (2^24)  0.0.0.0  127.255.255.255
10 16 16 16,384 (2^14)  65,536 (2^16)  128.0.0.0  191.255.255.255
110 24 8 2,097,152 (2^21)  256 (2^8)  192.0.0.0  223.255.255.255
NOTE: classful addressing was superseded by classless (AKA CIDR) addressing. Although this is the case it is common to discuss and have classful addresses referenced in documentation. While this technically shouldn't be done this network engineer hasn't seen much if any decrease in classful address references in my 25 years in the business.

Subnetting
In 1984 another evolution of IPv4 addressing was introduced, Internet Subnets with RFC 917. This explains both the need for and way to use part of the host portion of the address for a subnet, effectively creating something like nnnnnnn.ssssssss.hhhhhhhh.hhhhhhhh, where a "Class A" address is subnetted by taking 8 bits (as an example, more or less could be used of course) of the host address fields to allow for up to 256 subnets. This necessitated the creation of a way for hosts to know which portion of their address was used for the network and subnet vs. the host portion of the address, therefore the birth of the subnet mask. In the case listed above just about everyone familiar with IP addressing will know the mask would be 255.255.0.0. In August 1985 RFC 950, "Internet Standard Subnetting Procedure," further defines subnetting and subnet masks.

NOTE: In several early RFCs you'll see the term "catenet." This depricated term was defined in the early days of packet-switched networking and referred to a network of networks, or an internetwork or internet (lower case "i").

Governing Bodies
By the mid- to late-1980's organizations like the Internet Engineering Task Force (IETF), Internet Assigned Numbers Authority (IANA), its parent Internet Corporation for Assigned Names and Numbers (ICANN), and others were formed to steer, define and coordinate protocols, growth and enhancements of the Internet and Internet Protocols.

Classless IP Addressing - CIDR
In September 1993 IETF introduced Classless Inter-Domain Routing (CIDR) which is a method for allocating IP addresses and routing IP packets. CIDR replaces the previous addressing architecture of classful addresses, with the goal to slow the growth of routing tables on routers across the Internet, and to help slow the rapid exhaustion of IPv4 addresses. Once again the "experimental" IPv4 address space was modified to extend its life. Due to the lack of scalability of classful (A, B, C, etc.) addresses CIDR provides the ability to "subnet" an IPv4 address at virtually any bit value. This is done by appending the number of bits used for the network (technically network/subnet) to the IP address. For example 172.19.0.0/22 provides for a network with up to 1022 hosts. Much is available on CIDR around the web, including numerous subnetting/CIDR charts, but a few references are Wikipedia's "Classless Inter-Domain Routing", and RFC 1518, "An Architecture for IP Address Allocation with CIDR", and RFC 1519, "Classless Inter-Domain Routing (CIDR): an Address Assignment and Aggregation Strategy."

Private Addresses, NAT & PAT
As early as the 1980's it was apparent that IPv4's 32-bit address space would become exhausted. A variety of factors contributed to this with the main one being the explosive growth of networks and devices worldwide participating in and connecting to the Internet. So in addition to the aforementioned classful, classless and subnetting of IPv4 network address translation (NAT) and port address translation (PAT) were developed. Currently NAT & PAT are widely deployed, so much so that they are the norm. But, these actually break the end-to-end communications originally envisioned with the Internet and introduce other problems or challenges. This is something IPv6 addresses, but let's not get ahead of ourselves. NAT and more specifically PAT  has become a popular and essential tool in conserving global address space allocations in face of IPv4 address exhaustion.

Of course in order to effectively use NAT/PAT a set of private, non-publicly routable IP addresses had to be defined, which was done in February 1996 in RFC 1918, "Address Allocation for Private Internets." (See also, RFC 5735, "Special Use IPv4 Addresses.")

Table 3 - Private IPv4 Address Ranges
     10.0.0.0     -  10.255.255.255  (10/8 prefix)
     172.16.0.0   -  172.31.255.255  (172.16/12 prefix)
     192.168.0.0  -  192.168.255.255 (192.168/16 prefix)

IPv5
This discussion wouldn't be complete without touching on version 5. It was an experimental family of protocols for streaming voice, video, etc. called Internet Streaming Protocol in 1979, but was never fully developed. If you are so inclined see RFC 1190, "Experimental Internet Stream Protocol."

IPv6
Along with enhancing IPv4, by 1994 IETF began to define the next generation of IP with IPv6, sometimes called IPng. This took several years and in December 1998 RFC 2460, "Internet Protocol, Version 6 (IPv6) Specification" was published. Since IPv6 was defined about 20 years after IPv4 a lot of the former protocol's shortcomings were addressed. In addition to adding native security to IP with IPSec, restoring the end-to-end communications model (doing away with NAT), IPv6 increases the address space. A lot!

By using a 32 bit address IPv4 has a total of about 4.3 billion (2^32) available numbers. IPv6 is 128 bits, which provides a bit over 340 undecillion (2^128) addresses. Table 3 shows just how large this address space is. With such a large address space it is not necessary to define address classes (no "classful") in IPv6, nor is it necessary to use a subnet mask. Rather, since IPv6 was built upon the concept of Classless Inter-Domain Routing IPv6 addresses are written with a trailing /xx (example: 2001:db8:abcd:3f00::/64). In most cases organizations will receive a /64 or /48 address space from either an ISP or IANA, then they will be able to use CIDR to suit their needs.

Table 4
IP4 addresses (2^32)  - 4,294,967,296
IP6 addresses (2^128) - 340,282,366,920,938,463,463,374,607,431,768,211,456

Conclusion
Although IPv4 was an "experimental" addressing scheme born three decades ago it has seen numerous enhancements with classes, subnetting, CIDR and NAT/PAT to extend its lifespan, and it's still going strong. IPv6 has been around for nearly two decades and its use is definitely picking up steam but I believe IPv4 will be around for quite some time. In fact, it's likely the two will run in parallel perhaps for as long as the Internet exists.

Summary/Timeline
  • Mar 1977: IP addressing first defined and documented, starting with IPv0
  • Aug 1979: IPv4 defined (RFC 760)
  • Sep 1981: IPv4 classful addresses defined (RFC 791)
  • Oct 1984: IPv4 subnetting introduced (RFC 917)
  • Aug 1985: IPv4 subnetting further defined (RFC 950)
  • Sep 1993: IPv4 classless (AKA CIDR) addresses defined (RFC 1518, and RFC 1519)
  • Feb 1996  IPv4 private addresses and NAT/PAT  (RFC 1918, and RFC 5735)
  • Dec 1998 IPv6 defined (RFC 2460)

Thursday, March 6, 2014

How Did One Guy Build A World Class Web Infrastructure Serving Over One Billion Calls Per Day?

On Monday, December 15, 2008 I went to work for an online video start-up. On my first day our video players loaded on 384 of our partner's web pages. As you can see from the first chart that was actually a pretty good day for us (it was actually our biggest day to date) as we were averaging less than 100 pages a day. During my interview with the founder and CEO he confidently told me the company would be one of the 1000 busiest sites within one year. I believed him.

Chart 1 - 384 page views on Dec 15, 2008
I had a comfortable job as a Network Operations Director with a mature and stable company, but I felt like I needed a change so I was looking a little for something new. A colleague had recently found a job which had been posted on craigslist.org, so I was trolling IT position postings there. I was frustrated because most jobs wanted specific skills, like Active Directory, Exchange, routers, firewalls, etc. I didn't want to do just one thing or focus on a narrow area. Then I came across a post on craigslist looking for a utility player, one who had varied experience and could wear many hats. It was a match made in heaven. (I later found out this company was frustrated by not being able to find someone with a wide range of experience as all previous applicants had been siloed in specific areas. They had actually given up and pulled the post less than a day after I ran across it.)

Based on the CEO's statement that we were going to become one of the one thousand busiest web services on the Internet I was tasked with building a scale-able system that could grow rapidly (along with all other IT related duties, but that's another story...). Oh, and because we were a start-up funded solely by friends and family of our founder I had an extremely lean budget for equipment, facilities and personnel. Basically the budget was zero.

Admittedly I was a little naive, but I'm also optimistic and very determined. So I set out to do what I was asked.

At the time we were sharing a handful of servers with another start-up in a colo across the country. I had 20 year-old 1U Dell servers, a couple gigabit switches, two entry-level Cisco firewalls, and two low-end load balancers. I quickly put together a fairly lean list of servers and networking equipment I needed and tried to get a few hundred grand to buy that and setup at least one separate location. The answer came back that I couldn't even spend one tenth of what I needed & I had to figure out how to make things work without any capital expenditure.

Then on January 19-24, 2009 while I was trying to figure out how to work miracles we had our first Slashdot effect event when one of our partners had an article containing our player featured on politico.com (note: at the time we were mainly politically oriented, now we are a broad-based news, entertainment and sports organization). We went from averaging less than 100 player loads (AKA page views) per day to over 500,000 in a single day. Needless to say our small company was ecstatic, but I was a bit nervous. While our small infrastructure handled the spike, it did so just barely.

Chart 2 - January 19-24, 2009 Slashdot effect
When I started with this new company was when I was introduced to Amazon Web Services and started dabbling with EC2 and S3 right away. In fact, we started running our corporate website on EC2 a little over a month before I started, and we ran it on the exact same server for just over five years.

Admittedly I was somewhat hesitant to use AWS. First, the concept of every server having a public IP address, then the fact that they didn't have an SLA, and finally the only way to load balance was to build your own with something like HA Proxy on EC2 servers. But the compelling factors, elasticity, pay as you go, no CapEx, etc., were really attractive, especially to someone like me who didn't have any money for equipment, nor could I hire anyone to help build and maintain the infrastructure.

Sometime in the spring of 2009 when AWS announced Elastic Load Balancing I was swayed and fully embraced moving to "the cloud." I started right away copying our (~200 GB) video library and other assets to S3, and started a few EC2 servers on which I started running our web, database and application stacks. By August of 2009 we were serving our entire customer-facing infrastructure on AWS, and averaging a respectable quarter million page views per day. In October of that year we had our second 500,000+ day, and that was happening consistently.

Chart 3 - 2009 Traffic
Through most of 2009 our company had 1 architect, 0 DBA's (so this job defaulted to me), and 1 operations/infrastructure guy (me), and we were outsourcing our development. We finally started to hire a few developers and brought all development in-house, and we hired our first DBA, but it was still a skeleton crew. By the end of that year we were probably running 20-30 EC2 servers, had a couple ELB's, and stored and served (yes, served) static content on S3. Things were doing fairly well and we were handling the growth.

Chart 4 - Explosive Growth in 2010
2010 was a banner year for us. In Q1 we surpassed 1 million, 2 million and even 5 million page views per day. And by Q3 we were regularly hitting 10 million per day. Through it all we leveraged AWS to handle this load, adding EC2 servers, up-sizing servers, etc. And (this is one of my favorite parts) didn't have to do a thing with ELB as AWS scaled that up for us as needed, automatically.

We were still a skeleton crew, but finally had about ten people in the dev, database and operations group(s). Through this all and well beyond we never had more than one DBA, and one operations/infrastructure guy.

I can't say this growth wasn't without pain though. We did have a few times when traffic spikes would unexpectedly hit us, or bottlenecks would expose themselves. But throughout this time we were able to optimize our services making them more efficient, more able to grow and handle load, and even handle more calls per server driving costs (on a per call basis) down considerably. And, yes, we benefited greatly from Amazon's non-stop price reductions. I regularly reported to our CEO and others about how our traffic was growing exponentially but our costs weren't. Win, win, win!

I'm a bit of a data junky and I generate and keep detailed information on number of calls/hits to our infrastructure, amount of data returned per call, and ultimately cost per call. This has enabled me to keep a close eye on performance and costs. And I've been able to document when we've had numerous wins and fails. I've identified when particular deployments have begun making more calls or returning more data usually causing slower performance and always costing more money. I've also been able to identify when we've had big wins by improving performance and saving money.

The main way I've done this is to leverage available CPU capacity when servers have been underutilized on evenings and weekends. Currently on a daily basis I analyze close to 1 billion log lines, effectively for free. This is a high-level analysis looking at things like numbers of particular calls, bandwidth, HTTP responses, browser types, etc.

Starting in 2009 we really started to focus on making our systems more efficient and making them faster, more resilient and more scale-able. And I've been able to measure the results of those efforts and we recorded several wins, each time making our products faster, better and less expensive to deliver.

Chart 5 - More Growth in 2011
2011 was another banner year for us and we crossed the 20 million and 30 million page views per day thresholds. When our video products load on a given page as many as 20 calls are made to both static and dynamic content, roughly half of each type. All the static files (HTML, CSS, JS, images, video, etc.) are served through CDN's. But all the dynamic calls (embed, player services and analytics) are served through EC2 servers behind Elastic Load Balancers. And these are where I think we really shine. These are the services where we've really fine tuned their performance mentioned above.

Chart 6 - Continued Growth in 2012 and 2013
In 2012 and 2013 we saw more growth hitting as many 78 million page views in a single day, and at present an on average day our products load on 60 million pages across the web. This translates to about 500 million calls to static content served through CDN’s, and another 500 million daily calls to our web services (chart 7 shows four of our busiest web services, but not all of them) powered by web and database servers running in EC2 behind Elastic Load Balancers. ½ a billion dynamic services calls per day. Rather impressive!

Chart 07 - AWS CloudWatch Stats Showing Over 400,000 Calls Per Day
Not only have we been able to leverage the zero CapEx, low OpEx, high availability and scalability of AWS, but we were able to build all this with a very small team. In the fall of 2012 we had a couple of nearly 80 million page view days & at that time we had less than 10 people in the dev, database and operations groups (Note: to that point we never had more than 1 DBA and 1 network operations guy). Since I was the operations “group” up until that time I am blown away that we could build a world-class infrastructure serving at the scale we do with such a small crew. I believe it’s unheard of to build and run a system like ours with only 1 operations guy, and I know that wouldn't have been possible without AWS.

AWS Certified Solutions Architect

Although I've been working with Amazon Web Services for a few years now I only recently learned about AWS certifications. So today I went to an authorized testing center (for the first time in several years) and took a certification exam. And didn't do too bad.

Back in the day I took dozens of exams for Novell's CNE, Microsoft's MCSE, Citrix, and Cisco, so I wasn't too nervous about taking another one. All-in-all the AWS certification was fairly straight forward and comprehensive. The exam guide indicates that it covers topics from EC2, to S3, to RDS, ELB, CloudFront, CloudFormation, etc. And that's true. Study up and get certified.


Now I can officially use the AWS Certified Solutions Architect - Associate logo (above).