Thursday, April 19, 2018

AWS Certified SysOps Administrator Official Study Guide: Associate Exam Review

As I posted recently I've been using some great resources to prepare for some AWS exams. In my AWS Certified Solutions Architect Official Study Guide: Associate Exam review I raved about how good it is in assisting to prepare for the exam. Now I want to talk about the, "AWS Certified SysOps Administrator Official Study Guide: Associate Exam," which is what you need to help pass this exam.

In it, "The AWS Certified SysOps Administrator Official Study Guide: Associate Exam is a comprehensive exam preparation resource. This book bridges the gap between exam preparation and real-world readiness, covering exam objectives while guiding you through hands-on exercises based on situations you'll likely encounter as an AWS Certified SysOps Administrator. From deployment, management, and operations to migration, data flow, cost control, and beyond, this guide will help you internalize the processes and best practices associated with AWS. The Sybex interactive online study environment gives you access to invaluable preparation aids, including an assessment test that helps you focus your study on areas most in need of review, and chapter tests to help you gauge your mastery of the material. Electronic flashcards make it easy to study anytime, anywhere, and a bonus practice exam gives you a sneak preview so you know what to expect on exam day."

I would highly recommend getting this study guide and using it to help you achieve the, "AWS Certified SysOps Administrator – Associate" certification.

AWS Certified Solutions Architect Official Study Guide: Associate Exam Review

I've been working with AWS for 10 years now and love it more than ever! About four years ago when I first learned of AWS Certifications I went right out and took the Architecting on AWS Associates exam and was able to pass it the first time. When it came time to re-certify a couple years later I went in a little unprepared and barely missed passing so I started looking for some help.

That's when I found, "AWS Certified Solutions Architect Official Study Guide: Associate Exam." I got it and read through for the next few weeks brushing up on many topics I already knew, but also learning a lot of new material.

In this the authors do a great job of explaining the following topics and it will definitely help prepare you for the exam!

  • Mapping Multi-Tier Architectures to AWS Services, such as web/app servers, firewalls, caches and load balancers
  • Understanding managed RDBMS through AWS RDS (MySQL, Oracle, SQL Server, Postgres, Aurora)
  • Understanding Loose Coupling and Stateless Systems
  • Comparing Different Consistency Models in AWS Services
  • Understanding how AWS CloudFront can make your application more cost efficient, faster and secure
  • Implementing Route tables, Access Control Lists, Firewalls, NAT, and DNS
  • Applying AWS Security Features along with traditional Information and Application Security
  • Using Compute, Networking, Storage, and Database AWS services
  • Architecting Large Scale Distributed Systems
  • Understanding of Elasticity and Scalability Concepts
  • Understanding of Network Technologies Relating to AWS
  • Deploying and Managing Services with tools such as CloudFormation, OpsWorks and Elastic Beanstalk.

Wednesday, February 8, 2017

Disable TLS on Server 2012 R2 in the Registry with PowerShell

It's common knowledge that TLS is preferred over SSL because it provides better security, and because an industry-wide push to stop using SSL, use HTTPS exclusively (instead of HTTP), and increase security overall has been underway for a while. But it's also important to use the latest version of TLS. Fortunately Windows Server 2012 R2 supports all three current versions of TLS, 1.0, 1.1 and 1.2. But, what if your server requires the disabling of lower versions, like 1.0 or even 1.1? Sure, there are various resources on the Internet from .reg files to both paid and free utilities, but since I often work in environments restricting the use of such methods, and since I like to use the simplest native method possible I have a set of commands I run in PowerShell to both disable TLS 1.0 and 1.1, and explicitly create keys and enable TLS 1.2 (which aren't already in the registry for some reason).

Note: although this was written specifically for Server 2012 R2 these commands work on Server 2008 R2 as well.

After installing the latest version of PowerShell on new servers one of the next things I do is run the set of commands below. First though, we'll take a look at the current security (SCHANNEL) protocols on a new 2012 R2 server with:

Get-ChildItem -Path HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols -Recurse
View SCHANNEL Registry settings in PowerShell on Server 2012 R2.
Here is the set of commands I run to disable TLS 1.0 and 1.1 and explicitly enable TLS 1.2 on Windows Server 2012 R2:

#2012 R2 - Disable TLS 1.0, 1.1, enable 1.2 
$TLSProto = "TLS 1.0" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "TLS 1.0" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 0 –Type DWORD 

$TLSProto = "TLS 1.1" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "$TLSProto" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name DisabledByDefault –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name DisabledByDefault –Value 1 –Type DWORD 

$TLSProto = "TLS 1.2" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS" –Name "$TLSProto" New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name CLIENT New-Item
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto" –Name SERVER New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name Enabled –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\CLIENT" –Name DisabledByDefault –Value 0 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name Enabled –Value 1 –Type DWORD New-ItemProperty
"HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\PROTOCOLS\$TLSProto\SERVER" –Name DisabledByDefault –Value 0 –Type DWORD

Then run this again to verify the settings:

Get-ChildItem -Path HKLM:\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols -Recurse

And, don't forget to reboot for the changes to take effect.

Thursday, January 19, 2017

Shrinking EBS Windows Boot Volume

After migrating my physical server to AWS recently I needed to shrink the boot volume a bit. The original server's drive was ~1TB, so that's the size my boot EBS volume was after the migration, but since I only have about 125GB of used space I wanted to reduce the overall volume size to about 150GB. Not surprisingly AWS doesn't provide a native way to do this so I had to get creative. I found most of the steps on the AWS Developer Forum and have adapted them to my needs, along with adding a few. And just like with the physical server-to-cloud migration we'll do here what many say can't be done....

Step 1 - Create Snapshot of Volume
Using the AWS Console or AWS CLI create a snapshot of the volume you want to reduce, or an AMI of the instance. This will protect you in case something goes off the rails, making it quick and easy to recover.
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Snapshot of my root volume."
Step 2 - Shrink Volume
On the server in Disk Management, right-click the volume and select Shrink Volume. Select the desired size and let it run. Depending on a variety of factors this could take a while (several minutes to an hour or so) so be patient.

Step 3 - Stop Server and Detach Volume
When the volume shrink completes, stop (power off) the server. Preferably from within Windows, or use the AWS console or AWS CLI to do so. Then, detach the volume from the Windows server:
aws ec2 detach-volume --volume-id vol-1234567890abcdef0

Step 4 - Start Ubuntu EC2 Instance, Attach Volumes
Select the appropriate Ubuntu AMI (version and region) and launch it. Either through the web console or AWS CLI.
aws ec2 run instances --image-id-<AMI ID> --key-name <key> --count 1 --instance-type m1.large --security-group-ids sg-<SecGroup> --placement AvailabilityZone=<AZ>
Create a new EBS volume the size you want, making it at least the same size or larger than the shrunk volume size. Attach the volume you want to clone to the Ubuntu instance and choose a mount point. For this document we will use "sdo". "o" for Original (Note "sdo" in the AWS interface gets remapped to "xvdo" in ubuntu). Attach the new volume you want to clone to to the Ubuntu instance and choose a mount point. We will use "sdn". "n" for New. (Note "sdn" in the AWS interface gets remapped to "xvdn" in Ubuntu).
aws ec2 create-volume --size 150 --region <region%gt; --availability-zone <AZ> --volume-type gp2
aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-01474ef662b89480 --device /dev/sdo
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b89480 --device /dev/sdn
Step 5 - Connect to Ubuntu, Run Commands
Connect to the Ubuntu instance and elevate yourself to SU with sudo su.

View and Save partition information for the "original" disk:
fdisk -l -u /dev/xvdo
Setup partitions for the "new" disk:
fdisk /dev/xvdn
At the prompt (Command (m for help):
"n" to create a new partition
"p" to make it a primary partition
Select a partition number (match the original setup)
Enter the first sector (match the original setup)
Enter the last sector (for first partition match the original, for a second, use the default last sector)
"t" to set the partition type to 7 (HPFS/NTFS/exFAT) on all partitions
Repeat the above process for all needed partitions
"a" to set the boot flag on partition 1
"p" to review the setup
"w" to write changes to disk and exit

Run the following to verify settings on both "old" and "new" drives:
fdisk -l -u /dev/xvdo
fdisk -l -u /dev/xvdn

Copy the MBR (Master Boot Record). The MBR is on the first sector of the disk, and is split into three parts: Boot Code (446 bytes), Partition Table (64 bytes), Boot Code Signature = 55aa (2 bytes). We only want the boot code, and will copy it with the the "dd" command to do a direct bit copy from disk to disk:
dd if=/dev/xvdo of=/dev/xvdn bs=446 count=1
Clone the NTFS file system one partition at a time (/dev/xvdo1, /dev/svdo2):
ntfsclone --overwrite /dev/xvdn1 /dev/xvdo1
ntfsclone --overwrite /dev/xvdn2 /dev/xvdo2
Repeat for all partitions, monitoring progress:

Step 6 - Detach from Ubuntu, Attach to Windows
Detach both volumes from Ubuntu instance and attach new volume to Windows instance as device /dev/sda1:
aws ec2 detach-volume --volume-id vol-1234567890abcdef0
aws ec2 detach-volume --volume-id vol-1234567890abcdef1
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b8948a --device /dev/sda1
Step 7 - Verify and Cleanup
Start the Windows instance:
aws ec2 start-instances --instance-ids i-1234567890abcdef0
Note: it may take several minutes to start the instance, so don't be impatient or alarmed..... Once the instance starts, logon to the Windows instance, then run chkdsk to validate the drive and correct any errors:
chkdsk c: /f
Terminate Ubuntu instance:
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0
Finally, as good measure make an AMI of your instance or Snapshot of the volume.

Tuesday, January 17, 2017

Migrating Physical Windows Servers to AWS AMI - EC2

Doing What They Say Can't be Done
I've had to do this task a few times but because they were separated by a significant amount of time both the methods changed slightly and my memory of the exact steps faded. These are the new-and-improved way to convert a bare metal Windows server (I'm doing 2008 R2, but it will work with 2012 R2, etc.) into an EC2 AMI. It took me several days and multiple attempts (mostly due to the time it took to copy the 120 GB image to AWS) and some trial and error, but if you follow these steps you should have success moving your server to the cloud.

Although various commercial tools exist to perform cloud migrations I used all free tools for this physical-to-cloud migration.

Review these prerequisites for VM Import to help ensure success. This AWS document is a good reference for other necessities and steps as well. According to AWS a physical server-to-AMI conversion cannot be done, but we'll trick them a bit by converting to a Hyper-V VM from physical, then to an AMI, finally launching an EC2 instance.

Step 1 - Prepare Your Server
Prior to migration you should do a little house keeping to minimize potential issues and decrease the overall time the migration will take. Some suggestions are first, clean up your drive(s) by removing any unnecessary files and directories, this will make the virtual disk smaller and reduce time to copy files to AWS. Next, make sure at least one NIC has DHCP enabled (one of the things that will cause your import to fail). I also took the opportunity to make sure all apps and patches were up-to-date as well. I chose not to remove my server from the AD domain at this point - only after a successful import of the server into EC2.

Step 2 - Create VHD (Virtual Hard Disk) from Physical Server
This is simple with the free Sysinternals tool Disk2vhd. Download and run it. Select the volume(s) you want to turn into a VHD and the name of the destination VHD file (by default it uses the NetBIOS name of the server). Make sure to uncheck "use Vhdx" option as AWS will only allow you to import a VHD file and not a VHDX file. It is recommended that you save the VHD file to a drive other than one you are imaging, but since I didn't have another drive at the time I wasn't able to do that and the conversion worked fine. The server I am working on currently is named Alymere, so you'll see that name throughout.

Disk2vhd - convert server to virtual hard disk.
Step 3 - Import VHD to Hyper-V
Use Hyper-V Manager to import the VHD exported in the previous step. I had another server (at another location) which I was able to install Hyper-V on to perform these steps, but I suppose you could do this on the origin server if it's the only one you have. Maybe I'll try it and update later..... Start your newly imported VM to make sure it boots and works as a VM, then shut it down. One critical step is to remove any CD or DVD drives from the VM as these too will cause the EC2/AMI import to fail.

Step 4 - Export VM to VHD
Again, using Hyper-V Manager export the VM. This will create a few directories and possibly additional files, but the only one you need is the new VHD file - in my case this is Alymere.vhd (although it's the same name as the physical to virtual VHD file it is a different file).

Step 5 - Break up the VHD
If you've ever tried to copy a large file over the Internet you know it can be fraught with problems. So, for multiple reasons I used 7-zip to break the file into 100MB chunks. I did it as 7z format with a compression level of "normal." Although it took about four hours for the compression I saved about eight times that much time when uploading to AWS. My ~120GB VHD file compressed to 41GB of zipped files.

Step 6 - Copy/Upload File(s) to EBS
Since I will have to extract my hundreds of 100MB files back to the original VHD I copied them to an EBS volume on one of my existing EC2 servers over a VPN connected to my VPC using robocopy. One of the reasons for breaking the original file into many smaller ones is that if there's a problem with the copy (as is common over the Internet) I won't lose much progress - yes this can be done by using the /z (restart) switch with robocopy - which I would highly recommend, but I've had better experiencing breaking large files into smaller ones. Another reason I did this was that the office where the server resides has terrible upload speeds, so I copied the files to an external drive and had my business partner take it to another office (I'm in a different state). It still took 2-3 days to copy the files from both locations to AWS, but was considerably faster doing it the way we did - copying zipped files from two locations to EC2 simultaneously.

Step 7 - Reassemble VHD
Once the files were all copied to my EBS volume on my EC2 server I used 7-zip to extract the files to the original VHD file. As mentioned previously this whole process (zip, copy, unzip) took several days, but using the methods described I feel it was the most efficient way possible given the circumstances. If you have low upload bandwidth or huge files it may make sense to use the AWS Import/Export service, which I've used with great success previously.

Step 8 - Copy/Upload to S3
In order to use AWS's VM import/export the VHD file(s) have to reside in S3. Some tools (like my beloved CloudBerry) cannot copy files of this size (120 GB), so I used the AWS CLI. Namely, aws s3 cp:

aws s3 cp E:\Temp\Alymere\ALYMERE.VHD s3://<bucket name>/Alymere/

The AWS CLI displays copy progress, including speed.

aws s3 cp - Progress and completion.
Step 9 - Import VHD to AMI
This step requires a little extra work. Follow these steps to create an AMI role and policy necessary to access S3, along with the necessary .json files tailored to your needs. With that done run the command (adapted for your use):

aws ec2 import-image --description "Alymere" --disk-containers file://C:\AWS\JSON\containers.json

Depending on the size of the VM this will take from a few minutes to an hour or more. My 120GB VHD took about 90-120 minutes. Executing the export-image command will produce various output including a task ID, which can be used to check the progress of the import:

aws ec2 describe-import-image-tasks --import-task-ids import-ami-<task ID>

I ran this several times and could see it progressing.

Upon completion the message, "AWS Windows completed..." is displayed, along with the AMI ID.

Step 10 - Launch EC2 Instance
Finally, an EC2 instance can be launched from this newly created AMI.

aws ec2 run-instances --image-id ami-<task ID>

Step 11 - Post Launch
At this point I logged onto my newly created VM to verify all my applications and data were intact. Since they were I removed my "old" physical server from my domain and joined this one. Success!

Troubleshooting VM Import/Export
A variety of issues can cause problems, i.e. failure. I would suggest reading the prerequisites for VM Import and "Troubleshooting VM Import/Export" AWS pages before beginning to make sure you can avoid issues, then be able to troubleshoot if necessary.

Good luck and happy importing!

Saturday, December 17, 2016

HowTo: Install AWS CLI on Both Windows and Linux

Several years ago before the unified AWS CLI was released I wrote about installing the EC2 command line tools. Now it's time to update that.

It goes without saying that over the decade Amazon has been providing cloud services their interfaces and tools have matured greatly (along with their overall offerings). Early on Amazon didn't even have web console and we had to rely on a disparate offering of command line tools for managing S3, EC2, ELB, etc. Finally, in the fall of 2013 Amazon released the AWS CLI, which is a unified set of command line tools that works similarly across platforms (Windows, Linux, Mac) and across AWS services. It's definitely nice to use the same (or nearly the same) syntax on both Windows and Linux machines to perform various functions.

Installing AWS CLI on Linux (Ubuntu)
The easiest way I've found to install and update the AWS CLI on Ubuntu (and Debian-based linux distros) is using apt, with this simple one-liner:

sudo apt install awscli -y

Once installed or updated check the version with:

aws --version

Installing AWS CLI on Windows
Assuming you are installing on Windows 2008 R2 or later, we'll leverage the native capability of PowerShell to use Invoke-WebRequest (AKA wget) to retrieve the latest installer. I'm also assuming you have a directory c:\temp. If not, create it or use a directory of your choosing. Note: these steps can be used to update the AWS CLI to the latest version as well. If you aren't running at least PS v3 you can update PowerShell to get this functionality. And, you'll have to restart PowerShell if you are doing a new install of the AWS CLI to access them.

To download the latest version of the installer in PowerShell run:

wget -uri https://s3.amazonaws.com/aws-cli/AWSCLI64.msi -OutFile c:\temp\AWSCLI64.msi

Launch the installer (I'm using the /passive switch to accept defaults and not have to answer any questions):

c:\temp\AWSCLI64.msi /passive

In under a minute I am able to download and execute the installer, which works both for first time installs and updating to the latest version. Just like on Linux the version can be checked with (make sure the installer completed before running):

aws --version

Whether you are using the AWS CLI on Windows or Linux you can run the aws configure command to enter the default values; AWS Access Key, Secret, default region and default output type. Then you can run AWS commands to your heart's content.

Once the CLI is installed and configured you can run various commands to test connectivity to AWS services.

aws s3 ls s3:// (list all S3 buckets)

aws ec2 describe-instances

Friday, September 16, 2016

Install PowerShell v5 on Windows Server 2008 R2 or 2012 R2

Prior to updating to PowerShell version 5 make sure to be running the latest version of .NET Framework (.NET 4.6.1 as of this writing). See Checking .NET Versions.

Check Current Version of PowerShell
Run either $Host.Version or $PSVersionTable.PSVersion

Install latest version of .NET.

PowerShell 5 is part of Windows Management Framework (WMF) version 5 and can be downloaded from Microsoft (2008 R2 or 2012 R2). Select the correct download for your version of Windows Server and run the installer. After completion verify the version.

Wednesday, September 14, 2016

Install .NET 4.6.1 on Windows Server 2008 R2 or 2012 R2

Start by downloading .NET 4.6.1 from Microsoft. Launch the installer - I typically do this by running the command
from an elevated PowerShell. Accept the licensing agreement and wait....

Check Version(s) of .NET Installed
From an elevated PowerShell run the following set of commands:

Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP' -recurse |

Get-ItemProperty -name Version,Release -EA 0 |

Where { $_.PSChildName -match '^(?!S)\p{L}'} |

Select PSChildName, Version, Release, @{



      switch($_.Release) {

        378389 { [Version]"4.5" }

        378675 { [Version]"4.5.1" }

        378758 { [Version]"4.5.1" }

        379893 { [Version]"4.5.2" }

        393295 { [Version]"4.6" }

        393297 { [Version]"4.6" }

        394254 { [Version]"4.6.1" }

        394271 { [Version]"4.6.1" }




The result will look something like this.

Thursday, May 14, 2015

Use PowerShell to Save Excel Worksheet as CSV

I'm working on a PowerShell script to launch some virtual servers on VMWare based on input from a CSV file. To create the CSV file I'm using Excel, and I have more than one worksheet in the Excel workbook. Since it's a bit of a hassle to use Excel's GUI to save a given worksheet as a CSV, especially when this has to be done more than once, I set out to find a way to do this programatically. At first I was indifferent on using either a macro within Excel, or a PowerShell method. I decided on PowerShell because I'm already using a PS script to launch the servers, so I decided I would write a block of code to open my spreadsheet and create a CSV from the desired worksheet.

Excel spreadsheet file.
In my quest to figure this out I ran across a lot of pieces, but not a complete solution, and as is often the case on the web I couldn't find explanations of some of the solutions. That's what prompted me to save this all here. I'll step through some of the logic, then post the script in its entirety at the bottom. The best resource I found, although not complete (for my needs), was this great post from LazyWinAdmin.

First thing is to set some variables specific to my environment, the path and name to both my input (xlsx) and output (csv) files, and the name of the tab or worksheet in the spreadsheet.
$Sheet = "CreateVM-CA"
$xls = "C:\Temp\PC\PCTest.xlsx"
$csv = "C:\Temp\PC\$Sheet.csv"

NOTE: Make sure you save your Excel file as this script reads the file from disk & not what's in an open workbook.

Next we will open Excel, the workbook, and the specific worksheet we want to export. I'm also using "Excel.Visible = $False" to prevent the Excel GUI from opening, and "Excel.DisplayAlerts = $False" to prevent messages such as being prompted to overwrite the file if it already exists (see below).
$objExcel = New-Object -ComObject Excel.Application
$objExcel.Visible = $False
$objExcel.DisplayAlerts = $False
$WorkBook = $objExcel.Workbooks.Open($xls)
$WorkSheet = $WorkBook.sheets.item("$Sheet")

Prompted to replace existing file (when not using DisplayAlerts=$False)
A useful note (again thanks to LazyWinAdmin) is that we can verify excel has opened, and we are working with the desired workbook using the following, which lists the available worksheets within the opened workbook.
$WorkBook.sheets | Select-Object -Property Name

Listing Excel worksheets with PowerShell.
Saving the file. When saving the file we will, of course specify that it is a CSV, but also the file type (CSV) is designated with the number 6. For some reason you can't pass the number 6 directly in the SaveAs command, so we are using the variable xlCSV. In my discovery I ran across a list of several file types, but cannot find that now. It's out there somewhere....
$xlCSV = 6

Closing/quitting Excel. Initially I was using just the first command to quit Excel. However, I discovered that it remained open and subsequent executions of my script would fail to write the file because the open Excel process had the file locked. So after a little research and testing I found the second command which kills the process, and prevents the CSV file from being locked.

This little script works beautifully now and does just what I need by opening a specific worksheet from an Excel workbook, then saving it as a CSV file. Here's the entire script.
$Sheet = "CreateVM-CA"
$xls = "C:\Temp\PC\PCTest.xlsx"
$csv = "C:\Temp\PC\$Sheet.csv"
$objExcel = New-Object -ComObject Excel.Application
$objExcel.Visible = $False
$objExcel.DisplayAlerts = $False
$WorkBook = $objExcel.Workbooks.Open($xls)
$WorkSheet = $WorkBook.sheets.item("$Sheet")
$xlCSV = 6

Output CSV file.

Sunday, April 19, 2015

History of Internet Addressing

Volumes have been written about the Internet and its protocols so what I have here is nothing new or ground-breaking. It is, however, a good overview of the history of IP addressing, related Request For Comments (RFC) and other links.

I've been working with the Internet Protocols for a while, including data center builds, network operations and teaching various Cisco (CCNA), Microsoft (MCSE), Novell (CNE), etc. certification and training courses over the years. And although I do know a fair bit about the Internet and IP in particular I actually learned a few new things recently. Or I should say while doing a little research for a new project I had some thoughts gel in my mind which led me to record this information. This will be a useful resource for me, my future students, and hopefully you as well.

If you'd like to cut to the chase scroll down to the summary/timeline at the end.

Origins of The Internet
Host-to-host communications via packet-switched networks was in its infancy in the late 1960's with the Advanced Research Projects Agency Network (ARPANET), from which grew TCP/IP, AKA "The Internet Protocol Suite", with IP and IP addressing at its core. Two key pioneers of the Internet and the godfathers of IP, Bob Kahn and Vint Cerf, had to go through a lot of experimentation and work before finally coming up with what we know today as IPv4. First we'll discuss the IP versions prior to 4, then subsequent enhancements of IPv4 and finally the next generation of IP. The previous versions of IP where part of the initial experimentation. An example of one of these earlier versions is IPv2 as documented in Draft Internetwork Protocol Description Version 2. This and the other preceding versions are listed in table 1 along with the dates they were published.

Table 1
Version  Date
0        March 1977
1        January 1978
2        February 1978 version A
3        February 1978 version B
4        August 1979 version 4

In January of 1980 Jon Postel, who was editor and/or writer of many RFCs, published RFC 760, "DOD STANDARD, INTERNET PROTOCOL," the first IPv4 RFC. At this point it is important to note no concept of address classes, subnetting, CIDR, etc. were defined at this point, therefore they didn't exist. All of those things came along later to enhance the Internet Protocol addresses. So in this original definition IPv4 used the first 8 bits as the network address, and the last 24 bits as the "host" portion of the address, which was termed at the time Source or Destination Local Address (see RFC 760, pages 14 and 22 for exact wording).

In a 2011 interview, Vint Cerf, on of the Internet Founding Fathers said of IPv4, "I thought it was an experiment and I thought that 4.3 billion [IPv4 addresses] would be enough to do an experiment." He said he never thought, "this experiment wouldn't end." So for an "experiment" IPv4 is doing pretty damn good! Albeit with a few enhancements over the next several years.

Looking back to the 1970's computers hadn't yet reached the mainstream so it wasn't imagined by the Pioneers that within a few short years we would see an explosion of networks and devices connecting to the Internet. This is clearly evident in RFC 762 titled "Assigned Numbers," which lists 33 "networks" out of a possible 254 (0 and 255 being reserved). Again, IP address classes hadn't yet been defined.... This RFC was published in January 1980, but it also lists when the 0-3 Assigned IP versions were initially defined, giving us a little more insight into the timeline of IP address versions.

From this we can see that IP addressing began to be defined around March 1977 with IPv0. Then by August 1979 IPv4, the main version on which the Internet was built and is still widely in use today, was defined. Now, over 35 years on, the Internet runs mainly with the "experimental" IP (v4) addressing created by Vint Cerf and Bob Kahn.

Classful IPv4 Addresses
Fairly quickly it was apparent to these Founding Fathers that the intial 8/24 (nnnnnnnn.hhhhhhhh.hhhhhhhh.hhhhhhhh) addressing scheme initially defined by IPv4 needed an enhancement. In September 1981 RFC 791 defined IP address classes in order to "provide for flexibility in assigning address to networks and allow for the large number of small to intermediate sized networks the interpretation of the address field is coded to specify a small number of networks with a large number of host, a moderate number of networks with a moderate number of hosts, and a large number of networks with a small number of hosts." Since this was just an enhancement or further definition of the use of the 32 bit IPv4 address scheme it wasn't necessary to increment the version.

This enhancement to IPv4 is known as classful IP addressing. Table 2 provides a quick view of the classes A, B and C as a reference. Much more about these classes, etc. are available elsewhere on the Internet.

Table 2
Class Leading Bits Network Bits Host Bits Number of networks Addresses per network Start address End address
A 0 8 24 128 (2^7)  16,777,216 (2^24)
10 16 16 16,384 (2^14)  65,536 (2^16)
110 24 8 2,097,152 (2^21)  256 (2^8)
NOTE: classful addressing was superseded by classless (AKA CIDR) addressing. Although this is the case it is common to discuss and have classful addresses referenced in documentation. While this technically shouldn't be done this network engineer hasn't seen much if any decrease in classful address references in my 25 years in the business.

In 1984 another evolution of IPv4 addressing was introduced, Internet Subnets with RFC 917. This explains both the need for and way to use part of the host portion of the address for a subnet, effectively creating something like nnnnnnn.ssssssss.hhhhhhhh.hhhhhhhh, where a "Class A" address is subnetted by taking 8 bits (as an example, more or less could be used of course) of the host address fields to allow for up to 256 subnets. This necessitated the creation of a way for hosts to know which portion of their address was used for the network and subnet vs. the host portion of the address, therefore the birth of the subnet mask. In the case listed above just about everyone familiar with IP addressing will know the mask would be In August 1985 RFC 950, "Internet Standard Subnetting Procedure," further defines subnetting and subnet masks.

NOTE: In several early RFCs you'll see the term "catenet." This depricated term was defined in the early days of packet-switched networking and referred to a network of networks, or an internetwork or internet (lower case "i").

Governing Bodies
By the mid- to late-1980's organizations like the Internet Engineering Task Force (IETF), Internet Assigned Numbers Authority (IANA), its parent Internet Corporation for Assigned Names and Numbers (ICANN), and others were formed to steer, define and coordinate protocols, growth and enhancements of the Internet and Internet Protocols.

Classless IP Addressing - CIDR
In September 1993 IETF introduced Classless Inter-Domain Routing (CIDR) which is a method for allocating IP addresses and routing IP packets. CIDR replaces the previous addressing architecture of classful addresses, with the goal to slow the growth of routing tables on routers across the Internet, and to help slow the rapid exhaustion of IPv4 addresses. Once again the "experimental" IPv4 address space was modified to extend its life. Due to the lack of scalability of classful (A, B, C, etc.) addresses CIDR provides the ability to "subnet" an IPv4 address at virtually any bit value. This is done by appending the number of bits used for the network (technically network/subnet) to the IP address. For example provides for a network with up to 1022 hosts. Much is available on CIDR around the web, including numerous subnetting/CIDR charts, but a few references are Wikipedia's "Classless Inter-Domain Routing", and RFC 1518, "An Architecture for IP Address Allocation with CIDR", and RFC 1519, "Classless Inter-Domain Routing (CIDR): an Address Assignment and Aggregation Strategy."

Private Addresses, NAT & PAT
As early as the 1980's it was apparent that IPv4's 32-bit address space would become exhausted. A variety of factors contributed to this with the main one being the explosive growth of networks and devices worldwide participating in and connecting to the Internet. So in addition to the aforementioned classful, classless and subnetting of IPv4 network address translation (NAT) and port address translation (PAT) were developed. Currently NAT & PAT are widely deployed, so much so that they are the norm. But, these actually break the end-to-end communications originally envisioned with the Internet and introduce other problems or challenges. This is something IPv6 addresses, but let's not get ahead of ourselves. NAT and more specifically PAT  has become a popular and essential tool in conserving global address space allocations in face of IPv4 address exhaustion.

Of course in order to effectively use NAT/PAT a set of private, non-publicly routable IP addresses had to be defined, which was done in February 1996 in RFC 1918, "Address Allocation for Private Internets." (See also, RFC 5735, "Special Use IPv4 Addresses.")

Table 3 - Private IPv4 Address Ranges     -  (10/8 prefix)   -  (172.16/12 prefix)  - (192.168/16 prefix)

This discussion wouldn't be complete without touching on version 5. It was an experimental family of protocols for streaming voice, video, etc. called Internet Streaming Protocol in 1979, but was never fully developed. If you are so inclined see RFC 1190, "Experimental Internet Stream Protocol."

Along with enhancing IPv4, by 1994 IETF began to define the next generation of IP with IPv6, sometimes called IPng. This took several years and in December 1998 RFC 2460, "Internet Protocol, Version 6 (IPv6) Specification" was published. Since IPv6 was defined about 20 years after IPv4 a lot of the former protocol's shortcomings were addressed. In addition to adding native security to IP with IPSec, restoring the end-to-end communications model (doing away with NAT), IPv6 increases the address space. A lot!

By using a 32 bit address IPv4 has a total of about 4.3 billion (2^32) available numbers. IPv6 is 128 bits, which provides a bit over 340 undecillion (2^128) addresses. Table 3 shows just how large this address space is. With such a large address space it is not necessary to define address classes (no "classful") in IPv6, nor is it necessary to use a subnet mask. Rather, since IPv6 was built upon the concept of Classless Inter-Domain Routing IPv6 addresses are written with a trailing /xx (example: 2001:db8:abcd:3f00::/64). In most cases organizations will receive a /64 or /48 address space from either an ISP or IANA, then they will be able to use CIDR to suit their needs.

Table 4
IP4 addresses (2^32)  - 4,294,967,296
IP6 addresses (2^128) - 340,282,366,920,938,463,463,374,607,431,768,211,456

Although IPv4 was an "experimental" addressing scheme born three decades ago it has seen numerous enhancements with classes, subnetting, CIDR and NAT/PAT to extend its lifespan, and it's still going strong. IPv6 has been around for nearly two decades and its use is definitely picking up steam but I believe IPv4 will be around for quite some time. In fact, it's likely the two will run in parallel perhaps for as long as the Internet exists.

  • Mar 1977: IP addressing first defined and documented, starting with IPv0
  • Aug 1979: IPv4 defined (RFC 760)
  • Sep 1981: IPv4 classful addresses defined (RFC 791)
  • Oct 1984: IPv4 subnetting introduced (RFC 917)
  • Aug 1985: IPv4 subnetting further defined (RFC 950)
  • Sep 1993: IPv4 classless (AKA CIDR) addresses defined (RFC 1518, and RFC 1519)
  • Feb 1996  IPv4 private addresses and NAT/PAT  (RFC 1918, and RFC 5735)
  • Dec 1998 IPv6 defined (RFC 2460)