J Wolfgang Goerlich's thoughts on Information Security
Cost justifying 10 GbE networking for Hyper-V

By wolfgang. 21 September 2011 05:13

SearchSMBStorage.com has an article on 10 GbE. My team gets a mention. The link is below and on my Press mentions page.

For J. Wolfgang Goerlich, an IT professional at a 200-employee financial services company, making the switch to 10 Gigabit Ethernet (10 GbE) was a straightforward process. “Like many firms, we have a three-year technology refresh cycle. And last year, with a big push for private cloud, we looked at many things and decided 10 GbE would be an important enabler for those increased bandwidth needs."

10 Gigabit Ethernet technology: A viable option for SMBs?
http://searchsmbstorage.techtarget.com/news/2240079428/10-Gigabit-Ethernet-technology-A-viable-option-for-SMBs


My team built a Hyper-V grid in 2007-2008 that
worked rather nicely at 1 Gbps speeds.We assumed 80% capacity on a network link, a density of 4:1, and an average of 20% (~200 Mbps) per vm. In operation, the spec was close. We had a "server as a Frisbee" model that meant non-redundant networking. This wasn’t a concern because if a Hyper-V host failed (3% per year) it only impacted up to four hosts (%2 of the environment) for about a minute.

When designing the new Hyper-V grid in 2010, we realized this bandwidth was no longer going to cut it. Our working density is 12:1 with our usable density of 40:1. That meant 2.4 Gbps to 8 Gbps per node. Our 2010 model is "fewer pieces, higher reliability" and that translates into redundant network links. This was more important when a good portion of our servers (10-15%) would be impacted by a link failure.

Let’s do a quick back of the napkin sketch. Traditional 1 Gbps Ethernet would require 10 primary and 10 secondary Ethernet connections. That’s ten dual 1 Gbps adapters: 10 x $250 = $2,500. That’s twenty 1 Gbps ports: 20 x $105 = $2,100. Then there’s the time and materials cost for cabling all that up. Let’s call that $500. By contrast, one dual port 10 GbE adapter  is $700. We need two 10 GbE ports: 2 x $930 = $1,860. We need two cables ($120/per) plus installation. Let’s call that $400.

The total cost per Hyper-V host for 10 GbE is $2,960. Compared to the cost of 1 Gbps ($5,100), we are looking at a savings of $2,140. For higher density Hyper-V grids, 10 GbE is easily cost justified.

It took some engineering and re-organizing. We have been able to squeeze quite a bit of functionality and performance from the new technology. Cost savings plus enhancements? Win.

Tags:

Architecture | Systems Engineering | Virtualization

Matriux - Upgrade to 2.6.32-7 and install the GPL Hyper-V integration

By wolfgang. 14 December 2009 20:59

These steps will install Matriux into a Hyper-V vm (2008 or 2008 R2) and integrate the network and storage adapters.

Create a Hyper-V vm with the legacy network adapter and a 10 GB vhd.
Download Matriux and 
install onto the local vhd



Configure apt-get to download the Lucid (2.6.32-7) kernel.

 

sudo bash

nano /etc/apt/sources.list

 

# added by -JWG- for Hyper-V integration

# The Lucid repository contains the 2.6.32-7 kernel

deb http://archive.ubuntu.com/ubuntu/ lucid main

 

apt-get update

 

Install the kernel and then comment out the repositories.

 

apt-cache search linux-image-2.6.32

apt-get install linux-image-2.6.32-7-generic linux-headers-2.6.32-7-generic build-essential

 

nano /etc/apt/sources.list

Comment out the #deb line

 

Validate the kernel after rebooting to ensure we are on 2.6.32-7.

 

uname -r

 

Enable the GPL integration components.

 

uname -r

sudo bash

cd /lib/modules/2.6.32-7-generic/kernel/drivers/staging/hv

insmod hv_vmbus.ko

insmod hv_blkvsc.ko

insmod hv_netvsc.ko

insmod hv_storvsc.ko

 

Add the modules to the startup file.

 

nano /etc/initramfs-tools/modules

 

# added by -JWG- for Hyper-V integration

hv_vmbus

hv_blkvscb

hv_netvsc

hv_storvsc

 

update-initramfs -u

reboot

 

 

Confirm that the modules are loaded. You will have full network and disk integration. The mouse integration (Inputvsc) is currently provided by Citrix Project Satori and has not yet been patched to 2.6.32-7.

 

lsmod | grep vsc

 

Tags:

Hyper-V | Security | Virtualization

Matriux - Downgrade to 2.6.18 and install Hyper-V's integration components

By wolfgang. 14 December 2009 20:55

These steps will install Matriux into a Hyper-V vm (2008 or 2008 R2) and integrate the mouse, network adapter, and storage adapter.

Create a Hyper-V vm with the legacy network adapter and a 10 GB vhd.
Download Matriux and install onto the local vhd
Download the 
Linux Integration components for Windows Server 2008 R2 (LinuxIC v2.iso).
Download the
Citrix Project Satori mouse driver (Inputvsc.iso)

Configure apt-get to download the previous version of the kernel, which includes first flushing and renewing the encryption keyring.

 

sudo bash

 

apt-key list

apt-key del 437D05B5

apt-key del FBB75451

 

apt-key list should now return an empty list.

 

Install the keyring

apt-get install debian-archive-keyring

 

Load the key for the ftp.us.debian.org and security.debian.org.

 

cd /home/tiger/.gnupg/

mv gpg.conf gpg.con~

 

gpg --keyserver wwwkeys.eu.pgp.net --recv 9AA38DCD55BE302B

gpg --list-keys 9AA38DCD55BE302B

gpg --export 9AA38DCD55BE302B > 9AA38DCD55BE302B.gpg

apt-key add ./9AA38DCD55BE302B.gpg

apt-key list

 

Add the repositories to the end of the sources list, and update the apt list.

 

nano /etc/apt/sources.list

 

# Repository for older kernel versions

# added by -JWG- for Hyper-V integration

deb http://ftp.us.debian.org/debian etch main

deb http://security.debian.org/debian-security etch/updates main

 

cd /usr/src/

apt-get update

 

Install the kernel and then comment out the repositories.

 

apt-cache search linux-image-2.6.18

apt-get install linux-image-2.6.18-6-amd64 linux-headers-2.6.18-6-amd64 build-essential

 

nano /etc/apt/sources.list

Comment out the two #deb lines.

 

Modify the menu.lst file so it defaults to the 2.6.18-6 and reboot.

 

nano /boot/grub/menu.lst

default 2

reboot

 

Validate the kernel after rebooting to ensure we are on 2.6.18-6.

 

uname -r

Insert the LinuxIC v2.iso disk, copy locally, and install the drivers.

 

sudo bash

 

mkdir /opt/linux_ic

cd /opt/linux_ic

cp -R /media/CDROM/* /opt/linux_ic/

./setup.pl drivers

cat drvinstalls.err

 

The only error should be "make: udevcontrol: command not found" and "make: *** [install] Error 127". These simply indicate that we will need to manually add the services to the init modules file.

 

Insert the Inputsvc.iso disk.

 

mkdir /opt/inputvsc

cd /opt/inputvsc

cp -R /media/CDROM/* /opt/inputvsc/

./setup.pl drivers

cat drvinstall.err

 

Again, the only errors should be related to the modules. Edit that file now.

 

nano /etc/initramfs-tools/modules

 

# added by -JWG- for Hyper-V integration

netvsc

blkvsc

storvsc

inputvsc

 

update-initramfs -u

reboot

 

Confirm that the modules are loaded. Then it is play time.

 

lsmod | grep vsc

Tags:

Hyper-V | Security | Virtualization

Matriux - Penetration Testing from Hyper-V

By wolfgang. 13 December 2009 20:18

Matriux is a vulnerability assessment / penetration testing Linux distribution. The team's beta release was the beginning of this month, and I have been playing around with the distro for the past couple weeks. What can I say? I am a sucker for Latin motto's ("Aut viam inveniam aut faciam" or "I shall find a way or make one") and for cleanly laid out VA/PT toolsets.

The bonus, for those running Hyper-V, is that Matriux is a Kubuntu based and comes with the Jaunty kernel (2.6.28-13-generic). Setting up a Hyper-V security appliance is as simple as creating a vm, using the legacy network adapter, skipping the hard drive, and booting off the downloadable ISO. Matriux works right out of the box within Hyper-V.

You can compare this to the Slax VA/PT distros, which do not support the network adapter. Often times, these distros do not even support the mouse. Using the Matriux Live CD in Hyper-V is a breeze. For an environment to support a demo or an occassional vulnerability assessment, you cannot ask for more.

If you are doing regular assessments, there are a couple limitations with Hyper-V. The legacy network adapter performs at 100 Mbps (significantly slower than the 10 Gbps speed of the standard network adapter.) The Live ISO is read-only, too. The mouse integration is present, but it is not the seamless integration one is used with Windows vms. Oh, and the mouse integration does not work when connected to Hyper-V over RDP. To get full functionality, you will need to install Matriux into a vhd and install the Hyper-V integration components.

The Jaunty kernel does not support integration. You have two options: (1) downgrade Matriux's kernel to 2.6.18 and install Hyper-V's integration components; or (2) upgrade Matriux to the Lucid kernel (2.6.32-7) and enable the Hyper-V GPL code. Option (2) provides faster performance and is in-line with the Matriux planned Beta 2, but it does not support the full mouse integration.

Detailed steps for both options are in the links above. For those who want to skip to the chase and simply try out Matriux under Hyper-V, I have done the steps for you. You can download the security appliance from SimWitty's website. Enjoy!

Matriux beta (0.9.4) with 2.6.18-6 kernel
Matriux beta (0.9.4) with 2-6-32-7 kernel

Thank you to the Matriux team for a smooth, well done security distribution beta. Thanks goes, too, to Tom Houghtby for providing the Linux knowledge and guidance that made the integration possible.

jwg

Tags:

Hyper-V | Operations Security | Security | Virtualization

Building our own cloud

By wolfgang. 6 November 2009 10:29

I have been thinking a lot about IT service architecture. After all, my theme this year is "Security is Design". How can we maximize the benefits of new technologies while minimizing the security risks?

Take cloud computing. The buzz is that cloud computing reduces costs and increases scalability. Cloud computing, specifically with cloud hosting, does this by putting our servers in a multi-tenant environment and then charging based on utilization. So organizations get pay-as-you-go pricing that is shared across scores of customers (tenants). Add self-service and rapid provisioning, and you get a fast and flexible solution.

That makes the IT operations side of my brain happy. But then my IT security side chirps up.

Multi-tenant increases security risks as we no longer have end-to-end visibility and control coverage. Think of the property security of an apartment versus a private home.  Multi-tenant decreases responsiveness, too, as the service provider must balance the needs of his organization against the needs of yours. Think the customer service you get from your telephone utility versus your in-house telecommunications specialist. Above and beyond that, simply by being a new architecture, cloud computing will bring an entirely new set of risks that can only be identified with time.

So how can we balance the benefits and risks of cloud computing? One way is to bring the cloud computing technologies in-house. The basics are readily available: virtualization, rapid provisioning, self-service, resource pooling, charge back. A data center built on the cloud computing model, but leveraging the best of an internal IT team: responsiveness, responsibility, and business domain knowledge.

My team has been using the terms "in-house cloud" or "private cloud" to describe our efforts to achieve this balance. This week, vendors led by EMC launched www.privatecloud.com as a resource building such beasts. Check out their definition of private cloud. While the blog is VMware and EMC based, I wager it is only a matter of time before Microsoft and Compellent come out with comparable information.

Done right, private clouds or cloud computing built in-house will provide a smooth transition for organizations to get the benefits of this new architecture.

Tags:

Security | Systems Engineering | Virtualization

Virtualization and the physical security boundary

By wolfgang. 8 July 2009 04:52

There are several laws of information security. Ask ten InfoSec pros and you will likely get ten different lists of laws, but I wager every one of them will agree on a couple fundamentals. If an attacker can gain physical access to the computer, or if an attacker can modify the operating system, then the attacker can compromise the computer. The reason is physical access allows an attacker to bypass the OS and directly access the data, and bypass the security controls.

 

Now, switch gears and picture a virtual environment. The physical analog is the hypervisor. If an attacker can gain access to the hypervisor, he has the same abilities as if he had access to the physical computer. If an attacker can exploit the Windows or Linux server hosting Hyper-V or XenServer, then the attacker can compromise all virtual computers on the host.

 

It is a subtle shift in the way of thinking. In the past, only one server ran on one piece of hardware, and the security boundary was the server itself. Thus you would place a physical web server in the DMZ and physically wire it to the firewalls. Computers with different security postures (e.g., domain controllers) would be on separate physical hardware and wired into separate physical networks.

 

Thus the hypervisor should host servers that have relatively the same security posture. One should not, for instance, host domain controllers and public-facing web servers on the same hypervisor. Even if the public-facing web server is on a separate virtual network, you still run the risk of its compromise affecting the domain controllers.

 

The security boundary is the physical hardware, not the computer itself.

Tags:

Hyper-V | Security | Virtualization

Installing ARCserve on Hyper-V Core

By wolfgang. 28 June 2009 10:04

Hyper-V Core, or the Hyper-V role running on a Server Core installation of Windows Server 2008, provides only a command line interface. This makes installing management apps a bit tricky.

 

Take CA ARCserve Backup agent, for example. You cannot simply logon and run the installer. Rather, you need to use the management console that comes with ARCserve (r12.5). Use the management console to push out the agent to the Server Core.

 

The normal caveats apply to push installations. Both the management console and the Server Core computers should be on the same network. Both computers should be in the same Windows domain (or have a domain trust relationship setup.) Ensure the Windows firewall on the Hyper-V Core is accepting inbound file (CIFS) and procedure (RPC) requests. Once those are accomplished, pushing the agent is straightforward.

 

Similar procedures apply to Diskeeper and anti-virus software.

Tags:

Hyper-V | Virtualization

Disposable end-point model

By wolfgang. 26 May 2009 03:52

One project in my portfolio at the moment is building what I call a disposable end-point model. It is a low priority project, but an ongoing one. The goal is to deliver the best user experience at the lowest price-point.

Portability is a must. Think about the concerns over swine flu and the like. What is your pandemic plan? My pandemic plan, at least from a technology standpoint, is straightforward. People work from home over the vpn and run apps from Citrix. So the end-point devices must be portable and dual-use.

Yet traditional notebooks are expensive. My firm, like most, has an inventory of aging notebooks. These older computers are costly to maintain (studies show ~$1K per device per 2 years) and replace if lost or stolen (studies show ~$50K per incident).

The sweet spot are computers that are cheaper than supporting aging devices and disposable if lost or stolen. No local data means no security incident, which erases the risk exposure of stolen devices. These inexpensive computers should be light-weight and easily ported from office to home. So I am looking at netbooks, which run around $500.

I spoke with Jeff Vance, Datamation, about these ideas. He recently wrote an excellent article that summarizes the netbook market and how data center managers are looking to use the devices: Will Desktop Virtualization and the Rise of Netbooks Kill the PC?

Tags:

Architecture | Security | Virtualization

Virtualization and BCP Webinar Today

By wolfgang. 20 April 2009 08:01

How My Firm Reduced Costs and Delivers Agile IT Infrastructure through Virtualization

Virtualization has become an important part of many organizations’ IT strategy for 2009 and beyond.

The availability of both data and IT systems are at risk not only from natural disasters but also power outages, human error and hardware failures. To ensure that your company can quickly recover systems and data in the event of such an incident, it needs a reliable and cost-effective disaster recovery plan. The use of virtualization and data protection technology combined helps you control costs as your company grows, which is essential in any economic climate.

My firm has deployed CA ARCserve Backup and Microsoft Hyper-V server 2008 to create a simple and scalable disaster recovery environment. The combined solution is responsible for backing up around 36 terabytes of data every week.

Join this webcast to find out how CA ARCserve Backup, combined with Microsoft Hyper-V 2008, can work in tandem to protect many terabytes of data, and deliver an agile, cost-effective IT infrastructure.

In this webcast, you’ll also hear:

  • How to utilize CA ARCserve Backup to restore single files 
  • How CA ARCserve can backup your physical environment and also restore virtual instance 
  • Microsoft’s Virtualization Strategy 
  • The role Microsoft’s Hyper-V plays today and what you can expect in the upcoming release of Windows Server 2008 R2 
  • How in the event of a disaster, my firm is able to recover 86 physical servers to 12 standby servers in just two hours 
  • How my firm has been able to minimize not only downtime but also its spend on disaster recovery utilizing this combined solution 

Eric G. Pitcher
Eric Pitcher is vice president of technology strategy at CA, responsible for setting and communicating CA’s Recovery Management plans across the business unit, throughout CA and to partners and customers. Previously, Eric served as vice president of product management at CA, responsible for defining the process, requirements and product specifications for CA’s Recovery Management product lines. Prior to that, Eric worked as assistant vice president of CA’s research and development global SWAT team—a specialized task force designed to maximize the quality, customer satisfaction, and market competitiveness of CA’s storage management solutions.

Before joining CA, Eric was network and systems administrator at Universal Studios Florida and was responsible for server and network design, installation, administration and support on a network of more than 1,000 users. Eric earned a bachelor of science degree in business administration from the University of Central Florida.

Wolfgang Goerlich
J Wolfgang Goerlich, CISSP, CISA, is an information security professional who focuses on performance and availability. Mr. Goerlich is currently ... the network operations and security manager. With ten years of experience, Mr. Goerlich has a solid understanding of both the IT infrastructure and the business it enables.

Isaac Roybal
Isaac Roybal is a Product Manager in Windows Server managing the Server Virtualization, including Microsoft’s Hyper-V, and has been involved with IT for over twelve years. Seven of those years have been with Microsoft. Isaac’s career started in Systems and Network Engineering working with VMS, Windows Server since NT 3.51 and IIS 4 in various capacities.

Tags:

Business Continuity | Virtualization

Virtualization Webinar next Monday

By wolfgang. 13 April 2009 16:44

As I mentioned before, I have played around a bit with Hyper-V and virtualized my production and recovery systems. CA did a case study on the project.  This coming Monday, April 20 at 12:00 pm Eastern, I am doing a joint webcast with CA and Microsoft. The topic is still virtualization with the focus on disaster recovery. I doubt I will say anything new during the talk, excepting the talk will be much briefer than some others I have given on DR. CA’s going to talk a bit about their CDP, however, which is pretty cool stuff.

http://www.windowsitpro.com/go/seminars/CA/infrastructure_through_virtualization

Tags:

Business Continuity | Hyper-V | Virtualization

    Log in