Virtual machine drops ISCSI drives during Vmotion

celerra 1Recently during maintenance downtime early one Sunday morning, our VMware administrator was v-motioning numerous VM’s to patch the host environment blades on the Cisco UCS. One of the VM’s was a Windows 2008 R2 server that contained over 10 iSCSI connections to targets residing on an EMC Celerra NS-40G NAS. All migrations were going well until this particular VM was moved: all of the iSCSI drives disappeared from the Windows OS. A vMotion process is a transparent process and this should not have happened, but this time it had a negative effect on the VM: an unexpected glitch. This happens time to time. The iSCSI initiator in the Windows Server OS still registered the connections as connected and online, but the OS disk management would not see the drives.

celerra iSCSI targets

I disconnected and then re-connected the targets in the iSCSI client, but the Windows OS would still not see the drives. I restarted both the iSCSI initiator and Server service in the Windows OS with the same result. Re-scanning for storage in Windows disk management did not help. EMC tech support ran a check on the NAS end and came up with nothing. Eventually, I went into Celerra Manager and deleted the LUN masking for the target and then re-added it. The Windows Server OS was then able to see the drives. I was hesitant to do his at first, as I was unsure what effect it would have on the drive letter assignments on the server: it had no effect on that – all drives were reestablished as the previously were.

celerra iSCSI target LUN masking

Importance of disk offset

SAN Guy at CLARiiON BLOGS has a good article that explains disk alignment. A very important subject regarding performance, this issue has recently come up at my data center: a EHR application is experiencing performance issues due to fragmentation (of course), and improper disk alignment on the LUN. Our DBA just ran a SQL 2005 best practices on the database that resides on this particular LUN, and it spit out the recomendation of a 64k offset. SAN Guy taught me a few things on this important subject.

An overview of MAID

ubuntu-serversSGI, a technical computing manufacturer, produces a line of storage systems that incorporate green technologies. The COPAN line of storage products use the MAID storage methodology that presents the ability of powering down unused disks in the array while not in use, conserving power as a result. This is managed via internal “Power Managed RAID software” (Sgi, 2010). MAID, the acronym for Massive Array of Idle Disks, is suited for the environment that consists of long-term storage that requires write-once, read-occasionally data (WORO). MAID can be used in disk (virtual) libraries, also known as EDL (Enterprise Disk Library). This is energy efficient storage. SGI’s COPAN solution could be some tough competition against the Centera line from EMC. As of the time of this writing, I have not yet seen MAID technology on the archive systems from EMC.

Massive Array of Idle Disks (MAID) consists of a large disk group, consisting of hundreds or thousands of disks configured into RAID groups. Through internal power management code, only drives that are needed are activated at a time. This reduces drive wear and power consumption.


Sgi. MAID vs. Enterprise MAID. (2010). Retreived November 2, 2010 from

Thesis research: SSD vs. FC drive benchark tests – part I

typewriterI am writing my graduate thesis on the subject of Solid State Drive (SSD). By the way, D stands for DRIVE, not DISK, as SSD does not use disk. Now, with that out of the way…

I have been benchmarking a new SSD array that I have added to my companies SAN: an EMC CLARiiON Cx4-480 system running on 4Gb/s fiber. It will be 8Gb/s soon, but we are waiting on the NAS code (an EMC NS-40G) to catch up so it will support 8Gb/s: the firmware on the NAS only supports up to 4G currently. The SAN is held together with two Brocade 4900 FC switches.

About the disks that I will be testing and comparing:

Disks used: (5) EMC 70GB SSD and (5) 300GB FC disks.


66.639 raw capacity -FC SSD – Manufacturer: STEC – Model: ZIV2A074 CLAR72 – Serial: STM0000E9CFD – 4Gbps


268.403 raw capacity – FC – Manufacturer: SEAGATE – Model: STE30065 CLAR300 – Serial: 3SJ09XWW – 4Gbps

  • Created RAID5 (RAID group 100) on five SSD model ZIV2A074, 66.639GB each.
  • Creating RAID5 (RG 101) on five 300GB FC disks: Seagate 15K.7 ST3300657FC
  • LUN 104 is assigned drive letter W: (disk 3) (RG100) and named “SSD
  • LUN 108 is assigned drive letter X: (disk 4) (RG101) and named “FC

The test server was installed and set up with one dual-port 4Gb/s HBA. Windows Server OS Standard with 1GB RAM.

SAN Management: EMC Navisphere 6.28.21.

Network: 4Gb/s fiber. 2-Brocade 4900B FC switches. Host HBA: Emulex LightPulse FC2243.

Host connection via EMC PowerPath v5.2

Test I/O is generated by Microsoft SQLIOSim; I/O generation utility to similate I/O patterns found within versions of Microsoft SQL server. The versions simulated are Microsoft SQL Server 2005, SQL Server 2000, and Server 7.0. Brent Ozar, a SQL expert, has a good video on using SQLIO on his web site at I have learned some things from him and am using the tips on SQLIO for my benchmarking.

The monitoring will be done with EMC Navisphere Analyzer and SUN StorageTek Workload Analysis Tool (SWAT).

Here is a preliminary test on SSD vs. FC data rates using SQLIOSim to generate I/O and SWAT to record the results:


SSD performance survey data rate


FC performance survey data rate

So far there is not much of a difference. The Fiber Channel drives are keeping up with the SSD. Of course, this is a preliminary test and other tests at this time are giving similar results. I continue to plan my testing methodology.

Measuring disk performance

Here is a video on SearchDataCenter that I found that explains that noise can effect disk vibration:

About EMC Celerra NAS checkpoints

Here is a good intro on EMC Celerra NAS checkpoint technology. Checkpoints, also known as snapshots, are point-in-time images of a file system. These can be used for a quick system recovery in the event of file system corruption or loss:

EMC CLARiiON Cx4 integrated thin provisioning

virtualizationWe just upgraded to the new EMC CLARiiON Cx4 from the Cx3 over the weekend. The technician who arrived had done a good job: in about seven hours, the SAN was fully functional and we were back in business. We took this downtime to do some patching on host systems. The Cx4 has 8Gb fiber ports as compared to the Cx3 4Gb ports; however we could not use the 8Gb modules as the NAS, an EMC NS40g, contains code that is not 100% compatible with the Cx4 fiber module code. We will run it all on the 4Gb/s fiber modules for now until the NAS code catches up, and then the 8Gb modules will be installed. This is not a problem, as the 4Gb modules are still very fast and provide sufficient throughput for our electronic health records system (EHR).

One useful improvement about the Cx4 is the addition of thin pools – one can create RAID groups that use minimal storage and can expand dynamically when the need for more storage arises. See the white paper from EMC on virtual (thin) provisioning here.

LUN migration on EMC CLARiiON

dataminingSince we are upgrading our EMC CLARiiON CX3 to a CX4 this month, EMC has suggested that I get some LUN’s off of the vault (I didn’t put them there) and migrate them to another series of LUN’s. The vault is an array of disks used for storing the system cache – I never liked having other LUN’s on it. So, I am migrating LUN’s off of it, and this is taking some time due to the amount of data. This can be used to move the LUN to a bigger LUN or just to move it for performance reasons. To migrate a LUN is simple:

First, it is required to have some spare LUN’s on a RAID group(s) that are the same size or greater as the source LUN. Right-click on the LUN to migrate (source) and select migrate. A selection of prospective targets LUNs will be displayed. Select the target and the migrate. Very simple. The target will assume the identity (LUN number) of the source LUN when the migration is completed, and this is entirely transparent to the host that is using the LUN and there will be no interruption in LUN access on the host. The migration could take some time depending on the amount of data, of course.

If migrating to a larger LUN, it would be good if the disk were dynamic on the host, if possible. Then, one could just rescan the disks (on Windows Server) and the new larger space would be enabled. If the partition on the host is basic, one would have to use Microsoft Server’s DiskPart – an effective but risky utility.

A review of IP traffic on its effect within SAN

This brings iSCSI to mind. According to Network Magazine, iSCSI makes SAN data accessible to everyone, as this enables a node to access a storage target and set it up for access so that it appears to the OS as a local drive. The advantage of an IP transport in comparison to Fiber Channel is that it is possible to scale a SAN over a longer distance. “TCP/IP can run over any physical medium, so it offers a choice of topologies… and IP packets can travel across alternative LAN topologies just as easily ” (Hall, 2004). This can happen only if the network hardware is sufficient to support the additional Gigabit traffic that iSCSI causes, as iSCSI produces large amounts of traffic bursts. As TCP contains flow control that is very sensitive to loss and delay of packets which could result in negative performance impact, it is essential to implement the fastest switching and routing protocols such as Gigabit switches and router RIP. Currently, there are new 10Gbs Ethernet switches on the market that can more than handle the increase traffic bursts of iSCSI such as Intransa’s 10G Ethernet storage appliance that is designed for the demands of video on-demand, IPTV, digital imaging, video surveillance, and data mining” (Messmer, 2009) which demand those higher speeds. Let’s not forget the backup of data within the IP SAN. Data backup is the cause of a large amount of SAN traffic both on the wire and on the storage targets. Using a LAN-free approach, such as having a backup server within the SAN would eliminate much (IP) backup traffic so that the backup data is not flowing between the client node and the backup server, but only between the backup server and the storage target. As stated in Arkeia’s article on its new backup software, Arkeia Network Backup version 6.0, the “…Server for SAN option allows LAN-free backup for SAN environments, thus easing network congestion. It allows direct SAN connectivity of servers, storage arrays and tape libraries” (Connor, 2006).


iSCSI: Storage Networking Sans the SAN? — Farewell to Fibre Channel? iSCSI lets storage traffic run over a regular TCP/IP network.(Internet SCSI)(Storage Area Networks)(Transmission Control Protocol/Internet Protocol).Eric A. Hall. Network Magazine (May 1, 2004): p62. (3170 words) 

Messmer, E. (2009). Intransa debuts 10G Ethernet storage appliance; Latest StorStac appliance offers up to 180TB of storage capacity. Network World. Academic OneFile. Gale. BCR Regis University.

Connor, D. (2009). Arkeia rolls out new backup software. Network World. Academic OneFile. Gale. BCR Regis University.

IP as a viable transport for storage networks

netwrokIP is a viable transport method within SAN’s, as this is a great method for attaching additional client nodes to the SAN without the need of purchasing and installing additional (and expensive) HBA’s for the nodes. iSCSI is one of the tools that enables this to happen. With the use of a software iSCSI initiator, the node can view an assigned iSCSI file system on the storage end and attach to it as a local disk.

Another advantage of IP within a storage network is the utilization of the spanning tree protocol within TCP. Spanning tree monitors redundant links and assigns alternative network paths on demand in the event of a port failure. It also prevents any packet stream from accessing more than one path at a time, preventing packet or broadcast flooding in the network. Think of it as a traffic cop. Spanning Tree is valuable for SAN traffic, as it will keep the data going in the event of a link failure. This is known as transparent bridging in the text (p.81). With Fiber Channel, there is usually a second fiber switch that will enable continuity of the connection if the other goes down. It seems to me that IP can provide more links to the target.

Spanning Tree is being replaced presently in a few organizations by the new Flex Link protocol:

“Flex Link is a Layer 2 availability feature that can co-exist with spanning tree. This enhancement allows a convergence time of less than 50 milliseconds. In addition, this convergence time remains consistent regardless of the number of VLANs or MAC addresses configured on switch uplink ports. It is a pair of a Layer 2 interfaces, either switchports or port channels, that are configured to act as a backup to another Layer 2 interface. The feature provides an alternative solution to the spanning tree protocol (STP), and it allows users to turn off STP and still provide basic link redundancy”


Cisco. (2009). How to configure a Flex Link for link-level redundancy in Cisco Catalyst switches that run Cisco IOS. Retrieved July 16, 2009 from