Mass Update Media in vCloud Dicrector

In vCloud 5.1 it is very difficult to move media from LUN to LUN once it is uploaded. In vCloud 5.5 this gets better but it is still a pain.

One method I have found that works is to disable the LUN you want the media moved off of, and change something on the media record. Something such as adding a ' ' (space) to the description will cause vCloud Director to copy the media to a new LUN.

Below is a script to mass update media entires to allow them to move easier.

$MediaToUpdate = Get-Catalog -Org NAME | Get-Media

Foreach ( $Media in $MediaToUpdate ) {

$RefMedia = Get-Media -Id $Media.Id

Write-Host “Updating “ $RefMedia.Name $OldDescription = $RefMedia.Description

$OldDescription += "."

$Media.ExtensionData.Description = $OldDescription

$Media.ExtensionData.UpdateServerData

}

Just make sure the LUN you want cleaned off is 'disabled' and you have enough space on the other LUNs for the newly copied/moved media.

Relocating a VM in vCloud Director

Occasionally I need to move a VM or vApp from datastore to datastore, various reasons why. Sometimes I need to break the linked clone tree and force vCenter to consolidate it. Sure I can power off the VM and execute a 'consolidate' via the UI, but what if I need to keep the VM powered on? A storage vMotion does just that.

Sure you can fire up your vCenter Client or WebClient and move the VM that way. If you like your cloud / VM and like them running I would not do that.

Instead run this tid-bit of code in a PowerShell window.

To move a vApp:

$destDatastoreName = “LUN NAME"

$vm_names = Get-CIVApp "VAPP NAME" | get-civm

foreach ($vm in $vm_names) {

$dsQuery = Search-Cloud -QueryType Datastore -Name $destDatastoreName

$dsRef = New-Object vmware.vimautomation.cloud.views.reference

$dsRef.Href = "https://$($global:DefaultCIServers[0].name)/api/admin/extension/datastore/$($dsquery.id.split(':')[-1])" $vm.ExtensionData.Relocate($dsRef)

}

To move a single VM:

$destDatastoreName = “LUN NAME"

$vm_name = get-civm "VM NAME"

$dsQuery = Search-Cloud -QueryType Datastore -Name $destDatastoreName

$dsRef = New-Object vmware.vimautomation.cloud.views.reference

$dsRef.Href = "https://$($global:DefaultCIServers[0].name)/api/admin/extension/datastore/$($dsquery.id.split(':')[-1])" $vm.ExtensionData.Relocate($dsRef)

An important rule to remember... a VM can only live on 1 datastore, this means all VMDK's and the VMX of that VM must exist on the same datasore. Executing the above will make sure that happens. If you fell like breaking things and use the vSphere Client, keep this in mind.

Remove vApp Template With No Catalog Item Link

First... Did you know that vApp Templates and Catalog Items are two different things in vCloud Director? If so, did you know that you can have a vApp Tempalte that does not link to a catalog item? Finally, you can't have a Catalog Item without a vApp Tempalte backing though.

Well I guess you could. If you manage to get one of those, you will want to follow my guide on fixing that here.

Okay back to detecting if you have a vApp Template that is not mapped to a Catalog Item. To detect these 'hung' vApp Templates run the following script as a 'system administrator'

$VappTemplates = Search-Cloud -QueryType AdminVappTemplate

foreach ( $VappT in $VappTemplates ) { $vAppTemplate = Get-CIVAppTemplate -ID $VappT.Id

 $vapptid = $vapptemplate.id
 $CatItem = Search-Cloud -QueryType AdminCatalogItem -Filter Entity==$VappTId
 if ( !$CatItem ) { Write-Host "Get-CIVappTemplate -ID $($vAppTemplate.Id) " }

}

This will procduce an out put of 'Get-CIVappTemplate' and the ID of the hung vApp Template. It is up to you to pick what to do with that hung vApp Template. I recommend removing them and re-creating whatever they were. To do that append this to the output and run it in the same PowerShell window:

| Remove-CIVappTemplate -RemoveCatalogItem $False

Detecting and Removing Invalid vApps

After you run vCloud for a while, heck run any management product for a while, and you will have left overs, hung items, or even corrupt vApps.

With vCloud Director you can run the below to find those items and remove them.

$badvapp = Search-Cloud -QueryType AdminVapp | Where-Object { $_.Status -eq “UNRESOLVED" }

foreach ( $id in $badvapp ) {

Get-CIVApp -Id $id.id | Remove-CIVApp

}

This probably wont remove everything, those items that are left, generally can only be removed via editing the database (I would not recommend that).

Detecting and Removing Invalid vApp Templates

In the event that a vApp Template or Catalog Item fails during the upload process parts of that vApp Template can be left behind. To find and remove those items run the following:

$badvappt = Search-Cloud -QueryType AdminVappTemplate | Where-Object { $.Status -eq "FAILEDCreation" }

foreach ( $id in $badvappt ) {

Get-CIVAppTemplate -Id $id.id | Remove-CIVAppTemplate -RemoveCatalogItem $false

}

Run the first part to see what there is, run it all to remove broken vApp Templates.

You will need to download the custom powershell pack from here.

Modify Bulk vApp Network Attachments

With vCloud Director if you need to change the Organization vDC Network, for instance you run out of IP addresses. You can use the script below to change the attached network of all powered off vApps.

$vapps = search-cloud -QueryType VAppOrgNetworkRelation | Where-Object { $.OrgNetworkName -eq "Old Network Name" -and $.Status -eq "Stopped” }

foreach ($vapp in $vapps) {

get-civapp -id $vapp.id | Get-CIVAppNetwork -ErrorAction SilentlyContinue | Where-Object { $_.ConnectionType -eq "Routed" } | Set-CIVAppNetwork -ParentOrgNetwork "New Network Name"

Write-Host "Updated " $vapp.id " Owned by: " $Vapp.OwnerName

}

Simply change the New Network Name and Old Network Name in the commands above.

Query vCloud Director for Storage Profile Usage

Often times an organization will have multiple Storage Profiles, different class fo storage, etc. The simple command of:

Get-OrgVDC

Returns a summary or cummulative view of the storage profile usage. To get the actual usage for each profile, execute this command:

$OrgVdc = get-orgvdc (Org VDC Name)

search-cloud -querytype AdminOrgVdcStorageProfile | where { $_.VdcName -eq $orgvdc } | get-ciview

Find Broken Catalog Items in vCloud 5.x

Sometimes when uploading content into vCloud there is a chance that the content can be broken. This can cause programs to have errors. To find these items, run the following in PowerShell as the a 'system administrator'.

search-cloud -QueryType AdminCatalogItem | Where-Object { $_.Status -eq "Unknown" }

You can now remove them utilizing the API or vCO and the 'Remove Catalog Item' workflow.

Monitoring Compute Performance of a VM

Even when using vC Ops from VMware, vCenter metrics, vCloud Metrics, and Hyperic Montoring there are things that can be missed.

What if the datastore shows 1ms latency on the vSphere host, the CPU usage is low, the Memory is not being swapped, but a user insists it is just 'slow'. What would you check?

In our cloud deployment (25,000 VMs in a single vCloud instance, and we have 4); I have deployed 4 - 6 VMs per vCenter across differnet datastores, on different hosts. These VMs are 512MB of RAM, 16GB of Hard Drive, and 1 vCPU. On these VMs I installed CentOS, MySQL and sysbench.

sysbench for those that do not know runs artificial tests against the HDD, RAM, CPU and MySQL. Since we have the CPU, RAM and HDD monitored by other products, I configured it to only point at the MySQL server. The MySQL test will test every portion of the compute stack. While the RAM and CPU may report within acceptable ranges, they maybe high in those ranges. Combinded that may cause a performance impact.

My script runs every 5 minutes via a cron job, and logs the data to a SQL database, for later reporting and analysis.

I won't cover how to install sysbench or MySQL here, since that can be found else where on the internet. sysbench & MySQL

You will also need to install iSQL for CentOS, to write your data to a SQL server (if that is your target), those instructions can be found here.

After you have the above installed and ready, you can run the following to get a report on the status of your VMs performance:

sysbench --num-threads=16 --max-requests=10000 --test=oltp --oltp-table-size=500000 --mysql-socket=/var/lib/mysql/mysql.sock --oltp-test-mode=complex --mysql-user=root --mysql-password=VMware1! run > mysql.sysbench

$testvaule = cat mysql.sysbench | egrep " cat|transactions:" | awk {'print substr($3,2) '}

$date = date -u "+%F %R"

$machine = ifconfig eth0 | grep inet | awk '{ print substr($2,6) }’

$hostname = hostname

inssql="insert into TABLENAME VALUES (‘$date', ‘$machine', ‘$testvalue’,’$hostname')"

echo $inssql | isql HOSTNAME USERNAME PASSWORD

The above will insert the data in to a SQL table, for later reporting.

Keep in mind you do not want to make a 'monster' VM, the smaller the better, you want to tax all of the components. If you give it 4GB of RAM, then MySQL will cache all of the disk I/O in RAM and not report an accurate number. If you give it 4 vCPU's you may actually artificially lower your score by causing the hypervisor to schedule 4 vCPUs worth of tasks. In this case smaller is most definitely better.

This will not replace the other monitoring solutions you may have, but it will help to augment those solutions.

Ejecting CD ISO's in vCloud Director

With vCloud Director 5.1 every once in a while a CD will refuse to eject, or worse it will say it ejected the CD but it will not make the change in vCenter.

To get around this with PowerCLI connected to the vCenter use this command:

get-folder | get-VM | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$false

This will allow the CD to be ejected without having to Deploy, Eject, Re-Capture the vApp Template. To use for a normal vApp use the vApp Name/UUID.

vCloud Director and VMware Horizon Intergration

Requirements for SSO in vCloud Director

The SAML process and parties using them have formal roles and responsibilities:

  • Service Provider – An entity that receives the SAML Message from an Idenity Provider (vCloud Director )
  • Identity Provider – An entity that authenticates an user (Horizon)

The Identity provider must provide these fields in the response (case sensitive):

  • UserName
  • EmailAddress
  • FullName
  • Groups

While the only required field is UserName it is recommended that all fields be returned with proper information to allow for ease of management and administration of the users and their rights.

Configuring vCloud Director

In vCloud director you will first need to enable the use of a SAML Identity provider.

  1. Once logged in as a Organization Administrator or System Administrator go to the organization Administration page, click on Federation, then enable the ‘Use of SAML Identity Provider’

  2. The XML metadata file that you got from your SAML/SSO provider should be downloaded to the machine these actions are running from. Due to the use of special characters and copy/paste issues it is not recommended to paste in a value into the field. Instead use the upload function to upload the XML file in whole.

In the case of VMware's Horizon Product the metadata can be found here: https:///SAAS/API/1.0/GET/metadata/idp.xml

  1. Now close the organization tab or log out and back into the UI to get the ‘Groups’ and Import from SAML options in the UI. After that is completed you can now import users into vCloud Director.

Since SAML users and groups are not searchable users will have to be added one per line for each role. Another option for granting access to the organization is through the use of groups.

At this point the configuration of vCloud Director is complete.

Exporting XML Metadata from vCloud Director

We now need to export a file much like what we imported from the SAML provider, so that the SAML provider will trust and validate requests coming from vCloud Director. This file and related certificates are controlled on the Federation page where we imported the SAML XML file. At the bottom of the page click on the ‘Regenerate’ button to create a new certificate with a validity of 1 year.

It will prompt you with a warning about changing the certificate. If you have already setup a SAML provider relationship, changing this certificate will break that relationship until the SAML provider replaces their current information with the new certificate. Once the certificate is generated you can download it via this weblink: https://./cloud/org//saml/metadata/alias/vcd Save this file as an XML file, then provide this to your SAML provider for the next section.

Configuring Horizon for vCloud Director SSO

First you will need to login to the administrator interface for Horizon and create a new Application, then follow these configuration items for that application.

  1. Configure the Login Redirection URL to be the organization URL in vCloud Director. This is required for vCloud Director to work properly, since the authentication sequence must start with vCloud Director and not Horizon.

  2. Check the ‘Include the Destination in the response’

  3. Check the Sign the entire response.
  4. Check the ‘Sign the assertion’
  5. Remove the ‘Include Cert’ checkbox
  6. Select the configure via Meta-data XML option. Then upload the certificate (copy/paste) the certificate in to the text box given.
  7. Click on ‘Populate Attribute Mapping’
  8. Configure the Attribute Mapping to match your deployment of Hoizon.

Enabling SAML Providers in vCloud 5.1

As a system or organization administrator head to the 'Administration' page, and then the federation tab. Paste or upload the SAML configuration XML. Ensure the certificate at the bottom of the page is recent, if not click re-generate certificate. It will produce a certificate for 1 year.

After the certificate is re-newed enter this URL in to the web browser: cloud/org/[orgName]/saml/metadata/alias/vcd ; save the XML file on that page. This is the file you will need to provide the SAML or 3rd party authentication service with in order to authenticate you.

Now you have SSO or SAML or 3rd authentication, enabled on your organization. Once configured/enabled the Org login page will now use the SAML provider, meaning any local users will be locked out.

New Year... Brings Changes

With the New Year brings some new changes; I am transferring inside of VMware to a new group that is focusing on Cloud Consumption.

Cloud Consumption... WTH is that?

For the past, well as long as I have been in IT, I have been doing 'Ping, Power and Pipe'. With IaaS it is in my opinion it has been done, most any experienced IT person could stand up IaaS.

In my new role I will be focusing on methods to consume the cloud offerings. For example, how does one define a service (in the ITIL sense) from this 'Ping, Power, and Pipe'? Another example, what requirements does a consumer have? How can the cloud offering satisfy those requirements?

More importantly... "Define the services that consumers will utilize, while driving the architecture of the physical"

Happy New Years! 2013 will most definitely be a challenging year.

Symform Security Analysis

This morning on twitter I was asked if Symform "Secure Online Backup" was as secure as SpiderOak (my favorite online storage/backup solution). Here is my analysis from reading Symform's publically facing documents.

How it works:

According to Symform's website, your data is processed at the folder level. Meaning that all files in a folder are encrypted together. This does give the benefit of being able to de-duplicate the files inside that folder, it is not clear if this is at the block or file level though. If done at the block level this would be analogous to compression, if done at the file level I would image not much savings in space since it is uncommon for duplicate files in the same folder.

These files are encrypted with AES256 (good job there), but what generates this key? Since there would be a separate key per folder/container it would be near impossible for the end user to manage these keys. That means the Symform Cloud Control is generating and managing these keys for you the end user. (This is confirmed by their own documentation)

After your files are encrypted (in 64MB folder chunks), they are then broken into 1MB chunks. Parity fragments are then generated out of those 1MB chunks, 32 parity fragments to be exact. Those parity chunks are then sent out to the Symform Global Cloud Storage Network, and distributed to 96 devices.

Analysis:

That seems fine right? Your data is spread among 96 random devices on the internet, encrypted and secure.

Well that is the problem, who controls the decryption/encryption keys? Symform does. Who controls where your data is stored on this 'Global Cloud Storage Network? Symform does.

From a stand point of Trust No One, Symform fails the test. A hacker can still get your AES keys, and the location of those blobs on the internet. A government agency can still subpoena for your data, and have access to the decrypted data.

While it looks like a neat technology, SpiderOak still wins in my opinion, since I control the encryption keys and where my data resides (Amazon S3 storage).

Host Isolation Response Settings

Over the past few weeks I have had some people ask what I would recommend for host isolation addresses and responses, when using vSphere x.x with high availability (HA) enabled.

First off as with most everything in computers 'It depends'.

Items to consider with host isolation response is what type of storage is being utilized? Is it considered an outage if the vSphere hosts can talk to each other but the virtual machines (VMs) cannot? How stable is my management network? Is datastore heart-beating setup correctly? The list can of things to consider can be quite long.

Here is what I generally recommend…

NAS Based Storage Options (iSCSI or NFS)

First, set one of the host isolation response addresses to the virtual IP (VIP) or the IP of the storage array. This will tell you if you can at least talk to the array; chances are that is you cannot talk to the array your machines have already failed.

That brings me to the isolation response, since the VMs should already be dead, remember the NAS array and host are no longer able to communicate, I recommend 'Power Off'. If you were to leave the response to 'Leave Powered On' you can run into a chance of corruption on the guest disks. Think about it, the disk goes away for 60 seconds, then comes back. Do you think that VM is going to be able to recover, what are the odds of it corrupting that vital disk? What about 'Shutdown Guest', well that has a chance of the same exact thing happening.

Power Off in this setting is the safest setting, not only will it allow for that VM to be brought up elsewhere, you run a less chance of corruption or VM failure.

Fiber based storage (SAN)

With fiber the network isolation address does not matter as much, reason being the storage is not relying on the network for communication.

General Host Isolation Techniques

If it is considered a down time for the VM network to fail but not the management network, is there a way we can mitigate this?

Of course, put a vmkernel port on the virtual machine network, enable it for HA traffic, then place an isolation address of that networks gateway in to the HA configuration.

vCNS App - Spoofguard Deployment

Spoofguard records the IP address of each vNIC secured by vShield App. Spoofguard can be configured in two different modes, automatic and manual.

In Automatic mode the first IP that is presented for each vNIC is recorded and allowed to communicate without authorization. If the IP of the vNIC changes, the machine will no longer be able to communicate on the network until the new IP is approved.

In manual mode all IPs must be approved prior to being allowed to communicate; even existing virtual machines will be blocked until approved.

When enabling Spoofguard in an environment where there are existing virtual machines, it is recommended to enable Spoofguard in ‘Automatic’ mode. Once the vNICs are learned and reviewed it is then acceptable to switch to Spoofguard to ‘Manual’ mode, which will restrict all new virtual machines from communicating until approved. When deploying vShield App to a greenfield environment Spoofguard can be enabled with ‘Manual’ mode. Once Spoofguard has flagged a VNIC for approval, it must be approved prior to traffic being allowed to transmit. This is true even if Spoofguard is later disabled; all VNICs that are in ‘Require Approval’ will require approval before they can transmit.

When utilizing vShield App in a vCloud Director environment Spoofguard should not be enabled. The reason for this is that when vCloud Director brings a virtual machine online for the first time the machine will boot up and broadcast an IP (either DHCP or a pre-defined IP). When the machine is fully booted the guest customization script takes effect and changes the IP to the one defined in vCloud Director. Spoofguard will learn the first IP and not allow the VM to communicate on the network until the new IP is approved. It is possible to use vCenter Orchastrator to automate this approval process; that discussion and how to is out of the scope of this document.

Spoofguard currently supports only one IP per VNIC, so one cannot specify a secondary or failover IP.

iOS 6 - Ad Tracking

You can disable (well limit) the ad tracking that is done in iOS 6. This setting turns off the ability for advertisers to use what is called the 'Advertising Identifier'. This replaces the devices UUID which is a non-anonymous way of tracking usage. The advertising identifier is 'anonymous' (though further research needs to be done).

Here is how to disable the ability for advertisers to track you at all (once they move from the old deprecated UUID model).

  1. Fire up your iOS device
  2. Under Settings
  3. Find General
  4. Click on About
  5. Scroll to the bottom and select 'Advertising'
  6. Set the 'Limit Ad Tracking' to 'On'

You are done. Now go forth, untracked from iOS ads (the Apple ones anyway).

vCNS Edge 5.1 VIX_E_DISK_FULL ERROR Fix

VMware this morning released a KB article and a fix for the Edge devices filling up on disk space. While it did not kill the edge or cause an outage, it did stop you from being able to make changes to the Edge's configuration.

The official fix will be released on this site: https://my.vmware.com/web/vmware/info/slug/securityproducts/vmwarevcloudnetworkingandsecurity/51

The upgrade/patch will require the standard vCloud Network Security (vShield) upgrade process where the Edge device will need to be replaced with the new code. (READ: Possible outage of network for a few moments while the Edge is swapped)

How do you know if you have this problem?

In vCloud Director, attempting a reconfig fails with this error:

VIXEDISK_FULL

In vCloud Director, when looking at Edge Gateways, you receive this error:

Edge VM backing the edge gateway is unreachable

Just remember, when this happens it is not causing an outage or a network failure. The edge is basically running 'headless' at that time, and will not accept any ruleset changes.