Visualising the cloud - Amazon EC2 in HyperGlance

The old adage “If You Don't Monitor It, You Can't Manage It” holds true just as much today as in the past. People are pushing compute load to the cloud, both public and private, without implementing the robust measuring/monitoring solutions they do with their current infrastructure.

We hear the incumbent solution providers say they support the cloud while telling themselves and anyone else who will listen that no one will truly use the public cloud so you don’t need to bother. I am seeing both public and private cloud projects being kicked off by everyone from banks to government. Private clouds will certainly come first but public cloud is everyone’s goal.

Security is usually touted as the main reason large companies are reluctant to use something like Amazon’s EC2 and that is true with the current mind-set around security. But rest assured very soon we will need to use a new mind-set, one which recognizes that nothing can be deemed as being safe or trusted. The military are wrestling with this new paradigm, where they are not sure if another country or state has already hacked into and currently owns their data. The days of thinking a section of a company’s infrastructure, much less the entire company’s infrastructure, is trusted are gone.  We need to embrace the ethos of not trusting any data source, not even your own!

Amazon’s EC2 cloud is of course the current market 800 pound gorilla and with good reason. They created the current version of cloud (remember old versions like grid computing?) and it does an exceptional job of pushing new features and keeping the system up and running (although no one is perfect as they have had down time lately). Open Stack, Cloud Stack, Eucalyptus and others are desperately trying to gain market share; some have been more successful than others.

To enable better management of the cloud, we have built a HyperGlance collector to pull in data from the Amazon EC2 API to visualise it.

HyperGlance can pull in any structured data and create a topology as long as we have relationship data or something we can postulate connections with. Amazon EC2 is no different.  It has a very nice, well-known API that specifies a Region and Availability zone per VM instance so we can just create a topology that maps to those attributes. Once we have that data, we can query the API for any metrics and attributes for the instances and then overlay them onto the model.

We can also connect the topology to a physical device like a firewall which gives you a hybrid cloud view.  I see people using Nagios or similar tools to monitor the state of the internals of the Instance as Amazon can’t (and shouldn't  see inside.  We can also pull in the Nagios data and overlay onto the Amazon topology.

Next up is OpenStack with Quantum.  Networking is taking its place in the core of the stack, as it should. We are working on a collector to pull in OpenStack data via the API, then we will add on SDN data to that mix. We are working towards a true end-to-end view of I.T., from the applications down to the hardware.

VMware visualization using HyperGlance 1.3

Being able to visualize IT environments is something I have been working on for quite a while now and to start things off I have created a video that went longer than I expected, 10 minutes or so! I show off our new dynamic filtering capability. It really is easy to use and makes day to day tasks so easy. Have a look and see what you think.


Hyperglance 1.3 is out!

We have released HG 1.3 bringing great new features including Dynamic Filters. This allows the user to create filters using the GUI. It's very intuitive and works really fluently. I love how the Physics engine sorts out everything and the movement really catches the eye.

On the nodes you can do the following actions:


  1. Exclude (From the physics and Render)
  2. Glow
  3. Colour
  4. Icon Size
  5. Partially Hide
  6. Repel
  7. Icon Add to set




There is a smaller subset available on the interfaces/endpoints but we will increase over time.


I am planning on adding some youtube Videos in the coming days to show it off!!

Don’t lose visibility when moving your application load to the cloud

Automation and management; that is what VMware is concentrating on at the moment. The reasons are clear, as you scale up and move more application load to the “cloud” you have issues with managing the scale of change and you lose visibility to what is happening with both the applications and especially the infrastructure that the applications are running over.

Whether you move your application load to a virtual machine, an Colo outsourcer, a VM container provider like Amazon EC2, an application container provider like Heroku or to a pure SaaS solution like you lose visibility and control . Of course you gain efficiently, scale and the ability to be agile.

Traditional monitoring solutions concentrated on infrastructure, not applications. More and more people are realising that the applications and their dependencies are when they should be worrying about. Agent based monitoring is the only way to get real data when you move your application to a container totally outside of your infrastructure, but what do you do with that data. It soon gets overwhelming trying to make sense of dynamically changing load and containers that get created automatically.

Of course I am going to say that visualisation is the key, I helped create a way to visualise applications and their dependencies. For me, you have to get a handle on what is dependant in order to make sure your users aren’t affected when the inevitable happens and something fails. Also finding outliers and looking for trends is a great way to predict what might cause issues in the future and what can be removed to save resources.

Pretty much all cloud providers provide an API in some shape or form. Software providers are slowly reasling that they no longer can wall off the data they collect because it is only a part of the whole. Integration, correlation and visualisation, all three enable insight.

Humans see patterns and recognise correlations in milliseconds, why not utilise that powerful ability?