Benchmark Tools: The Holy Grail and the Single Source of the Truth

We have come to the conclusion of my 4-part series on Benchmark Tools. If you have not read the previous 3-parts, have a look at:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Two years ago, I attended Oracle OpenWorld 2006. As a systems person, I felt a bit uneasy being surrounded by Oracle DBAs. In the days leading up to my flight, I heard voices in my head asking for more CPU, more RAM, and more STORAGE. I felt almost at home at the pre-conference Extreme Weekend on Oracle Enterprise Manager Grid Control. I was placed in front of a machine with multiple Virtual Machines. The VMs were running an Oracle Database, another an Oracle Application server, and the third Grid Control. I went through all of the exercises, restarting Listeners in < 8 seconds, monitoring Linux servers and deploying patches. I came away thinking whoa, this is cool, lets get it up and running.

I drank even more Kool-Aid. I went to a few presentations given by Dell IT’s I/T Strategist and Oracle Grid Control Architect of the Year, Logan McLeod. Logan’s presentations detailed how Dell consolidated their operations and how Grid Control was a knight in shinning armor. He basically said that the motivation behind streamlining Dell IT’s services was two fold:

1. Dell’s corporate object is efficiency.
2. He knew that Dell IT’s services were growing, but he was not going to be able to hire any more analysts.

Logan described in detail how Oracle EM Grid Control became the Single Source of the Truth for Dell IT. He was able to make his analysts more efficient, manage their environment and innovate all with Grid Control. We were quite impressed with Logan and Dell IT. They were already running their operation the way we wanted to. Logan was even gracious enough to talk with our systems and DBA teams after the conference. Thanks Logan!

The Single Source of the Truth

My organization currently uses a few different tools to monitor and manage our environment. These tools include:

• Cacti – for SNMP-capable devices, used for trending switch ports, firewalls and server performance.
• Nagios – for system alerting and some trending using custom scripts for pSeries virtualization.
• VirtualCenter – VMware VI3 management and trending.
• syslog-ng – central syslog and regular expression matching and alerting.
• Oracle Enterprise Manager Grid Control – monitor Oracle databases and application servers.
• Other tools – Dell Open Manage, IBM Director, storage tools, MS perfmon, MySpeed, etc.

As you can see, we have good visibility into our environment, but have many different places to look for the truth. We are not unique, we grew quickly both with staff and server sprawl. We currently manage 200+ hosts, 200+ databases and 200+ heterogeneous devices from network switches to Blade Chassis to virtualized environments. We do not have a single source of the truth, a Change Management Database (CMDB). With Oracle EM, we can perform all of the functions of our current tools, plus integrate CMDB functions all wrapped under a single pane of glass.

Our vision for managing our environment parallels Oracle’s strategy for managing the business stack. There were two slides at this year’s Oracle OpenWorld 2007 that showed up in just about every session. The first image is Oracle’s representation on how Enterprise Manager (Cross-Layer Management) controls the business stack. The second image illustrates how the Cross-Layer Management box interacts with the rest of the enterprise. My organization’s application stack looks very similar to image 1. We see and are currently reaping the benefits from Oracle’s approach to enterprise management.

Image 1

Image 2

Our goal is to have all physical and virtual nodes that we manage installed with an EM Grid Control agent. We plan to leverage the current functions of EM to replace our current tools. Oracle EM is capable of:

• Trending performance on physical and virtual nodes
• Trending VMware ESX hosts
• Tranding SNMP network devices (future enhancement, unknow timeline)
• Acting as a CMDB with a change cart (CMDB in 10g, change cart planned for 11g)
• Database and application management (Oracle products, MS SQL & IIS, TomCat, etc)
• Patch management for operating systems, applications and databases
• Syslog and general log reporting & alerting
• Distributed shell and job control
• Remote application performance through Beacons
• and other reporting and operational functions we do not have today!

Conclusion

The most important part of computing is the service. It does not matter what the underlying hardware, operating system or application is. At the end of the day, the most salient part is the end-user’s perception of the response time of a service. I tried to demonstrate the tools my organization uses to test a service prior to deployment from different levels; disk, system, network and application. And once the service is deployed, the tools we use to ensure it is running as the end-user expects. I hope you enjoyed this series on Benchmark Tools.