OpenWorld 2008

After a week of catch-up, I can finally share my experiences at Oracle OpenWorld in San Francisco.  It was an excellent trip and conference.  We arrived early and headed to Yosemite.  It was about 3.5 hours from SF, but was well worth the 7 hours of driving round-trip.  Nature’s beauty cannot be captured in words or pictures!  The things I will remember most about the park is rock “scrambling”, drinking water from a mountain waterfall, and El Capitan.

The conference was packed with content.  I always walk away with two to three new products to look at and track, a few new contacts and some great memories of the various conference events.  This year the products I will be tracking are Oracle BeeHive, Enterprise Manager Configuration Change Console, Enterprise Manager Real User Experience and EEnterprise Manager 10.2.0.5.

Oracle BeeHive is a newly rewritten collaboration platform based on open standards and an object model based on some of Oracle’s technology stack objects such as SOA and BEPL.  It looks extremely promising, but we thought that of Oracle Collaboration Suite.  Collaboration Suite seemed like a duct tape approach and many of the technologies available to develop BeeHive simply were not available.  We will definitely be installing and testing, hopefully that process fairs a bit better.  We are told that the install is completely self contained except the database and completes in about 70 minutes.

Enterprise Manager Configuration Change Console is a product acquired by Oracle.  It currently is a standalone product but will eventually be integrated in to EM.  CCC installs kernel modules or uses OS API’s to “catch” events in real time.  It audits everything from file open, file change, logins, etc all in real time.  It will not only log the event, but will keep revision histories.  Since it is working at the OS/kernel layer, it is much more efficient and accurate than standard diff approaches. Oracle’s OnDemand hosting uses this for all their production databases, applications and systems.

Enterprise Manager Real User Experience is a package the works with a mirror port to track and log all application traffic.  It records full TCP transactions from client to web server to database, reporting on the actual user performance.  It also acts as a troubleshooting tool to replay user sessions from the past.  It actually is able to show you the HTML of the session.

Finally, Enterprise Manager 10.2.0.5 brings many enhancements.  The features I am most excited about are the enhancements to it’s integration with Oracle VM.  EM will now be able to perform many of the functions of Virtual Center such as Live Migration, High Availability, Migration based on policies, schedules and service levels.  Very cool!

Larry Ellison keynote was OK.  He announced a couple of new Oracle hardware products developed in conjunction with HP.  I am not sure how that relationship occurred, the past couple of years Micheal Dell was on stage with Larry.  The new products are Oracle Exadata and Oracle Database Machine.  The Exadata is basically a 2u server with 2-socket quad-core processors and 12 disk drives.  The Exadata acts as storage for an Oracle Database with the advantage of running parallel queries returning only data and not data blocks. A very interesting concept.  Larry claims it is the faster data warehouse storage on the planet.  We probably do not have a need for it yet, but an interesting partnership and piece of engineering.

The session were outstanding. I learned a great deal about the products above, how to manage our environment with EM and some excellent Linux troubleshooting and general configuration tips that we will be standarizing on.  The conference renewed my excitement about EM and managing our environment using the Oracle ideal “manage many as one.”

Types of Virtualization: Para- vs. HW- vs. Full-

Recently, I gave a presentation on Oracle VM. Oracle VM is Oracle’s new virtualization product built on the Xen 3.1 hypervisor. I created a slide that listed the types of virtualization OVM supports and the guest OS’s supported. It occurred to me that my audience would probably want to understand the types of virtualization Xen supports and the caveats with Windows. There are many buzz words that surround virtualization: paravirtualization, VT, VMM, hardware assisted, full virtualization. To be honest, I was not 100% confident in my understanding of the types of virtualization to present to my peers.

A group of my colleagues and I sat down for about 1 hour trying to pipoint the differences both in technology and terms each vendor uses. We reviewed white papers, FAQs, diagrams and explanations by experts. The three of us came to a general consensus and here are our results.

Full Virtualization

• Hypervisor contains the Virtual Machine Monitor (VMM)
• Guest OS unchanged, Host can be any kind of hardware
• Guest uses generic device drivers

Hardware Virtualization (a.k.a. Hardware Assisted)

• VMM moved to hardware
• Requires VT hardware
• Guest OS unchanged, uses generic device drivers

Paravirtualization

• Guest OS requires paravirtualized drivers
• Thin layer for hypervisor, guest has access to Host hardware
• Host any kind of hardware

Our conclusions were based on the following resources:

• Redhat Virtualization Home
• Dell: An Overview of Xen Virtualization
• VMware: Understanding Full Virtualization, Paravirtualization, and Hardware Assist

The interesting conclusion we arrived at is  Xen-based virtualization does not support full virtualization.  Looking back through the history of Xen, we saw that Xen became prevalent when VT hardware was already in the mainstream.  My opinion is that because VMM functions were already available in hardware, the Xen developers opted to not build full virtualization functionality into the hypervisor.  In contrast, VMware and others were around prior to VT enabled hardware and needed to build a hypervisor capable of full virtualization.

If anyone has any further insight, I welcome your comments.

Benchmark Tools: The Holy Grail and the Single Source of the Truth

We have come to the conclusion of my 4-part series on Benchmark Tools. If you have not read the previous 3-parts, have a look at:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Two years ago, I attended Oracle OpenWorld 2006. As a systems person, I felt a bit uneasy being surrounded by Oracle DBAs. In the days leading up to my flight, I heard voices in my head asking for more CPU, more RAM, and more STORAGE. I felt almost at home at the pre-conference Extreme Weekend on Oracle Enterprise Manager Grid Control. I was placed in front of a machine with multiple Virtual Machines. The VMs were running an Oracle Database, another an Oracle Application server, and the third Grid Control. I went through all of the exercises, restarting Listeners in < 8 seconds, monitoring Linux servers and deploying patches. I came away thinking whoa, this is cool, lets get it up and running.

I drank even more Kool-Aid. I went to a few presentations given by Dell IT’s I/T Strategist and Oracle Grid Control Architect of the Year, Logan McLeod. Logan’s presentations detailed how Dell consolidated their operations and how Grid Control was a knight in shinning armor. He basically said that the motivation behind streamlining Dell IT’s services was two fold:

1. Dell’s corporate object is efficiency.
2. He knew that Dell IT’s services were growing, but he was not going to be able to hire any more analysts.

Logan described in detail how Oracle EM Grid Control became the Single Source of the Truth for Dell IT. He was able to make his analysts more efficient, manage their environment and innovate all with Grid Control. We were quite impressed with Logan and Dell IT. They were already running their operation the way we wanted to. Logan was even gracious enough to talk with our systems and DBA teams after the conference. Thanks Logan!

The Single Source of the Truth

My organization currently uses a few different tools to monitor and manage our environment. These tools include:

• Cacti – for SNMP-capable devices, used for trending switch ports, firewalls and server performance.
• Nagios – for system alerting and some trending using custom scripts for pSeries virtualization.
• VirtualCenter – VMware VI3 management and trending.
• syslog-ng – central syslog and regular expression matching and alerting.
• Oracle Enterprise Manager Grid Control – monitor Oracle databases and application servers.
• Other tools – Dell Open Manage, IBM Director, storage tools, MS perfmon, MySpeed, etc.

As you can see, we have good visibility into our environment, but have many different places to look for the truth. We are not unique, we grew quickly both with staff and server sprawl. We currently manage 200+ hosts, 200+ databases and 200+ heterogeneous devices from network switches to Blade Chassis to virtualized environments. We do not have a single source of the truth, a Change Management Database (CMDB). With Oracle EM, we can perform all of the functions of our current tools, plus integrate CMDB functions all wrapped under a single pane of glass.

Our vision for managing our environment parallels Oracle’s strategy for managing the business stack. There were two slides at this year’s Oracle OpenWorld 2007 that showed up in just about every session. The first image is Oracle’s representation on how Enterprise Manager (Cross-Layer Management) controls the business stack. The second image illustrates how the Cross-Layer Management box interacts with the rest of the enterprise. My organization’s application stack looks very similar to image 1. We see and are currently reaping the benefits from Oracle’s approach to enterprise management.

Image 1

Image 2

Our goal is to have all physical and virtual nodes that we manage installed with an EM Grid Control agent. We plan to leverage the current functions of EM to replace our current tools. Oracle EM is capable of:

• Trending performance on physical and virtual nodes
• Trending VMware ESX hosts
• Tranding SNMP network devices (future enhancement, unknow timeline)
• Acting as a CMDB with a change cart (CMDB in 10g, change cart planned for 11g)
• Database and application management (Oracle products, MS SQL & IIS, TomCat, etc)
• Patch management for operating systems, applications and databases
• Syslog and general log reporting & alerting
• Distributed shell and job control
• Remote application performance through Beacons
• and other reporting and operational functions we do not have today!

Conclusion

The most important part of computing is the service. It does not matter what the underlying hardware, operating system or application is. At the end of the day, the most salient part is the end-user’s perception of the response time of a service. I tried to demonstrate the tools my organization uses to test a service prior to deployment from different levels; disk, system, network and application. And once the service is deployed, the tools we use to ensure it is running as the end-user expects. I hope you enjoyed this series on Benchmark Tools.

Benchmark Tools: Part III – Network Testing

In Part I – Disk I/O and Part II – Application Testing, we discussed tools to help you benchmark and test your applications within the confines of a system or lab. Once your application has been tested and hits the real world, consistent level of performance almost always falls on the network. In our department, if a member campus is having problems, we perform some quick application tests and then dive into the network; WAN and LAN. If I had to guess, 70% of our support calls are due to local network problems or some in-transit problem over the WAN.

The most difficult parts of troubleshooting network performance are the lack of control over the full communication and the subjective nature of performance. Often times a network administrator has control of only half of the communication and reproducing what an end-user is seeing is next to impossible without a site visit. We have deployed a few different pieces of software to assist us with network performance trending and real-time analysis.

The first tool we discovered a number of years ago is called Qcheck. Qcheck is a powerful network troubleshooting when deployed with cross-platform Performance Endpoints. Qcheck can test, in rea-time:

• Response time
• Throughput
• Streaming
• Traceroutes

To deploy Qcheck, you install one or more of the Performance Endpoints in locations you want to test end-to-end network performance. Then you use Qcheck console to control the Endpoints, running tests between yourself and an endpoint or between two endpoints. Qcheck is an excellent tool and returns a bunch of valuable information. Its one caveat is that is does not run from a command line and cannot be scheduled or trended over time.

The lack of trending made us look for other alternatives. We wanted to be able to see network peaks as they related to support calls or general performance complaints. One of our sister organizations found a tool called MySpeed Server.

MySpeed Server allows you to host your own Internet speed tests such as Ookla Net Metrics found on some of these well known Internet speed test sites:

• http://www.speakeasy.net/speedtest/
• http://www.speedtest.net/

MySpeed provided us with a much needed feature, the ability to deploy an agent to remote sites that would run a speed test during a predefined interval. We even requested a feature to be able to deploy an agent that can dial back to 2 or more MySpeed servers, which the developers graciously wrote for us.

MySpeed was inexpensive to implement and has paid many dividends in troubleshooting and proving choke points for our applications.

Here are some other tools that can be of assistance in trending your network or quickly tabulating how fast something should take:

• Cacti
• http://www.speedguide.net/conversion.php

The final part of the Benchmark Tools series will be a summary of the tools discussed and a look forward on our future direction for performance trending and monitoring. The final part will be entitled “The Holy Grail and the Single Source of the Truth”. Stay tuned!

Benchmark Tools Series:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Benchmark Tools: Part II – Application Testing

In Part I – Disk I/O, we discussed two standalone tools to measure disk performance. In Part II, I hope to present some viable tools to measure the performance of your web applications and Oracle databases. My organization is continuously being asked to size and architect systems to run various applications. We use a couple of tools to perform load tests for SCT Banner and other applications we host.

The most common tools we use are:

• BadBoy
• jMeter
• SwingBench

BadBoy is …

a powerful tool designed to aid in testing and development of complex dynamic applications. Badboy makes web testing and development easier with dozens of features including a simple yet comprehensive capture/replay interface, powerful load testing support, detailed reports, graphs and much more!

Best of all, Badboy is Cheap or FREE depending on your use (Read the license agreement to learn more)

BadBoy is a browser recording tool that can output jMeter test files.

jMeter is …

a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

We use BadBoy to record and create scripts that are then used as input test files in jMeter. jMeter’s wiki has a link to a step-by-step tutorial entitled Test your Web Application’s Performance on how to use BadBoy and jMeter. With this method, we have worked with our member campuses to produce a significant registration load for SCT Banner.

Swingbench is …

a free load generator (and benchmarks) designed to stress test an Oracle database (9i,10g,11g).

Our DBAs use Swingbench to stress Oracle databases. Swingbench has proven a useful, cross-platform tool to stress standalone databases and RAC databases. It has a very pretty looking, real time interface.

Benchmark Tools Series:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Benchmark Tools: Part I – Disk I/O

Intro

I would like to share with you some of the tools we use to measure performance, stress our environment and perform load tests in our environment. Along with running the tests, we also need to measure and track performance. In this multi-part post, I will discuss some of the tools my department currently uses to accomplish these tasks.

Background

My department is an application hosting facility for a University system. We run a wide variety of applications with a vast majority running Oracle. We also have a large install base of MS SQL Server and Lotus Domino. We support almost every OS (Linux, AIX, Solaris, HP-UX, AIX, Windows and OpenVMS) and will support just about any application run by the University system. This means we need to use tools that are cross platform and somewhat application agnostic.

Part I – Disk I/O

My department is embarking on a new storage platform, the HP XP24000. We also acquired HP Storage Essential Tools to trend and track our environment. But Storage Essentials really does not stress test the environment.

I have been doing some research on how we can measure the relative performance between LUNs being migrated from our IBM ESS800 to the HP HP24000. We are changing the underlying storage, cache, and LUN layout. With the new subsystem, we are using multiple “LDEV” (HP’s name for LUNs) sizes and RAID levels. We now have a mix between 20, 100, and 300 GB LDEVs, with the RAID levels of 4+4 (RAID 10) and 7+1 (RAID 5). Our ESS800, on the other hand, has a uniform 35GB LUN size and 6+1 (RAID 5). Our strategy with the ESS800 was to assign multiple LUNs to a system and then use logical volume management to stripe the LUNs.

Using the LVM strategy with the ESS800, we have not encountered disk I/O bottlenecks. In moving to the XP, we are abandoning this strategy, unless needed. We are going to keep it as simple as possible, assigning larger LDEVs and growing only with concatenated volumes. We plan to compare the relative performance comparing the ESS800 to the XP24000. We are hoping that the XP24000 will blow away the ESS800. It has faster drives, 15k compared to 10k, and almost 10x the amount of cache and backend processors.

I have found two tools to measure disk I/O performance:

• IOzone Filesystem Benchmark
• Oracle I/O Numbers Calibration Tool (ORION)

IOzone Filesystem Benchmark

IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.

Iozone is useful for performing a broad filesystem analysis of a vendor’s computer platform.

IOzone can be downloaded from http://www.iozone.org/

Oracle I/O Numbers Calibration Tool (ORION)

ORION is the Oracle I/O Numbers Calibration Tool designed to simulate Oracle I/O workloads
- Without having to create and run an Oracle database
- Using the Oracle database’s I/O libraries
- Using small I/Os to simulate OLTP workloads
- Using large I/Os to simulate data warehouses

ORION is useful for understanding the performance capabilities of a storage system, either to uncover performance issues or to size a new database installation.

ORION can be downloaded from http://www.oracle.com/technology/software/tech/orion/index.html

These sure beat `time dd if=/dev/zero of=/data/file1 bs=1024 count=10000`

Benchmark Tools Series:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Oracle VM – Xen Hypervisor

Oracle did what we expected and hoped for, they released their virtualization product, Oracle VM, today at OpenWorld. Oracle VM (aka Oracle Virtual Server and OVS) is Oracle’s implementation of the Xen Hypervisor and more importantly management tools for your virtualized infrastructure.

The Oracle VM product will be freely available on OTN on Wednesday. Oracle will be selling support for Oracle VM at about the same rate as Unbreakable Linux support.

I am sure the Linux purest are grumbling heatedly over this announcement. But for those who have to support real systems, it will be a huge step in the right direction. Oracle is also releasing a Oracle VM Manager, which manages the Xen environment in a web-based application similar to Virtual Center for VMware. The next release of Oracle Enterprise Manager, 11g, will have a full featured management pack for Oracle VM. Both management products are very promising and have been the missing piece of Xen going mainstream.

I talked to one of the members of the product team in depth. He gave me a demo of the Oracle VM Manager and discussed the history behind Oracle VM. I guess they wanted to release the product months ago, but had to re-write pieces of the Xen Hypervisor to work in an enterprise fashion. We also discussed some of the future steps Oracle VM will be taking, and without disclosing too much information, they are going down the right path IMHO.

I would suggest browsing Oracle FAQ on Oracle VM at http://www.oracle.com/technologies/virtualization/docs/ovm-faq.pdf

For more information see, http://www.oracle.com/technologies/virtualization/index.html