Party like it is 1234567890

“At 11:31:30pm UTC on Feb 13, 2009, Unix time will reach 1,234,567,890.
Where will you be at this momentous second?” – from Bell Labs

This will be Friday, February 13th at 1831 and 30 seconds EST. If you want to find out what time it will be in your local time, try this Perl script courtesy of Matias Palomec:

perl -e ‘print scalar localtime(1234567890),”\n”;’

Source: One of those magic times: On Friday the 13th!

IBM’s 10 Essential Tricks for Admins

IBM posted a great reference for the newbie and schooled Linux system administrator.  On their Developers’ Works site, they have a post called Lazy Linux: 10 Essential Tricks for Admins.

Learn these 10 tricks and you’ll be the most powerful Linux systems administrator in the universe…well, maybe not the universe, but you will need these tips to play in the big leagues. Learn about SSH tunnels, VNC, password recovery, console spying, and more. Examples accompany each trick, so you can duplicate them on your own systems.

These tricks are all invaluable to any Linux SA, and most will work on other Unix flavors.  My particular favorites are:

• Trick 3: Collaboration with screen – I hope to make a more in depth HOWTO.
• Trick 6: Remote VNC session through an SSH tunnel – a must for Oracle DBAs and remote support.
• Trick 8: Command-line scripting and utilities – a good CLI for loop comes in handy often.

StoreVault: Hidden Low-End Netapp

iSCSI and virtualization are the rage, but finding a solution that can work in a High Availibility model with VMware and be affordable for a small business is difficult.  There are a number of really good iSCSI solutions ranging from Dell’s Equallogic to Lefthand Networks.  The retail prices of these solutions are in the range of $30k-60k.  In developing a cost effective solution for a local medium business, I was turned on to StoreVault by a VAR.  Enterprise iSCSI features for about $10k.

StoreVault is a division of Netapp offering Small to Medium sized businesses (SMB) a enterprise storage solution based on Netapp’s Data OnTap infrastructure.  StoreVault offers two models of storage arrays, the S300 and S550.  The chassis are limited in their expandability offering a total of 12 drives and are limited to a single controller.  StoreVault does offer a cold stand-by chassis option for a faction of the price of the hot chassis.  Basically the cold stand-by chassis is a chassis, power supply and controller.  The drives and SD card are transferred to the stand-by in the event of a failure.

A side from those limitations, it offers many of the same features as the main Netapp product line including:

* RAID-DP
* WAFL (Write Anywhere File Layout) file system
* CIFS and NFS
* NDMP
* Snapshots
* FlexVol
* SnapManager for Exchange and SQL Server
* Replication

The medium small business actually purchased three units and a cold stand-by.  They currently have them deployed in a VMware Server configuration utilizing Snapshot and NDMP.  The goal is to eventually replicate to a central location.  The implementation has been running for about 9 months very successfully.  They have run into two problems, one was user error and the other a hardware failure with a drive.  In both cases, StoreVault support worked to resolve the problem in a timely manner and then followed up which was unexpected.

StoreVault does not make sense everywhere, but should be considered if you are in the SMB or department demographic.

Unix Crosses Over Productivity

I try to keep my approach to productivity and life as simple as possible.  I do not like a bunch of clutter in my personal or work life.  I am not sure where this philosophy stems from, I am sure childhood, so far it has served me well.

I am drawn to systems that are simple, non-obtrusive and get out of your way when they are no longer necessary.  I think this mentality is why I am also drawn to Unix.  Unix has many simple, single use utilities such as cat, more, ps, kill, etc, that serve a single purpose.  Yet, when strung together rival more complex programs.

Lifehacker ran an intriguing article entitled Applying Unix Philosophy to Personal Productivity.  The article applies the 17 rules of Unix philosophy listed in Eric Steven Raymond’s book free online book, The Art of Unix Programming to productivity.  If you enjoy Unix, personal productivity and GTD, then this article is right up your alley.

The article helped me start inspecting my Unix geekdom.  I thought I was fairly proficient in Unix having used it day-in and day-out for about 10 years.  It occurred to me that I had never heard of the 17 rules of Unix philosophy.  Here they are from Wikipedia’s Unix Philosophy entry:

Rule of Modularity: Write simple parts connected by clean interfaces.
Rule of Clarity: Clarity is better than cleverness.
Rule of Composition: Design programs to be connected to other programs.
Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
Rule of Simplicity: Design for simplicity; add complexity only where you must.
Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
Rule of Transparency: Design for visibility to make inspection and debugging easier.
Rule of Robustness: Robustness is the child of transparency and simplicity.
Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.
Rule of Least Surprise: In interface design, always do the least surprising thing.
Rule of Silence: When a program has nothing surprising to say, it should say nothing.
Rule of Repair: When you must fail, fail noisily and as soon as possible.
Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for “one true way”.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.

Other References:

• Basics of the Unix Philosophy

Benchmark Tools: Part II – Application Testing

In Part I – Disk I/O, we discussed two standalone tools to measure disk performance. In Part II, I hope to present some viable tools to measure the performance of your web applications and Oracle databases. My organization is continuously being asked to size and architect systems to run various applications. We use a couple of tools to perform load tests for SCT Banner and other applications we host.

The most common tools we use are:

• BadBoy
• jMeter
• SwingBench

BadBoy is …

a powerful tool designed to aid in testing and development of complex dynamic applications. Badboy makes web testing and development easier with dozens of features including a simple yet comprehensive capture/replay interface, powerful load testing support, detailed reports, graphs and much more!

Best of all, Badboy is Cheap or FREE depending on your use (Read the license agreement to learn more)

BadBoy is a browser recording tool that can output jMeter test files.

jMeter is …

a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

We use BadBoy to record and create scripts that are then used as input test files in jMeter. jMeter’s wiki has a link to a step-by-step tutorial entitled Test your Web Application’s Performance on how to use BadBoy and jMeter. With this method, we have worked with our member campuses to produce a significant registration load for SCT Banner.

Swingbench is …

a free load generator (and benchmarks) designed to stress test an Oracle database (9i,10g,11g).

Our DBAs use Swingbench to stress Oracle databases. Swingbench has proven a useful, cross-platform tool to stress standalone databases and RAC databases. It has a very pretty looking, real time interface.

Benchmark Tools Series:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Benchmark Tools: Part I – Disk I/O

Intro

I would like to share with you some of the tools we use to measure performance, stress our environment and perform load tests in our environment. Along with running the tests, we also need to measure and track performance. In this multi-part post, I will discuss some of the tools my department currently uses to accomplish these tasks.

Background

My department is an application hosting facility for a University system. We run a wide variety of applications with a vast majority running Oracle. We also have a large install base of MS SQL Server and Lotus Domino. We support almost every OS (Linux, AIX, Solaris, HP-UX, AIX, Windows and OpenVMS) and will support just about any application run by the University system. This means we need to use tools that are cross platform and somewhat application agnostic.

Part I – Disk I/O

My department is embarking on a new storage platform, the HP XP24000. We also acquired HP Storage Essential Tools to trend and track our environment. But Storage Essentials really does not stress test the environment.

I have been doing some research on how we can measure the relative performance between LUNs being migrated from our IBM ESS800 to the HP HP24000. We are changing the underlying storage, cache, and LUN layout. With the new subsystem, we are using multiple “LDEV” (HP’s name for LUNs) sizes and RAID levels. We now have a mix between 20, 100, and 300 GB LDEVs, with the RAID levels of 4+4 (RAID 10) and 7+1 (RAID 5). Our ESS800, on the other hand, has a uniform 35GB LUN size and 6+1 (RAID 5). Our strategy with the ESS800 was to assign multiple LUNs to a system and then use logical volume management to stripe the LUNs.

Using the LVM strategy with the ESS800, we have not encountered disk I/O bottlenecks. In moving to the XP, we are abandoning this strategy, unless needed. We are going to keep it as simple as possible, assigning larger LDEVs and growing only with concatenated volumes. We plan to compare the relative performance comparing the ESS800 to the XP24000. We are hoping that the XP24000 will blow away the ESS800. It has faster drives, 15k compared to 10k, and almost 10x the amount of cache and backend processors.

I have found two tools to measure disk I/O performance:

• IOzone Filesystem Benchmark
• Oracle I/O Numbers Calibration Tool (ORION)

IOzone Filesystem Benchmark

IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.

Iozone is useful for performing a broad filesystem analysis of a vendor’s computer platform.

IOzone can be downloaded from http://www.iozone.org/

Oracle I/O Numbers Calibration Tool (ORION)

ORION is the Oracle I/O Numbers Calibration Tool designed to simulate Oracle I/O workloads
- Without having to create and run an Oracle database
- Using the Oracle database’s I/O libraries
- Using small I/Os to simulate OLTP workloads
- Using large I/Os to simulate data warehouses

ORION is useful for understanding the performance capabilities of a storage system, either to uncover performance issues or to size a new database installation.

ORION can be downloaded from http://www.oracle.com/technology/software/tech/orion/index.html

These sure beat `time dd if=/dev/zero of=/data/file1 bs=1024 count=10000`

Benchmark Tools Series:

• Benchmark Tools: Part I – Disk I/O
• Benchmark Tools: Part II – Application Testing
• Benchmark Tools: Part III – Network Testing
• Benchmark Tools: The Holy Grail and the Single Source of the Truth

Bootable Images in every geeks bag

Tuxme.com had a post on 7 downloadable CD images that will give you super powers.  All of the images and tools have saved my butt at some point in my career.  A definite read (download) for every system admin!

Blades and Virtualization

Virtualization, server consolidation and blade servers are hot topic for datacenters everywhere.  Here are some of my daily RSS reads for these topics:

• Virtualization.info
• Blade Watch
  
• VMTN Blog
• Parallels Virtualization

Finding an Unlinked Open File (and other lsof uses)

Here is the scenario:

• the output of `df` shows a filesystem at 100% capacity
• adding up `du -sk *` does not match the usage in `df`

What is happening is someone launched a program or script that opens a file perhaps for logging purposes. Then, the user deletes the file from the filesystem. Since the file handle is open, the user is only deleting the inode associated (unlink) with the open handle. The process continues to write to the open file handle filling up your filesystem in an almost undetectable manner.

I have run into this a number of times with application developers. You can artificially recreate this scenario by starting syslogd and then delete /var/log/messages. Syslogd will actually contiune to write to the data portion that the inode was pointing to.

Question: So how do you detect this?

Answer: Can be found in A Quick Start for Lsof, Finding an Unlinked Open File section.

Finding an Unlinked Open File section:

a. Finding an Unlinked Open File
=================================

A pesky variant of a file that is filling a file system is an
unlinked file to which some process is still writing. When a
process opens a file and then unlinks it, the file’s resources
remain in use by the process, but the file’s directory entries
are removed. Hence, even when you know the directory where the
file once resided, you can’t detect it with ls.

This can be an administrative problem when the unlinked file is
large, and the process that holds it open continues to write to
it. Only when the process closes the file will its resources,
particularly disk space, be released.

Lsof can help you find unlinked files on local disks. It has an
option, +L, that will list the link counts of open files. That
helps because an unlinked file on a local disk has a zero link
count. Note: this is NOT true for NFS files, accessed from a
remote server.

You could use the option to list all files and look for a zero
link count in the NLINK column — e.g.,

$lsof +L
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
…
less 25366 abe txt VREG 6,0 40960 1 76319 /usr/…
…
> less 25366 abe 3r VREG 6,0 17360 0 98768 / (/dev/sd0a)

Better yet, you can specify an upper bound to the +L option, and
lsof will select only files that have a link count less than the
upper bound. For example:

$ lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
less 25366 abe 3r VREG 6,0 17360 0 98768 / (/dev/sd0a)

You can use lsof’s -a (AND) option to narrow the link count search
to a particular file system. For example, to look for zero link
counts on the /home file system, use:

$ lsof -a +L1 /home

CAUTION: lsof can’t always report link counts for all file types
– e.g., it may not report them for FIFOs, pipes, or sockets.
Remember also that link counts for NFS files on an NFS client
host don’t behave as do link counts for files on local disks.

Looking Glass is here

Sun Microsystems has released version 1.0 of its Looking Glass software, a Java-based 3D user interface.  Version 1.0 is available for Linux, Solaris x86 and Windows.  Sun has some demo movies of Looking Glass on their site.

Here are a couple from YouTube:

Looking Glass on Ubuntu

Looking Glass on Windows XP