IT god exposed as false idol by quirks of Java – until he laid his hands on the server

Look on my works and despair. After you've emptied the trash

Who, Me? Sometimes, IT professionals can appear as gods to users. Sometimes their mere presence can cause problems to miraculously disappear. In today's Java-based tale, a reader recalls the all-too-brief moment when he became a database deity. Welcome to Who, Me?

Our story comes from "Bob" (not his name) and is set amid the fallout from the dotcom bubble and the dying embers of the corporate Y2K dollar bonfire.

Bob, having tired of dealing with the memory-corrupting bombs left in C++ code by junior developers, was seeking a more productive environment. "Java," he said, "seemed like the go to language of choice."

"By this time, we had a few small to middle-sized projects under our belts and felt that we knew what we were doing."

That said, there was still the odd issue. A Java app had been developed to process large CSV files but it was usually quicker to simply kill the thing after processing than wait for it to kick off the next batch. Strange, but it wasn't critical. Nobody bothered to look into it any further.

Things were going well. So well in fact that the team was handed its first Java project: a corporate web application running on Tomcat with a SQL Server handling the data wrangling. More than a hundred users hammered the system, using it to record sales calls and the outcomes.

The administration team was trimmed from 40 to 20 people and the sales team increased, such was the efficiency of the application.

"Never once did we spill a tear as another admin person was delivered a pink slip," boasted Bob. "No, these were demonstrations of just how mighty we were. We were the gods of efficiency."

And so it went on. SQL queries were carefully crafted and tuned to within an inch of their electronic lives. Other managers hopped on the efficiency train, and Bob's app spread its tentacles far and wide within the organisation. Those not made redundant loved their Java overlords. "We were masters of our domain," recalled Bob, happily.

Until something went wrong. The wunderapp developed a mystery ailment and started becoming unresponsive for a few seconds. Normally this would just be an annoyance, but stir telephony into the equation and those short seconds can seem a lifetime.

Seasoned Java developers are doubtless mashing the comment button at this very moment. But for Bob's team, it was a mystery.

Logs were checked and deadlocks searched for. Tomcat, SQL Server, and Java documentation was inspected. Nothing leapt out except… something called Java Garbage Collection (GC).

The gang investigated further, becoming more and more convinced that the pausing problem was somehow connected to the GC. However, they also began to feel less godlike and more out of their depth. A call was duly placed to the local Sun Microsystems office for help.

"Well," remarked Bob, "it turns out that the local Sun office didn't actually support Java and in fact there was no way to obtain any support and zero interest in doing so.

"And that is perhaps why Sun is no more," he added, drily.

But how to solve the problem? Memory! Of course! More memory would sort things out!

And yes, it initially seemed like the problem had gone away. But, alas, it had not. Sure, the pause was delayed, but when it inevitably arrived,the stoppage was now well over a minute rather than taking a few seconds.

"At this point," said Bob, "we were no longer the corporate heroes. Instead, we were the bastards in the basement responsible for all of the woes of the organisation."

Things were bad, and there was a real chance the gang might find themselves staring down the barrel of the redundogun if the problem wasn't solved. Bob was hauled to the corporate office that housed the system and told home time would not come until the pauses were history.

He sat next to the server, awaiting the dread pause and amusing himself by fiddling with SQL queries and reading more articles on the Java GC when it happened again. Heads popped up above partitions like meerkats, telephones began ringing, and angry noises could be heard from the offices of management.

It was at this point that an event Bob called "The Miracle of the Server Room" occurred.

"OK, yes," he admitted, "I wasn't in the server room but it sounds much more iconic than 'The Miracle of the Open Planned Office Desk with a Server Sitting On It'."

At his wit's end, Bob had rested his hands on the server.

"And with the laying of my hands, a great miracle occurred and the people once again came to worship at my altar (ego), for a vision had come to me, of fast spinning disks and the heads thereof fervently thrashing all in my name to perform a garbage collection."

In other words, he'd felt the vibration of the drive within the box and realised the page file was being hammered.

"If you don't know how Java GC works," Box explained, "when memory is first allocated it is placed in a heap called 'Eden'. Most allocations are only short-lived and get collected whilst still in Eden. After a period of time allocations that haven't been collected are moved into a Tenured heap. Most Java GC runs ignore the Tenured heap."

There's a bit more to it these days, but in a nutshell the Tenured heap Bob's talking about doesn't often see a lot of action and so was a prime candidate for paging out to disk. Right up until Java got short of memory and paged the Tenured Heap back into memory for a Garbage Collection.

Reading back the page file was resulting in a pause. The addition of more memory had just increased the size of the file, thus making the problem even worse.

"Oh, and that little Java app that processed CSV files? If we had only stopped to find out what the problem was, we would have already known that GC and page files don't mix."

While modern SSDs mean the laying of hands on a server might not cause a visit from the inspiration angel (via vibrating disks) nowadays, "those were the days of the real IT gods," said Bob, wistfully.

"And, for a moment... I was one of them."

Ever soared high in the IT world, only to have your wings clipped when you realised you were not quite as clever as you thought? Or were you the one wielding the clippers? Confess all with an email to Who, Me? ®