IT god exposed as false idol by quirks of Java – until he laid his hands on the server

Look on my works and despair. After you've emptied the trash

Who, Me? Sometimes, IT professionals can appear as gods to users. Sometimes their mere presence can cause problems to miraculously disappear. In today's Java-based tale, a reader recalls the all-too-brief moment when he became a database deity. Welcome to Who, Me?

Our story comes from "Bob" (not his name) and is set amid the fallout from the dotcom bubble and the dying embers of the corporate Y2K dollar bonfire.

Bob, having tired of dealing with the memory-corrupting bombs left in C++ code by junior developers, was seeking a more productive environment. "Java," he said, "seemed like the go to language of choice."

"By this time, we had a few small to middle-sized projects under our belts and felt that we knew what we were doing."

That said, there was still the odd issue. A Java app had been developed to process large CSV files but it was usually quicker to simply kill the thing after processing than wait for it to kick off the next batch. Strange, but it wasn't critical. Nobody bothered to look into it any further.

Things were going well. So well in fact that the team was handed its first Java project: a corporate web application running on Tomcat with a SQL Server handling the data wrangling. More than a hundred users hammered the system, using it to record sales calls and the outcomes.

The administration team was trimmed from 40 to 20 people and the sales team increased, such was the efficiency of the application.

"Never once did we spill a tear as another admin person was delivered a pink slip," boasted Bob. "No, these were demonstrations of just how mighty we were. We were the gods of efficiency."

And so it went on. SQL queries were carefully crafted and tuned to within an inch of their electronic lives. Other managers hopped on the efficiency train, and Bob's app spread its tentacles far and wide within the organisation. Those not made redundant loved their Java overlords. "We were masters of our domain," recalled Bob, happily.

Until something went wrong. The wunderapp developed a mystery ailment and started becoming unresponsive for a few seconds. Normally this would just be an annoyance, but stir telephony into the equation and those short seconds can seem a lifetime.

Seasoned Java developers are doubtless mashing the comment button at this very moment. But for Bob's team, it was a mystery.

Logs were checked and deadlocks searched for. Tomcat, SQL Server, and Java documentation was inspected. Nothing leapt out except… something called Java Garbage Collection (GC).

The gang investigated further, becoming more and more convinced that the pausing problem was somehow connected to the GC. However, they also began to feel less godlike and more out of their depth. A call was duly placed to the local Sun Microsystems office for help.

"Well," remarked Bob, "it turns out that the local Sun office didn't actually support Java and in fact there was no way to obtain any support and zero interest in doing so.

"And that is perhaps why Sun is no more," he added, drily.

But how to solve the problem? Memory! Of course! More memory would sort things out!

And yes, it initially seemed like the problem had gone away. But, alas, it had not. Sure, the pause was delayed, but when it inevitably arrived,the stoppage was now well over a minute rather than taking a few seconds.

"At this point," said Bob, "we were no longer the corporate heroes. Instead, we were the bastards in the basement responsible for all of the woes of the organisation."

Things were bad, and there was a real chance the gang might find themselves staring down the barrel of the redundogun if the problem wasn't solved. Bob was hauled to the corporate office that housed the system and told home time would not come until the pauses were history.

He sat next to the server, awaiting the dread pause and amusing himself by fiddling with SQL queries and reading more articles on the Java GC when it happened again. Heads popped up above partitions like meerkats, telephones began ringing, and angry noises could be heard from the offices of management.

It was at this point that an event Bob called "The Miracle of the Server Room" occurred.

"OK, yes," he admitted, "I wasn't in the server room but it sounds much more iconic than 'The Miracle of the Open Planned Office Desk with a Server Sitting On It'."

At his wit's end, Bob had rested his hands on the server.

"And with the laying of my hands, a great miracle occurred and the people once again came to worship at my altar (ego), for a vision had come to me, of fast spinning disks and the heads thereof fervently thrashing all in my name to perform a garbage collection."

In other words, he'd felt the vibration of the drive within the box and realised the page file was being hammered.

"If you don't know how Java GC works," Box explained, "when memory is first allocated it is placed in a heap called 'Eden'. Most allocations are only short-lived and get collected whilst still in Eden. After a period of time allocations that haven't been collected are moved into a Tenured heap. Most Java GC runs ignore the Tenured heap."

There's a bit more to it these days, but in a nutshell the Tenured heap Bob's talking about doesn't often see a lot of action and so was a prime candidate for paging out to disk. Right up until Java got short of memory and paged the Tenured Heap back into memory for a Garbage Collection.

Reading back the page file was resulting in a pause. The addition of more memory had just increased the size of the file, thus making the problem even worse.

"Oh, and that little Java app that processed CSV files? If we had only stopped to find out what the problem was, we would have already known that GC and page files don't mix."

While modern SSDs mean the laying of hands on a server might not cause a visit from the inspiration angel (via vibrating disks) nowadays, "those were the days of the real IT gods," said Bob, wistfully.

"And, for a moment... I was one of them."

Ever soared high in the IT world, only to have your wings clipped when you realised you were not quite as clever as you thought? Or were you the one wielding the clippers? Confess all with an email to Who, Me? ®

Similar topics

Broader topics

Other stories you might like

  • Did hoodwink Americans with IRS facial-recognition tech, senators ask
    Biz tells us: Won't someone please think of the ... fraud we've stopped

    Democrat senators want the FTC to investigate "evidence of deceptive statements" made by regarding the facial-recognition technology it controversially built for Uncle Sam. made headlines this year when the IRS said US taxpayers would have to enroll in the startup's facial-recognition system to access their tax records in the future. After a public backlash, the IRS reconsidered its plans, and said taxpayers could choose non-biometric methods to verify their identity with the agency online.

    Just before the IRS controversy, said it uses one-to-one face comparisons. "Our one-to-one face match is comparable to taking a selfie to unlock a smartphone. does not use one-to-many facial recognition, which is more complex and problematic. Further, privacy is core to our mission and we do not sell the personal information of our users," it said in January.

    Continue reading
  • Meet Wizard Spider, the multimillion-dollar gang behind Conti, Ryuk malware
    Russia-linked crime-as-a-service crew is rich, professional – and investing in R&D

    Analysis Wizard Spider, the Russia-linked crew behind high-profile malware Conti, Ryuk and Trickbot, has grown over the past five years into a multimillion-dollar organization that has built a corporate-like operating model, a year-long study has found.

    In a technical report this week, the folks at Prodaft, which has been tracking the cybercrime gang since 2021, outlined its own findings on Wizard Spider, supplemented by info that leaked about the Conti operation in February after the crooks publicly sided with Russia during the illegal invasion of Ukraine.

    What Prodaft found was a gang sitting on assets worth hundreds of millions of dollars funneled from multiple sophisticated malware variants. Wizard Spider, we're told, runs as a business with a complex network of subgroups and teams that target specific types of software, and has associations with other well-known miscreants, including those behind REvil and Qbot (also known as Qakbot or Pinkslipbot).

    Continue reading
  • Supreme Court urged to halt 'unconstitutional' Texas content-no-moderation law
    Everyone's entitled to a viewpoint but what's your viewpoint on what exactly is and isn't a viewpoint?

    A coalition of advocacy groups on Tuesday asked the US Supreme Court to block Texas' social media law HB 20 after the US Fifth Circuit Court of Appeals last week lifted a preliminary injunction that had kept it from taking effect.

    The Lone Star State law, which forbids large social media platforms from moderating content that's "lawful-but-awful," as advocacy group the Center for Democracy and Technology puts it, was approved last September by Governor Greg Abbott (R). It was immediately challenged in court and the judge hearing the case imposed a preliminary injunction, preventing the legislation from being enforced, on the basis that the trade groups opposing it – NetChoice and CCIA – were likely to prevail.

    But that injunction was lifted on appeal. That case continues to be litigated, but thanks to the Fifth Circuit, HB 20 can be enforced even as its constitutionality remains in dispute, hence the coalition's application [PDF] this month to the Supreme Court.

    Continue reading

Biting the hand that feeds IT © 1998–2022