This article is more than 1 year old

How one Ukrainian software maker planned for survival as invaders approached

Set priorities, expect confusion, keep emergency instructions simple – that's just for starters

At the start of the year, when it looked likely that Russia would invade Ukraine, Kyiv-based MacPaw began making a plan for operating during wartime.

CTO Vira Tkachenko described at the Jamf Nation User Conference how the 500-person Mac and Windows software maker prepared for the conflict. Risk assessment is something every organization has to deal with, but when there's an army massing on your borders, it concentrates the mind.

"It was quite very terrifying for us and we decided to plan," Tkachenko said, adding they initially relied on reports from Western intelligence agencies about the anticipated start of the invasion. The first step, she said, was to establish priorities, because you can't do everything.

"Our priority number one was about the security and stability of our services," she explained, "because for a customer, it doesn't matter whether you have, let's say some tornado or war or pandemic, they expect that the services they use, they just should work."

There was an even more pressing priority – which Tkachenko referred to as "priority zero" – the physical security of company team members. About 70 percent of employees are in Ukraine, she said, then added that she wouldn't cover personnel security in her presentation and would focus on the technical side of business continuity.

A MacPaw spokesperson told The Register that presently 10 MacPaw team members have been mobilized to support the Ukrainian war effort and none have been killed or injured to date.

Prepare the plan

The first step to prepare involved drawing up a list of possible risks.

"We were expecting that there will be no internet in all of Ukraine, so that our infrastructure and cellular towers will be damaged and we couldn't operate because of no internet connectivity," said Tkachenko. "The next risk we saw was about the physical occupation of our offices by invaders, so it won't be possible to enter our offices."

Then there was the risk of attacks on company services. "Modern war, it's not only about military stuff, it's also about cyber war," said Tkachenko. "And as for today, we have, like, the biggest cyber war happening in the world."

The MacPaw team identified other risks: the increased phishing attacks on employees and on company social media accounts; unauthorized access from lost or captured devices; the disruption of the hardware supply chain; and the effect of sanctions and the consequences of operating from a war zone, such as the reluctance of suppliers to maintain relationships.

Tkachenko said MacPaw's 2022 War Outbreak IT Business Continuity Plan "was created in Google Spreadsheet, one of my favorite instruments as a manager." It describes service assets like Okta, Github, and AWS, emergency procedures for safeguarding the system, backup procedures, responsible team members, and so on.

"We had two procedures," she explained. "The main one and backup one, if the main one won't work. And the instructions were very simple, with the least directions we needed to follow to do something. And it's a great idea to keep it very simple. Because in times of emergency, you're getting very emotional. You can't do anything really complex so it's better to keep it simple."

The business also created an emergency team – two people representing each product or service who could fix any issue that came up or knew who to ask. These people, said Tkachenko, were asked to move somewhere safe or at least to western Ukraine, where the threat of invasion looked less imminent.

Emergency lockdown

At that point, a code freeze was instituted. "Only people from the emergency team can have the right to make changes or approve them," Tkachenko explained. "And the job was to assess if the change is safe enough to be made and is it really necessary to do so right now."

Remote work was also on the list of preparations, but Tkachenko said that the COVID-19 pandemic had already forced MacPaw to adapt. Having the war arrive after the pandemic meant it was already prepared for remote work, she said.

MacPaw also adopted the Signal messaging app as a backup communication channel in case the company's Slack channel was no longer viable. And it planned to move its services completely to cloud infrastructure. Most of the biz's tools were already cloud services but the MacPaw still needed on-prem software and that had to change. The preparation for war basically helped MacPaw accelerate its transition to the cloud.

The company bought some laptops in advance, expecting disruptions that might prevent that, and also invested in two satellite internet systems, the Ground Control MCD-4800 BGAN Satellite Terminal and the Cobham Explorer 710 - BGAN Internet Terminal.

Tkachenko was not particularly enthusiastic about either system. The hardware was expensive and so was the data plan, she said. And the systems were difficult to buy. A system administrator spent about a week setting the satellite terminals up, but the speed was not great.

Tkachenko had nicer things to say about SpaceX Starlink, which activated its satellite internet service for Ukraine shortly after the invasion. In March, she had a Starlink receiver. "It was a huge deal," she said. "Not only for us but for our military especially."

Shots fired

When the invasion started on February 24, MacPaw was as prepared as it could be. "I heard a siren at five o'clock in the morning," Tkachenko recalled. "And it was a very shocking experience. Even if you were kind of expecting it, it's still a shock."

Following a call with the company's executive team, the emergency plan was activated. Not everything went as anticipated of course. Tkachenko described how a laptop sent to an employee two days prior to the invasion never made it. The device eventually was geolocated to an area near Kyiv and reported to the police but hasn't been recovered.

It also became difficult to understand who was working and from where because employees had to move around, often without much advance notice. So company developers put together an open source Slack system called Together App for keeping track of employees.

"It asks where you are and gets a geolocation, then it asks if you are safe and are able to work," said Tkachenko, "because for several weeks we were experiencing quite a bad emotional state of our employees and almost no one was able to work."

In summary, Tkachenko said, don't neglect planning, even though you can't understand the emergency until it happens. Talk to an organization that has had an experience similar to what you anticipate. Write your emergency procedures down and make them simply to follow, because people can't do complicated tasks under stress. And expect swift changes.

Tkachenko believes MacPaw handled the crisis fairly well and noted that the emergency team was dissolved by March 14 as the conflict became more routine.

"We managed to ensure our company's security and stability in a time of war," she said. "We released several big updates during that time and even released some new products. So it's possible to operate normally even in difficult times." ®

More about

TIP US OFF

Send us news


Other stories you might like