Oh no, you're thinking, yet another cookie pop-up. Well, sorry, it's the law. We measure how many people read us, and ensure you see relevant ads, by storing cookies on your device. If you're cool with that, hit “Accept all Cookies”. For more info and to customize your settings, hit “Customize Settings”.

Review and manage your consent

Here's an overview of our use of cookies, similar technologies and how to manage them. You can also change your choices at any time, by hitting the “Your Consent Options” link on the site's footer.

Manage Cookie Preferences
  • These cookies are strictly necessary so that you can navigate the site as normal and use all features. Without these cookies we cannot provide you with the service that you expect.

  • These cookies are used to make advertising messages more relevant to you. They perform functions like preventing the same ad from continuously reappearing, ensuring that ads are properly displayed for advertisers, and in some cases selecting advertisements that are based on your interests.

  • These cookies collect information in aggregate form to help us understand how our websites are being used. They allow us to count visits and traffic sources so that we can measure and improve the performance of our sites. If people say no to these cookies, we do not know how many people have visited and we cannot monitor performance.

See also our Cookie policy and Privacy policy.

This article is more than 1 year old

Chef reviews internal update plans after 'degradation incident'

Automation vendor manually fixes Hosted Chef problem

Chef fans faced going hungry on Friday afternoon as the automation vendor’s hosted service degraded itself by having a little lie-down in the back pantry.

Chef issued a mea culpa explaining the “Hosted Chef Service Degradation Incident”, which saw users presented with 404 errors at the end of “successful” Chef Client runs, or empty responses using Knife runs as a client on their workstations.

“This was a result of 2 of our 16 frontend nodes being left in a incorrect state following a routine deployment,” the statement continued. We’ll overlook this and other grammar fails in Chef’s statement, seeing as the firm clearly had other things on its mind at the time.

The snafu was down to Chef operations performing a overnight deployment to upgrade Chef Server to 12.4 - one of a number of updates last week addressing security vulnerabilities. The deploy tool failed to configure the reporting service on two of its hosts leaving them in a “functional, but degraded state”.

A condition many of our readers will be familiar with, particularly on a Friday. The nodes still passed health checks, meaning no red flags were raised.

Once the problem was noticed by Chef, “service engineers manually reconfigured the two incorrectly running nodes which immediately resolved the increased error rates.”

We’ll leave you to discuss whether there’s a smidgeon of irony in that last statement.

As a result, said Chef, “additional host-level monitoring is being put in place to catch this type of issue more rapidly in the future. Also, ELB health checks are being updated to more throughly [sic] test all components before a node is placed into service behind and load balancer.

A footnote added that a previous version of the posting said the upgrade had “accidentally stated the upgrade was to Chef Server 12.6. The upgrade was actually to 12.4, which is the latest release.” ®

Similar topics

TIP US OFF

Send us news


Other stories you might like