CircleCI suffering 'degraded performance' for hours

DevOps platform tells devs it is ‘continuing to work on a longer-term fix’

Updated DevOps outfit CircleCI is suffering from performance issues that have been ongoing for most of the day, with developers left waiting for the platform to come back online.

The service keeps an eye on the likes of GitHub and Bitbucket and fires off a build when commits are made. It then tests builds and pings out notifications if things go wrong. The platform also features a managed cloud service.

However, today things went a bit awry. CircleCI described the issue as "degraded performance," which had begun with "database read/write delays." Meanwhile, over on Twitter, the firm promoted a Lego raffle as users became increasingly antsy.

As the day dragged on, the company upgraded the situation to an "Incident" as it searched for the root cause.

By the afternoon, it told users: "We are applying changes to ensure that builds are running as usual. Thank you for bearing with us."

At the time of publication, it had "deployed some mitigation to get more work through our system and continue to work on a longer-term fix," but devs were still reporting issues.

The problem with developers is that they are all too familiar with code changes that upset the apple cart, and (as is often the case) several took to Twitter to vent their frustration at the situation.

CircleCI's DevOps wares were originally limited to being a cloud-only SaaS when the company launched in 2011, with a self-hosted version not available until 2015's Server product arrived.

In 2019, the software testing and delivery company suffered a security incident involving "a third-party analytics vendor." It said at the time that the attacker was able to "improperly access some user data in our vendor account, including usernames and email addresses associated with GitHub and Bitbucket, along with user IP addresses and user agent strings."

No source code, build logs or other production data had been at risk, it said, nor was any authentication or password data lost. It said it would upgrade security together with the vendor.

The Register contacted the company to get its take on the current outage and will update should there be a response.

In the meantime, the (so far) near day-long outage is a reminder of the risks of cloud dependency. CircleCI is not the only service to experience a wobble (dare we mention GitHub and its woes?) but when one gets used to something working just so, its failure is felt all the more keenly.

Still, all the more time to enter that Nintendo Lego competition, eh? ®

Updated to add

CircleCI tells The Register that the issue has now been fixed. Let us know if not in the forums.

Similar topics

Broader topics

Narrower topics

Other stories you might like

  • Lenovo halves its ThinkPad workstation range
    Two becomes one as ThinkPad P16 stands alone and HX replaces mobile Xeon

    Lenovo has halved its range of portable workstations.

    The Chinese PC giant this week announced the ThinkPad P16. The loved-by-some ThinkPad P15 and P17 are to be retired, The Register has confirmed.

    The P16 machine runs Intel 12th Gen HX CPUs, but only up to the i7 models – so maxes out at 14 cores and 4.8GHz clock speed. The laptop is certified to run Red Hat Enterprise Linux, and can ship with that, Ubuntu, and Windows 11 or 10. The latter is pre-installed as a downgrade right under Windows 11.

    Continue reading
  • US won’t prosecute ‘good faith’ security researchers under CFAA
    Well, that clears things up? Maybe not.

    The US Justice Department has directed prosecutors not to charge "good-faith security researchers" with violating the Computer Fraud and Abuse Act (CFAA) if their reasons for hacking are ethical — things like bug hunting, responsible vulnerability disclosure, or above-board penetration testing.

    Good-faith, according to the policy [PDF], means using a computer "solely for purposes of good-faith testing, investigation, and/or correction of a security flaw or vulnerability."

    Additionally, this activity must be "carried out in a manner designed to avoid any harm to individuals or the public, and where the information derived from the activity is used primarily to promote the security or safety of the class of devices, machines, or online services to which the accessed computer belongs, or those who use such devices, machines, or online services."

    Continue reading
  • Intel plans immersion lab to chill its power-hungry chips
    AI chips are sucking down 600W+ and the solution could be to drown them.

    Intel this week unveiled a $700 million sustainability initiative to try innovative liquid and immersion cooling technologies to the datacenter.

    The project will see Intel construct a 200,000-square-foot "mega lab" approximately 20 miles west of Portland at its Hillsboro campus, where the chipmaker will qualify, test, and demo its expansive — and power hungry — datacenter portfolio using a variety of cooling tech.

    Alongside the lab, the x86 giant unveiled an open reference design for immersion cooling systems for its chips that is being developed by Intel Taiwan. The chip giant is hoping to bring other Taiwanese manufacturers into the fold and it'll then be rolled out globally.

    Continue reading

Biting the hand that feeds IT © 1998–2022