Think GitHub and Git but for data – and you've got FlockerHub and fli

ClusterHQ debuts information time machine for better production testing

Updated Flocker is a mouthful. It's an open-source container data volume orchestrator, which means it helps migrate data when containers shift hosts. It makes data volumes portable within clusters.

Two years into its life, it's spawned a hosted service called FlockerHub. Its creator, ClusterHQ, describes it and its command line companion fli as the equivalent of GitHub and Git for data.

"FlockerHub and fli, we think, can help DevOps teams test better," said Michael Ferranti, ClusterHQ's VP of marketing, in a phone interview with The Register.

A developer survey conducted by the company suggests that code quality can be improved, a finding that could more or less be assumed. The firm polled 386 people, from companies large and small, 41 per cent of whom described themselves as DevOps team members, 37 percent as developers, and the remainder cited association with operations, QA, and security.

Among the respondents, 43 per cent said they spend between 10 and 25 per cent of their time debugging application errors discovered in production, a chore that cuts into time that might otherwise be used developing new features.

The top challenges these IT professionals cited were:

• Inability to recreate production environments in testing (33 per cent).
• Testing difficulties arising from the interdependence of external systems (27 per cent).
• Testing against unrealistic data prior to production (26 per cent).

By providing version control for Docker data volumes, ClusterHQ contends it can help DevOps teams manage and distribute data volumes to any host in a compliance-friendly fashion.

FlockerHub works with Docker Compose, allowing Dockerized applications to be created with actual data, to ensure realistic testing scenarios.

With commands issued through fli, DevOps personnel can take snapshots of Linux databases and data volumes and push them to FlockerHub, a process similar to commits in Git. Also like Git, fli sends incremental changes rather than complete databases.

"Every time you take a snapshot, you're only pushing changes," said Ferranti. "That means transfers are faster and consume less bandwidth. And if you ever need to go back in time to before some change was made, any of these snapshots is consistent and can be mounted to a container, thus giving you a sort of time machine for your data."

It's a concept that Apple already has trademarked for its data backup software. But it's nonetheless apt for developers: Fetching real data from the past can help identify bugs discovered in production.

FlockerHub is scheduled to be available in beta on November 8. And fli should be available for download then too. ®

Updated to add on December 22

ClusterHQ has shut down with immediate effect. Its source code is still open on GitHub.