Dell adds crowdsourced regression tests to Boomi

Integration cloud gets ETL boost, Hadoop connector

Dell has announced a new service for its fully buzzword-compliant "cloud integration platform", Boomi, which taps the power of the crowd to help tap the power of the cloud.

When you rely on cloudy applications that are tweaked and run by a third party, you are on their release cycle and you are not in control of when things change and when they don't. But with an integration service like Boomi, customers create the integration scripts to link applications together through the service, and as Dell updates its integration service four times a year, those customers want to know that the Boomi hacks – called Atoms – that they use to link on-premises applications to infrastructure, platform, and software clouds and to social networks will continue to work without fail.

To that end, Dell is letting Boomi customers submit their Atoms to Dell ahead of each release cycle so they can be part of the regression testing pool to make sure that release doesn't break compatibility.

Rick Nucci, general manager of the Boomi unit, tells El Reg that this new process, called Boomi Assure, is designed to ensure the quality and relevance of regressions that are executed on each Boomi AtomSphere release.

Boomi Assure – which Dell calls "the industry's first crowd-sourced regression testing" – is a simple idea, and obviously useful once you think about it. As far as Nucci knows, neither Informatica or IBM's Cast Iron unit are offering such regression testing on their respective integration tools.

The announcement of the Assure Atom-testing program coincides with the Summer 12 Boomi AtomSphere release, which has the required and ubiquitous ease-of-use enhancements that all software seems to be always getting, as well as actual new code allowing for peppier extract, transform, and load (ETL) operations, and for linking apps to Hadoop data-munching clusters.

The ETL enhancements are a big deal, says Nucci. "What we used to think of as traditional SOA integration and traditional ETL are converging, and the lines are blurring." And so Boomi has to act like an ETL tool as well as a way to link disparate apps together.

With the Summer 12 release, AtomSphere now sports a join function, which will allow for multiple data sources (such as a relational database and a flat-file database) to be joined in a single Atom integration process and kept in-memory in the AtomSphere cloud for fast access.

The Summer 12 release also now has hooks into the bulk copy functions of relational databases that allows for information to be loaded directly into database tables rather than having to go in through ODBC or JDBC calls. The difference in speed with the bulk copy function can be minutes instead of hours using these other methods, which are really aimed at linking end users on an individual basis to data sources so they can update records as part of a transaction.

The AtomSphere integration cloud also now sports a connector into the Hadoop Distributed File System, the underpinning of the Hadoop big data muncher. You can connect applications running in-house or out on the cloud somewhere to in-house Hadoop clusters, or even cloudy data-munchers like Amazon's Elastic MapReduce service.

This particular connector works one-way, allowing you to pump data into Hadoop for it to be chewed upon. You have to use other Hadoop tools to extract the refined information from the Hadoop cluster, and it seems reasonable that Dell or third parties will create connectors linking the output of these tools (Pig, Hive, HBase, and so forth) back into applications through Boomi. The SDK for creating Atoms is free, so there is nothing to stop coders from doing this.

Hire a new service-oriented architect

Dell is also announcing a resource container called SOA Worker that runs in the AtomSphere cloud and provides a consistent and predictable set of resources to execute real-time data transformations on that cloud.

The capacity of the SOA Worker container is reserved and persistent, which is something that certain customers are asking for, according to Nucci, because they want to expose the results of their integrations and data transformations to partners and end users under certain circumstances.

The exact amount of resource in a SOA Worker container was not divulged, but it costs $250 per month, is an add-on feature to everything but SOA Framework Enterprise Edition, and is scalable elastically – meaning that if you need more oomph for a data transformation, you pay Dell for some extra containers and it will scale with automatic load-balancing across those multiple SOA Worker containers.

The Summer 12 release of the AtomSphere cloud has a bunch of security enhancements. For example, it now supports single sign-on using the Security Assertion Markup Language (SAML) variant of XML. SAML keeps passwords inside the corporate firewall as authentication is done for applications outside the firewall, which makes it hard to phish. Google,, Cisco Systems, VMware, and others have put their weight behind the SAML standard.

The access control in the Boomi service is also now more granular. Before, you either had access to an integration or you didn't – but now, for instance, you can give a programmer access to test data as they do their coding, but not give them access to the production data that will be used during the actual integration.

Dell Boomi integrations over time

Dell Boomi integrations over time (click to enlarge)

The use of the Boomi service has been growing steadily, as you can see here at the public Boomi dashboard. On Monday, Boomi processed 654,274 integrations either on the Atom cloud or on internally installed AtomSphere platforms, and has done 18.4 million integrations in the past 30 days.

These integrations can range from something as trivial as a single transaction to the transformation of a 20GB file from one data source to another. There are 2,854 unique Atoms deployed on the cloud or on premises, and there are over 2.6 million mappings and 18,864 functions that are being indexed by the Boomi Suggest service.

Boomi Suggest is an online wizard that helps companies do application and data mappings, and is based on the mappings created by existing users. At this point, about 80 per cent of the mappings that customers do, and map functions that customers create, are auto-generated from this pool of existing maps. The Atom is the runtime engine where these mappings run, in Boomi lingo. Boomi Suggest was put out in the Summer 10 release.

Dell is a bit cagey about what the Boomi service costs, but Nucci says that a typical SMB customer spends around $20,000 per year and enterprise shops spend around $100,000 per year. ®

Similar topics

Other stories you might like

Biting the hand that feeds IT © 1998–2021