Developers - Make life easier by grabbing more of the data life cycle

How MongoDB provides capabilities which help you do more with less

Sponsored Feature Welcome to the new development team. It's a lot like the old development team, but with two main differences: managers are asking devs to work in more joined-up ways. They finally understand that they must unlock the data across disparate systems in their organization so that they can use it to whip up new digital services for customers and increase internal efficiencies.

That's easier said than done, argues Sahir Azam, chief product officer at MongoDB. Developers are crucial in creating those new services, but it's difficult for already-stretched teams to suddenly become data architects.

"The scarcest resource in modern business is developer talent," says Azam. "Every organization is going through some sort of digital transformation, and the demand for developers to help with that has taken off."

This transformation places increasing demands on developers during a period when their skills are hard to find. So with IDC predicting a shortage of 4m developers by 2025 (up from 1.4m in 2021), it's important to make coders as productive as possible.

One of the biggest hurdles to productivity is data fragmentation. "A lot of these modern applications don't use just a single database," Azam explains. Instead, they rely on several technologies from multiple online transaction processing databases of different types through to streaming, analytics and search systems. Historically, those different technologies have been isolated across different platforms that require different development skills to build applications . "A lot of cost goes into operating and managing and duplicating all those environments" Azam adds.

How about a unified data platform?

Over the last few years, MongoDB has moved to solve that problem by bringing in not just more of the things that developers have grown to know and love about the database, but also capabilities that they need as their remit expands and stretches - essentially creating a developer data platform. At its MongoDB World event last June and over the course of the year, the company has since doubled down with a slew of additional new features.

MongoDB hopes that developers will find this platform familiar and comfortable – after all MongoDB is at the heart of it. But they should also be able to expand the use cases that it caters to and the applications it supports. "This has the capability to replace a lot of those disparate data tools" Azam says.

The data platform should also help companies to consolidate vendors, saving in software licenses and support costs, MongoDB believes. It will consolidate organizations' data footprints for example, make their skill sets more adaptable, and generally help them to do more with less.

Better search and support for time series data

Three years ago, MongoDB launched Atlas Search, an embedded full-text search solution in the Atlas multi-cloud database service. Previously, many organizations would have defaulted to a separate search engine to address their search needs, and then figured out the plumbing between the systems themselves. But the developer data platform integrates the database, search engine, and sync mechanism so that developers can build relevance-based search directly into applications.

Atlas Search is based on Apache Lucene, a popular library underpinning several of the most widely used search engines. Because it's integrated into the developer data platform, developers can leverage a single query language for both database and search operations. Atlas Search provides a full suite of features to help developers deliver search experiences to their users, including autocomplete, fuzzy matching, relevance scoring, geospatial queries, and faceted search, among others.

One customer is real estate company Keller Williams, which has been honing its use of Atlas Search to power its customer-facing search experience. It implemented auto-completion of user searches for ease of use, and then went one step further. Location-based search is important to users looking for properties, so the company built geolocation features into its Atlas Search-based implementation that change the auto-completion results based on where customers are searching. Keller Williams has also included a form of fuzzy searching capability that returns results based on incorrect spelling, along with synonyms for search terms.

MongoDB Time Series is another platform addition that helps developers avoid data silos. Time series data is common across industries, but can be challenging to work with due to its enormous storage footprint and the difficulty of unlocking real-time insights. Time Series empowers developers to handle this directly within MongoDB with native support for the entire data lifecycle, including storage optimization and performant data analysis. One customer, Norwegian startup and industrial IoT-as-a-service provider DigiTread Connect, leverages the simplicity and performance of Time Series to deliver solutions for their customers faster.

Application-driven analytics and queryable encryption

In its quest to build a unified data platform for both analytic and transactional workloads, MongoDB has also included a wide range of capabilities to help development teams build richer application experiences that rely on automatic, low-latency processing of live data. This includes rich aggregations and indexing strategies, as well as dedicated analytics nodes for workload isolation. The company has also added capabilities that help teams better manage their entire data lifecycle. Atlas Data Lake now allows developers to cost-effectively store large amounts of historical data without the need for complex extraction, transformation, and loading tools.

Developers can set up automated pipelines that extract data from Atlas clusters to provide consistent copies of their cluster data for analytical queries. When used in conjunction with Atlas Data Federation, developers can create a virtual database that queries across various sources, including data lake datasets, multiple Atlas clusters, HTTPS endpoints, and Amazon S3 storage buckets.

Azam also singles out another technology - queryable encryption - a feature which allows database users to search their data without decrypting it. The system is based on work with experts from Brown University who formed a company called Aroki that MongoDB acquired. It uses a cryptographic primitive approach known as structured encryption to find the appropriate information while leaving it encrypted, enabling companies to query sensitive data for everything from insurance claims to healthcare without having to see and be liable for that data.

This is different from the homomorphic encryption that some companies have pursued in the past. That approach focuses on conducting calculations with encrypted data. Structured encryption concentrates on finding it, but the Aroki approach can handle more complex queries than other databases' attempts to query encrypted data on the client side, Azam says.

"Queryable encryption allows us to make that data use much more powerful," he says. "You can start to move beyond point queries towards range, prefix, and suffix-based queries across multiple values, which has previously been impossible."

Queryable Encryption is still not generally available at the time of writing, but the reasons for that are sound; MongoDB has open-sourced parts of the technology so that third parties can validate it. Initial feedback from academics has been positive but the company wants to be thorough, so that customers can be confident in it when it goes live later in 2023.

You're speaking my language

The developer data platform also incorporates capabilities covering a range of different areas, including the general availability of Atlas Serverless, a version of the cloud-based MongoDB Atlas database that will expand and contract based on workload all without interaction from a developer or ops personnel. That's now part of a broader technology stack encompassing everything from security tools and advanced search through to new performance features.

One of these performance features is columnar indexing. This allows developers to create their own indexes across columns of data rather than rows. The advantage here is speed. Unlike row-based indexes, columnar ones need not retrieve each core document to extract and index its data, which speeds up the aggregations necessary for analytical queries. The company claims up to a 15-fold performance increase compared to conventional row-based indexes in some cases.

While MongoDB is a NoSQL database, that doesn't mean that it can't play nicely with SQL-based tools. After all, many companies are likely to straddle both environments. While their developers are likely to prefer MongoDB's document data platform to build applications, their business analysts may choose to use relational business intelligence tools like Tableau or Power BI for data analysis, for example.

Last year, MongoDB declared its Atlas SQL Interface available for public preview, serving to create an endpoint to easily connect SQL-based analytics tools to Atlas using custom MongoDB Connectors for select programs (eg Tableau and Power BI) or drivers (JDBC/ODBC) that provide connection to a broad range of other tools.

Built with MongoSQL, a SQL-92 compatible dialect optimized for the document data model, analysts can use custom SQL commands and native BI tool features to control their data visualization directly from their analytics dashboards; the Atlas SQL Interface, Connectors, and Drivers eliminate the need for data replication and ETL, preserving the richness of the document model while facilitating faster time to insights on live application data.

This support for SQL access tools eases the transition and learning curve for developers whose companies Azam hopes will eventually transfer their relational database assets to MongoDB. And to help these teams with mapping and moving relational data to documents, MongoDB launched Relational Migrator.

"It's easy to rush into using MongoDB quickly," he says, though warning that you can trip up unless you think about the right way to model the relationships between data elements to take advantage of the document format. "This tool makes mapping rows, tables and their foreign key relationships to documents a very visual experience that's easier to understand. And when teams are ready to start migrating data and changing application code, the tool helps there too."

Today's developers don't just code algorithms; they manipulate complex data from multiple sources that feed their programs for optimal results. If the software is the engine, then the data is the fuel that makes it go. MongoDB seems set on combining the two, honing its expanded tech stack along with educational resources so that organizations can continue their digital transformation journey.

Sponsored by MongoDB.