Himanshumali, Manager – Solutions Architect Corp APAC, MongoDB
Progress in innovation is routinely blocked by overly complex data architectures.
Innovation is closely linked to an organization’s success and profitability. Today, innovation is often delivered through software. According to the 2022 MongoDB Report on Data and Innovation, developers and technology leaders are broadly aligned on the importance of innovation, with 81% agreeing that regularly building new, innovative applications and features is crucial to the long-term success of their organization.
Yet, the report also found that progress in innovation is routinely blocked by overly complex data architectures which slow teams down and consume considerable time and resources, making it difficult to create new and compelling products and features. The report also found 63% of respondents describing their organization’s data architecture as complex and 86% said complexity was a limiting factor when it came to innovation and one of the main reasons for complexity.
The report also highlighted some of the most common reasons for this complexity. How many of these do you experience every day:
1. Need to meet regulatory constraints
The number one cause of complex architecture was regulatory compliance. These requirements often lead to companies to incorporate numerous steps and extra technologies to ensure they can comply and overcome the fact their data is stored in siloed and disparate locations. This problem then multiples as soon as an app or service expands to a new country, where different laws may require another set of processes or choices that further complicate the underlying infrastructure.
2. Need to move quickly and create new products or features
While most developers agree that developing new, innovative applications and features is critical to innovation, the pressure to add new features more swiftly, also leads to data complexity. The reason is fairly obvious, for every new service or feature you want to create will, almost by definition, have new requirements. Say you want to add powerful search to your website, you’ll need to add a search database. Perhaps you want a mobile app that syncs with your order system? Yep, that’s potentially going to require a whole raft of new technologies to enable. When you add in the time pressure to ship new products, developers are incentivised to grab the most expedient option for building their new service. This may have short term advantages, but when it comes time to make changes, collate data or ensure a smooth customer experience, all those different technologies can make it an essentially impossible task.
3. Lack of coordination and planning
In the age of big data and modern software experiences, companies need to rethink how they approach processes, workflows and look at building a data-driven culture. Often, the lack of alignment and understanding becomes a big obstacle. So it’s no surprise that lack of coordination and planning was the third most commonly cited cause of architectural complexity.
4. Lack of internal technology standards
A large enterprise is always going to have a large and disparate estate of applications and services. However, when an organisation has a set of technology standards that all developers need to follow it makes it a vastly easier exercise to break down silos, connect different data sources and have developers build better experiences. Technology standards should evolve and should have some flexiblity, but they are also a huge asset. In many companies they simply don’t exist as no one got round to enforcing them and as they adopt new layers of technology, they create even more IT complexity. That ”spaghetti architecture” slows down the pace of development and requires time and budget for maintenance and management that should be spent on driving innovation.
5. Lack of knowledge or skills
Given the dynamic nature of big data technologies continuous upskilling and reskilling are key to ensuring overall success of the business. It also, of course plays a key role in avoiding complexity. Not only is the technology changing, but so are the people in the organisation, with people coming and going all the time. They can leave behind systems that no one else can operate which leads to new people creating ever more systems and introducing more technologies. More spaghetti.
With the right approaches and technologies, technology professionals can be better equipped to overcome barriers to innovation and to working with data. Through powerful data models and integrated developer data platforms that suit multiple use cases, teams can limit the number of technologies that need to be managed, and help simplify data architectures.
Addressing the challenge of data complexity can radically improve developer productivity, giving developers the most intuitive way to work with data and a repeatable, consistent experience across multiple application requirements — from the core database to analytics, mobile, and search.
