We’ve entered a new era in which more data management and delivery happens at the edge — the closest place, person, or device to where the data is actually generated. While there is still a significant role for central data repositories, companies simply can’t compete by relying exclusively on this pinwheel of activity model.
Instead, they are transitioning to using data fabrics that overlay all of their data systems, from edge to cloud, no matter where they are. This fabric relies on a federated metadata model that ultimately provides every data consumer with an integrated and holistic view of the data needed.
A recent white paper published by Hitachi Vantara (Edge-to-Cloud Data Fabric: A New Architecture for Data Management) laid out the components of this new federated data architecture and how they should work together to ensure access, transformation, and delivery of the right data to the right place at the right time.
With the right amount of data transformation, quality, and stewardship it is possible to create a data fabric that can manage and govern the data as it moves from the edge to the cloud. In such an architecture, data can move in a myriad of ways, in and out of centralized data centres, through company controlled or public clouds, as well as third-party services and applications, where it will be integrated and then pushed out to where it is needed. The key goal of the architecture is to provide coordinated management and governance and expedite the rate at which data can be acted on.
In this blog, I want to outline how this process works in real-life, using an example from the automotive industry. First, however, it’s worth giving a quick overview of how a federated architecture functions.
The Edge To Cloud Data Landscape: A federated architecture spans four centres of data that must be integrated effectively. They include:
- The Core: traditional data centres where data processing and running of applications occurs. Many edge devices stream data into the core
- Public Cloud: services and applications in one or multiple public cloud platforms, including those like Azure, Oracle, and AWS, where companies have the process capacity to compute, store, and run applications that handle any data
- Third-Party Applications and Services: companies also use third-party applications and services over networks that are accessed from remote locations. These are generally SaaS-based applications that enable businesses to use application functionality to conduct key services like credit checks, traffic reports or delivery tracking — but the data is still controlled by external companies
- Remote Physical Data Hubs: these are smaller data centres that aggregate data or are designed for special purpose processing. They can be hosted in cloud or data centres
Companies collect and analyze data in each of these locations. The federated architecture then operates like a distributed supply chain in which data in each of these areas can be connected and accessed by users. What is crucial to keep in mind is that creating the right architecture is predicated on assessing the best place for specific data to be processed and used. All of the locations provided needed data and computing capabilities, resulting in a more complex data environment in which more data than ever before is collected and put to use. To ensure this happens, a strong federated architecture must include:
- Diversification or ensuring data work is done wherever it fits most naturally into the larger architecture
- Distribution or dividing up workloads across multiple instances of a device or application to allow for processing to occur in unlikely places
- Orchestration or the ability for companies to bring distribution and diversification together to create optimal application functionality. Data must be orchestrated so it can flow from the edge to the cloud and back
As a real-life example, lets walk through how this works in the automotive industry to break ground in expanding the scope of automation and optimization.
Federated applications work by distributing work over a large landscape of data collection, data management, and computing technology and using a data fabric to provide a way to govern, manage, and make the data useful.
The power of this architecture is easiest to understand in the context of an industry.
The shift to an edge-to-cloud landscape has disrupted the manufacturing process of the automotive industry as much, if not more so, than the impact self-driving of cars.
While we’ve grown accustomed to computers in our cars controlling everything from the engine to our Spotify playlists, the change in how much technological innovation is incorporated into the creation of cars is only just beginning.
The reason for this is that auto manufacturers are just beginning to fully digitize their operations in order to foster a better, faster, and cheaper production process that also enables them to respond to consumer demand for greater mass customization.
One illustrative example of a federated architecture is the way that several systems that take advantage of edge data create large detailed model of the automotive business that can lead to new types of planning and optimization.
Consider the way that predictive maintenance systems can be used to prevent outages on the welding robots used on factory floors. The failure of a welding robots can cause outages and delays that damage productivity.
The robots are equipped with sensors that send data about their health. This data could include the oil levels, oil heat, electrical currents, pressure on the joints, and temperature of the welding robots. But there are numerous other environmental factors that may be leading indicators of problems.
So, the first example of a federated architecture is one that brings together all the data, processing what can be processed locally, possibly with the help of machine learning and AI, and then passing the rest along. This allows operational decision making to occur at the places the data originates.
This predictive maintenance system provides advance warning of when maintenance is needed which can then feed applications that are operating at the level of the entire factory. Needed maintenance from all parts of a line can be scheduled as part of a factory-wide production planning system that can include input from other applications in the federated architecture.
With a functioning factory model, it is then possible to create a company wide model that incorporates models of customer demand and workforce planning.
With a data fabric relaying on a and federated architecture, companies are able to catch problems earlier on and do preventive maintenance, avoiding downtime and saving money. They can also then implement processes that help to ensure optimal operations at much larger levels of scope from the production line, to the factory floor, to the company, and possible to the industry and supply chain as well.
In this way federated applications that run on an edge-to-cloud data fabric become both wider, incorporating a broader scope of data and computing, but also also become deeper, supporting models with vastly larger scope.
Paul Lewis is the global chief technology officer at Hitachi Vantara. He can be reached at firstname.lastname@example.org.