Integration landscapes can quickly feel overwhelming. When you are zoomed in on the technical components, it soon becomes impossible to wrap your head around all the different implementations. The cognitive load becomes too great for any human to handle efficiently.
To make things even more challenging, with the move to the cloud, integrations are nowadays distributed services. The term ”Integration” previously represented a single component deployed to an integration platform like BizTalk; we now have publishers and subscribers running as different serverless components and supporting resources like queues and storage. We might have data pipelines ingesting data to a database exposed through an API.
The key to scaling your governance model efficiently is having a level of abstraction to help us reason about the integration landscape. A good abstraction makes it easier to see the forest for all the trees and is still valid after the teams decide to refactor the implementation for technical reasons.
At Integrio, we have decided to use the Backstage System Model as our mental model for reasoning about the integration landscape. This allows us to get a clearer view of relationships and responsibilities within the landscape without focusing too much on implementation details.
Backstage uses five concepts to model their technology (further explained below):
- Domains are a high-level grouping of systems
- Systems encapsulate the implementation of APIs
- APIs are the boundaries between different components and systems
- Components are pieces of software
- Resources are physical or virtual infrastructure needed to operate a system
The parts that are most relevant to us are Systems, APIs, Components, and Resources. Each is explained below.
System
A system is an abstraction on top of technical components like serverless functions, databases, and queues needed to fulfill a business need. These abstractions make it easier to reason about our integration environment and allow us to let the technical solutions evolve without having to rewrite the whole map. The system remains the same if we refactor an integration from one Logic App to two Function Apps.
Components
A component represents a part of the business need implemented in a serverless component, a container, or any other compute instance. It also represents the system’s technical design; for example, the publisher and receiver of a message interchange should be implemented as two separate FaaS solutions rather than one.
Resources
Resources represent infrastructure components that support a component, like a database to store data or a queue to enable pub/sub.
APIs
APIs are the interface between components and systems. APIs can be private, used for internal integration needs between components and systems, or external for use by external parties.
System Archetypes
When we map our integration landscape to the system model, we see that only a few system types are genuinely distinct. We refer to these foundational types as system archetypes.
These system archetypes provide a foundational approach to solving recurring integration challenges. They abstract away specific technologies and focus on the solution’s essential elements. Archetypes are intended to remain stable over time, representing core principles and high-level approaches broadly applicable across different systems and contexts.
In contrast, our Golden Paths represent our opinionated approach to implementing an archetype. They are continuously updated to align with our tech radar.
There are three system archetypes: Pub/Sub, API service, and Aggregator.
Pub/Sub
The pub/sub archetype is for loosely coupled, asynchronous message exchanges between 2 or more parties.
This example system consists of 3 components and one resource. There is also an API not represented in the diagram, the API that the queue exposes with a well-defined schema.
Check out Revision Blog to learn more about Pub/Sub from a pattern perspective.
API Service
The API Service archetype is used to expose data to internal and external parties in a reliable and performant way. Data is exposed through a synchronous API, and notifications are published to indicate when data has been changed.
This example comprises 1 component, 2 APIs (the synchronous API and the notifications), and two resources.
Aggregator
An API Service is typically designed as a single entity or a set of tightly related entities. Still, we commonly need a combination of these entities to compose a single message that can be delivered to an external system. This is what the aggregator archetype is for. An aggregator typically receives a notification from an API Service that some data an external party needs has been updated. The aggregator calls API services to collect the data needed to create a higher-order event that can be imported into another system.
Summary
The Backstage System Model gives an abstraction level to help us reason around the integration landscape. By having all components belong to a system and each system is derived from any of the three system archetypes, we can see the forest for all the trees.