Enterprise Data Hub Architecture for a Large-Scale Transportation Service Provider

• Project: Enterprise Data Fabric and integration hub with real-time analytics and migration support
• Duration: Ongoing
• Technologies: Microsoft Fabric (Lakehouse, Spark Notebooks, Data Pipelines, OneLake, Power BI semantic models), Azure Logic Apps, Microsoft Entra ID, fabric-cicd, .NET microservices

The client is a large logistics and transportation company processing between 3,000 and 5,000 orders daily across a network of more than 50 branches. The business relies on a large number of operational systems, including TMS platforms, telematics, payroll, and legacy IBM i (AS/400) solutions.

Challenges

A key constraint of the project was the nature of the logistics business itself. Operating in a 24/7 environment, the client could not afford even minimal downtime. Any disruption in data flows or system availability would have had a direct impact on operations, making a seamless, zero-interruption transition a critical requirement that their existing setup could not guarantee.

At the same time, the client lacked a unified source of truth. They had accumulated years of historical data, fragmented across multiple sources, including APIs, legacy databases, and SFTP exchanges. This fragmentation significantly undermined the reliability of analytics, leading to inconsistencies in reporting and making the decision-making process increasingly complex and risky.

Another major challenge was the absence of master data synchronization. Core business entities such as drivers, equipment, and assets were distributed across eight different systems without proper alignment. This resulted in frequent duplication, inconsistencies, and errors in critical processes such as payroll, compliance, and fleet management.

In addition, the client had no reliable mechanism to validate the correctness of the migration process. As data was being transferred between legacy and new systems, there was no way to verify its integrity or ensure consistency in real time. This lack of visibility introduced significant operational and technical risks.

The instinctive response to a situation like this was to defer the data layer until the migration is complete and clean up the data after system stabilization. At first glance, it felt like the safer path, but in practice, it meant operating blind during the most critical window of the transformation. Errors in how the new systems process data go undetected until they surface in payroll, billing, or compliance reporting, at which point the damage is already done, and recovery is expensive.

To mitigate these challenges and ensure a controlled digital transformation, Velvetech recommended building the data foundation in parallel with the migration, since it was the safest way to turn the transition from a leap of faith into a continuously validated process. The client agreed, and that decision defined everything that followed.

The Process

Data Fabric Architecture Design and Layered Isolation

We began by designing a Data Fabric architecture on Microsoft Fabric that could operate independently of ongoing system changes and serve as a stable analytical backbone. Instead of waiting for the migration to complete, we introduced a Medallion Architecture with Bronze, Silver, and Gold layers, built on Fabric Lakehouse, Spark Notebooks, and Data Pipelines, to decouple ingestion, transformation, and business logic.

The Bronze layer was designed as a resilient ingestion layer capable of handling high-frequency data streams from all sources. It incorporated incremental loading patterns and monitoring mechanisms to detect upstream schema changes, ensuring that data ingestion remained stable despite continuous updates from vendors and internal systems.

The Silver layer was introduced as a critical normalization and historization layer, effectively becoming the system of record. Here, data from legacy and new systems were standardized, reconciled, and aligned into a unified schema. This layer also incorporated flexible data contracts, allowing schema changes to be managed through configuration rather than requiring constant code modifications.

The Gold layer was built as a business-facing layer responsible for aggregations, KPIs, and analytical data models. By isolating it from upstream volatility, we ensured that reporting remained stable even as underlying systems evolved.

Given the dynamic nature of source systems, we prioritized the enterprise data hub architecture’s resilience in the transportation system. The separation between Bronze and Silver layers acted as a buffer against upstream changes, ensuring that API or schema updates required minimal adjustments confined to the ingestion layer.

At the same time, the implementation of flexible data contracts enabled rapid adaptation to evolving data structures without disrupting downstream processes. This reduced maintenance overhead and increased the overall agility of the system.

Integration Hub and Master Data Synchronization

All client data were stored in a fragmented way, which is why we suggested implementing a centralized Integration Hub within the Data Fabric. This layer unified all incoming and outgoing data flows, replacing a complex network of point-to-point integrations with a controlled and scalable transportation data architecture.

A key focus was master data consistency. Critical entities such as drivers, equipment, and assets existed across eight systems without synchronization. We introduced a unified identity and master data management layer with real-time bidirectional synchronization. This approach treated master data as an operational stream rather than static records, eliminating duplication and ensuring consistency across all platforms.

Identity Governance via Microsoft Entra ID

We established a single identity authority for the entire enterprise and wired it directly into the architecture from day one. Role provisioning across all 16 business departments became fully automatic: the right access is granted at onboarding and revoked at offboarding, with no manual steps. Identity state stays synchronized in real time across all connected operational platforms. Administrators get a single governed interface to manage role assignments, review the audit trail, and make changes without touching the underlying pipelines.

Analytics Enablement

To resolve one of the biggest clients’ pains — the absence of transparency and visibility — we needed to transform the reporting capabilities. Legacy manual reporting processes were replaced with automated Power BI dashboards built on curated Gold-layer datasets with report-level SSO and RBAC enforced through Entra groups. This reduced reporting lag from 3 days to approximately 15 minutes and enabled near real-time analytics across all the departments of the transportation organization.

Data Fabric as a Live Validation Engine

One of the most deliberate architectural decisions we made was building the data layer during the migration rather than deferring it until TMS stabilization. By constructing the Data Fabric during the migration rather than after it, we were able to use the warehouse itself as a live validation tool, which wouldn’t have been possible in a sequential approach.

With both legacy IBM i data and new TMS data flowing through the same pipeline simultaneously, we configured the Gold layer to cross-check outputs from the new system against the established baseline from the old one, throughout the entire migration window. Data errors introduced by the transition were caught at the data layer before they reached payroll, billing, or compliance reporting.

Operational Maturity Assurance through CI/CD Pipelines and AI Automation

To ensure platform stability and reliability, we fine-tuned a full CI/CD pipeline with gated artifact promotion from pre-production to production, so that every schema update, new integration, or model change is tested before it reaches live data. A dedicated SAP-side monitoring component, et-sap-fabric-watch, tracked the health of legacy system feeds throughout the migration.

With the data foundation stabilized, we also layered AI-driven automation tooling on top of the unified operational data, having reduced recurring manual toil for project management and operations teams. The clean, continuously updated Gold-layer data provided the reliable foundation that AI automation requires, making the tooling both more accurate and easier to maintain as the underlying systems continue to evolve.

The Result

As a result of the project, the client established a stable and unified data foundation that supported daily operations throughout the migration and ensured their stability. Data flows became predictable and resilient despite continuous changes in source systems, significantly reducing operational risks.

The introduction of centralized data management improved consistency across key business entities, which increased data reliability. At the same time, reporting evolved into a faster and more automated process, enabling quicker access to insights. The solution also brought greater transparency to the migration process, allowing the client to maintain confidence in data accuracy while transitioning to the new system.