top of page
logistics-transportation-container-cargo-ship-cargo-plane-with-working-crane-bridge-shipya

Transforming Data Ingestion for TransLogix Global: A Real-Time Logistics Solution using MS Fabric

Project Overview​

TransLogix Global, a logistics company, relied on daily manual uploads of shipment logs stored as JSON files. This process was time-consuming and prone to errors, causing delays in updating shipment statuses across business dashboards and operational systems.

Objective

To automate and optimize the data ingestion process, ensuring that only new or updated shipment logs are loaded each day into the Lakehouse, enabling real-time visibility into logistics operations.​

Solution Overview​

I designed and implemented an incremental data ingestion pipeline using Microsoft Fabric. The solution uses watermarking logic to detect and load only new or modified shipment files into a Delta Lake table in the Bronze layer of the Lakehouse.​

​

Incremental Pipeline 1.png

​​

What I Built
​

​​Watermarking Logic with Spark SQL

​

I created a dedicated watermark table using Spark SQL, initialized with a base timestamp. On every pipeline execution, the timestamp updates dynamically. This enables the pipeline to filter and process only the new files whose Log Timestamp is greater than the last recorded watermark and the pipeline trigger time.

​​Incremental Copy Activity

​

The pipeline uses a copy activity that references the watermark to apply dynamic filters at runtime. This ensures only newly added or updated shipment log files are ingested, avoiding duplication and reducing resource usage.​

​​​Lakehouse Architecture (Bronze Layer)


Source: Raw JSON shipping logs stored in the Lakehouse Files area.
Destination: A managed Delta Lake table named ShippingLogs in the Bronze layer (Tables section), storing raw but queryable data.
The use of Delta Lake’s transactional storage ensures data integrity and consistency with every ingestion run.
​

​Business Impact​

  • Automation of daily ingestion removed manual intervention entirely

  • Significantly reduced data latency, enabling near real-time shipment tracking

  • Improved decision-making for the operations team with fresh, consistent data

  • Designed for scale – The pipeline can easily accommodate growing data volumes

​

Key Takeaways

  • Implementing incremental ingestion with watermarking ensures efficiency and data freshness.

  • Delta Lake provides reliability through ACID transactions and easy rollback capabilities.

  • Microsoft Fabric offers a unified platform for data ingestion, storage, and transformation, simplifying pipeline orchestration.

​

bottom of page