Operational Reporting Platform for Pilbara Mining Operation

The Problem

The technical challenge was straightforward to describe and deceptively hard to execute: take data from six independent source systems — three historians (OSIsoft PI), two SQL Server databases, and one legacy system accessible only via CSV export on a network share — and produce a single, reconciled daily operational report.

The source systems had no shared identifiers. Equipment referenced in the PI historians by tag name was referenced in the SQL databases by asset ID, and in the CSV exports by a human-readable name that had drifted from the official asset register over the years. Previous attempts at integration had bogged down trying to build a comprehensive asset mapping layer before writing a single line of reporting logic.

Data volumes were modest (roughly 50,000 data points per day across all sources), but latency requirements were firm: the report had to be available by 6:00am, with data no more than 15 minutes stale. The site had unreliable connectivity between the process control network and the corporate network, so the architecture needed to tolerate intermittent source availability.

The Approach

Architecture Decision: Pipeline, Not Warehouse

We deliberately avoided building a data warehouse. The client’s previous failed projects had both started with data warehouse designs, and both had stalled because the schema design phase became an unbounded exercise in modelling every possible future reporting need.

Instead, we built an event-driven pipeline that ingests source data, applies transformation and reconciliation logic, and materialises reporting views in PostgreSQL. The pipeline only models what the current reports need. When new reports are required (phases two and three), new pipeline stages and materialised views are added — the existing ones don’t change.

This was a deliberate trade-off: we sacrificed the flexibility of ad-hoc querying across a normalised warehouse in exchange for a system that could be built, tested, and deployed incrementally.

Source Integration

Each source system got its own ingestion adapter:

• PI historians: OPC-DA read via a thin C# bridge (the PI SDK is Windows/.NET only) that publishes to Azure Service Bus. We evaluated writing a direct OPC-DA client in Rust but the protocol complexity and the existing PI SDK made the C# bridge the pragmatic choice.
• SQL Server databases: Direct query via scheduled jobs in the Rust pipeline, using tiberius (the Rust TDS client). Polling interval: 5 minutes.
• CSV exports: File watcher on the network share that picks up new files, parses, validates, and publishes to Service Bus. The validation step catches ~4% of files that have formatting inconsistencies — these get flagged for manual review rather than silently ingested.

Reconciliation: The Hard Part

The asset identity mapping was the core technical challenge. Rather than building a comprehensive mapping table upfront, we took an incremental approach:

1. We built a fuzzy matching layer that proposes mappings between the different identifier systems based on string similarity, temporal correlation (do these two identifiers show correlated data patterns?), and spatial proximity (are they on the same part of the plant?).
2. Proposed mappings surface in the operator dashboard for human confirmation.
3. Confirmed mappings are persisted and applied automatically going forward.

This meant the system started with zero mappings and built its mapping table through use. After two weeks of parallel running, 94% of assets were mapped. The remaining 6% were genuinely ambiguous (duplicate equipment names on different parts of the plant) and required the site team to clarify.

Why Rust

We chose Rust for the pipeline for three reasons:

1. Reliability: The pipeline runs unattended. Rust’s type system and ownership model meant we caught entire categories of bugs at compile time — null references, data races in the concurrent ingestion stages, unhandled error cases on source system timeouts.
2. Performance: Not because 50,000 data points per day is a demanding volume, but because we wanted the entire pipeline to complete within the latency budget with margin. End-to-end processing (all sources to materialised views) runs in 11 minutes. The budget was 60.
3. Deployment simplicity: A single compiled binary, deployed to Azure App Service. No runtime dependencies, no garbage collector tuning, no dependency version conflicts in production.

Testing Strategy

• Unit tests for all transformation and reconciliation logic. The fuzzy matching layer has particularly thorough property-based tests using proptest, generating random asset names and verifying that the similarity scoring is stable and symmetric.
• Integration tests against Docker containers running PostgreSQL and a mock Service Bus, validating the full pipeline from ingestion through to materialised views.
• Parallel running in production for two weeks: the automated pipeline ran alongside the manual process, and a comparison report flagged any discrepancies above 0.1%.

Deployment

The pipeline runs on Azure App Service (Linux, B2 tier). PostgreSQL is on Azure Database for PostgreSQL Flexible Server. Service Bus handles the async messaging between the C# bridge and the Rust pipeline.

CI/CD is GitHub Actions: build, test, deploy on merge to main. Deployment is blue-green via App Service deployment slots. Rollback is a slot swap — under 30 seconds.

The Results

• End-to-end latency: 11 minutes from source data availability to materialised report. Budget was 60 minutes.
• Data reconciliation: 94% of assets auto-mapped within 2 weeks of parallel running. 100% mapped within 4 weeks of production use.
• Accuracy: 99.7% agreement with manually compiled figures during parallel running. The 0.3% discrepancy was a known rounding difference in the PI historian, not a pipeline bug.
• Uptime: Zero unplanned outages in the first 6 months. The pipeline has tolerated 3 source system outages (PI historian restarts) gracefully — it logs the gap, continues with available sources, and backfills when the source comes back.
• Deployment frequency: 12 production deployments in the first 6 months, zero rollbacks.

In hindsight, the one thing we’d reconsider is the C# bridge for PI historian access. It works, but it’s the one component that requires a Windows runtime environment and .NET dependency management. If we were starting today, we’d evaluate the newer PI Web API (REST-based) more seriously — it wasn’t stable enough at the time of our evaluation, but it’s matured since.