The grid’s new gold: Harnessing DSO data to unlock flexibility

Europe’s distribution grids sit at the center of the energy transition. EVs, heat pumps, induction cooking, and distributed PV are reshaping low-voltage demand into volatile, bidirectional flows. Traditional forecasting and reinforcement cycles were designed for steadier, more predictable networks; they now struggle to anticipate where stress will emerge and how quickly it will become an issue.

Regulation is pushing in the same direction. Across Europe, distribution system operators (DSOs) are tasked to prepare granular development plans aligned with national transmission system operator (TSO) scenarios, quantify electrification drivers and flexible resources, and demonstrate that investment choices are efficient, targeted, and future-proof. This mandate effectively turns DSOs into local system managers. They must connect more load and distributed generation while using flexibility as a practical lever to phase CAPEX, manage temporary constraints, and avoid reinforcing every bottleneck by default.

THE DATA PARADOX

Rich in data, poor in actionable insight

DSOs already hold the ingredients for this shift: smart meter readings, asset registers, GIS, SCADA, maintenance history, point-of-delivery (POD) records, and power-quality measures. The issue is not the volume of data, but its readiness for planning decisions:

• Topological and asset data — lines, substations, components, and network configuration
• Customer and connection data — POD characteristics, contracts, and connection status
• Metering and power-quality data — energy flows, voltage, and stability signals
• Operational data — outages, maintenance, and continuity records

Most datasets were created for billing, reporting, or operational control, not predictive grid planning. They often remain in a “bronze” state: fragmented across systems, incomplete, poorly aligned with topology, and not consistently available where smart meter penetration is limited. The first mile is therefore decisive: raw, inconsistent signals must become trusted inputs for investment and flexibility decisions.

Moving to a gold standard requires more than a new platform. DSOs need data engineering discipline: cleansing, validating, enriching, and versioning information at the micro level and transforming versioned information into reusable layers that feed forecasting, network planning, and flexibility assessment.

Internal data is only part of the puzzle. Hyperlocal weather, mobility data, demographic trends, and EV charging patterns can be connected via APIs to make local scenarios more realistic and forward-looking. A turning point is on the horizon — many DSOs are investing in modern data platforms and scalable compute. However, the leap in value comes when those tools pair with strong governance, clear ownership, and a business view of the decisions that the data must improve.

AN INDUSTRIAL APPROACH TO DATA INTELLIGENCE

To convert data into strategic insight, DSOs should move from isolated analytics projects to a data-as-a-product model. Data products are curated, documented, reusable assets designed for specific business consumers, with clear owners, quality standards, and service levels. A practical architecture uses three refinement layers:

1. Bronze layer (raw ingestion) — stores GIS, AMI, SCADA, and other sources in their original form, creating a governed historical record for audit and future use cases
2. Silver layer (refinement and reconstruction) — cleanses, validates, harmonizes, and enriches raw data, correcting quality issues, rebuilding missing information, and adding business context
3. Gold layer (business-ready intelligence) — delivers reusable data products optimized for analytics, planning, and decision-making, such as customer profiles, asset risk scores, and flexibility-potential maps:

• Customer-segment profiles— classify customers into behavioral archetypes and support granular, bottom-up load forecasting
• Low-voltage substation health score — combines topology, asset, and customer profiles into dynamic risk indicators
• Local flexibility potential — quantifies flexibility needs by location, kW, and hours/year under future scenarios

This is also a delivery model choice. External specialists should work as an extension of the DSO’s data science and planning teams, not as a separate analytics provider. The DSO defines operational rules, thresholds, and business logic, including what level of error is acceptable for planning purposes.

The technical team industrializes the data in pipelines with transparent validation, tolerance levels, and exception handling. This creates a shared language between planners and data engineers.

The engineering work is highly practical: account for daylight-saving shifts, detect metering errors, rebuild missing POD-to-substation mappings, track version topology changes over time, and reconcile outages or maintenance events with measurements. These tasks are not secondary hygiene activities; they determine whether advanced models produce results that planners can trust and regulators can understand. Once this foundation is in place, future use cases start from validated data rather than repeated cleaning exercises. The data backbone becomes an enduring capability that can continuously feed planning, forecasting, and flexibility tools.

UNLOCKING FLEXIBILITY VALUE THROUGH GRID INTELLIGENCE

Gold-layer data products turn grid intelligence into operational value. They give planners a clearer view of future load growth, asset stress, and flexibility needs at the decision-making level: the customer segment, feeder, transformer, and local cluster.

For network development planning, bottom-up scenarios based on classified customer behaviors replace generic assumptions. This improves regulatory granularity, forecast accuracy, and the targeting of reinforcements, especially in low-voltage areas where electrification evolves unevenly, and a small number of assets can drive local peaks.

AI models, including neural networks, can infer the energy signatures of EVs, heat pumps, or PV generation from validated load profiles, even when customers have not explicitly declared those assets. These inferred signatures also support more transparent scenario building because planners can link future peaks to observable adoption patterns.

For flexibility, a local flexibility-potential product enables a CAPEX versus OPEX comparison by substation and time window. Planners can distinguish structural congestion from short, localized peaks, estimate the kW and hours needed, and map where a flexibility service can defer investment. The result is a shift from reactive reinforcement to optimized capital allocation and more credible flexibility procurement.

BUILDING A FLEXIBILITY-READY, DATA-CENTRIC DSO

Distribution management has become a data-driven discipline. For DSO leaders, the priority is to turn data from an operational byproduct into a strategic asset that shapes planning, investment, and market facilitation. The transformation should start with business decisions, not technology alone, and requires executive sponsorship across network, IT, and regulatory functions.

To embark on this transformation, leaders should reflect on a series of critical questions:

1. Data readiness. Do we know which datasets are trusted enough to inform planning and where quality gaps still limit decisions?
2. Lighthouse use case. Which decision (regulatory compliance, CAPEX optimization, fault prediction, or flexibility procurement) would benefit most from a first strategic data product?
3. Operating model. Do planning, operations, IT, and data teams share rules, accountability, and a role (e.g., data product manager) to translate business needs into reusable assets?

Successfully navigating this complexity demands a clear commitment to three strategic imperatives:

1. Make data governance a C-level priority. Ownership, quality, lineage, and accountability must be managed as business requirements (not IT housekeeping) with clear escalation paths for critical datasets.
2. Choose products over projects. Build durable data products that serve multiple use cases, reduce or eliminate duplicate effort, and compound value over time rather than isolated analyses that disappear after delivery.
3. Build a data-driven culture. Ensure all decisions, from long-term planning to daily operations, are validated by reliable, explainable information and understood by business teams, so adoption extends beyond the analytics function.

Case study: Enabling bottom-up flexibility analytics through neural reconstruction

A leading DSO used deep neural networks to reconstruct missing customer-level load curves. The challenge was structural: in many low-voltage networks, only part of the customer base had complete hourly 2G smart meter data. Even where meters were installed, communication gaps, corrupted readings, and outages created visibility gaps that weakened forecasting, transformer stress analysis, and bottom-up flexibility calculations. Without a complete customer-level view, planners risked underestimating local peaks or overbuilding against poorly understood constraints.

The solution was a dual-model reconstruction engine: one network for standard customers below 1,000-kWh monthly net exchange and one for larger consumers. Each model was trained on filtered, coherent data that aligned contractual attributes, actual measurements, and inferred consumption or production behavior, ensuring the training set was both statistically robust and operationally meaningful. This filtering step was essential to avoid training on inconsistent meter readings or customer records.

The architecture combines multi-head self-attention, residual connections, and squeeze-and-excitation blocks, enabling the model to learn both local hourly patterns and longer-term dependencies like weekday/weekend behavior and seasonality. In inference mode, it assigns a complete 24-hour curve to each POD for each missing day. GPU acceleration allows a full DSO perimeter to be reconstructed in about five minutes, supporting scalable daily or monthly refresh cycles.

In 2024, the engine reconstructed more than 74 million daily load curves, with typical monthly validation errors below 10% versus transformer-level energy balances. This accuracy threshold matters because overload detection, flexibility estimation, and anomaly spotting must remain reliable enough to support planning and operational choices. Validation against transformer balances also kept the model anchored to physical grid behavior.

Figure 1 shows how the neural network fills in missing portions of a customer’s daily profile by generating reconstructed curves consistent with the surrounding measured days. This restores data continuity when smart meter readings are unavailable.

Figure 2 compares reconstructed hourly estimates with measured transformer data. The reconstructed curve follows the main shape and dynamics of the real load profile, supporting a consistent bottom-up view where customer-level data is incomplete.

The lesson is strategic:DSOs do not need to wait for universal smart meter coverage before running bottom-up grid analytics. With strong data-quality controls and scalable reconstruction pipelines, machine learning can restore visibility across millions of low-voltage customers and provide an hourly-resolution foundation for advanced planning tools. Reconstruction becomes the bridge between today’s partial measurements and tomorrow’s fully observable grid.

Case study: Turning flexibility from a technical concept into an investment decision

Where can flexibility credibly replace grid reinforcement, and where is immediate CAPEX unavoidable? As electrification accelerated across the service area, a DSO needed to distinguish structural bottlenecks requiring infrastructure upgrades from localized, time-bound constraints that could be managed more efficiently through flexibility. Traditional planning identified load growth but did not always show the economic nature of the constraint.

Arthur D. Little (ADL) supported the client in building a bottom-up flexibility business case at secondary-substation level. Starting from the client’s 2024 network baseline, reconstructed load patterns were projected to 2029 to assess how local stress could evolve under electrification assumptions and where flexibility could have measurable value.

The method quantified flexibility potential at the transformer level. Reconstructed POD profiles were aggregated to medium-voltage/low-voltage transformers, creating high-resolution simulations of loading conditions and making it possible to compare stress across substations consistently.

Flexibility potential was identified by comparing these load profiles with technical capacity thresholds. A 75% saturation level acted as an early stress indicator; each threshold violation was then classified according to duration and severity. This made the model actionable for planners because it converted a technical loading profile into a decision rule.

Short overloads, typically lasting between two and four hours per day and remaining below full capacity, were treated as non-structural and addressable through flexibility. Persistent or severe overloads, including events above full capacity or lasting longer than four hours, were classified as structural and linked to reinforcement.

For each nonstructural gap, the model calculated both the energy volume above the threshold and the duration of the event, translating network stress into actionable flexibility requirements expressed in MWh and hours. The analysis showed a material increase in localized stress. In 2024, 56 substations experienced nonstructural overloads, corresponding to about 1,600 violation hours and 19 MWh of flexibility-relevant energy. In the 2029 scenario, this rose to 122 substations, 7,800 hours, and 163 MWh. The increase confirmed that constraints were expanding but that many remained localized and time-bound. This changed the planning discussion in three ways:

1. It segmented constraints by economic logic. Not every congestion justifies immediate reinforcement, and some can be addressed with targeted flexibility procurement during limited time windows. This distinction is critical to avoid overinvesting where an operational measure is sufficient.
2. It created a fact base for CAPEX deferral and phasing. It did this by showing where overloads are concentrated, how often they occur, and how large the flexibility need is. Engineering resources can therefore be directed toward structural issues first.
3. It made flexibility bankable. Rather than treating flexibility as a generic innovation topic, the DSO could quantify an addressable perimeter, map it geographically, and assess where local procurement mechanisms could complement or compete with grid expansion, including through transparent cost comparisons.

Figure 3 shows a mapping of secondary substations with nonstructural overload events. Marker size reflects stress intensity, while the data panel details selected-substation indicators such as load exceedances, connected customers, and demand mix. This helps identify priority areas where flexibility can substitute or complement reinforcement.

By Arthur D. Little

02/07/2026

Fuente: Arthur D. Little

https://www.adlittle.com/en/insights/viewpoints/grid%E2%80%99s-new-gold-harnessing-dso-data-unlock-flexibility