Proving the concept for smart metering data management

IBM Research works with AMT-SYBEX to develop and test Smart DTS

Published on 11-Mar-2011

Validated on 12 Nov 2013

"We owe a great deal to IBM, both for the Informix technology itself, and for the fantastic support from all levels of the organisation. Everything we needed to make this project a success was made available to us. We plan to continue to work with the IBM team to develop Smart DTS and help build a smarter planet." - Gordon Brown, DTS Product Owner, AMT-SYBEX

Customer:
AMT-SYBEX

Industry:
Computer Services, Energy & Utilities

Deployment country:
United Kingdom

Solution:
Database Management, Optimizing IT

Overview

As a provider of solutions for energy and utilities companies across Europe, AMT-SYBEX is constantly looking to develop new products and services that help its customers meet the challenges of an ever-changing industry. When the UK government announced that the introduction of smart metering technologies to all homes would be a top priority, AMT-SYBEX immediately recognised that the proposed rollout presented both a challenge and an opportunity that would soon be repeated in other global energy markets.

Business need:
To meet the demands of smart metering for UK energy and utilities companies, AMT-SYBEX identified an opportunity to extend its existing Data Transfer Solution (DTS) to load, validate, store and provide smart meter interval and event data to external systems. The challenge was to build a platform that could process the potentially enormous data volumes quickly enough, without requiring enormous investments in new hardware.

Solution:
Working with IBM® Research at the Hursley Innovation Centre, AMT-SYBEX developed and tested Smart DTS – a solution which embeds IBM Informix® TimeSeries DataBlade™ technology, which is designed specifically to handle time series data such as sequential meter readings. During a proof of concept, the team simulated a system that received and processed data from 10 million smart meters. The solution processed more than 200,000 transactions per second, and completed the job within 96 minutes on a single eight-processor server.

Benefits:
Demonstrates that Smart DTS is capable of handling real-world smart metering challenges. Provides data that helps energy and utilities companies monitor consumption more accurately and design new tariffs to smooth out peaks in demand. Enables the development of new services such as real-time energy monitoring for consumers. Increases the practical potential of micro-generation initiatives.

Case Study

As a provider of solutions for energy and utilities companies across Europe, AMT-SYBEX is constantly looking to develop new products and services that help its customers meet the challenges of an ever-changing industry. When the UK government announced that the introduction of smart metering technologies to all homes would be a top priority, AMT-SYBEX immediately recognised that the proposed rollout presented both a challenge and an opportunity that would soon be repeated in other global energy markets.

“With traditional metering solutions, a utilities company will typically collect one reading per meter per quarter,” explains Gordon Brown, DTS Product Owner at AMT-SYBEX. “So if you have 10 million meters, that’s only 40 million data points per year. With smart meters that send a new reading every half an hour, if you have 10 million meters, that’s 20 million data points per hour! The massive increase in data volumes has proven to be one of the main challenges of creating viable smart metering solutions, because with a traditional relational database structure, it’s impossible to handle that much sequential data unless you make enormous investments in hardware.”

Transforming the energy industry

If AMT-SYBEX could crack this problem, there was an opportunity to develop a new solution that would help its clients not only meet government requirements for smart metering, but also develop new and innovative services that may help to improve efficiency and enhance customer service.

“Smart metering will enable fine-grained monitoring of consumption, both for the companies themselves and for individual consumers,” says Brown. “If utilities companies can gain a more accurate idea of consumption patterns, they will be able to design tariffs that reward consumers for consuming less power during times of peak demand, and consumers will gain a better understanding of how they can save money by altering their usage patterns.

“For example, there’s often a demand spike in the early evening because people come home from work and turn a variety of appliances on. If energy companies can show their customers the impact that this has on their energy consumption, and can incentivise them to avoid it, it will help to smooth out the peaks in demand and improve the overall efficiency of the generation and distribution network.”

Making more out of microgeneration

“Another example is microgeneration,” he adds. “If people have solar panels or turbines on their land, they can sell the electricity they generate back to the National Grid. If smart metering reveals when the periods of peak demand are, it might be possible to persuade these people to use their own generators to supply their energy locally during these periods – reducing the strain on supply at peak times.”

Revolutionary time series technology

For these dreams to be realised, AMT-SYBEX needed to solve the data problem, and began working with IBM Research at the Hursley Innovation Centre to find a solution.

“Our Data Transfer Solution (DTS) is already the solution of choice for data transfer between the many separate partners in the energy industry supply ecosystem,” comments Brown. “We wanted to extend this solution to handle smart metering, and created Smart DTS, which is built on IBM Informix TimeSeries DataBlade technology.”

The Informix database is built specifically to handle time series data, such as meter readings. Whereas a traditional relational database creates a new row every time a new data point is collected, Informix creates a single row for each meter, and simply appends subsequent readings from the same meter to that row.

“The way that the data is structured with Informix TimeSeries DataBlade means that you can process, validate, analyse and manipulate all the readings from a single meter in a very efficient manner,” says Brown. “It’s dramatically faster than a relational data model, and it also cuts storage requirements by approximately 55 percent.”

Proving the concept

The combined team from AMT-SYBEX and IBM created a proof-of-concept for Smart DTS that would simulate the data load created by 10 million smart meters, each submitting a new reading every 30 minutes. The Smart DTS prototype needed not only to record the data as it came in, but also to subject it to a number of complex validations and calculations – for example, aggregating consumption over the past 24 hours and comparing it to expected readings, to ensure that the meter is working correctly.

“IBM provided a single mid-range server with eight quad-core processors,” says Brown. “On this one machine, Smart DTS was able to deliver more than 200,000 database transactions per second, and completed the full processing run (including complex validation and estimation of values that fail the validation process) for daily volumes from 10 million meters within just 96 minutes. It’s hard to tell whether you are comparing apples with oranges, but we believe that this level of performance is right at the forefront of what anyone has achieved in the smart metering space so far.”

A partnership for a smarter planet

IBM and AMT-SYBEX have continued to develop and optimise the solution, and recently gave a demonstration to one of the UK’s largest electricity companies which achieved nearly twice the performance of the prototype, using a server with 16 CPUs. This demonstrated the near-linear scalability of the offering.

“The clients who have seen Smart DTS so far have been very impressed, and we’re expecting a lot of interest in the product as soon as we release it to market in Europe,” concludes Brown. “We owe a great deal to IBM, both for the Informix technology itself, and for the fantastic support from all levels of the organisation. Everything we needed to make this project a success – from access to the top technical experts and industry specialists, through to hardware and facilities – was made available to us. We plan to continue to work with the IBM team to develop Smart DTS and help build a smarter planet.”

Products and services used

IBM products and services that were used in this case study.

Hardware:
Power Systems

Software:
Informix TimeSeries DataBlade, Informix Dynamic Server

Legal Information

© Copyright IBM Corporation 2011. IBM United Kingdom Limited, PO Box 41, North Harbour, Portsmouth, Hampshire, PO6 3AU. Produced in the United Kingdom, March 2011. All Rights Reserved. IBM, the IBM logo, ibm.com, DataBlade and Informix are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. A current list of other IBM trademarks is available on the Web at “Copyright and trademark information” at http://www.ibm.com/legal/copytrade.shtml. Other company, product or service names may be trademarks, or service marks of others. DTS and Smart DTS are products of AMT-SYBEX. IBM and AMT-SYBEX are separate companies and each is responsible for its own products. Neither IBM nor AMT-SYBEX makes any warranties, express or implied, concerning the other’s products. References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program or service is not intended to imply that only IBM’s product, program or service may be used. Any functionally equivalent product, program or service may be used instead. All customer examples cited represent how some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. IBM hardware products are manufactured from new parts, or new and used parts. In some cases, the hardware product may not be new and may have been previously installed. Regardless, IBM warranty terms apply. This publication is for general guidance only. Photographs may show design models.