The free energy market in Brazil is growing rapidly and as entry barriers for new players are reduced, more and more trading agents are joining CCEE. In recent years, the industry’s regulator, the National Agency of Electric Energy (ANEEL), announced plans to change the energy pricing model which would have farreaching consequences for all players. Dario Almeida, Executive Systems Architecture Manager at CCEE, explains further: “Previously, prices were calculated weekly. This meant that energy prices did not always adequately reflect the relationship between generation and demand, leading to the need to charge agents high system charges due to the price-cost imbalances. With the new model, prices are closer to the operating reality, as they are calculated daily, on an hourly basis, and published the day before the operation. Thus, the price curve tends to follow the real demand curve more realistically, reducing the need for additional charges and significantly minimizing the overall costs of the system. This would also allow market agents to have contracts with near realtime prices to avoid losses. Looking to the future, we were very much in favor of this approach, but realized that the resulting massive data volume increase would give us some technical challenges.”

The data warehouse solution in place was Oracle and preliminary testing showed that some hourly pricing queries could take more than five hours to return, obviously defeating the objective and causing business intelligence (BI) timeouts. However, Oracle worked very effectively as a transaction database platform, so CCEE looked for a big data option that would integrate with Oracle and the suite of BI tools in use at the organization. An initial load of 10TB of historical data would need to be loaded onto the new platform, with high volumes of incremental data added on a daily basis without negatively affecting system performance.

Infrastructure cost savings through OpenText performance and data compression

CCEE engaged IMExperts, an IT consultancy firm specializing in big data and artificial intelligence. Eduardo Valladares, Sales Director at IMExperts: “Time was of the essence for this project, as there was only a year to implement the new pricing model. For CCEE the decision was almost binary; a new solution either could or could not support the required data volume levels. Our previous experience with Vertica told us that it could be a great contender. We ran a proof-of-concept (POC) with Vertica and Oracle running the most demanding queries against the new hourly pricing model. Vertica demonstrated a performance improvement of over 97 percent which was great news.” Some of the heaviest hourly queries that took more than five-hours to run in Oracle would run in less than 5 minutes in OpenText™ Vertica™. In addition to the impressive performance improvements achieved in hourly queries, we also saw a 40 percent increase in access performance to the 200 standard BI reports that we make available to our stakeholders.

CCEE’s in-house data management team partnered closely with IMExperts during the phased OpenText implementation, engaging in very effective knowledge transfer so that CCEE analysts were able to maintain and manage the system themselves. With very strict data protection requirements and complex accounting and account settlement algorithms to manage, the strong partnership between all parties was the key to success. OpenText™ was able to compress historical data by over 55 percent and because Oracle is now focused on transaction processing only, its capacity has increased. This means CCEE could postpone planned additional processor investment and introduce a lower-cost data storage infrastructure to realize further cost savings. They chose to license OpenText based on data volumes, rather than server or processor units. This predictable pricing model works well in rapid growth scenarios such as CCEE.

Leveraging the power of Vertica to manage our huge data volumes with high performance and then presenting the near real-time results in Tableau-created executive dashboards gave us a whole new dimension. We created a web-based self-service tool for our staff which enables better trend analysis and data-driven decision making.

Dario Almeida
Executive Systems Architecture Manager, CCEE

Data analytics drives improved decision making and stakeholder communication

During the implementation OpenText was fully integrated with Oracle and Tableau for visual BI. It also supports all existing Oracle security policies required by CCEE agents to maintain full data confidentiality while keeping response times as expected. Almeida on the OpenText integration with Tableau: “We were already Tableau power users and had created some really useful dashboards. Leveraging the power of Vertica to manage our huge data volumes, then presenting the near real-time results in Tableau-created executive dashboards gave us a whole new dimension. We created a web-based self-service tool for our staff which enables better trend analysis and data-driven decision making. We also created new dashboards with security indicators to show market players that their operations are run safely. In the first month of introducing this it was accessed over one million times, a huge success! Sophisticated data analytics enables innovation and improves our communication.”

Previously, only transactional analytics services were offered to stakeholders, but now CCEE agents can access their near real-time analytics directly from OpenText via API-services in a secure SOA architecture, with sub-second response times. As CCEE data analysts are exploring OpenText’s cognitive features, i.e., machine learning and artificial intelligence, they expect to release value-add services to clients this year.

Reduced TCO and improved platform stability

“Our information is completely transparent which helps in our annual audit process,” says Almeida. “Even though we are a relatively small organization, our core data management processes are supported by a robust enterprise-level infrastructure. Vertica manages all our security authorization processes. All data is housed in one database and we provide web-access to over 10,000 agents. With the very sensitive data we hold, we can only do this if access is fully secured. Thankfully, we could build our authorization features in Vertica and have a full auditing track.”

“We love the Vertica platform stability and ease of management,” says Almeida. “We only require half a fulltime employee (FTE) to maintain our Vertica [now part of OpenText™] solution which means our cost of ownership has reduced. Our platform availability has increased from 99.3 percent to 99.9 percent. Bearing in mind that our contracts grew by more than 300% in the last five years, and we now manage close to 80,000 contracts, this stability and scalability give us the confidence that Vertica [now part of OpenText™] will continue to serve us well into the future.”

He concludes: “Our stakeholders see a huge benefit in the move to hourly pricing which accurately reflects the energy demand curve, smoothing out the previous market imbalances. We could only make this happen successfully with Vertica [now part of OpenText™] and we are pleased with the collaboration between our inhouse team and IMExperts. Together, we have created a scalable data analytics solution to benefit all energy market participants.”

We love the Vertica platform stability and ease of management. We only require half a full-time employee (FTE) to maintain our Vertica solution which means our cost of ownership has reduced. Our platform availability has increased from 99.3 percent to 99.9 percent.

Dario Almeida
Executive Systems Architecture Manager, CCEE

About Chamber of Electric Energy Commercialization

CCEE acts as operator of the Brazilian electric energy market. It aims to facilitate a competitive, sustainable, and safe trading environment, involving thousands of associates. One of its main activities is to account for the purchase and sale of electricity, verifying the differences between the amounts contracted and the amounts generated or consumed by market agents. It records the contracts signed between buyers and sellers, in order to measure the physical amounts of energy handled by agents.