Deploy Machine Learning at scale with Kedro and Cortex.dev

ML model deployment in production is still an area that lacks conformity, nomenclature, and patterns. Aside from few technology companies who started the journey early on, for late adopters of ML practices, it’s pretty much the wild west when it comes to the standards of model deployment. Some other challenges include the culture of dataContinue reading “Deploy Machine Learning at scale with Kedro and Cortex.dev”

Power generation prediction using realtime inference with Spark Structured Streaming and Kafka

A modern power plants generate an enormous amount of data, with the ever growing number of high frequency sensors, some of the applications of this data are : the protection against problems induced by combustion dynamics for example, and help improve the plant heat rate, and optimize power generation. If we take a sensor thatContinue reading “Power generation prediction using realtime inference with Spark Structured Streaming and Kafka”

Deploy PJM Electricity Load Forecast Model on AWS SageMaker

in the energy industry, forecasting the grid load is vital for various commercial optimizations around Day Ahead and Real Time trading, but it also help Independent Power Providers (IPPs) allocate the right generation unites. we talked previously about energy markets, CAISO, PJM, ERCOT , and others. In this article the goal is not to talkContinue reading “Deploy PJM Electricity Load Forecast Model on AWS SageMaker”

Predicting Electricity Wholesale Prices using AWS Machine Learning

Electricity Wholesale Markets Most of the nation’s wholesale electricity sales happen in a competitive market managed by Independent System Operator (ISO), with over 200 million customers in these areas and over $120 billion in annual energy transactions taking place. Under the Federal Power Act, these markets are overseen by the Federal Energy Regulatory Commission (FERC),Continue reading “Predicting Electricity Wholesale Prices using AWS Machine Learning”

Graph databases and Energy: Modeling LMPs and CRRs

Graph Databases : Graph databases are NoSQL databases which use the graph data model comprised of vertices, which is an entity such as a person, place, object or relevant piece of data and edges, which represent the relationship between two nodes. Graph databases are particularly helpful because they highlight the links and relationships between relevantContinue reading “Graph databases and Energy: Modeling LMPs and CRRs”

Using Smart Meter Data for consumer electricity usage forecasting

Electric energy consumption is essential for promoting economic development and raising the standard of living. In contrast to other energy sources, electric energy cannot be stored for large-scale consumption. From an economic viewpoint, the supply and demand of electric energy must be balanced at any given time. Therefore, a precise forecasting of electric energy consumptionContinue reading “Using Smart Meter Data for consumer electricity usage forecasting”

My books recommendations for AI and ML

With AI and machine learning creeping in every industry, the paradigm of IT as we know it is also changing, ML is a natural extension of the software revolution we have seen in the last decades, knowing how to utilize ML in your industry will be a key element for success and growth in theContinue reading “My books recommendations for AI and ML”

Storing your ML Models with parameters

Often when training machine learning models you find yourself creating different estimators and tuning this parameter or that to get the results you want, you may also find yourself wanting to save the results of those iterations, to save you time in the future. that’s what I’m trying to address in this post, having someContinue reading “Storing your ML Models with parameters”

Querying Ercot public dataset using AWS Glue and Athena

ERCOT is an acronym for Electric Reliability Council of Texas, it manages the flow of electric power to more than 25 million Texas customers — representing about 90 percent of the state’s electric load. As the independent system operator for the region, ERCOT schedules power on an electric grid that connects more than 46,500 miles ofContinue reading “Querying Ercot public dataset using AWS Glue and Athena”

Automating AWS server builds from ServiceNow requests.

A lot of organizations are using ServiceNow to manage requests for creating new compute resources in the could ( i.e servers, lambda functions, containers …) Usually, CMP tools like Scalr, or even ServiceNow Cloud Management can ( to a certain degree ) automate this process, but there are two major issues with these approaches :Continue reading “Automating AWS server builds from ServiceNow requests.”