Monday, 23 July 2018

AWS Glue now offers additional Apache Spark metrics for ETL jobs

You can now do better profiling and debug the ETL jobs with the addition of Apache Spark metrics. With the latest update, the customers can now easily keep a record of the running metrics such as memory usage, CPU load of the driver and executors, data shuffles among executors and bytes read and written from the AWS Glue Console. The additional ETL job metrics added by the AWS Glue will help plan CPU capacity, debug code and identify data issues. In Amazon CloudWatch you can set up alarms on certain job conditions. From the AWS Glue jobs you can collect the metrics and visualize them on the Amazon CloudWatch and AWS Glue console to identify and fix issues. 

No comments:

Post a Comment

Yes, Cloud Cost Optimization Is Real and It’s Saving Big Bucks

It all started with a short message in the team chat: “ Hey… why is our cloud bill twice as high this month? ” Raj, a DevOps engineer at a f...