How to Achieve 60% AWS Cost Optimization with Functions and Tags?

AWS, Cost Optimization, DevOps
AWS Cost

Introduction

In today’s rapidly evolving digital landscape, cloud computing has become the backbone of many businesses, enabling scalability, flexibility, and cost-effectiveness. Amazon Web Services (AWS) is a popular choice among enterprises for its wide range of services and capabilities. However, as businesses grow and their AWS infrastructure expands, managing costs can become a significant challenge. This blog presents a case study on how a Madgical Techdom, leveraged serverless functions and resource tagging to optimize AWS cost(expenses) during off hours and weekends, resulting in substantial cost savings.

The Challenge: Rising AWS Cost Optimization

Madgical Techdom, despite office hours being over, experienced a surge in AWS costs due to their machines continuing to run. Their infrastructure comprised various AWS services, including EC2 instances, RDS databases, S3 buckets, and more. While AWS offered cost optimization tools, the company struggled to identify areas where expenses could be reduced without impacting performance.

Identifying Off-Peak Hours for AWS Cost Optimization

To tackle the cost optimization challenge, Madgical Techdom analyzed its website traffic patterns using AWS CloudWatch metrics and logs. They discovered that their peak customer activity occurred during specific hours, primarily during weekdays. Outside these peak hours, the website experienced significantly lower traffic and resource utilization. This observation presented a prime opportunity for cost reduction by leveraging serverless functions and resource tagging effectively.

Harnessing Serverless Functions in AWS Cost Optimization

Madgical Techdom decided to adopt a serverless architecture for certain non-critical functions that could be processed asynchronously. They implemented AWS Lambda, a serverless computing service, to offload resource-intensive tasks during off-peak hours. By employing serverless functions, the company reduced the need for running and maintaining EC2 instances 24/7, resulting in considerable cost savings.

For instance, Madgical Techdom implemented a serverless function to process user-generated images uploaded to their platform. During peak hours, when the load was high, the company continued to rely on EC2 instances for faster processing. However, during off-peak hours, the Lambda function seamlessly handled image processing tasks, eliminating the need for idle EC2 instances and reducing costs significantly.

Resource Tagging for Enhanced AWS Cost Management

Madgical Techdom recognized the importance of resource tagging to gain granular visibility into their AWS expenses. They strategically assigned tags to their resources, including EC2 instances, RDS databases, and S3 buckets. The tags provided essential metadata that helped in identifying and classifying resources based on their usage, importance, and department ownership.

By leveraging resource tags, Madgical Techdom could generate accurate cost reports using AWS Cost Explorer. They could identify resource-wise cost breakdowns and gain insights into areas where expenses could be reduced. For instance, by analyzing the costs associated with specific tags, such as “development,” “testing,” or “staging,” the company could identify idle or underutilized resources during off-peak hours and take appropriate action to optimize costs.

Setup AWS Cost Optimization Operations

This section contains information on utility functions for AWS operations and it can be deployed using the Serverless Framework.

AWS Operations:

  • Get the IP address of an instance e.g. https://<serveless-endpoints>/?instance_id=<instanceId>
  • Start an instanceId e.g. https://<serveless-endpoints>/manageInstance?instance_id=<instanceId>&action=start
  • Stop an instanceId e.g. https://<serveless-endpoints>/manageInstance?instance_id=<instanceId>&action=stop

Prerequisites

Setup for the AWS Cost Optimization operations

1. Clone this repository: 

git clone https://github.com/madgicaltechdom/aws-operations.git

2. Navigate to the project directory: 

cd aws-operations

3. Deploy the function to your AWS account using the Serverless Framework: 

serverless deploy

4. Copy the endpoints url from the output command line.

Usage

You can invoke the endpoint using: 

https://<serveless-endpoints>/?instance_id=<instanceId>

You can invoke the Lambda function using the Serverless Framework: 

serverless invoke --function getPublicIp --path data.json

You can also test the function by calling the endpoint that is created when you deploy the function. This endpoint will be shown in the output of the serverless deploy command.

Configuration

You can configure the function and the events that trigger it in the serverless.yml file.

Schedule timing to start and stop the EC2 Instances for the AWS Cost Optimization

Many businesses are focused on lowering the costs of running AWS EC2 instances. Terraform can help you save money by automating EC2 instance management. This solution enables you to automate the start and stop of your instances based on your business requirements.

Non-production machines can be turned off after hours and on weekends. The machines can be started whenever working hours begin. This might result in more than 50% of your ec2 instances being stopped, saving you a lot.

Prerequisites

  • Install terraform video.
  • Set up your AWS account video.
  • Create a programmatic user with the permissions specified in the permission.json file.
  • To schedule ec2 instances, we must have tagged them with the ‘environment’ tag.

Cronjob Fundamentals

This cron job is made up of several fields, each separated by a space:

[Minute] [Hour] [Day_of_Month] [Month_of_Year] [Day_of_Week]
  • The first field is for minutes (0-59).
  • The second field is for hours (0-23).
  • The third field is for days of the month (1-31).
  • The fourth field is for months (1-12).
  • The fifth field is for days of the week (0-7, where both 0 and 7 represent Sunday).
  • The sixth field is for the year by default it takes the current year.

Usage

1. Clone this repository to your local machine by running the below command:

git clone https://github.com/madgicaltechdom/Schedule-Idle-EC2-Instances-with-Terraform-Automation.git

2. Navigate to the repository directory by running the below command:

cd Schedule-Idle-EC2-Instances-with-Terraform-Automation

3. Log in to your AWS Account, search for the EC2, and open the instance in which you want to add scheduling, in the below options click on the “Tags”, then the “Manage tags” button. Here you need to add a tag, click on the “Add new tag” button, in the Key field select “environment”, and in Value select “qa” or “prd” according to your need then click on the “Add Tag” button.

Instance Tags-1
Instance Tags-2

Changes

4. Optional, if you want to add another tag then first you need to add that tag in the workspace_to_environment_map variable in the variable.tf file and use that tag for scheduling.

variable "workspace_to_environment_map" { type = map(string) default = { qa = "qa" prd = "prd" } }

5. To match your requirements, modify the stopping time value in the file variable.tf. In this case, “30 14” is UTC time, which corresponds to 8 p.m. IST. For timing, please refer to the chart we printed on the last of this file. Additionally, the machine is shut off at 8 p.m every Monday to Saturday. You can customize your days according to your need.

variable "cron_stop" { description = "Cron expression to define when to trigger a stop of the DB" default = "30 14 ? * MON-SAT *" }

6. Change the starting time value in the file variable.tf to suit your needs. For timing, please refer to the chart in the last of this file. In this case, “30 03” denotes UTC time, which corresponds to 9 a.m. IST. Additionally, the machine is turned on at 9 a.m every Monday to Saturday. You can customize your days according to your need.

variable "cron_start" { description = "Cron expression to define when to trigger a start of the DB" default = "30 03 ? * MON-SAT *" }

7. Change the AWS access key value in the file variable.tf to meet your requirements.

variable "access_key" { description = "value of access key" default = "" }

8. Change the AWS secret key value in the file variable.tf to meet your requirements.

variable "secret_key" { description = "value of secret key" default = "" }

How to Apply

9. Create a new workspace for each environment you want to deploy, for example for qa(testing):

terraform workspace new qa

10. Initialize Terraform by running the below command:

terraform init

11. Run the below command to preview the changes:

terraform plan

12. Run the below command to apply the changes:

terraform apply

Verify Machines Status

1. In the AWS account, search for “Cloudwatch,” then click on “Rules” in the “Events” section. Two rules are added, as shown in the image below. You can check the details of each rule by clicking on it.

Cloudwatch Rules-1
Cloudwatch Rules-2
Cloudwatch Rules-3

2. For checking the Execution Point search for the “System Manager” then click on the by clicking on “Automation” in the “Change Management” section:

System Manager

NOTE:- If you want to schedule other machines then first add the tag on those machines and then run this script again.

Results and Cost Savings

By adopting serverless functions and resource tagging, Madgical Techdom achieved significant cost reductions during off hours and weekends. The following highlights the outcomes of their cost optimization efforts:

  1. Reduced EC2 Instance Costs: By offloading non-critical tasks to serverless functions, Madgical Techdom minimized the need for running and maintaining EC2 instances 24/7, resulting in a substantial reduction in EC2 costs during off-peak hours.
  2. Eliminated Idle Resources: Leveraging resource tagging and cost reports, Madgical Techdom identified idle resources during off hours and took corrective measures. They terminated or downsized underutilized instances and databases, reducing unnecessary expenses.

Results and Benefits

By implementing these cost optimization strategies, Madgical Techdom achieved significant cost savings during off-peak hours. Additionally, they managed to reduce their AWS expenses by up to 60% without impacting their application’s performance or availability. The serverless architecture allowed them to scale resources on demand and pay only for the computing power they used. The tagging strategy improved their cost-tracking capabilities and enabled them to make data-driven decisions to optimize their infrastructure further.

Conclusion

Optimizing AWS expenses can be a challenging task, especially for organizations with complex infrastructures and dynamic workloads. However, by leveraging serverless functions and implementing a comprehensive tagging strategy, businesses like Madgical Techdom can effectively optimize costs during off hours and weekends.

I hope this article proves beneficial to you. In case of any doubts or suggestions, feel free to mention them in the comment section below.

References

You may also like