BOTO3: Find users in AWS IAM with no MFA enabled

AWS IAM provides an additional layer of security for IAM users with multi factor authentication. When mfa enabled, users will be prompted to enter the authentication code, after providing the username and the password. AWS supports Virtual MFA devices, U2F security key, Hardware MFA devices etc. Virtual MFA device, especially Google Authenticator is commonly used … Continue reading BOTO3: Find users in AWS IAM with no MFA enabled

How I passed GCP Associate Cloud Engineer certification

I recently passed the GCP Associate Cloud Engineer certification after preparing for 2 months. In this post, I wrote about how I prepared for the certification, study materials used etc., I attended the Google cloud summit Bangalore’19 which piqued my interest in Google Cloud Platform. Given the interest and demand around multi-cloud experience in the … Continue reading How I passed GCP Associate Cloud Engineer certification

Boto3: Python script to find AWS IAM access keys older than 90 days

To access AWS programmatically via CLI or api, we can use IAM access keys provided by AWS. The access key consists of an access key ID and a secret access key. We can generate these keys in the IAM console and configure these keys in our local laptop by running aws configure command. These keys … Continue reading Boto3: Python script to find AWS IAM access keys older than 90 days

EC2 api RequestLimitExceeded error: How to fix it using cloudwatch, cloudtrail

RequestLimitExceeded - The error message which almost every AWS user is familiar with. We often encounter this error in operations interacting with AWS services. AWS imposes a hard limit on the maximum number of api calls allowed for their services and it can’t be increased. Last month, we encountered this error for EC2 operations in … Continue reading EC2 api RequestLimitExceeded error: How to fix it using cloudwatch, cloudtrail

How to increase the HDFS capacity of AWS Elastic Mapreduce EMR cluster

emr hdfs

In this tutorial, we’re going to see how to increase the hdfs capacity of a running EMR cluster. Sometime back, we received an alert that HDFSutilization was high on one of our cluster. Upon checking, the usage is an expected one but we under provisioned the storage capacity during the creation of the cluster and … Continue reading How to increase the HDFS capacity of AWS Elastic Mapreduce EMR cluster

AWS EMR Uniform Instance groups

In this post, I wrote about the AWS EMR uniform instance groups overview, advantages and caveats of using it. AWS EMR architecture contains master node, core node(s) and task nodes.  If you’re new to EMR, refer https://www.hadoopandcloud.com/aws/amazon-emr/  for a quick introduction. While creating the cluster, you have two configuration options for the nodes - instance … Continue reading AWS EMR Uniform Instance groups

Script to delete thousands of delete marker in a S3 bucket

s3 delete marker

Last week, we got an incident that some of the data are missing in a versioning enabled S3 bucket. The S3 bucket in question has lifecycle policy enabled, which expires the objects after 90 days (adds delete marker) and permanently delete them after 30 days of becoming previous versions. Reported missing data belong to the … Continue reading Script to delete thousands of delete marker in a S3 bucket