By John Fahl, IOD Expert
Once you work with AWS long enough, you realize it changes all the time.
Some things that were difficult last year are now easy.
Who remembers ELB IPs moving on you? Normally, when you use a CNAME (like you’re supposed to) it doesn’t matter, but I’ve moved a few apps that used hardcoded IPs in old applications. Now, you can just abandon the legacy ELB and use their NLB for that issue. NLB and ALB were big improvements over traditional ELB.
I remember researching for a few weeks for a solution to encrypt my EBS volumes. Just when I was about to decide upon a solution, AWS announced the ability to encrypt EBS volumes for free at the hardware level!
Even experienced AWS users stumble when changes are announced, but any seasoned pro finds work-arounds. And when we can’t figure something out on our own, we turn to our virtual colleagues. With that in mind, here are a few of my AWS tricks.
1. Dedicated Tenancy: Be Careful What You Wish for
When you are creating a new VPC, you may wonder if you need dedicated tenancy. You may not be sure if your PCI application requires it, but better to be safe, right? There are cost and design implications by picking dedicated tenancy, detailed here. Be completely sure before you pick dedicated tenancy because you cannot change it.
This is the screen innocently staring you in the face as you make such a subtle decision:
Depending on what AWS offerings and Instance Types you want to utilize in your environment, it won’t be available. If you’re a year into this environment, you will be kicking yourself (like I have). You won’t be able to use T2 series hardware, Elasticache and other features…
WAIT A SECOND
Now you can change it!
IOD is a content creation and research company working with some of the top names in IT. Our philosophy is experts are not writers and writers are not experts, so we pair tech experts with experienced editors to produce high quality, deeply technical content. The author of this post is one of our top experts. You can be too! JOIN US.
2. Share that Amazon EBS Key? Nah
If your AWS environment spans over regions or different accounts, you will run into challenges encrypting volumes or Amazon Machine Images (AMIs). Amazon will allow you to share the keys. Here is another trick that is more secure, but you are still sharing them — why do that? Why take the risk in exposing a production key in a development account? Why risk sharing a key with a customer and have it compromised? By default, you can share unencrypted AMIs to different accounts and copy them to different regions (in the same account).
What good does copying an UNENCRYPTED AMI do if you want encrypted AMIs, you ask? It reduces the risk for key compromise and you still can get your encrypted AMI.
If you are like me, you want AMIs that are predictable and, in essence, identical across all accounts. If I build a packer image in account 1 that is fully patched, hardened, and has a few services, I want it the same everywhere. I use Jenkins to do this as one job, but you can do this through a number of tools or even Lambda.
This assumes you have a custom or default KMS key per account/region.
- Create your AMI in an account, share it to other accounts, and copy it to other regions. You will now have an unencrypted yet identical image across your environment.
- Copy the AMI and then encrypt it with your local key of choice.
Now you have an encrypted AMI with your region/account specific KMS key without sharing the keys from your source account.
Are you an AWS expert? Have a few tricks of your own? JOIN US.
3. AWS Pen/Vuln Test Trick
Do you ever request for penetration or vulnerability testing from AWS? I do. A lot. If you are working in a cloud environment that deals with regulatory accreditations such as HIPAA, SOC, PCI DSS, ISO, and the lot, you’ve had to appease the auditing deities with your vulnerability reports. AWS will let you scan your resources, but you have to request permission to do so.
Simple enough, right? Well, when you get to this section of the pen test request:
You will learn that you need to provide the IP address and InstanceID of each resource you want to scan. If you have 20 servers, this isn’t too bad. You can open the AWS Console and copy/paste the data in, but what if it is a thousand servers, or five thousand? You might want to use a script to make your life better rather than cussing out your security manager.
Using a script makes this pretty easy. Between the AWS CLI (or SDK) and jq, you can grab the data from AWS, parse and format it, then copy it into the AWS form.
This assumes you have an AWS IAM key pair that allows read access to EC2 from a linux bash:
aws ec2 describe-instances –region=us-east-1 | jq ‘.”Reservations”[].”Instances”[] | .PrivateIpAddress + ” ” + .InstanceType’ | sed s/\”//g
This should return a report like so:
10.89.2.255 i-73ca2fe2
10.89.13.124 i-063d67af5a491a125
10.45.36.4 i-0f98ff1d29c619391
10.45.31.12 i-043243abc0f6fdd69
…
4. Adding a Custom Certificate to IAM
You’ve gone out to a trusted CA and bought a wildcard certificate for your AWS servers and now you want to install it into AWS so that your ELB can make use of it. But where is that option in the console?! I don’t see it! Surely they haven’t overlooked the capability to do this … after all, they provide the Amazon Certificate Manager to create certs (Openssl CA in a pretty bow).
You won’t find a section in the AWS Console to upload your cert. You have to upload it when you’re creating an Amazon Elastic Load Balancer (ELB). I find this a bit cumbersome and use the CLI to do this work, especially if I’m uploading many certs.
After purchasing your certificate from a trusted CA, or using an internal CA, use the CLI to upload it:
aws iam upload-server-certificate –server-certificate-name wildcard.iod.com –certificate-body file://wildcard.iod.com.pem –private-key file://wildcard.iod.com.key –certificate-chain file://trustedCA.pem
Now you can list your new certs with:
aws iam list-server-certificates
You will get back all certs managed by AWS:
{
“ServerCertificateMetadataList”: [
{
“ServerCertificateId”: “ASCAIQESO5RTXV6TTIDL4”,
“ServerCertificateName”: “wildcard.iod.com”,
“Expiration”: “2018-04-19T12:00:00Z”,
“Path”: “/”,
“Arn”: “arn:aws:iam::366389857854:server-certificate/wildcard.iod.com”,
“UploadDate”: “2017-08-09T17:35:59Z”
}
}
5. Kicking the NIC
In a perfect DevOps world, all your instances would line up like cattle. It’s not online and cooperating? Terminate it on sight. But for those of you who have long living instances or perhaps lift and shifted from a legacy datacenter environment. Maybe you have old servers you moved to AWS and you need them to live a few years longer. I’ve heard all the reasons.
You get a notice from CloudWatch that a critical instance is failing Health Checks. You log into AWS and see the dreaded 1/2. Now what? Sometimes it will be complete failure, for which I have no remedy other than a trusty backup (see CPM).
Before you terminate that failed instance, try to revive it by “kicking the NIC.” Simply follow this procedure to shock your instance back into prod.
- Create a new Elastic Network Interface (ENI) in AWS.
- Make sure it is in the same subnet and Availability Zone as your troubled Instance.
- Make sure it is using the same Security Group.
- Attach it to the trouble instance (note the new ENI IP address)
- Try logging in to the new ENI IP. If you are successful:
- In Windows, open ncpa.cpl (network interfaces) and disable, then re-enable the primary network interface-this will fix the issue logging into it
- In Linux, sudo ifconfig eth0 down, then ifconfig eth0 up (or the interface that is failed)
- Log out and try logging in on the original IP, if it works, you’re good, detach the new ENI and destroy it.
On a weekly basis, IOD publishes original top tech content written by one of our experts. We hire and pay freelancers from all over the world to research and write about new technologies.
Summary
This list is not exhaustive, nor does it contain hacks outside of what should be possible. It is simply an attempt to make your life easier when it comes to tasks that can be painful in AWS. There are more tools in my toolbelt of cloud wrangling. If you enjoyed these tricks, check out my next blog discussing what I learned migrating an ancient data center to the cloud. Stay tuned!
= = =
John Fahl is a Devops Manager and Fantasy Fiction writer who lives in Philadelphia, PA. After serving in the Navy, he’s built his portfolio working for major defense contractors, the federal government, and reputable start-ups.
