We've made it to the final day, the 12th day of AWS!
Talking of RDBMSs, with all of the Data Lakes, Data Warehousing, NoSQL solutions available on AWS, it is nice to still have access to a good old RDBM, which Aurora provides, with both MySQL and PostgreSQL flavors.
To talk about Redshift we probably need to talk about Data Warehousing, as Redshift is a fully managed Data Warehouse package. This is not putting your data in a ‘digital warehouse’ to go and gather virtual dust on the back of a virtual shelf somewhere and is more akin to a single source of truth for the state of a business and its information. Whilst Redshift is based on PostgreSQL and can be queried with normal SQL, it is not meant to be a DataBase per se.
Extract, Transform & Load (ETL) is the name of the game when it comes to Data Pipelines.
The Extract portion will acquire data from some source(s), which will then pass through Transform where some alteration may be needed to that data, to then end up Loaded into another storage format, such as Redshift, S3, to name a couple.
Despite being what most people think about when mentioning AWS, I left their Elastic Compute Cloud (EC2) until now, mostly because personally I don’t use it all that much.
With the advent of Lambda, EC2 can feel a bit more laborious as there is a lot more set up involved than with serverless solutions.
Speaking of CloudWatch… all things logging belong here. CloudWatch will track metrics on your EC2 instances, SQS, SNS entries and more. You will have graphing of data, and be able to keep track of everything happening on your AWS stack.
On the 7th day of AWS, I would like to mention something that almost stays hidden behind the scenes when using Serverless, but is critical to its use.
CloudFormation allows you to spin up a stack of resources from a config file. It deals with the order of operations, timing and error handling on your behalf and makes it pretty easy to keep on top of all the elements of the stack.
Simple Notification Service (SNS) is another great tool for distributed operations but is different from SQS in various ways including:
Simple Queue Service (SQS) is an indispensable tool for many batch operations and concurrent events within, and even outside of, the AWS infrastructure. There are two types of queues: Standard or FIFO. The standard queue, where messages enter the queue and exit in the most performance effective manner, can be batched into groups of messages and can be used to trigger Lambdas.
AWS Identity and Access Management (IAM) is where you set up users and groups to create and assign roles and permissions to those users. You may want to assign a user with specific access to certain functionality that will be used by a Lambda, or a user login for someone to log into AWS and view logs. All of these things are done here in IAM.
For a while now I have been thinking about writing a blog post about Amazon Web Services (AWS) tools that have become a part of my toolkit recently. However, there are plenty of wonderful in-depth articles about specific parts of AWS and I really only wanted to focus on some of the parts to help demystify the storm of acronyms and code names that can make the AWS environment seem a bit opaque.
