AWS re:Invent 2019 has come and gone, and now the collective audience has to sort through the massive list of AWS announcements released at the event. According to the AWS re:Invent 2019 Recap communication, AWS released 77 products, features and services in just 5 days! Many of the announcements were in the Machine Learning (ML) space (20 total), closely followed by announcements around Compute (16 total), Analytics (6 total), Networking and Content Delivery (5 total), and AWS Partner Network (5 total), amongst others. In the area of ML, things like AWS DeepComposer, Amazon SageMaker Studio, and Amazon Fraud Detector topped the list. While in the Compute, Analytics, and Networking space, Amazon EC2 Inf1 Instances, AWS Local Zones, AWS Outposts, Amazon Redshift Data lake, AWS Transit Gateway Network Manager, and Inter-Region Peering were at the forefront. Here at 2nd Watch we love the cutting-edge ML feature announcements like everyone else, but we always have our eye on those announcements that key-in on what our customers need now – announcements that can have an immediate benefit for our customers in their ongoing cloud journey.
All About the Network
In Matt Lehwess’ presentation, Advanced VPC design and new capabilities for Amazon VPC, he kicked off the discussion with a poignant note of, “Networking is the foundation of everything, it’s how you build things on AWS, you start with an Amazon VPC and build up from there. Networking is really what underpins everything we do in AWS. All the services rely on Networking.” This statement strikes a chord here at 2nd Watch as we have seen that sentiment in action. Over the last couple years, our customers have been accelerating the use of VPCs, and, as of 2018, Amazon VPCs is the number one AWS service used by our customers, with 100% of them using it. We look for that same trend to continue as 2019 comes to an end. It’s not the sexiest part of AWS, but networking provides the foundation that brings all of the other services together. So, focusing on newer and more efficient networking tools and architectures to get services to communicate is always at the top of the list when we look at new announcements. Here are our takes on these key announcements.
One exciting feature announcement in the networking space is Inter-Region Peering for AWS Transit Gateway. This feature allows the ability to establish peering connections between Transit Gateways in different AWS Regions. Previously, connectivity between two Transit Gateways could only be done through a Transit VPC which included the overhead of running your own networking devices as part of the Transit VPC. Inter-Region peering for AWS Transit Gateway enables you to remove the Transit VPC and connect Transit Gateways directly.
The solution uses a new static attachment type called a Transit Gateway Peering Attachment that, once created, requires an acceptance or rejection from the accepter Transit Gateway. In the future, AWS will likely allow dynamic attachments, so they advise you to create unique ASNs for each Transit Gateway for the easiest transition. The solution also uses encrypted VPC peering across the AWS backbone. Currently Transit Gateway inter-region peering support is available for gateways in US East (Virginia), US East (Ohio), US West (Oregon), EU (Ireland), and EU (Frankfurt) AWS Regions with support for other regions coming soon. You also can’t peer Transit Gateways in the same region.
(Source: Matt Lehwess: Advanced VPC design and new capabilities for Amazon VPC (NET305))
On the surface the ability to connect two Transit Gateways is just an incremental additional feature, but when you start to think of the different use cases as well as the follow-on announcement of Multi-Region Transit Gateway peering and Accelerated VPN solutions, the options for architecture really open up. This effectively enables you to create a private and highly-performant global network on top of the AWS backbone. Great stuff!
AWS Transit Gateway Network Manager
This new feature is used to centrally monitor your global network across AWS and on premises. The Transit Gateway network manager simplifies operational complexity of managing networks across regions and remote locations. This AWS feature is another to take a dashboard approach to provide a simpler overview of your resources that may be spread over several regions and accounts. To use it, you create a Global Network within the tool which is an object in the AWS Transit Gateway Network Manager service that represents your private global network in AWS. It includes your AWS Transit Gateway hubs, their attachments, and on-premises devices, sites, and links. Once the Global Network is created, you extend the configuration by adding in Transit Gateways, information about your on-premises devices, sites, links, and the Site-to-Site VPN connections with which they are associated, and start using it to visualize and monitor your network. It includes a nice geographic world map view to visualize VPNs (if they’re up/down impaired) or Transit Gateway Peering connections.
There’s also a nice Topology feature that shows VPCs, VPNs, Direct Connect gateways, and AWS Transit Gateway-AWS Transit Gateway peering for all registered Transit gateways. It provides an easier way to understand your entire global infrastructure from a single view.
Another key feature is the integration with SD-WAN providers like Cisco, Aviatrix, and others. Many of these solutions will integrate with AWS Transit Gateway Network Manager and automate the branch-cloud connectivity and provide end-to-end monitoring of the global network from a single dashboard. It’s something we look forward to exploring with these SD-WAN providers in the future.
AWS Local Zones
AWS Local Zones in an interesting new service that addresses challenges we’ve encountered with customers. Although listed under Compute and not Networking and Content Delivery on the re:Invent 2019 announcement list, Local Zones is a powerful new feature with networking at its core.
Latency tolerance for applications stacks running in a hybrid scenario (i.e. app servers in AWS, database on-prem) is a standard conversation when planning a migration. Historically, those conversations would be predicated by their proximity to an AWS region. Depending on requirements, customers in Portland, Oregon may have the option to run a hybrid application stack, where those in Southern California may have been excluded. The announcement of Local Zones (initially just in Los Angeles) opens up those options to markets that were not previously available. I hope this is the first of many localized resource deployments.
That’s no Region…that’s a Local Zone
Local Zones are interesting in that they only have a subset of the services available in a standard region. Local Zones are organized as a child of a parent region, notably the Los Angeles Local Zone is a child of the Oregon Region. API communication is done through Oregon, and even the name of the LA Local Zone AZ maps to Oregon (Oregon AZ1= us-west-2a, Los Angeles AZ1 = us-west-2-lax-1a). Organizationally, it’s easiest to think of them as remote Availability Zones of existing regions.
As of December 2019, only a limited amount of services are available, including EC2, EBS, FSx, ALB, VPC and single-zone RDS. Pricing seems to be roughly 20% higher than in the parent region. Given that this is the first Local Zone, we don’t know whether this will always be true or if it depends on location. One would assume that Los Angeles would be a higher-cost location whether it was a Local Zone or full region.
All the Things
To see all of the things that were launched at re:Invent 2019 you can check out the re:Invent 2019 Announcement Page. For all AWS announcements, not just re:Invent 2019 launches (e.g. Things that launched just prior to re:Invent), check out the What’s New with AWS webpage. If you missed the show completely or just want to re-watch your favorite AWS presenters, you can see many of the re:Invent presentations on the AWS Events Youtube Channel. After you’ve done all that research and watched all those videos and are ready to get started, you can always reach out to us at 2nd Watch. We’d love to help!
AWS re:Invent always presents us with a cornucopia of new cloud capabilities to build with and be inspired by, so listing just a few of the top takeaways can be a real challenge.
There are the announcements that I would classify as “this is cool, I can’t wait to hack on this,” which for me, a MIDI-aficionado and ML-wannabe, would include DeepComposer. Then there are other announcements that fall in the “good to know in case I ever need it” bucket such as AWS LocalZones. And finally, there are those that jump out at us because “our clients have been asking for this, hallelujah!”
I’m going to prioritize this list based on the latter group to start, but check back in a few months because, if my DeepComposer synthpop track drops on SoundCloud, I might want to revisit these rankings.
“AWS Compute Optimizer uses machine learning techniques to analyze the history of resource consumption on your account and make well-articulated and actionable recommendations tailored to your resource usage.”
Our options for EC2 instance types continues to evolve and grow over time. These evolutions address optimizations for specialized workloads (e.g., the new Inf1 instances), which means better performance-to-cost for those types of workloads.
The challenge for 2nd Watch clients (and everyone else in the Cloud) is maintaining an up-to-date knowledge of the options available and continually applying the best instance types to the needs of their workloads on an ongoing basis. That is a lot of information to keep up on, understand, and manage, and you’re probably wondering, “how do other companies deal with this?”
The ones managing it best have tools (such as CloudHealth) to help, but cost optimization is an area that requires continual attention and experience to yield the best results. Where AWS Compute Optimizer will immediately add value is surfacing inefficiencies at zero cost of 3rd party tools to get started. You will need to have the CloudWatch agent installed to gather OS-level metrics for the best results, but this is a standard requirement for these types of tools. What remains to be seen in the coming months is how Compute Optimizer compares to the commercial 3rd party tools on the market in terms of uncovering overall savings opportunities. However, the obvious advantage for 3rd party tools remaining unaffected by this change will be their ability to optimize across multiple cloud service providers.
“Amazon Elastic Container Service (ECS) now supports Windows group Managed Service Account (gMSA), a new capability that allows ECS customers to authenticate and authorize their Windows containers with network resources using an Active Directory (AD). Customers can now easily use Integrated Windows Authentication with their Windows containers on ECS to secure services.”
This announcement was not part of any keynote, but thanks to fellow 2nd Watcher and Principal Cloud Consultant, Joey Yore, bringing it to my attention, it is definitely making my list. Over the course of the past year, several of our clients on a container adoption path for their .NET workloads were stymied by this very lack of Windows gMSA support.
Drivers for migrating these .NET apps from EC2 to containers includes easier blue/green deployments for faster time-to-market, simplified operations by minimizing overall Windows footprint to monitor and manage, and cost savings also associated with the consolidated Windows estate. The challenge encountered was with the authentication for these Windows apps, as without the gMSA feature, the applications would require a time-intensive refactor or leverage an EC2 based solution with management overhead. This raised questions about the commitment of AWS to Windows containers in the long term, and thankfully, with this release, it signals that Windows is not being sidelined.
#3 AWS Security Hub Gets Smarter
Third on the list is a 2-for-1 special because security and compliance is one of the most common areas our clients have come to us for help. Cloud gives builders all of the tools they need to build and run secure applications, but defining controls and ensuring their continual enforcement requires consistent and deliberate work. In response to this need we’ve seen AWS releasing more services that streamline activities for security operations teams. In that list of tools are Amazon GuardDuty, Amazon Macie, and, more recently, AWS Security Hub, which these two selections integrate with:
“AWS IAM Access Analyzer generates comprehensive findings that identify resources that can be accessed from outside an AWS account. AWS IAM Access Analyzer does this by evaluating resource policies using mathematical logic and inference to determine the possible access paths allowed by the policies. AWS IAM Access Analyzer continuously monitors for new or updated policies, and it analyzes permissions granted using policies for their Amazon S3 buckets, AWS KMS keys, Amazon SQS queues, AWS IAM roles, and AWS Lambda functions.”
If you’ve worked with IAM, you know that without deliberate design and planning, it can become an unwieldy mess quickly. Disorganization with your IAM policies means you run the risk of creating inadvertent security holes in your infrastructure, which might not be immediately apparent. This new feature to AWS Security Hub streamlines the process for surfacing those latent IAM issues that may have otherwise gone unnoticed.
“Amazon Detective is a new service in Preview that makes it easy to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. Amazon Detective automatically collects log data from your AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct faster and more efficient security investigations.”
The result of Amazon’s acquisition of Sqrrl in 2018, Amazon Detective is another handy tool that helps separate the signal from the noise in the cacophony of cloud event data generated across accounts. What’s different about this service as compared to others like GuardDuty is that it builds relationship graphs which can be used to rapidly identify links (edges) between events (nodes). This is a powerful capability to have when investigating security events and the possible impact across your Cloud portfolio.
“EC2 Image Builder is a service that makes it easier and faster to build and maintain secure images. Image Builder simplifies the creation, patching, testing, distribution, and sharing of Linux or Windows Server images.”
2nd Watch clients have needed an automated solution to “bake” consistent machine images for years, and our “Machine Image Factory” solution accelerator was developed to efficiently address the need using tools such as Hashicorp Packer, AWS CodeBuild, and AWS CodePipeline.
The reason this solution has been so popular is that by having your own library of images customized to your organizations requirements (eg, security configurations, operations tooling, patching), you can release applications faster, with greater consistency, and without burdening your teams’ time or focus watching installation progress bars when they can be working on higher business value activities.
What’s great about AWS releasing this capability as a native service offering is that it is making a best-practice pattern even more accessible to organizations without confusing the business outcome with an array of underlying tools being brought together to make it happen. If your team wants to get started with EC2 Image Builder but you need help with understanding how to get from your current “hand crafted” images to Image Builder’s recipes and tests, we can help!
“AWS Outposts is a fully managed service that extends AWS infrastructure, AWS services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility for a truly consistent hybrid experience. AWS Outposts is ideal for workloads that need low latency access to on-premises applications or systems, local data processing, and to securely store sensitive customer data that needs to remain anywhere there is no AWS region, including inside company-controlled environments or countries.”
It’s 2019, and plants are now meat and AWS is hardware you can install in your datacenter. I will leave it to you to guess which topic has been more hotly debated on the 2nd Watch Slack, but amongst our clients, Outposts has made its way into many conversations since its announcement at re:Invent 2018. Coming out of last week’s announcement of Outposts GA, I think we will be seeing a lot more of this service in 2020.
One of the reasons I hear clients inquiring about Outposts is that it fills a gap for workloads with proximity or latency requirements to manufacturing plants or another type of strategic regional facility. This “hyper-local” need echoes the announcement for AWS Local Zones, which presents a footprint for AWS cloud resources targeting a specific geography (Los Angeles, CA initially).
Of course, regional datacenters and other hyperconverged platforms exist to run these types of workloads already, but what is so powerful about Outposts is that it brings the Cloud operations model back to the datacenter and the same cloud skills that your teams have developed and hired for don’t need to be stunted to learn a disparate set of skills on a niche hardware vendor platform that could be irrelevant 3 years from now.
I’m excited to see how these picks and all of the new services announced play out over the next year. There is a lot here for businesses to implement in their environments to drive down costs, improve visibility and security, and dial in performance for their differentiating workloads.
Head over to our Twitter account, @2ndWatch, if you think there should be others included in our top 5 list. We’d love to get your take!
With a week full of sessions, bootcamps and extra-curriculars at AWS re:Invent, you might not have had time to make it to our breakout session.
Watch ‘The Simple Path to AMS’ On-Demand
Learn how to accelerate your journey to the cloud by using AWS Managed Services (AMS), including the process for assessing, migrating and operationalizing your infrastructure from your on-premise datacenter or existing cloud environment to AMS. Discover key steps to streamline this process using automation and infrastructure as code to set up network connectivity, access management, logging, monitoring, backups and configuration as well as integration points for an existing managed service provider to seamlessly work with AMS.
Thursday marked the last full day of AWS re:Invent 2019 and the morning after another outstanding 2nd Watch party. If you attended, it is understandable if you were unable to make Werner Vogels’ keynote address. Have no fear, 2nd Watch’s Victoria Geronimo has recapped all the highlights for you in her blog post, or you can watch it here. This year, Vogels focused more on how AWS builds to support microservices instead of on new announcements. As usual, his t-shirt choice was a huge topic of conversation.
It has been another great week here in Vegas, and again I am amazed at all the new and interesting people we get to talk to during this conference. It is truly a global experience getting to talk to people from all over the world and some AWS Heroes. I hope we got a chance to meet you at the 2nd Watch booth. If you needed some relaxation time, AWS provided plenty of areas and opportunities to play including Broomball, Dodgeball and the final party, re:Play, which featured Anderson Paak, as well as A-Trak, Jamestown Revival, Jen Lasher, Miya Folick, and STS9.
A few of the interesting announcements on Thursday included:
Amazon Fraud Detector, a fully managed service that makes it easy to identify potentially fraudulent online activities such as online payment fraud and the creation of fake accounts
UltraWarm, a fully managed, low-cost, warm storage tier for Amazon Elasticsearch Service that takes a new approach to providing hot-warm tiering in Amazon Elasticsearch Service, offering up to 900TB of storage at almost a 90% cost reduction over existing options
Advanced Query Accelerator (AQUA) for Amazon Redshift is a new distributed and hardware-accelerated cache that enables Redshift to run up to 10x faster than any other cloud data warehouse
As usual, the announcements this week show that AWS continues to listen to its customers and release services to fill those needs. There are still sessions going on today and thousands heading to the airport. Travel safe and see everyone next year November 30 – December 4, 2020 in Las Vegas.
AWS re:Invent kicked off Day 4 with its third and final keynote speech, delivered by Amazon.com’s CTO, Dr. Werner Vogels. 2nd Watch was up early with the birds and excited to hear about the architectural investments AWS is making this year.
The name of the game of this keynote was micro. From Nitro to Fargate to use cases at Vanguard, AWS had microservices up on its moodboard because it allows for more efficient services, greater customization, and speedier deployments.
Vogels began the keynote on Nitro System, AWS’ innovative virtualization platform that will be supporting core services. Hypervisors are integral to the cloud and offer multiple functionalities – protecting hardware, virtualizing storage and CPU, etc. Vogels described them as “monolithic,” which translates to fewer customization options. Inspired by microservices, Nitro modularizes each of these functionalities and offloads them onto hardware and software. By breaking each of these capabilities apart, we can tailor each and build a more efficient and less costly platform for our AWS services. Nitro is at the heart of many of AWS’ newer initiatives, including Outposts, VMware, Bare Metal, and Live Updates.
Vogels also gifted us security nerds with good news – Nitro encrypts everything out of the box and passes on Domain 0 (aka Dom0). Dom0 is a critical functionality of Xen hypervisors and has special privileges allowing it to access hardware. This means that if someone can exploit Dom0, they have direct access to the hardware. In fact, Nitro prohibits admin access entirely.
Clare Liguori, Principal Software Engineer for AWS, took the stage next to reinforce re:Invent’s earlier announcement that Fargate will be supporting EKS. With a clever trivia game demonstration, Clare showed how Fargate allows for better capacity and efficiently isolates containers. She also spent time on Firecracker – a lightweight micro-VMs for containers and serverless applications. She even threw the repo on screen for us! https://github.com/firecracker-microvm/firecracker.
Next, Jeff Dowds, IT Executive from Vanguard, presented a compelling story of how Vanguard developed a Cloud Construction Team to adopt a serverless environment, quickly get a cloud initiative in practice, and reduce overall computing costs.
Reiterating the micro theme, Vogels came back on to discuss the theory behind their cell-based architecture, which enables evolvable architecture and reduces blast radius. Things fail all the time, and if it’s a cell rather than a monolith, it’s harder to bring down the whole architecture. Vogels then discussed Amazon Builder’s Library, which empowers users with write ups on how Amazon architects its own resources in AWS – truly a great resource for anyone building on AWS.
Next up, Sebastian de Halleax, COO of Saildrone, presented how they use AWS to innovate in new ways on the unexplored territory of oceans (wow!). Their Ocean Surveyor is mapping the sea floor and sending data in realtime using AWS. They’ll use this innovation to see how the sea affects humans and vice versa. It’s pretty great to see how AWS is leveraged to solve real world issues outside of the IT realm.
Next up was Dr. Martin Hofmann, Group CIO of Volkswagen AG, which is leveraging AWS Autonomous Factories to innovate their supply chain management. Vogels noted how most equipment is old and outdated at manufacturing plants. Autonomous Factories – used at AWS warehouse and at Volkswagen – envision automating not only manufacturing but repairing of machines. Think Wall-E!
Wednesday is the heart of the AWS re:Invent conference. Between Keynotes from Andy Jassy on Tuesday and Werner Vogels on Thursday was a keynote that many did not pay attend to, as it was partner focused. The Global Partner Summit Keynote by Doug Yeum, head of AWS worldwide channels and alliances, was where the partner community learned about cloud-powered innovation and the opportunity that it creates for partners. There were many examples of the huge partner ecosystem and how it impacts companies in their cloud journey. One thing is obvious at this conference, the number of services that AWS is delivering provides a basis for innovation in the partner community. You can read more about the Keynote in a blog post by Evan Luchese Leon here. It’s obvious that the partner community is strong, growing and supported by AWS with the intent of continuing to drive innovation and support for everyone with new products and supporting services.
Stepping back for a moment, it’s amazing to look at the numbers. Over 65,000 people were attending move than 1,100 sessions during the day. It was also apparent that the shear amount of excitement and walking required during the day is starting to wear on people as their step counters are telling them just how much they are moving during the day. And don’t worry if you missed a few of the sessions this week. Just head over to YouTube: https://reinventvideos.com/, filter for 2019 and enjoy them on-demand!
A few of the interesting announcements on Wednesday included:
There are sure to be some really interesting announcements from Werner Vogels during his keynote today.
If you are still cruising the expo for the last hour of the show, don’t forget to stop by the 2nd Watch booth, meet some of the team and lets discuss what you are seeing as well as your challenges and opportunities for the future in AWS.