Mind the Gap! The Leap from Legacy to Modern Applications 

Legacy to Modern Applications | modernizing legacy applications

Most businesses today have evaluated their options for application modernization. Planned movement to the cloud happened ahead of schedule, driven by the need for rapid scalability and agility in the wake of COVID-19.

Legacy applications already rehosted or replatformed in the cloud saw increased load, highlighting painful inefficiencies in scalability and sometimes even causing outages. Your business has likely already taken some first steps in app modernization and updating legacy systems. 

Of the seven options to modernize with legacy systems outlined by Gartner, 2nd Watch commonly works with clients who have already successfully rehosted and replatformed applications. To a lesser extent, we see mainframe applications encapsulated in a modern RESTful API or replaced altogether. Businesses frequently take those first steps in their digital transformation but find themselves stuck crossing the gap to a fully modern application. 

What are common issues and solutions businesses face as they move away from outdated technologies and progress towards fully modern applications? 

Keeping the Goal in Mind 

Overcoming the inertia to begin a modernization project is often a lengthy process, requiring several months or as much as a year or more to complete the first phases. Development teams require training, thorough and careful planning must occur, and unforeseen challenges are encountered and overcome. Through it all, the needs of the business never slow down, and the temptation to halt or dramatically slow legacy modernization efforts after the initial phases of modernization can be substantial. 

No matter what the end state of the modernization journey looks like, it can be helpful to keep it at the forefront of the development team’s minds. In today’s remote and hybrid working environment, that’s not as easy as keeping a whiteboard or poster in a room. Sprint ceremonies should include a brief reminder of long-term business goals, especially for backlog or sprint reviews. Keep the team invested in the business and technical reasons and the question “why modernize legacy applications” at the forefront of their minds. Most importantly, solicit their feedback on the process required to accomplish the long-term strategic goals of the business. 

With the goal firmly in your development team’s minds, it’s time to tackle tactics in migrating from legacy apps to newer systems. What are some of these common stumbling blocks on the road to refactoring and rearchitecting legacy software? 

(Related article: Rehost vs Refactor vs Replatform | AppMod Essentials) 

Refactoring 

Refactoring an application can encompass a broad set of areas. Refactoring is sometimes as straightforward as reducing technical debt, or it can be as complex as breaking apart a monolithic application into smaller services. In 2nd Watch’s experience, some common issues when refactoring running applications include: 

  • Limited knowledge of cloud-based architectural patterns.
    Even common architectures like 2- and 3-tier applications require some legacy code changes when an application has moved from a data center to a cloud service provider or among cloud service providers. Where an older application may have hardcoded IP addresses or DNS, a modern approach to accessing application tiers would use environment variables configured at runtime, pointing at load balancers. 
  • Lack of telemetry and observability.
    Development teams are frequently hesitant to make changes quickly because there are too many unknowns in their application. Proper monitoring of known unknowns (metrics) and unknown unknowns (observability) can demystify the impact of refactoring. For more context around the types of unknowns and how to work with them in an application, Charity Majors frequently writes on the topic. 
  • Lack of thorough automated tests.
    A lack of automated tests also slows the ability to make changes because developers cannot anticipate what their changes might break. Improved telemetry and observability can help, but automated testing is the other side of the equation. Tools like Codecov can initially help improve test coverage, but unless carefully attended, incentivizing a percentage of test coverage across the codebase can lead to tests that do not thoroughly cover all common use cases. Good unit tests and integration testing can halt problems before they even start. 
  • No blueprint for optimal refactoring.
    Without a clear blueprint for understanding what an optimally refactored app looks like, development and information technology teams can become frustrated or unclear about their end goals. Heroku’s Twelve-Factor App methodology is one commonly used framework for crafting or refactoring modern applications. It has the added benefit of being applicable to many deployment models – single- or multiple-server, containers, or serverless. 

Rearchitecting

Rearchitecting an application to leverage better capabilities, such as those found in a cloud service provider’s Platform-as-a-Service (PaaS) or Software-as-a-Service (SaaS) options, may present some challenges. The most common challenge 2nd Watch encounters with clients is not fully understanding the options available in modern environments. Older applications are the product of their time and typically were built optimally for the available technology and needs. However, when rearchitecting those applications, sometimes development teams either don’t know or don’t have details about better options that may be available. 

Running a MySQL database on the same machine as the rest of the monolithic application may have made sense when initially writing the application. Today, many applications can run more cheaply, more securely, and with the same or better performance using a combination of cloud storage buckets, managed caches like Redis or Memcached, and secrets managers. These consumption-based cloud options tend to be significantly cheaper than managed databases or databases running on cloud virtual machines. Scaling automatically with end-user demand and reduced management overhead are additional benefits of software modernization. 

Rearchitecting an application can also be frustrating for experienced systems administrators tasked with maintaining and troubleshooting production applications. For example, moving from VMs to containers introduces an entirely different way of dealing with logs. Sysadmins must forward them to a log aggregator instead of storing them on disk. Autoscaling a service can mean the difference between identifying which instances – of potentially dozens or hundreds – had an issue instead of a small handful of them. Application modernization impacts every person involved with the long-term success of that application, not just developers and end-users. 

Conclusion 

Application Modernization is a long-term strategic activity, not a short-term tactical activity. Over time, you will realize the benefits of the lower total cost of ownership (TCO), increased agility, and faster time to market. Recognizing and committing to the future of your business will help you overcome the short- and mid-term challenges of app modernization. 

Engaging a trusted partner to accelerate your app modernization journey and lead the charge across that gap is a powerful strategy to overcome some of the highlighted problems. It can be difficult to overcome a challenge with the same mindset that led to creating that challenge. An influx of different ideas and experiences can be the push development teams need to reach the next level for a business. 

If you’re wondering how to modernize legacy applications and are ready to work with a trusted advisor that can help you cross that gap, 2nd Watch will meet you wherever you are in your journey. Contact us to schedule a discussion of your goals, challenges, and how we can help you reach the end game of modern business applications. 

Michael Gray, 2nd Watch Senior Cloud Consultant 


Why You Need to Modernize Your Media Supply Chain

The demand for direct-to-consumer services and media content is continuously growing, and with that, audiences are raising their expectations of media and entertainment companies. Agile and innovative companies, such as Netflix, YouTube, and Amazon Prime, have arguably created and continue to enable the current viewership trends.

These streaming services have disrupted the traditional media landscape by empowering audiences to watch any content wherever and whenever they want. To accommodate new audience behaviors, relevant media companies use technologies to support the modern-day digital media supply chain, which has become increasingly complex to manage.

However, legacy media companies have something that audiences still want: content. Most of these institutions have massive budgets for content production and enormous existing media libraries that have latent revenue potential. For example, legacy media brands own nostalgic cult classics, like “The Office,” that viewers will always want to watch, even though they have watched these episodes multiple times before.

As the volume of content consumption and demand increases, media organizations will find that a traditional media supply chain will constrain their ability to grow and meet customers in their preferred venues, despite owning a broad range of content that viewers want to watch. In order to keep up with audience demand, media companies will need to transform their media supply chains, so that they can distribute their media quickly and at scale, or they risk falling behind. Cloud technologies are the key to modernizing digital asset management, metadata models, quality control, and content delivery networks.

The Challenges of a Traditional Media Supply Chain

There are a lot of moving parts and behind-the-scenes work for media and entertainment businesses to push media assets to audiences. The media supply chain is the process used to create, manage, and deliver digital media from the point of origin (creator, content provider, content owner, etc.) to the destination (the audience.) For the right content and best experience to reach users on devices and platforms of their choice, digital media files must pass through various stages of processing and different workflows.

Media supply chain management is challenging and if there are inefficiencies within this  process, issues that will ultimately affect the bottom line will crop up. The following are top challenges of media supply chain management:

Decentralized Assets

The content wars are in full swing, and as a result, the media and entertainment industry has seen an influx of divestitures, mergers, and acquisitions. Organizations are accumulating as much content as possible by bolstering their media production with media acquisition, but as a result, content management has become more difficult. With more content comes more problems because this introduces more siloed third-party partners. As companies merge, the asset management system becomes decentralized, and media files and metadata are spread across different storage arrays in different datacenters that are managed by different MAMs with various metadata repositories.

Reliance on Manual Processes

Legacy media companies have been around much longer than modern technologies. As a result, some of these organizations still do many media production and distribution tasks manually, especially when it comes to generating, reviewing, and approving metadata. Metadata is essential for sorting, categorizing, routing, and archiving media content, as well as making the content accessible to a global, diverse audience. Using manual processes for these functions not only severely slows down a business, but they are also susceptible to human-error.

Quality of Media Assets

Today, consumers have the latest technology (4K TVs, surround sound systems, etc.), which requires the highest quality version of content sources. With dispersed content libraries and team, working derivative edits to meet localization and licensing requirements and locating native frame rate masters can be a challenging and time-consuming problem to tackle.

Benefits of Using Cloud Technology to Modernize the Media Supply Chain

Cloud-based technologies can help manage and resolve the issues typically encountered in a media supply chain. If media organizations do not utilize cloud solutions to modernize their supply chain, they risk being less agile to meet global audience demand, incurring higher costs to deliver media, and eroding viewership.

Legacy media brands are recognizing the consequences of not adopting modern technology to support their media supply chains, and recently, we’ve seen established media corporations partnering with cloud service providers to undertake a digital transformation. A recent and newsworthy example of this is the MGM and AWS partnership. MGM owns a deep library of film and television content, and by leveraging AWS, MGM is able to distribute this content with flexibility, scalability, reliability, and security to their audiences. AWS offers services and tools to modernize MGM’s media supply chain to be able to distribute content across multiple platforms quickly and at scale.

Businesses don’t need to strike historic deals with cloud service providers to receive the same benefits. By transforming into a cloud-based framework, any media company can reap the following major benefits of modernizing their media supply chain:

Scale and Agility

This point cannot be repeated enough because, again, customer media consumption is rapidly increasing, and businesses must find a way to meet those demands in order to retain customers and remain competitive. With cloud computing, the media supply chain is no longer limited to the capacity of on-premise data centers or the capital expenditure budget that was forecasted a year earlier. Using cloud technology allows organizations to be dynamic and flexible to adjust for growing demand. Businesses can easily scale services up (or even down) based on audience demands by simply adding (or removing) more cloud resources, which is easier and more forgiving than having to add more infrastructure or being stuck with wasted databases.

Cost Effective

Cloud services employ pay-as-you-go billing, which allows companies to pay for what they use rather than paying a fixed cost that may not fit their needs later on down the road. Most importantly, using the cloud removes the maintenance and operational costs associated with  maintaining data center footprints. The costs of server hardware, power consumption, and space for traditional data centers can really add up, especially because these costs are inflexible based on actual consumption. Utilizing cloud technology provides flexibility in billing and trims down maintenance costs.

Automation and Efficiency

Cloud services offer tools that can handle abstract operational complexities, like metadata management, that were historically done manually. These automation and AI features can dramatically reduce the need to manually generate this metadata because it implements machine learning and video, audio, and image recognition to largely automate the generation, review, and approval of metadata. Harnessing the power of automation frees up teams’ resources and time and redirects that energy on impactful, business-differentiating activities.

Data-Driven Decisions

Large audiences also means large amounts of data. Massive volumes of both structured and unstructured data requires increased processing power, storage, and more. Cloud computing has the scalable infrastructure to rapidly manage huge spikes of real time traffic or usage. Moreover, cloud service providers offer a variety of analytic tools that enable extract, transform, and loading of enormous datasets to provide meaningful insights quickly. Media companies can harness this data to improve user experiences and optimize supply chains, all of which greatly affects their bottom line.

How do I Get Started in my Media Supply Chain Transformation?

The process is less daunting than you think, and there are experienced cloud advisors and consulting firms who can point you in the right direction. At 2nd Watch, we embrace your unique modernization journey to help transform and modernize your business and achieve true business growth through cloud adoption. To learn more about our media cloud services, visit our Media and Entertainment page or talk to someone directly through our Contact Us page.


Riding the Digital Transformation: Why Enterprises Are Reengineering Their Cloud Infrastructure

Post 2020, how are you approaching the cloud? The rapid and unexpected digital transformation of 2020 forced enterprises worldwide to quickly mobilize workers using cloud resources. Now, as the world returns to an altered normal, it’s time for organizations to revisit their cloud infrastructure components with a fresh perspective. Hybrid work environments, industry transformations, changing consumer behavior, and growing cyber threats have all effected the way we do business. Now it might be time to change your cloud.

Risk mitigation at scale

Avoiding potential missteps in your strategy requires both wide and narrow insights. With the right cloud computing infrastructure, network equipment, and operating systems, organizations can achieve better risk mitigation and management with cloud scalability. As you continue to pursue business outcomes, you have to solve existing problems, as well as plan for the future. Some of these problems include:

  • Scaling your cloud platform and infrastructure services quickly to keep up with increasing and/or unexpected demand.
  • Maximizing cloud computing services and computing power to accommodate storage, speed, and resource demands.
  • Prioritizing new and necessary investments and delivery models within a fixed budget.
  • Innovating faster to remain, or gain, competitive advantage.

Overall, to avoid risk, you need to gain efficiency, and that’s what the cloud can do. Cloud infrastructure, applications, and Software as a Service (SaaS) solutions are designed to decrease input, and increase output and effectiveness. The scalability of cloud services allows enterprises to continue growing and innovating, without requiring heavy investments. With continuous cloud optimization, you’re positioned to adapt, innovate, and succeed regardless of the unknown future.

Application modernization for data leverage

Much of the digital transformation started with infrastructure modernization and the development of IaaS as a base line. Now, application modernization is accelerating alongside a changing migration pattern. What used to be simply ‘lift and shift’ is now ‘lift and evolve.’ Enterprises want to collaborate with cloud experts to gain a deeper understanding of applications as they become more cloud native. With a constant pipeline of new applications and services, organizations need guidance to avoid cloud cost sprawl and streamline environment integration.

As application modernization continues, organizations are gaining access to massive amounts of data that are enabling brand new opportunities. This requires a new look at database architectures to make sure you’re unlocking value internally and potentially, externally. While application modernization and database architecture are interconnected, they can also transform separately. We’re starting to see people recognize the importance of strategic cloud transformations that include the entire data footprint – whether it’s the underlying architecture, or the top level analytics.

Organizations are getting out of long-term licensing agreements, monetizing their data, gaining flexibility, cutting costs, and driving innovation, customer value, and revenue. Data is pulled from, and fed into, a lot of different applications within constantly changing cloud environments, which brings both challenges and opportunities. Enterprises must transform from this to that, but the end goal is constantly changing as well. Therefore continuous motion is necessary within the digital transformation.

Changing core business strategies

One thing is for sure about the digital transformation – it’s not slowing down. Most experts agree that even after pandemic safety precautions are eliminated, the digital transformation will continue to accelerate. After seeing the speed of adoption and opportunities in the cloud, many enterprises are reevaluating the future with new eyes. Budgets for IT are expanding, but so is the IT skills gap and cybersecurity incidents. These transitions present questions in a new light, and enterprises should revisit their answers.

  • Why do you still have your own physical data center?
  • What is the value in outsourcing? And insourcing?
  • How has your risk profile changed?
  • How does data allow you to focus on your core business strategy?

Answering these questions has more enterprises looking to partner with, and learn from, cloud experts – as opposed to just receiving services. Organizations want and need to work alongside cloud partners to close the skills gap within their enterprise, gain skills for internal expansion in the future, and better understand how virtualized resources can improve their business. It’s also a way to invest in your employees to reduce turn-over and encourage long-term loyalty.

Security and compliance

At this point with security, compliance, and ensuring business continuity, enterprises must have solutions in place. There is no other way. Ransomware and phishing attacks have been rising in sophistication and frequency year-over-year, with a noticeable spike since remote work became mainstream. Not only does your internal team need constant training and regular enforcement of governance policies, but there’s a larger emphasis on how your network protections are set up.

Regardless of automation and controls, people will make mistakes and there is an inherent risk in any human activity. In fact, human error is the leading cause of data loss with approximately 88% of all data breaches caused by an employee mistake. Unfortunately, the possibility of a breaches is often made possible because of your internal team. Typically, it’s the manner in which the cloud is configured or architected that creates a loophole for bad actors. It’s not that the public cloud isn’t secure or compliant, it’s that it’s not set up properly. This is where many enterprises are outsourcing data protection to avoid damaging compliance penalties, guarantee uninterrupted business continuity, and maintain the security of sensitive data after malicious or accidental deletion, natural disaster, or in the event that a device is lost, stolen or damaged.

Next steps: Think about day two

Enterprises who think of cloud migration as a one-and-done project – we were there, and now we’re here – aren’t ready to make the move. The cloud is not the answer. The cloud is an enabler to help organizations get the answers necessary to move in the direction they desire. There are risks associated with moving to the cloud – tools can distract from goals, system platforms need support, load balancers have to be implemented, and the cloud has to be leveraged and optimized to be beneficial long-term. Without strategizing past the migration, you won’t get the anticipated results.

It can seem overwhelming to take on the constantly changing cloud (and it certainly can be), but you don’t have to do it alone! Keep up with the pace and innovation of the digital transformation, while focusing on what you do best – growing your enterprise – by letting the experts help. 2nd Watch has a team of trusted cloud advisors to help you navigate cloud complexities for successful and ongoing cloud modernization. As an Amazon Web Services (AWS) Premier Partner, a Microsoft Azure Gold Partner, and a Google Cloud Partner with over 10 years’ experience, 2nd Watch provides ongoing advisory services to some of the largest companies in the world. Contact Us to take the next step in your cloud journey!

-Michael Elliott, Director of Marketing


Innovation is Key to Surviving the COVID 19 Pandemic

[et_pb_section admin_label=”section”]
[et_pb_row admin_label=”row”]
[et_pb_column type=”4_4″][et_pb_text admin_label=”Text”]The COVID 19 pandemic has affected all of us in one way or another, but possibly the hardest hit are small and medium businesses (SMBs) that continue to close shop across the country. Yelp estimates that 60% of the closures due to the pandemic are now permanent.

With jobs lost, businesses shut down, and the economic hardship many now face, many consumers are understandably cautious in terms of spending. Besides, the lockdown and its impact will continue to pose operational challenges to businesses for the next few months. Many business are going into 2021 facing lower consumer spending and continued restrictions on the movement of people.

This brings us to the question –what can really make a difference and help businesses survive the pandemic? The short answer – innovation!

The very entrepreneurial spirit of Americans – adapting to change, overcoming challenges in creative ways, and utilization of available technologies to reduce business disruption – has helped businesses stay afloat in a time when many are sinking fast.

Here are some tips for businesses to overcome the potentially debilitating effects of the pandemic – around 6 focus areas – to not only survive, but create an advantage enabling them to surpass the competition:

  • Revolutionalize the Delivery of Services
  • Leverage Remote Workforce
  • Cloud Transformation
  • Automation
  • Service Innovation
  • Upskill and Leap to Next Level

1.      Revolutionize the Delivery of Services

It’s difficult to predict when the lockdown will end and normalcy will return to everyday business operations. That doesn’t mean you have to close shop and go home. It just means that you must find new ways of serving your clients. And, a variety of establishments are already doing it. Consider these:

  • Restaurants are focusing on take-outs and food ordering apps to serve their customers.
  • Schools are conducting their classes via online video apps. Even ivy league institutes have been doing this for some time now.
  • Fitness trainers are hosting out-of-home workouts for their clients.

The list goes on. The good thing about these options is that these businesses are not inventing new technologies; they are just changing how they operate using available technologies.

2.      Leverage Remote Workforce

Since the beginning of the lockdown, you’ve been likely bombarded with news and social media updates on how other businesses are resorting to a remote workforce to ensure business continuity. Perhaps, you’ve given it some thought too, but are wary of the unknown and uncertainties that come with it. So, here’s some motivation to help you take the plunge:

  • The availability of technologies like Zoom, Google Meet, and other team collaboration apps have made remote working seamless for various employees in roles such as accounting, HR, IT support, and so on.
  • As employees spend more time at home with their loved ones, their productivity is increasing. That’s right, the remote workforce is more productive and happier Clearly, the fears of loss in productivity are now proving to be unfounded.
  • Working from home reduces the employees’ expenses on commute. Employees who witness these savings may deliver better results for the company, driven by higher employee satisfaction.

3.      Cloud Transformation

Cloud transformation was already a hot trend among businesses in every industry and economy worldwide before the pandemic. COVID-19 has only accelerated the adoption of the cloud. Here are some ways the cloud is helping businesses in the times of the pandemic:

  • Improved Resilience

With everyone locked in their homes, internet usage has skyrocketed. However, the people who maintain and upkeep digital technologies in organizations are stuck at home too. So, the digital surge combined with limited maintenance has resulted in increased outages.

Cloud systems offer superior resilience against outages – built-in data backups, recovery tools, and robust security features – that limit disruptions significantly.

  • Improved Scalability

The elastic nature of the cloud makes it possible for organizations to manage unexpected changes in consumer demand seamlessly and effortlessly. They can scale their digital resources up or down to match the consumer demand and pay only for the resources they utilize.

  • Improved Data-driven Innovation

The cloud is designed for data. It offers organizations powerful data manipulation tools they can utilize to extract highly useful insights from their consumer data. They can improve their existing products, develop new ones, or target new markets or audiences based on these insights. The potential is unlimited.

There are several more benefits of cloud that have made it a powerful enabler of business, even pre-pandemic:

4.      Automation

Automation is enabling businesses to streamline their operations and simultaneously improve customer satisfaction. Chatbots, for instance, allow businesses to engage their audiences in natural, human-like conversations. They offer quick, accurate, and timely responses to your potential customers, and even guide them through the purchase journey.

Likewise, robot process automation (RPA) can be used to automate a sizeable number of repetitive, low-level tasks, thereby freeing up your precious resources to focus on what really matters.

Another more common example of automation is self-service checkouts. Solutions like chatbots and self-service checkouts offer a win-win solution for both consumers and businesses. The increased customer satisfaction, lower costs of operations, and enhanced performance all add up to fuel your revenues.

5.      Service Innovation

A bit of creativity can go a long way in attracting previously ignored customers. For instance, meal buckets like Family Specials, Friends Specials, Weekend Combos, etc. can fetch high-value, low-volume orders from both current and new customers for restaurants.

McDonalds, for instance, has gone all out on its drive-thru and digital experience push. As part of its digital push, it has installed new point of sale (POS) systems that allow its customers to browse the menu, select items, make payment, and place an order with zero in-store contact. This minimizes the risk of infection, while demonstrating to their customers that the company cares about their health.

Likewise, fitness centers can use fitness apps to track their customers’ exercise regimen daily and offer them personalized advice. Such a personalized experience can go a long way in winning the loyalty of customers.

6.      Upskill and Leap to Next Level

For many businesses that witness frenzied activity during non-pandemic times, the lockdown can be a terrific opportunity to undertake the business improvement projects they’ve been putting off due to time crunch.

Free and premium online learning resources like Coursera, Datacamp, Codecademy, EDX, FuturLearn, and several others offer basic to specialized courses on almost everything under the sun. Upskilling your staff allows you to target new customers, new markets, and new needs of the consumers. You may emerge out of this lockdown stronger.

Conclusion

These are not normal times. Businesses cannot continue to operate as they did under normal times and expect to come out unscathed. As it goes, they must undertake desperate measures, exercise their creativity, and transform their business to generate new avenues of revenue. They must reinvent how they do business.

To examine more cloud-driven opportunities for your business and identify new sources of revenue, get in touch.

-Mir Ali, Field CTO[/et_pb_text][/et_pb_column]
[/et_pb_row]
[/et_pb_section]


Automating Windows Server Builds with Packer

[et_pb_section fb_built=”1″ _builder_version=”3.22.3″][et_pb_row _builder_version=”3.22.3″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.0.47″][et_pb_text _builder_version=”3.22.7″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”]

It might not be found on a curated list of urban legends, but trust me, IT IS possible (!!!) to fully automate the building and configuration of Windows virtual machines for AWS.  While Windows Server may not be your first choice of operating systems for immutable infrastructure, sometimes you as the IT professional are not given the choice due to legacy environments, limited resources for re-platforming, corporate edicts, or lack of internal *nix knowledge.

Complain all you like, but after the arguing, fretting, crying, and gnashing of teeth is done, at some point you will still need to build an automated deployment pipeline that requires Windows Server 2016.  With some PowerShell scripting and HashiCorp Packer, it is relatively easy and painless to securely build and configure Windows AMIs for your particular environment regardless of environment variables.

Configuration Options

Let’s dig into an example of how to build a custom configured Windows Server 2016 AMI.  You will need access to an AWS account and have sufficient permissions to create and manage EC2 instances.  If you need an AWS account, you can create one for free.

I am using VS Code and built-in terminal with Windows Subsystem for Linux to create the Packer template and run Packer, however Packer is available for several many Linux distros, Mac OS, and Windows.

First, download and unzip Packer:

/*
~/packer-demo:\> wget https://releases.hashicorp.com/packer/1.3.4/packer_1.3.4_linux_amd64.zip 
https://releases.hashicorp.com/packer/1.3.4/packer_1.3.4_linux_amd64.zip 
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.1.183, 151.101.65.183, 151.101.129.183, ... 
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.1.183|:443... connected. 
HTTP request sent, awaiting response... 200 OK 
Length: 28851840 (28M) [application/zip] 
Saving to: ‘packer_1.3.4_linux_amd64.zip’ 
‘packer_1.3.4_linux_amd64.zip’ saved [28851840/28851840] 
~/packer-demo:\> unzip packer_1.3.4_linux_amd64.zip 
Archive:  packer_1.3.4_linux_amd64.zip   
 inflating: packer
*/

/* ~/packer-demo:\> wget https://releases.hashicorp.com/packer/1.3.4/packer_1.3.4_linux_amd64.zip https://releases.hashicorp.com/packer/1.3.4/packer_1.3.4_linux_amd64.zip Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.1.183, 151.101.65.183, 151.101.129.183, ... Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.1.183|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 28851840 (28M) [application/zip] Saving to: ‘packer_1.3.4_linux_amd64.zip’ ‘packer_1.3.4_linux_amd64.zip’ saved [28851840/28851840] ~/packer-demo:\> unzip packer_1.3.4_linux_amd64.zip Archive:  packer_1.3.4_linux_amd64.zip   inflating: packer */

Now that we have Packer unzipped, verify that it is executable by checking the version:

/*
~/packer-demo:\> packer --version
1.3.3
*/

/* ~/packer-demo:\> packer --version 1.3.3 */

Packer can build machine images for a number of different platforms.  We will focus on the amazon-ebs builder, which will create an EBS-backed AMI.  At a high level, Packer performs these steps:

  1. Read configuration settings from a json file
  2. Uses the AWS API to stand up an EC2 instance
  3. Connect to the instance and provision it
  4. Shut down and snapshot the instance
  5. Create an Amazon Machine Image (AMI) from the snapshot
  6. Clean up the mess

The amazon-ebs builder can create temporary keypairs, security group rules, and establish a basic communicator for provisioning.  However, in the interest of tighter security and control, we will want to be prescriptive for some of these settings and use a secure communicator to reduce the risk of eavesdropping while we provision the machine.

There are two communicators Packer uses to upload scripts and files to a virtual machine: SSH (the default) and WinRM.  While there is a nice Win32 port of OpenSSH for Windows, it is not currently installed by default on Windows machines, but WinRM is available natively in all current versions of Windows, so we will use that to provision our Windows Server 2016 machine.

Let’s create and edit the userdata file that Packer will use to bootstrap the EC2 instance:

/*
 
# USERDATA SCRIPT FOR AMAZON SOURCE WINDOWS SERVER AMIS
# BOOTSTRAPS WINRM VIA SSL
 
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine -Force -ErrorAction Ignore
$ErrorActionPreference = "stop"
 
# Remove any existing Windows Management listeners
Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse
 
# Create self-signed cert for encrypted WinRM on port 5986
$Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName "packer-ami-builder"
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint -Force
 
# Configure WinRM
cmd.exe /c winrm quickconfig -q
cmd.exe /c winrm set "winrm/config" '@{MaxTimeoutms="1800000"}'
cmd.exe /c winrm set "winrm/config/winrs" '@{MaxMemoryPerShellMB="1024"}'
cmd.exe /c winrm set "winrm/config/service" '@{AllowUnencrypted="false"}'
cmd.exe /c winrm set "winrm/config/client" '@{AllowUnencrypted="false"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/client/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{CredSSP="true"}'
cmd.exe /c winrm set "winrm/config/listener?Address=*+Transport=HTTPS" "@{Port=`"5986`";Hostname=`"packer-ami-builder`";CertificateThumbprint=`"$($Cert.Thumbprint)`"}"
cmd.exe /c netsh advfirewall firewall add rule name="WinRM-SSL (5986)" dir=in action=allow protocol=TCP localport=5986
cmd.exe /c net stop winrm
cmd.exe /c sc config winrm start= auto
cmd.exe /c net start winrm
 
*/

/* # USERDATA SCRIPT FOR AMAZON SOURCE WINDOWS SERVER AMIS # BOOTSTRAPS WINRM VIA SSL Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope LocalMachine -Force -ErrorAction Ignore $ErrorActionPreference = "stop" # Remove any existing Windows Management listeners Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse # Create self-signed cert for encrypted WinRM on port 5986 $Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName "packer-ami-builder" New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint -Force # Configure WinRM cmd.exe /c winrm quickconfig -q cmd.exe /c winrm set "winrm/config" '@{MaxTimeoutms="1800000"}' cmd.exe /c winrm set "winrm/config/winrs" '@{MaxMemoryPerShellMB="1024"}' cmd.exe /c winrm set "winrm/config/service" '@{AllowUnencrypted="false"}' cmd.exe /c winrm set "winrm/config/client" '@{AllowUnencrypted="false"}' cmd.exe /c winrm set "winrm/config/service/auth" '@{Basic="true"}' cmd.exe /c winrm set "winrm/config/client/auth" '@{Basic="true"}' cmd.exe /c winrm set "winrm/config/service/auth" '@{CredSSP="true"}' cmd.exe /c winrm set "winrm/config/listener?Address=*+Transport=HTTPS" "@{Port=`"5986`";Hostname=`"packer-ami-builder`";CertificateThumbprint=`"$($Cert.Thumbprint)`"}" cmd.exe /c netsh advfirewall firewall add rule name="WinRM-SSL (5986)" dir=in action=allow protocol=TCP localport=5986 cmd.exe /c net stop winrm cmd.exe /c sc config winrm start= auto cmd.exe /c net start winrm */

There are four main things going on here:

  1. Set the execution policy and error handling for the script (if an error is encountered, the script terminates immediately)
  2. Clear out any existing WS Management listeners, just in case there are any preconfigured insecure listeners
  3. Create a self-signed certificate for encrypting the WinRM communication channel, and then bind it to a WS Management listener
  4. Configure WinRM to:
    • Require an encrypted (SSL) channel
    • Enable basic authentication (usually this not secure as the password goes across in plain text, but we are forcing encryption)
    • Configure the listener to use port 5986 with the self-signed certificate we created earlier
    • Add a firewall rule to open port 5986

Now that we have userdata to bootstrap WinRM, let’s create a Packer template:

/*
~/packer-demo:\> touch windows2016.json
*/

/* ~/packer-demo:\> touch windows2016.json */

Open this file with your favorite text editor and add this text:

/*
{
    "variables": {
        "build_version": "{{isotime \"2006.01.02.150405\"}}",
        "aws_profile": null,
        "vpc_id": null,
        "subnet_id": null,
        "security_group_id": null
    },
    "builders": [
        {
            "type": "amazon-ebs",
            "region": "us-west-2",
            "profile": "{{user `aws_profile`}}",
            "vpc_id": "{{user `vpc_id`}}",
            "subnet_id": "{{user `subnet_id`}}",
            "security_group_id": "{{user `security_group_id`}}",
            "source_ami_filter": {
                "filters": {
                    "name": "Windows_Server-2016-English-Full-Base-*",
                    "root-device-type": "ebs",
                    "virtualization-type": "hvm"
                },
                "most_recent": true,
                "owners": [
                    "801119661308"
                ]
            },
            "ami_name": "WIN2016-CUSTOM-{{user `build_version`}}",
            "instance_type": "t3.xlarge",
            "user_data_file": "userdata.ps1",
            "associate_public_ip_address": true,
            "communicator": "winrm",
            "winrm_username": "Administrator",
            "winrm_port": 5986,
            "winrm_timeout": "15m",
            "winrm_use_ssl": true,
            "winrm_insecure": true
        }
    ]
}
*/

/* { "variables": { "build_version": "{{isotime \"2006.01.02.150405\"}}", "aws_profile": null, "vpc_id": null, "subnet_id": null, "security_group_id": null }, "builders": [ { "type": "amazon-ebs", "region": "us-west-2", "profile": "{{user `aws_profile`}}", "vpc_id": "{{user `vpc_id`}}", "subnet_id": "{{user `subnet_id`}}", "security_group_id": "{{user `security_group_id`}}", "source_ami_filter": { "filters": { "name": "Windows_Server-2016-English-Full-Base-*", "root-device-type": "ebs", "virtualization-type": "hvm" }, "most_recent": true, "owners": [ "801119661308" ] }, "ami_name": "WIN2016-CUSTOM-{{user `build_version`}}", "instance_type": "t3.xlarge", "user_data_file": "userdata.ps1", "associate_public_ip_address": true, "communicator": "winrm", "winrm_username": "Administrator", "winrm_port": 5986, "winrm_timeout": "15m", "winrm_use_ssl": true, "winrm_insecure": true } ] } */

Couple of things to call out here.  First, the variables block at the top references some values that are needed in the template.  Insert values here that are specific to your AWS account and VPC:

  • aws_profile: I use a local credentials file to store IAM user credentials (the file is shared between WSL and Windows). Specify the name of a credential block that Packer can use to connect to your account.  The IAM user will need permissions to create and modify EC2 instances, at a minimum
  • vpc_id: Packer will stand up the instance in this VPC
  • aws_region: Your VPC should be in this region. As an exercise, change this value to be set by a variable instead
  • user_data_file: We created this file earlier, remember? If you saved it in another location, make sure the path is correct
  • subnet_id: This should belong to the VPC specified above. Use a public subnet if needed if you do not have a Direct Connect
  • security_group_id: This security group should belong to the VPC specified above. This security group will be attached to the instance that Packer stands up.  It will need, at a minimum, inbound TCP 5986 from where Packer is running

Next, let’s validate the template to make sure the syntax is correct, and we have all required fields:

/*
~/packer-demo:\> packer validate windows2016.json
Template validation failed. Errors are shown below.
Errors validating build 'amazon-ebs'. 1 error(s) occurred:
* An instance_type must be specified

/* ~/packer-demo:\> packer validate windows2016.json Template validation failed. Errors are shown below. Errors validating build 'amazon-ebs'. 1 error(s) occurred: * An instance_type must be specified

Whoops, we missed instance_type.  Add that to the template and specify a valid EC2 instance type.  I like using beefier instance types so that the builds get done quicker, but that’s just me.

            "ami_name": "WIN2016-CUSTOM-{{user `build_version`}}",
            "instance_type": "t3.xlarge",
            "user_data_file": "userdata.ps1",
*/

"ami_name": "WIN2016-CUSTOM-{{user `build_version`}}", "instance_type": "t3.xlarge", "user_data_file": "userdata.ps1", */

Now validate again:

/*
~/packer-demo:\> packer validate windows2016.json
Template validated successfully.
*/

/* ~/packer-demo:\> packer validate windows2016.json Template validated successfully. */

Awesome.  Couple of things to call out in the template:

  • source_ami_filter: we are using a base Amazon AMI for Windows Server 2016 server. Note the wildcard in the AMI name (Windows_Server-2016-English-Full-Base-*) and the owner (801119661308).  This filter will always pull the most recent match for the AMI name.  Amazon updates its AMIs about once a month
  • associate_public_ip_address: I specify true because I don’t have Direct Connect or VPN from my workstation to my sandbox VPC. If you are building your AMI in a public subnet and want to connect to it over the internet, do the same
  • ami_name: this is a required property, and it must also be unique within your account. We use a special function here (isotime) to generate a timestamp, ensuring that the AMI name will always be unique (e.g. WIN2016-CUSTOM-2019.03.01.000042)
  • winrm_port: the default unencrypted port for WinRM is 5985. We disabled 5985 in userdata and enabled 5986, so be sure to call it out here
  • winrm_use_ssl: as noted above, we are encrypting communication, so set this to true
  • winrm_insecure: this is a rather misleading property name. It really means that Packer should not check if encryption certificate is trusted.  We are using a self-signed certificate in userdata, so set this to true to skip certificate validation

Now that we have our template, let’s inspect it to see what will happen:

/*
~/packer-demo:\> packer inspect windows2016.json
Optional variables and their defaults:
  aws_profile       = default
  build_version     = {{isotime "2006.01.02.150405"}}
  security_group_id = sg-0e1ca9ba69b39926
  subnet_id         = subnet-00ef2a1df99f20c23
  vpc_id            = vpc-00ede10ag029c31e0
Builders:
  amazon-ebs
Provisioners:
 
Note: If your build names contain user variables or template
functions such as 'timestamp', these are processed at build time,
and therefore only show in their raw form here.
*/

/* ~/packer-demo:\> packer inspect windows2016.json Optional variables and their defaults: aws_profile = default build_version = {{isotime "2006.01.02.150405"}} security_group_id = sg-0e1ca9ba69b39926 subnet_id = subnet-00ef2a1df99f20c23 vpc_id = vpc-00ede10ag029c31e0 Builders: amazon-ebs Provisioners: Note: If your build names contain user variables or template functions such as 'timestamp', these are processed at build time, and therefore only show in their raw form here. */

Looks good, but we are missing a provisioner!  If we ran this template as is, all that would happen is Packer would make a copy of the base Windows Server 2016 AMI and make you the owner of the new private image.

There are several different Packer provisioners, ranging from very simple (the windows-restart provisioner just reboots the machine) to complex (the chef client provisioner installs chef client and does whatever Chef does from there).  We’ll run a basic powershell provisioner to install IIS on the instance, reboot the instance using windows-restart, and then we’ll finish up the provisioning by executing sysprep on the instance to generalize it for re-use.

We will use the PowerShell cmdlet Enable-WindowsOptionalFeature to install the web server role and IIS with defaults, add this text to your template below the builders [ ] section:

/*
"provisioners": [
        {
            "type": "powershell",
            "inline": [
                "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerRole",
                "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServer"
            ]
        },
        {
            "type": "windows-restart",
            "restart_check_command": "powershell -command \"& {Write-Output 'Machine restarted.'}\""
        },
        {
            "type": "powershell",
            "inline": [
                "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
                "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown"
            ]
        }
    ]
*/

/* "provisioners": [ { "type": "powershell", "inline": [ "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerRole", "Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServer" ] }, { "type": "windows-restart", "restart_check_command": "powershell -command \"& {Write-Output 'Machine restarted.'}\"" }, { "type": "powershell", "inline": [ "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule", "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown" ] } ] */

Couple things to call out about this section:

  • The first powershell provisioner uses the inline Packer simply appends these lines in order into a file, then transfers and executes the file on the instance using PowerShell
  • The second provisioner, windows-restart, simply reboots the machine while Packer waits. While this isn’t always necessary, it is helpful to catch instances where settings do not persist after a reboot, which was probably not your intention
  • The final powershell provisioner executes two PowerShell scripts that are present on Amazon Windows Server 2016 AMIs and part of the EC2Launch application (earlier versions of Windows use a different application called EC2Config). They are helper scripts that you can use to prepare the machine for generalization, and then execute sysprep

After validating your template again, let’s build the AMI!

/*
~/packer-demo:\> packer validate windows2016.json
Template validated successfully.
*/
/*
~/packer-demo:\> packer build windows2016.json
amazon-ebs output will be in this color.
==> amazon-ebs: Prevalidating AMI Name: WIN2016-CUSTOM-2019.03.01.000042
    amazon-ebs: Found Image ID: ami-0af80d239cc063c12
==> amazon-ebs: Creating temporary keypair: packer_5c78762a-751e-2cd7-b5ce-9eabb577e4cc
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-0384e1edca5dc90e5
==> amazon-ebs: Waiting for instance (i-0384e1edca5dc90e5) to become ready...
==> amazon-ebs: Waiting for auto-generated password for instance...
    amazon-ebs: It is normal for this process to take up to 15 minutes,
    amazon-ebs: but it usually takes around 5. Please wait.
    amazon-ebs:  
    amazon-ebs: Password retrieved!
==> amazon-ebs: Using winrm communicator to connect: 34.220.235.82
==> amazon-ebs: Waiting for WinRM to become available...
    amazon-ebs: WinRM connected.
    amazon-ebs: #> CLIXML
    amazon-ebs: System.Management.Automation.PSCustomObjectSystem.Object1Preparing modules for first 
use.0-1-1Completed-1 1Preparing modules for first use.0-1-1Completed-1 
==> amazon-ebs: Connected to WinRM!
==> amazon-ebs: Provisioning with Powershell...
==> amazon-ebs: Provisioning with powershell script: /tmp/packer-powershell-provisioner561623209
    amazon-ebs:
    amazon-ebs:
    amazon-ebs: Path          :
    amazon-ebs: Online        : True
    amazon-ebs: RestartNeeded : False
    amazon-ebs:
    amazon-ebs: Path          :
    amazon-ebs: Online        : True
    amazon-ebs: RestartNeeded : False
    amazon-ebs:
==> amazon-ebs: Restarting Machine
==> amazon-ebs: Waiting for machine to restart...
    amazon-ebs: Machine restarted.
    amazon-ebs: EC2AMAZ-LJV703F restarted.
    amazon-ebs: #> CLIXML
    amazon-ebs: System.Management.Automation.PSCustomObjectSystem.Object1Preparing modules for first 
use.0-1-1Completed-1 
==> amazon-ebs: Machine successfully restarted, moving on
==> amazon-ebs: Provisioning with Powershell...
==> amazon-ebs: Provisioning with powershell script: /tmp/packer-powershell-provisioner928106343
    amazon-ebs:
    amazon-ebs: TaskPath                                       TaskName                          State
    amazon-ebs: --------                                       --------                          -----
    amazon-ebs: \                                              Amazon Ec2 Launch - Instance I... Ready
==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance, attempt 1
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating unencrypted AMI WIN2016-CUSTOM-2019.03.01.000042 from instance i-0384e1edca5dc90e5
    amazon-ebs: AMI: ami-0d6026ecb955cc1d6
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.
 
==> Builds finished. The artifacts of successful builds are:
==> amazon-ebs: AMIs were created:
us-west-2: ami-0d6026ecb955cc1d6
*/

/* ~/packer-demo:\> packer validate windows2016.json Template validated successfully. */ /* ~/packer-demo:\> packer build windows2016.json amazon-ebs output will be in this color. ==> amazon-ebs: Prevalidating AMI Name: WIN2016-CUSTOM-2019.03.01.000042 amazon-ebs: Found Image ID: ami-0af80d239cc063c12 ==> amazon-ebs: Creating temporary keypair: packer_5c78762a-751e-2cd7-b5ce-9eabb577e4cc ==> amazon-ebs: Launching a source AWS instance... ==> amazon-ebs: Adding tags to source instance amazon-ebs: Adding tag: "Name": "Packer Builder" amazon-ebs: Instance ID: i-0384e1edca5dc90e5 ==> amazon-ebs: Waiting for instance (i-0384e1edca5dc90e5) to become ready... ==> amazon-ebs: Waiting for auto-generated password for instance... amazon-ebs: It is normal for this process to take up to 15 minutes, amazon-ebs: but it usually takes around 5. Please wait. amazon-ebs: amazon-ebs: Password retrieved! ==> amazon-ebs: Using winrm communicator to connect: 34.220.235.82 ==> amazon-ebs: Waiting for WinRM to become available... amazon-ebs: WinRM connected. amazon-ebs: #> CLIXML amazon-ebs: System.Management.Automation.PSCustomObjectSystem.Object1Preparing modules for first use.0-1-1Completed-1 1Preparing modules for first use.0-1-1Completed-1 ==> amazon-ebs: Connected to WinRM! ==> amazon-ebs: Provisioning with Powershell... ==> amazon-ebs: Provisioning with powershell script: /tmp/packer-powershell-provisioner561623209 amazon-ebs: amazon-ebs: amazon-ebs: Path : amazon-ebs: Online : True amazon-ebs: RestartNeeded : False amazon-ebs: amazon-ebs: Path : amazon-ebs: Online : True amazon-ebs: RestartNeeded : False amazon-ebs: ==> amazon-ebs: Restarting Machine ==> amazon-ebs: Waiting for machine to restart... amazon-ebs: Machine restarted. amazon-ebs: EC2AMAZ-LJV703F restarted. amazon-ebs: #> CLIXML amazon-ebs: System.Management.Automation.PSCustomObjectSystem.Object1Preparing modules for first use.0-1-1Completed-1 ==> amazon-ebs: Machine successfully restarted, moving on ==> amazon-ebs: Provisioning with Powershell... ==> amazon-ebs: Provisioning with powershell script: /tmp/packer-powershell-provisioner928106343 amazon-ebs: amazon-ebs: TaskPath TaskName State amazon-ebs: -------- -------- ----- amazon-ebs: \ Amazon Ec2 Launch - Instance I... Ready ==> amazon-ebs: Stopping the source instance... amazon-ebs: Stopping instance, attempt 1 ==> amazon-ebs: Waiting for the instance to stop... ==> amazon-ebs: Creating unencrypted AMI WIN2016-CUSTOM-2019.03.01.000042 from instance i-0384e1edca5dc90e5 amazon-ebs: AMI: ami-0d6026ecb955cc1d6 ==> amazon-ebs: Waiting for AMI to become ready... ==> amazon-ebs: Terminating the source AWS instance... ==> amazon-ebs: Cleaning up any extra volumes... ==> amazon-ebs: No volumes to clean up, skipping ==> amazon-ebs: Deleting temporary keypair... Build 'amazon-ebs' finished. ==> Builds finished. The artifacts of successful builds are: ==> amazon-ebs: AMIs were created: us-west-2: ami-0d6026ecb955cc1d6 */

The entire process took approximately 5 minutes from start to finish.  If you inspect the output, you can see where the first PowerShell provisioner installed IIS (Provisioning with powershell script: /tmp/packer-powershell-provisioner561623209), where sysprep was executed (Provisioning with powershell script: /tmp/packer-powershell-provisioner928106343) and where Packer yielded the AMI ID for the image (ami-0d6026ecb955cc1d6) and cleaned up any stray artifacts.

Note: most of the issues that occur during a Packer build are related to firewalls and security groups.  Verify that the machine that is running Packer can reach the VPC and EC2 instance using port 5986.  You can also use packer build -debug win2016.json to step through the build process manually, and you can even connect to the machine via RDP to troubleshoot a provisioner.

Now, if you launch a new EC2 instance using this AMI, it will already have IIS installed and configured with defaults.  Building and securely configuring Windows AMIs with Packer is easy!

For help getting started securely building and configuring Windows AMIs for your particular environment, contact us.

-Jonathan Eropkin, Cloud Consultant

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]


AWS re:Invent Keynote Recap – Thursday

Thursday’s General Session Keynote kicked off with Amazon CTO, Werner Vogels, taking the stage to deliver additional product and services announcements with the inclusion of deeper, technical content.  Revisiting his vision for 21st Architectures from the 1st Re:Invent in 2012, Werner focused on what he sees as key guiding principles for next-gen workloads.

  1. Voice represents the next major disruption in computing. Stressing this point, Werner announced the general availability of Alexa for Business to help improve productivity by introducing voice automation into your business.
  2. Use automation to make experimentation easier
  3. Encryption is the ‘key’ to controlling access to your data. As such, encrypting data (at rest and in transit) should be a default behavior.
  4. All the code you should ever write is business logic.

 

Werner also highlighted the fact that AWS now has over 3,951 new services released since 2012. These services were not built for today but built for the workloads of the future.  The goal for AWS, Werner says, is to be your partner for the future.

One of the highlights of the keynote was when Abby Fuller, evangelist for containers at AWS, came on stage to talk about the future of containers at AWS.  She demoed the use of Fargate which is AWS’s fully managed container service. Think of Fargate as Elastic Beanstalk but for containers.  Per AWS documentation “It’s a technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances. All you need to do is build your container image, specify the CPU and memory requirements, define your networking and IAM policies, and launch. With Fargate, you have flexible options to closely match your application needs and you’re billed with per-second granularity.”

The Cloud9 acquisition was also a highlight of the keynote.  Cloud9 is a browser-based IDE for developers.  Cloud9 is completely integrated with AWS and you can create cloud environments, develop code, and push that code to your cloud environment all from within the tool.  It’s really going to be useful for writing and debugging lambda functions for developers that have gone all in on serverless technologies.

New Announcements

AWS Lambda Function Execution Activity Logging – Log all execution activity for your Lambda functions. Previously you could only log events but this allows you to log data events and get additional details.

AWS Lambda Doubles Maximum Memory Capacity for Lambda Functions – You can now allocate 3008MB of memory to your AWS Lambda functions.

AWS Cloud9 –  Cloud9 is a cloud based IDE for writing, running, and debugging your code.

API Gateway now supports endpoint integrations with Private VPCs –  You can now provide access to HTTP(S) resources within your Amazon Virtual Private Cloud (VPC) without exposing them directly to the public Internet.

AWS Serverless Application Repository –   The Serverless Application Repository is a collection of serverless applications published by developers, companies, and partners in the serverless community.

We expect AWS to announce many more awesome features and services before the day ends so stay tuned for our AWS  re:Invent 2017 Products & Services Review and 2017 Conference Recap blog posts for a summary of all of the announcements that are being delivered at AWS re:Invent 2017.

 

— Brent Clements, Sr. Cloud Consultant, 2nd Watch


2nd Watch Enterprise Cloud Expertise On Display at AWS re:Invent 2017

AWS re:Invent is less than twenty days away and 2nd Watch is proud to be a 2017 Platinum Sponsor for the sixth consecutive year.  As an Amazon Web Services (AWS) Partner Network Premier Consulting Partner, we look forward to attending and demonstrating the strength of our cloud design, migration, and managed services offerings for enterprise organizations at AWS re:Invent 2017 in Las Vegas, Nevada.

About AWS re:Invent

Designed for AWS customers, enthusiasts and even cloud computing newcomers, the nearly week-long conference is a great source of information and education for attendees of all skill levels. AWS re:Invent is THE place to connect, engage, and discuss current AWS products and services via breakout sessions ranging from introductory and advanced to expert as well as to hear the latest news and announcements from key AWS executives, partners, and customers. This year’s agenda offers a full additional day of content for even more learning opportunities, more than 1,000 breakout sessions, an expanded campus, hackathons, boot camps, hands-on labs, workshops, expanded Expo hours, and the always popular Amazonian events featuring broomball, Tatonka Challenge, fitness activities, and the attendee welcome party known as re:Play.

2nd Watch at re:Invent 2017

 2nd Watch has been a Premier Consulting Partner in the AWS Partner Network (APN) since 2012 and was recently named a leader in Gartner’s Magic Quadrant for Public Cloud Infrastructure Managed Service Providers, Worldwide (March 2017). We hold AWS Competencies in Financial Services, Migration, DevOps, Marketing, and Commerce, Life Sciences and Microsoft Workloads, and have recently completed the AWS Managed Service Provider (MSP) Partner Program Audit for the third year in a row. Over the past decade, 2nd Watch has migrated and managed AWS deployments for companies such as Crate & Barrel, Condé Nast, Lenovo, Motorola, and Yamaha.

The 2nd Watch breakout session—Continuous Compliance on AWS at Scale—will be led by cloud security experts Peter Meister and Lars Cromley. The session will focus on the need for continuous security and compliance in cloud migrations, and attendees will learn how a managed cloud provider can use automation and cloud expertise to successfully control these issues at scale in a constantly changing cloud environment. Registered re:Invent Full Conference Pass holders can add the session to their agendas here.

In addition to our breakout session, 2nd Watch will be showcasing our customers’ successes in the Expo Hall located in the Sands Convention Center (between The Venetian and The Palazzo hotels).  We invite you to stop by booth #1104 where you can explore 2nd Watch’s Managed Cloud Solutions, pick up a coveted 2nd Watch t-shirt and find out how you can win one of our daily contest giveaways—a totally custom 2nd Watch skateboard!

Want to make sure you get time with one of 2nd Watch’s Cloud Journey Masters while at re:Invent?  Plan ahead and schedule a meeting with one of 2nd Watch’s AWS Professional Certified Architects, DevOps, or Engineers.  Last but not least, 2nd Watch will be hosting its annual re:Invent after party on Wednesday, November 29. If you haven’t RSVP’d for THE AWS re:Invent Partner Party, click here to request your invitation (Event has passed)

AWS re:Invent is sure to be a week full of great technical learning, networking, and social opportunities.  We know you will have a packed schedule but look forward to seeing you there!  Be on the lookout for my list of “What to Avoid at re:Invent 2017” in the coming days…it’s sure to help you plan for your trip and get the most out of your AWS re:Invent experience.

 

–Katie Laas-Ellis, Marketing Manager, 2nd Watch

 

Gartner Disclaimer

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

About 2nd Watch

2nd Watch is an AWS Premier tier Partner in the AWS Partner Network (APN) providing managed cloud to enterprises. The company’s subject matter experts, software-enabled services and cutting-edge solutions provide companies with tested, proven, and trusted solutions, allowing them to fully leverage the power of the cloud. 2nd Watch solutions are high performing, robust, increase operational excellence, decrease time to market, accelerate growth and lower risk. Its patent-pending, proprietary tools automate everyday workload management processes for big data analytics, digital marketing, line-of-business and cloud native workloads. 2nd Watch is a new breed of business which helps enterprises design, deploy and manage cloud solutions and monitors business critical workloads 24×7. 2nd Watch has more than 400 enterprise workloads under its management and more than 200,000 instances in its managed public cloud. The venture-backed company is headquartered in Seattle, Washington. To learn more about 2nd Watch, visit www.2ndwatch.com or call 888-317-7920.


Standardizing & Automating Infrastructure Development Processes

Introduction

Let’s start with a small look at the current landscape of technology and how we arrived here. There aren’t very many areas of tech that have not been, or are not currently, in a state of fluctuation. Everything from software delivery vehicles and development practices, to infrastructure creation has experienced some degree of transformation over the past several years. From VMs to Containers, it seems like almost every day the technology tool belt grows a little bigger, and our world gets a little better (though perhaps more complex) due to these advancements. For me, this was incredibly apparent when I began to delve into configuration management which later evolved into what we now call “infrastructure as code”.

The transformation of the development process began with simple systems that we once used to manage a few machines (like bash scripts or Makefiles) which then morphed into more complex systems (CF Engine, Puppet, and Chef) to manage thousands of systems. As configuration management software became more mature, engineers and developers began leaning on them to do more things. With the advent of hypervisors and the rise of virtual machines, it was only a short time before hardware requests changed to API requests and thus the birth of infrastructure as a service (IaaS). With all the new capabilities and options in this brave new world, we once again started to lean on our configuration management systems—this time for provisioning, and not just convergence.

Provisioning & Convergence

I mentioned two terms that I want to clarify; provisioning and convergence. Say you were a car manufacturer and you wanted to make a car. Provisioning would be the step in which you request the raw materials to make the parts for your automobile. This is where we would use tools like Terraform, CloudFormation, or Heat. Whereas convergence is the assembly line by which we check each part and assemble the final product (utilizing config management software).

By and large, the former tends to be declarative with little in the way of conditionals or logic, while the latter is designed to be robust and malleable software that supports all the systems we run and plan on running. This is the frame for the remainder of what we are going to talk about.

By separating the concerns of our systems, we can create a clear delineation of the purpose for each tool so we don’t feel like we are trying to jam everything into an interface that doesn’t have the most support for our platform or more importantly our users. The remainder of this post will be directed towards the provisioning aspect of configuration management.

Standards and Standardization

These are two different things in my mind. Standardization is extremely prescriptive and can often seem particularly oppressive to professional knowledge workers, such as engineers or developers. It can be seen as taking the innovation away from the job. Whereas standards provide boundaries, frame the problem, and allow for innovative ways of approaching solutions. I am not saying standardization in some areas is entirely bad, but we should let the people who do the work have the opportunity to grow and innovate in their own way with guidance. The topic of standards and standardization is part of a larger conversation about culture and change. We intend to follow up with a series of blog articles relating to organizational change in the era of the public cloud in the coming weeks.

So, let’s say that we make a standard for our new EC2 instances running Ubuntu. We’ll say that all instances must be running the la official Canonical Ubuntu 14.04 AMI and must have these three tags; Owner, Environment, and Application. How can we enforce that in development of our infrastructure? On AWS, we can create AWS Config Rules, but that is reactive and requires ad-hoc remediation. What we really want is a more prescriptive approach bringing our standards closer to the development pipeline. One of the ways I like to solve this issue is by creating an abstraction. Say we have a terraform template that looks like this:

# Create a new instance of the la Ubuntu 14.04 on an
provider "aws" { region = "us-west-2"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type = "t2.micro"

tags {
Owner	= "DevOps Ninja" Environment = "Dev" Application = "Web01"
}
}

This would meet the standard that we have set forth, but we are relying on the developer or engineer to adhere to that standard. What if we enforce this standard by codifying it in an abstraction? Let’s take that existing template and turn it into a terraform module instead.

Module

# Create a new instance of the la Ubuntu 14.04 on an

variable "aws_region" {} variable "ec2_owner" {} variable "ec2_env" 
{} variable "ec2_app" {}
variable "ec2_instance_type" {}

provider "aws" {
region = "${var.aws_region}"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type =
"${var.ec2_instance_type}"

tags {
Owner	= "${var.ec2_owner}" Environment = "${var.ec2_env}" Application = 
"${var.ec2_app}"
}
}

Now we can have our developers and engineers leverage our tf_ubuntu_ec2_instance module.

New Terraform Plan

module "Web01" { source =
"git::ssh://git@github.com/SomeOrg/tf_u buntu_ec2_instance"

aws_region = "us-west-2" ec2_owner = "DevOps Ninja" ec2_env	= "Dev"
ec2_app	= "Web01"
}

This doesn’t enforce the usage of the module, but it does create an abstraction that provides an easy way to maintain standards without a ton of overhead, it also provides an example for further creation of modules that enforce these particular standards.

This leads us into another method of implementing standards but becomes more prescriptive and falls into the category of standardization (eek!). One of the most underutilized services in the AWS product stable has to be Service Catalog.

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. AWS Service Catalog allows you to centrally manage commonly deployed IT services, and helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.

The Interface

Once we have a few of these projects in place (e.g. a service catalog or a repo full of composable modules for infrastructure that meet our standards) how do we serve them out? How you spur adoption of these tools and how they are consumed can be very different depending on your organization structure. We don’t want to upset workflow and how work gets done, we just want it to go faster and be more reliable. This is what we talk about when we mention the interface. Whichever way work flows in, we should supplement it with some type of software or automation to link those pieces of work together. Here are a few examples of how this might look (depending on your organization):

1.) Central IT Managed Provisioning

If you have an organization that manages requests for infrastructure, having this new shift in paradigm might seem daunting. The interface in this case is the ticketing system. This is where we would create an integration with our ticketing software to automatically pull the correct project from service catalog or module repo based on some criteria in the ticket. The interface doesn’t change but is instead supplemented by some automation to answer these requests, saving time and providing faster delivery of service.

2.) Full Stack Engineers

If you have engineers that develop software and the infrastructure that runs their applications this is the easiest scenario to address in some regards and the hardest in others. Your interface might be a build server, or it could simply be the adoption of an internal open source model where each team develops modules and shares them in a common place, constantly trying to save time and not re-invent the wheel.

Supplementing with software or automation can be done in a ton of ways. Check out an example Kelsey Hightower wrote using Jira.

“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” – John Gall

All good automation starts out with a manual and well-defined process. Standardizing & automating infrastructure development processes begins with understanding how our internal teams can be best organized.  This allows us to efficiently perform work before we can begin automating. Work with your teammates to create a value stream map to understand the process entirely before doing anything towards the efforts of automating a workflow.

With 2nd Watch designs and automation you can deploy quicker, learn faster and modify as needed with Continuous Integration / Continuous Deployment (CI/CD). Our Workload Solutions transform on-premises workloads to digital solutions in the public cloud with next generation products and services.  To accelerate your infrastructure development so that you can deploy faster, learn more often and adapt to customer requirements more effectively, speak with a 2nd Watch cloud deployment expert today.

– Lars Cromley, Director of Engineering, Automation, 2nd Watch


Automating Windows Patching of EC2 Autoscaling Group Instances

Background

Dealing with Windows patching can be a royal pain as you may know.  At least once a month Windows machines are subject to system security and stability patches, thanks to Microsoft’s Patch Tuesday. With Windows 10 (and its derivatives), Microsoft has shifted towards more of a Continuous Delivery model in how it manages system patching. It is a welcome change, however, it still doesn’t guarantee that Windows patching won’t require a system reboot.

Rebooting an EC2 instance that is a member of an Auto Scaling Group (depending upon how you have your Auto Scaling health-check configured) is something that will typically cause an Elastic Load Balancing (ELB) HealthCheck failure and result in instance termination (this occurs when Auto Scaling notices that the instance is no longer reporting “in service” with the load balancer). Auto Scaling will of course replace the terminated instance with a new one, but the new instance will be launched using an image that is presumably unpatched, thus leaving your Windows servers vulnerable.

The next patch cycle will once again trigger a reboot and the vicious cycle continues. Furthermore, if the patching and reboots aren’t carefully coordinated, it could severely impact your application performance and availability (think multiple Auto Scaling Group members rebooting simultaneously). If you are running an earlier version of Windows OS (e.g. Windows Server 2012r2), rebooting at least once a month on Patch Tuesday is an almost certainty.

Another major problem with utilizing the AWS stock Windows AMIs with Auto Scaling is that AWS makes those AMIs unavailable after just a few months. This means that unless you update your Auto Scaling Launch Configuration to use the newer AMI IDs on a continual basis, future Auto Scaling instance launches will fail as they try to access an AMI that is no longer accessible. Anguish.

Automatically and Reliably Patch your Auto-Scaled Windows instances

Given the aforementioned scenario, how on earth are you supposed to automatically and reliably patch your Auto-Scaled Windows instances?!

One approach would be to write some sort of an orchestration layer that detects when Auto Scaling members have been patched and are awaiting their obligatory reboot, suspend Auto Scaling processes that would detect and replace perceived failed instances (e.g. HealthCheck), and then reboot the instances one-by-one. This would be rather painful to orchestrate and has a potentially severe drawback that cluster capacity is reduced by N-1 during the rebooting (maybe more if you don’t take into account service availability between reboots).

Reducing capacity to N-1 might not be a big deal if you have a cluster of 20 instances but if you are running a smaller cluster of something— say 4, 3, or 2 instances—then that has a significant impact to your overall cluster capacity. And, if you are running on an Auto Scaling group with a single instance (not as uncommon as you might think) then your application is completely down during the reboot of that single member. This of course doesn’t solve the issue of expired stock AWS AMIs.

Another approach is to maintain and patch a “golden image” that the Auto Scaling Launch Configuration uses to create new instances from. If you are unfamiliar with the term, a golden-image is an operating system image that has everything pre-installed, configured, and saved in a pre-baked image file (an AMI in the case of Amazon EC2). This approach requires a significant amount of work to make this happen in a reasonably automated fashion and has numerous potential pitfalls.

While it prevents a service outage by replacing the unavailable public AMI with a stock AMI, you still need a way to reliably and automatically handle this process. Using a tool like Hashicorp’s Packer can get you partially there, but you would still have to write a number of Providers to handle the installation of Windows Update and anything else you need to do in order to prep the system for imaging. In the end, you would still have to develop or employ a fair number of tools and processes to completely automate the entire process of detecting new Windows Updates, creating a patched AMI with those updates, and orchestrating the update of your Auto Scaling Groups.

A Cloud-Minded Approach

I believe that Auto Scaling Windows servers intelligently requires a paradigm shift. One assumption we have to make is that some form of configuration management (e.g. Puppet, Chef)—or at least a basic bootstrap script executed via cfn-init/UserData—is automating the configuration of the operating system, applications, and services upon instance launch. If configuration management or bootstrap scripts are not in play, then it is likely that a golden-image is being utilized. Without one of these two approaches, you don’t have true Auto Scaling because it would require some kind of human interaction to configure a server (ergo, not “auto”) every time a new instance was created.

Both approaches (launch-time configuration vs. golden-image) have their pros and cons. I generally prefer launch-time configuration as it allows for more flexibility, provides for better governance/compliance, and enables pushing changes dynamically. But…(and this is especially true of Windows servers) sometimes launch-time configuration simply takes longer to happen than is acceptable, and the golden-image approach must be used to allow for a more rapid deployment of new Auto Scaling group instances.

Either approach can be easily automated using a solution like to the one I am about to outline, and thankfully AWS publishes new stock Windows Server AMIs immediately following every Patch Tuesday.  This means, if you aren’t using a golden-image, patching your instances is as simple as updating your Auto Scaling Launch Configuration to use the new AMI(s) and preforming a rolling replacement of the instances. Even if you are using a golden-image or applying some level of customization to the stock AMI, you can easily integrate Packer into the process to create a new patched image that includes your customizations.

The Solution

At a high level, the solution can be summarized as:

  1. An Orchestration Layer (e.g. AWS SNS and Lambda, Jenkins, AWS Step Functions) that detects and responds when new patched stock Windows AMIs have been released by Amazon.
  2. A Packer Launcher process that manages launching Packer jobs in order to create custom AMIs. Note: This step is only required If copying AWS stock AMIs to your own AWS account is desired OR if you want to apply customization to the stock AMI. Either use case requires that the custom images are available indefinitely. We solved this problem by creating a Packer Launcher process by creating an EC2 instance with a Python UserData script that launches Packer jobs (in parallel) to create copies of the new stock AMIs into our AWS account. Note: if you are using something like Jenkins, this could be handled by having Jenkins launch a local script or even a Docker container to manage launching Packer jobs.
  3. A New AMI Messaging Layer (e.g. Amazon SNS) to publish notifications when new/patched AMIs have been created
  4. Some form of an Auto Scaling Group Rolling Updater will be required to replace exiting Auto Scaling Group instances with new ones based on the Patched AMI.

Great news for anyone using AWS CloudFormation… CFT inherently supports Rolling Updates for Auto Scaling Groups! Utilizing it requires attaching an UpdatePolicy and adding a UserData or cfn-init script to notify CloudFormation when the instance has finished its configuration and is reporting as healthy (e.g. InService on the ELB). There are some pretty good examples of how to accomplish this using CloudFormation out there, but here is one specifically that AWS provides as an example.

If you aren’t using CloudFormation, all hope is not lost. With Hashicorp Terraform’s ever increasing popularity for deploying and managing AWS infrastructure as code, Terraform has still yet to implement a Rolling Update feature for AWS Auto Scaling Groups. There is a Terraform feature request from a few years ago for this exact feature, but as of today, it is not yet available, nor do the Terraform developers have any short-term plans to implement it. However, several people (including Hashicorp’s own engineers) have developed a number of ways to work around the lack of an integrated Auto Scaling Group Rolling Updater in Terraform. Here are a few I like:

Of course, you can always roll your own solution using a combination of AWS services (e.g. SNS, Lambda, Step Functions), or whatever tooling best fits your needs. Creating your own solution will allow you added flexibility if you have additional requirements that can’t be met by CloudFormation, Terraform, or other orchestration tool.

The following is an example framework for performing automated Rolling Updates to Auto Scaling Groups utilizing AWS SNS and AWS Lambda:

a.  An Auto Scaling Launch Config Modifier worker that subscribes to the New AMI messaging layer performs an update to the Auto Scaling Launch Configuration(s) when a new AMI is released. In this use case, we are using an AWS Lambda function to subscribe to an SNS topic. Upon notification of new AMIs, the worker must then update the predefined (or programmatically derived) Auto Scaling Launch Configurations to use the new AMI. This is best handled by using infrastructure templating tools like CloudFormation or Terraform to make updating the Auto Scaling Launch Configuration ImageId as simple as updating a parameter/variable in the template and performing an update/apply operation.

b.  An Auto Scaling Group Instance Cycler messaging layer (again, an Amazon SNS topic) to be notified when an Auto Scaling Launch Configuration ImageId has been updated by the worker.

c.  An Auto Scaling Group Instance Cycler worker that will perform replacing the Auto Scaling Group instances in a safe, reliable, and automated fashion. For example, another AWS Lambda function that will subscribe to the SNS topic and trigger new instances by increasing the Auto Scaling Desired Instance count to a value of twice the current number of ASG instances.

d.  Once the scale-up event generated by the Auto Scaling Group Instance Cycler worker has completed and the new instances are reporting as healthy, another message will be published to the Auto Scaling Group Instance Cycler SNS topic indicating scale-up has completed.

e.  The Auto Scaling Group Instance Cycler worker will respond to the prior event and return the Auto Scaling group back to its original size which will terminate the older instances leaving the Auto Scaling Group with only the patched instances launched from the updated AMI. This assumes that we are utilizing the default AWS Auto Scaling Termination Policy which ensures that instances launched from the oldest Launch Configurations are terminated first.

NOTE: The AWS Auto Scaling default termination policy will not guarantee that the older instances are terminated first! If the Auto Scaling Group is spanned across multiple Availability Zones (AZ) and there is an imbalance in the number of instances in each AZ, it will terminate the extra instance(s) in that AZ before terminating based on the oldest Launch Configuration. Terminating on Launch Configuration age will certainly ensure that the oldest instances will be replaced first. My recommendation is to use the OldestInstance termination policy to make absolutely certain that the oldest (i.e. unpatched) instances are terminated during the Instance Cycler scale-down process.  Consult the AWS documentation on the Auto Scaling termination policies for more on this topic.

In Conclusion

Whichever solution you choose to implement to handle the Rolling Updates to your Auto Scaling Group, the solution outlined above will provide you with a sure-fire way to ensure your Windows Auto Scaled servers are always patched automatically and minimize the operational overhead for ensuring patch compliance and server security. And the good news is that the heavy lifting is already being handled by AWS Auto Scaling and Hashicorp Packer. There is a bit of trickery to getting the Packer configs and provisioners working just right with the EC2 Config service and Windows Sysprep, but there are a number of good examples out on github to get you headed in the right direction. The one I referenced in building our solution can be found here.

One final word of caution... if you do not disable the EC2Config Set Computer Name option when baking a custom AMI, your Windows hostname will ALWAYS be reset to the EC2Config default upon reboot. This is especially problematic for configuration management tools like Puppet or Chef which may use the hostname as the SSL Client Certificate subject name (default behavior), or for deriving the system role/profile/configuration.

Here is my ec2config.ps1 Packer provisioner script which disables the Set Computer Name option:

$EC2SettingsFile="C:\\Program
Files\\Amazon\\Ec2ConfigService\\Settin
gs\\Config.xml"
$xml = [xml](get-content
$EC2SettingsFile)
$xmlElement =
$xml.get_DocumentElement()
$xmlElementToModify =
$xmlElement.Plugins
foreach ($element in
$xmlElementToModify.Plugin)
{
if ($element.name -eq
"Ec2SetPassword")
{
$element.State="Enabled"
}
elseif ($element.name -eq
"Ec2SetComputerName")
{
$element.State="Disabled"
}
elseif ($element.name -eq
"Ec2HandleUserData")
{
$element.State="Enabled"
}
elseif ($element.name -eq
"Ec2DynamicBootVolumeSize")
{
$element.State="Enabled"
}
}
$xml.Save($EC2SettingsFile)

Hopefully, at this point, you have a pretty good idea of how you can leverage existing software, tools, and services—combined with a bit of scripting and automation workflow—to reliably and automatically manage the patching of your Windows Auto Scaling Group EC2 instances!  If you require additional assistance, are resource-bound for getting something implemented, or you would just like the proven Cloud experts to manage Automating Windows Patching of your EC2 Autoscaling Group Instances, contact 2nd Watch today!

 

Disclaimer

We strongly advise that processes like the ones described in this article be performed on a environment prior to production to properly validate that the changes have not negatively affected your application’s functionality, performance, or availability.

 This is something that your orchestration layer in the first step should be able to handle. This is also something that should integrate well with a Continual Integration and/or Delivery workflow.

 

-Ryan Kennedy, Principal Cloud Automation Architect, 2nd Watch