When it comes to managing and monitoring IT spending, the cloud has created a new layer of complexity. Consider the fact that AWS provides as many as 28,000 service offerings, generating up to millions of billing line items each month. This creates budgeting and planning problems for CIOs because there’s no easy way to interpret what percentage of cloud spending is going toward storage, compute, or network, along with specific applications, projects and services. IT departments also need a way to merge cloud costs with on-premise IT costs to see the full picture of infrastructure spending across key categories. As well, with many individuals from different departments procuring their own AWS resources, a company can have dozens of unmanaged and unlinked accounts. This creates gaps in financial tracking and spend management and prevents a company from taking advantage of volume discounts.
IT needs a unified model to categorize cloud and non-cloud costs together, and automation to map line items into the IT cost model each month. To automate the mapping process, 2nd Watch and Apptio have worked together on a mapping table that specifies where each Amazon product fits within a standard cost model. This mapping is now embedded in the Apptio Cost Transparency application, a solution for integrating AWS usage with billing, cost categorization, modeling of total costs including internal labor and self-service analytics. This allows IT organizations to categorize cloud costs into trackable categories such as Cloud Windows in Compute or Cloud Archive in Storage.
Determining the run costs of an application is another goal, and requires mapping cloud resources, such as servers and storage, to individual applications. Many IT organizations have not yet adapted their management processes to track application relationship data for cloud infrastructure. Linked accounts and tagging are two ways to get around these hurdles on AWS.
Many enterprises have several AWS accounts at the team and departmental levels in order to encourage agility, but these unlinked accounts create gaps in cost and operational management. To unify unlinked accounts across an organization, companies can use the Apptio application to link individual accounts into one “master account” paid through an IT cost center. This provides visibility into enterprise spend on AWS yet still maintains business-unit level tracking. It also enables savings with volume discounts, that’s not possible when spending goes across several individual AWS accounts.
AWS tags help group usage and expenses across shared key resources like databases. Tagging helps accomplish the problem of mapping AWS resources back to specific business projects, such as “Marketing Web Staging” and “Marketing Web Production.” Detailed tagging can help answer questions such as, how much of an entire application portfolio is comprised of AWS services or what percentage of which projects are using cloud resources? One thing to keep in mind is that AWS tags are applied only to individual accounts. AWS tagging is ideal for environments where you need to share resources across multiple workloads.
There are some limitations to this manual approach for managing individual accounts and tagging, however – managing numerous logins and passwords, going through the AWS setup process for each individual account, creating and controlling a tagging schema, etc. For a more scalable approach to managing AWS accounts and tagging, consider solutions like our 2W Insight billing application, which enables grouping of tags across AWS accounts and provides tools to track and analyze cloud costs by cost center, business unit department, etc. For more information on 2W Insight, contact us.
To learn more about best practices for managing and tracking cloud spending, download our Analyzing Cloud Costs white paper.
-Jeff Aden, EVP, Marketing & Strategic Business Development
AWS enables enterprises to trade capital expense for variable expense, lower operating costs and increase speed and agility. As enterprises begin to deploy cloud services across their business, it is critical to have a standardized approach to allocate usage costs to the appropriate department or cost center. By tracking costs at the cost center level, Enterprises gain visibility throughout their organization – and specifically who is spending precious IT funds.
To allocate costs, usage must first be grouped. AWS provides two methods to group usage; Resources Tags and AWS accounts. Each method is useful but also comes with downsides.
Using AWS Tagging to group usage
- Grouping by tag enables enterprises to run all of their workloads (applications) in a single AWS account, simplifying management within the AWS console.
- A tagging schema needs to be created, universally deployed and tightly controlled.
- Care has to be taken to ensure all individual AWS resources are tagged properly as any mistake in tagging will cause a resource to be left out of the group and not reported properly.
- Many AWS resources are un-tagable, which will require the creation and maintenance of a separate cost distribution scheme to allocate those costs across the enterprise.
- Reserved Instance (RI) discounted usage pricing cannot be linked to a single tag group and can result in significant costing inaccuracies.
Using Multiple AWS Accounts to group usage
- Using individual AWS accounts for each workload provides the most accurate and detailed reporting of costs and usage.
- By creating a separate AWS account for each workload, enterprises can track all associated costs (including RIs) and allocate them to cost centers, departments and/or business units.
- When using AWS accounts to group usage, each account must be manually set up.
- There is no method of sharing resources, such as databases, with multiple workloads as each workload is located in separate AWS accounts.
Given the challenges of both “account based” and “tag based” grouping, we have found that the tracking methodology needs be aligned to the applications or workloads. For deployments where the resources are 100% dedicated to a specific workload, grouping by AWS accounts is ideal as it is the only way to ensure fully accurate costing. Using AWS tagging should be used when you need to share resources across multiple workloads, however enterprises must note that the costing will not be 100% accurate when using tag groups.
Tracking and Allocating Costs for Workloads with Dedicated Resources
As stated above, workloads that do not need to share resources should be set up in unique AWS accounts. This can be accomplished by establishing individual AWS accounts for each workload and mapping them directly to your enterprise organizational structure. The example below illustrates how a complex enterprise can organize its cloud expenses and provide showback or chargeback reports across the enterprise.
In this example, the enterprise would receive two bills for their cloud usage – Business Unit 1 and Business Unit 2. Under each business unit there are multiple levels of cost centers that roll up to each subsequent higher level – which is typical with many Enterprise organizations. In this example, AWS accounts are created for each project/workload then rolled up to provide consolidated usage information by department and business unit. This type of structure enables:
- The owners at the “resources and workload cost accrual and tracking” levels to track their individual usage by AWS accounts, which captures 100% of the cost associated with each AWS account
- The management of department level to view the consolidated usage for their respective cost centers and workloads
- The management of each business unit to view usage by department and AWS account and receive a bill for its entire consolidated usage
This provides a reliable and accurate methodology to track and allocate cloud usage based on your distinct enterprise organizational structure. It does, however, require a disciplined approach to creating new projects and updating your expense management platform to provide executive-level dashboards and the ability to drill-down to detailed consumption reports by cost center. This enables Enterprise IT to provide executive-level transparency while keeping excessive resource consumption under control and reduce IT costs.
Tracking and Allocating Costs for Workloads with Shared Resources
In many organizations there is a need to share key resources, such as databases, across multiple workloads. In these cases it is a best practice to use AWS tags to group your expenses. This method requires careful set up of resources and the creation of a schema to allocate shared resources and resources that cannot be tagged across the enterprise.
Tagging allows enterprises to assign its own metadata to each tag-able resource. Tags do not have any semantic meaning to AWS resources and are interpreted strictly as a string of characters. Tags are made up of both a “Key” and a “Value”. AWS allows up to 10 Keys for each resource, and each Key can have can have unlimited values enabling very detailed grouping of resources. Tagging should be set up based on the needs of the organization and the AWS architecture design. The image below illustrates how to establish a tagging scheme for a 2-Tier Auto-scalable Web Application.
As the project moves from Web Sandbox to Web Staging to Web Production, you can use tags to track usage. When the application is in the Sandbox all resources are tagged with the key “Web Sandbox” and the appropriate value (Environment, Owner, App and/or IT Tower). When the project moves to “Web Staging” you simply replace the original key and values with the ones associated with the next step in development.
While there is no one-size-fits-all solution to AWS expense management, deploying one or both of these methods can provide you the visibility necessary to successfully meet the tracking and analytical needs of your enterprise.
-Tim Hill, Product Manager
As firms progress through the transition from traditional IT to the AWS Cloud, there is often a moment of fear and anxiety related to managing cost. The integration and planning group has done an excellent job of selecting the right consulting partner. Contracts have been negotiated by legal and procurement. Capital funding has been allocated to cover the cost of application migrations. Designs are accepted and the project manager has laid out the critical path to success. Then at the last hour, just before production launch, the finance team chimes in – “How are we going to account for each application’s monthly usage?”
So much planning and preparation is put into integration, because we’ve gone through this process with each new application. But moving to the public cloud presents a new challenge, one that’s easily tackled with a well-developed model for managing cost in a usage-based environment.
AWS allows us to deploy IaaS (Infrastructure as a Service), and that infrastructure is shared across all of our applications in the cloud. With the proper implementation of AWS Resource Tags, cloud resources can be associated with unique applications, departments, environments, locations and any other category for which cost-reporting is essential.
Firms must have the right dialog in the design process with their cloud technology partner. Here’s an outline of the five phases of the 2nd Watch AWS Tagging Methodology, which has been used to help companies plan for cloud-based financial management:
Phase 1: Ask Critical Questions – Begin by asking Critical Questions that you want answered about utilization, spending and resource management. Consider ongoing projects, production applications, and development environments. Critical Questions can include: Which AWS resources are affecting my overall monthly bill? What is the running cost of my high availability features like warm standby, pilot light or geo-redundancy? How do architectural changes or application upgrades change my monthly usage?
Phase 2: Develop a Tagging Strategy – The Cloud Architect will interpret these questions and develop a tagging strategy to meet your needs. The strategy then turns into a component of the Detailed Design and later implemented during the build project. During this stage it’s important to consider the enforcement of standards within the organization. Configuration drift is when other groups don’t use standardized AWS Resource Tags, or those defined in a naming convention. Later when it’s time for reporting, this will create problems for accounting and finance.
Phase 3: Determine Which AWS Resources Are In Scope – Solicit feedback from your internal accounting department and application owners. Create a list of AWS Resources and applications that need to be accounted for. Refer frequently to AWS online documentation because the list of taggable resource types is updated often.
Phase 4: Define How Chargebacks and Showbacks Will Be Used – Determine who will receive usage-based reports for each application, environment or location. Some firms have adopted a Chargeback model in which the accounting team bills the internal groups who have contributed to the month’s AWS usage. Others have used these reports for Showback only, where the usage & cost data is used for planning, forecasting and event correlation. 2W Insight offers a robust reporting engine to allow 2nd Watch customers the ability to create, schedule and send reports to stakeholders.
Phase 5: Make Regular Adjustments For Optimization – Talk to your Cloud Architect about automation to eliminate configuration drift. Incorporate AWS tagging standards into your cloud formation templates. Regularly review tag keys and values to identify non-standard use cases. And solicit the feedback of your accounting team to ensure the reports are useful and accurate.
Working with an AWS Premier Consulting Partner is critical to designing for best practices like cost management. Challenge your partner and ask for real-world examples of AWS Resource Tagging strategies and cost reports. Planning to manage costs in the cloud is not a step that should be skipped. It’s critical to incorporate your financial reporting objectives into the technical design early, so that they can become established, standardized processes for each new application in the cloud.
For more information, please reach out to Zachary Bonugli firstname.lastname@example.org.
– Zachary Bonugli, Global Account Manager
In an effort to simplify the Reserved Instances (RI) model, AWS announced yesterday a change in the model based on customer feedback and purchasing patterns.
AWS will move from three types of RIs – Fixed Price: Heavy, Medium and Light Utilization RIs – to a single type with three payment options. All continue to provide capacity assurance and discounts when compared to On-Demand prices.
The three new payment options give you flexibility to pay for the entire RI upfront, a portion of the RI upfront and a portion over the term, or nothing upfront and the entire RI over the course of the term.
What does this mean for you? These changes will really benefit predictable workloads that are running >30% of the time. In cases where usage is less consistent, it may be better for companies to stick with on-demand rates. We’ve developed some related research on usage trends. Meanwhile, our role as a top AWS partner continues to be simplifying procurement of all AWS products and services.
Download the AWS Usage infographic
Read more about the new RI model.
One of the things “everyone knows” about migrating to the Cloud is that it saves companies money. Now you don’t need all those expensive data centers with the very physical costs associated with them. So companies migrate to the Cloud and are so sure they will see their costs plummet… then they get their bill for Cloud usage and experience sticker shock. Typically, this is when our customers reengage 2nd Watch – they ask us why it costs so much, what they can do to decrease their costs, and of course everyone’s favorite – why didn’t you tell me it would be so much?
First, in order to know why you are spending so much you need to analyze your environment. I’m not going to go into how Amazon bills and walk you through your entire bill in this blog post. That’s something for another day perhaps. What I do want to look into is how to enable you to see what you have in your Cloud.
Step one: tag it! Amazon gives you the ability to tag almost everything in your environment, including ELB’s, which was most recently added. I always highly recommend to my customers to make use of this feature. Personally, whenever I create something manually or programmatically I add tags to identify what it is, why it’s there, and of course who is paying for it. Even in my sandbox environment, it’s a way to tell colleagues “Don’t delete my stuff!” Programmatically, tags can be added through CloudFormation, Elastic Beanstalk, auto scaling, CLI, as well as third party tools like Puppet and Chef. From a feature perspective, there are very few AWS components that don’t support tags, and more are constantly being added.
That’s all well and good, but how does this help analytics? Tagging is actually is the basis for pretty much all analytics, and without it you have to work much harder for far less information. For example, I can tag EC2 instances to indicate applications, projects, or environments. I can then run reports that look for specific tags – how many EC2 instances are associated with Project X and what are the instance types? What are the business applications using my various RDS instances? – and suddenly when you get your bill, you have the ability to determine who is spending money in your organization and work with them on spending it smartly.
Let’s take it a step further and talk about automation and intelligent Cloud management. If I tag instances properly I can automate tasks to control my Cloud based on those tags. For example, maybe I’m a nice guy and don’t make my development team work weekends. I can set up a task to shutdown any instance with “Environment = Development” tag every Friday evening and start again Monday morning. Maybe I want to have an application only online at month end. I can set up another task to schedule when it is online and offline. Tags give us the ability to see what we are paying for and the hooks to control that cost with automation.
I would be remiss if I didn’t point out that tags are an important part of using some great 2nd Watch offerings that help manage your AWS spend. Please check out 2W Insight for more information and how to gain control over and visibility into your cloud spend.
-Keith Homewood, Cloud Architect
IT infrastructure is the hardware, network, services and software required for enterprise IT. It is the foundation that enables organizations to deliver IT services to their users. Disaster recovery (DR) is preparing for and recovering from natural and people-related disasters that impact IT infrastructure for critical business functions. Natural disasters include earthquakes, fires, etc. People-related disasters include human error, terrorism, etc. Business continuity differs from DR as it involves keeping all aspects of the organization functioning, not just IT infrastructure.
When planning for DR, companies must establish a recovery time objective (RTO) and recovery point objective (RPO) for each critical IT service. RTO is the acceptable amount of time in which an IT service must be restored. RPO is the acceptable amount of data loss measured in time. Companies establish both RTOs and RPOs to mitigate financial and other types of loss to the business. Companies then design and implement DR plans to effectively and efficiently recover the IT infrastructure necessary to run critical business functions.
For companies with corporate datacenters, the traditional approach to DR involves duplicating IT infrastructure at a secondary location to ensure available capacity in a disaster. The key downside is IT infrastructure must be bought, installed and maintained in advance to address anticipated capacity requirements. This often causes IT infrastructure in the secondary location to be over-procured and under-utilized. In contrast, Amazon Web Services (AWS) provides companies with access to enterprise-grade IT infrastructure that can be scaled up or down for DR as needed.
The four most common DR architectures on AWS are:
- Backup and Restore ($) – Companies can use their current backup software to replicate data into AWS. Companies use Amazon S3 for short-term archiving and Amazon Glacier for long-term archiving. In the event of a disaster, data can be made available on AWS infrastructure or restored from the cloud back onto an on-premise server.
- Pilot Light ($$) – While backup and restore are focused on data, pilot light includes applications. Companies only provision core infrastructure needed for critical applications. When disaster strikes, Amazon Machine Images (AMIs) and other automation services are used to quickly provision the remaining environment for production.
- Warm Standby ($$$) – Taking the Pilot Light model one step further, warm standby creates an active/passive cluster. The minimum amount of capacity is provisioned in AWS. When needed, the environment rapidly scales up to meet full production demands. Companies receive (near) 100% uptime and (near) no downtime.
- Hot Standby ($$$$) – Hot standby is an active/active cluster with both cloud and on-premise components to it. Using weighted DNS load-balancing, IT determines how much application traffic to process in-house and on AWS. If a disaster or spike in load occurs, more or all of it can be routed to AWS with auto-scaling.
In a non-disaster environment, warm standby DR is not scaled for full production, but is fully functional. To help adsorb/justify cost, companies can use the DR site for non-production work, such as quality assurance, ing, etc. For hot standby DR, cost is determined by how much production traffic is handled by AWS in normal operation. In the recovery phase, companies only pay for what they use in addition and for the duration the DR site is at full scale. In hot standby, companies can further reduce the costs of their “always on” AWS servers with Reserved Instances (RIs).
Smart companies know disaster is not a matter of if, but when. According to a study done by the University of Oregon, every dollar spent on hazard mitigation, including DR, saves companies four dollars in recovery and response costs. In addition to cost savings, smart companies also view DR as critical to their survival. For example, 51% of companies that experienced a major data loss closed within two years (Source: Gartner), and 44% of companies that experienced a major fire never re-opened (Source: EBM). Again, disaster is not a ready of if, but when. Be ready.
-Josh Lowry, General Manager – West