Cloud Architecture Strategy, Success & ROI from junior Sys Admin to CEO

Tags

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

NOTE: This paper is meant to be read in chunks, not in one long sitting

 

Overview

Revolution – the Cloud?

Strategy in the Cloud / Virtualization Arena

Strategy

Data Protection and Compliance

Cloud Infrastructure

Education & Creativity in any Environment

A Few Cloud Tools to Start With

Compute Instances

Storage

Load Balancers

WAF

VPC

S3 & Glacier & Google Nearline

Redshift

CloudFormation

CloudWatch

CloudTrail

Trusted Advisor

IAM

Data Centers and Cloud

Conclusions, for now…

Intelligence and Law Enforcement

FEMA

 

 

Overview

In tackling this personal transition into the Cloud/Virtualization strategy and architecture realm, it dawned on me that in addition to all the studying and gaining AWS certifications, I would also need to write more papers on the cloud to prove my knowledge base since I did not have that “day-to-day” practical ‘hands-on’ cloud work experience many companies want (i.e. 2 – 10 years).

Firstly, this paper is meant to be read and enjoyed while at the same time, prodding, provoking and tickling the gray matter in both hemispheres of your brain, to envisage, possibly, some things you had not considered hitherto (yes, a good word, eh?). This paper is about looking down the road, long-term to visualize where a company might wish to be in the cloud. Moving to the cloud is not something anyone should take lightly AND, everyone should be aware there are multiple areas where a company will have to pay for various cloud services. It is a great deal different than having a data center where you pretty much know the exact CapEx and OpEx year over year.

Take note, this paper is not just for upper management and the executive ranks, this paper is also for you out there who are the day-to-day members toiling away on your GUI dashboards (Azure, HPE, AWS, etc.) or performing CLI (old time mainframe workers should know this set up from working on the old TSO screens or even Cisco router engineers using IOS commands) calls to get your job done. It is for you because you, the workers, get to intimately see the ins and outs of what is really happening – you know what is going on and how amazing the interlocking cloud processes work to speed a specific function along to fruition.

You and your company will have to do more research to prepare for in gaining a cloud footprint. Then, as the year goes on, you may have to re-evaluate what you are paying for on the firm’s bottom line – that ‘in the black’ ROI every CFO, COO, CEO is hoping to achieve.

Revolution – the Cloud?

Yes, revolution in the Cloud, or Virtualization. This is because it is not just some everyday game changer; it is “The Game Changer”. Every success oriented individual on the planet should know this by now. It is not going to go away, it is only to improve and get better as time marches on… At some point in the very near future, the cloud will not only touch many people, it will touch ‘everyone’s’ lives in some facet or fashion. “You” will be affected the cloud in any number of ways, from:

  • IoT – the internet of things will become even more prevalent than it is today, overtly or covertly
  • Telemedicine – doctors, nurses and surgeons having a ‘face-to-face’ with you, anywhere
  • V2V – all vehicles (or rather that is the premise) will ‘talk’ to each other to avoid vehicular fatalities; or something as simple as communicating to other vehicles that ‘hey, my car has a tire blow out, please let me pull over immediately to the shoulder, to get out of traffic’ in order to avoid traffic jams or preventable accidents
  • Education – where HIGHER quality knowledge can be shared and obtained by anyone, anywhere in the world
  • Care takers – to keep an eye on elderly or medically incapacitated individuals
  • Children – security and safety as well as catching them ‘before’ the child can cause any mischief…
  • Political – where individuals can tap into the cloud to do their own research instead of relying on one news outlet (or one country’s news) to see what is going on – look at Russia and how Putin controls all of the media in that country (do you truly believe everything you see/hear on Russia Today?)

The cloud is going to be a continual revolutionary force as it cuts across race, gender, age, religious, political and economic strata around the globe.

Oh yes, do not forget about the militaristic impacts that will start to be felt as time moves on in ground, sea, air and space areas as they open up to more cloud utilities…

Strategy in the Cloud / Virtualization Arena

This paper is not strictly geared towards one specific vendor, although, my training and certifications are primarily on AWS.

Strategy

So, as you or your decision-makers look down the road to see what virtualization can do and where it can take you, you simply cannot only look at the short-term – you absolutely must look at the long-term as well. That long-term view is where you will make your ROI realizations come to fruition.

Your costs, benefits, productivity and success are going to spring from that longer view. Yes, you may reap some rewards in the short run but you have to be realistic to know that you will also flounder and lose some profit as you start winding your way through virtualization, the cloud.

Your micro and macro outlooks must be in sync and they must be realistic so you do not trapped by the Pie-In-The-Sky thought process and become disillusioned before you get to far in the cloud.

You will have to look at the various cloud / virtualization tools needed and used by individuals working in the cloud and those companies who are successful in embracing the cloud for better efficiency in reaching a highly successful annual ROI. Tell me, which will make your company’s brand/reputation stand out as a company that others will want to buy products from, work with or to invest in:

  1. A firm using only single, or even multiple data centers or
  2. A firm using data centers and/or virtualization technologies to house and share their data

Remember, if a data center goes down for any variety of reasons – fire, flooding, loss of power, you had best have a completely up to snuff disaster recovery / business continuity (DR/BC) plan in place.  That has been tested. And years ago, I have worked in a data center, as well as taking part in the latter stages of planning, deploying and testing. Take note, if you do have only a data center to work with, you will have to understand the phases of cold site, warm site and hot sites.

If a cloud infrastructure goes down, you have the option of ‘easily’ moving to your back up DR/BC plan. I forgot to mention that today; the cloud is basically code, or rather, Infrastructure as Code (IaC). Your firms’ infrastructure is comprised of YAML, JSON, Chef or other scripting language that makes your strategic vision come to life. If the east coast were to go under, your load balancers can easily switch over to your ‘cloud’ resources on the west coast or to Australia or to London…

You want your firm to be as nimble and agile as possible in order to weather the multiple fluctuations of day to day business and social media life. You want to:

  • Ensure your infrastructure can endure those last minute deluges of hits because some celebrity tweeted about your product. You want your firm to have that capability of elasticity everyone refers to in order to scale up your resources (computing instances, database instances, storage) during peak demands.
  • Be able to handle and overcome DDoS attacks and not suffer the fate of multiple entities (Brian Krebs or that incident at the Code Spaces ISP)

Data Protection and Compliance

With the growing levels of the need for data protection and compliance for all of the various national (i.e – HIPAA and PCI) and international (i.e. the European Union GPDR), are you equipped to protect all of your customers and users, globally. If you are an American firm doing business across the pond, or allowing those cousins over there to shop your store in the U.S., you will have to protect those customers according to the EU standard in addition to the U.S. standards.

How well are you going to protect your firms’ data?  Are you going to use encryption for data at rest, data in transit? Are you going to have SSL termination on your load balancers, or behind them?

What about on your databases, are you going to set up fine based granular access (FBGA) or more?

These are some of the areas you will need to consider, specifically encryption (laptop whole disk, database, SAN to name a few), long term to aid in preventing breaches. Or if you are breached, the hacker cannot do anything with the data as it will be useless to them (until they gain unfettered access to some Quantum based computer…).

Cloud Infrastructure

The hardware in use is mostly commodity components now or may use specialized RAM (i.e. the latest Intel 3D Xpoint collaboration with Micron), CPUs, storage (like the NVMe and PCIe to remove latencies in any server/storage communication). There are some remarkable exceptions out there, look at Googles’ TPU (Tensor Processing Unit), a massively parallel processing conglomeration to handle global voice queries when we use Google (ok google). 3D Xpoint uses 3 data bits per cell to create denser and faster storage.

Overall, the key is in the software, we keep coming back to that… The software algorithms with the use of specialized components, as mentioned above, continue to get better (faster, smarter, smaller & much less expensive).

Companies like AWS now allow you to RE-SIZE your storage volumes and computing instances pretty much on the fly now instead of having to:

  • Take a snapshot, stop & detach the storage volume and then add/attach a newer, larger volume or
  • Do more programming to gain a larger, faster compute instance – you almost have real time continuity if you need to gain that better instance.

Besides those areas above, you have to determine what kind of setup you want. Do you want:

  1. IaaS – Where a 3rd party vendor handles the hardware
  2. PaaS – Where you gain a platform to do developmental work for new applications
  3. SaaS – Where you obtain everything you need to run a business, software (and hardware) provided by a 3rd party vendor, you just pay for the use of that software

You have to decide how much of which venue you want to control and manage and how comfortable you are with someone else controlling hardware/software they run for your use. I.E. – such as a vendor managing your encryption keys.

You have to understand your strategy, from inception to implementation. Take application migration; data migration; or database migration as examples. You will need to take into account the time necessary to undertake a database migration, say from Oracle to Aurora on AWS (there is supposed to be a large saving in moving to Aurora). There are two phases to it (besides the planning and testing) – there are the data migration itself and there is the database schema as well. So plan accordingly…

You will have to understand that your initial costs will be higher and that you will recoup those costs over time, even when your firm consumes more services:

  • CloudWatch (logging any and all of the resources being used)
  • CloudTrail (audit to see who is trying to access what and from where)
  • CDN Queries (Cloud Distribution Network – for getting content to consumers globally)

Education & Creativity in any Environment

Look, virtualization is knocking down the previously insurmountable barriers to knowledge and to business success. There can no more excuses to connect to the ‘Net and gain knowledge, that is, in most western developed countries. We have to continue to work on helping those in developing nations and those in repressive/authoritarian countries to gain access to the world – to see the truth that is out there and not the completely sanitized content these authoritarian (okay, dictatorial) countries and leaders want their citizens to see.

The problem however, comes in where a company, for the benefit of its shareholders and executives, prefer profits over open and equitable access to all. This kind of impediment will and does prevent a faster spread of knowledge, free flowing collaboration. One aspect of this is where the U.S. may see the end of net neutrality in order to maximize profits for larger corporations. This paper is not going to go into any depth on this topic.

On the positive side, virtualization already opens doors to more innovation and collaboration and inclusiveness – this will continue to grow over time. And it may even accelerate as we move forward.

Every day, more and more individuals around the globe are realizing the benefits of collaborating with others who may bring a different aspect to a problem set that creates successful synergy to the group in resolving that problem. Yes, synergy and creativity, those are two wonderful words (for this writer at any rate) that spring from education and knowledge. The more cognitive diversities that can be introduced to a challenge, the more possibilities can be put on the table to fix something that is highly complex.

There are multiple areas of where we see and can look forward to in using the cloud. Scientific, Finance, Health Care, Energy and Infrastructure (nuclear, water, power) are some of the other primary areas to use the creativity of all the people involved. These are areas that need to be in the cloud to realize cost savings and benefits. Benefits such as collaborating on database usage, not small isolated databases but national / international databases (i.e. database warehouse) where people can pull/share unique/valuable info to visualize (yes, 3D VR or AR) fixes in multiple areas, such as:

  • Chemistry
  • Microbiology
  • Physics (astrophysics)
  • Bioethics
  • Geriatrics

A Few Cloud Tools to Start With

Now, using various tools in and for the cloud is not some event where your firm has an intention of eliminating a mass number of personnel in moving to the cloud. No, that is not the case. It is having better tools to use for your cloud success as well as in better education for your employees as you move to and work in the cloud environment.

Using the cloud means you want to be able to scale when you need to and be able to use as powerful compute, storage and networking components you can afford in order to make a profit for the firm.

Using the cloud also means you can have multiple areas of security in your infrastructure, from the VPC to CloudFront’s CDN set up; using WAFs at different points; using query string authentication; S3 ACL for buckets or bucket policies;

The following is not nearly all inclusive, far too many areas to cover in great depth here.

Compute Instances:

You are going to want to know, whether you can use compute instances for a:

  • short term (on-demand or spot) or whether you can use for critical use cases or impromptu situations for demonstrations of some product
  • reserved type instances to gain better pricing over 1 year or 3 years – steady, consistent use every day or every week day
  • can they be optimized to handle large workloads

Storage:

You will want to know about the storage types, for daily use or for compliance:

  • what kind of encryption – who manages the keys
  • can the storage be modified at will
  • how fast, or, how optimized is that storage to work with the compute instances
  • and you will need to know whether you want magnetic storage (slower but less expensive), SSD (faster), optimized SSD (higher, faster throughput) or optimized serial magnetic storage (not good for booting up with but for fast serial transactions)

Load Balancers:

  • Elastic load balancer (classic) ELB – that works primarily at layer 4 or an
  • Application load balancer (newer) ALB – that can work at layer 7 (http / https) and can deal effectively with DDoS attacks

WAF:

A web application firewall you can use at your CDN or in your VPC where your various cloud tiers reside (web, application or database) to filter access based on:

  • IP or location
  • Conditions – to allow or deny
  • Strings that your application is looking for
  • Suspicious behavior

VPC:

A virtual private cloud where you can isolate components of that VPC (web, application or database) and:

  • Allow only certain users or IPs access
  • Have private high speed, extremely low latency between corporate assets
  • Set up network access lists to deny / allow specific IPs (individual, subnets) to certain VPC components
  • Set up security groups to allow (explicitly) users, groups, protocols, etc to certain VPC components
  • VPC endpoints to ensure secure connections within the VPC
  • VPC Peering – where you can have certain branches of your corporation connect to each other and share certain data
  • VPC Flow logs to either, complement or replace IDS/IPS instances/agents and to use in conjunction with CloudWatch alarms

S3 & Glacier & Google Nearline:

Storage options that you can select and pay for what your firm requires:

  • Google and AWS are constantly going back and forth as to the better costs for storage, real time and archival
  • You can set up version control to secure and prevent accidental deletions and for compliance purposes
  • What about frequent or infrequent
  • You can move Terabyte and Petabyte amounts of data from your on-premise location to the cloud – MUCH better than you could via a 10 or 100 MB fiber connection (try 1 Gb or 10 Gb connection… or even AWS Snowmobile {truck comes to you gain that huge amount of data you have})

Redshift:

  • Massively parallel processed data warehouse – at a terabyte and petabyte scale

CloudFormation:

  • This service is a money maker for many firms – this is where you build your cloud infrastructure (if you are not using OpsWorks or Elastic Beanstalk), this is the IaC mentioned previously – you have absolute control in how you can build your infrastructure and save those templates for use in other regions, in other countries – why reinvent the wheel when you can use these templates or tweak them (and save them) for other uses

CloudWatch:

  • To log all traffic or only specific traffic
  • To log traffic in some regions only or in all regions (the much better choice in order to detect suspicious and illegal data patterns)
  • You want this to more efficiently run your business, you can create dashboards to discover the heavier used data patterns; compute/storage/databases instances; or the less used resources

CloudTrail:

  • To audit API calls throughout all of your VPCs
  • To log traffic in some regions only or in all regions (the much better choice in order to detect suspicious and illegal activities)

Trusted Advisor:

  • To check and find out where you can save more money on underutilized resources
  • To discover potential security threats

IAM:

  • To create specific and explicit roles for various staff members and vendors and partners
  • To create roles with specific permissions, using the AWS managed policies and inherent permissions or create your own inline policies and permissions
  • To create cross account (other AWS account holders) policies / roles to access your VPC resources

Etc.

Data Centers and Cloud

Basically, you want to create resiliency and redundancies across your AWS or other cloud vendor resources. When you talk to your firm’s CEO, President, CFO, CIO and others, you want to be able to talk up how your firm can withstand and flourish in different scenarios by using certain cloud infrastructures.

You will want to show them that ‘yes, there are charges for multiple services in any cloud/virtualization venture.’ But you should also have a companion slide set of how data centers have their own possible hiccups and problems and expenses.

  • Hardware – you have to budget for better servers every two to three years to take advantage of energy use and heat output and faster CPUs – are you taking advantage of, or even following, the technological trends in hardware to save your firm money in latency, energy savings, productivity – i.e. NVMe, PCIe, 3D NAND gains (smaller and smaller dies – 10 nm), more specialized ASICs, more energy efficient servers, lighter & thinner & more efficient desktop monitors, etc., etc., etc.
  • Licensing – you have to renew these on an annual or every several period for your software applications
  • Space – you need space for the servers (or mainframes)
  • Cooling – do you know about the heat island effect in a data center, where the hot and cold spots are – do you regulate it properly – can you cool your servers down adequately – cooling IS expensive
  • Disaster Recovery – do you have a hot site with duplicate equipment (and the requisite licensing and servers and cooling)
  • What about emergencies, does data center have a fire fighting or power outage process in place
  • What about your data communication lines, are you set up to handle network outages, in your data center or across the globe via different TelCos, IXP (i.e. LINX in London or Boston Internet Exchange)

With cloud infrastructures, well, let’s just say you do not have to worry about equipment/software obsolescence. The cloud vendor will always try to stay on their game and keep you as a customer satisfied by having the latest hardware and software versions possible.

You just have to ensure that your contract with them covers this as well as much of the services they offer. You are ultimately the one that needs to understand and be aware of where you can and will be charged along with what kind of capabilities you will gain.

You will need to know what kind of response time for the different problems your cloud may encounter (i.e. network latency, storage access issues, corrupted compute instances that are not caused by you).

Conclusions, for now…

Your organization is only as good as the people that you have onboard. The same goes for the tools those people are able to employ as they need. If you want data (on anything) ‘now’ – you have to equip your staff with better tools and not scrimp. If you scrimp on any of the necessary cloud tools your people need, you should already know what kind of end results you will accrue.

You must recognize that the strategic advantages is most always going to go to the firm with the better or best tools (not referring to bleeding edge). If you attempt to do any effort on the cheap, as many of you know, that is going to be the end result of what your staff provide will provide you.

For those of you who might believe ‘the Cloud’ is:

  1. Not going to work
  2. Not good enough or if
  3. You are waiting for it to become better

You can no longer wait. “The Cloud” is here to stay. The technology is a given at this point. Look at IBM (Watson and IBM’s cloud), Microsoft (Azure), Google Cloud, Amazon (AWS), Rack Space, HPE; these firms and others working in the IaaS, PaaS and SaaS spaces already know the promise of this technology.

Bottom line, well, pun intended – the bottom line is, success. AND if your firm is doing a ‘Good’ to ‘GREAT’ job, there should be a fair modicum of profit for the company (employees, shareholders, partners AND customers). If the firm is doing a great job, it means the firm is using the right and best tools along with the productivity of great employees. AND, there I go again, AND it means that your customers are coming back to your store front (brick and mortar OR web page OR CDN) for more AND spreading the word, enhancing your brand / reputation.

Furthermore, it means your firm’s employees across all divisions are able to:

  • see trouble in advance, such as using monitoring tools to see patterns and failures and fix a problem before it becomes magnified
  • management can use predictive analysis tools from the various cloud tools – something like Kinesis and Elastic MapReduce and even using Aurora and RedShift – to discover potential problems in the supply chain, a continent away maybe because a CloudWatch alarm kicked in and sent an SNS message
  • Customers can do their own API calls to gain answers to a possible problem
  • And on, and on, and on….

Intelligence and Law Enforcement

The implication for law enforcement and national security intelligence agencies is staggering. Think about how they can likely collaborate and synthesize what they have in order to prevent or stop (at least significantly minimize) future attacks on people and cities.

This type of scenario has been a dream for years/decades (and not to worry, this is all information available in the public domain for years, no secrets here) – to share data. What better way to do so than using the cloud to tie together all the information in databases and then based on the individuals’ need to know accesses, allow those analysts/agents to pull data to make their analysis. And then allow them to share that analysis across the spectrum so that “other” agents and analysts do not have to completely reinvent the wheel. To date, some of those similar databases have been siloed, meaning you cannot always see or find critical data you are looking for (I have seen this from experience).

Remember the ‘Underwear Bomber’ who set himself on fire? Well, part of the delay in uncovering this idiot was that he was in several databases, because multiple agencies typed in his Arabic name using different versions of the name (i.e. Farook, Farruk, Farruuk, etc). No wonder it took forever to find more info on him…

FEMA

Then you have to look at FEMA (Federal Emergency Management Agency). This is an institution that could use cloud based resources to manage and more efficiently handle local, state and/or federal situations.

  • Database in the cloud for missing people – individuals anywhere could plug in information on their missing family members and have this information replicated in near real time across all connected databases with those missing members being able to jump on the cloud database from somewhere else to indicate they are alive and well and where they can be found
  • Emergencies – floods, earthquakes, fires – any FEMA member (firemen, police, FEMA employees, DHS agents) can set up, strategize and manage any of these scenarios virtually instead of trying to meet somewhere that is impassable at some location, meaning that all members will not be able to attend in a timely manner
  • And, well, you get the gist of where this can successfully go for FEMA

What was covered in this short paper (in comparison to what more could have been covered in more depth) is just the tip of the iceberg for your corporations cloud journey.

The Cloud is knocking and it is up to you to see where it can take you, your company and employees. To have near instant access for your employees, customers, partners, executive staff and shareholders is amazing. To be able to gain deep insight into any part of your cloud in near real time is also amazing.

Sure, call me starry eyed. I will admit it, I am just a hyped up, bed rock solid tech guy that immediately (with eyes open) and easily adapts to new forks in the technological road. I am one of those who can realistically see what’s coming down the road. Why not optimize the way a firm does business and bring the employees of that company along with them? As well as the customers who enjoy doing business with that company. Why continue to work with outdated tech?

You will have to bite the bullet and pull the plug at some point and recognize it as sunk costsThen you can revisit and recapitalize your CapEx and OpEx costs for a better business future…

  

 

Definitions

AR      Augmented Reality

CDN    Content Distribution Network – where you push content from to customers who are calling for it (i.e. via API calls)

GDPR  General Data Protection Regulation (EU)

HPE    Hewlett-Packard Enterprise

HIPAA Health Insurance Portability and Accountability Act (medical / privacy)

IaaS    Infrastructure as a Service – a form of cloud computing that provides virtualized computing resources over the Internet – a third-party provider hosts hardware, software, servers, storage and other infrastructure components on behalf of its users. Users are responsible for managing applications, data, run-time, middleware, and OSes.

IXP     Internet exchange point or just IX – a physical infrastructure through which Internet service providers (ISPs) and Content Delivery Networks (CDNs) exchange Internet traffic between their networks

PaaS   Platform as a Service – Used for applications, and other development, while providing cloud components to software. What developers gain with PaaS is a framework they can build upon to develop or customize applications. PaaS makes the development, testing, and deployment of applications quick, simple, and cost-effective. With this technology, enterprise operations, or a third-party provider, can manage OSes, virtualization, servers, storage, networking, and the PaaS software itself. Developers, however, manage the applications. 

PCI     Payment Card Industry – more formally known as the PCI-DSS (Data Security Standard) for various levels of credit card security – levels 1 – 4 (1 is for smaller merchants with 4 for the largest global multinationals)

SaaS   Software as a Service – The largest cloud market and it is still growing quickly. SaaS uses the web to deliver applications that are managed by a third-party vendor and whose interface is accessed on the clients’ side. Most SaaS applications can be run directly from a web browser without any downloads or installations required, although some require plugins. i.e. Customer Relationship Management applications like Salesforce, productivity software suites like Google Apps, and storage solutions brothers like Box and Dropbox. 

VR      Virtual Reality