Consider the following scenario.
An Independent Software Provider (ISV) developed a financial application for a global investment firm that serves global conglomerates, leading central banks, asset managers, brokerage firms and governmental bodies. The development strategy for the application encompassed a thought through DevOps plan with cutting-edge agile tools. This has ensured zero downtime deployment at maximum productivity. The app now handles financial transactions in real-time at an enormous scale, while safeguarding sensitive customer data and facilitating uninterrupted workflow. One unfortunate day, the application crashed and this investment firm suffered a severe backlash (monetarily and morally) from its customers.
Here is the backstory:
Application’s workflow exchange had crossed its transactional threshold limit, and lack of responsive remedial action crippled the infrastructure. The intelligent automation brought forth by DevOps was confined mainly to the development and deployment environment. The IT operations, thus, remained susceptible to challenges.
Decoupling DevOps and RunOps - The Genesis of Site Reliability Engineering (SRE)
A decade or two ago, companies operated with a legacy IT mindset. IT operations consisted mostly of administrative jobs without automation. This was the time when the code writing, application testing and deploying was done manually. Around 2008 to 2010, automation started getting prominence. Now developers and ops teams worked concurrently toward continuous integration and continuous deployment — backed by the agile software movement. The production team was mainly in charge of the runtime environment. However, they lacked skillsets to manage IT operations, which resulted in application instability, as depicted in the scenario above.
Thus, DevOps and RunOps were decoupled, paving the way for SRE — a preventive technique to infuse stability in IT operations.
Software-First Approach: Brain Stem of SRE
“SRE is what happens when you ask a software engineer to design an operations team,” said Benjamin Treynor Sloss, VP of engineering at Google.
This means an SRE function is run by IT operational specialists who code. These engineers implement a software-first approach to automate IT operations and preempt failures. They apply cutting edge software practices to integrate dev and Oops on a single platform, and execute test codes across the continuous environment. Therefore, they carry advanced software skills, including DNS configuration, remediating server, network and infrastructure problems, and fixing application glitches.
The software approach codifies every aspect of IT operations to build resiliency within infrastructure and applications. Thus, changes are managed via version control tools and checked for issues leveraging test frameworks, while following the principle of observability.
Parameter | DevOps | SRE |
Monitoring vs. Remediation | DevOps typically deals with the pre-failure situation. It ensures conditions that do not lead to system downtime. | SRE deals with the post-failure situation. It needs to have a postmortem for root cause analysis. The main aim to ensure maximum uptime and weed out failures for long tern reliability. |
Role in Software Development Life Cycle (SDLC) | DevOps is primarily concerned with the efficient development and effective delivery of software products. It must ensure Zero Downtime Deployment (ZDD). It also requires identifying blind spots within infrastructure and application. | SRE is managing IT operations efficiently once the application is deployed. It must ensure maximum application uptime and stability within the production environment. |
Speed and Cost of Incremental Change | DevOps is all about rolling out new update/features, faster release cycle, quicker deployment and continuous integration, and continuous development. The cost of achieving all this is not of significance. | SRE is all about instilling resilience and robustness in the new updates/features. However, it expects small changes at frequent intervals. This gives a larger room to measures in case of possible failure. The bottom line is efficient testing and remediation to bring down the cost of failure. |
Key Measurements | DevOps' measurement plan revolves around CI/CD. It tends to measure process improvements and workflow productivity to maintain a quality feedback loop. | SRE regulates IT operations with some specific parameters like Service Level Indicators (SLIs) and Service Level Objectives (SLOs). |
The Principle of Error Budget
SRE engineers verify the code quality of changes in the application by asking the development team to produce evidence via automated test results. SRE managers can fix service level objectives (SLOs) to gauge the performance of changes in the application. They should set a threshold for permissible minimum application downtime, also known as error budget. If a downtime during any changes in the application is within the scale of the error budget, then SRE teams can approve it. If not, then the changes should be rolled back for improvements to fall within the error budget formula.
Calculating Error Budget
A simple formula to calculate error budget is:
(system availability percentage) minus (SLO benchmark percentage)
Please refer to the System Availability Calculator cheat sheet below.
Availability Cheat Sheet
Availability Level | Downtime per year | Downtime per quarter | Downtime per month | Downtime per week | Downtime per day | Downtime per hour |
90% | 36.52 days | 9.13 days | 3.04 days | 16.80 hours | 2.40 hours | 6.00 minutes |
95% | 18.26 days | 4.57 days | 1.52 days | 8.40 hours | 1.20 hours | 3.00 minutes |
99% | 3.65 days | 21.91 hours | 7.30 hours | 1.68 hours | 14.40 minutes | 36.00 seconds |
99.5% | 1.83 days | 10.96 hours | 3.65 hours | 50.40 minutes | 7.20 minutes | 18.00 seconds |
99.9% | 8.77 hours | 2.19 hours | 43.83 minutes | 10.08 minutes | 1.44 minutes | 3.60 seconds |
99.95% | 4.38 hours | 1.10 hours | 21.91 minutes | 5.04 minutes | 43.20 seconds | 1.80 seconds |
999.99% | 52.59 minutes | 13.15 minutes | 4.38 minutes | 1.01 minutes | 8.64 seconds | 0.36 seconds |
99.999% | 5.26 minutes | 1.31 minutes | 26.23 seconds | 6.05 seconds | 0.86 seconds | 0.04 seconds |
Suppose the system availability is 95% and your SLO threshold is 80%.
Error budget: 95% - 80% = 15%
Availability | SLA/SLO Target | Error budget | Error Budget per Month (30 days) | Error Budget per Quarter (90 days) |
95% | 80% | 15% | 108 hours | 324 hours |
Error Budget/Month: 108 hours. (At 5% downtime, per day downtime hours is 1.2 hours. Therefore, for 15% it is 1.2 x 3 = 3.6. So, for 30 days it will be 30 x 3.6 = 108 hours).
Error Budget/Quarter: 108*3 = 324 hours.
Quick trivia:
Breaking monolithic applications lets us derive SLOs at a granular level.
Cultural Shift: A Right Step Toward Reliability and Scalability
Popular SRE engagement models such as Kitchen Sink (aka Everything SRE), infrastructure teams and embedded SREs, tend to build dedicated teams that lead to a “Silo” SRE environment. The problem with the Silo environment is that it promotes a hands-off approach, which results in a lack of standardization and coordination between teams. So, a sensible approach is shelving off a project-oriented mindset and allowing SRE to grow organically within the whole organization. It starts by appraising the teams of customer principles and instilling a data-driven method for ensuring application reliability and scalability.
Organizations must identify a change agent who would create and promote a culture of maximum system availability. The agent can champion this change by practicing the principle of observability, where monitoring is a subset. Observability essentially requires engineering teams to be vigilant of common and complex problems hindering the attendance of reliability and scalability in the application. See the principles of observability below.
The principles of observability follow a cyclical approach, which ensures maximum application uptime.
Step Zero: Unlocking Potential of Pyramid of Observability
Step zero is making employees aware of end-to-end product detail — technical and functional. Until an operational specialist knows what to observe, the subsequent steps in the pyramid of observability remain futile.
Also, remember that this culture shift isn’t achievable overnight. It will be successful when practiced sincerely over a few months.
DevOps vs. SRE
People often confuse SRE with DevOps. SRE and DevOps are complementary practices that drive quality in the software development process and maintain application stability.
Let’s analyze four key fundamental differences between DevOps and SRE.
Parameter | DevOps | SRE |
Monitoring versus Remediation | DevOps typically deals with the pre-failure situation. It ensures conditions that do not lead to system downtime. | SRE deals with the post-failure situation. It needs to have a postmortem for root cause analysis. The main aim to ensure maximum uptime and weed out failures for long tern reliability. |
Role in Software Development Life Cycle (SDLC) | DevOps is primarily concerned with the efficient development and effective delivery of software products. It must ensure Zero Downtime Deployment (ZDD). It also requires identifying blind spots within infrastructure and application. | SRE is managing IT operations efficiently once the application is deployed. It must ensure maximum application uptime and stability within the production environment. |
Speed and Cost of Incremental Change | DevOps is all about rolling out new update/features, faster release cycle, quicker deployment and continuous integration, and continuous development. The cost of achieving all this is not of significance. | SRE is all about instilling resilience and robustness in the new updates/features. However, it expects small changes at frequent intervals. This gives a larger room to measures in case of possible failure. The bottom line is efficient testing and remediation to bring down the cost of failure. |
Key Measurements | DevOps' measurement plan revolves around CI/CD. It tends to measure process improvements and workflow productivity to maintain a quality feedback loop. | SRE regulates IT operations with some specific parameters like Service Level Indicators (SLIs) and Service Level Objectives (SLOs). |
Conclusion – SRE Teams as Value Center
A software product is expected to deliver uninterrupted services. The ideal and optimal condition is maximum uptime with 24/7 service availability. This requires unmatched reliability and ultra-scalability.
Therefore, the right mindset will be to treat SRE teams as a value center, which carries a combination of customer-facing skills and sharp technical acumen. Lastly, for SRE to be successful, it is necessary to create SLI driven SLOs, augment capabilities around cloud infrastructure, a smooth inter-team coordination and thrust automation and AI within IT operations.
Comments