There’s a tweet of @sadserver that says: “DevOps is a software engineering culture and practice of putting horrors into containers and then talking about Kubernetes at conferences”
What is DevOps?
Nothing is permanent except change
We try to order and categorize our roles and our place in the world to make our jobs and our life more manageable, but over categorization and bureaucracy can leave us trapped in a straightjacket where human imagination and innovation can get suffocated. So we eventually breakout out of it only to come to a point where things have become chaotic so we bring back the order and rigidity, and so the cycle repeats itself. This is roughly the cycle in which progress occurs.
DevOps is the most recent instance of us realizing the rigidity of our organizations. As the IT industry grew, over the last few decades, it also accumulated a fair bit of bureaucratic overheads on top of it. Chief among these was the differentiation between people who wrote the software – the Developers – and people who operated, monitored and troubleshoot the software – the Ops personnel. The DevOps movement, as it is called, was the realization that this distinction is one with little to no merit.
The DevOps mindset encourages members from both categories should communicate more freely. Some organizations have started to consolidate them into a single DevOps team. But it goes deeper than that. DevOps is more about mindset than the organizational structure at your company. The DevOps mindset encourages developers to know more about the real-world and it encourages the operators to know more about software development. It is useful for several reasons.
The first reason is buried in the fact that computers are meant to be automated, not manually tended like an automobile. If your job involves running a set of test and build routines and then deploying the app into production, you will find yourself writing a lot of scripts for automating your work. That will make the routine more reliable and quicker. In other words, even as an Operator, you will find yourself writing a fair bit of code yourself.
Conversely, if you are a developer you can’t bury your head in sand and just use your favorite language and framework to build applications. No! You need to learn a fair bit about the production environment, the real-world, to make a good product. For example, knowing a little bit about AWS CloudWatch can let you write software with better monitoring and tracing support.
“DevOps is about confronting the real-world constraints and opportunities with an operator’s pragmatism and analytical thinking, and a developer’s creativity and ingenuity. ”
Before asking how it is done, you should consider asking why.
If applied properly, the DevOps philosophy can give your product a competitive edge, make your employees happier, and your management smoother. These are the most obvious, short term goals but over the long run, it will bring forth a generation of readable code, elegant design, and smarter technocrats. Done properly, DevOps may very well be our legacy.
As stated earlier, DevOps is a mental framework where people with one set of responsibilities, say Ops, they are empathetic about the complications and troubles that people from related departments, like Development, face. A mutual understanding between the two aspects of the software industry can work wonders, given the right mindset.
For example, if the developers are cognizant of the environment in which their code will be running, they will try and optimize the code accordingly. If the Ops team notice a certain bug creeps up repeatedly, they can add it as one of the tests in the test suite used by the developers. They can even take this further by automating the process of tracking a bug, so it is much quicker. This symbiosis could help reduce friction, save time and result in better products overall.
“The above scenario, in my mind, summarizes what DevOps really mean. DevOps is not about containerization, or scalability, or microservices or any other buzzword that the software industry is spewing these days. DevOps really is about being pragmatic.”
It has more to do with the human workflow and day-to-day activities of IT professionals. That said, I will still be talking about it largely in the context of cloud computing, and cloud-native applications because its where it is more immediate and obvious.
What DevOps is NOT?
If what you are doing involves everything, you end up doing nothing.
If marketing is to be believed, DevOps can solve any problem under and inside the Sun. Sadly, the more omnipotent you make a concept, the harder it becomes to follow it. So let’s narrow our field of vision by eliminating what DevOps is not.
1. Not System Administration
People tend to equate system administration to Ops. Let’s be clear here, DevOps, or even Ops, is not comparable to your traditional sysadmin job. More often than not, physical infrastructure and its management is delegated to cloud service providers like AWS, Azure, DigitalOcean, etc. If your company is using one of these services, you don’t need a ‘traditional’ sysadmin. That job has long since been decoupled from companies writing and shipping software.
There are a few exceptions to this rule such as the cloud service providers themselves, CDN providers and several other organizations responsible for maintain the Internet backbone upon which IT industry run. Also system administration, by no means, is a ‘dead job’ some of the most bleeding edge research facilities, universities, banks, government institutes, etc need to run and maintain theirs on private servers and networks.
2. Not a set of tools
“DevOps is not a set of tools. It uses a set of tools to automate testing, building, bug tracking, and deployment.”
Tools like JIRA, Jenkins, Ansible, Puppet and even Docker fit incredibly well with the DevOps mindset. Docker helps make the development environment similar to the production one, JIRA helps developers stay on top of the issues being reported, Jenkins lets you build a lot of CI/CD and so on and so forth.
All of these tools have various commercial and open source alternatives, but DevOps as a concept is independent of them. Later on, we will be seeing some of these DevOps related tools in much greater detail.
Adopting the DevOps Model
Splitting the problem, not the team, into smaller pieces.
There are a few key concepts and practices that might be disruptive, at first, but are essential in the long run, when you are trying to adopt the DevOps model. Some of the changes are related to human behavior and the rest are technical.
DevOps means greater freedom for the individual to experiment and learn about different technologies. However, with this freedom comes additional responsibility. The range of problems you are expected to deal with grows, and so does the level of competence expected from you.
This can be overwhelming. Under the traditional model, the humans were split based upon the skill sets Under the DevOps model you might want to use a different approach. The application itself can be broken down into manageable microservices. Then a team, or an individual, can take responsibility for a particular microservice rather than the whole complex application.
Adoption of a microservice architecture is not necessary, but quite helpful if you are trying to adopt DevOps as a culture.
2. Continuous Integration
Quarterly release cycles and “scheduled” deployments slow down the thought processes. Ideas lose their charm, sitting on a to-do list, and humans lose enthusiasm. Central to the DevOps culture is the practice of continuous integration, or CI, where you push changes to a central repository and then a set of automated tests validates that the entire app works correctly, after the merge.
This works beautifully with microservice architecture. If a build fails, then you know that it is your particular microservice that was changed and you will turn your attention to that. If it were all a giant monolith, incremental updates would have triggered bugs in the remotest part of your code base making incremental updates an impractical idea.
3. Continuous Delivery
Integrating your changes with the existing code base is fine, but what about releasing it into production? Is it a bridge too far? While many insist that this is indeed reckless and unwise, you should look into Continuous Delivery options if they are well-suited for your application. Why push a giant update every quarter or every six months, when you can push new features incrementally.
This will give your product a competitive edge. While your competitors are waiting for their next release cycle you are rolling out new features to the users continuously.
If security and reliability is a concern, you can use the flexibility of source control to differentiate between the production-ready branch and the one that is on the bleeding edge. A good example this is the FreeBSD operating system, which offers its users STABLE, CURRENT and RELEASE branch and OpenSUSE rolling update model, Tumbleweed.
4. Observability and Debugging
The nastiest of the bugs are encountered in production, the ones that escape test cases and subtly hurt performance are the hardest to catch. The only way to monitor them is to make your code more observable. The proper DevOps team believes strongly in the value of observability.
Services like Amazon CloudWatch can be integrated into your applications to monitor and log the app’s state, in real-time. Besides real-time monitoring, aggregating time series data can also benefit the developers to better optimize for trends in usage pattern and user behavior.
Observability and Debugging ought to be first class citizens when it comes to DevOps.
DevOps in The Cloud
Try shooting for the stars. If you miss, you will still reach the cloud.
DevOps model has an intimate relationship with the cloud and cloud-native apps. Cloud is where each microservice can be individually updated. Cloud is where your repositories are host and cloud is where the building and testing happens. Cloud is costly.
That last point is rather crucial. Both the cloud vendors and the customers want to make sure that you get the best bang for your buck. It is an immensely competitive market, which is good news if your app can be a host on it. AWS is the best example of it, their pay-as-you-go model and billing on per second basis has motivated developers to optimize their code to the nth degree. A deep understanding of both the cloud platform and the nature of your application is needed to make things work optimally.
Cloud is essential for the majority of requirements a DevOps team might have. Everything from the source code repositories, to the building, testing and deployment can be run on the cloud. Conversely, the attention of a healthy DevOps team, with the perfect interplay between the development and operations, would soon turn towards optimizing their app to use the least amount of memory, compute time and offer the best user experience.
Benefits of DevOps
Its effects on the organization, management and employees.
If executed properly, and that’s a big ‘If’, DevOps mindset can be mutually beneficial to all the participants involved.
- The organization benefits as the product development is accelerated. This gives them an increased competitive edge at the cost of zero additional human recruitment. All that’s needed is some encouragement for different groups to communicate more freely.
- The employees benefit as they step outside their comfort zone and learn about the intricacies involved in other aspects of computing.
- The cloud vendors benefit as they learn more about the developer’s need and modify their platform accordingly.
- Users get better products and services and smoother updates.
If you think that DevOps mindset is feasible for your business/workplace, you should definitely put this idea on the table. The end result of this philosophy is definitely worth the additional adjustment period.
Tools for your DevOps Team
Tools are meant for us, not the other way around.
When it comes to the addition of tools in your arsenal you ought to be very careful. A good rule of thumb is to add more tools, only when the need arises and not before. But as a starting point, here are a few tools that you ought to be using regardless of the complexity of your application.
- Git and Remote Repository: Even projects maintained by a single individual need source control. You may want to branch out and try new experimental features, you may want to rollback to a previous version, and a version control system, like git, let’s you do that. It is unambiguously the first tool you will be adding to your toolkit. You can take the version control and host it on the cloud using services like GitHub, AWS CodeCommit, BitBucket or GitLab.
- CI/CD Pipeline: Once you have a version control, you can take it a step further. Everytime you push a commit to your repo, it can automatically try to build the entire application for you to see if nothing’s broken. A lot of open source and cloud hosted solutions are available that will automatically build, test and deploy for you. The last part, deploy, is not something you want to take lightly, but it is a good idea to automate build and test anyways.
Few good examples are CircleCI, Jenkins and AWS CodeDeploy. Bear in mind, that this will add an additional layer of complexity that everyone will need to know and understand.
- Standardize your production, staging and development environment. Pick one technology, like Docker or Vagrant. Furthermore, pick one image, like an Ubuntu image for Vagrant or Alpine base image for Docker and stick to it. Nothing wastes more time than disparity between different environments. Avoid it at all cost. When the base image is updated, make sure it is updated throughout all the environments. Which brings us to last point.
- Configuration management system. Config management systems can automatically setup a server to your particular specification. Want to create a user with a this username and that SSH key, write an Ansible playbook and it will handle that for you, no matter what the underlying Operating system is. Config management systems contain the blueprint of your infrastructure which they can automatically bring to life by talking to your Cloud provider’s API. You can store this blueprint as a git repo and go full circle with the tools all interdependent on one another. Although that is advisable only for the most sophisticated and complex of software projects.
A few good examples of configuration management systems are Chef, Puppet, and Ansible. Although there’s nothing wrong with writing plain Bash or Python scripts to achieve the same results.
Each new tool makes your life a little bit easier but also adds an additional layer of complexity. If your source repository goes offline, it could mean that your CI system would stop working or your Ansible playbooks are now out of your reach. So don’t take these tools lightly.
DevOps and its Greatest Challenge
The Journey is more important than the end.
The biggest challenge towards automation has always been human nature, also known as ‘transition cost’. A good example of this was the introduction of the autopilot feature in airplanes. The autopilot was not perfect initially, and it still requires human intervention quite a bit. But it also allowed humans to be more careless, more imprecise. The problem is exacerbated further if are not aware of the workings of the automation. A lot of airplane incidents have occurred precisely because autopilots behaved how they are supposed to, but the pilots labored under wrong assumptions.
The transition phase is the part where computers are slowly doing more and more of our work for us, but still, require a little bit of human intervention. In the world of DevOps, this is not a phase, in this world, the humans are always going to be in the loop, or at least in the final analysis of the things. At the same time, we will always be adding more and more of automation, a kind of meta-automation to push IT forward. Not doing that would be similar to not eating fruits from your own garden. It would be folly.
DevOps places itself at this very frontier of transition. It is, definitionally, the place where we tentatively let our fragile, nascent software go out in the wild, but at the same time monitor and nurture it, and in turn, learn from its failings in the wild.
The challenges that a DevOps mindset addresses are asymptotic. There is an ever-receding end towards which it aims, and though the end may very well be out of our reach, the journey in and of itself is not devoid of merits.
ClickIT -- DevOps is what we do!
Adopting the DevOps culture is important if you want to maintain that competitive edge in the free market. If done improperly, it can make things go from bad to worse. Moving to a cloud-native environment, with lots of moving parts can be daunting for startups and old school players alike. We can help you with this transition. ClickIT can handle the transition towards the Cloud in a way that is beneficial to your business without you having to worry about any of it.
Feel free to reach out to us, to know more.