Why we do not operate in the production environment

When it comes to DevOps, a requirement is that the development and operations teams need to collaborate. They need to share responsibility for designing, implementing, and maintaining the systems that they are working on.

What is the best way to silo responsibilities and access though? Especially when it comes to the riskier production environment?

An ongoing argument in many development circles is whether or not developers should have access to operate in the production environment.

Our view on this is no. Here's why.

Why take the unnecessary risk

The answer is simple - we want to ensure that our applications and systems are reliable and functioning as expected. It's important that we do so before putting them into use. In a production environment, there is always the potential for things to go wrong. This of course can lead to unexpected downtime or system instability.

Why take the risk when we can avoid it by testing our code in a controlled environment first?

There are other benefits to testing in a non-production environment as well. Developers can take their time to investigate and fix issues without impacting users. They can also try out new features and functionality. All without having to worry about potential negative consequences.

In this blog article, we take a look at these and other reasons why we don’t operate in the production environment.

The reasons why we don't!

There are a few reasons why we don't operate in the development environment.

It can make software harder to manage when making changes in production

When changes are made in production, it can be difficult to track those changes and ensure that they are properly implemented. This can lead to errors and potential instability in the software.

There is the possibility that when they implement changes in this environment, that they break ops processes they’re unfamiliar with.

They may spend all their time fighting fires.

If there are constantly issues popping up in production, developers may find themselves spending all their time fixing those issues. They could instead be working on new features or improvements. This can impact the overall quality of the software.

This time could be better spent on software upgrades or exploring new development opportunities that may exist. These issues should rather be identified before the production environment. Ultimately it will save a lot of development time.

Potential liability concerns and privacy implications

There are potential legal implications of making changes in a production environment. If something goes wrong, the company may be held liable.

This is why it’s important to have proper testing and approval procedures in place before making any changes to production systems.

Additionally, there are privacy implications to consider. If personal data is involved, any changes made in production could potentially lead to a data breach.

This is why it’s essential to have adequate security measures in place and to test any changes thoroughly. This has to happen before implementing them in a production environment.

It can be difficult to roll back changes

If something goes wrong after making a change in production, it can be difficult to undo that change. This can lead to further instability and downtime.

It’s much easier to roll back changes in a testing or development environment. That's why it’s generally preferable to make changes in those environments first.

Production environments are often complex

Production environments are typically more complex than other types of environments. These include both development and testing environments. This complexity can make it difficult to properly test changes before making them in production.

It’s important to understand the complexities of the production environment. It's also important to have adequate testing procedures in place before making any changes.

There are often multiple stakeholders involved

When making changes in a production environment, there are often multiple stakeholders involved. This can make it difficult to get everyone on board with the changes.

It’s important to ensure that all stakeholders are properly briefed on the changes and that they understand the risks involved. Ideally all these changes are made prior to the production environment.

Downtime can be costly

If something goes wrong after making a change in production, it can lead to downtime. This downtime can be costly for businesses, as it can impact revenue and reputation.

It’s important to minimize the risk of downtime by thoroughly testing any changes before implementing them in production.

Take a look at Uber for example, which was recently hacked. It's believed this took place because a change was made in their production environment without being properly tested. As a result, the personal data of millions of users was compromised.

This just goes to show how important it is to avoid making changes in production.

Conclusion

In conclusion, there are many reasons why we do not operate in a production environment. From liability and privacy concerns to the potential for downtime, it’s just not worth the risk.

We understand that there are sometimes circumstances where making a change in production is unavoidable. However, it’s important to understand the risks involved and to take proper precautions before making any changes.

‍When we start work with our customers, we are more than happy to help them set up a proper pipeline for a smarter software development process - from testing to deployment.

‍