Hey InfoSec, what are you doing to protect your DevOps Team?

DevOps, serverless applications and containers are just a few of the latest advancements in a developer’s toolbox. For development teams, this means that the time to market (TTM) is faster—especially for Agile teams. So, how and what are security operations teams doing to ensure that security is keeping pace with this rapid development? Most are attempting to incorporate a security engineer within their development teams—which is a great first step—but there are multiple layers that you need to ensure you are protecting your organization beyond your source-code.

DevOps or DevSecOps is a great step forward, but the adoption (and implementation) of DevOps is still difficult for many organizations. Even if an organization has implemented DevSecOps, there are still major concerns that every security team needs to be aware of.

No matter the size of the organization—from a small tech startup to a global enterprise—rapid deployment is key for maintaining a competitive advantage. Security must provide the necessary rights for your development to team to their jobs done. Developers and security teams alike need to be able to install any tools, libraries, etc. when they are needed.

You have heard about NPM being compromised right? What about Arch Linux? Python? What security controls do you have in place to detect and prevent this for your own organization?

Monitor all the environments

Security teams have a bevy of concerns that need to be addressed daily from concerns of compromised upstream third-party packages to their production systems. But what about the deployment environments? QA environments? Are you monitoring them as well as you do production? You should be.

In many situations (but not necessarily a best practice), a development or QA environment is a duplication of a known good environment (like production). This means that this environment may have production data that your security teams should be aware of. These development environments should have the same or similar security controls as they do with their production environments and should be monitored (and logged) as you would with any production system.

There are many other controls you can add, but the point is that you need to make sure that your team is accurately monitoring these environments and has visibility into any changes made to any environment.

If your development or QA environments are stuffed full of non-production data, then there are still risks you need to be aware of.

Keys, tokens, CI/CD, oh my!

Do your developers have access to your CI/CD (continuous integration and continuous deliver) build and deploy environments? Do they have access to development environments either locally or in AWS/Azure/GCP/etc.?

If you answered yes, then you must ensure that you are monitoring your developers activity (especially on their local system) closely! All it takes is visiting a bad site or a succumbing to a phishing email, and you could be in serious danger.

If your developers access cloud based resources, they typically use a set of credentials or access/secret keys to access these environments. Guess where they are stored? Yep, their local system. Here’s a quick list of places to look:

AWS CLI:
 ls ~/.aws
 ~/.aws/credentials
 ~/.aws/config
Azure CLI:
 $HOME/.azure (*nix & macOS)
 %USERPROFILE%\.azure (Windows)

If these credentials on your developer’s machines are compromised, it could be devastating for your organization. A malicious actor can modify your CI/CD environments and alter your code base without ever being detected in compiled binaries or libraries.

Additionally, you should prevent developers from accidentally committing secrets to repositories. There are several ways to detect this, but one way is to use git-secrets from AWS.

Make sure that your security team understands where secrets are kept, how they are used, and what the consequences are for not monitoring or alerting on potential misuse of these credentials.

Containerization

As organizations move to containers, serverless and infrastructure as code, information security teams should be working with development teams to establish an internal container repository that is continually monitored by the security team. If a new package or library is needed, then make sure that a security engineer (on your development team) reviews the code before merging into your master branch. Additionally, you should start implementing automated scanning of your repositories to help identify any potential secrets left in commits and to help identify any vulnerable third-party libraries being imported or used in your product.

Development team + security

Imagine you are on a development team, and someone from some other department came in and told you that your work sucks and there are security holes left and right. How would you react? Well, probably not too great, especially if you feel that your code is pretty good!

This type of interaction with development teams happens all the time, and we need to approach this problem differently. This is why we need to embed security engineers with our development teams. This engineer should be able to identify security issues, create backlog items (tickets), and be able to work (code) those backlog items. This key individual is both a developer and a security specialist within the development team.

Now, imagine receiving feedback from your co-team member. The reaction by the rest of the development team would be more understanding, and they would have a direct line of communication—including building a long-term relationship between developers and security. As they become comfortable, the remaining developers on the team don’t want to add more work for their teammate. So, do you think they will take their time or stop and think about their implementation? I do.

Additionally, this security engineer should be brought into discussions with the product manager/owner and architect at the early stages, as they are the de facto security personnel and should be part of initial design decisions that may impact how your product handles security (think authentication, shared secrets, communications, etc.)

Wrap up

All security professionals are busy. There too many alerts and not enough hours in the day. Even if we cannot respond to every alert, security operations center (SOC) team members need to understand what the mission of their organization is, where critical assets are, and what to look for in this ever evolving threat landscape. By working with development teams, we can help protect our respective organizations and begin to ingrain security in our development life cycle.