Alcide Rapid 7 Logo Alcide Rapid 7 Logo
Alcide has been acquired by Rapid7- a leading provider of security analytics and automation. Learn more

DevOps KPIs: Continuous Integration /Delivery to Security Automation

March 7, 2019

blog banner-Measure Succes-2

The post is part two of a two blog series on DevOps KPIs. 
After reviewing asset management and monitoring as KPIs in the first part of the blog, this post will cover continuous integration/ continuous delivery and security automation tools.

Continuous Integration / Continuous Delivery

This technique is all the rage, and for good reason. Continuous integration and continuous delivery can make the difference between releasing software once a quarter (or year) and performing dozens of releases to production within a day. There is a lot to measure in this KPI, and it is one of the most difficult indicators to measure since every environment is very different. It is important to understand how many code commits don’t make it past your integration environment. These blockages could be the results of cultural problems such as poor programming practices, poor code quality, or terrible testing. However, if features stay in “dev” or “integration” for quite some time, there is an underlying issue. In most cases, it centers around automated testing.

How do you measure “good testing?” This is one of the most difficult questions enterprises face – especially when automated testing is added after monolithic applications are already written. In simple terms, good testing is about creating good business logic and integrating testing that meets “most” use cases while covering each function and class. Immediate feedback on failed or successful automated tests speeds the automation process. The less automated testing is built, the more QA has to test by hand. Failing to integrate good business and integration testing slows down the entire “CD” part of the pipeline significantly. 

On the subject of pipelines, good DevOps shops should be utilizing deployment pipelines whenever possible. Tools such as Jenkins, CodefreshCircleCI, TravisCI, (and most code repo solutions like Gitlab) offer rich feature sets to aid in pipeline creation and management. Let the tools help you automate better software delivery.

Below are questions to ask when developing CI/CD pipelines:

  • Is each application in a pipeline? If not, and the application is manually tested, it will never keep up with the pace of cloud-born innovation. If you are dealing with a giant monolith legacy app, it’s time to design its replacement (starting with testing).
  • What is your test coverage? Going for 100% is unrealistic. Shooting for over 80% is more appropriate. Remember to actively write tests for new failures, fixed bugs, and new features; these are always good practices in a DevOps process.
  • Do you have notification built into your pipelines? If so, great! Do you have chaining logic to react to those failures? Even better. Treat your pipelines like an application by including error handling, visibility, and monitoring.


Security Automation 

An often overlooked but critical KPI is security automation. The easiest way to ensure security compliance is to start with security practices in the development environment. Too often, security is applied at production, which means it’s not part of the environment’s end-to-end process. Securing applications and networks at the development environment level will give you more confidence that applications will interoperate properly at the production level.

Network security automation is critical. Automating controls such as AWS Security Groups and Network Access Controls (NACLs) will minimize network attack exposure through code and can even maintain potential configuration drift. Tools like Terraform and AWS CloudFormation can configure only the necessary ports to be open and analyze and close ports that were opened in error. 

Furthermore, the process of requesting, applying, and managing certificates and SSL can be a burden unless they are automated into the lifecycle of asset management. Many regulatory controls such as HIPAA, SOC, and PCI require end-to-end encryption of sensitive data. The more that control is in code, the easier it is to maintain the security posture. Seeing that controls are being automatically maintained will also provide auditors with peace of mind when evaluating your environment.

Here at Alcide we provide code-to-production security controls that are designed for DevOps, by embedding Dev., and DevOps know-how with an active, dynamic analysis of network, permission and configurations of Kubernetes deployments.

When measuring for security, ask yourself the next questions:

  • Are your network controls under code control and maintaining least privilege access? If not, you are entering the realm of “not knowing what you don’t know.” Network controls are easy things to automate. This step should be mandatory for any public-facing network.
  • If you are governed by security compliance, how much of your security is automated? The higher the percentage of automation, the easier the audit.



Success in measuring your KPIs is not about getting to full automation immediately. It is about understanding your biggest issues, automating them, and improving them over time. It’s about improving your environment by using the right tools. Take control, implement thorough and meaningful monitoring, deliver with confidence, and do it securely. And then repeat. 


Elad Ishay is the Head of DevOps at Alcide

Subscribe for updates, fresh insights, stories and tips

The latest posts delivered to your inbox