Jenkins and CloudBees CI Plugin Hygiene
Plugins are the best and worst parts of Jenkins and CloudBees CI. Jenkins, and ostensibly CloudBees CI, is built for the primary purpose of delivering a reliable, performant, and customizable CI platform.
Plugins, the Duality of Jenkins
Jenkins, and ostensibly CloudBees CI, is built for the primary purpose of delivering a reliable, performant, and customizable CI platform. I've said it to all of my clients, and now I'm saying it to you: plugins are the best and worst parts of Jenkins and CloudBees CI. This will likely never change, as the core of Jenkins is built around extensibility, and Jenkins' primary purpose is integration with all of the tools in your toolbelt.
Plugins provide a ton of capabilities not natively enabled in the CI tool, and as such makes Jenkins the most flexible and expandable CI tool out there. This enables functionality like customized authentication providers, integrated tools like Artifactory and Maven, and even fun such as minigames or automated functionality when a built completes.
With this flexibility comes risk, however, and plugins not maintained will blossom into a performance, security, and stability nightmare. It is my hope that this post will aid you in developing healthy plugin habits for a secure and reliable CI experience. Most of these tips are from my own experience with enterprise-scale clients I work with, but they should apply from the smallest of teams all the way up the chain to the thousands of developers hammering on projects all at the same time.
Just keep in mind that any time you run a plugin, you are installing third-party code on your controller. This can be from a big corporation like IBM, or from Joe Schmoe who creates and maintains a plugin in his spare time. This means plugins can be wholly trustworthy or could be abandoned entirely on a whim. Hopefully, this document will help you understand the risks associated and how to best determine whether a plugin is worth using or keeping.
Understanding Plugin Hygiene
The term "plugin hygiene" is intended to get across that plugins are something to be monitored and maintained, much like one's own personal physical hygiene. Much like our own bodies, plugins left untouched and unchecked will result in smells that might be difficult to track down. Poor plugin hygiene's greatest dangers are security and performance impacts, but can also result in compatibility problems with new Jenkins releases if left unmanaged. Ensuring your plugins are maintained, updated, and removed as needed ensures that you've got the smallest attack surface for bad actors, fewer problems when upgrading Jenkins or CBCI, and performance improvements.
Ideally, the rule of thumb is the fewer the plugins, the better.
Scrubbing Those Plugins
With plugins, less is more. The fewer the plugins, the fewer moving parts you have for compatibility issues or widespread security/performance concerns. Ensuring your plugins only meet the absolute necessary needs for your Jenkins or CBCI controller is critical.
Is This Plugin Necessary?
Is there a mighty need?
Almost all plugins run directly on the controller and rarely offload onto an agent. This means that you're using up controller resources to execute what a plugin is doing, rather than offloading that onto an agent an alleviating the stressor on your controller itself, which is how users interact with the CI system. Avoid plugins that can be replaced with CLI commands!
Determining usage for a plugin is pretty critical at this juncture. Does this plugin fulfill a required activity for your controller, such as authentication, infrastructure management (Kubernetes, Amazon EC2, etc.), or organization within the UI (such as CloudBees' Folders Plus plugin)? As mentioned above, many plugins will fulfill a need that would be far more efficient, reliable, and secure from the command line than a plugin within Jenkins itself.
It's also really important to periodically review your plugins to determine if they should stay on your controller or not. To review whether plugins are in use or not, you can utilize CloudBees' Plugin Usage Analyzer plugin to determine that if you're on CBCI. If you're running OSS Jenkins, take a look at this script written by my colleague Mayank Gangwal. It uses Groovy to achieve some level of plugin usage within a controller and may be a good tool to identify how active a plugin is on your controller.
Keep Plugins Updated
Older does not mean wiser.
Security is key with all tooling, and even more so nowadays with the risk of early poisoning attacks and backdoors within code. With the advent of the SBOM (Software Bill of Materials), and deeper tracking of tooling and dependencies within the build chain to ensure security of generated artifacts, it's critical to have as safe and secure of a build environment as possible.
Much like with vehicle maintenance to ensure that your multi-thousand pound personal transportation machine is reliable and safe, plugin updates prevent you from falling victim to the tens of thousands of security vulnerabilities found in software yearly.
Plugin upgrades should be done as soon as possible. Plugins, like pipelines and any other changes to your production Jenkins or CBCI environment, should go through a pre-production environment to be sure of compatibility with the version of Jenkins or CBCI it's being installed to.
Many enterprises tend to upgrade plugins alongside their Jenkins or CBCI instances, which is acceptable short of critical vulnerabilities being discovered in either the core of Jenkins or within a plugin itself. These are most frequently done quarterly, but the cadence varies from organization to organization. Always prioritize critical vulnerabilities!
Without testing, you could see compatibility problems or issues with the pipelines or other dependencies that rely on these plugins. Always test and lock your versions to avoid unintentional upgrades in a production setting. Utilizing a Plugin Catalog and providing specific versions via your own artifact repository (Nexus or Artifactory) will give you full control over what plugins are available, and which versions are available for install.
Vet Your Plugins From the Start
Always perform background checks before hiring.
Making good decisions on whether a plugin should even be considered a candidate for installation is key. There are several tools at your disposal to determine whether a plugin is worth trying or not
1: Check the plugin's health score.
The health score of a plugin can be determined from the plugin page, or from the Jenkins website. This health score is designed to give an at-a-glance visual if a plugin is following best practices and is up-to-date, maintained, and actively seeing development and usage. The higher the score, the more trust you can place in implementing that plugin.
2: If using CBCI, be sure it's supported.
Double-check whether a plugin is supported if you're on CloudBees CI. The CloudBees Assurance Program (CAP) provides support for all the plugins in this CAP designation. If you're paying for CloudBees Support, be sure to try to place focus on using fully supported plugins for the best experience. You can determine if a plugin falls under the CloudBees Assurance Program by searching the plugin here, which will tell you where it falls.
3: Ask yourself if it can be done via CLI.
As mentioned earlier in this doc, if it can be done via CLI it should be! Maven builds using the Maven plugin will build Maven on the controller and will not run on an agent, even if within an agent block in a Pipeline. Maven's CLI tools are well documented and thoroughly understood, so installing Maven on your agent (or including it in your image if it's an ephemeral agent) and calling it within a stage specifically on an agent from a sh or bat command is the most reliable and performant method. Another obvious instance is the Artifactory plugin, which is barely maintained and ultimately eats up controller resources to pull in and upload artifacts from builds.
Avoid plugins that replicate tooling that could much easier and efficiently be used directly from the command line!
4: Regularly Review Plugin Dependencies
Plugins will often depend on other plugins to provide functionality. It is very important to also understand and maintain the dependency plugins in addition to the selected plugins from above. Dependencies can be determined in a couple of ways. The first is via the plugin's page on the Jenkins.io site.
The second is to use a script to calculate the dependencies. It's an older script, but written by my colleague Kenny Younger. It relies on a CasC bundle, so if you're not using CasC you'll need to either write something yourself from the Update Center JSON or use the Jenkins site to manually validate.
Dependency management is just as important as managing the plugins overall. They can fall victim to the same security vulnerabilities and other issues (ownership, configuration, best practices) that any other plugin can experience. Do this whenever you update plugins to ensure all is acceptable when upgrading.
Indicators of Problematic Plugins
Why is my instance barely working?!
It's important to keep a close eye on your systems. Monitoring performance using Prometheus, alongside other tools such as Splunk or DataDog and DynaTrace, will give tons of insights into your system. Keeping a close, watchful eye on the experiences your users have within your Jenkins or CBCI controllers is key to ensuring satisfaction and ability to operate.
If you're seeing or hearing of performance slowdowns or frequent build failures (also known as brittle pipelines), dig into logs and keep an eye on what is causing these slowdowns. Within the pipeline context, if they're calling an inordinate amount of plugins (or even just one less-than-reliable plugin), this could be an indicator that you may want to phase out specific plugins causing these issues.
Within the UI, if you're relying heavily on some plugins for certain functionality (Blue Ocean comes to mind), you're looking at needing to entirely remove key functionality customers may rely on. It's important to understand these plugins, their use cases, and the performance implications of installing or using it.
Google is your friend! If you're seeing slowdowns, do some digging on the plugin to identify if others experience similar issues. The GitHub repos for these plugins are generally a treasure trove of what problems may exist, especially in the Issues section that get reported by end users.
Plugin Management the Automated Way
It is highly recommended you use a tool like Configuration as Code (JCasC for OSS Jenkins, CloudBees CasC for CBCI) to maintain your configurations in an automated fashion. This provides multiple benefits:
- Controller configurations become portable and reusable
- Plugins are declared as code and are easier to organize and troubleshoot
- SCM management gives the option for following code best practice/principle, such as Pull Request and Code Reviews
Utilizing the tools I laid out above in conjunction with Configuration as Code, it becomes almost trivial to maintain your systems once CasC is implemented. With CloudBees' CasC implementation, it offers even more features such as automated plugin dependency calculation and installation, URL pinning within the plugins.yml file, and management of items (such as jobs, folders, etc.) within the CI platform itself.
Do yourself a favor and stop managing Jenkins from the UI! It is an intense amount of extra effort to continually manage controllers, especially at scale, without a CasC tool.
Case Study: Chip Manufacturer
One of my clients, a large chip manufacturer based in the USA, was struggling with plugin management in general on their CloudBees CI implementation. Frequent crashes, poorly constructed pipelines that relied heavily on plugins, and UI performance problems plagued their platform. Plugins were upgraded, alongside the CI platform itself, once a year. Plugins were installed without oversight or any sort of approval/vetting process, which led to lots of monolithic snowflake environments with huge plugin manifests. It was making upgrades and maintenance a nightmare.
By applying process around what I've outlined in this document, they've been able to completely transform how their system maintenance is executed. Trimming the overall plugin load on every controller, establishing as standardized set of plugins, and using Configuration as Code to install and manage the installed plugins while restricting common user administrative access to the controllers went a huge distance in improving reliability and maintainability for their whole fleet. At the onset, they were only operating in approximately two dozen controllers. Now they're able to operate a fleet of almost 400 controllers with automation scripts written by their teams, in addition to heavy use of Configuration as Code.
Process and close monitoring are key for this client to ensure they have a maintainable, reliable environment for their developers to take advantage of.
Conclusion
Plugin hygiene is a must for a functional, usable CI system with Jenkins or CBCI. Without keeping a close eye and building good process for managing your plugins, you run the risk of security problems, poor performance, and a lack of reliability that a good CI system (or any IT system, really) should otherwise have.
Being proactive with how you handle plugins and ensuring close maintenance of your systems is key to being successful with CI.
Tim is a DevOps Consultant working for CloudBees. This information is provided as a service to any Jenkins user, wrought from experiences in the trenches with huge enterprises trying to successfully build a CI platform that scales well.