Automate your New Relic Alerts configuration
Being a software engineer these days is not only about writing good code. It’s also about deploying your code to production and checking it works all the time.
Ocado Technology uses New Relic to monitor its services, either across our e-commerce applications or inside our highly-automated warehouses. There are multiple environments for the code we create and deploy, and each of these requires a separate configuration for alerting policies. Some of these alerting policies are common between environments, while some are unique for production, testing, etc.
When you have dozens of services running across multiple environments, configuring the various alerts for each service is a lot of repeatable work which is also error-prone. As it turns out, human data entry can be pretty inconsistent! Since the process of defining alerting policies was manual, we started to struggle when we had to introduce the same configuration change for multiple services in New Relic. After a thousand and one clicks, we decided we had had enough and began to explore other possibilities.
We wanted to find a way to configure alerts policies programmatically, which would ensure that our Alerts policies are consistent, and our configuration is not changed by an accident. The solution we came up with was to use New Relic REST API to create a tool tailored for our needs:
- Create missing alerting policies
- Update rules for existing policies
- Keep configuration in repository and run synchronisation automatically when a change is introduced
- Don’t repeat yourself: extract common configuration and use it when needed
- Detect invalid configuration before pushing changes to New Relic
We concentrated our attention on developing the tool and implemented it straight away. It proved to be the perfect solution to our problem; we no longer needed to browse through New Relic pages and introduce changes manually, and whenever we had to introduce a change that affected all our services, we only modified the element responsible for common alerting rules – Git commit, push, and wait for pipeline to execute. The consequence of these modifications was that our processing time decreased significantly. This had impact on business stories as well – because declaring alerting rules now proved to be a simple task, we added additional layers of checks to our applications.
With this tool, we are able to automate whole process. It reads configuration from repository and performs required changes to Alerts policies via New Relic REST API. We have extracted common alerting policies to a shared space that is used by all services and separated services that have unique policies.
Now we would like to share our efforts with rest of the community. We have published the New Relic Alerts Configurator library on GitHub. Using this library, you can configure New Relic alerting policies via Java code.
The library covers most of the popular alerting policy REST APIs, but not all of them. We are looking to introduce new features like NRQL support, but we also encourage you to add new features to help others using New Relic.
We hope you find our tool useful. Also, for those of you interested to know more about how we use New Relic, here is a presentation from our COO Anne Marie Neatham at FutureStack 2017 in London:
Konrad Bielak, Software Engineer