Use Keyword Checks

I always prefer using keyword checks over simple HTTP tests. The reason is simple: less false negatives. Almost always, there is some keyword or HTML snippet that should always be on the page, even if it’s as simple as a Terms of Service link or a page title.

By using keyword checks, you can avoid a whole host of stupid errors that can cause your simple HTTP monitor to stay green when it should be telling you there is a problem. Things I’ve seen in production that keyword checks have identified and alerted me to:

• IIS returning 200 responses when it was showing exception pages
• Elasticsearch indices being broken, causing pages to render no results
• APIs being shipped with bugs and returning incorrect JSON responses

Ideally, all of these things would have been caught in a staging environment or by developers/QA during testing. Sometimes that doesn’t happen and if there’s something wrong in production, it’s always better to know about it.

Monitor All Your Domain Variations

I’d recommend creating a monitor for all variants of your domain. For example, http://, http://www., https:// and https://www.. This can catch a few types of issues:

• DNS misconfiguration
• Reverse proxy misconfiguration
• SSL certificate problems

One issue with this strategy is that these alerts can become a bit noisy if you have a full-scale outage that impacts all the domains. To handle that scenario, you can funnel your alerts into a platform like PagerDuty or OpsGenie and deduplicate or aggregate these alerts. If some of the alerts are less important (for instance, 99% of traffic comes through https://www.), you could always disable alerting on the other domains while still having visibility into their status when you visit your uptime checker.

(Speaking of SSL certificate problems, you can use this free service to monitor your SSL certificate expiration for you and find out about any upcoming certificate expirations before you find out with your uptime checker.)

Monitor 3rd-Party Dependencies

This isn’t something I’d necessarily recommend using extensively… After all, your service provider should be monitoring these heh. However, it can be still be useful in helping you gauge whether a service provider is living up to its SLA or your uptime requirements.

At my last company, we were hit with an outage of a pretty critical reverse geocoding API we were using on one of our mapping pages. The first time this went out, we found out via our CEO… Not good. Setting up an alert on this API let us know that the service was having pretty consistent intermittent outages that were impacting our users and we were able to swap in a replacement that performed much better.

Monitor More Than Just Your Homepage

Finally, you should take advantage of the large number of monitors most services let you make.

• Monitor your public APIs to make sure that your meeting your SLAs.
• Monitor the pages that are most important to your visitors.
• Monitor your customer support pages to ensure those are always up and available.

Your site is much more than just your homepage, so make sure to monitor all of it!

Conclusion

Hopefully, you’ve found some of these tips enlightening and can apply them to the sites your responsible for so you can respond to outages sooner and get those 3 9s of reliability!

Any thoughts? Did I miss something? Send me an email.

If your sole experience has been doing development work, these can be particularly tough challenges to deal with… Operations is a big field with a huge set of tools that can take a while to get up to speed on. It doesn’t help that you might be having to learn about and implement these tools while under duress. You will probably have a lot of questions when you’re first getting started.

• What is worth monitoring?
• What tools are at my disposal?
• What problems do those tools help me solve?
• What should I add monitoring around first?

I’m going to answer some of these questions below based on my experience operating Leafly, ranked 1,112 on Alexa at the time of writing and serving around 750 million requests per month when I was last working there.

This is for developers that need to get a real quick handle on operations.

Must-Haves

Below you’ll find the bare minimum monitoring solutions I’d suggest having in place, in order of what I think is most important to implement first in terms of bang for your buck. If you implement everything I talk about in this article, you should be done in under a day.

Finally, one thing to keep in mind is that there is no “one size fits all” approach to monitoring. You will want the results of your monitoring to drive what you add monitoring to next. I’ll dive further into that topic in a future blog post.

Uptime Monitoring

To start with, you’ll want to start using a good uptime-monitoring solution. These services are dead simple — they ping specific HTTP endpoints periodically from different locations around the world and report on whether they are up or not. I’m partial to Pingdom (Paid), but I’ve had good success with UptimeRobot (Freemium) as well. It’s worth noting that some APMs (e.g., NewRelic) also offer this service if you’d like to keep your monitoring stack completely self-contained within a single tool.

To use, simply add a list of endpoints to the service to track — for example, your homepage, sign up page and other critical pages of your app. If you are running multiple services and/or multiple domains, you’ll want to have checks on those too.

You should be able to get good coverage of your services in under 10 minutes. As far as cost is concerned, these services are relatively cheap and often free — To start out with, you won’t be paying more than $10/month. These alerts are also fairly hard to screw up — False positives are extremely rare. If our uptime monitoring reported an issue… There was always an issue. Sometimes it was external (e.g., AWS CloudFront was down), but there was always customer impact.

Because of the ease of setup, low price and “idiot-proof” nature of uptime monitoring, I’d always recommend setting up this type of monitoring first. It can tell you if your servers are overloaded, your deploys are really zero-downtime, critical third-party services are going down and more.

Application Performance Monitoring

The second most valuable piece of monitoring you can add to your applications are APM tools such as NewRelic or Scout. Often times these tools also include uptime monitoring and exception monitoring baked in, making them a truly one-stop shop.

These tools usually work by adding a library and some configuration to your web application. So just like uptime monitoring, they are relatively low effort to add to your projects. APM solutions are usually free to start with, but the cost grows dramatically as you scale. Setting up meaningful alerts with these tools can also be tricky — you’ll likely have a lot of false positives as you gain experience with what these tools are telling you. The big win though that you will get instantly is deep, deep insight into your application that will help you guide your future monitoring and development efforts.

APMs will help you determine what pages are slow and why. They can help with the following questions:

• Are these pages slowing down the rest of the site by hogging all your computing resources?
• Did your latest deploy introduce a large performance regression?
• Is the database is overloaded?
• Is a key third-party service is slow?
• Did your cache unexpectedly emptied?

APMs will be the most valuable tool in your monitoring toolbox after you get past the somewhat steep learning curve and information overload. You will want to have this tool open during and after every deploy as well as when you are having outages or poor performance.

Exception Monitoring

Finally, the last piece of critical monitoring I’d ensure you have in place is exception monitoring. Many APM solutions include this, so it’s worth checking if you get it “for free” before adding in another tool. Even if your APM has something, depending on the volume of your exceptions (and quality of the APM), you might want to pull in a 3rd party to do this work for you anyway. I’ve had good experiences with Rollbar and Sentry personally, but don’t stress too much about choice here. Most solutions are really similar, so just pick one.

Just like APMs, these are usually just library additions that hook into your application’s global exception handlers and log failures in your app to an external service. Exception monitoring can help with some of the following:

• Easily view and search stack traces for rapid root cause analysis.
• Grouping similar exceptions so you can understand the impact.
• Integrate with your bug tracking systems so you can triage and assign bugs to developers.
• Attach metadata to your exceptions so you can see for example which users are being impacted by failures.

Exception monitoring is cheap or free at low volume and by and large, you always want to know when your application has an unhandled exception so the information is always valuable.

There is a decent amount of overlap with what an APM can tell you (e.g., if your cache is down, you will probably get exceptions from your cache connector), but for digging into specific errors that your users are seeing it’s usually hard to beat a dedicated exception monitoring interface. Just like your APM, you will want this open during and after every deploy.

Conclusion

Based on my experience running a fairly large website that was in a state of absolute disarray when we took over, these are the absolute critical pieces of monitoring infrastructure you cannot live without. There are a lot of other pieces to a successful monitoring solution that I’m planning to cover in future posts: meaningful alerting and what that means, incident management platforms, Slack-Ops and exploratory monitoring to name a few. Sign up for my mailing list if you are interested.

Any thoughts? Did I miss something? Send me an email.

While traveling through the UK and Balkans, I’ve kept on hand the following items:

• Keys and Wallet: Left pocket that is zipper and button secured
• Passport: Left pocket that is zipper, button and velcro secured pocket
• Phone and Handkerchief: Right pocket that is zipper secured

The above is my perfect configuration for security and ease of access. With the exception of my keys and phone, all of the above items only leave my pockets when they are in use and they never change pockets.

This has a bunch of small, but meaningful quality of life improvements. In the morning, I can just slip on my pants, unplug my phone, lock the door and I’m ready to go for the day. This was especially useful when we were living out of a van in the UK – I never lost anything to the crevasses of our tiny VW Caddy. When I’m at a border crossing, I always know where my passport is and that it’s safe and secure. Coins and keys are always kept away from my phone, keeping my screen scratch-free.

You can certainly experience pant continuity with any pair of trousers – At the minimum, I’d probably recommend a pair of pants with zippers. But I think the ClothingArts pants are probably the most ideally suited for it – The triple secure pockets put me at ease carrying my valuables around new cities. I don’t have to worry about my things falling out when I sit down or my valuables getting stolen by thieves. They can also hold quite a bit of stuff without getting bulky or disorganized.

Long story short, I don’t worry about my things anymore. It’s markedly improved my traveling experience. I’d recommend any long-term traveler try out Pant Continuity to reduce their packing weight, simplify their lives and relieve your worries.

Appnouncements is aimed at making it as easy as possible for businesses to embed beautiful release notes right inside their app. Every mobile app I’ve ever worked on has needed this feature over the course of its development – After all, if none of your users know you’ve released a feature or fixed important bugs, what was the point? That being said, it was still difficult to prioritize it in between more pressing work. After my launch, there will be no excuse for that again 😊.

It has been a blast developing it over the past few months and I’m excited to talk about it more on here before and after launch whether it’s a success or failure!

Look interesting?

If this looks interesting to you, visit Appnouncements and sign up for the mailing list to find out when it launches. Also, feel free to contact me directly via email or LinkedIn to ask about early access, coupons or just to talk about your needs.