After Cloudbleed: Security at Growella

Cloudbleed

Yesterday, it was revealed that Cloudflare, a popular web security and optimization company, was accidentally leaking potentially-sensitive information due to a parser bug.

The potential impact of the bug — dubbed “Cloudbleed” — was discovered by Google’s Project Zero team, who immediately reached out to Cloudflare to fix the bug.

What does Cloudbleed mean for me?

There will no doubt be dozens of think-pieces about Cloudflare’s role as what amounts to a TLS “man-in-the-middle” (MitM) and steps we can take as a community to recover. There are already guides for consumers who are confused by the implications of a bug like Cloudbleed, and what they can do to stay safe.

In short, reset your passwords and consider de-activating and re-activating multi-factor authentication.

If you’re following good operational security (“OpSec”) practices, the impact of things like Cloudbleed should be minimal. That itself is a blog post in itself, but some best practices you should absolutely be adhering to include:

Never re-use passwords across sites.

While it might be easy for you to remember “[email protected]$$w0rd”, all it takes is one service to be compromised before attackers have a record of “[email protected] uses this password, let’s try that everywhere”.

Instead, use a password vault like 1Password (paid) or LastPass (free) — with either service, you still only have to remember a single password, but can easily have unique passwords across every site you frequent.

Use strong, randomly-generated passwords.

In the early days of the internet, an 8-character password containing numbers and letters was considered secure (we were so young and naive). Today, we have the ability to run tens of millions of possible username/password combinations against a service every second, and every extra character — especially those of the non-alphanumeric variety — can help improve your password entropy (a fancy word for “strength”).

Both of the password generators mentioned above allow you to generate random, secure passwords; push that entropy slider to the max.

Warning: once you’re used to having unique, secure passwords everywhere, you’ll be amazed at how much the services who do this poorly stick out.

Enable multi-factor authentication

Multi-factor authentication (“MFA”, also referred to as “Two-factor authentication” or “2FA”), is becoming widely available across many services. The basic concept is something you know (a password) something you have (an authentication device, like your phone).

Using MFA, a simple username/password combination is no longer enough to access your account; you’ll also be prompted for a unique code, available through your authentication device that rotates as often as once every 30 seconds. Even if an attacker gets access to your username and password, he or she would need your authentication device in their possession as well.

Multi-factor authentication is a must for any service that deals with potentially-sensitive information, like financial services (sadly, often behind the times on consumer-facing OpSec), email, social media, and more. If you’re running a site on WordPress, there are a number of great plugins to enforce multi-factor authentication for users (we love Eric Mann’s Dovedi, but there are also great solutions from Duo, Authy, and more).

Another interesting option is Tozny, which will generate a one-time use, “magic link”. After setting it up and installing the app on your phone, clicking “Login with Tozny” on the WordPress login screen will prompt your phone to ask “Would you like to login to https://example.com?”, no password necessary.

For a very practical example of MFA saving the day, imagine an attacker has gained access to your email account. He or she could look in the archive or trash and see the history of sites you’ve signed up for. Then, using that information, an he/she could reset your passwords on any number of sites, wreaking havoc for you. If you had MFA enabled, however, the attacker would be stuck at the login screen unless he/she also had your authentication device.

Demand HTTPS, everywhere

Even a few years ago, implementing HTTPS was something of a painful process; the certificates cost money, the process varies depending on your system, and your sysadmins had to remember to renew them before they expire and users start getting browser warnings.

In the last few years, however, that’s all changed. New certificate authorities like LetsEncrypt offer free SSL certificates with easy, automated renewals; in fact, we use LetsEncrypt for SSL certificates on every site we maintain, even internal-only staging sites.

Cloudflare also gained a lot of attention not too long ago when they started supporting SSL for all customers, even those on the free tier. Since their services work by sitting between your web server and the public web (hence the “man-in-the-middle” reference earlier), they’re able to offer free HTTPS connections between users and your site (though for true security, you should still generate a valid SSL certificate on your web server).

Admittedly “HTTPS Everywhere” isn’t a simple fix, especially for organizations with a lot of content across many properties, but it’s a noble cause. If you missed our post on LoopConf, Wired Lead Engineer Zack Tollman gave a fantastic talk about the struggles they’ve encountered at Wired in moving to HTTPS.

At the very least, make sure you see that green padlock in your browser’s address bar on any page where you’re entering sensitive information; HTTPS doesn’t guarantee total security on the server you’re communicating with, but it does mean that your connection to that server is protected.

Growella’s web security practices

One of Growella’s core values is ethical behavior, and this extends to user security. From the time we launched our “coming soon” splash page, Growella has redirected all traffic over HTTPS.

All of our web servers are running Ubuntu 16.04, a popular and well-maintained Linux distribution, locked down to prevent the most common attack vectors. System updates are run at least once a week on every server, including staging, and access to the servers is locked down tight using OpenSSH.

On the application side, we’re subscribed to Sucuri’s WPScan Vulnerability Database, getting a daily digest of known WordPress core and plugin vulnerabilities. Running composer update is a near-daily exercise, ensuring that WordPress core, plugins, and libraries are all up-to-date with the latest security patches.

Both staging and production enforce multi-factor authentication via Dovedi, and everyone with access to those sites has been encouraged to change their passwords.

Additionally, any access to Growella over insecure networks (coffee shops, hotels, airplanes…we’re kind of all over the place) is handled through our company VPN. After all, you can never be too careful!

Cloudflare usage

In the name of transparency, Growella does use Cloudflare. Fortunately, we have confirmation from Cloudflare (as of this morning) that no data is known to have been leaked from Growella due to Cloudbleed:

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare’s systems. If you haven’t yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers’ sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we’ve worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare’s customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince

Cloudflare, Inc.

Co-founder and CEO

I normally wouldn’t be so quick to trust a “you’re cool, no worries” email, but we also have the benefit of not having collected any user information. We have several new features rolling out in the next few weeks to help connect readers to our partners, but none of that has been rolled out yet, so your information is safe.

Growella puts a strong emphasis on protecting user privacy, and has no intention of sharing information that hasn’t been explicitly volunteered. If you have concerns, please read through Growella’s privacy policy and reach out if you have any questions or concerns.

Leave a Reply