On Thursday September 15 news broke out that Uber’s internal systems were breached. Our sympathies are with Uber’s security team. Uber has one of the best security teams and they seem to have followed every best practice before the breach. Yet an alleged teenager easily broke into the crown jewels of Uber’s infrastructure (Domain admin, AWS admin, Duo admin, Vsphere admin and Slack). Earlier this year a similar thing had happened at the two most premier identity companies(Okta and Microsoft). Another teenage gang Lapsu$ managed to break into their systems.

We started Procyon to prevent precisely these kinds of attacks. So we find it useful to do post incident analysis to reflect on ourselves. Could it have happened to us? What could have been done to prevent it. There are plenty of blog posts out there proclaiming how humans are the weakest link and more cybersecurity training or more awareness with some zero-trust pixie dust could have prevented it. This is not going to be one of those superficial blog posts. This is going to be a deeply technical analysis of the breach with concrete recommendations on prevention.

What do we know about the breach

Uber has published some details on the breach, given they are still investigating it. But there is quite a bit of OSINT information available out there. Our friends at GitGuardian have done a great job of summarizing it here. We highly recommend reading it in its entirety.

 

 

1. Attacker managed to social engineer an Uber contractor using a technique known as MFA push-bombing. Compromised user seems to be an IT contractor with access to highly privileged accounts. Attacker seems to have bought the contractor’s username and password from the darkweb. Then they bombarded the employee with 2FA push notification requests. Employee eventually gave in and approved it.
2. Once the MFA was approved, the attacker got access to Uber’s corporate VPN. Once they were on the VPN, they started scanning Uber’s internal network. They found an internal network share that contained some powershell script.
3. One of the powershell scripts had the admin credential for Thycotic privileged access management system(PAM). PAM systems are the secret storage vaults used by almost every organization.
4. This secret vault had stored keys to the kingdom. It has secrets for everything including (i) Domain admin account (ii) Duo admin account (iii) AWS admin account (iv) Vsphere admin account (v) Onelogin (vi) Slack (vii) Sentinel one and (viii) G Suite
5. Once the attacker had access to all these admin accounts, it was game over. The attacker could have done a lot more damage given he had access to highly privileged accounts.

How do we prevent it?

Lets dig into each step of the attack and see how to prevent it in detail.

Step 1: Social engineer an employee to gain initial access

This is an easy one to prevent. Just use FIDO2 authentication. Google made an announcement many years ago that ever since they rolled out FIDO2 authenticator keys, none of their 85,000 employees have been phished or social engineered. If you are interested in learning more about the underlying technology, you can research more about FIDO2 and Webauthn standard.

Just a few years ago, you needed a hardware token like Yubikey to achieve this. Now all our Iphones, Macbooks and Windows 11 laptops are capable of being FIDO2 authenticators. Apple, Google, Microsoft have all announced their support for PassKey. This is possible because of what are called TPM chips(Apple calls them “Secure Enclave” chips) that are widely available in all of our devices. The key mechanism in these chips is, they can generate a private key inside it in such a way that you can never read the private key out. Whenever we need to authenticate a device, we can simply ask the TPM chips to sign a random string to prove the possession of the private key.

One thing that remains a weak spot in this area is: onboarding new devices. Many solutions let you onboard a new device by using your username and password. In such cases it is trivial for the attacker to onboard his own device into your identity. We see that only Apple’s Passkey solution has done a decent job of making this secure. In Apple’s implementation, you need to use one of your existing devices to onboard the new device into PassKey.

Obviously we would highly recommend Procyon’s authenticator apps to solve this problem(pardon our bias). We not only prevent phishing and social engineering attacks like everyone else. We also have very sophisticated onboarding methods to prevent attackers from onboarding his device into your identity. This topic will require a complete blog post by itself.

Step 2: Gain access to VPN and scan the internal network

This one is relatively easy to prevent as well. Just dont use a VPN. Move away from the corporate network mindset. Call it zero trust access, or BeyondCorp or identity defined perimeter. Call it whatever you want, but prevent blanket access to everything inside the corporate network to users. This is a fairly mature product segment with many good options to choose from. Most of the vendors in this segment also support integration with EDR products to determine the health of the device before granting access.

One weak spot in this area remains the integration between the Identity providers and the zero trust access providers. Most zero trust vendors still rely on basic single-sign-on to authenticate the user. Most of them store identity materials like private keys and bearer tokens on filesystem or in memory. These are all trivially compromised by hackers just like passwords. We need much tighter integrations between identity providers and zero trust access vendors, preferably using FIDO2/Webauthn standard for both.

Step 3: Get admin credential to secrets vault from the script

This is actually much harder to solve than it appears at first glance. Lot of smart people were surprised that Uber was storing admin credentials to a secret vault in a powershell script. Weren’t the secret vaults supposed to solve this problem to begin with? This is actually a fundamental flaw with secret vaults.

Secret vaults are really good at consolidating secret storage and providing access control on who can access what secrets. Most of them integrate with your single-sign-on systems to grant human users access to the secrets, which means you can require 2FA or a FIDO2 authenticator to access it. But if a non-human user needs to access the secret, like a script or an API client or a CI/CD system, you need to generate a permanent API key so that the script can access the vault. Basically, you traded one kind of secret(like a database password) to another kind of secret(like an API key to the vault). You have not fundamentally solved the problem. This is known in the industry as the “Inception” problem(also known as the “turtles-all-way-down” problem).

The way to solve this is to use identities instead of secrets. Imagine every workload had a unique identity similar to humans. Then we can grant access to specific identities instead of generating a secret and storing it with the workload. If a developer is running a script on his laptop, we can simply use the TPM based device identity to grant him/her access. Similarly if your workload is running in a cloud provider, you can use the instance identity to do the same thing. Most cloud providers have good integration with instance identities and their own secret managers. You can assign an instance role to workloads and then use that to access secrets manager, with the help of instance metadata servers(you have to be little careful with MDS attacks here, otherwise you could end up with the capital one breach scenario). There are similar solutions for Kubernetes pods and service meshes like Istio. Some of these are not as secure because they store the identity keys in the filesystem and can be stolen just like the secrets. But the ones provided by the cloud provider themselves are pretty good as long as you can trust the cloud provider’s own infrastructure.

But these are only a small subset of scenarios where we need secrets. We often see scenarios like:

1. Developers bundle service account credentials with the scripts because it needs to run in multiple environments like a developer’s laptop, a CI/CD pipeline, a cloud VM instance or even on an on-premise system. Since there is no unified way to get workload identities across these environments, the easiest thing to do is just bundle the credentials with the script.
2. We see devops teams storing secrets in password managers like 1Password or BitWarden and then sharing it across the team so that anyone in the team can use those secrets
3. We see an endless number of stories of developers just committing the secrets into the code because it is convenient. This problem has become so bad that we have a cottage industry of tools that scan your code base for secrets.
4. We see plenty of examples of IaC tools like Terraform or Helm fetching the secrets from vaults and provisioning it into workloads. If Terraform can fetch the secret, the attacker can fetch it too.

For these other-90% of the scenarios we have built the Procyon workload identity solution. It works very similar to our TPM based device identities. It uses a private key as the workload identity. It then splits this private key into multiple parts using MPC cryptography and stores them on multiple instances. This way even if the attacker compromises an instance he can not steal the private key. Attacker has to compromise all instances at the same time to retrieve the key. Once the workload has a non-compromisable identity, we can use it just like human identities. We can grant access to identities instead of passing around secrets. Procyon workload identity solution integrates with many popular CI/CD systems today and we are adding more. It also has a simple API and command line tool to make it easy to use it with your existing workload. It provides a unifying workload identity layer so that you can use it in multiple environments as discussed above. It even integrates with cloud provider’s instance identities when used inside a cloud provider environment. This solution is in early access beta and please contact us if you want to learn more.

Step 4:Extract all credentials from secrets vault

Secret managers and vaults are necessary evil for the most part. But a good privileged access management system should avoid exposing them to the end users. It should act as a transparent proxy and inject real credentials into the session without exposing them to the users. It should provide ways to provision secrets into specific workloads(with specific workload identities) without exposing them to the users.

In this section we will look at different techniques for avoiding secrets all together. Before we jump into discussing how to avoid secrets all together, we need to acknowledge there are some secrets that are completely unavoidable in most organizations. They are:

1. Database passwords: Database passwords are a necessary evil for the most part. We are not going to be able to convince database vendors to support FIDO2 anytime soon. Use a modern PAM solution to inject the credentials into database logins without exposing them to end users. A good PAM solution will also help you rotate these passwords frequently.
2. Root admin accounts: These are also unavoidable in most modern Iaas and SaaS applications. Everything from AWS, Azure, GCP to Github has some form of a root account. Lock them down completely and limit access to them only during break-glass procedures.
3. Legacy application admin accounts: Treat these secrets very similar to database passwords. Store them in a modern PAM solution and inject them into sessions using a transparent proxy.

Workload identity federation

One of the emerging techniques for avoiding secrets sprawl is to use Workload Identity federation instead of passing secrets between services owned by different organizations. For example AWS has introduced “IAM Roles Anywhere”. You can use this to federate workload identities from outside AWS itself. GCP has introduced a robust “Workload Identity Federation” capability. Github has introduced OIDC for Github actions which can be used with cloud providers instead of storing secrets. We see more and more SaaS providers are adding support for some form of federation. This is a promising way forward.

Even when you use workload identity federation, there is a secret being used behind the scenes. There is an identity provider or a certificate authority that is issuing identity assertions that the other party has to trust. The private key associated with the workload identity provider or the certificate authority needs to be stored in a CloudHSM so that no one can ever extract it. Only the public key of the workload identity provider should be sent to the other parties so that they can verify the workload identity assertions.

Another thing to keep in mind is you need to make sure the attacker can not trick your workload identity provider to issue assertions to the attacker. When AWS and Github actions participate in OIDC federation, each party is trusting the other party to protect their own infrastructure from compromise. This is not an easy thing to do. If you are not sure how to do this well, it is better to use a solution like Procyon workload identity federation to solve this for you instead of trying to do this yourself.

Conclusion

Preventing credential sprawl in the cloud is a complex problem. Even the most sophisticated organizations are struggling with it. Traditional security tools and threat detection tools are not very suitable for detecting these identity threats. We would love to work with you on understanding your challenges and help you solve them. Please get in touch with us at [email protected].