How to find the right testing tool for Okta, Auth0, and other SSO solutions

Implementing a single sign-on solution can be complicated, especially if you have apps that are not in the SSO vendor's catalog. These tools can simplify the process.

If you have bought a single sign-on (SSO) product, how do you know that is operating correctly? That seems like a simple question, but answering it isn’t so simple. Configuring the automated sign-ons will require understanding of the authentication protocols they use. You will also need to know how your various applications use these protocols—both on-premises and SaaS—to encode them properly in the SSO portal.

While the major SSO vendors support hundreds of apps, some of them—especially your own home-grown ones—may not already be in their catalogs. This means you might have the non-trivial effort to write your own custom login scripts. Some SSO vendors have tools or consultants (available either as part of the initial purchase or for an extra fee) to help with this.

Wouldn’t it be nice if you could run an automated testing tool to find out where you slipped up, or where your SSO software is failing? This can be useful in three scenarios:

• When your app vendors make small changes to their login screens, and these changes could trip up your automation routines that will cause your SSO login to fail. The major SSO vendors claim they have ways to stay on top of this, but if they don’t, your users will usually let you know before you can figure the issue out.
• When you are rolling out a new app, either a public SaaS app or something that you have developed internally and want to make sure your SSO connection is operating correctly.
• When a malicious actor leverages their way into your enterprise. (More on this later.)

To cover all these scenarios, you have two basic choices: The first is to use the testing tools written by your SSO vendor. The problem is that finding these tools isn’t easy. Many vendors treat these tools as more of a side development project rather than something that they offer with their mainstream product lines.

The second choice is to consider a third-party testing tool. A growing collection of these tools can help you determine if you have set up the numerous variables, cryptographic tokens, and policies correctly. The caveat is that they work with only a few of the SSO products—at least so far. While the absence of a test tool shouldn’t sway your SSO purchase decision, it’s good to know if one is available for your eventual implementation.

Another option is to employ a cloud access security broker (CASB) tool. These do other things beside automate the login to your SaaS apps, such as add another security layer.

Testing tools from major SSO vendors

When you first stand up your SSO implementation, you typically have a small test group of users to ensure that it is working as intended for a small sampling of your apps. Some SSO vendors can create a test environment to work out the bugs, such as provisioning and deprovisioning users, and identifying unknown apps (or at least unknown to the IT organization) that will need further attention.

Here is where things get interesting. If this is your first exposure to the Security Assertion Markup Language (SAML), OpenID Connect, and Open Authorization (OAuth) protocols, you will want to have some help in how your SSO vendor implements them. All the vendors have copious (and almost unreadable) documentation. That is where these test tools can come in handy, because they don’t require you to study the protocol syntax but offer a more human-friendly interface. To get a better idea of what lies ahead, you might read this blog from Testim, which is a general software automation testing provider. They cover the steps to set up typical SSO tests.

Two SSO vendors have details on SSO tests that are good starting places, particularly if you don’t know the difference between a SAML service provider and identity provider. You don’t need to be a customer to use most of these tools.

• NetIQ's (rebranded by Microfocus as CyberRes) series of public SAML libraries include a wider collection of programming languages.
• OneLogin’s series of SAML test tools are available in different languages (PHP, Python, Ruby, Java and .NET). One nice plus is that they are all available on GitHub. Recently, they released new workflow automation products that should make it easier to support general provisioning tasks, and they have other connectors that can help support custom app sign-ons.

You probably should have a separate test instance for your entire web app environment so you can experiment with SSO without harming your production systems. Four vendors have put these together:

• NetIQ (in addition to the tools above), has its Access Manger OAuth Playground where you can try things out if you are using their own identity management product.
• CyberArk has various SAML preview tools to validate the code generated for its apps.
• Ping Identity has its OAuth Playground, also for this purpose. You can experiment with OAuth and OpenID Connect scripts without harming your production systems. Ping has other tools, such as a SAML decoder where you can examine the XML request commands.
• Okta has something similar with its Integration Network. This has pre-built connectors to thousands of apps and a lot more: tools for additional security, analytics, and run-of-the-mill SaaS apps. There is also copious documentation on how to create your own integrations, along with wizards on creating SAML and OpenID Connect integrations. While that is a lot to review, at least the chances are great that they have come across most of the apps that your users run.

If you are using other SSO products, you will have to resort to a patchwork of tools:

• Auth0 offers help to troubleshoot SAML implementations, but the manual steps to resolve various error messages can be tedious. You can select a special debug mode to get a more verbose series of error messages. They have also built a service that maintains changes to the services once they are created. Even if you aren’t using Auth0, this is a useful ready reference if you want to learn about troubleshooting SAML.
• Matt Fuller offers helpful debugging tips for users of Amazon’s SSO and other identity management implementations for its web services.
• Finally, this open-source SAML tracing tool lets you debug and troubleshoot your SAML message traffic from inside your Chrome browser.

Third-party SSO testing tools

The problem with the SSO vendor toolset is that you have no way to automate tests across your entire applications infrastructure. This is where the third-party products come into play, such as Testim mentioned earlier. Testim is a general software testing automation vendor, of which there are dozens if not hundreds (such as Silk and Selenium, for example). While Testim doesn’t offer any SSO-specific tools, you can cobble together a way to test your SSO implementation if you have specialists on staff (or outsourced) that favor a particular automation program. Other software testing companies cover SSO implementations:

Canny

Canny works with Azure Active Directory and OneLogin. An Okta version is in beta. It requires embedded Canny widgets to pass SSO token. Pricing starts at $50 a month.

Dorothy

Elastic’s Dorothy works only with Okta and requires Elastic Security. Pricing starts at $16 a month.

SSOScan

SSOScan is an open-source vulnerability scanner for Facebook SSO only.

Walrus

Walrus works with Okta, Auth0, IBM, ForgeRock, and NetIQ. It is based on its general purpose automated testing tool. Pricing is not disclosed.

Many of these testing tools come from the general testing automation space, which means these vendors can test a lot more than your SSO applications. That is both good and bad news. Good because it shows that the automation vendors are finally paying attention to resolving SSO issues. Bad because the number of SSO customers is still a small subset of each vendor’s user base, which means finding a knowledgeable support person might be a challenge.

Actually, finding any support is a challenge. None of these vendors would respond to our queries. Another downside is that getting documentation about the SSO-related features will require some effort, too. (In Walrus’s case, getting their pricing proved impossible.)

The tools offer common features:

• Automated failure detection. If a login script fails, you can identify it before your users call to complain.
• Quick creation of new test scripts. Once you master the ins and outs of the tool, you can generate new scripts within a few minutes. Writing scripts should also be easier than debugging SAML or JSON code error messages.
• Software vulnerability testing. (I cover this below.)

If you use Okta’s SSO module, you can try out most of these automation tools and see which one you like the best. If you use Auth0 (which Okta acquired earlier this year but is maintaining as an independent product), Ping, or something else, then you have more limited options until vendors expand their support.

Security vulnerability testing

Putting together an SSO implementation, you will likely have a few issues to ensure that a bad actor doesn’t slip through your infrastructure. In an extreme example, a hacker was able to poison cryptographic tokens for escalating privilege for Active Directory.

Other circumstances are more common, such as:

• Can you easily figure out if users that are no longer employed have been successfully deprovisioned from your logins?
• Can you search to find unneeded admin users or simple passwords or misconfigured network zones?
• Can you parse a vendor’s SSO log and if so, how is this done? (Elastic’s Dorothy tool, for example, uses Fleet for precisely this purpose.)

You might think that general-purpose vulnerability scanners can do some of these tasks, but again, not necessarily focus on the sanctity of your SSO system. Enter SSOScan, which is the only general-purpose SSO vulnerability scanner I could find, made even more limited because it will only work with Facebook’s SSO logins. Because it is open source, you can examine how it is built and perhaps modify it to work with your SSO implementation. That may be more effort than it is worth if you don’t use and don’t care about Facebook integrations.

Testing your SSO configuration can be frustrating. Some new third-party tools can help, particularly if you run Okta’s SSO environment. Other SSO vendors have been slow to develop better automated methods. If you use Okta, Ping, CyberArk or Auth0 SSO, start with their tools. If you have something else, start with either OneLogin or NetIQ’s code libraries. Then add your automation layer with the tool most familiar to you.