Auth0 Internal Dev Tools: Unleashing Engineering Potential

Auth0 is continuously growing. To share a couple of examples, we just hit a new milestone: 2.5B+ monthly logins securely authenticated and we ended 2018 with a nearly 80 percent increase in both sales and new customers.

We are growing and scaling the organization at a really fast pace, and engineering is no exception. This presents some interesting and fun challenges:

1. Promoting and developing the culture that fuels this endeavor.
2. Hiring and retaining great engineers.
3. Minimizing duplicate work and time to be productive as new people come in.

This blog post is focused on how Auth0 is addressing part of that last challenge; by building a dedicated team to unleash the potential in Auth0 developers to deliver business value with confidence.

Developer Tools Team at Auth0

In the same way that Auth0's goal is to make identity simple for developers, the Developer Tools team goal is to let our engineers fully focus on development and providing value for our customers. We want to provide the best tools to keep our developers happy and engaged shipping new features instead of struggling every day with exhausting and repetitive tasks that could be automated.

We defined the mindset we want to adopt to develop our tools and outlined a roadmap with the following vision:

Provide toolchains that Auth0 engineering teams can use to improve their workflow, automate manual tasks, simplify complex processes, and implement a set of company-wide standards.

What we do

1. Work closely with the engineering teams to understand their pain points, provide tools and guidance.

2. Investigate existing tools and develop new ones to solve their needs:

   • Development Environment tools.
   • Change Management tools.
   • Debugging and Performance tools.
   • Incident Response tools.
   • Work with the SRE team to integrate site reliability tools into the platform.

3. Recommended patterns & practices:

   • Identify patterns currently in use across our organization.
   • Provide developer tools for Scaffolding and Code Snippets.
   • Promote the adoption and tooling for the best engineering practices, e.g.: contract testing.

What we don’t do

1. While we provide general guidance, we don’t do consultancy, i.e.: we neither do work at a particular team's requests to tackle specific issues, provide advice on demand, or do an investigation by request nor develop customized solutions to solve a specific team's needs.
2. We don’t provide advice or work on particular issues on demand: our tools are developed following a product approach.
3. We don’t embed team members on another team to build tooling. We might do it for discovery, but development happens within the Development Tools team.
4. We don’t debug and profile internal services, we provide the toolchain for that.
5. We don’t do SRE (Site Reliability Engineering), we might provide tooling that could improve the experience during their analysis.
6. We don’t investigate a random tool or technology on demand. We do it if it is related to reducing developer pain.

"Learn how our engineers stay focused on solving the hard problems of identity for our customers."

Tweet This

Tools Development Mindset

We are approaching the development of the tools as internal products, guided by these principles:

1. Ownership: We own the tools that we ship, following a “You Build It, You Run It” principle.

   • We operate what we build: taking care of the CI/CD, incident response, etc.
   • We take care of not just the coding but also keeping the product running in good shape.

2. Extensibility: We develop these tools taking extensibility into account from scratch.

   • For example, any engineer should be able to easily extend the CLI that we provide to interact with Auth0’s local development environment.

3. Internal Collaboration: We engage our users to contribute with the tool evolution, collecting their feedback and welcoming their contributions to the code and documentation (via pull requests on git repositories).

We aim to follow an iterative process to develop our tools (just like our Product Engineering teams), working closely with our internal customers.

The process adopted by our team suits the lean startup cycle model.

-><-

Lean Startup Model by Jeff Patton is licensed under CC BY 2.0.

"The Auth0 Developer Tools team shares three key principles to develop a tools-development mindset that helped them create successful internal products for developers."

Tweet This

Our Tools

Some of the tools that we currently maintain and regularly work on had a great impact and adoption rates within the organization but were built prior to adopting the internal dev tools as product mindset mentioned above. It was originally hard for us to learn from the tools once released without any kind of usage metrics. For that reason, we focused on existing solutions by:

• Improving documentation, both user guides and internal playbooks concisely describing deployment, troubleshooting, and contribution guidelines.
• Instrumenting usage and performance metrics to measure the impact of our iterative improvements.

Vivaldi

-><-

Photo by Guillaume Bolduc on Unsplash.

Vivaldi is the tool on which we have invested most of our efforts so far. Named after the Italian musical composer, it is a Docker container-based development environment that allows engineers to run a local Auth0 on their machine. Through a command line interface (CLI), Vivaldi provides commands to quickly interact with our service instances running locally in order to:

• Update dependencies.
• Run tests.
• Attach a debugger.
• Run diagnosis tools.
• Execute performance profilers.

-><-

The initial version of the tool had a great impact on the development cycle and was widely adopted. Before Vivaldi, developers needed to manually spin up the services they were working on, fighting with dependencies with their local environment. Now they can rely on an isolated and replicable environment, being able to spin up the entire services cluster relatively quickly.

As our teams and number of services grow, we are now actively working with teams to better implement a fast development loop without relying too much on running all Auth0 services locally. We are already planning the next generation of Vivaldi, which will support future growth in a scalable way.

Slack Bots

-><-

Photo by Franck V. on Unsplash.

We developed an internal Slack bot platform called Gynoid during a hackathon and it was quickly adopted by our engineering organization. The platform is customized to satisfy the needs of our developers:

• To build Slack bots easily: just create a JavaScript file, add metadata, and you are done!.
• To install Slack bots straight from a GitHub repository.
• To extend existing bots from different sources: one bot, multiple functions, different repositories.
  • We call these extensions Droids. To install new extensions, we have a Gynoid that serves as a wrapper for this library. Our Gynoid Droid is responsible for registering and extending our bots directly from Slack.

A notable Droid is one that serves as a concierge for any Slack channel, properly named @concierge. It works like this:

• A channel that needs a person on duty to address direct questions invites @concierge to the channel.

-><-

• A teammate is assigned the role of @concierge.

-><-

• When someone needs to ask anything to a team, for example, they visit the team's channel and ask the question directly to @concierge.

-><-

• The teammate with the role of @concierge is sent a direct message through Slackbot with the concierge request.

-><-

• The concierge has now the responsibility to research and to address the request directly. If the request is outside of the scope of the team, the request is redirected to the appropriate channel

The concierge droid helps us avoid unnecessary noise in Slack — especially since it is widely used within our company as a good substitute for @here and @channel when asking questions. It's very useful when you have an "on-call" rotation in your team and people need to reach the teammate on call through a Slack channel.

Roadmap

We are currently working on the next major version of our main tools, focusing on:

• Scalability improvements: Vivaldi, our Docker-based development environment, will run an increasing number of services without compromising the developer machine resources.
• Robustness: Gynoid, our slack bots platform, will have end-to-end tests and its dependencies will be upgraded to the latest stable versions, allowing our team and contributors from other teams to introduce changes or develop new droids with confidence.

Additionally, we’re in the early stages (discovery, viability analysis, and ideation) for a unified internal CLI (command-line interface): we already have multiple CLIs that provide different types of functionality. As the number of CLIs keeps growing, it will become hard to keep track of them.

Some teams will decide to use a different tech stack, which may require different runtime dependencies and a distinct CLI to better manage it. However, we could create a unified CLI, customizable and extensible to meet the needs of different tech stacks. Having a unified CLI will help internal toolchains scale while staying maintainable.

Conclusion

We aim to provide an optimal developer experience, unified across the engineering organization. To do this we take ownership of the toolchains, iterate to improve them and encourage teams to use these tools that have already proven their value on making developers more productive.

We dedicate our Developer Tools team efforts to the discovery, building, maintenance, and promotion of reusable solutions so that the rest of the engineers don’t have to struggle with repetitive tasks, walk through slow development cycles, or constantly switch contexts to take care of maintenance tasks that can be automated.

We want Auth0 engineers to be focused on building new features that solve the hard problems of identity for our customers.