Enabling Teams to Make Data Driven Decisions

As FlightStats has evolved from a startup with fewer than 10 employees to a stable, revenue generating company with more than 70 employees, we have had to evolve our infrastructure, technology tools and team structure to meet the needs of our growing customer base.  The main challenge in doing so is scaling the fast-paced, high-output, startup-like environment that has been successful for us in the past.

When companies are small, information flows easily between people and teams.  Engineers talk to executives, product managers talk to accountants, and almost everyone talks to customers.  Because of this free flow of information, smaller companies are more agile, meaning they can pivot or change directions quickly.

As a company grows, those lines of communication become hard to maintain.  Engineering teams become isolated, finding it hard to understand the business decisions being made by executives.  Revenue analysis and customer communication become specialized roles further decreasing communication to the engineers designing and building the products.  Successful companies find ways to bridge these gaps.  Specifically, they enable their teams to make good decisions with the best available data.

Since good decisions require good data, FlightStats has heavily invested in ensuring that our teams have the data that they need to make the best decisions possible.  We have realigned our engineering teams around products and provided the teams with the necessary tools to understand not only the performance of the apps they build and support, but also the business cases for the customers they serve.

When re-aligning teams, the goal was to make each team as independent as possible.  Each team was designed be a startup-like group of 5-10 people, including engineers, a product manager, an IT/DevOps engineer, and a sales and marketing representative.  We try to limit dependencies between teams as much as possible.  This reduces communication complexity and prevents teams from being blocked by each other, reducing the overhead of task swapping and enabling the highest possible velocity.  Our teams are given great freedom in the tools and technologies that they use to implement their products.  They are also given the responsibility of identifying and justifying a market before bringing a product to fruition.

Making good decisions requires good data of all sorts: application performance metrics, business metrics, revenue data, corporate strategic goal alignment, etc.  At FlightStats, we use New Relic to record and analyze both application performance and business metrics.  Our office is covered with dashboards displaying this information in real time.  Teams use these dashboards for everything from debugging technical issues to making priority decisions around products and projects.  We track the costs of infrastructure, coverage of our flight status data, volume of itineraries loaded, volume of incoming flight status data, load on our website and APIs, external threats to our network and many other metrics.  They are all displayed visually in one dashboard or another.  This data has allowed our teams to stay very agile.  They adjust their priorities daily based on the data they see in their dashboards.

Dashboarding performance metrics can also lead to large business savings.  After creating a dashboard showing web and mobile traffic over time, our Web and Mobile Apps (WMA) Team noticed large spikes in traffic to our web properties.  On further inspection, the traffic was identified as scraper bots harvesting large quantities of data from our website.  Since FlightStats is a data company we take the scraping of our data very seriously.  It costs us not only in revenue, but in opportunity cost and in the bandwidth and infrastructure it takes to handle the additional request load.  The WMA team used the dashboard data to write software to detect and block scraping attempts and also to dashboard the results for monitoring.  Scraping is a constant arms race so this software is maintained and monitored daily.

Our metrics also allow our teams to make better decisions about how to deploy their apps.  We push all of our cost data from Amazon Web Services (the public half of our hybrid cloud architecture) into New Relic Insights.  This allows our teams to track the costs of the compute and storage infrastructure that hosts their apps.  Parameters like EC2 Instance Size can be tuned to meet performance requirements while controlling the cost to operate the app.

Company-wide metrics help the executive team build new strategies and explore new market opportunities.  Inbound data throughput metrics from our 500+ sources of data helps show us sources that may be underperforming leading to overall data quality loss.  Reach and geographic coverage are also monitored to help prioritize future data acquisition and processing efforts.  These same metrics also help our support team to send timely answers to customer support requests and to alert our data provider partners if there is a problem on their end.

FlightStats is a data company, so we naturally tend toward using good data to make data-driven decisions.  Keeping these decisions as close to the teams as possible allows us to stay agile and change direction quickly to meet the ever-changing, heterogeneous needs of our growing customer base.  We continue to make investments in internal metric gathering and business intelligence systems to make the best data-driven decisions possible.