FlightStats Open Source Project: How to easily share data among your teams

If you’re using big data to fuel your business, whether it’s for product development or insight, then there is one key question you must be able to answer:

How are you addressing the technical challenges of data management?

The ability to capture and store vast amounts of data has existed for many years; however, the technical capacity organizations have to aggregate and exchange these disparate volumes of information is still lagging behind. Many of today’s commonly used data management systems do little to contain cost, reduce risk or accelerate product development.

The problem with traditional databases

If you’re working with big data, then your teams need to be able to easily access the same data at all times as well as safely replicate it for their own projects. Additionally, the team that maintains the data needs to be able to organize it in a way that makes sense to the entire company.

Traditionally, companies have relied on databases to store and share data; however, most databases are not built to support big, fast or diverse data. Managing your data in a traditional database can put unnecessary burden on your teams.

In the end, using a traditional database to manage large amounts of varying data will slow your development process to a crawl and costs will stack up quickly. As your business evolves into a more data-centric organization, you must acknowledge the importance of having a more sophisticated approach to managing data.

Luckily, we’ve come up with a solution for you. We call it the Hub.

Why use the Hub

At FlightStats, we removed many of the risks and complexities associated with data management by creating the Hub. Unlike your typical database, the Hub allows you to easily share data among your teams, which results in increased productivity and lower overhead for your business. The best part is that the Hub can be used within any type of business that manages any type of data.

1. Add as much data as you want

We specifically built the Hub to be Cloud-like, giving it the ability to be scaled in elastic ways. Our Hub has a lot of data in it, but since it’s flexible, it can grow without any additional cost or development. If you need more data, then you can add more without any limitations.

2. Remove risk

Your teams have to be moving or getting data at all times. At FlightStats, we often have multiple teams working with a common dataset at the same time. This can introduce a lot of risk if you don’t have the right processes in place.

It’s critical to have protections that prevent too much modification when more than one person is working with the data. Your teams need to be able to safely replicate data to consume or send at any time.

With a traditional database, data replication could cause it to break; however, the Hub contains a simple, no risk process of replication that will prevent teams from losing or damaging data with unwanted changes. Once the data is in the Hub, it’s written in there forever. If you want to make a change to a dataset, then you have to replicate it as a brand new addition. The original version will always be there and so will each new version you create when you make a modification or change.

3. Develop products faster

Since traditional databases are not easy to run in a Cloud environment, the development process can become complicated. It would not be uncommon for a typical database to make your product development take an entire year.

At FlightStats, we have been known to take an idea and build a whole product within the timeframe of a week. This is largely due to the Hub and how easy it is to work with. We managed to accelerate our go-to-market ability by making data more accessible.

4. Keep data organized

Whenever you have a vast amount of data, you have to organize it in some way; however, you should keep your data ordering as simple as possible. Organizing data in more than one way can quickly become too complex.

You’ll want to organize data in a way that ensures every team member is speaking the same language. The better you can organize your data, the more productive your teams will be because they’ll spend less time trying to understand data files and more time working with those data files.

With the Hub, all data is sorted and organized by time. It’s like going into a library and instead of the Dewey Decimal System it’s everything is organized by time. If you came into a library and only had to know what time it was, then it’ll be much more simple to understand.

Start using the Hub right now

Sharing and transporting large amounts data easy isn’t easy. The Hub is the ideal solution to the data management challenge If you want to build your business to be stronger and more competitive. And it’s free to use right now. We made the Hub open source because we believe in giving back to the developers and data providers who have shared so much with us.

Interested in getting started? You can access the Hub through the Flightstats Github Project. If you have any questions, please feel free to reach out to us.