Time Series Database with a Dynamic Schema — what the HOG do we need this for?

Getting to know the ROI of time-series data

For those coming from the world of SQL databases, time-series data can be a little challenging at first.

Perhaps the biggest challenge is the sheer volume of data generated by systems. A traditional SQL database can handle time-series data, but as you scale, the overhead of maintenance and operations starts to outweigh the value of the data.

In addition, an SQL database also requires you to define in advance the schema for the data you plan to store. Is the data a string or an integer? Does it conform to a specific character length? There are endless permutations of these questions involved in defining a schema. Of course, the advantage of a defined schema is that it lets the SQL database process queries quickly.

Having to define a schema in advance is problematic for time-series data. A project involving timestamped data often comes with unknowns, so your system needs to be flexible enough to accommodate changing needs.

For example, you might need to collect this type of data from many different sources, each producing results in a different format. What’s more, you might not even know the actual number of endpoints your database will collect metrics from when you get started.

With all these challenges, it might seem like time-series data is a burden. It isn’t.

All you need are the right tools to easily unlock the tremendous value and deeper insights that time-series data provides.

A time-series database is a database specifically built to handle timestamped data. Using a time-series database with a dynamic schema approach (also called an implicit schema or schema-on-write) eliminates these problems right out of the box.

Another time-saving benefit is that this type of database can accept any data that conforms to configurable ingestion protocols, setting a low bar for acceptable data.

Users can simplify the process by simply configuring an agent to collect data from each component of their system. This step ensures that the metrics conform to the ingestion protocol. Once you’ve configured the data collection agent and the metrics are loaded into the database, a schema is created. As new data arrives in the time-series database, the schema is continually updated as needed.

The benefits of a dynamic schema for time series

Evidence continues to build showing that users can benefit from time series’ dynamic-schema flexibility. Here are a couple of popular examples.

In a DevOps monitoring situation, you may not know all the information coming from a particular server, system, or infrastructure. In this situation, you might get hundreds or thousands of metrics, and defining a schema for them would be a very labor-intensive process. A dynamic-schema system removes this planning step and, as a result, speeds up the development process.

The Internet of Things is another area where a dynamic schema offers an advantage. Suppose you deploy equipment in the field, and that equipment has its own set of sensors or systems that generate metrics. You may perform remote updates to this equipment that change the type of information the device sends back to your database.

If you have to spend time coordinating changes to incoming data in your data stores, the update process becomes much more complicated and cumbersome. If you want to configure and improve the software on your IoT devices in the field and start collecting new metrics with an update or a change to a metric name, you don’t want data to be lost or deleted from the database.

With a dynamic schema, you keep collecting whatever metrics your devices generate, and the database adapts the schema to them as they arrive.

In both examples above, the speed of development and deployment with a dynamic schema is significantly higher than with an SQL database. It can take a company weeks to add one field to an SQL database schema, and that may require several meetings with database administrators and infrastructure engineers.

Final thoughts

If you want to build applications that use time-series data, use a database that supports a dynamic schema for the best results.

This will not only speed up development, letting you deploy faster at the beginning of your project, but it will also make your applications more flexible as they evolve. That’s because you can add new dimensions on the fly to quickly build new features, generate more robust data, and improve the user experience. In short, a time-series database with a dynamic schema lets you build awesome applications in next to no time.