I’m an engineer that loves data and especially loves making data easier to work with. I’ve worked in a wide range of environments where data is a foundational asset, from medical research labs and healthcare supply chain companies, to corporate insurance companies and video streaming platforms.

No matter the environment or business task at hand, one thing always remained true: Turning data into insights at a rate faster than it’s collected is very difficult! Over the past decade, there has been a sea of software projects, particularly in the open source world, that have successfully tackled the hardware limitations of processing massive data at rest. These accomplishments have given rise to new critical roles in the tech industry, such as Data Engineer and Data Infrastructure Engineer. These are the heavy-lifters who take the firehoses pouring data into their organizations and bend it to their will into something useful.

I’d like for this blog to cover the next revolution that I think is happening today. If the past decade answered the question of “How can we process all of this data?”, I believe the next decade will answer the question of “How can we make use of all of this processed data?”. Specifically, the bridge between online transaction processing systems and online analytical processing systems will be shortened and streamlined. The time between collecting mountains of data and understanding what that data is signaling will trend towards becoming an instantaneous event. The series of “Extract, Transform, and Load” cadences will merge into a single process that takes in raw streams of data and outputs actionable business insights.

In the future, I hope to look back at this blog as a somewhat highlight-reel of this next chapter in such an exciting space!