At Ascend, we believe it’s time to rethink data engineering from the ground up.
As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Ascend has always been at the forefront of innovation, and with Project Inception, we’re setting a new standard. This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent.
A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.
Today, we will unravel the mystery behind “Project Inception,” provide a glimpse into the design principles behind the latest generation of our platform, and illuminate what’s in store for the future of Ascend users.
Plus, we’ll offer a teaser of what’s to come: an exclusive live demo that you simply cannot miss.
Let’s dive in!
The Evolution of Data Engineering
Over the past decade, we’ve seen tremendous change in the data engineering landscape. As companies scale their data and analytics capabilities, their data pipelines and operations have grown increasingly complex. The new scaling challenge is no longer the volume, velocity, or variety of your data, but rather, the speed, efficiency, and accuracy of your teams.
Many organizations start simple – ingesting data from a few sources, doing some transformations, and loading into a data warehouse. But as they add more data sources, implement real-time streaming pipelines, and build out a modern data lake, the complexity compounds quickly.
We’ve watched dozens of customers, from small startups to extremely large enterprises, across industries, encounter the same challenges with scaling their data platforms. As they ingest more data and build more pipelines, they end up grabbing a patchwork of tools from the broad data ecosystem. While each tool solves a specific need, stitching them together requires huge amounts of custom integration and scaffolding.
It’s time we stop Data Plumbing, and
get back to Data Engineering.
As data pipelines sprawl across disparate systems, it becomes harder to trace data lineage, track data quality, and quickly pinpoint issues when they arise. Lack of unified metadata makes it difficult to optimize pipelines, avoid redundant processing, and quantify data engineering costs. And data teams spend more and more time on plumbing and wiring, and less on delivering valuable data products.
While the data engineering ecosystem provides powerful point solutions, most companies still end up building custom scaffolding to wire everything together. This has led to common patterns emerging as teams scale their data platforms. There is a clear need for a unified foundation that removes one-off integration code and provides built-in metadata, automation, and observability.
The Need for a Unified Foundation
Project Inception stems from our desire to address these recurring challenges by establishing a more standard and unified core foundation for data engineering platforms. Our inspiration was drawn from the parallel between the foundational principles of Kubernetes, with its unified metadata and domain-specific control plane for container orchestration, and the void within the data engineering domain.
Over the past several quarters, we have embarked on a journey to architect an advanced control plane that propels data platforms into a new era, aiming to provide a extensibility and automation to the data engineering ecosystem.
What excites us most is not just the solution itself, but the potential for data teams. They will no longer need to wrap their processes around the technology; instead, they’ll be able to plug into a far more powerful and dynamic ecosystem.
The mission of Project Inception is to provide a data platform that’s not just extensible, but intelligent — one that anticipates, adapts, and augments the myriad tasks that fall under the data engineering umbrella. The convergence of metadata, AI, and automation will be the bedrock of a new world of possibilities in data automation.
Unveiling the Core of Inception
Imagine a platform where your data pipeline isn’t just a series of operations strung together, but an intelligent, dynamic entity. A system that doesn’t just run your code, but can adapt as things change around it. This is the heart of Project Inception – a powerful metadata core, DataAware automation, extensible capabilities, and AI Assistants, all designed to empower you to build data pipelines that are intuitive, adaptive, and intelligent.
At its core, it’s about empowering data engineers to work smarter, not harder. This means not just finding efficiencies in data processing but also codifying the best practices and intelligence in the very fabric of the platform.
In simple terms, Project Inception aims to create a new standard in data platforms. It’s a canvas for data engineers to create their masterpieces, with the platform taking care of the heavy lifting — pipeline optimization, monitoring, scaling, and much more. This level of automation is what will set Inception apart, enabling data teams to focus on what really matters — the data itself.
Beyond the Horizon: What Data Engineers Can Expect
While we won’t reveal specifics just yet (you’ll have to stay tuned for more!), future users of Inception can anticipate a suite of features that will redefine their approach to data. With its unified pipeline definition, automation engine, and an array of AI-infused capabilities, Inception will empower data engineers to craft resilient and adaptive data systems.
Unified Metadata
The assimilation of a unified metadata system means that every process is connected, and context is king. The typical data pipeline run can emit tens of thousands of telemetric, performance, and profile events that are key to understanding your data’s lineage, quality, and usage. Most of this data is never collected, and lost forever. Inception allows for a whole new level of comprehension and management that is paramount in data operations.
Unmatched Efficiency
Inception redefines efficiency with the level of automation it brings to the table. Building and managing data pipelines becomes a breeze, with the platform adapting to environmental changes and realigning strategies accordingly. Automation isn’t just about speed; it’s about maintainability, resilience, and predictability.
Scalability and Adaptability
No matter the volume or velocity of data, the platform ensures that your pipelines can handle the load. Inception employs a multi-tier scaling model that not only supports the largest of workloads, it can also scale down to conserve resources when possible.
Developer Ecosystem
Inception isn’t just a platform; it’s an ecosystem. Just as Kubernetes extended its capabilities through custom operators, we are creating a playground for developers. This will invite a collaborative effort to expand the platform’s capabilities, enriching it continuously with innovative features and toolsets.
Join Us on the Journey
Curiosity piqued? We hope so! We’re opening up private previews starting in May, and you can join us for a series of exclusive demos starting May 22nd. These interactive sessions will provide an inside look at Project Inception’s capabilities. Our product experts will demo key features and answer all your questions.
Space is limited, so save your seat today.
I can’t wait to embark on this exciting journey with you.
See you at the demos!
– Sean Knapp
Founder & CEO
FAQs
When will Project Inception be available?
Project Inception will be available for public preview in June 2024.
How does Inception differ from previous versions for the Ascend platform?
Inception introduces a suite of new features and enhancements, focusing on unification of metadata, automation, and AI assistants to optimize data pipelines.
Can I participate in the private preview?
Current Ascend customers are eligible to join the private preview. Please contact our team for more information.
