Monte Carlo

APRIL 24, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. Check out Part 1 of the build vs buy guide to catch up. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Towards Data Science

MARCH 9, 2023

Code implementations for ML pipelines: from raw data to predictions Photo by Rodion Kutsaiev on Unsplash Real-life machine learning involves a series of tasks to prepare the data before the magic predictions take place.

Machine Learning 79

Machine Learning Building Datasets Scala 79

Christophe Blefari

APRIL 21, 2023

If a model do not respect a contract it will not build. In dbt vocabulary build means run + other things. Building a ChatGPT Plugin for Medium. Fast News ⚡️ Building a Flink self-serve platform on Kubernetes at scale — Instacart engineering team migrated from Flink on EMR to Flink on Kubernetes.

Raw Data 130

Raw Data Data Datasets SQL 130

Snowflake

NOVEMBER 27, 2023

Welcome to Snowflake’s Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. Traveling over hard ground on the way to building something important is what inspires me. Hum is harnessing frontier AI to transform content and audience data into actionable insights and personalized experiences.

Raw Data 99

Raw Data Relational Database Consulting Architecture 99

AltexSoft

MAY 12, 2022

Commercial audio sets for machine learning are definitely more reliable in terms of data integrity than free ones. The same relates to those who buy annotated sound collections from data providers. Audio data labeling. Building an app for snore and teeth grinding detection. Commercial datasets.

Machine Learning 122

Machine Learning Building Datasets Healthcare 122

Monte Carlo

JANUARY 6, 2023

A data-informed product strategy Product teams are tasked with being data-informed at all stages in the product life cycle, from idea generation and product definition, to validating prototypes and building a commercially successful product. But so often data is not a first-class citizen in product launches.

Building 52

Building Project BI Data 52

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. Lack of governance Our platform lacked governance policies for metric definitions.

SQL 82

SQL Metadata Raw Data Government 82

Snowflake

APRIL 7, 2023

This was the first year that startups had the chance to build with our Native Applications Framework (currently in private preview), and we were thrilled to see the number of entries that included a native app. It transforms multiple financial and operational systems’ raw data into a common, friendly data model that people can understand.

Raw Data 108

Raw Data Portfolio Building SQL 108

Monte Carlo

JANUARY 12, 2023

Today, we have a data team of about 20 professionals organized among three teams. The first team is responsible for building the backend data infrastructure. The second team, data services , are more application or front-end oriented. Also, Snowflake scalability and features like data sharing were very attractive.

Architecture 52

Architecture Building Data Warehouse BI 52

Monte Carlo

MARCH 28, 2023

And so it almost seems unfair that new ideas are already springing up to disrupt the disruptors: Zero-ETL has data ingestion in its sights AI and Large Language Models could transform transformation Data product containers are eyeing the table’s thrown as the core building block of data Are we going to have to rebuild everything (again)?

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

dbt Developer Hub

DECEMBER 5, 2021

We’ve also included some sample raw data to add to your warehouse so you can run these projects yourself! It's more important to think about how features build upon themselves (and each other) rather than how quickly they do so.* You can use this repository to benchmark the maturity of your own dbt project.

Project 52

Project Building Metadata BI 52

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Data Engineering Podcast listeners can sign up for a free 2-week sandbox account, go to dataengineeringpodcast.com/tonic today to give it a try!

Data Management 100

Data Management Management MongoDB Scala 100

Monte Carlo

FEBRUARY 9, 2023

For those unfamiliar, data vault is a data warehouse modeling methodology created by Dan Linstedt (you may be familiar with Kimball or Imon models ) created in 2000 and updated in 2013. Data vault collects and organizes raw data as underlying structure to act as the source to feed Kimball or Inmon dimensional models.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Knowledge Hut

DECEMBER 26, 2023

Businesses benefit at large with these data collection and analysis as they allow organizations to make predictions and give insights about products so that they can make informed decisions, backed by inferences from existing data, which, in turn, helps in huge profit returns to such businesses. What is the role of a Data Engineer?

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Towards Data Science

APRIL 3, 2023

And so it almost seems unfair that new ideas are already springing up to disrupt the disruptors: Zero-ETL has data ingestion in its sights AI and Large Language Models could transform transformation Data product containers are eyeing the table’s thrown as the core building block of data Are we going to have to rebuild everything (again)?

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Cloud Academy

JANUARY 27, 2022

A data engineer is an engineer who creates solutions from raw data. A data engineer develops, constructs, tests, and maintains data architectures. Let’s review some of the big picture concepts as well finer details about being a data engineer. You’ll learn how to load, query, and process your data.

Data Engineering 52

Data Engineering Data Engineer Certification Engineering 52

Meltano

OCTOBER 5, 2022

This includes various day-to-day activities, from reducing development time and improving data quality to providing guidance and support to data team members. This involves continually striving to reduce wasted effort, identify gaps and correct them, and improve data development and deployment processes. The best part?

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

Precisely

JANUARY 5, 2023

For some companies that do have a formal strategy, that strategy may be little more than a technical exercise, the primary purpose of which is to lay out the nuts and bolts of data management, compliance, and similar baseline requirements. Data governance plays a critical role in any effective data strategy.

Data Governance 52

Data Governance Government Data Raw Data 52

The Modern Data Company

MARCH 1, 2024

Beyond the Analogy: A Closer Look at Data Products Think of a Data Product as a Lego brick. Each brick (data) has specific properties and can be combined with others (transformations) to build something larger and purposeful (insights). Collaboration: Data Products facilitate collaboration through Data Contracts.

Database-centric 40

Database-centric Data Lake Government Retail 40

dbt Developer Hub

SEPTEMBER 28, 2022

So let’s say that you have a business question, you have the raw data in your data warehouse , and you’ve got dbt up and running. If your analyst is not trained as an analytics engineer, this is the point that they will need to hand the project over to a data engineer to build the model. Or are you?

Engineering 52

Engineering Raw Data Datasets BI 52

Knowledge Hut

JANUARY 30, 2024

In today's world, where data rules the roost, data extraction is the key to unlocking its hidden treasures. As someone deeply immersed in the world of data science, I know that raw data is the lifeblood of innovation, decision-making, and business progress. What is data extraction?

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

ProjectPro

FEBRUARY 16, 2023

Business Intelligence and Artificial Intelligence are popular technologies that help organizations turn raw data into actionable insights. While both BI and AI provide data-driven insights, they differ in how they help businesses gain a competitive edge in the data-driven marketplace.

Business Intelligence 52

Business Intelligence BI Data Mining Raw Data 52

The Modern Data Company

MAY 10, 2023

Companies need more than definitions. In a world where technology evolves, and data assets have exploded in volume, it helps to know the best use cases for each of these solutions and when to avoid them. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

DataKitchen

OCTOBER 5, 2021

A DataOps Engineer owns the assembly line that’s used to build a data and analytic product. While car companies lowered costs using mass production, companies in 2021 put data engineers and data scientists on the assembly line. Imagine if a car company asked the engineers who designed cars to also build them.

Engineering 60

Engineering Raw Data SQL Data Engineering 60

RandomTrees

DECEMBER 20, 2023

Thus, Edge AI reduces downtime or latency by combining Edge computing and AI processes. Furthermore, next gen edge computing devices successfully integrate AI processes as an essential component. As a result, users can build data and support user requirements without interacting with physical locations, saving them time and money.

IT 52

IT Hospitality Healthcare Deep Learning 52

ProjectPro

JUNE 9, 2021

As we proceed further into the blog, you will find some statistics on data engineering vs. data science jobs and data engineering vs. data science salary, along with an in-depth comparison between the two roles- data engineer vs. data scientist. vs. What does a Data Engineer do? What is Data Science?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Knowledge Hut

JUNE 30, 2023

This article suggests the top eight data engineer books ranging from beginner-friendly manuals to in-depth technical references. What is Data Engineering? It refers to a series of operations to convert raw data into a format suitable for analysis, reporting, and machine learning which you can learn from data engineer books.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

AUGUST 31, 2023

Why we originally built features with SQL Feature engineering and construction isn’t much different than other modern data pipeline architectures. You start with raw data from a source, combine it with other data, and then transform it into the desired state for your machine learning model to consume.

SQL 64

SQL Data Science Machine Learning Algorithm 64

The Modern Data Company

MAY 10, 2023

Companies need more than definitions. In a world where technology evolves, and data assets have exploded in volume, it helps to know the best use cases for each of these solutions and when to avoid them. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

Companies need more than definitions. In a world where technology evolves, and data assets have exploded in volume, it helps to know the best use cases for each of these solutions and when to avoid them. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

Monte Carlo

OCTOBER 2, 2023

Within their lakehouse, multiple layers of data are available to different teams for different business applications. The bronze layer is the “landing zone” of raw data. The silver layer is cleansed and queryable by multiple data teams. But that still left one question: build or buy ? And Monte Carlo delivered.

Pharmaceutical 72

Pharmaceutical Healthcare Machine Learning Architecture 72

Knowledge Hut

NOVEMBER 27, 2023

A Data Scientist earns about 25% more than a computer programmer. A person with a die-hard passion to work on large datasets and to draw meaningful insights can definitely begin their journey in becoming a great data scientist. What is Data in Data Science? Data is the essence of Data Science.

Data Science 52

Data Science Datasets Machine Learning Algorithm 52

Monte Carlo

MARCH 10, 2022

Monte Carlo’s Barr Moses sat down with Snowflake Director of Product Management Chris Child to talk about building data platforms at scale, how awesome data teams approach data quality, the role of data observability tools in the modern data stack, and more. Just get the raw data in there.

Engineering 52

Engineering Management Raw Data Scala 52

Precisely

FEBRUARY 6, 2023

If digital transformation initiatives are to deliver on their promises, they need accurate, consistent, contextualized, and rich data. What Is Data Integrity? Until recently, the business community has lacked a clear and consistent definition of data integrity.

Data Integration 52

Data Integration Data Governance Government Raw Data 52

Knowledge Hut

JUNE 20, 2023

Factors Data Engineer Machine Learning Definition Data engineers create, maintain, and optimize data infrastructure for data. In addition, they are responsible for developing pipelines that turn raw data into formats that data consumers can use easily.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. However, the medallion architecture brings a clear bucketing of data to align with the organization's delivery strategy from raw data → filer & clean data, → business metrics.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

DataKitchen

FEBRUARY 2, 2023

And the tools for acting on data are consolidating: Tableau does data prep, Altreyx does data science, Qlik joined with Talend, etc. Most data governance tools today start with the slow, waterfall building of metadata with data stewards and then hope to use that metadata to drive code that runs in production.

Metadata 52

Metadata BI Government ETL Tools 52