Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

DoorDash Engineering

NOVEMBER 28, 2023

It is amusing for a human being to write an article about artificial intelligence in a time when AI systems, powered by machine learning (ML), are generating their own blog posts. We also shed light on how we drove value by taking a user-centered approach while building this internal tool.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Data Engineers are engineers responsible for uncovering trends in data sets and building algorithms and data pipelines to make raw data beneficial for the organization.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. At the end of 2022 we decided to collect the blogs we enjoyed the most over the year. For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

Pipeline-centric 52

Pipeline-centric Database-centric Data Warehouse SQL 52

Snowflake

JUNE 28, 2023

Streamlit has rapidly become the de facto way to build UIs for LLM-powered apps. Get smarter about your data with native LLMs Additionally, Snowflake is building LLMs directly into the platform to help customers boost productivity and unlock new insights from their data. Read this blog. And with LLMs, it’s no different.

Pipeline-centric 75

Pipeline-centric Unstructured Data Government Data 75

Towards Data Science

MAY 19, 2023

Data engineers spend countless hours troubleshooting broken pipelines. Data plays a central role in modern organisations; the centricity here is not just a figure of speech, as data teams often sit between traditional IT and different business functions. More can be found in this blog. Know when to build and when to buy.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Snowflake

JUNE 28, 2023

It provides familiar APIs for various data centric tasks, including data preparation, cleansing, preprocessing, model training, and deployments tasks. In the warehouse model, users can seamlessly run and operationalize data pipelines, ML models, and data applications with user-defined functions (UDFs) and stored procedures (sprocs).

Python 52

Python Accessible Accessibility Pipeline-centric 52

Lyft Engineering

APRIL 3, 2023

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders. This blog mostly focuses on the mode selector to explain how rankings have evolved in the past years and briefly touches on the post request cross-sells.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87

U-Next

AUGUST 17, 2022

SQL – A database may be used to build data warehousing, combine it with other technologies, and analyze the data for commercial reasons with the help of strong SQL abilities. Pipeline-centric: Pipeline-centric Data Engineers collaborate with data researchers to maximize the use of the info they gather.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Data Engineering Weekly

APRIL 2, 2023

link] Tweet Search System (EarlyBird) Design [link] Google AI: Data-centric ML benchmarking - Announcing DataPerf’s 2023 challenges Data is the new code: it is the training data that determines the maximum possible quality of an ML solution. I echoed a similar statement here. Seriously, come on, Redshift team!!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Workfall

AUGUST 29, 2023

Learn how to build your very own Amazon Price Tracker using Python. In this blog, we will cover: About Python About Amazon About BeautifulSoup Library Hands-On Conclusion About Python Python is a high-level, interpreted, and versatile programming language known for its simplicity and readability. appeared first on The Workfall Blog.

Python 93

Python Pipeline-centric Programming Language Coding 93

DataKitchen

AUGUST 27, 2021

DataOps is fundamentally about eliminating errors, reducing cycle time, building trust and increasing agility. The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Lake Data Analytics 98

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. DoorDash writes about its journey to build a Metric Layer for experimentation.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

NOVEMBER 15, 2021

This blog discusses quantifications, types, and implications of data. The activity in the field of learning with limited data is reflected in a variety of courses , workshops , reports , blogs and a large number of academic papers (a curated list of which can be found here ). Quantifications of data. Addressing the challenges of data.

Unstructured Data 88

Unstructured Data Pipeline-centric Database-centric Entertainment 88

Cloudera

OCTOBER 17, 2023

This blog is for anyone who was interested but unable to attend the conference, or anyone interested in a quick summary of what happened there. Use cases such as fraud monitoring, real-time supply chain insight, IoT-enabled fleet operations, real-time customer intent, and modernizing analytics pipelines are driving development activity.

Database-centric 77

Database-centric Kafka Pipeline-centric Database 77

Cloudera

APRIL 22, 2021

Cloudera has partnered with Cisco in helping build the Cisco Validated design (CVD) for Apache Ozone. Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Cloudera will publish separate blog posts with results of performance benchmarks.

Pipeline-centric 100

Pipeline-centric Data Lake Hadoop Metadata 100

DataKitchen

FEBRUARY 21, 2023

As a result, a less senior team member was made responsible for modifying a production pipeline. Build analytic data systems that have modular, reusable components. Build components that are idempotent on data. To add fuel to the fire, a few weeks ago, one of the most talented people in the group left.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Snowflake

AUGUST 21, 2023

In this blog post, we’ll review the core data mesh principles, highlight how both organizations and modern data platforms are putting those principles into action, and demonstrate just how achievable a secure and efficient data mesh architecture can be. It’s also crucial that teams consider the price tag of the data mesh.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Picnic Engineering

DECEMBER 19, 2023

In this blog, we’d like to give you a glimpse into some of the major developments in Picnic Tech in 2023. Combining efficient incident handling, establishing resilience by design, and strict adherence to SLOs are pivotal in ensuring our services remain resilient, reliable, stable, and user-centric. Join us and have a read! The impact?

Transportation 52

Transportation Pipeline-centric Database-centric Python 52

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 103

Data Pipeline Data Engineering Data Engineer Engineering 103

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Pinterest Engineering

OCTOBER 31, 2023

At Pinterest, we are evolving our compute platform to provide an application-centric and fully managed compute API for the 90th percentile of use cases. Architecture PinCompute is a regional Platform-as-a-Service (PaaS) that builds on top of Kubernetes. PinPod is the basic building block for general purpose compute at Pinterest.

Architecture 97

Architecture Pipeline-centric Accessible Accessibility 97

DataKitchen

NOVEMBER 4, 2021

The DataKitchen Platform serves as a process hub that builds temporary analytic databases for daily and weekly ad hoc analytics work. These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. . The DataOps Advantage .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Databand.ai

DECEMBER 20, 2022

He is also an open-source developer at The Apache Software Foundation and the author of Hysterical , a popular blog on tech careers and topics like data, coding, and engineering. When he’s not creating videos, he might be creating his next patent to add to the 200+ he already has.

Data Analytics 52

Data Analytics Google Cloud Data Science Data Mining 52

Monte Carlo

JULY 28, 2022

These are particularly frustrating, because while they are breaking data pipelines constantly, it’s not their fault. As Convoy Head of Product, Data Platform, Chad Sanderson wrote in our blog , “The data in production tables are not intended for analytics or machine learning. He suggested : “Private vs. public methods.

Pipeline-centric 52

Pipeline-centric Software Engineer Software Engineering Database-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Knowledge Hut

MARCH 22, 2024

In this blog post, I will guide you through crafting a compelling response to "Why should we hire you," provide examples, discuss “Why should we hire you answers,” and highlight common pitfalls to avoid in the process.

Recruitment 52

Recruitment Healthcare Pipeline-centric Hospitality 52

Databand.ai

DECEMBER 29, 2022

Chad writes on data management, contracts, and products on his Substack blog and serves as an advisor and investor to several startups. He recently joined Databand’s MAD Data Podcast to talk about how his team is building one of the most advanced experimentation and machine learning platforms in the world from the ground up.

BI 52

BI Consulting Data Science Data Governance 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. Elasticsearch has become ubiquitous as an index centric datastore for search and rose in tandem with the popularity of the internet and Web2.0.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

phData: Data Engineering

JULY 18, 2023

This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data. Legacy tools force users to manually build out processes that can be automated by the Modern Data Stack. Why Migrate to a Modern Data Stack?

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Building the KPay payment system. Pillar 1 – Business function: Payment processing pipeline.

Kafka 91

Kafka Pipeline-centric Architecture Database-centric 91

DataKitchen

JUNE 24, 2021

Agility is absolutely the cornerstone of what DataOps presents in the build and in the run aspects of our data products.”. We took over an entire building where we co-located all the resources to ensure very high speeds, and high effectiveness of the team.” Be business-centric. GE also worked hard to democratize data. “We

Pipeline-centric 84

Pipeline-centric Manufacturing Education Data Cleanse 84

Airbnb Tech

JUNE 1, 2021

On the one hand, core_data standardized table consumption and allowed users to quickly identify which tables to build upon. Furthermore, pipelines built downstream of core_data created a proliferation of duplicative and diverging metrics. Figure 1: Event sources and dimension sources are the fundamental building blocks of Minerva.

Datasets 52

Datasets Pipeline-centric Metadata Data Science 52

Knowledge Hut

JULY 7, 2023

Best Certification Courses in Full-Stack Development In this blog, we will delve into the top 20 Full-Stack Developer courses that can help you embark on this exciting career journey or take your existing skills to new heights. Looking to explore more Full-Stack Web Development Courses with certification? Keep reading! So, let's begin.

Certification 52

Certification MongoDB Database-centric Programming 52

RandomTrees

NOVEMBER 29, 2023

But this approach would also be more susceptible to cost and build factors. Normally traditional warehouses found it very hard to implements an agile system of approach in building a traditional data/enterprise warehouse for their business needs. So, what is a data vault model or modelling approach?

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

Scott Logic

NOVEMBER 6, 2023

Testing Library describes themselves as “a family of packages that helps you test UI components in a user-centric way”. Those discussions eventually led to this series of blog posts. These tests can also be run as part of a build pipeline, including as part of automated PR checks. He has a fantastic blog.

Accessible 52

Accessible Accessibility Coding Pipeline-centric 52