Start Data Engineering
JUNE 14, 2024
1. Introduction 2. Run Data Pipelines 2.1. Run on codespaces 2.2. Run locally 3. Projects 3.1. Projects from least to most complex 3.2. Batch pipelines 3.3. Stream pipelines 3.4. Event-driven pipelines 3.5. LLM RAG pipelines 4. Conclusion 1. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up!
Confessions of a Data Guy
JUNE 20, 2024
After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. I wish. Everything has changed in 10 years, yet nothing has changed in 10 years, how is that even possible? Sometimes I wonder if I’ve learned anything […] The post What I’ve Learned After A Decade Of Data Engineering appeared first on Confessions of a Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Waitingforcode
JUNE 24, 2024
Last May I gave a talk about stream processing fallacies at Infoshare in Gdansk. Besides this speaking experience, I was also - and maybe among others - an attendee who enjoyed several talks in software and data engineering areas. I'm writing this blog post to remember them and why not, share the knowledge with you!
Rockset
JUNE 21, 2024
I’m excited to share that OpenAI has completed the acquisition of Rockset. We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI. From the start, our vision at Rockset was to fundamentally transform the way data-driven applications were built. We developed our search and analytics database, taking full advantage of the cloud, to eliminate the complexity inherent in the data infrastructure needed for these apps.
Speaker: Jay Allardyce
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
KDnuggets
JUNE 20, 2024
Image by author Model deployment is the process of trained models being integrated into practical applications. This includes defining the necessary environment, specifying how input data is introduced into the model and the output produced, and the capacity to analyze new data and provide relevant predictions or categorizations.
databricks
JUNE 12, 2024
Over the last year, we have seen a surge of commercial and open-source foundation models showing strong reasoning abilities on general knowledge tasks.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Engineering at Meta
JUNE 12, 2024
As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of computation required to train large language models (LLMs). Traditionally, our AI model training has involved a training massive number of models that required a comparatively smaller number of GPUs.
Robinhood
JUNE 6, 2024
This acquisition will bring Bitstamp’s globally-scaled crypto exchange to Robinhood, with retail and institutional customers across the EU, UK, US and Asia. This strategic combination better positions Robinhood to expand outside of the US and will bring a trusted and reputable institutional business to Robinhood. Expected to close in the first half of 2025, subject to customary closing conditions, including regulatory approvals.
Edureka
JUNE 7, 2024
Is AI taking over the world? Umm, not yet, at least. However, according to a recently published report , almost 35% of global companies report using AI to optimize their business. In this article, we will take a closer look at two of the most talked about and widely used AI technologies of 2024 – generative AI and predictive AI. Table of Contents Generative AI vs Predictive AI – Comparison Table Generative AI 101: A Revolutionary Cocktail of Technology and Art How Does Generative AI
KDnuggets
JUNE 20, 2024
Understanding LLMs is pivotal in unlocking the full potential of AI-driven solutions across various domains. As we navigate the process of building AI-driven solutions, it is essential to approach the development and deployment of LLMs with a focus on responsible AI practices.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
In an era where efficiency is king, are you leveraging the right tools to transform your manufacturing processes? A Manufacturing Execution System (MES) is critical for enhancing operational efficiency, reducing waste, and optimizing energy usage—key factors for improving your bottom line and lowering your carbon footprint. Join Nikhil Joshi, a manufacturing technology expert with 18+ years of hands-on experience, in this new webinar as he uncovers the secrets of MES and how to best utilize thes
databricks
JUNE 12, 2024
“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire.
Snowflake
JUNE 27, 2024
Despite the seemingly nonstop conversation surrounding AI, the data suggests that bringing AI into enterprises is still easier said than done. There’s so much potential and plenty of value to be captured — if you have the right models and tools. Implementing advanced AI today requires a solid data foundation as well as a set of solutions, each demanding its own tools and complex infrastructure.
Cloudera
JUNE 7, 2024
A constant flow of breaking news from the data lakehouse space is making notable tech headlines this week. On Tuesday, Databricks announced that it will acquire Tabular, a data management company founded by the creators of Apache Iceberg, Ryan Blue, Daniel Weeks, and Jason Reidfor. The deal was for an unconfirmed sum, but some reports suggest that amount to be between $1B and $2B (and allegedly outbidding Snowflake).
Confluent
JUNE 25, 2024
Leverage Confluent’s data streaming platform to continuously ingest, process, and analyze logs to strengthen your cybersecurity and SIEM.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Data Engineering Weekly
JUNE 16, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Databricks: Open Sourcing Unity Catalog This week brought many exciting developments, with Snowflake and Databricks announcing open-source catalogs.
KDnuggets
JUNE 21, 2024
Want to learn AI from the best of resources? Check out these free AI courses from top universities.
databricks
JUNE 13, 2024
Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.
Snowflake
JUNE 10, 2024
Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. However, as applications and pipelines grow in complexity, understanding what’s happening beneath the surface becomes increasingly crucial. A lack of visibility hinders the development and maintenance of high-quality applications and pipelines, ultimately impacting customer experience.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Cloudera
JUNE 17, 2024
Pride Month is underway and we at Cloudera are looking forward to joining the global celebration of diversity, equity and the ongoing effort for LGBTQ+ ( L esbian, G ay, B isexual, T ransgender, Q ueer/ Q uestioning) rights and recognition. Pride Month serves as a reminder that the fight for equality and equity for members of the LGBTQ+ community is not over.
Knowledge Hut
JUNE 4, 2024
A cyber security plan agrees on the security policies, procedures, and controls required to protect an organization against threats, risks, and vulnerabilities. A cyber security plan can also outline the precise steps to take to respond to a breach. A cyber security plan sets the typical actions for activities such as the encryption of email attachments and restrictions on the use of social media.
Data Engineering Weekly
JUNE 30, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Ozge Demirci, Jonas Hannane & Xinrong Zhu: Who Is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms The economic impact of Gen AI is widely speculated, and we see few signs of impact.
KDnuggets
JUNE 10, 2024
Learn SQL and databases through free courses, tutorials, tools, guides, books, practice exercises, projects, awesome lists, and other resources.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
databricks
JUNE 3, 2024
We are excited to announce that we have agreed to acquire Tabular, Inc, a data management company founded by Ryan Blue, Daniel Weeks.
Snowflake
JUNE 5, 2024
Each year, we are humbled and honored to look back on the contributions from the Snowflake Partner Network (SPN) and recognize their hard work with the Snowflake Partner Awards. Our partners help drive customer success and build an ever-expanding open ecosystem of solutions built on the AI Data Cloud. In the midst of this year’s AI Data Cloud Summit , we announced the 2024 Snowflake Partner Awards, recognizing 36 partners that are winning together with Snowflake and honoring them for their conti
Uber Engineering
JUNE 6, 2024
Dive into how we tackled flakiness among thousands of tests in our CI pipelines with a modularized and configurable approach across our diverse codebase, strengthening reliability of testing infrastructure, improving efficiency in identifying and resolving issues.
Scott Logic
JUNE 2, 2024
It’s June! It’s Pride month! Rainbows! Love is love! We’re your ally! Buy stuff with rainbows on! Let’s come to your Pride parade, but make sure you tone it down a bit! More rainbows! Buy our products! Look, we’ve put a rainbow on it! We love everyone ! We love absolutely everyone, in a very non-specific way! We definitely love sparkly unicorn rainbows!
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Data Engineering Weekly
JUNE 10, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Cube Research: Crystallizing Snowflake Summit 2024 We should officially call the first week of June the data engineering week, as two major data companies are running their developer conference.
KDnuggets
JUNE 25, 2024
This article intends to provide practical advice for becoming a better data scientist by focusing on five different areas of proficiency. Whether you are starting out, or looking to get grounded after years as a practitioner, jump in and elevate your game.
databricks
JUNE 13, 2024
We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance.
Snowflake
JUNE 6, 2024
Snowflake announced the global winners of the sixth annual Data Drivers Awards, the premier data awards that honor Snowflake customers who are leading their organizations and transforming their industries with the AI Data Cloud. This year’s winners of the Data Drivers Awards include data leaders from across global organizations, including Caterpillar, Bentley, Mitsubishi Corporation, Zoom and more.
Speaker: Evelyn Chou
Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.
Expert insights. Personalized for you.
239 
Let's personalize your content