Start Data Engineering
OCTOBER 22, 2022
1. Introduction 2. Data project template 2.1. Prerequisites 2.2. Setup infra 2.3. Tear down infra 3. Set up data infrastructure 3.1. Run data infra on your laptop with containers 3.2. Manage cloud infrastructure with code 4. Set up development workflow 4.1. CI: Automated tests & checks before the merge with GitHub Actions 4.2. CD: Deploy to production servers with GitHub Actions 4.3.
Data Engineering Podcast
OCTOBER 30, 2022
Summary Business intelligence has grown beyond its initial manifestation as dashboards and reports. In its current incarnation it has become a ubiquitous need for analytics and opportunities to answer questions with data. In this episode Amir Orad discusses the Sisense platform and how it facilitates the embedding of analytics and data insights in every aspect of organizational and end-user experiences.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Pragmatic Engineer
OCTOBER 27, 2022
This issue was written in Oct 2022, sent out to all subscribers of The Pragmatic Engineer Newsletter in October 2022. The observations on how Big Tech hiring will slow down have since been validated, with Meta not only laying off in November, but also rescinding offers in January 2023, and Amazon doing the same. If you want to get the pulse of the industry in your inbox, subscribe.
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? If you only came here to know this, my answer is no. Betteridge’s Law strikes again! But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
Jesse Anderson
OCTOBER 19, 2022
I have a calendar reminder that tells me when I founded Big Data Institute. It just told me I founded the company eight years ago. The reminder is called “Independent Anniversary.” It’s the day I split off and executed my vision for an independent, big data consulting company. Independence has all sorts of manifestations. For you, it’s an independent look at technology and vendors from someone who’s worked at a vendor (Cloudera) and worked in distributed systems for even longer.
KDnuggets
OCTOBER 24, 2022
Preprocessing data for machine learning models is a core general skill for any Data Scientist or Machine Learning Engineer. Follow this guide using Pandas and Scikit-learn to improve your techniques and make sure your data leads to the best possible outcome.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
OCTOBER 30, 2022
Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.
The Pragmatic Engineer
OCTOBER 20, 2022
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe. Pollen, the events festival tech startup, went bankrupt in August after raising more than $200M in venture funding. In an exclusive investigative article , I covered the events and details leading up this bankruptcy.
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? If you only came here to know this, my answer is no. Betteridge’s Law strikes again! But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Jesse Anderson
OCTOBER 5, 2022
My guest this week is Brent Dykes , Founder and Chief Data Storyteller at Analytics Hero. Before he founded his own company, he was at Omniture, Adobe, and Domo. Analytics Hero is a consulting business based around data storytelling Data storytelling was a new concept to me. Brent defines it as “as a structured approach for communicating insights to a targeted audience using narrative elements and explanatory visuals.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
KDnuggets
OCTOBER 19, 2022
This post is a summary of 2 distinct frameworks for approaching machine learning tasks, followed by a distilled third. Do they differ considerably (or at all) from each other, or from other such processes available?
Teradata
OCTOBER 12, 2022
ClearScape Analytics provides robust functionality giving people across the organization the ability to efficiently execute their roles in the analytics process on a common platform.
Data Engineering Podcast
OCTOBER 23, 2022
Summary The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases that continue to be adopted due to their proven reliability and robust features. MariaDB is one of those default options that has continued to grow and innovate while offering a familiar and stable experience.
The Pragmatic Engineer
OCTOBER 10, 2022
Part of this article was originally published in The Scoop #27 , for subscribers of The Pragmatic Engineer Newsletter last week. I decided to publish this section for everyone to read after the Business Insider article claiming that 15% of Facebook employees - 12,000 people - may lose their jobs started to spread within the media. The Business Insider article was not specific to software engineers but still spread heavily within tech circles.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Confluent
OCTOBER 4, 2022
Confluent’s new Stream Designer is the industry’s first visual interface for rapidly building, testing, and deploying streaming data pipelines natively on Apache Kafka.
Cloudera
OCTOBER 17, 2022
As we wrap up Hispanic Heritage month this #ClouderaLife Spotlight features Elias Avila, senior staff proactive support engineer for Cloudera. In this spotlight, we talk about his career in technology and his philosophy for getting the most out of work in terms of satisfaction and advancement. We also talk about his upbringing in the primarily Mexican American community of Salinas, California, and the important role Hispanics play in California’s Central Valley. .
KDnuggets
OCTOBER 14, 2022
Leveraging sparse matrix representations for your data when appropriate can spare you memory storage. Have a look at the reasons why, see how to create sparse matrices in with Python, and compare the memory requirements for standard and sparse representations of the same data.
AltexSoft
OCTOBER 13, 2022
Take a look at the featured image above. Beautiful, isn’t it? The interesting thing is, it isn’t a painting drawn by some famous artist, nor is it a photo taken by a satellite. The image you see has been generated with the help of Midjourney — a proprietary artificial intelligence program that creates pictures from textual descriptions. Neural nets can create images, video, and audio content that not every person can.
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
Data Engineering Podcast
OCTOBER 16, 2022
Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development of products and services that support the open lakehouse. In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to.
Teradata
OCTOBER 13, 2022
Implementing a cookieless identity solution will help businesses maintain advertising efforts amid the phaseout of third-party cookies.
Confluent
OCTOBER 6, 2022
Current 2022 is a wrap! Here are some of the top keynote speeches, exciting new data streaming technologies, popular sessions, and where to find videos online.
Cloudera
OCTOBER 11, 2022
A recent VentureBeat article , “4 AI trends: It’s all about scale in 2022 (so far),” highlighted the importance of scalability. I recommend you read the entire piece, but to me the key takeaway – AI at scale isn’t magic, it’s data – is reminiscent of the 1992 presidential election, when political consultant James Carville succinctly summarized the key to winning – “it’s the economy”.
Advertisement
Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.
KDnuggets
OCTOBER 12, 2022
Do you want to learn PyTorch for machine learning and deep learning? Check out this 24 hour long video course with accompanying notes and courseware for free. Did I mention it's free?
Netflix Tech
OCTOBER 12, 2022
A conversation between Travis Brooks, Netflix Product Manager for Experimentation Platform, and George Khachatryan, OfferFit CEO Note: I’ve known George for a little while now, and as we’ve talked a lot about the philosophy of experimentation, he kindly invited me to their office (virtually) for their virtual speaker series. We had a fun conversation with his team, and we realized that some parts of it might make a good blog post as well.
Data Engineering Podcast
OCTOBER 16, 2022
Summary Logistics and supply chains are under increased stress and scrutiny in recent years. In order to stay ahead of customer demands, businesses need to be able to react quickly and intelligently to changes, which requires fast and accurate insights into their operations. Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics.
Teradata
OCTOBER 25, 2022
Developing an IT sustainability strategy can bring major positive change across the enterprise, lowering costs and optimizing resource use.
Speaker: David Bard, Principal at VP Product Coaching
In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.
DataKitchen
OCTOBER 14, 2022
The post Podcast: Scaling DataOps first appeared on DataKitchen.
Cloudera
OCTOBER 6, 2022
Digital transformation. Everyone has their own ideas about what digital transformation means, so I decided to look up a few definitions. . Gartner : “Digital transformation can refer to anything from IT modernization (for example, cloud computing), to digital optimization, to the invention of new digital business models.”. CIO blog post : “Digital transformation is a foundational change in how an organization delivers value to its customers.”.
KDnuggets
OCTOBER 11, 2022
A Data Science Enablement Team consists of people from various departments like marketing, sales, product development, etc. They are responsible for providing the necessary tools and resources to help the data scientists do their job more efficiently.
Confluent
OCTOBER 3, 2022
Announcing Confluent for Startups! Get started with Apache Kafka, leverage our data streaming expertise, and set your business up with the best infrastructure for scale and success.
Speaker: Jon Harmer, Product Manager for Google Cloud
Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.
Expert insights. Personalized for you.
147 
Let's personalize your content