Medium Data Engineering
SEPTEMBER 16, 2023
First time I tried to shift from Data science and Machine learning to data engineering all my focus was on three main skills : Continue reading on Medium »
Analytics Vidhya
SEPTEMBER 20, 2023
Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise. This article presents the top 20 data engineering project ideas with their source code.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Pragmatic Engineer
SEPTEMBER 26, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover a fresh industry trends: Cloud Developent Environments — which is analysis full subscribers have received 3 weeks ago.
Medium Data Engineering
SEPTEMBER 5, 2023
I started working with data analytics in 2015. Since then, the industry has doubled many times over.
Speaker: Dr. Greg Loughnane and Chris Alexiuk
Technology professionals developing generative AI applications are finding that there are big leaps from POCs and MVPs to production-ready applications. They're often developing using prompting, Retrieval Augmented Generation (RAG), and fine-tuning (up to and including Reinforcement Learning with Human Feedback (RLHF)), typically in that order. However, during development – and even more so once deployed to production – best practices for operating and improving generative AI applications are le
Marc Lamberti
SEPTEMBER 22, 2023
Wondering how to share data between tasks? What are XCOMs in Apache Airflow? Well, you are at the right place. In this tutorial, you will learn about XComs in Airflow. What they are, how they work, how you can define them, how to get them, and more. If you checked my course “Apache Airflow: The Hands-On Guide”, Aiflow XCom should not sound unfamiliar.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
SEPTEMBER 10, 2023
Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its im
Rock the JVM
SEPTEMBER 17, 2023
By Lucas Nouguier Hey everyone, Daniel here. Lucas’ story is shared by lots of beginner Scala developers, which is why I wanted to post it here on the blog. I’ve watched thousands of developers learn Scala from scratch, and, like Lucas, they love it! If you want to learn Scala well and fast, take a look at my Scala Essentials course at Rock the JVM.
The Pragmatic Engineer
SEPTEMBER 19, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of seven topics from today’s subscriber-only issue on How Big Tech does QA. To get full issues twice a week, subscribe here.
Engineering at Meta
SEPTEMBER 7, 2023
Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We wanted to build a decentralized (or federated) app that would enable people to post content that is viewable by anyone on other social apps, and vice versa.
Advertisement
Are you thinking of adding enhanced data matching and relationship detection to your product or service? Do you need to know more about what to look for when assessing your options? The Senzing Entity Resolution Buyer’s Guide gives you step-by-step details about everything you should consider when evaluating entity resolution technologies. You’ll learn about use cases, technology and deployment options, top ten evaluation criteria and more.
databricks
SEPTEMBER 12, 2023
Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval.
KDnuggets
SEPTEMBER 16, 2023
This tutorial offers a comprehensive hands-on walkthrough of machine learning with Scikit-learn. Readers will learn key concepts and techniques including data preprocessing, model training and evaluation, hyperparameter tuning, and compiling ensemble models for enhanced performance.
Data Engineering Podcast
SEPTEMBER 17, 2023
Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products.
Snowflake
SEPTEMBER 5, 2023
Companies want to train and use large language models (LLMs) with their own proprietary data. Open source generative models such as Meta’s Llama 2 are pivotal in making that possible. The next hurdle is finding a platform to harness the power of LLMs. Snowflake lets you apply near-magical generative AI transformations to your data all in Python, with the protection of its out-of-the-box governance and security features.
The Pragmatic Engineer
SEPTEMBER 28, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in today’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here. Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems.
Engineering at Meta
SEPTEMBER 12, 2023
Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products. In 2021, we discovered a vulnerability in the Meta Quest 2’s Android-based OS that never made it to production but helped us find new ways to improve the security of Meta Quest products. We’re sharing our journey to get arbitrary native code execution in the privileged VR Runtime service on the Meta Quest 2 by exploiting a memory corruption v
Knowledge Hut
SEPTEMBER 6, 2023
As a seasoned software developer with almost a decade of experience in the tech industry, I vividly remember the excitement of taking my first web development course. Back then, I was just starting my journey as a front-end web developer, and that course was a stepping-stone that transformed my career. Today, I am thrilled to share my insights on some of the top software development courses available, hoping to empower aspiring developers like you to find the perfect path to success.
KDnuggets
SEPTEMBER 20, 2023
KDnuggets' latest cheat sheet covers 10 curated hands-on projects to boost data science workflows with ChatGPT across ML, NLP, and full stack dev, including links to full project details.
ArcGIS
SEPTEMBER 19, 2023
In collaboration with NOAA, we used the Presence-Only Prediction (Maxent) tool to predict snow crab habitat under changing climate conditions.
Snowflake
SEPTEMBER 18, 2023
As data continues to become more complex, it is critical to have effective ways to present this information. With the explosion of AI/ML, users want to be able to interact with their data and ML models. However, building such data apps has not been easy. Any data practitioner or product owner will attest to how it takes a lot of steps to build a data app.
The Pragmatic Engineer
SEPTEMBER 22, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in yesterday’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here. Two weeks ago, a JavaScript runtime and toolkit called Bun was released and took the Node.js world by storm. Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of s
databricks
SEPTEMBER 28, 2023
We are excited to announce public preview of GPU and LLM optimization support for Databricks Model Serving! With this launch, you can deploy.
Robinhood
SEPTEMBER 1, 2023
Robinhood Markets. Inc. (Nasdaq:HOOD) today announced that it has successfully purchased all 55,273,469 shares Earlier this year, we shared that our Board of Directors authorized us to pursue purchasing most or all of the 55 million remaining Robinhood shares that Emergent Fidelity Technologies, Ltd. had bought in May 2022. The proposed share purchase underscored the confidence that the Board of Directors and management team have in our business and the success of this effort is another step in
KDnuggets
SEPTEMBER 29, 2023
Cloud notebooks are game-changers for data science, providing free access to computing, pre-built environments, collaboration features, and third-party integrations - everything you need to enhance your workflow.
ArcGIS
SEPTEMBER 22, 2023
This Climate Week renews focus on nature. Learn more about how ArcGIS supports nature-related assessments to run sustainable organizations.
Snowflake
SEPTEMBER 7, 2023
Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Functions or procedures written by users in these languages are executed inside of Snowpark’s secure sandbox environment , which runs on the warehouse.
Precisely
SEPTEMBER 25, 2023
77% of data and analytics professionals say data-driven decision-making is the top goal for their data programs. Data-driven decision-making and initiatives are certainly in demand, but their success hinges on … well, the data that supports them. More specifically, the quality and integrity of that data. It seems obvious enough, but checking that your data is up to the task and taking any necessary steps to improve and maintain its quality can be easier said than done.
databricks
SEPTEMBER 20, 2023
This blog post is in collaboration with Greg Rokita, AVP of Technology at Edmunds. Long envisioned as a key milestone in computing, we've.
Knowledge Hut
SEPTEMBER 25, 2023
In the realm of cybersecurity, the pursuit of expertise is not only a passion but a necessity. As I embarked on the journey to strengthen my skills and broaden my horizons, the Certified Information Systems Security Professional (CISSP) credential emerged as a key qualification for professional recognition. However, achieving the CISSP-Information Systems Security Architecture Professional (ISSAP) specialization demanded more than just dedication—it necessitated guidance through the best CISSP t
KDnuggets
SEPTEMBER 12, 2023
With a shortage of talent and an abundance of opportunity, there’s never been a better time to launch or advance your data science career with the SAS Academy for Data Science. Read on to find out everything you need to become a SAS Certified Data Scientist.
ArcGIS
SEPTEMBER 29, 2023
Esri basemaps are designed to be used at multiple scales, but a static map needs everything in one view. How doe we get around that?
Snowflake
SEPTEMBER 13, 2023
Snowflake and Salesforce have built on our existing partnership to unify the full breadth of customer and business data and generate actionable insights for our customers. We are happy to announce the general availability of Bring Your Own Lake (BYOL) Data Sharing with the Snowflake Data Cloud from Salesforce Data Cloud. Organizations can now leverage Salesforce data directly in Snowflake via zero-ETL data sharing to accelerate decision-making and help streamline business processes.
Towards Data Science
SEPTEMBER 27, 2023
Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. Intro Data lake tables are mostly utilized by data engineering teams using big data compute engines, such as Spark or Flink, as well as by data analysts and scientists creating models and reports with heavy SQL query engines, such as Trino or Redshift.
Expert insights. Personalized for you.
98 
Let's personalize your content