Analytics Vidhya

NOVEMBER 8, 2023

Introduction Imagine yourself in command of a sizable cargo ship sailing through hazardous waters. It is your responsibility to deliver precious cargo to its destination safely. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip. In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya.

Data Pipeline 247

Data Pipeline Big Data Data Data Engineering 247

The Pragmatic Engineer

NOVEMBER 9, 2023

The below topic was sent out to full subscribers of The Pragmatic Engineer , three weeks ago, in The Pulse #66. I have received several messages from people asking if they can pay to “unlock” this information for others, given how vital it is for software engineers. It is vital, and so I’m sharing this with all readers, without a paywall.

Software Engineering 233

Software Engineering Software Engineer Engineering Coding 233

Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doing to shine some light on PostgreSQL to make it easier to understand how to keep it running smoothly.

PostgreSQL 147

PostgreSQL SQL Data Lake Database 147

Zalando Engineering

NOVEMBER 8, 2023

Introduction This blog post describes a recent contribution from Zalando to the Postgres JDBC driver to address a long-standing issue with the driver’s integration with Postgres’ logical replication that resulted in runaway Write-Ahead Log (WAL) growth. We will describe the issue, how it affected us at Zalando, and detail the fix made upstream in the JDBC driver that fixes the issue for Debezium and all other clients of the Postgres JDBC driver.

PostgreSQL 139

PostgreSQL Database Java Bytes 139

Speaker: Dr. Greg Loughnane and Chris Alexiuk

Technology professionals developing generative AI applications are finding that there are big leaps from POCs and MVPs to production-ready applications. They're often developing using prompting, Retrieval Augmented Generation (RAG), and fine-tuning (up to and including Reinforcement Learning with Human Feedback (RLHF)), typically in that order. However, during development – and even more so once deployed to production – best practices for operating and improving generative AI applications are le

Process

Waitingforcode

NOVEMBER 7, 2023

Checkpoints are a well-known fault-tolerance mechanism in stream processing. But what does it have to do with Delta Lake?

Process 130

Process IT 130

KDnuggets

NOVEMBER 7, 2023

To solve the conundrum of ensuring the quality of AI models in production — especially given the emergence of LLMs — we are thrilled to announce the official launch of Giskard, the premier open-source AI quality management system.

Management 120

Management Systems 120

ArcGIS

NOVEMBER 7, 2023

From oriented imagery to engaging thematic map series, there is something for everyone in this release of ArcGIS Pro 3.2.

143

143

Engineering at Meta

NOVEMBER 8, 2023

New optional features in WhatsApp have helped make calling on WhatsApp more secure. “Silence Unknown Callers” is a new setting on WhatsApp that not only quiets annoying calls but also blocks sophisticated cyber attacks. “Protect IP Address in Calls” is a new setting on WhatsApp that helps hide your location from other parties on the call. Privacy and security are at the core of WhatsApp.

Metadata 114

Metadata Data Process Data Process 114

Cloudera

NOVEMBER 7, 2023

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? The vast tapestry of data types spanning structured, semi-structured, and unstructured data means data professionals need to be proficient with various data formats such as ORC, Parquet, Avro, CSV, and Apache Iceberg tables, to cover the ever growing spectrum of datasets – be they images, videos, sensor data, or other type of media content

Hadoop 107

Hadoop Systems Unstructured Data Data Architecture 107

Advertisement

Are you thinking of adding enhanced data matching and relationship detection to your product or service? Do you need to know more about what to look for when assessing your options? The Senzing Entity Resolution Buyer’s Guide gives you step-by-step details about everything you should consider when evaluating entity resolution technologies. You’ll learn about use cases, technology and deployment options, top ten evaluation criteria and more.

Technology

KDnuggets

NOVEMBER 6, 2023

Thinking about getting into the data analytical world but do not know where to start? Have a look at these 5 FREE university courses on data analytics.

Data Analytics 129

Data Analytics Data Data Science 129

ArcGIS

NOVEMBER 10, 2023

This blog summarizes new capabilities on multidimensional raster, STAC, trajectory data, and image processing in ArcGIS Pro 3.

Data Science 125

Data Science Data Process 125

databricks

NOVEMBER 7, 2023

Apache Spark™ 3.5 and Databricks Runtime 14.0 have brought an exciting feature to the table: Python user-defined table functions (UDTFs). In this blog p.

Python 113

Python Data Engineering Data Engineer Engineering 113

Medium Data Engineering

NOVEMBER 10, 2023

Imagine a magical language that lets you talk to databases and make them spill their secrets – that’s SQL! It’s the superhero of data analytics, making sense of heaps of information.

Data Analytics 98

Data Analytics SQL Data Database 98

KDnuggets

NOVEMBER 7, 2023

No, they’re not the same jobs! Learn what responsibilities, skills, and tools used make them different. Then, choose the right career path for you.

Data Science 129

Data Science Data Engineering Data Engineer Data 129

Knowledge Hut

NOVEMBER 9, 2023

In the competitive job market of today, the Certified Scrum Developer training is one thing that can set you apart from the rest. A successful Scrum Developer is committed to delivering continuous improvement. The dedication and coursework that is needed for the achievement of a CSD certification will help you to sharpen your skills leading you to become a much better practitioner of Scrum.

Certification 98

Certification Project Accessible Accessibility 98

DoorDash Engineering

NOVEMBER 7, 2023

At Doordash, we value every chance to boost order conversions in the app. When users fail to complete a purchase after adding items to their carts, we send push notifications such as the one shown in Figure 1 to remind them that their orders are still pending. It has been difficult, however, to determine whether users actually have abandoned their carts or instead are simply browsing for more items or different merchants within the app.

Data 96

Data Designing Process Coding 96

Medium Data Engineering

NOVEMBER 10, 2023

In the dynamic field of data analytics, Microsoft Excel VBA (Visual Basic for Applications) emerges as a potent ally, offering the ability to transform raw data into actionable insights.

Data Analytics 98

Data Analytics Raw Data Data Data Engineering 98

KDnuggets

NOVEMBER 6, 2023

From November 6 (07:00 PST) to November 20 (07:00 PST), enjoy free unlimited access to 365 Data Science's comprehensive curriculum, interactive courses, practical data projects, and earn industry-recognized certificates—all at no charge.

Data Science 116

Data Science Accessible Accessibility Data 116

Knowledge Hut

NOVEMBER 9, 2023

2017 is making a boom of mass layoffs. While taking up a job, we usually consider employment security is a pre-eminent thing. A jolt, mass layoffs in each and every sector are eliciting panic among the employees and youths as well. Every job seeker in this planet requires stability and a risk-free environment. The Recession has been badly affecting the IT sector by unexpectedly slicing the labor-force because of the inclusion of the new advanced technologies and reduced market growth.

Certification 98

Certification IT Project Government 98

databricks

NOVEMBER 8, 2023

This blog is the first of a series of blog posts highlighting industry-leading data providers we collaborate with and Marketplace data providers. Special.

Data 103

Data Management Manufacturing 103

Medium Data Engineering

NOVEMBER 9, 2023

On a crisp evening filled with palpable anticipation, Europe’s grand stage was set as two top-notch footballing teams, Real Madrid and Braga, collided head-on in the Champions League.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

KDnuggets

NOVEMBER 9, 2023

Time is the one thing developers can never get back. The author, discusses the value of low code/no code platforms backed by AI in promoting faster development times and increased business agility.

Coding 96

Coding 96

Knowledge Hut

NOVEMBER 9, 2023

Project Management Professional (PMP) certification, sponsored by the Project Management Institute (PMI), is the most recognized and respected certification credential in the field of project management. To achieve PMP certification, each candidate must satisfy all educational and experiential requirements established by PMI, agree to adhere to a code of professional conduct, and must demonstrate an acceptable and valid level of understanding and knowledge of project management.

Certification 98

Certification Project Management Government 98

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love. A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ.

Kafka 95

Kafka Java Data Ingestion Software Engineering 95

Medium Data Engineering

NOVEMBER 7, 2023

ทำไม NocNoc ถึงใช้ Data Lakehouse? ทำไมเราไม่ใช้ Databricks?

AWS 98

AWS Data Data Engineering Data Engineer 98

KDnuggets

NOVEMBER 10, 2023

The blog covers cheat sheets on SQL, statistics, pandas, data visualization, scikit-learn, Git, and theoretical data science concepts.

Data Science 110

Data Science SQL Data 110

Knowledge Hut

NOVEMBER 9, 2023

While contracting for Agile development, both the parties agree on the scope & vision at the outset but the project is developed iteratively. The traditional contracting practice seeks to agree with “what” & “how” about the project but Agile contract agrees just with “what” leaving the “how” to be managed at the development stage by both the parties.

Project 97

Project Certification Management Programming 97

databricks

NOVEMBER 6, 2023

In Apache Spark™, Python User-Defined Functions (UDFs) are among the most popular features. They empower users to craft custom code tailored to their u.

Python 104

Python Coding Data Engineering Data Engineer 104

Medium Data Engineering

NOVEMBER 9, 2023

In today’s digital age, data is the new gold, and the ability to extract insights from this wealth of information is a skill highly sought after across industries.

Data Analysis 98

Data Analysis Data Data Engineering Data Engineer 98

KDnuggets

NOVEMBER 6, 2023

Cultivate your data science expertise with KDnuggets' Back to Basics pathway, which includes Python, data manipulation, and visualization.

Data Science 109

Data Science Python Data Programming 109

Knowledge Hut

NOVEMBER 9, 2023

Scrum is one of the popular project management methodologies at present. It is very different from the conventional project management methods and offers a fresh perspective to get the work done. In a Scrum team, the employees do not have the usual title roles such as developer, tester etc. They all work together to achieve the common goal in a short span of time and then move on to the next goal.

Certification 97

Certification Project Management Process 97

ArcGIS

NOVEMBER 7, 2023

With ArcGIS Enterprise 11.2, the parcel fabric user type extension is replaced by the Advanced Editing user type extension.

Data Management 115

Data Management Government Data Management 115