Sizing Lakebase Search: lessons from a real app and 4 million passages
Slide Hub is an internal Databricks app. Field engineers use it to find and reuse...
RADAR & RSS
DE News Radar
Last updated: Oct 15, 12:00 AM UTC
Tracking the latest research and engineering blogs in the Data Engineering space.
Scroll for more →
Unlock powerful insights and forecasts with Predictive AI Functions
Data teams all get asked the same questions: "what will sales be next quarter?,"...
Integrating Netflix’s Foundation Model into Personalization applications
Authors: Divya Gadde, Ko-Jen Hsiao, Dhaval Patel and Moumita Bhattacharya The motivation for building the Netflix Foundation Model for personalization was la...
Power real-time applications and agents with Document Intelligence
At Databricks, our enterprise customers receive millions of documents every single...
DataJunction as Netflix’s answer to the missing piece of the modern data stack
by Olek Gorajek, Samuel Redai, Yian Shang Intro Our Netflix journey from data to dashboards, particularly within the space of the experimentation platform, a...
Measuring the Impact of Personalized Recommendations
By Kevin Zielnicki, Guy Aridor, Aurélien Bibaut, Allen Tran, Winston Chou, and Nathan Kallus Introduction Personalization is everywhere on the internet. It s...
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan Introduction In an earlier post, we introduced the TimeSeries Abstrac...
Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II
By Yogesh Nagarur Introduction In Part 1 of this series, we shared how Netflix built the Ads Event pipeline and approached it as a system design problem: rep...
Netflix Conductor: The Next Chapter
Netflix Conductor : The Next Chapter by Aravindan Ramkumar on behalf of the Conductor team Conductor is the workflow orchestration engine Netflix uses to sti...
Running Apache Spark experiments in my sleep (and on a plane)
By Prashanth Gedde Narayanaswamy As one of Netflix’s largest and critical data pipelines, member-sessionizer consolidates tens of billions of hourly client e...
The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale
By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia Introdu...
Netflix Stratum Media Processing: Automated Container Right-Sizing
By Violetta Pidvolotska and Naveen Mareddy Stratum is Netflix’s internal serverless media processing platform, shaped by over a decade of experience developi...
Reliable GPU compute for large training & inference jobs
Authors: Nick Rutigliano, Argha C, Samartha Chandrashekar How we detect failing GPUs and network fabric, remediate machines automatically, and let long-runni...
Introducing Private Network Gateway
Databricks serverless compute is fully managed, intelligent, and versionless, allowing...
Reducing Qwen training time by 6x on AI Runtime
Training or finetuning a multi-billion parameter foundation model requires significant compute...
The Self-Organizing Data Stack
Today, a data engineer does two jobs: they decide which tables should exist(Data Modeling), and they hand-wire the orchestration engine—schedules, dependenci...
File Data Ingestion with Materialized Lake Views in Microsoft Fabric (Preview)
File data ingestion with materialized lake views (MLVs) in Microsoft Fabric is now available in preview. Instead of building separate ingestion pipelines to ...
A practical guide to real-time analytics in 2026
Data has always been central to business decision-making, and most analytical workloads...
What's New in Lakeflow Designer - October 2026 Roundup
We launched Lakeflow Designer to general availability in June, and since then, over...
Fabric September 2026 Feature Summary
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
Everyone's adopting AI agents. Almost no one can prove the ROI. Here's why
Amber Roberts, Staff Technical Marketing Manager at Databricks, has spent the last...
Databricks is a Visionary the Gartner® Magic Quadrant™ for Analytics and Business Intelligence Platforms
We are incredibly excited to announce that Databricks has made its inaugural appearance...
Fabric App functions to use secure server-side custom logic (Preview)
Fabric App functions is a new capability that allows builders to add secure, server-side logic directly to their Fabric Apps without standing up separate inf...
My workspace governance improvements (Generally Available)
My workspace governance features, which give administrators governance controls over their users' personal My workspaces, are now generally available. As My ...
What is HTAP? A plain-English guide to hybrid transactional/analytical processing
Consider a payment system processing a purchase. Before approving the transaction,...
What’s new for Business Events and real-time events in Microsoft Fabric
Microsoft Fabric helps teams react when those moments happen. Real-Time Hub provides a common place to discover event sources, connect consumers, and move fr...
A new look and new capabilities in the Capacity Metrics App (Preview)
As announced at FabCon 2026 in Barcelona, we're releasing meaningful enhancements to the Capacity Metrics app in preview, designed to accelerate time to insi...
Custom SQL Pools in Fabric Data Warehouse (Generally Available)
Custom SQL Pools in Microsoft Fabric Data Warehouse are now generally available, along with statement type classification. Custom SQL Pools give administrato...
Data Engineering Weekly #291
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the ones happy teams run on....
Planning in Microsoft Fabric: From insight to action
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
Fabric Influencers Spotlight: September 2026
See what the people who implement Fabric are working on, learning, and sharing this month!
Data Engineering Weekly #290
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a...
Bringing governed analytics into the flow of work: Fabric Analytics at FabCon Europe 2026
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
Build, deploy, and govern Microsoft Fabric at scale
If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents” ...
Thinking Like a Data Architect
Last year I wrote about thinking like a data engineer and shared my journey in data engineering. After building large-scale data infrastructure for various l...
Modernize your Azure Data Estate with Microsoft Fabric: A guided path from Azure Data Factory and Azure Synapse
Authors: Faisal Mohamood and Bogdan Crivat If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft ...
Trusted AI starts with Microsoft Fabric, Real-Time Intelligence, and IQ
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
From prompt to production: What's new in Fabric Apps
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
FabCon and SQLCon Barcelona 2026: What’s new in Microsoft OneLake and its rapidly growing ecosystem
If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents” ...
FabCon/SQLCon Barcelona 2026: What’s new in Fabric Data Factory
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
Connecting apps, databases, and AI on one foundation: new innovations at SQLCon and FabCon EU 2026
If you haven't already, check out FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents for a complete look at a...
Data Engineering Weekly #289
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a...
On-premises data gateway September 2026 release
The September 2026 release of the on-premises data gateway (version 3000.334) continues our commitment to delivering secure, reliable, and manageable connect...
Data Engineering Weekly #288
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a...
Upcoming changes to legacy Teams and Outlook activities in Fabric Data Factory pipelines
Beginning October 30, 2026, users will no longer be able to add new legacy Teams or legacy Outlook activities to pipelines. These legacy activities will no l...
Soft delete for On-premises data gateways in Microsoft Fabric (Preview)
Deleting a gateway is currently permanent, so administrators must recreate it if it is removed accidentally. With soft delete, supported gateways are retaine...
Data Engineering Weekly #287
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a...
Data Engineering Weekly #286
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
Data Engineering Weekly #285
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
Building an Operational Ontology: An E-Commerce Walkthrough
Reads travel from a shared model to agents, apps, and people. Writes enter through an audited action gate and write back to the systems of record that own th...
Data Engineering Weekly #284
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
Data Engineering Weekly #283
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
What an Ontology for AI Agents Actually Needs
The word ontology now covers almost every kind of semantic ambition in data and AI. It can mean a metrics layer that consistently calculates “revenue”. It ca...
Data Engineering Weekly #282
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
On Benchmarking
https://www.reddit.com/r/LinusTechTips/comments/13qji93/user_benchmark_meme/ A benchmark is useful only when you can explain what it measured, why it stopped...
Data Engineering Weekly #281
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single p...
Data Engineering Weekly #280
Editor’s Note: leetdata.ai & aidataengineer.io updates Last week we were busy fixing some bugs and upgrading a few features. aidataengineer.io now supports s...
Data Engineering Weekly #279
How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating under a single ...
Data Engineering Weekly #278
Deep Dive: See how Dagster uses AI internally What does it actually look like to use AI as part of your day-to-day engineering workflow? At Dagster, we’ve de...
Beyond Redaction: Anatomy of a Privacy-Safe Data Platform
The hard problem is not making personal data disappear. It is preserving the minimum useful properties for an allowed purpose, limiting who can link the data...