DataPrincipal Weekly, July 9th, 2026
The data stack rebuilt itself around AI agents this week: MCP servers went GA in DataHub Cloud 2.0 and Weaviate 1.38, as ClickHouse scaled LogHouse to 431 PiB.
🤖 MCP servers went GA across DataHub Cloud 2.0 and Weaviate 1.38 as the data stack rebuilt itself around AI agents.
📊 ClickHouse scaled its internal LogHouse platform to 431 PiB and 1.59 quadrillion rows across three cloud providers.
🐘 pgEdge ColdFront tiers older PostgreSQL rows to Iceberg and Parquet on object storage, readable through standard SQL.
✍️ The Week in Data (4 minute read)
SQLite is the most widely deployed database engine on earth. It is in your phone and your browser as you read this. This week, its maintainers shipped patch 3.53.3, and the changelog credits most of the fixes to AIs. Everywhere else in data, the week ran the other way. Vendors shipped MCP servers to general availability so AI agents could walk straight into their catalogs and databases as customers. The whole industry spent the week selling to AI, while one of its most-used tools spent the week being audited by that same AI. That double role, customer and auditor at once, is what you now have to manage, and how you handle it will show up in your data budget and your on-call load for the next two years.
In a single week, the Model Context Protocol became a shipped default. DataHub Cloud 2.0 moved its MCP Server and a Claude Marketplace connector to general availability, with OAuth and SSO, so an agent reaches your catalog without a human ever pasting an API token. Weaviate 1.38 shipped its own MCP Server to GA, with 4 tools sitting at a /v1/mcp endpoint. If you have not met MCP, picture every restaurant in a city agreeing to print its menu in one shared format, so a courier who has never walked in knows exactly what to order. That is handy right up to the moment the courier no longer needs you to explain anything, and neither does the company that sent it.
The engines under that interface started rebuilding to face it. RisingWave 3.0 now calls itself a real-time context layer for interactive AI, with native vector search (FLAT and HNSW indexes) and embeddings that re-embed only the rows that changed. Materialize is attacking the same target from the opposite side, holding computed search entities as SQL views that update themselves as the source data moves, with no hand-maintained pipeline behind them for an agent to read. The two designs are opposites, but both exist because a batch job is too slow for the customer every engine suddenly wants. And that customer has a budget.
The budget is where this turns dangerous for anyone who sells you data infrastructure. A maintained connector now costs roughly $100 a year when agents handle the building and the upkeep. Put that number next to your current vendor invoice. dbt shipped dbt Wizard, a CLI for agentic data work that will migrate a project from MotherDuck to Iceberg or push dbt Core up to Fusion, with validation subagents checking the work as it runs. If a connector really costs $100 a year, a decade of vendor business models falls apart, and some of them are on your renewal list.
This is where SQLite comes back. Cheaper pipeline code, written faster by agents, still carries bugs, and this week produced a concrete one. Lance 8.0.0 shipped a fix for a merge_insert bug that silently dropped rows whenever a leading payload column was entirely null. It threw no error, and it did not crash. The rows were there yesterday and gone today, with nothing in the logs to say which ones. That is the failure you cannot afford to take on trust from a fast, always-on author. So the validation subagents in dbt Wizard are the one thing standing between you and a silent loss you will not notice for months. If agents write the pipelines, something has to check them. Per that SQLite changelog, the thing doing the checking is increasingly another agent. That leaves the agent as both the largest new consumer of your data infrastructure and one of the main things finding its faults. You are now trusting the same technology to build your pipelines and to catch its own mistakes.
ClickHouse drove its internal LogHouse to 1.59 quadrillion rows across 431 PiB, peaking at 190 million rows a second. Downloading that much data over a fast home internet line would take about 120 years. An independent benchmark clocked the minimal-dependency Hardwood Parquet reader at two to three times the reference Java implementation on a 64-core machine. The old hard problems are all still hard. They were not the thing that could reprice your entire stack in a week.
Next week I am watching two things. Whether more catalogs and stores ship MCP to GA instead of preview, and whether a third engine picks up the RisingWave and Materialize agent-context framing, because a third adopter would turn two launches into a market-wide move. And I am watching the bug trackers. SQLite said out loud that AI is finding most of its defects. Every other project running the same tools is seeing the same pattern right now and deciding whether to admit it.
🔬 Deep Dives
A quadrillion rows across three clouds: scaling LogHouse (12 minute read)
ClickHouse scaled its internal LogHouse logging platform from 19 PiB to 431 PiB (1.59 quadrillion rows) across 36 cells on three cloud providers and more than 30 regions, sustaining peak writes of 80 GiB/s uncompressed and 190 million rows per second. The design uses geosharding for writes, async inserts for small-batch ingestion, and a three-level distributed table hierarchy with shard pruning for sub-second cross-region queries.
Benchmarking Hardwood 1.0, a JVM Parquet reader, on a Threadripper 9980X (9 minute read)
An independent benchmark ran Gunnar Morling’s Hardwood Parquet reader against Apache Parquet Java on a 64-core Threadripper 9980X, measuring roughly 2x faster full columnar scans and roughly 3x faster filtered scans on a single core. It also found moderate decoder thread counts of 4 to 8 outperformed the default multi-threaded setting.
Materialize: search is how agents see the world (5 minute read)
Materialize maintains computed search entities as SQL views that update automatically as source data changes. The approach removes custom view-maintenance pipelines and batch reprocessing for agent-facing search.
Apache Hudi: building indexes on a moving target (9 minute read)
A post in Hudi’s Merge-on-Read series describes adding new indexes to a live table without pausing writes, treating INDEXING as a first-class timeline action alongside commits and compactions with a two-stage inflight-versus-active partition lifecycle. It walks through a bootstrap-and-catchup pattern in which an indexer scans historical data while concurrent writers append metadata into the same file groups.
BCBS 239 data lineage and AI governance (8 minute read)
A DataHub engineering post states that the lineage infrastructure required for BCBS 239 banking-regulation compliance is the same infrastructure needed for AI governance, noting only 2 of 31 globally systemically important banks reached full compliance nearly a decade after the 2016 deadline. It specifies four required capabilities: column-level tracking, cross-system end-to-end visibility, versioned audit history, and business-glossary integration.
One Postgres cluster, many apps (8 minute read)
A PlanetScale post walks through hosting multiple isolated applications on a single Postgres cluster using separate logical databases and per-app roles, with REVOKE and GRANT statements enforcing least-privilege isolation. The setup is automated with Pulumi and TypeScript.
🚀 Launches & Tools
RisingWave 3.0 (7 minute read)
RisingWave 3.0 adds native vector search with FLAT and HNSW indexes and incremental embedding that updates only affected rows instead of full re-embeddings. It expands the Apache Iceberg integration into a full table engine with V2 and V3 format support and a DataFusion-powered query engine.
DataHub Cloud 2.0 (6 minute read)
DataHub Cloud 2.0 moves its MCP Server OAuth and Claude Marketplace connector to general availability so employees reach the catalog through SSO without manual API tokens. It adds an AI Tool Audit Dashboard in public beta with CSV export, a Context Documents Home aggregating Notion, Confluence, and GitHub sources, and new ingestion connectors for ThoughtSpot, TimescaleDB, SAP HANA, and Airbyte.
Weaviate 1.38 (6 minute read)
Weaviate 1.38 brings the HFresh disk-based vector index, which is SPFresh-inspired with built-in RQ-1 quantization, and the built-in MCP Server to general availability, exposing four tools at the /v1/mcp endpoint. It also restructures async replication to run cluster-wide from a single scheduler and adds a preview Boost API for query-time rescoring.
Apache Flink 2.3.0 (6 minute read)
Apache Flink 2.3.0 adds SQL operators FROM_CHANGELOG and TO_CHANGELOG for converting between append-only and changelog tables, plus materialized-table DDL parity and refresh control via a new START_MODE clause. The release implements 15 FLIPs and ships an experimental native S3 filesystem built on AWS SDK v2.
ClickHouse 26.6 (8 minute read)
ClickHouse 26.6 adds initial support for streaming continuous queries over MergeTree tables and multi-stage distributed scatter-gather execution across worker nodes. It introduces an EXPLAIN WHATIF command for estimating skip-index selectivity before building indexes, a system.documentation table, and geospatial support via GeoJSON and Mapbox Vector Tiles.
pgEdge ColdFront for PostgreSQL (3 minute read)
pgEdge ColdFront is a transparent data-tiering layer that moves older PostgreSQL rows to Apache Iceberg and Parquet on S3-compatible object storage while keeping them readable and writable through standard SQL, running on unpatched PostgreSQL 16, 17, and 18. It uses DuckDB’s columnar engine for analytical queries on cold data and offers tiered, decoupled, and partition-only modes.
Lance 8.0.0 (3 minute read)
Lance 8.0.0 ships as a major version of the Lance columnar format, adding FM-Index scalar indexes for substring search, branch-aware table version operations, and support for the TOS and GooseFS object stores. It fixes a merge_insert bug that silently dropped matching rows when a leading payload column was entirely null.
dbt Core 2.0.0-alpha.3 (3 minute read)
dbt Core 2.0.0-alpha.3 adds automatic latest-version view pointers for versioned models, JavaScript UDF support for Snowflake and BigQuery, and catalog federation between Snowflake Horizon and Unity Catalog. It also introduces a configurable maximum seed size for hashing and a reservation config for routing BigQuery queries.
📈 Opinions & Advice
Build versus buy is over: a connector now costs $100 a year (5 minute read)
A dltHub post argues that agentic development has changed connector economics so a maintained connector costs roughly $100 per year. It contends teams can own pipeline code with AI agents handling development and upkeep.
dbt Wizard use cases (9 minute read)
A dbt Labs post describes dbt Wizard, a CLI for agentic data work on dbt projects, covering use cases such as migrating from MotherDuck to Iceberg plus BigQuery and upgrading dbt Core to dbt Fusion. It describes validation subagents that check work as it proceeds.
💎 Gems & Repos
Hardwood Hardwood is an open-source, minimal-dependency implementation of Apache Parquet for the JVM with multi-threaded page decoding and a CLI for inspecting schemas, metadata, indexes, and dictionaries. Optional dependencies are limited to compression libraries such as snappy and zstd.
datacontract-cli datacontract-cli is an open-source command-line tool implementing the Open Data Contract Standard, used to lint contracts, connect to data sources, run schema and quality tests, and export to multiple formats in CI/CD or as a Python library. The latest release, 1.0.10, is dated 2026-07-08.
⚡ Quick Links
Apache Airflow 3.3.0rc1: a release candidate with Task SDK 1.3.0rc1 published ahead of the 3.3.0 stable launch.
Milvus 2.6.19: adds configurable concurrency for function-runner text tokenization, enables JSON shredding by default, and adds an optional zero-copy path to reduce search-result serialization overhead.
TimescaleDB 2.28.2: a bugfix release correcting bgw_job_stat_history and chunk_constraint migration issues introduced in 2.28.1 and fixing column ordering on first and last based sparse indexes.
SQLite 3.53.3: a patch release fixing defects present in 3.53.0 through 3.53.2, with release notes attributing most fixed problems to AIs.
That is the week in data. See you next Thursday 👋 You can connect with me on LinkedIn.