In this post, we'll move from theory to practice with a hands-on example, building an end-to-end demo RAG application from the ground up.

Workflow Overview

As illustrated in the figure below, our RAG application consists of five main components:

• User-Facing API: The primary interface for the recommendation service.

• Product Service (Retrieval): For our e-commerce use case, this service manages access to and updates of product information. It represents the "retrieval" phase of RAG.

• LLM Service (Generation): This service builds context from the retrieved product information and connects to an LLM to generate user-facing responses. This constitutes the "generation" phase.

• Product Store: A Hudi table that stores the primary product information, which is built incrementally and serves as the single source of truth.

• Vector Store: Responsible for storing product information embeddings and handling similarity search requests from the Product Service.

For the user-facing API, we define an async endpoint, /recommend, to handle incoming requests. This endpoint processes natural language queries from users and uses product and LLM services to retrieve context and generate response. It also runs a periodic sync task that incrementally reads the latest product information from the Product Store, converts it to vectors, and upserts them into the vector store to keep it up-to-date.

The product service is responsible for embedding the user's query and performing a similarity search on the vector store. When the similarity search returns relevant product information, the Product Service passes this context to the LLM Service.

The LLM service then uses this context to compose a prompt, which it sends to an LLM API. Once a response is generated, the service formats it and delivers the final API response to the user.

Let's now examine each step, highlighting the key code snippets involved.

API Server

We use FastAPI to quickly prototype the API endpoint /recommend as:

@app.post("/recommend")
async def get_recommendations(request: RecommendationRequest) -> dict:
    return recommendation_service.get_recommendations(request.query)

Also define a background sync task to be run periodically while the server app is running, keeping the vector store up-to-date:

async def background_sync(svc: RecommendationService):
    while True:
        try:
            logger.info("Starting background sync...")
            svc.sync_data_source()
            logger.info("Background sync completed")
        except Exception as e:
            logger.error(f"Background sync failed: {e}")

        await asyncio.sleep(config.update_interval)

Hudi Incremental Read

The sync_data_source() function leverages the Hudi-rs library—a native Rust implementation of Hudi—to incrementally read from the products table. The key snippet below shows how we initialize the Hudi table and perform an incremental read:

# Initialize the Hudi table from its base path
hudi_table = HudiTableBuilder.from_base_uri(
  config.hudi_table_path
).build()

# Perform an incremental read starting from the last sync timestamp
incr_batches = hudi_table.read_incremental_records(start_ts, None)

The start_ts parameter is set to the timestamp of the last sync, which is maintained as a property within the product service. This ensures that only new or updated records are fetched during each cycle, making the data sync process highly efficient.

Upsert Embeddings

To prepare the product data for the vector store, we first need to process the raw text. The product names and descriptions must be broken down into smaller, semantically meaningful 'chunks.' This is a crucial step because embeddings are more effective when generated from concise, focused pieces of text.

While chunking strategies can be quite complex and highly dependent on the data, we'll use the semantic-text-splitter library for this demo. It allows us to divide the text into coherent semantic units. In the code, we define a SemanticChunker with configurable settings to handle this process.

from semantic_text_splitter import TextSplitter


class SemanticChunker:
    def __init__(self):
        self.splitter = TextSplitter(
            capacity=config.max_chunk_tokens,
            overlap=config.chunk_overlap_tokens,
        )

Once the text is chunked, the next step is to convert these chunks into numerical representations called embeddings. For this, we use the powerful SentenceTransformers library, which transforms each text chunk into a dense vector (as a NumPy array) that captures its semantic meaning.

class EmbeddingService:
    def __init__(self):
        self.model = SentenceTransformer(config.embedding_model)

    def create_embeddings(
        self, chunks: list[ProductChunk]
    ) -> np.ndarray:
        texts = [chunk.content for chunk in chunks]
        return self.model.encode(
            texts, convert_to_tensor=True
        ).cpu().numpy()

For the vector store, we chosen Qdrant, a high-performance vector database. The embedding arrays are packaged into PointStruct objects—Qdrant's primary data structure—before being saved into a collection.

class VectorStore:
    def __init__(self):
        self.client = QdrantClient(":memory:")
        self.collection_name = "products"
    
    def upsert(
        self, chunks: list[ProductChunk], embeddings: np.ndarray
    ):
        points = [
            PointStruct(
                id=chunk.chunk_id,
                vector=embeddings[i] * chunk.importance_score,
                payload={
                    "product_id": chunk.product_id,
                    "content": chunk.content,
                    "importance_score": chunk.importance_score,
                },
            )
            for i, chunk in enumerate(chunks)
        ]

        self.client.upsert(
            collection_name=self.collection_name,
            points=points,
            wait=False,
        )

This process of combining incremental reads with vector upserts ensures that the vector store remains synchronized with the latest product information, making it ready for accurate similarity searches.

Product Service

When a user query is received, the product service uses the same embedding model to encode the query into a vector. It then performs a similarity search against the vector store to retrieve the top-K most relevant product chunks.

class VectorStore:
    def __init__(self):
        self.client = QdrantClient(":memory:")
        self.collection_name = "products"

    def search(self, query_vector: np.ndarray) -> list[ProductChunk]:
        top_k = int(config.vector_search_top_k)
        results = self.client.search(
            collection_name=self.collection_name,
            query_vector=query_vector,
            limit=top_k,
            score_threshold=config.similarity_threshold,
        )

Because each chunk is associated with a product ID, the service can reconstruct a list of relevant product information from the search results. This list forms the context that is passed to the LLM Service for the final generation step.

LLM Service

The LLM service receives the list of ProductSearchResult objects from the product service to use as context. It then composes a prompt incorporating this information and sends it to the connected AI service provider (in this case, OpenAI).

class LLMService:
    def __init__(self):
        import openai

        self.client = openai.OpenAI(api_key=config.openai_api_key)

    def generate_recommendation(
        self,
        query: str,
        matching_products: list[ProductSearchResult],
    ) -> str:
        """Generate recommendations with chunk context"""
        context = self._prepare_context(matching_products)
        prompt = f"""
User Query: {query}

Based on these context:

{context}

(End of context)

Instructions:
1. Provide product recommendations to the user with brief explanations.
2. Only recommend the products shown in the context.
3. Do not mention the product IDs in the recommendations.
"""
        try:
            response = self.client.chat.completions.create(
                model=config.openai_model,
                messages=[
                    {
                        "role": "system",
                        "content": "You are a helpful e-commerce recommendation assistant.",
                    },
                    {"role": "user", "content": prompt},
                ],
                max_tokens=1000,
                temperature=0.7,
            )
            return response.choices[0].message.content or ""
        except Exception as e:
            logger.error(f"LLM generation failed: {e}")
            return "Unable to generate recommendations at the moment."

Once OpenAI returns a response, the LLM service forwards it to the main recommendation service, which formats and delivers the final API response to the user.

Test Run

To test the end-to-end workflow, we first need a Hudi table populated with sample e-commerce product data. The schema can be minimal, as the key information is contained in the name and description columns.

First, create the Hudi table using Spark SQL:

CREATE TABLE products (
    id BIGINT,
    ts BIGINT,
    name STRING,
    description STRING
) USING HUDI
TBLPROPERTIES (
    primaryKey = 'id',
    preCombineField = 'ts'
);

Next, insert some realistic sample products into the table. Also, set your OpenAI secret key as an environment variable: OPENAI_API_KEY.

Now, you can start the API server by running ./uvicorn.sh. Upon startup, the application will bootstrap the vector store with the initial records from the Hudi table. As you upsert more records into the Hudi table, the periodic sync process will incrementally process these changes and keep the vector store up-to-date.

To test the API, issue a sample e-commerce query to the endpoint, like: "I like drinking coffee and listening to music with a wireless headphone."

➜ curl -X POST "http://localhost:8000/recommend" \
  -H "Content-Type: application/json" \
  -d '{"query": "I like drinking coffee and listening to music with wireless headphone"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2483  100  2401  100    82    409     13  0:00:06  0:00:05  0:00:01   516
{
  "success": true,
  "query": "I like drinking coffee and listening to music with wireless headphone",
  "message": "Based on your love for coffee and music, here are two product recommendations that would enhance your experience:\n\n1. **Wireless Bluetooth Headphones**: Enjoy your coffee while immersing yourself in premium audio quality with these state-of-the-art wireless headphones. They feature active noise cancellation, a long battery life of 30 hours, and comfortable padding, making them perfect for listening to music without distractions. The water-resistant design also means you can wear them while enjoying your coffee outdoors or during workouts.\n\n2. **Programmable Coffee Maker**: Elevate your coffee experience with this professional-grade stainless steel coffee maker. It offers programmable brewing options and a thermal carafe that keeps your coffee hot for hours. With features like a brew strength selector and a self-cleaning function, you can enjoy a perfect cup of coffee tailored to your taste, all while you relax and listen to your favorite tunes. \n\nThese products will complement your routines beautifully!",
  "products": [
    {
      "id": 1001,
      "matched_content": [
        "Experience premium audio quality with these state-of-the-art wireless Bluetooth headphones. Featuring active noise cancellation technology, 30-hour battery life, and premium comfort padding. Perfect for music lovers, commuters, and professionals who demand crystal-clear sound quality. The headphones include a portable charging case, multiple ear tip sizes, and support for high-resolution audio codecs. Compatible with all major devices including smartphones, tablets, and laptops. Water-resistant design makes them ideal for workouts and outdoor activities."
      ]
    },
    {
      "id": 1004,
      "matched_content": [
        "Professional-grade stainless steel coffee maker with programmable brewing options and thermal carafe. Brews up to 12 cups of perfect coffee with precision temperature control and optimal extraction time. Features include auto-start timer, brew strength selector, and self-cleaning function. The thermal carafe keeps coffee hot for hours without a heating plate, preserving flavor and aroma. Built-in water filtration system removes impurities for better taste. Compact design fits most kitchen countertops while the sleek stainless steel finish complements any décor. Includes permanent gold-tone filter and measuring scoop."
      ]
    }
  ]
}

The response payload contains the AI-generated recommendation in the message field, along with the most relevant product information retrieved by the product service. You can find the complete, runnable code for this project in this GitHub repository.

Recap

In this ending part of the blog, we successfully built a RAG-based AI recommender for e-commerce, leveraging FastAPI, Apache Hudi, Qdrant, and OpenAI. While this application serves as a comprehensive local demonstration, transitioning to a production environment requires further consideration.

Real-world data is far more complex and varied, which needs significant tuning in several areas, including chunking strategies, embedding model selection, similarity search parameters, and LLM prompt engineering.

Ultimately, the success of any production-grade RAG application hinges on a solid data foundation. A reliable and efficient data lakehouse is not just a prerequisite but the core component for this architecture to thrive.

Follow me on LinkedIn and X for more updates.

• linkedin.com/in/xushiyan/

• x.com/_xushiyan

In this two-part blog, I'll structure the content as follows:

• Part 1: I'll introduce RAG from a conceptual perspective and explore why solid data architecture is crucial for AI success.

• Part 2: We'll get hands-on with working code and demonstrate the complete end-to-end flow of an AI recommender.

By the end of this two-part blog series, you'll have a solid understanding of RAG basic concepts and know how to build AI apps. You'll get hands-on experience running incremental queries on Hudi tables, building vector search indexes with Qdrant, and setting up a FastAPI app that handles user questions and connects to OpenAI—so you can see a complete working example in action.

RAG is gonna be Ubiquitous

Retrieval-Augmented Generation (RAG) has emerged as the dominant paradigm for building AI applications that need to work with private or domain-specific data. At its core, RAG consists of two essential steps: retrieval and generation. While Large Language Models (LLMs) possess impressive general knowledge, they don't know anything about your company's internal documents, customer data, or proprietary information. RAG solves this by first retrieving relevant pieces of your private data based on a user's query, then feeding this context to the LLM for generation. This approach allows the model to ground its responses in your specific data, transforming a general-purpose AI into a knowledgeable assistant that can reason about your unique business context.

RAG is set to become ubiquitous because it solves a real problem every organization faces: how to make AI work with their specific data. Companies have tons of proprietary information—internal docs, customer records, product manuals—that could make AI incredibly useful if the models could actually access it. RAG provides a practical way to connect your existing data with powerful LLMs, putting AI that understands your business context within reach of most organizations. As more businesses realize they can get contextually aware AI with manageable engineering effort, RAG adoption will continue to explode.

Data Preparation

Before you can retrieve relevant information, your data needs to be transformed into a format that machines can efficiently search and understand. This preparation process involves two critical steps that lay the foundation for effective retrieval.

Chunking

The first step involves splitting your data into manageable chunks. This chunking process is essential because you don't want to retrieve massive documents for the LLM to process—there are limits on the context window that LLMs can effectively work with. Each chunk should contain enough context to be meaningful on its own while remaining small enough to fit comfortably within the LLM's processing capabilities. Choosing a good chunking strategy—whether by paragraphs, sentences, or semantic boundaries—is critical for improving your RAG system's efficiency and accuracy.

Embedding

Once your data is chunked, the next step involves converting these chunks into vectors using an embedding model. Vectors are arrays of numbers that encode human-understandable information into a machine-understandable format, where the dimensions work together to encode semantic meaning in complex patterns. To understand this concept, imagine a 3D vector in a coordinate system representing a point in space, where each element represents distances on the x, y, and z axes. Similarly, text embeddings use hundreds or thousands of dimensions to mathematically compare the similarity of different text chunks, even when they use completely different words to express similar concepts.

Retrieval

The retrieval process takes your prepared data and finds the most relevant pieces to answer a user's query. This involves converting the user's question into the same vector format and then searching your knowledge base for similar content.

With all your data chunks converted to vectors and stored in a vector database, it's time to retrieve the ones most relevant to a user's query. The user's question first gets converted into a vector using the same embedding model that processed your data chunks. Then, similarity functions calculate how close this query vector is to each stored vector in your database. Common similarity functions include cosine similarity, Euclidean distance, and dot product—each representing different mathematical approaches to measuring the "distance" between two vectors in high-dimensional space. Vector databases support these calculations out-of-the-box and can quickly return the top-k most similar vectors along with their associated original data chunks. These retrieved chunks then serve as the contextual foundation for the LLM to generate its response.

Re-ranking

In practice, retrieval typically involves a two-stage process. After the initial vector database search, there's an additional step called re-ranking that significantly improves accuracy. Re-ranking uses a cross-encoder model to perform a more sophisticated comparison between the user's query and the candidates returned by the vector database. While the distance-based similarity search is fast and efficient, it's not always precise in capturing semantic relevance. Cross-encoder models, though more computationally intensive, provide much more accurate similarity assessments. This two-stage approach dramatically boosts overall retrieval accuracy while maintaining good performance.

Generation

Once the retrieval process identifies and refines the most relevant context from your datasets, the generation step becomes relatively straightforward. You combine the retrieved context with the user's original query and send this information to an LLM via API calls, which then returns a response to the user.

In practice, you'll typically compose the context information and user query into a prompt template. This template allows you to specify detailed instructions for the LLM to follow, tailoring the response format, tone, and focus to match your specific business needs. For example, you might instruct the model to cite sources, maintain a professional tone, or format answers in a particular structure that aligns with your application's requirements.

From Data to AI

Now you have gone through the end-to-end flow of a RAG system, but it's still not the full picture of reality. Where do you gather and store the data for chunking? Your data is changing dynamically—how do you deal with updates to ensure your RAG apps are using up-to-date data? And what if the data contains noise that could cause your system to send misleading information to users? There is no shortcut here: a solid data platform serving high-quality data is the pre-requisite to fully utilizing AI's power.

Without a proper data infrastructure, even the most sophisticated LLM models can't reach their full potential—it's the classic "garbage in, garbage out" principle that still holds true in the age of AI. To fully harvest the fruits your RAG apps can offer, you'll likely spend a significant portion of your efforts—sometimes around 70%—on data engineering: ingesting and cleaning data, scaling clusters appropriately, setting up monitoring and alert systems, and managing access. These efforts ensure that you have an efficient, reliable, and business-ready data source for AI applications.

“Garbage In, Garbage Out”

Data quality fundamentally defines the lower bound of your AI inference results. No matter how sophisticated your models are, poor-quality input data will inevitably lead to subpar AI performance. This becomes even more critical in RAG systems, where AI responses are directly grounded in the retrieved information.

The medallion architecture provides a foundation to build quality control mechanisms, systematically refining data through bronze, silver, and gold layers. See this previous blog. Raw data often contains inconsistencies, duplicates, and formatting issues that lead to embeddings calculated from bad data, resulting in poor retrieval accuracy. The gold layer provides business-ready, curated datasets perfect for AI consumption. Even for unstructured binary data like images and videos, you'll want structured tables in the gold layer to track properly formatted metadata for them, enabling LLMs to have more accurate context for generating high-quality responses.

AI applications are also compute-intensive—from semantic chunking to calculating embeddings, from performing similarity searches to querying LLMs for response generation—each step involves heavy compute resources and significant costs. This makes two things essential: first, ensuring the data you put through this process is high-quality and worth the compute investment; second, running the process only on changes that affect the system. Rather than reprocessing entire table snapshots, you should work with incremental changes to update just a subset of all the embeddings.

Why Hudi for Incremental Processing?

Hudi was designed with incremental processing in mind from day one. Hudi's timeline essentially tracks changes made to the table, and through commit files, we can easily fetch the files that contain changes during a specific time window. With the relevant files identified, Hudi uses the record-level meta field _hoodie_commit_time to further filter down the records to be returned. The detailed design discussion can be found in this post.

This timeline and record-level filtering mechanism makes Hudi tables perfectly suitable for the medallion architecture, where changes propagate through multiple layers. The same semantics of incremental processing can be applied throughout your data lakehouse as a standard practice, establishing good data engineering and architecture patterns that make data operations and maintenance easier.

Looking into the future, Hudi has also made roadmap plans to include vector search index support out-of-the-box, which will further simplify the architecture across the entire data storage layer. You'll be able to incrementally calculate embeddings and store them in a Hudi table, then perform similarity searches with Hudi readers. This creates a more unified reader and writer stack for your data architecture.

Recap

In this first part of the blog, we've introduced RAG at a conceptual level by walking through the end-to-end flow. We've also discussed the importance of good data architecture for AI applications and how Hudi's incremental processing capabilities can support such architecture. In part 2, we'll go through an end-to-end implementation of RAG, building the AI recommender example that we introduced at the beginning of this blog.

Follow me on LinkedIn and X for more updates.

• linkedin.com/in/xushiyan/

• x.com/_xushiyan

Think of this series as the follow-up to my earlier "Apache Hudi: From Zero To One" series, where I dove deep into Hudi's design concepts based on version 0.x. The good news is that almost everything from that earlier series still applies to Hudi 1.x, so it's still great for building a solid foundation—in fact, I'd highly recommend reading through that series first, or checking out the consolidated e-book version available at this link. While that earlier series was pretty heavy on concepts and internals, this one's going to be different. I'm aiming for a good mix of theory and practical stuff—complete with sample code and real examples you can actually use. My goal is to help you not just understand what makes Hudi so powerful, but get you up and running with these features quickly through hands-on, practical guidance.

Let's start with one of Hudi's core performance features: the multi-modal index and file pruning. This is what makes your queries fast by helping engines figure out exactly which files to read and which ones to skip entirely.

The Multi-Modal Index in Hudi

Every data lakehouse table—whether it uses Delta, Hudi, or Iceberg—contains a metadata directory that describes the data stored in that table. For Hudi tables, this is the .hoodie/ directory, and you can learn more about Hudi's complete storage layout (including this metadata directory) in this post.

Hudi's multi-modal index lives within the .hoodie/metadata/ directory and has an interesting design: it's actually implemented as its own Hudi Merge-on-Read table, known as the metadata table. This metadata table gets updated synchronously alongside any write operations to your main data table, ensuring everything stays consistent and in sync.

Indexing is what separates a good lakehouse table from a great one—it can make or break your read and write performance. The challenge is that different queries need different types of indexes: range pruning relies on min/max values, point lookups need exact value matching, and vector searches use similarity calculations to find the closest matches. There's no single "one-size-fits-all" index that can handle everything efficiently.

That's why lakehouse tables need versatile indexing capabilities to perform well across all kinds of workloads. Hudi was actually a pioneer in this space, introducing the multi-modal index back in version 0.11 in 2022. The "multi-modal" name reflects how the underlying metadata table is partitioned by different index types, with each index using its own record schema designed for its specific purpose.

Files, Partitions, and Statistics

When you have a collection of columnar files like Parquet stored somewhere, partitioning by some columns at the physical storage level is the best indexing you can get without a lakehouse format—but it's also very coarse-grained and basic. Here's where it gets limiting: if your table is partitioned by column A and someone runs a query filtering on column B (like "find all records where column B > X"), the query engine can't do much optimization. It still has to list all partitions and files, then scan through and filter all the records.

When you create a table in Hudi 1.x, three essential indexes are automatically enabled in the metadata table: files, partition_stats, and column_stats. These indexes provide the core information that query engines need to plan and execute queries efficiently. For more details about how query engines work with Hudi tables, check out this earlier post.

When we say a query engine "supports Hudi," it means the engine has a component that understands Hudi's table layout, including how to read from the metadata table. Here's how the query planning process works:

First, the query engine reads the files index to get a list of partitions to examine. Then it uses the partition_stats index to prune that list by comparing your query predicates against partition-level statistics like min, max, and count values. For example, if your query is looking for records where price >= 300, any partitions with a max price below 300 can be completely skipped.

With the pruned partition list in hand, the engine goes back to the files index to get the actual file lists for each remaining partition. But it's not done yet—it can prune those file lists even further using the column_stats index, which provides the same kind of statistics but at the file level instead of the partition level.

This multi-layered pruning process means the query engine only reads the files it actually needs, significantly reducing the amount of data it has to process.

Running in Spark SQL

Let's see file pruning in action by creating a Hudi table with sample data and running some Spark SQL queries. We'll start by creating a table with both partition_stats and column_stats disabled to establish a baseline.

CREATE TABLE order (
    order_id STRING,
    price DECIMAL(12,2),
    order_status STRING,
    update_ts BIGINT,
    shipping_date DATE,
    shipping_country STRING
) USING HUDI
PARTITIONED BY (shipping_country)
OPTIONS (
    primaryKey = 'order_id',
    preCombineField = 'shipping_date',
    hoodie.metadata.index.column.stats.enable = 'false',
    hoodie.metadata.index.partition.stats.enable = 'false'
);

And insert some sample data:

INSERT INTO order VALUES
('ORD001', 389.99, 'PENDING',    17495166353, DATE('2023-01-01'), 'A'),
('ORD002', 199.99, 'CONFIRMED',  17495167353, DATE('2023-01-01'), 'A'),
('ORD003', 59.50,  'SHIPPED',    17495168353, DATE('2023-01-11'), 'B'),
('ORD004', 99.00,  'PENDING',    17495169353, DATE('2023-02-09'), 'B'),
('ORD005', 19.99,  'PENDING',    17495170353, DATE('2023-06-12'), 'C'),
('ORD006', 5.99,   'SHIPPED',    17495171353, DATE('2023-07-31'), 'C');

The query for our test is as below:

SELECT order_id, price, shipping_country
FROM order
WHERE price > 300;

This query looks for orders with price greater than 300, which only exist in the partition of shipping_country=A. After running the SQL, here's what we see in the Spark UI:

Spark read all 3 partitions and 3 files to find potential matches, but only 1 record from partition A actually satisfied the query condition.

Enable column_stats

Now let's enable column_stats while keeping partition_stats disabled. Note that we can't do it the other way around—partition_stats requires column_stats to be enabled first.

CREATE TABLE order (
    order_id STRING,
    price DECIMAL(12,2),
    order_status STRING,
    update_ts BIGINT,
    shipping_date DATE,
    shipping_country STRING
) USING HUDI
PARTITIONED BY (shipping_country)
OPTIONS (
    primaryKey = 'order_id',
    preCombineField = 'shipping_date',
    hoodie.metadata.index.column.stats.enable = 'true',
    hoodie.metadata.index.partition.stats.enable = 'false'
);

Running the same SQL gives us this in the Spark UI:

Now it shows all 3 partitions but only 1 file was scanned. Without partition_stats, the query engine couldn't prune partitions, but column_stats successfully filtered out the non-matching files. The compute cost of examining those 2 irrelevant partitions and their files could have been avoided with partition_stats enabled.

Enable column_stats and partition_stats

Now let's enable partition_stats as well. Since both indexes are enabled by default in Hudi 1.x, we can simply omit those additional configs from the CREATE statement.

CREATE TABLE order (
    order_id STRING,
    price DECIMAL(12,2),
    order_status STRING,
    update_ts BIGINT,
    shipping_date DATE,
    shipping_country STRING
) USING HUDI
PARTITIONED BY (shipping_country)
OPTIONS (
    primaryKey = 'order_id',
    preCombineField = 'shipping_date'
);

Running the same SQL gives us this in the Spark UI:

Now we see the full pruning effect happened—only 1 relevant partition and 1 relevant file were scanned, thanks to both indexes working together.

Configure columns to be indexed

By default, Hudi indexes the first 32 columns for both partition_stats and column_stats. This limit prevents excessive metadata overhead—each indexed column requires computing min, max, null-count, and value-count statistics for every partition and data file. In most cases, you only need to index a small subset of columns that are frequently used in query predicates. You can specify which columns to be indexed to reduce the maintenance costs:

CREATE TABLE order (
    order_id STRING,
    price DECIMAL(12,2),
    order_status STRING,
    update_ts BIGINT,
    shipping_date DATE,
    shipping_country STRING
) USING HUDI
PARTITIONED BY (shipping_country)
OPTIONS (
    primaryKey = 'order_id',
    preCombineField = 'update_ts',
    'hoodie.metadata.index.column.stats.column.list' = 'price,shipping_date'
);

The config hoodie.metadata.index.column.stats.column.list applies to both partition_stats and column_stats. By indexing just the price and shipping_date columns, queries filtering on price comparisons or shipping date ranges will already see significant performance improvements.

Recap

In this post, we explored Hudi's multi-modal index from a storage layout perspective and demonstrated the file pruning capabilities of the three default indexes in Hudi 1.x: files, partition_stats, and column_stats. Through our SQL examples, you can see how these indexes could dramatically reduce the number of files that need to be scanned, which is crucial for query performance.

In upcoming posts, we'll explore the multi-modal index's additional capabilities and discuss how its design benefits not just performance, but also scalability, extensibility, and maintenance.

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

Throughout the last nine posts, I have explored Hudi concepts pertinent to release 0.14, ideas that are relevant across most of the 0.x versions. For the blog series finale, I aim to cast a glance into the future and delve into the exciting new features in the upcoming 1.0 release. In doing so, this ending post will effectively accomplish the purpose of the series: guiding readers from the foundational beginnings to the groundbreaking future - from zero to one.

The Hudi Stack

Let's take a step back to our initial discussion in the first post and revisit the Hudi stack, a framework that has remained its relevance across both the 0.x and 1.x versions. Illustrated below, this stack functions on top of storage systems, executing read and write operations against open file formats. It is structured into three layers: the transactional database, the programming API, and the user interface.

The transactional database layer, viewed as "the Hudi core", comprises several key components: the table format defines the storage layout, the table services keep the table optimized, the indexes speed up reads and writes, the concurrency control upholds the isolation principle, the lake cache elevates the read efficiency, and the metaserver centralizes the metadata access. These components act together to establish a robust foundation, delivering a database experience for Hudi Lakehouses.

The programming API layer introduces a suite of writer and reader interfaces, standardizing the integration with various execution and query engines. These APIs empower users across the ecosystem to fully harness Hudi's advanced capabilities such as efficient upserts and incremental processing.

The user interface layer provides a higher level of integrated tools that broadly fall into two categories: platform services, which include ingestion utilities, catalog sync tools, and admin CLI; and query engines such as Spark, Flink, Presto, Trino, among others. The diverse array of tools further aids users in adopting Hudi and building comprehensive Lakehouse solutions.

Release 1.0 Highlights

While the Hudi stack remains consistent in version 1.0, the new release features redesigns and updates at the table format level compared to the 0.x versions. These changes, along with other innovative new features, have enhanced overall efficiency and throughput, significantly upgrading Hudi Lakehouse's capabilities.

LSM Tree Timeline

The Hudi Timeline fundamentally consists of a series of immutable transaction logs that record all changes made to a table. In the 0.x versions, the volume of transaction logs increases linearly over time. To optimize storage use, older Timeline instants are archived for optimizing the storage while at the cost of increased compute during access. For Hudi 1.0, a key design goal is to support a near-infinite Timeline that balances optimized storage with efficient access. To achieve this, Log-Structured Merge-Tree (LSM Tree) is adopted to define the Timeline layout in 1.0 tables.

LSM Tree is a multi-layered data structure designed for high write throughput. At the top layer, Hudi Timeline stores the transactions as active instants, having individual Avro files recording the metadata for each transaction's states: requested, inflight, and completed. When exceed a certain threshold, the active instants will be flushed to Parquet files, forming the first storage-optimized layer. The transactions are grouped and sorted chronologically, and the file names contain time range information, allowing efficient retrieval through manifest files. When the number of Parquet files stored at one level exceeds a certain limit, the files will be compacted and pushed down to the next level as larger files with more Timeline instants, further improving storage efficiency. Additionally, these highly compressed Parquet files are optimized for query performance, particularly when time-range filters or specific columns are targeted, ensuring fast data retrieval.

Non-Blocking Concurrency Control

When a streaming writer is present in a concurrent writing scenario, contention could frequently arise due to random updates (e.g., running a separate GDPR deleter job). Using Optimistic Concurrency Control (OCC) in such scenario can lead to repeated retries, thereby wasting compute resources. Hudi has adopted Multi-Version Concurrency Control (MVCC) to prevent blocking and retry behaviors due to contention among a single-writer and table service runners. Hudi also offers early conflict detection for OCC to reduce resource wastage upon retries. To advance further, Hudi 1.0 introduces Non-Blocking Concurrency Control (NBCC) for MOR tables to maximize writer throughput.

NBCC allows multiple writers to persist updating Log Files freely to the same File Slice, and defers the conflict resolution to the compaction stage. Different from the 0.x versions, Log Files in 1.0 also record commit completion time in addition to the starting time. This new piece of information enables proper sorting for the Log Files and helps determine File Slice boundaries. Merging semantics based on a configurable ordering field are applied to the updating records during compaction. To resolve the clock skew issue, TrueTimeGenerator is implemented to ensure monotonically increasing timestamps for all writers' commits.

File Group Reader & Writer

Since the very beginning, Hudi has incorporated the concept of record keys, a design choice that unlocks significant potential for record level operations. Paired with the File Group model, this approach lays a robust foundation for efficient upserts and look-ups. In Hudi 1.0, File Group Reader and Writer APIs are introduced to fully capitalizes on the design advantages offered by the record keys and the File Group model.

On the writer side, Hudi 1.0 employs partial updates, which involve only the updated columns and values, to greatly reduce the Log File sizes. By leveraging Hudi's advanced indexing systems, records targeted for updates are efficiently located, and positional information can be encoded alongside the data log blocks. On the reader side, having the minimized Log File data and the positional information to pinpoint the updating rows and columns, a snapshot query against an un-compacted File Slice can be fully optimized.

Expression Index

In the 0.x versions, Hudi has supported a variety of indexing capabilities, including the Bucket Index and Record-level Index, among others. To enhance flexibility and improve access speeds, Hudi 1.0 introduces the Expression Index, enabling faster retrieval methods and incorporating partitioning schemes into the indexing system.

For example, consider a column "ts" that holds Epoch timestamps. Users might want to filter the data by different time precisions, such as monthly or hourly. By building an Expression Index on the "ts" column using the following SQL, it's possible to achieve effective data-skipping without the need to physically partition the table by hour or add a separate "hour" column.

CREATE INDEX ts_hour ON hudi_table USING column_stats(ts) options(func='hour');

Hudi stores user-created index definitions under a dedicated directory under the metadata path .hoodie/. These definitions inform query engines about the available indexes, facilitating more optimized query planning. The index entries are maintained under separate partitions within the Metadata Table, which serves as the indexing subsystem for the enclosing Hudi table. When writers commit changes, all the available Expression Indexes are updated to reflect these changes, in a manner similar to other enabled indexing features in the Metadata Table. This ensures that indexes remain up-to-date, maintaining high levels of efficiency for both read and write operations.

Recap

In this post, we revisited the Hudi stack diagram, and introduced four noteworthy features set to debut in the upcoming 1.0 release: the LSM Tree Timeline, Non-Blocking Concurrency Control, File Group Reader & Writer, and the Expression Index. In short, Hudi 1.0 makes a defining release, setting a new standard in its development trajectory. As a closing note, here is a key excerpt from the 1.x RFC that succinctly captures the essence of this significant upgrade:

We propose Hudi 1.x as a reimagination of Hudi, as the transactional database for the lake, with polyglot persistence, raising the level of abstraction and platformization even higher for Hudi data lakes.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

Over the course of the last eight posts, I've explored many topics and internal designs of Hudi, including its storage layout, read and write operations, indexing, table services, and concurrency control mechanisms. The timing now feels right to broaden our perspective and start implementing some practical pipelines to streamline data flow into Hudi. In this post, my focus will shift to Hudi Streamer, a comprehensive data ingestion tool designed for deploying production-grade pipelines for Hudi tables. Given its versatility, a topic I will delve into further within this blog, I frequently liken it to a "Swiss Army knife" for importing data into Lakehouses.

Overview

Hudi Streamer is a Spark application1 designed to offer a wide range of customizable interfaces for managing the write process to Hudi tables. It enables users to configure source data, define schemas, schedule table services, keep data catalogs in sync, and so on. The diagram below presents a high-level view of Hudi Streamer's components and their workflow.

The workflow illustrates an ingestion pipeline, typically configured step by step by users. With Hudi Streamer, the setup process will be greatly simplified with its rich set of options. The key to master this tool is to understand what the options are meant for and how to configure them properly. Below, we provide explanations for some of the foundational options.

• The "--table-type" (CoW or MoR), "--table-name" (for identifying the table), and "--target-base-path" (physical location of the table) are three required properties for writing to a Hudi table.

• The "--continuous" flag indicates whether Hudi Streamer should operate in an ongoing manner or execute for once. If the flag is present, the application will keep fetching source data and writing to storage in a loop. Without the flag, Hudi Streamer performs one-time data fetching and writing before terminating. The "continuous" mode is ideal when there is a steady stream of data from upstream sources, whereas the "run-once" mode is tailored for batch or bootstrap use cases.

• The "--min-sync-interval-seconds" works with the "continuous" mode, specifying the shortest allowable interval in seconds between ingestion cycles. For instance, if an ingestion operation requires 40 seconds to complete and the min-sync-interval is configured to 60 seconds, Hudi Streamer will pause for 20 seconds before initiating the subsequent ingestion cycle. This pause ensures the interval adheres to the minimum set duration. Conversely, if the ingestion duration extends to 70 seconds, surpassing the minimum interval, the application immediately proceeds to the next cycle without any delay. This functionality is crucial for ensuring that sufficient data accumulates at the upstream source for processing, thereby preventing the inefficiency of handling numerous small-scale writes.

• The "--op" option represents the type of operation to be executed by Hudi Streamer, which fundamentally serves as another Hudi writer. It supports three write operations: UPSERT (default), INSERT, and BULK_INSERT. For a comprehensive review of write operations, please revisit post 3.

• The "--filter-dupes" flag corresponds to the write client configuration hoodie.combine.before.insert=false|true. This setting allows users to pre-combine records by keys within the incoming batch, effectively reduce the amount of data to process. The flag is applicable when the write operation is set to INSERT or BULK_INSERT, however it should not be present when "--op" is set to UPSERT, since we don't want to lose potential updates before merging them with on-storage versions.

• The "--props" and "--hoodie-conf" options offer flexible ways to take in arbitrary Hudi properties. The former points to a file containing a collection of properties, and the latter accepts a single configuration in the format of "key=value". It is important to note that the properties specified through "--hoodie-conf" take precedence over those extracted via "--props".

In the forthcoming sections, we will delve into the major components depicted in the workflow diagram, offering a detailed exploration of additional options.

Source

Source is an abstraction for providing upstream source data for Hudi Streamer. Its primary responsibility is fetching data from the source system as an input batch for processing and writing. By extending the Source abstract class, Hudi Streamer can be seamlessly integrated with a wide range of data systems. Designed with a platform vision from day one, Hudi currently offers more than a dozen of Source implementations off-the-shelf, as shown in the following picture.

To use a Source for Hudi Streamer, set its fully qualified class name to "--source-class" and configure some Source-specific properties where applicable. For example, a KafkaSource would require setting hoodie.streamer.source.kafka.topic. You may consult the configurations page for more details. Additionally, the "--source-limit" option sets an upper limit on the data amount to read during each fetch, enhancing control over the ingestion process.

Transformer

Upon retrieving incoming data from the Source, it often becomes necessary to perform lightweight transformations, such as adding or dropping specific columns or flattening the schema. The transformer interface facilitates these modifications in a straightforward yet effective manner. It processes a Spark Dataset and outputs the transformed version of the Dataset, enabling seamless data manipulation to meet the requirements of the ingestion pipeline.

The "--transformer-class" option takes in one or many class names of Transformer implementations. When multiple Transformers are given, they are applied sequentially, i.e., the output of one serving as the input for the next. This chained approach provides flexibility and facilitates code maintenance.

Run Table Services

Table services, as introduced in post 5, can be easily managed by Hudi Streamer alongside data writing. When configured as "async", compaction and clustering will be scheduled inline by the Hudi write client internal to Hudi Streamer, and will be executed asynchronously by HoodieAsyncTableService, which uses a thread pool to submit and control table service jobs.

While async table service jobs are running, it might not always be desirable to write new data, for instance, the same cluster that is executing the table services may not have enough resources to perform ingestion. Furthermore, it's sometimes advisable to avoid running too many concurrent compaction or clustering jobs to prevent resource contention. Use "--max-pending-compactions" and "--max-pending-clustering" to limit the outstanding table service operations, and when the limits are reached, no new ingestion job will be scheduled.

When running ingestion jobs and table service jobs concurrently within the same Spark application, it's crucial to appropriately allocate the cluster's resources to ensure optimal performance and efficiency. Hudi Streamer facilitates this by enabling users to input scheduling configurations through specific options. These configurations play a key role in managing how resources are distributed between the ingestion, compaction, and clustering processes.

# for ingestion
--delta-sync-scheduling-weight
--delta-sync-scheduling-minshare

# for compaction
--compact-scheduling-weight
--compact-scheduling-minshare

# for clustering
--cluster-scheduling-weight
--cluster-scheduling-minshare

The properties shown will be used to generate an XML file, which is then referenced by the Spark property spark.scheduler.allocation.file. To activate these settings, users should set spark.scheduler.mode=FAIR for the Spark application.2 For more explanation on the scheduling mechanism, please consult this Spark documentation page.

Sync to Catalogs

Data catalogs play a crucial role in the data ecosystem, and Hudi supports multi-catalog sync out-of-the-box via its SyncTool classes. Hudi Streamer can integrate with SyncTools through the "--sync-tool-classes" option, which takes in a list of SyncTool class names:

# for AWS Glue Data Catalog
org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool

# for Google BigQuery
org.apache.hudi.gcp.bigquery.BigQuerySyncTool

# for Hive Metastore
org.apache.hudi.hive.HiveSyncTool

# for DataHub
org.apache.hudi.sync.datahub.DataHubSyncTool

After each write, if the catalog sync is enabled using the "--enable-sync" flag, each of the configured SyncTools will run synchronously in sequence to upload metadata to the target data catalog. For example, if the write created some new partitions and added a new column to the table, the AwsGlueCatalogSyncTool will update the partition list and the schema stored in the catalog table.

For SyncTools to function properly, users should supply additional SyncTool-specific properties through the "--props" or "--hoodie-conf" options. For detailed configurations, please refer to this section of the documentation page.

Other Notable Features

The Schema Provider, specified through "--schemaprovider-class", serves the schema for reading from the Source and writing to the target table. A notable implementation of this is the SchemaRegistryProvider, which proves particularly useful when integrating with a KafkaSource. This implementation enables Hudi Streamer to access Kafka's schema registry, ensuring that data ingested from Kafka is accurately interpreted and processed.

The "--checkpoint" and "--initial-checkpoint-provider" facilitate pausing and resuming data fetching from the Source, avoiding data loss or duplication. The "--post-write-termination-strategy-class" allows for a graceful shutdown of Hudi Streamer in the "continuous" mode. The "--run-bootstrap" flag instructs the Hudi Streamer to perform a one-time bootstrap operation for a new Hudi table.

Recap

In this post, we've provided an overview of Hudi Streamer's workflow, followed by an in-depth exploration of its diverse options in Hudi Streamer. These discussions aim to highlight Hudi's rich platform capabilities in building an end-to-end ingestion pipeline. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In this post, we will explore the topic of incremental processing in Hudi, addressing the missing piece mentioned in post 2. We'll start with a concise overview of the incremental architecture before examining two related features in Hudi: incremental query and change data capture (CDC).

Overview

Incremental processing, a technique of extracting, loading, and transforming (ELT) subsets of data to keep end results up-to-date, has become a standard in constructing data pipelines for data lakehouses. Unlike traditional methods, which often involve pulling a complete data snapshot for storage overwriting or using costly join operations to identify updates, modern data lakehouses typically utilize a storage format inherently supportive of incremental processing to simplify the architecture. Benefiting from the native support, the medallion architecture has gained popularity and has been adopted in production pipelines by numerous companies. This architecture is characterized by three key layers: the bronze layer, essential for reprocessing needs; the silver layer, ensuring data quality; and the gold layer, delivering business value.

In the next sections, we'll discuss how Hudi achieves incremental processing, which is well-suited to supporting a robust implementation of the medallion architecture.

Incremental Query

Hudi effectively tracks changes in the form of transaction logs by persisting commit metadata within the Timeline, and thereby naturally facilitates incremental processing which, in most cases, relies on timestamp-based checkpointing. Hudi's incremental query feature is enabled through these configurations:

hoodie.datasource.query.type=incremental
hoodie.datasource.read.begin.instanttime=202305150000
hoodie.datasource.read.end.instanttime=202305160000 # optional

These allow for the retrieval of data that has changed within a defined time window. For more usage examples, please check out the documentation page. A few things to note on the behaviors:

• Setting hoodie.datasource.read.begin.instanttime=0 effectively requests all changes made to the table from the very beginning of its history.

• Omitting hoodie.datasource.read.end.instanttime will result in fetching the changes up to the most recent completed commit in the table.

• The data returned by incremental queries contains records that were updated during the specified time window1. These records are matched to their versions corresponding to the latest completed commit in the table. If hoodie.datasource.read.end.instanttime is set, the records will align with the commit denoted by this specified end time.

• When the begin time is set to 0 and the end time is omitted, the incremental query effectively becomes equivalent to a snapshot query, retrieving all the latest records in the table.

Now that we have an understanding of the behavior of incremental queries, we are prepared to delve into the details. The following diagram shows the workflow involved in fetching incremental data from a Hudi MoR table.

Incremental queries follow the read flow as depicted in post 2, implementing two internal APIs: collectFileSplits() and composeRDD(). The implementation is largely divided into these steps:

• collectFileSplits() is responsible for identifying all files relevant to the query. This function derives start and end timestamps based on user input to define a specific time range. This time range is then used to filter commits on the Timeline.

• Hudi's Timeline, comprising a series of transaction logs, inherently represents the changes made over time. With a specified time range, it becomes straightforward to filter down to the relevant files needed for the composeRDD() function to process.

• In a Hudi table, each record includes a metadata field named _hoodie_commit_time, which links the record to a specific commit in the Timeline. During the process of loading target files for records, incremental queries construct a commit time filter to further minimize the amount of data read. This filter is pushed to the level of file reading, allowing composeRDD() to be optimized to load only those records that are intended to be returned.

Change Data Capture

Incremental queries effectively reveal which records have been changed and their final states. However, they don't provide specific details about the nature of these changes. For instance, if record X is identified as having been modified, the incremental query doesn't clarify its column values prior to the update, or whether it was a newly inserted record. Additionally, it doesn't indicate if any records were hard-deleted. To address these limitations, Hudi 0.13.0 introduced Change Data Capture (CDC). This enhanced format of incremental processing provides a more comprehensive view of data modifications, including inserts, updates, and deletes, thereby enabling a clearer understanding of the changes within the dataset.

To enable the CDC functionality, users need to set this table property hoodie.table.cdc.enabled=true2. Writers writing to the table will honor this setting and activate the process of creating CDC log files alongside Base Files. Thanks to Hudi's file grouping mechanism, these CDC log files are included in the same File Groups that hold the changed data. This makes it easy to extend table services like cleaning, and facilitate recovery operations like restore, to manage both CDC log files and data files altogether for more coherent file management.

To pull the CDC data, users just need to set the incremental format to CDC when performing incremental queries. Time-range related behaviors still apply to the CDC query format.

hoodie.datasource.query.type=incremental
hoodie.datasource.query.incremental.format=cdc
hoodie.datasource.read.begin.instanttime=202305150000
hoodie.datasource.read.end.instanttime=202305160000 # optional

The following diagram shows a brief overview of how writer and reader interact with CDC files and data.

On the writer side, Hudi's write handle holds the information3 about the intended operations for the writing records (insert, update, or delete). This information is then encoded into a specific CDC log file format, containing four fields as shown in the diagram. The nullable "before" and "after" fields store the complete record snapshot before and after the change. Users have the flexibility to reduce the volume of logged data by adjusting hoodie.table.cdc.supplemental.logging.mode: use DATA_BEFORE to skip the "after" field, or set OP_KEY_ONLY to store record key instead of "before" and "after" fields.

On the reader side, CDC log files are loaded to construct the results, following a process similar to that of normal incremental queries (whose incremental format is called latest_state). If both "before" and "after" fields are logged, the results will be directly extracted from the CDC log files. In case of a less verbose logging mode is used, the results will be computed on-the-fly by looking up existing records in the table. This is essentially a trade-off between saving storage space and the efficiency of running CDC queries.

Richer Insights

The introduction of CDC capabilities greatly enhances Hudi tables' usage, supporting a wider range of scenarios and offering valuable insights. Take, for example, an account balance subject to frequent debit and credit transactions. Without CDC, periodic snapshot queries or the latest_state incremental queries might only see small or no change in the balance, potentially missing critical fluctuations. Through CDC queries, all changes are revealed, offering a comprehensive view of the account's activities. This level of details would be essential to enable fraud detection algorithms to take actions accordingly.

Recap

In this post, we provided a concise introduction to incremental processing and the medallion architecture, followed by an in-depth exploration of Hudi's approach to supporting incremental queries and Change Data Capture (CDC). Finally, we discussed the significance of CDC in deriving valuable business insights. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In the previous post, we concluded our discussion on table services by exploring the intricacies of the clustering process and space-filling curves. With the knowledge gained in previous posts, we can seamlessly transition to the next topic: concurrency control, focusing specifically on managing concurrency for writers and table services.

A Primer on Concurrency Control

Every commit to a Hudi table is a transaction, whether it stems from adding new data or executing a table service job. Concurrency control is about orchestrating concurrently executed transactions to ensure correctness and consistency while maintaining optimal performance. A wealth of valuable resources is available online, such as this course and this paper. This primer aims to offer just enough context for the subsequent sections that delve into Hudi's implementation of concurrency control.

In databases, ACID are 4 essential properties to maintain the integrity and reliability of transactions. In the chart provided below, I've presented a brief summary of ACID, attempting to leave a clear and easily memorable overview.

Atomicity requires that each transaction be treated as an indivisible unit of work; any changes made by the transaction should be rolled back in the event of a halfway failure. Consistency is about application-specific constraints; for example, a primary key field cannot have duplicates, or the product price column must be non-negative. Isolation ensures concurrent transactions are isolated from each other, resulting in making changes as though they are executed sequentially. Durability mandates the preservation of committed data on storage, ensuring resilience against incidents such as hardware failures.

If the Isolation property is not honored, concurrent transactions will incur read and write anomalies, such as dirty read/write, lost update, etc. While enforcing a strictly serial execution of all transactions can eliminate the anomalies, this severely impacts performance, rendering the system practically unusable. Therefore, we should allow concurrent execution for performance, and coordinate them in equivalence to a serial schedule for correctness. In other words, what we need is a serializable schedule.

MVCC (Multi-Version Concurrency Control) and OCC (Optimistic Concurrency Control) stand out as two widely adopted strategies for enforcing serializable schedules in various database systems. MVCC keeps multiple record versions on storage and associates them with monotonically increasing transaction IDs (e.g., timestamps). OCC "optimistically" allows concurrent transactions to proceed on their own first and resolves any conflicts later. Hudi adopted MVCC for handling single writer with concurrent table services without locking. In later releases, OCC implementation was added to support multi-writer scenarios. In the upcoming sections, we will explore how Hudi employs these strategies in dealing with concurrent writers and table services.

MVCC in Hudi

Timeline and File Slices serve as the foundation to Hudi's MVCC implementation. Timeline uses monotonically increasing commit start time to keep track of transactions to the table. File Slices handle record versioning and correspond to transaction timestamps. One layer above these, Hudi constructs a view object, namely TableFileSystemView, providing APIs to return the table's most recent storage states, such as the latest File Slices under a partition path, and File Groups that undergo clustering. Writers and readers always consult the table file-system view for deciding where to perform the actual IO operations. This design provides read-write isolation out-of-the-box since the new data writing does not interfere with readers accessing past versions.

When a write operation is in progress, a commit action indicating this write will be marked as "requested" or "inflight" on the Timeline. This makes the table file-system view aware of the ongoing action, ensuring that table service planners do not include the File Slices currently being written for subsequent execution. This logic also holds true in the scenario of concurrent table service jobs. Hudi's table services are idempotent operations because the plans containing information about which File Slices to read are persisted. Therefore, retries in the event of failure won't impact the final result.

While a compaction could be on-going, any new write to the MoR table would either route new records to new File Groups or append updates/deletes to Log Files. The Base File that the compaction job is producing will be excluded by the view to prevent misuse. When clustering is pending, users can configure the writer's behavior in case of updating a File Group that undergoes clustering: abort the write, rollback the clustering, defer to later conflict resolution (OCC), or perform dual-write to both the source and target clustering File Groups. Cleaning is always executed in a way that retains the latest File Slices, keeping the deletes clear of new writes.

OCC in Hudi

An OCC protocol typically comprises of three phases: read, validation, and write. In the read phase, concurrent writers perform the necessary IO operations to complete their work in isolation. The validation phase involves collecting the list of changes from each writer and determine if any conflicts exist. Lastly, during the write phase, all changes will be accepted if no conflicts are found, or if conflicts arise, the changes from writer with the later transaction time will be rolled back. This is similar to the GitHub workflow, where contributors can submit pull requests to the upstream repository. The merging will be blocked for pull requests that have conflicts, akin to the validation phase in OCC.

As concurrent updates could lead to write anomalies, Hudi implements OCC at the file-level granularity to handle multi-writer scenarios. To enable this feature, users need to set "hoodie.write.concurrency.mode" to OPTIMISTIC_CONCURRENCY_CONTROL and configure a locker provider accordingly.1 The following diagram demonstrates how OCC is integrated into Hudi's write flow.

Highlight some key steps in the diagram:

• Write client 1 is writing t1.commit and first acquires a lock from the lock provider, which is usually implemented using an external-running server such as Zookeeper, Hive Metastore, or DynamoDB.

• While holding the lock, write client 1 can exclusively check the Timeline to see if any concurrent commits have been completed before its own attempt. In this example, t2.commit by write client 2 is the only candidate Timeline instant to check against and it's still inflight, therefore client 1 can proceed to commit and release the lock.

• Write client 2 is writing t2.commit and acquires the lock after client 1 releases it. In the pre-commit phase, the changed files by client 2, obtained from WriteStatus, conflict with the changed files by client 1, derived from t1.commit. Consequently, client 2 will abort the write.

Aborted writes will be rolled back, implying the deletion of all the written files, both for data and metadata, as if the writes never occurred. While this fulfills Atomicity in the ACID properties, it could also be wasteful, particularly when the conflict chances are high. Hudi offers an early-conflict-detection mode for OCC. In this mode, before the actual files are written, lightweight marker files are created in a temporary folder. These markers serve as a preliminary step for conflict checking. For a detailed explanation of the design and implementation of early conflict detection, please refer to this community talk.

Recap

In this blog post, we went through a brief overview of the concurrency control topic before delving into the implementation details of two strategies, MVCC and OCC, within Hudi. These strategies are adeptly employed to address scenarios involving a single writer with table services and multiple writers. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In the previous post, we covered the concept of table service and discussed compaction, cleaning, and indexing. To conclude this topic, we will now delve into the remaining service - clustering.

Overview

In the context of machine learning, clustering is a technique used to categorize data points into groups, unveiling underlying structures within the dataset. Many clustering algorithms employ specific methods to measure distances between data points, thereby determining the groups they belong to. When talking about clustering within the data storage domain, we can consider records as the data points and physical files as the groups. Therefore, the clustering process can be viewed as putting "proximate" records into the same files. You might naturally pose two follow-up questions: a) How can we determine if records are "proximate"? b) Why is clustering necessary?

To illustrate the concept of "proximity", let’s use the analogy of a 2-dimensional plane with X and Y axes. In this analogy, if a dataset’s schema has two columns, X and Y, and the records will be considered "close" when the value pairs (X, Y) are close to each other on the 2D plane. In case of wide schema with numerous of columns, more dimensions should be added accordingly. While visualizing high-dimensional spaces is challenging for 3-dimensional beings like ourselves, the proximity can still be determined mathematically, allowing computers to process the information.

Clustering, in the context of data storage, stands as a valuable optimization technique to improve the storage layout by preserving data locality for better read efficiency. There are three main motivations to perform clustering:

• Low-latency high-throughput writes often result in too many small files, hurting the query performance. A clustering task that consolidates and rewrites these data files into larger ones can effectively address the issue, especially when executed asynchronously to the writer.

• During the process of rewriting data files, "proximate" records are more likely to be clustered in the same files, thereby facilitating data-skipping techniques. Clustered records tend to show better alignment with the file-level statistics like column min/max values, allowing data files to be skipped more effectively based on given predicates.

• Reading clustered data can also take advantage of cache systems. The principle of spatial locality suggests that, following the access of certain data elements, nearby data elements are likely to be accessed in the near future. As clustered data exhibits good locality, utilizing block cache (e.g., in HDFS) can increase the hit rate, resulting in faster reads.

Clustering Workflow

Similar to other table services mentioned in post 5, clustering can be run in three modes: inline, semi-async, and full-async. Users are encouraged to consult the official documentation and configure these flags to control the running mode as needed.

hoodie.clustering.inline
hoodie.clustering.schedule.inline
hoodie.clustering.async.enabled

Since clustering involves rewriting data, a .replacecommit will be generated upon the completion of the table service job, indicating that the eligible File Groups have been rewritten into new ones. The clustering workflow, consisting of scheduling and execution, is illustrated below.

The clustering workflow is akin to compaction. During the scheduling phase, eligible partitions and File Slices are selected based on ClusteringPlanStrategy. Users have the flexibility to define partition patterns (e.g., using Regex) to target specific partitions. Within these partitions, File Slices meeting certain criteria - such as not undergoing another pending compaction or not qualifying as small files - are added to HoodieClusteringGroups. These entities store information about the input and output for subsequent clustering execution. Typically, HoodieClusteringGroup adheres to size limits, such as the maximum total bytes of File Slices to include for rewriting. The total number of HoodieClusteringGroups is also capped by default, preventing unintentional submission of resource-intensive clustering jobs.

The execution phase involves high-level steps as below:

• Deserialize the clustering plan

• Load the designated input File Slices

• Merge the loaded records

• Bulk Insert the merged records to new File Groups

• Report write statistics through the returned WriteStatus

Users can customize the execution by supplying their own implementation of ClusteringExecutionStrategy. By default, each HoodieClusteringGroup defined in a clustering plan will be submitted as a separate job to perform parallel rewriting of File Slices.

For File Groups undergoing a clustering process, writers will, by default1, abort if updates or deletes on those File Groups are intended. However, failing writes in the case of running table services may not be ideal. Other pluggable strategies exist that allow updates to proceed, followed by resolving conflicts or enforcing dual-writes on both the old and new File Groups.

We have illustrated how the clustering workflow looks like as a Hudi table service. However, one crucial piece of information was missing: where does the record "proximity" we mentioned in the overview come to play during the process? This occurs at the Bulk Insert step, where records are re-partitioned and sorted according to hoodie.layout.optimize.strategy, which I’ll elaborate on in the next section.

Layout Optimization Strategies

Hudi offers three layout optimization strategies, namely Linear, Z-order, and Hilbert. Each of these defines how records should be sorted during Bulk Insert. The default strategy is Linear, which performs lexicographical sorting. The other two, Z-order and Hilbert, are known as space-filling curves that sort and preserve good spatial locality.

The Linear strategy is highly effective for datasets where record "proximity" relies on just one column. For instance, consider a table containing transaction records with a timestamp column. Analysts, for most of the time, run queries to fetch all records between transaction time A and B. Given that the records are considered "close" as long as the transaction timestamps are close, Linear is a perfect strategy due to sorting by the timestamp significantly preserve the locality.

The Linear strategy may not perform well with datasets that require two or more columns to determine record "proximity". Take, for example, a house inventory dataset with columns for latitude and longitude. Lexicographical sorting of latitude followed by longitude would group geographically distant house records together simply based on the proximity of latitude. In such cases, sorting algorithms capable of handling N-dimensional records are needed.

Space-filling curves are specifically designed to map N-dimensional points to one dimension. The term "space-filling" originates from the process where a curve traverses through the space, hitting all the possible points to fill it. Once the curve is straightened, all the multi-dimensional points are mapped to a one-dimensional space and assigned a single-value coordinate. Among various curve-drawing methods, Z-order and Hilbert, as shown below, are two approaches that can effectively preserve spatial locality through this mapping - the majority of nearby points on the curve are also close to each other in the original space.2

When we treat records as multi-dimensional points, drawing a Z-order or Hilbert curve essentially defines the way to sort them. Given that spatial locality is well preserved, actual "nearby" records are more likely to be stored in the same files. This fulfills the proximity condition explained in the overview and enhances read efficiency.

Recap

In this post, we completed the topic of table services by elaborating on clustering. Additionally, we discussed the space-filling curves and how they are used in the clustering process to optimizes the storage for reads. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

The previous four posts in this series have delved into the details about read and write, offering ample context to the new topic addressed in this post, i.e., table services. The following content will be divided into two parts: the first segment will introduce the high-level concepts of table services, while the second part will cover three specific table services - compaction, cleaning, and indexing.

Overview

Table services can be defined as a type of maintenance job that operates on a table without adding new data. When ingesting new records, we often prioritize low latency, which may lead to making trade-offs and leave storage sub-optimized. Running table service jobs results in an improved storage layout, paving the way for more efficient read and write processes in the future.

A table service job comprises two steps: scheduling and execution. The scheduling step aims to generate a plan of execution, while the execution step carries out the plan and makes actual changes to the table. We can categorize the methods of running table services in Hudi into three modes: Inline, Semi-async, and Full-async, as depicted below, to provide flexibility for various real-world scenarios.

In the inline mode, both "schedule" and "execute" occur synchronously after the writer commits, making them "inline". This requires the simplest operational effort, as the two steps are automatically executed in sequence within the existing writer process. However, as an evident trade-off, it may introduce significant latency to the writing process.

The semi-async mode maintains inline scheduling and separates execution from it, i.e., execute asynchronously to the writer process. In this mode, users have the flexibility to deploy the service runner as a separate job or even to a different cluster, which might be necessary due to high computational requirement of the service execution.

The full-async mode is the most flexible mode that decouples table service running from writer processes. This is particularly helpful in managing a large number of tables in a lakehouse project, where a dedicated scheduler can be employed to optimize both scheduling and execution.1

As of release 0.14.0, Hudi offers four table services: compaction, clustering, cleaning, and indexing. In the following sections, we will explore compaction, cleaning, and indexing, reserving clustering in a subsequent post.

Table Services

Compaction

Recall from post 1 on storage layout that a File Slice can contain multiple Log Files and a Base File in MoR tables. As new data comes in, we evolve the File Slice by merging all Log Files against the Base File, creating a new version of the File Slice represented in a new Base File. This process is called compaction and is specific to MoR tables. However, this doesn’t apply to CoW tables, as new Base Files are generated automatically upon writes and have no Log Files to undergo compaction.

There are quite a few configurations to manage when scheduling and executing compaction. The official documentation provides detailed examples that showcase the usage. In this post, our focus is on the generalized internal workflow illustrated in the diagram below.

The scheduling step determines whether compaction is necessary based on the configurable CompactionTriggerStrategy. If deemed true, it generates a compaction plan and saves it to the Timeline as a .compaction.requested action. Users can set the triggering threshold based on factors such as number of commits or elapsed time. If the criteria are met, a compaction plan generator will scan the table based on the CompactionStrategy, which essentially controls which File Slices should be compacted, and produces CompactionOperation for each File Slice to formulate a plan.

The execution step loads all the serialized CompactionOperation from the plan and runs them in parallel. Depending on the presence of the Base File in the target File Slice, either MergeHandle or CreateHandle will be used to write the merged records in a new File Slice. Similar to a write process, a group of WriteStatus will be returned, reporting statistics collected during the execution, and a .commit action will be saved on the Timeline, marking the success of the compaction.

Compaction jobs can be quite resource-intensive due to the high write-amplification when re-writing Base Files. An experimental table service named Log Compaction2 was initially introduced in release 0.13.0 to address the write-amplification issue by only compacting Log Files into larger ones.

Cleaning

For incoming data, Hudi tables continually add File Slices to represent newer versions, taking more disk space. Cleaning is the table service designed to reclaim storage space by deleting old and unwanted versions. For detailed usage information, please refer to the documentation page.

Similar to compaction, Hudi utilizes CleaningTriggerStrategy to determine if cleaning is required at the time of scheduling. Currently, the only supported triggering criterion is the number of commits. After N commits (as configured), a cleaning planner will scan relevant partitions and determine if any File Slice meets the criteria for cleaning, as defined by HoodieCleaningPolicy. Physical paths of either Base Files or Log Files from the eligible File Slices will be used to generate a group of CleanFileInfo. A cleaning plan is then formulated based on that and saved into a .clean.requested action. As of now, three cleaning policies are supported: clean-by-commits, clean-by-file-versions, and clean-by-hours.

The cleaning execution is relatively straightforward: after loading the plan and deserializing the CleanFileInfo, the job performs file-system deletes for the target files in parallel. Statistics are initially collected at the partition level, and then aggregated and saved into a .clean action, indicating the completion.

Indexing

The indexing table service was first added in release 0.11.0 as an experimental feature. Currently, it is designed for building indexes for the metadata table. We will refrain from delving into the indexing process as it requires prior knowledge of the metadata table. Instead, I will provide a brief overview of the design. For further learnings, I recommend consulting the official documentation, this blog, and RFC-45.

In post 4, we mentioned an indexing API named updateLocation() that is required by certain indexes to keep the indexing data in sync with the written data. From a table service perspective, we can view it as indexing running in inline mode, i.e., scheduled inline and executed inline. The current indexing service is considered as being in full-async mode. The metadata table can be seen as another index type that encompasses multiple indexes, also known as a multi-modal index. As the data table size grows, updating the metadata table inline with each write can be time-consuming. Therefore, we need the async table service to maintain high write-throughput while keeping the indexes up-to-date.

Recap

In this post, we introduced the concept of table service, delved into the detailed processes of compaction and cleaning, and briefly touched on indexing. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In the previous post, we walked through Hudi’s write operation flow. Among all the steps involved, indexing is a crucial one that verifies the existence of records in the table and helps achieve efficient update and delete operations. This post will introduce the indexing APIs and explore various types of indexes. Please note that the indexes covered in this post are intended for writers, which differs from reader-side indexing.1

Indexing APIs

Writer indexing abstractions are defined in HoodieIndex. I'll describe some key APIs below to provide a high-level understanding of what indexing entails.

• tagLocation(): when a set of input records is passed to the index component during writing, this API is invoked to tag each record, determining whether it is present in the table, and then associating it with its location information. The resulting set of records is referred to as "tagged records". In the HoodieRecord model, the "currentLocation" field will be populated by this tagging process.

• updateLocation(): after writing to storage, certain indexes require location information to be updated to synchronize with the data table. This process is only executed during the post-IO phase for those applicable index types.

• isGlobal(): Hudi categorizes indexes into global and non-global types. Global indexes identify unique records across all table partitions, hence being "global" in relation to the table. Non-global indexes, on the other hand, validate uniqueness at the partition level. Typically, non-global indexes exhibit better performance due to their smaller scan space. However, they are not suitable for tables with records that can shift between partitions.

• canIndexLogFiles(): due to the implementation specifics, certain indexes are able to index on Log Files for Merge-on-Read tables. This characteristic affects how writers create file-writing handles: when this is true for the configured index, inserts will be routed to Log Files through AppendHandle.

• isImplicitWithStorage(): this is a characteristic that indicates whether the index is implicitly "persisted" along with data files on storage. Some indexes store their indexing data separately.

Index Types

Hudi offers several out-of-the-box index types to suit different traffic patterns and table sizes. Selecting the most appropriate index for each table is a crucial tuning step. This article well-explains the significance of making the right choices. In the following sections, I will illustrate the internal workings of writer indexes to enhance understanding.

Simple Index

The Simple Index is a non-global index and currently serves as the default type. The primary concept behind it involves scanning all Base Files within the relevant partitions to determine whether incoming records match any of the extracted keys.

From the left-join operation, if an input record matches an extracted key, the join result will include the location information, which will then be used to populate the "currentLocation" field of the HoodieRecord. This produces the so-called "tagged records". Those unmatched records will be kept as-is and union-ed with the tagged records for further processing.

The Simple Index has a global version known as the Global Simple Index. Unlike its non-global counterpart, it matches input against Base Files from all partitions rather than just the relevant ones. When a record's partition value is updated, the respective File Group is loaded, which also includes Log Files for MoR tables, for an additional tagging step: it merges the incoming record with its existing old version and tags the merged result to the location in the new partition.

Since the Simple Indexes tend to load all Base Files at either the partition level or the table level, they are well-suited for traffic patterns having random or evenly-distributed data access.

Bloom Index

The Bloom Index follows a similar high-level flow to the Simple Index. However, the distinguishing concept behind the Bloom Index lies in its approach to minimizing the number of keys and files for look-ups while maintaining a low read cost.

The Bloom Index employs a 2-stage filtering to reduce the number of keys and files for look-ups.

• The first stage involves comparing input keys against an interval tree constructed using minimum and maximum record key values stored in Base Files’ footers. Keys falling out of these ranges represent the new inserts, while the remaining keys are considered candidates for the next stage.

• The second stage checks the candidate keys against deserialized Bloom filters, which help determine the definitively absent keys and the potentially present keys. Actual file look-ups are then carried out using the filtered keys and the associated Base Files, which subsequently return the key and location tuples for tagging.

Please note that the filtering process before the look-ups only involves reading the file footers, thereby incurring low read costs.

Just like the Simple Index, the Bloom Index also has a global version known as the Global Bloom Index. It operates similarly to the non-global version, albeit at the table level, and employs the same logic as the Global Simple Index for handling partition-update scenarios.

Bucket Index

The Bucket Index is designed based on hashing, allowing us to consistently map a key to a File Group using a fixed hashing function, eliminating the need for any disk reads and resulting in significant time savings.

The Bucket Index comes in two variations - the Simple Bucket Index and the Consistent Bucket Index. The Simple Bucket Index assigns a fixed number of buckets, each mapping to one File Group, which in turn limits the total number of File Groups in the table. This leads to drawbacks on handling data skewness and scaling out.

On the other hand, the Consistent Bucket Index is designed to overcome the drawbacks by dynamically re-hashing an existing bucket into sub-buckets when the corresponding File Group exceeds a certain size threshold.

HBase Index

The HBase Index is implemented using an externally running HBase server. It stores the mappings between a record key and the relevant File Group information, and it is a global index. This offers efficient look-ups for tagging in most cases and can readily scale out as the table size increases. However, the drawback is the operational overhead involved in managing an additional server.

Record Index

The Record Index is a newly added feature in release 0.14.0 and operates logically similar to the HBase Index: it is also a global index that saves the mappings of record keys and File Groups. The key improvement lies in keeping the indexing data local to the Hudi tables, thus avoiding the cost of operating an extra server. Please refer to this blog for a detailed discussion.

Recap

In this post, we discussed Hudi indexing APIs for writers, delved into the detailed flows of the Simple Index and the Bloom Index, and briefly introduced the Bucket Index, the HBase Index, and the Record Index. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In the previous post, we discussed Hudi query types and their integration with Spark. In this post, we will delve into the other aspect - write flows, with Spark as the example engine. There are numerous configurations and settings you can adjust when it comes to writing data. Therefore, this post does not aim to serve as a complete usage guide. Instead, my primary goal is to present the internal data flows and break down the steps involved. This will provide readers with a deeper understanding of running and fine-tuning Hudi applications. For various practical usage examples, please consult Hudi’s official documentation page.

Overall Write Flow

The picture below illustrates the typical high-level steps involved in a Hudi write operation within the context of an execution engine. I will provide a brief introduction to each step in this section.

Create write client

A Hudi write client serves as the entry point for write operations, and Hudi write support is achieved by creating an engine-compatible write client instance. For instance, Spark utilizes the SparkRDDWriteClient, Flink employs the HoodieFlinkWriteClient, and Kafka Connect generates the HoodieJavaWriteClient. Typically, this step involves reconciling user-provided configurations with the existing Hudi table properties and subsequently passing the final configuration set to the client.

Transform input

Before a write client processes the input data, several transformations occur, including the construction of HoodieRecords and schema reconciliation. Let's delve deeper into the HoodieRecord, as it is a fundamental model in the write paths.

Hudi identifies unique records using the HoodieKey model, which consists of "recordKey" and "partitionPath". These values are populated by implementing the KeyGenerator API. This API offers flexibility in extracting and transforming custom fields into the key based on the input schema. For usage examples, please refer to the documentation page.

Both "currentLocation" and "newLocation" consist of a Hudi Timeline's action timestamp and a FileGroup's ID. Recalling the logical FileGroup and FileSlice concepts from post 1, the timestamp points to a FileSlice within a specific FileGroup. The "location" properties are employed to locate physical files using logical information. If "currentLocation" is not null, it indicates where a record with the same key exists in the table, while "newLocation" specifies where the incoming record should be written.

The "data" field is a generic type that contains the actual bytes for the record, also known as the payload. Typically, this property implements HoodieRecordPayload, which guides engines on how to merge an old record with a new one. Starting from release 0.13.0, a new experimental interface, HoodieRecordMerger, has been introduced to replace HoodieRecordPayload and serve as the unified merging API.

Start commit

At this step, a write client always checks if there are any failed actions remaining on the table's Timeline and performs a rollback accordingly before initiating the write operation by creating a "requested" commit action on the Timeline.

Prepare records

The provided HoodieRecords may optionally undergo deduplication and indexing based on user configurations and the operation type. If deduplication is necessary, records with the same key will be merged into one. If indexing is required, the "currentLocation" will be populated if the record exists.

The topic of indexing logic with various index types is crucial and warrants a dedicated post. For the purpose of understanding write flows, it is important to remember that an index is responsible for locating physical files for the given records.

Partition records

This is an essential pre-write step that determines which record goes into which FileGroup and, ultimately, which physical file. Incoming records will be assigned to update buckets and insert buckets, implying different strategies for subsequent file writing. Each bucket represents one RDD partition for distributed processing, as is the case with Spark.

Write to storage

This is when the actual I/O operations occur. Physical data files are either created or appended to using file writing handles. Before that, marker files may also be created in the .hoodie/.temp/ directory to indicate the type of write operation that will be performed for the corresponding data files. This is valuable for efficient rollback and conflict resolution scenarios.

Update index

After data is written to disk, there may be a need to immediately update the index data to ensure read/write correctness. This applies specifically to index types that are not synchronously updated during writing, such as the HBase index hosted in an HBase server.

Commit changes

In this final step, the write client will undertake multiple tasks to correctly conclude the transactional write. For example, it may run pre-commit validation if configured, check for conflicts with concurrent writers, save commit metadata to the Timeline, reconcile WriteStatus with marker files, and so on.

Write Operations

Upserting data is a common scenario in Lakehouse pipelines. In this section, we will delve into the Upsert flow for CoW table in detail, followed by a brief overview of all other supported write operations.

Upsert

1. Write client starts the commit and creates the "requested" action on Timeline.

2. Input records undergo the preparation step: duplicates are merged, and target file locations are populated by the index. At this point in the process, we have the exact records to be written and know which of those exist in the table, along with their respective locations (FileGroups).

3. Prepared records are categorized into "update" and "insert" buckets. Initially, a WorkloadProfile is constructed to gather information on the number of updates and inserts in the relevant physical partitions. This data is then serialized into an "inflight" action on the Timeline. Subsequently, based on the WorkloadProfile, buckets are generated to hold the records. For updates, each updating FileGroup is assigned as an update bucket. In the case of inserts, the small-file handling logic comes into play: any BaseFile smaller than a specified threshold (determined by hoodie.parquet.small.file.limit) becomes a candidate for accommodating the inserts, with its enclosing FileGroup being designated as an update bucket. If no such BaseFile exists, insert buckets will be allocated, and new FileGroups will be created for them later.

4. The bucketized records are then processed through file-writing handles for actual persistence to storage. In the case of records in the update buckets, "merge" handles are used, resulting in the creation of new FileSlices within the existing FileGroups (achieved by merging with data from the old FileSlices). For records in the insert buckets, "create" handles are utilized, leading to the creation of entirely new FileGroups. This process is done by HoodieExecutors, which employ a producer-consumer pattern for reading and writing records.

5. Once all data has been written, the file-writing handles return collections of WriteStatus that contain metadata about the writes, including the number of errors, the number of inserts performed, the total written size in bytes, and more. This information is sent back to the Spark driver for aggregation. If no errors have occurred, the write client will generate commit metadata and persist it as a completed action on the Timeline.

Upserting to a MoR table follows a very similar flow, with a different set of conditions to determine the types of file-writing handles used for both updates and inserts.

Insert & Bulk Insert

The Insert flow is very similar to Upsert, with the key difference being the absence of an indexing step. This implies that the entire writing process is faster (will be even faster if deduplication is turned off), but it may result in duplicates in the table.

Bulk Insert follows the same semantics as Insert, meaning it can also result in duplicates due to the absence of indexing. However, the distinction lies in the absence of small-file handling for Bulk Insert. The records partitioning strategy is determined by setting BulkInsertSortMode or can be customized by implementing BulkInsertPartitioner. Bulk Insert also enables row-writing mode by default for Spark, bypassing Avro data model conversion at the "transform input" step and working directly with the engine-native data model Row. This mode gives even more efficient writes.

Overall, Bulk Insert is generally more performant than Insert but may require additional configuration tuning to address small-file issues.

Delete

The Delete flow can be viewed as a special case of the Upsert flow. The primary difference is that, during the "transform input" step, input records are transformed into HoodieKeys and passed on to subsequent stages, as these are the minimum required data for identifying the records to be deleted. It's important to note that this process results in a hard delete, meaning that the target records will not exist in the new FileSlices of the corresponding FileGroups.

Delete Partition

Delete Partition follows a completely different flow compared to those introduced above. Instead of input records, it takes a list of physical partition paths, which is configured via hoodie.datasource.write.partitions.to.delete. Because there are no input records, processes such as indexing, partitioning, and writing to storage do not apply. Delete Partition saves all the FileGroup IDs of the target partition paths in a .replacecommit action on the Timeline, ensuring that subsequent writers and readers treat them as deleted.

Insert Overwrite & Insert Overwrite Table

Insert Overwrite completely rewrite partitions with the provided records. This flow can be effectively seen as a combination of Delete Partition and Bulk Insert: it extracts affected partition paths from the input records, marks all existing FileGroups in those partitions as deleted, and creates new FileGroups to store the incoming records.

Insert Overwrite Table is a variation of Insert Overwrite. Instead of extracting affected partition paths from input records, it fetches all partition paths of the table for the purpose of overwriting.

Recap

In this post, we have explored the common high-level steps in Hudi write paths, delved into the CoW Upsert flow with a detailed explanation of record partitioning logic, and introduced all other write operations. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

In the previous post, we discussed the data layout within a Hudi table and introduced the two table types, CoW and MoR, along with their respective trade-offs. Building on that, we will now explore how read operations work in Hudi.

There are several engines, such as Spark, Presto, and Trino, integrated with Hudi that enable you to execute analytical queries. Although the integration APIs may differ, the fundamental process in distributed query engines remains consistent. This process entails interpreting the input SQL, creating a query plan for execution on worker nodes, and collecting the results to return to users.

In this post, I have selected Spark1 as the example engine to illustrate the flow of read operations and provide code snippets to showcase the usage of various Hudi query types. I will begin by introducing Spark queries with a primer, then delve into the Hudi-Spark integration points, and finally, explain the different query types.

Spark Query Primer

Spark SQL is a distributed SQL engine that performs analytical tasks for large-scale data. A typical analytics query begins with user-provided SQL, aiming to retrieve results from a table on storage. Spark SQL takes this input and proceeds through multiple phases, as depicted in the diagram below.

During the analysis phase, the input is parsed, resolved, and converted into a tree structure that works as an abstraction of the SQL statement. The table catalog is consulted for information such as table names and column types.

At the logical optimization step, the tree is evaluated and optimized at the logical layer. Some common optimizations include predicate pushdown, schema pruning, and null propagation. This step generates a Logical Plan that outlines the necessary computations for the query. Since it is a logical representation, the Logical Plan lacks the specifics needed for running on actual nodes.

Physical planning serves as the bridge between the logical layer and the physical layer. A Physical Plan specifies the precise manner in which computations should be executed. For instance, in a Logical Plan, there may be a join node indicating a join operation, whereas in the Physical Plan, the join operation could be specified as a sort-merge join or a broadcast-hash join, depending on size estimates from the relevant tables. The optimal Physical Plan is selected for code generation and actual execution.

The three phases are features provided by Catalyst Optimizer. For further study on this topic, you may explore excellent talks like the ones linked here and here.

During execution, a Spark application operates on the foundational data structure known as RDD (Resilient Distributed Dataset). RDDs are collections of JVM objects that are immutable, partitioned across nodes, and fault-tolerant due to the tracking of data lineage information. As the application runs, the planned computations are performed: RDDs are transformed and acted upon to produce results. This process is also commonly referred to as "materializing" the RDDs.

Data Source API

While Catalyst Optimizer is formulating query plans, connecting to the data source becomes advantageous, enabling optimizations to be pushed down. Spark's Data Source API is designed to provide extensibility for integrating with a wide range of data sources. Some sources are supported out-of-the-box, such as JDBC, Hive tables, and Parquet files. Hudi tables, owing to the specific data layout, represent another type of custom data source.

Spark-Hudi Read Flow

The diagram below illustrates some key interfaces and method calls in the Spark-Hudi read flow.

1. DefaultSource serves as the entry point of the integration, defining the data source’s format as org.apache.hudi or hudi. It provides a BaseRelation, which Hudi uses to implement the data extraction process.

2. buildScan() is a core API to pass filters to data sources for optimizations. Hudi defines collectFileSplits() for gathering relevant files.

3. collectFileSplits() passes all the filters to a FileIndex object that helps identify the necessary files to read.

4. FileIndex locates all the relevant FileSlices for further processing.

5. composeRDD() is invoked after FileSlices are identified.

6. FileSlices are loaded and read out as RDDs. For columnar files like Base Files in Parquet, this read operation minimizes the transferred bytes by reading only the necessary columns.

7. RDDs are returned from the API for further planning and code generation.

Please note that the steps mentioned above provide only a high-level overview of the read flow, omitting details such as support for schema-on-read and advanced indexing techniques like data skipping using a metadata table.

The flow is common to all Hudi query types with Spark2. In the following sections, I will explain how the various query types work. All of them, except for Read-Optimized, are applicable to both CoW and MoR tables.

Snapshot Query

This is the default query type when reading Hudi tables. It aims to retrieve the latest records from the table, essentially capturing a "snapshot" of the table at the time of the query. When performed on MoR tables, the merging of Log Files with the Base File occurs and results in some performance impact.

After launching a spark-sql shell with Hudi dependency3, you may run these SQLs to setup a MoR table with one record inserted and updated.

create table hudi_mor_example (
  id int,
  name string,
  price double,
  ts bigint
) using hudi
tblproperties (
  type = 'mor',
  primaryKey = 'id',
  preCombineField = 'ts'
) location '/tmp/hudi_mor_example';

set hoodie.spark.sql.insert.into.operation=UPSERT;
insert into hudi_mor_example select 1, 'foo', 10, 1000;
insert into hudi_mor_example select 1, 'foo', 20, 2000;
insert into hudi_mor_example select 1, 'foo', 30, 3000;

You can execute a snapshot query by running a SELECT statement as shown below, and it will retrieve the latest value of the record.

spark-sql> select id, name, price, ts from hudi_mor_example;
1       foo     30.0    3000
Time taken: 0.161 seconds, Fetched 1 row(s)

Read-Optimized (RO) Query

RO query type is designed as a trade-off for lower read latency with potentially older results, and therefore, it is exclusively applicable to MoR tables. When conducting such queries, collectFileSplits() will only fetch Base Files for FileSlices.

The provided setup code above automatically generates a catalog table named hudi_mor_example_ro, which specifies a property hoodie.query.as.ro.table=true. This property instructs query engines to always perform RO queries. Running the SELECT statement below returns the original value of the record since the subsequent updates have not yet been applied to the Base File.

spark-sql> select id, name, price, ts from hudi_mor_example_ro;
1       foo     10.0    1000
Time taken: 0.114 seconds, Fetched 1 row(s)

Time Travel Query

By specifying a timestamp, users can request a historical snapshot of a Hudi table at the given time. As previously discussed in post 1, FileSlices are associated with specific commit times and, therefore, support filtering. When performing time travel queries, the FileIndex locates only the FileSlices that correspond to, or are just older than, the specified time if there is no exact match.

spark-sql> select id, name, price, ts from hudi_mor_example timestamp as of '20230905221619987';
1       foo     30.0    3000
Time taken: 0.274 seconds, Fetched 1 row(s)

spark-sql> select id, name, price, ts from hudi_mor_example timestamp as of '20230905221619986';
1       foo     20.0    2000
Time taken: 0.241 seconds, Fetched 1 row(s)

The first SELECT statement executes a time travel query precisely at the deltacommit time of the latest insert, providing the most recent snapshot of the table. The second query sets a timestamp earlier than the latest insert’s, resulting in a snapshot as of the second-to-last insert.

The timestamp in the example follows the Hudi Timeline’s format 'yyyyMMddHHmmssSSS'. You may also set it in the form of 'yyyy-MM-dd HH:mm:ss.SSS' or 'yyyy-MM-dd'.

Incremental Query

Users can set a starting timestamp, with or without an ending timestamp, to retrieve changed records within the specified time window. If no ending time is set, the time window will include the most recent records. Hudi also offers full Change-Data-Capture (CDC) capabilities by enabling additional logs on the writer's side and activating CDC mode for incremental readers. Further details will be covered in a separate post dedicated to incremental processing.

Recap

In this post, we provided an overview of Spark's Catalyst Optimizer, explored how Hudi implements the Spark Data Source API for reading data, and introduced four distinct Hudi query types. In the upcoming post, I will demonstrate the write flow to further enhance our understanding of Hudi. Please feel free to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack

• Post 1: A first glance at Hudi's storage format

• Post 2: Dive into read operation flow and query types

• Post 3: Understand write flows and operations

• Post 4: All about writer indexes

• Post 5: Introduce table services: compaction, cleaning, and indexing

• Post 6: Demystify clustering and space-filling curves

• Post 7: Concurrently run writers and table services

• Post 8: Read and process incrementally

• Post 9: Hudi Streamer - a "Swiss Army knife" for ingestion

• Post 10: Becoming "One" - the upcoming 1.0 highlights

Follow on LinkedIn

After dedicating approximately 4 years to working on Apache Hudi, including 3 years as a committer, I decided to start this blog series with the intention of presenting Hudi's design and usage in an organized and beginner-friendly manner. My aim is to ensure the series is easy to follow for people with some knowledge of distributed data systems. The series will comprise 10 posts, each delving into a key aspect of Hudi. (Why 10? Purely a playful nod to 0 and 1, echoing the series title :) ) The ultimate goal is to help readers understand Hudi with both breadth and depth, enabling them to confidently utilize and contribute to this open-source project.1

Hudi Overview

Hudi is a transactional data lake platform that brings database and data warehouse capabilities to the data lake. The Hudi stack shown below clearly illustrates the major features of the platform.

At its core, Hudi defines a table format that organizes the data and metadata files within storage systems, allowing for features such as ACID transactions, efficient indexing, and incremental processing to be achieved. The remainder of this post will explore the format details, essentially showcasing the structure of a Hudi table on storage and explaining the roles of different files.

Storage Format

The diagram below depicts a typical data layout of a Hudi table under the table's base path in storage.

There are two main types of files:

• Metadata files located in <base_path>/.hoodie/

• Data files stored within partition paths for partitioned tables, or under the base path for non-partitioned tables

Metadata

The <base path>/.hoodie/hoodie.properties file contains essential table configurations, such as table name and version, which both writers and readers of the table will adhere to and utilize.

Alongside hoodie.properties, there are meta-files that record transactional actions to the table, forming the Hudi table's Timeline.

# an example of deltacommit actions on Timeline
20230827233828740.deltacommit.requested
20230827233828740.deltacommit.inflight
20230827233828740.deltacommit

These meta-files follow the naming pattern below:

<action timestamp>.<action type>[.<action state>]

The <action timestamp>

• marks when an action was first scheduled to run.

• uniquely identifies an action on Timeline.

• is monotonically increasing across different actions on a Timeline.

The <action type> shows what kind of changes were made by the action. There are write action types, such as commit and deltacommit, which indicate new write operations (insert, update, or delete) that occurred on the table. Additionally, there are table service actions, such as compaction and clean, as well as recovery actions like savepoint and restore. We will discuss different action types in more detail in future posts.

The <action state> can be requested, inflight, or completed (without a suffix). As the names suggest, requested indicates being scheduled to run, inflight means execution in progress, and completed means that the action is done.

The meta-files for these actions, in JSON or AVRO format, contain information about the changes that should be applied to the table or that have been applied. Keeping these transaction logs makes it possible to recreate the table's state, achieve snapshot isolation, and reconcile writer conflicts through concurrency control mechanisms.

There are other metadata files and directories stored under .hoodie/. To give some examples, the metadata/ contains further metadata related to actions on the Timeline and serves as an index for readers and writers. The .heartbeat/ directory stores files for heartbeat management, while the.aux/ is reserved for various auxiliary purposes.

Data

Hudi categorizes physical data files into Base File and Log File:

• Base File contains the main stored records in a Hudi table and is optimized for read.

• Log File contains the records' changes on top of its associated Base File and is optimized for write.

Within a partition path of a Hudi table (as shown in the previous layout diagram), a single Base File and its associated Log Files (which can be none or many) are grouped together as a File Slice. Multiple File Slices constitute a File Group. Both the File Group and the File Slice are logical concepts designed to enclose physical files, simplifying access and manipulation for both readers and writers. By defining these models, Hudi can

• fulfill both read and write efficiency requirements. Typically, Base File is configured as a columnar file format (e.g., Apache Parquet) and Log File is set to a row-based file format (e.g., Apache Avro).

• achieve versioning across commit actions. Each File Slice is tied to a specific timestamp of an action on the Timeline, and the File Slices within a File Group essentially track how the contained records evolved over time.

Table Types

Hudi defines two table types - Copy-on-Write (CoW) and Merge-on-Read (MoR). The layout differences are as follows: CoW has no Log File compared to MoR, and write operations result in .commit actions instead of .deltacommit. Throughout our discussion, we have been using MoR as the example. Understanding CoW becomes straightforward once you grasp MoR - you can treat CoW as a special case of MoR where records in a Base File and the changes are implicitly merged into a new Base File during each write operation.

When choosing the table type for a Hudi table, it is important to take into account the read and write patterns, as there are some implications:

• CoW has high write amplification due to rewriting records in the new File Slices for every write, while read operations will always be optimized. This is well-suited for read-heavy analytical workloads or small tables.

• MoR has low write amplification because changes are buffered in Log Files and batch-processed to merge and create new File Slices. However, read latency is affected since inflight merging of Log Files with Base File is required for reading the latest records.

Users may also opt to only read Base Files of an MoR table to obtain efficiency while sacrificing result freshness. We will discuss more about Hudi's different read modes in forthcoming posts. As the Hudi project evolves, the merging costs associated with reading from MoR tables has been optimized over past releases. It is foreseeable that MoR will become the preferred table type for most workload scenarios.

Recap

In this initial post of the zero-to-one series, we have explored the fundamental concepts of Hudi's storage format to elucidate how metadata and the data are structured within Hudi tables. We also briefly explained the different table types and their trade-offs. As shown in the overview diagram, Hudi serves as a comprehensive lakehouse platform offering features across various dimensions. In the upcoming nine posts, I will progressively cover other significant facets of the platform. Please don't hesitate to share your feedback and suggest content in the comments section.

Apache Hudi has a thriving community - come and engage with us via Slack, GitHub, LinkedIn, X (Twitter), and YouTube!

Engage on Slack