Elasticsearch：如何在 ES|QL 中使用 FORK 及 FUSE 命令来实现混合搜索 - 9.1+

Elastic 中国社区官方博客

1645人浏览 · 2025-11-17 11:22:57

Elastic 中国社区官方博客 · 2025-11-17 11:22:57 发布

我们知道在 DSL 中我们可以轻松地使用 RRF 及线性来针对多路搜索进行混合搜索。我们可以使用如下的命令来进行 RRF 混合搜索：

GET /imdb_movies/_search?pretty
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "term": {
                "overview": "clueless slackers"
              }
            }
          }
        },
        {
          "knn": {
            "field": "overview_dense",
            "query_vector_builder": {
              "text_embedding": {
                "model_id": ".multilingual-e5-small_linux-x86_64",
                "model_text": "clueless slackers"
              }
            },
            "k": 5,
            "num_candidates": 5
          }
        },
        {
          "standard": {
            "query": {
              "text_expansion": {
                "overview_sparse": {
                  "model_id": ".elser_model_2_linux-x86_64",
                  "model_text": "clueless slackers"
                }
              }
            }
          }
        }
      ],
      "rank_window_size": 5,
      "rank_constant": 1
    }
  },
  "size": 3,
  "fields": [
    "names",
    "overview"
  ],
  "_source": false
}

请详细阅读之前的文章 “Elasticsearch：介绍 retrievers - 搜索一切事物”。在上面，我们使用了 RRF。有关 RRF 的介绍，请阅读文章 “Elasticsearch：倒数排序融合 - Reciprocal rank fusion (RRF)”。

我们也可以使用线性组合来平衡混合搜索。请详细阅读文章 “平衡尺度：利用权重使倒数排序融合 (RRF) 更加智能”。

{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "retriever": {
            "standard": {
              "query": {
                "match": {
                  "cuisine_type": "Italian"
                }
              }
            }
          },
          "weight": 0.4
        },
        {
          "retriever": {
            "standard": {
              "query": {
                "match": {
                  "menu_items": "cacio e pepe"
                }
              }
            }
          },
          "weight": 0.6
        }
      ]
    }
  }
}

随着 ES|QL 的推出，越来越多的查询会使用 ES|QL 来做查询，那么我们该如何实现混合搜索呢？

Elastic 在 9.1 中开始推出 FORK 及 FUSE 命令来帮助我们来实现这种混合搜索。

准备数据

我们首先创建如下的数据集：

PUT /people
{
  "mappings": {
    "properties": {
      "id": {
        "type": "integer"
      },
      "name": {
        "type": "text"
      },
      "description": {
        "type": "text",
        "copy_to": "des_semantic"
      },
      "des_semantic": {
        "type": "semantic_text",
        "inference_id": ".multilingual-e5-small-elasticsearch"
      },
      "sex": {
        "type": "keyword"
      },
      "age": {
        "type": "integer"
      },
      "address": {
        "type": "text"
      },
      "location": {
        "type": "geo_point"
      }
    }
  }
}

POST /_bulk
{ "index" : { "_index" : "people", "_id" : "1" } }
{ "id": 1, "name" : "John Doe", "description" : "A software developer", "sex" : "Male", "age" : 30, "address" : "123 Elm Street, Springfield", "location": {"lat": 37.7749, "lon": -122.4194} }
{ "index" : { "_index" : "people", "_id" : "2" } }
{ "id": 2, "name" : "Jane Smith", "description" : "A project manager", "sex" : "Female", "age" : 28, "address" : "456 Maple Avenue, Anytown", "location": {"lat": 40.7128, "lon": -74.0060} }
{ "index" : { "_index" : "people", "_id" : "3" } }
{ "id": 3, "name" : "Alice Johnson", "description" : "A graphic designer", "sex" : "Female", "age" : 26, "address" : "789 Oak Lane, Metropolis", "location": {"lat": 34.0522, "lon": -118.2437} }
{ "index" : { "_index" : "people", "_id" : "4" } }
{ "id": 4, "name" : "Bob Brown", "description" : "A marketing specialist", "sex" : "Male", "age" : 32, "address" : "321 Pine Street, Gotham", "location": {"lat": 41.8781, "lon": -87.6298} }
{ "index" : { "_index" : "people", "_id" : "5" } }
{ "id": 5, "name" : "Charlie Davis", "description" : "An IT analyst", "sex" : "Male", "age" : 29, "address" : "654 Cedar Blvd, Star City", "location": {"lat": 29.7604, "lon": -95.3698} }
{ "index" : { "_index" : "people", "_id" : "6" } }
{ "id": 6, "name" : "Diana Prince", "description" : "A diplomat", "sex" : "Female", "age" : 35, "address" : "987 Birch Road, Themyscira", "location": {"lat": 39.9526, "lon": -75.1652} }
{ "index" : { "_index" : "people", "_id" : "7" } }
{ "id": 7, "name" : "Evan Wright", "description" : "A journalist", "sex" : "Male", "age" : 27, "address" : "213 Willow Lane, Central City", "location": {"lat": 33.4484, "lon": -112.0740} }
{ "index" : { "_index" : "people", "_id" : "8" } }
{ "id": 8, "name" : "Fiona Gallagher", "description" : "A nurse", "sex" : "Female", "age" : 31, "address" : "546 Spruce Street, South Side", "location": {"lat": 32.7157, "lon": -117.1611} }
{ "index" : { "_index" : "people", "_id" : "9" } }
{ "id": 9, "name" : "George King", "description" : "A teacher", "sex" : "Male", "age" : 34, "address" : "879 Elm St, Smallville", "location": {"lat": 39.7392, "lon": -104.9903} }
{ "index" : { "_index" : "people", "_id" : "10" } }
{ "id": 10, "name" : "Helen Parr", "description" : "A full-time superhero", "sex" : "Female", "age": 37, "address" : "123 Metro Avenue, Metroville", "location": {"lat": 47.6062, "lon": -122.3321} }

如上所示，我们的 people 索引含有一个 description 字段。它是 text 类型的。而另外一个字段 des_semantic 其实是一个密集向量。它的类型是 semantic_text。其向量值由 E3 多语言模型产生。

我们可以针对 des_semantic 字段做如下的向量查询：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _score 
     | WHERE MATCH(des_semantic, "coder")
     | SORT _score DESC | LIMIT 2
     | KEEP description
  """
}

我们做如上的查询：

FORK

FORK 处理命令会创建多个执行分支，对相同的输入数据进行操作，并将结果合并为一个输出表。

FORK ( <processing_commands> ) ( <processing_commands> ) ... ( <processing_commands> )

说明

FORK 处理命令会创建多个执行分支，对相同的输入数据进行操作，并将结果合并为一个输出表。会添加一个区分列（_fork）来标识每一行来自哪个分支。

结合 FUSE 命令，FORK 可以实现混合搜索，用来合并并给多个查询的结果打分。想要了解更多关于使用 ES|QL 做搜索的内容，请参考 ES|QL for search。

例子：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _score 
     | FORK (WHERE MATCH(des_semantic, "coder"))
            (WHERE MATCH(description, "journalist"))
     | SORT _score DESC
     | KEEP description
     | LIMIT 2
  """
}

很显然，我们得到了搜索既是 jounalist 也是是 coder 的搜索结果。虽然这种结果是很好，返回了多路查询的结果，但是毕竟 WHERE MATCH(description, "journalist") 使用的是 BM25 的搜索方法，而 HERE MATCH(des_semantic, "coder") 使用的是向量搜索的方法。两种打分的方式是不一样的。相似性的分数在 0-1 之间。

我们再次修改我们的查询方法：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _score 
     | FORK (WHERE MATCH(des_semantic, "coder"))
            (WHERE MATCH(description, "journalist"))
     | KEEP description, _score
     | LIMIT 2
  """
}

在上面，我们保留了 _score 的值：

FUSE 命令

从上面的结果中，我们可以看出来，FORK 命令虽然可以帮我进行多路召回，但是他们各自的打分体现是不同的。我们需要使用一种方法把多路查询的结果来进行统一打分，并最终给出结果。 FUSE 就是为这种混合搜索而生。FUSE 处理命令会合并多个结果集的行，并分配新的相关性分数。

FUSE 与 FORK 命令结合，可以实现混合搜索，用来合并并给多个查询的结果打分。

FUSE 的工作方式包括：

合并具有匹配 <key_columns> 值的行
使用指定的 <fuse_method> 算法，根据 <group_column> 和 <score_column> 的值分配新的相关性分数

提示：FUSE 用于搜索用例：它会合并已排序的结果集并计算相关性。想了解更多 ES|QL 中搜索的工作方式。

示例：

使用 RRF

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "coder" | SORT _score DESC)
            (WHERE description: "journalist" | SORT _score DESC)
     | FUSE
     | KEEP description, _score
     | LIMIT 2
  """
}

注意：针对上面的搜索，我们必须在 METADATA 里指名 _id, _index 及 _score。否则会有错误！

上面的搜索是针对两种不同的职业来进行搜索的。比较少见。我们直接针对 coder 来进行搜索：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "coder" | SORT _score DESC)
            (WHERE description: "coder" | SORT _score DESC)
     | FUSE
     | KEEP description, _score
     | LIMIT 2
  """
}

为了更加清楚地说明问题，我们添加一个 search_type 来展示：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "coder" | SORT _score DESC | EVAL search_type = "semantic" )
            (WHERE description: "coder" | SORT _score DESC | EVAL search_type = "bm25")
     | FUSE
     | KEEP description, _score, search_type
     | LIMIT 2
  """
}

我们可以直接查询 software developer：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "software developer" | SORT _score DESC | EVAL search_type = "semantic" )
            (WHERE description: "software developer" | SORT _score DESC | EVAL search_type = "bm25")
     | FUSE
     | KEEP description, _score, search_type
     | LIMIT 2
  """
}

使用线性组合

FUSE 也可以使用线性分数组合：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "software developer" | SORT _score DESC | EVAL search_type = "semantic" )
            (WHERE description: "software developer" | SORT _score DESC | EVAL search_type = "bm25")
     | FUSE LINEAR
     | KEEP description, _score, search_type
     | LIMIT 2
  """
}

我们也可以使用定制的 weights。FUSE 允许你基于 _fork 列的值为分数指定不同的权重，让你可以控制每个查询分支在最终结果中的相对重要性。

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "software developer" | SORT _score DESC | EVAL search_type = "semantic" )
            (WHERE description: "software developer" | SORT _score DESC | EVAL search_type = "bm25")
     | FUSE LINEAR WITH { "weights": { "fork1": 0.7, "fork2": 0.3 }, "normalizer": "minmax" }
     | KEEP description, _score, search_type
     | LIMIT 2
  """
}

归一化分数

当使用线性组合将语义查询和词法查询的结果合并时，我们建议先对每个结果集的分数进行归一化。

下面的示例使用 minmax 分数归一化。这意味着在合并行之前，分数会归一化为 0 到 1 之间的值：

POST _query?format=txt
{
  "query": """
     FROM people METADATA _id, _index, _score 
     | FORK (WHERE des_semantic: "software developer" | SORT _score DESC | EVAL search_type = "semantic" )
            (WHERE description: "software developer" | SORT _score DESC | EVAL search_type = "bm25")
     | FUSE LINEAR WITH { "normalizer": "minmax" }
     | KEEP description, _score, search_type
     | LIMIT 2
  """
}