elasticsearch通用工具类

　　这几天写了一个关于es的工具类，主要封装了业务中常用es的常用方法。

　　本文中使用到的elasticsearch版本6.7，但实际上也支持es7.x以上版本，因为主要是对springboot提供的：ElasticsearchRestTemplate 提供的API做的二次封装。目的是：让不懂es的开发人员新手也能轻松上手。

整个工程分为es-api与es-server。

es-api为对外公共jar,包含了es映射实体；

es-server包含了具体的es工具类与Repository接口(下文会提到)。并通过necos配置中心统一管理es配置参数。

外部业务模块可引入es-api jar maven依赖，由Jar提供的入口，通过httpClient或feign调用(springcloud分布式项目)到es-server服务上的es工具类，得到需要的数据。

这里仅以springcloud分布式项目简单为例

业务方用户服务user 引入es-api maven

/**
 * @author: shf
 * description: es-server feign接口
 */
public interface EsServerClient {
    @PostMapping(value = "/queryList", produces = {"application/json"})
    public <T> List<T> queryList(@RequestBody T t);
}

在user服务中创建feignClient继承自es-api中的EsServerClient

@FeignClient(contextId = "esFeignClient", name = "es-server")
public interface EsFeignClient extends EsServerClient {
}

在user服务的代码中即可调用

@AutoWired
public EsFeignClient esFeignClient;
 
 
public void test() {
   //.......如业务方Dto与es映射实体转换 等省略
  //....
  EmployeeEs employee = new EmployeeEs();
  List queryList = Stream.of(employee.new QueryRelation<String>("张三", EntityEs.SHOULD, 5F), employee.new QueryRelation<String>("李四", EntityEs.SHOULD, 20F)).collect(Collectors.toList());
  employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, queryList).put(EmployeeEs::getUserAge, employee.new RangeRelation(20, EntityEs.GTE, null, null, EntityEs.MUST)));
  //排序查询
  employee.setOrderMap(new EsMapUtil().put(EmployeeEs::getUserId, SortOrder.DESC));
  List<EmployeeEs> employeeEs = esFeignClient.queryList(employee);
  //.....employeeEs与业务方Dto转换
}

三、具体实现

es-api 引入es依赖

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>transport</artifactId>
    <version>6.7.0</version>
</dependency>
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>6.7.0</version>
</dependency>

排除后重新引入对应的Es版本6.7,避免因版本不一致导致的一些坑。

es-server 服务 application.yml配置es

spring:
 elasticsearch:
  rest:
    #ES的连接地址，多个地址用逗号分隔
    uris: localhost:9200
    username: kibana
    password: pass
    #连接超时时间
    connection-timeout: 1000
    #读取超时时间
    read-timeout: 1000

一、映射实体

1、与ES mapping结构对应的映射实体：EmployeeEs

①设置的字段与该es对应的该索引完全对应，不存在多余字段。

②项目中引入了spring-boot-starter-data-elasticsearch，所以可直接使用注解形式设置索引信息与mapping结构信息，详见示例

③@JsonIgnoreProperties({"orderMap","pageNumber","pageSize","highlightFields","preTags","postTags","fieldQueryMap","scrollId","aggregationMap","multiLayerQueryList"})

作用：在保存文档到es时忽略父类EntityEs中的功能性字段，下文会提到

④@EsRepository(EmployeeEsRepository.class)

作用：通过注解过去该映射对应的Repository接口

View Code

2、Repository接口：EmployeeEsRepository

/**
 * @author: shf
 * description: 可根据映射实体设置自动生成mapping结构；支持bean的增删改查操作
 * date: 2022/2/23 10:47
 */
@Component
public interface EmployeeEsRepository extends CrudRepository<EmployeeEs,Long> {
}

二、功能性实体类：EntityEs

①所有字段非es索引中的mapping属性字段，均为功能性字段，如排序、高亮、分页设置和一些常量设置等，详见贴码

②所有es映射实体类均需继承该实体

View Code

三、小工具：EsMapUtil

说明：封装了一个map工具，编码简洁链式调用，应用了方法引用特性避免了字符串硬编码造成单词拼错的情况。

/**
 * @author: shf description: 函数式接口 便于方法引用获取实体字段名称
 * date: 2022/3/4 13:41
 */
@FunctionalInterface
public interface IGetterFunction<T> extends Serializable{
    Object get(T source);
}

View Code

四、Es通用工具类EsService

View Code

五、单元测试

1、根据映射实体设置生成索引与mapping

说明：启动项目会自动加载自动创建，如需要手动创建可调用方法

@Test
public void indexCreateTest() {
    System.out.println(esService.indexCreate(EmployeeEs.class));
}

2、删除指定索引

@Test
public void indexDelete() {
    System.out.println(esService.indexDelete("employee_index"));
}

3、判断指定索引是否存在

@Test
public void indexExists() {
    System.out.println(esService.indexExists(EmployeeEs.class));
}

4、保存单个bean

@Autowired
private EsService esService;
 
@Test
public void saveBean() {
    System.out.println("-----es测试start...");
 EmployeeEs employee = EmployeeEs.builder().userId(9L).userName("张三1").userCode("abc").userAge(22).userMobile("12345678987")
        .remarks("今天天气好晴朗~").birthDay(new Date()).build();
 esService.saveBean(employee);
 System.out.println("-----es测试end...");
}

5、批量保存

@Test
public void saveList() throws ParseException {
    System.out.println("-----es测试start...");
 EmployeeEs employee  =  EmployeeEs.builder().userId(1L).userName("张三").userCode("abc").userAge(18).userSex("男").userMobile("12345678987").birthDay(new SimpleDateFormat("yyyy-MM-dd").parse("1995-9-20")).build();
 EmployeeEs employee1 = EmployeeEs.builder().userId(2L).userName("李四").userCode("abc").userAge(10).userSex("女").userMobile("12345678987").birthDay(new SimpleDateFormat("yyyy-MM-dd").parse("1995-6-20")).build();
 EmployeeEs employee2 = EmployeeEs.builder().userId(3L).userName("王五").userCode("abc").userAge(10).userSex("男").userMobile("12345678987").birthDay(new SimpleDateFormat("yyyy-MM-dd").parse("1995-5-20")).build();
 EmployeeEs employee3 = EmployeeEs.builder().userId(4L).userName("赵六").userCode("abc").userAge(20).userSex("男").userMobile("12345678987").birthDay(new SimpleDateFormat("yyyy-MM-dd").parse("1995-8-20")).build();
 EmployeeEs employee4 = EmployeeEs.builder().userId(5L).userName("董七").userCode("abc").userAge(20).userSex("女").userMobile("12345678987").birthDay(new SimpleDateFormat("yyyy-MM-dd").parse("1995-8-20")).build();
 
 esService.saveList(Lists.newArrayList(employee,employee1,employee2,employee3,employee4));
 System.out.println("-----es测试end...");
}

6、根据ID删除指定Bean

@Test
public void deleteByBean() {
    EmployeeEs employee = EmployeeEs.builder().userId(1L).build();
 esService.deleteByBean(employee);
}

7、批量删除

@Test
public void deleteList() {
    EmployeeEs employee = EmployeeEs.builder().userId(1L).build();
 EmployeeEs employee1 = EmployeeEs.builder().userId(2L).build();
 esService.deleteAllByBeanList(Lists.newArrayList(employee,employee1));
}

8、删除该索引下所有数据

@Test
public void deleteAll() {
    esService.deleteAll(EmployeeEs.class);
}

9、修改数据(ID不能为空)

@Test
public void updateTest() throws IllegalAccessException {
    System.out.println("-----es测试start...");
 EmployeeEs employee = EmployeeEs.builder().userId(5L).userName("张一").userCode("abcD").userAge(19).build();
 esService.updateByBean(employee);
 System.out.println("-----es测试end...");
}

10、根据ID查询指定Bean

@Test
public void queryById() {
    EmployeeEs employeeEs = esService.queryBeanById("2", EmployeeEs.class);
 System.out.println(employeeEs);
}

11、批量查询

/**
 * 涉及到了组合查询bool、权重、范围查找、排序
 */
@Test
public void queryList() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
 EmployeeEs employee = new EmployeeEs();
 List queryList = Stream.of(employee.new QueryRelation<String>("张三", EntityEs.SHOULD, 5F), employee.new QueryRelation<String>("李四", EntityEs.SHOULD, 20F)).collect(Collectors.toList());
 employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, queryList).put(EmployeeEs::getUserAge, employee.new RangeRelation(20, EntityEs.GTE, null, null, EntityEs.MUST)));
 //排序查询
 employee.setOrderMap(new EsMapUtil().put(EmployeeEs::getUserId, SortOrder.DESC));
 List<EmployeeEs> employeeEs = esService.queryList(employee);
 System.out.println(employeeEs);
 System.out.println("-----es测试end...");
}

打印出来的语句：

查询结果：

12、二级属性查询

@Test
public void querySecondList() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
 EmployeeEs employee = new EmployeeEs();
 employee.setFieldQueryMap(new EsMapUtil().putStr("userName.trueName", employee.new QueryRelation<String>("张三", EntityEs.MUST)));
 List<EmployeeEs> employeeEsList = esService.queryList(employee);
 System.out.println(employeeEsList);
 System.out.println("-----es测试end...");
}

先准备一条测试数据插入

看下上面查询语句的结果：

由于userName的二级属性trueName的类型是keyword，所以是term精确查找

@Test
public void querySecondList() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
 EmployeeEs employee = new EmployeeEs();
 employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, employee.new QueryRelation<String>("张三", EntityEs.MUST)));
 //employee.setFieldQueryMap(new EsMapUtil().putStr("userName.trueName", employee.new QueryRelation<String>("张三", EntityEs.MUST)));
 List<EmployeeEs> employeeEsList = esService.queryList(employee);
 System.out.println(employeeEsList);
 System.out.println("-----es测试end...");
}

13、分页查询

@Test
public void queryPage() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
    EmployeeEs employee = new EmployeeEs();
    List<EntityEs.QueryRelation> queryList = Stream.of(employee.new QueryRelation<String>("张三", EntityEs.SHOULD), employee.new QueryRelation<String>("李四", EntityEs.SHOULD)).collect(Collectors.toList());
    employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, queryList));
    //分页
    employee.setPageNumber(0);
    employee.setPageSize(2);
    List<EmployeeEs> employeeEs = esService.queryPage(employee);
    System.out.println(employeeEs);
    System.out.println("-----es测试end...");
}

14、游标分页查询

@Test
public void queryScrollPage() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
    String scrollId = "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAADJEWMVFWMVBnb1ZSZDZsV1k2Y2JjLVlldw==";
    EmployeeEs employee = new EmployeeEs();
    if (StringUtils.isNotBlank(scrollId)) {
        //如果前端有传scrollId
        employee.setScrollId(scrollId);
        List<EmployeeEs> employeeEs = esService.queryScrollPage(employee);
        System.out.println(employeeEs);
    } else {
        List<EntityEs.QueryRelation> queryList = Stream.of(employee.new QueryRelation<String>("张三", EntityEs.SHOULD, 20F), employee.new QueryRelation<String>("李四", EntityEs.SHOULD)).collect(Collectors.toList());
        employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, queryList));
        //每页页数
        employee.setPageSize(4);
        List<EmployeeEs> employeeEs = esService.queryScrollPage(employee);
        System.out.println(employeeEs);
    }
    System.out.println("-----es测试end...");
}

15、多层bool套bool查询

@Test
public void queryMuiltiLayer() throws IllegalAccessException, NoSuchFieldException {
    System.out.println("-----es测试start...");
    EmployeeEs employee = new EmployeeEs();
    
    //多层bool查询
    employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserSex, employee.new QueryRelation("男", EntityEs.MUST)));
    List<EntityEs.QueryRelation> multiLayerUserAgeList = Stream.of(employee.new QueryRelation(10, EntityEs.SHOULD), employee.new QueryRelation(18, EntityEs.SHOULD)).collect(Collectors.toList());
    EntityEs.MultiLayerRelation multiLayerRelation = employee.new MultiLayerRelation(EntityEs.MUST, new EsMapUtil().put(EmployeeEs::getUserAge, multiLayerUserAgeList));
    employee.setMultiLayerQueryList(Lists.newArrayList(multiLayerRelation));
 
    List<EmployeeEs> userEs = esService.queryList(employee);
    System.out.println(userEs);
    System.out.println("-----es测试end...");
}

示例：查找年龄为10或者18岁的男性员工

//错误案例
EmployeeEs employee = new EmployeeEs();
List<EntityEs.QueryRelation> ageList = Lists.newArrayList(employee.new QueryRelation(10, EntityEs.SHOULD), employee.new QueryRelation(18, EntityEs.SHOULD));
employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserSex, employee.new QueryRelation("男", EntityEs.MUST))
    .put(EmployeeEs::getUserAge,ageList)
);
List<EmployeeEs> userEs = esService.queryList(employee);
System.out.println(userEs);

结果年龄为20的赵六也被查询了出来。原因是：当should遇到must和filter时就不是或者了，而是应该的意思。可通过must嵌套一层解决。

修改为多层bool查询

EmployeeEs employee = new EmployeeEs();
//修改为多层bool查询
employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserSex, employee.new QueryRelation("男", EntityEs.MUST)));
List<EntityEs.QueryRelation> multiLayerUserAgeList = Stream.of(employee.new QueryRelation(10, EntityEs.SHOULD), employee.new QueryRelation(18, EntityEs.SHOULD)).collect(Collectors.toList());
EntityEs.MultiLayerRelation multiLayerRelation = employee.new MultiLayerRelation(EntityEs.MUST, new EsMapUtil().put(EmployeeEs::getUserAge, multiLayerUserAgeList));
employee.setMultiLayerQueryList(Lists.newArrayList(multiLayerRelation));
 
List<EmployeeEs> userEs = esService.queryList(employee);

16、高亮查询

@Test
public void highlightTest() throws NoSuchFieldException, IllegalAccessException {
    System.out.println("-----es测试start...");
    EmployeeEs employee = new EmployeeEs();
    employee.setFieldQueryMap(new EsMapUtil().put(EmployeeEs::getUserName, employee.new QueryRelation<String>("张三", EntityEs.MUST))
        .put(EmployeeEs::getRemarks, employee.new QueryRelation<String>("天气", EntityEs.MUST)));
    employee.setHighlightFields(Lists.newArrayList("remarks"));
    //默认<em></em>，可自定义高亮标签
    employee.setPreTags("<h1>");
    employee.setPostTags("</h1>");
    List<EmployeeEs> employeeEs = esService.queryList(employee);
    System.out.println(employeeEs);
    System.out.println("-----es测试end...");
}

17、聚合查询

@Test
public void queryForAggregation() throws NoSuchFieldException, IllegalAccessException {
    System.out.println("-----queryForAggregation-es测试start...");
    EmployeeEs employee = new EmployeeEs();
    employee.setAggregationMap(new EsMapUtil().put(EmployeeEs::getUserAge, EntityEs.COUNT));
    Map employeeEsAggMap = esService.queryForAggregation(employee);
    System.out.println("返回结果:" + employeeEsAggMap);
    System.out.println("-----queryForAggregation-es测试end...");
}

将上面的EntityEs.COUNT改为EntityEs.SUM 求和运行结果：

elasticsearch通用工具类

elasticsearch通用工具类

三、具体实现

一、映射实体

1、与ES mapping结构对应的映射实体：EmployeeEs

2、Repository接口：EmployeeEsRepository

二、功能性实体类：EntityEs

三、小工具：EsMapUtil

四、Es通用工具类EsService

五、单元测试

1、根据映射实体设置生成索引与mapping

2、删除指定索引

3、判断指定索引是否存在

4、保存单个bean

5、批量保存

6、根据ID删除指定Bean

7、批量删除

8、删除该索引下所有数据

9、修改数据(ID不能为空)

10、根据ID查询指定Bean

11、批量查询

12、二级属性查询

13、分页查询

14、游标分页查询

15、多层bool套bool查询

16、高亮查询

17、聚合查询

Recommend

老狗啃爬虫-动态页面爬取之Selenium

老狗咬咬乐-SpringCloud概述

Pico Store：音乐应用《PartyOn》上线，0元嗨唱VR版KTV

What TPMs Do and What Software Engineers Can Learn From Them

外媒：苹果Studio Display显示器运行iOS 15.4系统

Devolver Digital’s Tentacular Launches March 24 For Quest, SteamVR

OnePlus Nord 3 leak suggests it’ll have better fast charging than OnePlus 10 Pro

内存衰退周期要来了吗？

腾讯云携手高阳金信发布“银行核心系统无感上云”联合方案-品玩

美国新法案寻求禁止价值超过 50 亿美元的并购交易

About Joyk