7

ELK-学习笔记–elasticsearch的mapping |坐而言不如起而行！ 二丫讲梵

 2 years ago
source link: http://www.eryajf.net/5129.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
ELK-学习笔记–elasticsearch的mapping |坐而言不如起而行！ 二丫讲梵
> 日志管理 > ELK > <十八>ELK-学习笔记–elasticsearch的mapping
本文预计阅读时间 14 分钟

以往采集日志都是比较简单的操作，没有过多深入es的mapping等内容，不过有时候技能都是基于需求驱动的。

现有日志内容如下：

  1. {"sign":"test-log","@timestamp":"2020-07-05T17:43:12+08:00","type":"filter","sale_id":2084090132,"sale_uri":"2003261352dvxv50","shop_id":47516579,"shop_uri":"1910201845lawpvt","cat_id":4,"sec_cat_id":4001,"rule":"startprice","description":"拍品起拍价\u003e0","score":0,"arguments":"{\"startPrice\":2600}"}

因为后期会对日志中一些内容进行聚合计算，因此要求日志中score字段写入之后是float类型，但是如果什么都不指定，那么默认写入之后，会分配一个其他的类型。

两种解决方式。

第一：创建索引的时候指定mapping

  1. PUT test-index
  2. {
  3. "mappings" : {
  4. "properties" : {
  5. "score" : {
  6. "type" : "float"
  7. }
  8. }
  9. }
  10. }

返回结果：

  1. {
  2. "acknowledged" : true,
  3. "shards_acknowledged" : true,
  4. "index" : "test-indexa"
  5. }

查看索引mapping：

  1. GET test-index/_mapping
  1. {
  2. "test-index" : {
  3. "mappings" : {
  4. "properties" : {
  5. "score" : {
  6. "type" : "float"
  7. }
  8. }
  9. }
  10. }
  11. }

这样写进来之后对应的 score字段就是float类型了。

但是这样有一个问题，因为刚刚是指定了单个索引的mapping，正常情况下，我们的日志索引都会按天来存，那么新的索引就无法自动进行对照了。接下来要引入索引模板的配置定义。

  1. PUT _template/template_test
  2. {
  3. "index_patterns": ["test*"],
  4. "order" : 1,
  5. "settings" : {
  6. "number_of_shards": 1,
  7. "number_of_replicas" : 2
  8. },
  9. "mappings" : {
  10. "properties" : {
  11. "score" : {
  12. "type" : "float"
  13. }
  14. }
  15. }
  16. }

创建一个索引模板，只要是以test开头的索引，那么创建索引并写入进来之后，对应的score字段就应该是float类型了。

  1. GET test-index-2020-03-30/_mapping
  2. {
  3. "test-index-2020-03-30" : {
  4. "mappings" : {
  5. "properties" : {
  6. "@timestamp" : {
  7. "type" : "date"
  8. },
  9. "@version" : {
  10. "type" : "text",
  11. "fields" : {
  12. "keyword" : {
  13. "type" : "keyword",
  14. "ignore_above" : 256
  15. }
  16. }
  17. },
  18. "arguments" : {
  19. "type" : "text",
  20. "fields" : {
  21. "keyword" : {
  22. "type" : "keyword",
  23. "ignore_above" : 256
  24. }
  25. }
  26. },
  27. "batch" : {
  28. "type" : "text",
  29. "fields" : {
  30. "keyword" : {
  31. "type" : "keyword",
  32. "ignore_above" : 256
  33. }
  34. }
  35. },
  36. "cat_id" : {
  37. "type" : "long"
  38. },
  39. "description" : {
  40. "type" : "text",
  41. "fields" : {
  42. "keyword" : {
  43. "type" : "keyword",
  44. "ignore_above" : 256
  45. }
  46. }
  47. },
  48. "host" : {
  49. "type" : "text",
  50. "fields" : {
  51. "keyword" : {
  52. "type" : "keyword",
  53. "ignore_above" : 256
  54. }
  55. }
  56. },
  57. "path" : {
  58. "type" : "text",
  59. "fields" : {
  60. "keyword" : {
  61. "type" : "keyword",
  62. "ignore_above" : 256
  63. }
  64. }
  65. },
  66. "rule" : {
  67. "type" : "text",
  68. "fields" : {
  69. "keyword" : {
  70. "type" : "keyword",
  71. "ignore_above" : 256
  72. }
  73. }
  74. },
  75. "sale_id" : {
  76. "type" : "long"
  77. },
  78. "sale_uri" : {
  79. "type" : "text",
  80. "fields" : {
  81. "keyword" : {
  82. "type" : "keyword",
  83. "ignore_above" : 256
  84. }
  85. }
  86. },
  87. "score" : {
  88. "type" : "float"
  89. },
  90. "sec_cat_id" : {
  91. "type" : "long"
  92. },
  93. "shop_id" : {
  94. "type" : "long"
  95. },
  96. "shop_uri" : {
  97. "type" : "text",
  98. "fields" : {
  99. "keyword" : {
  100. "type" : "keyword",
  101. "ignore_above" : 256
  102. }
  103. }
  104. },
  105. "sign" : {
  106. "type" : "text",
  107. "fields" : {
  108. "keyword" : {
  109. "type" : "keyword",
  110. "ignore_above" : 256
  111. }
  112. }
  113. },
  114. "type" : {
  115. "type" : "text",
  116. "fields" : {
  117. "keyword" : {
  118. "type" : "keyword",
  119. "ignore_above" : 256
  120. }
  121. }
  122. }
  123. }
  124. }
  125. }
  126. }

71cfeb93ly1gf8jb767t5j20rs14fajk.jpg

2，logstash处理。

还有一种相对简便的方案是在lgostash层面来做，让日志在从logstash转过来的时候，指定某些字段的类型，配置如下：

  1. input {
  2. kafka {
  3. bootstrap_servers => "192.168.0.1:9092"
  4. group_id => "test-index"
  5. consumer_threads => 6
  6. topics => ["test-index"]
  7. client_id => "test-index"
  8. codec => "json"
  9. check_crcs => "false"
  10. }
  11. }
  12. filter {
  13. mutate {
  14. convert => {
  15. "score" => "float"
  16. }
  17. }
  18. }
  19. output {
  20. elasticsearch {
  21. hosts => ["http://192.168.0.2:9208"]
  22. index => "test-index-%{+YYYY-MM-dd-HH}"
  23. }
  24. }

实际生产中，也会利用这一功能，对NGINX的access日志进行一些特殊处理：

  1. input {
  2. kafka {
  3. bootstrap_servers => "192.168.0.1:9092"
  4. group_id => "nginx_access"
  5. consumer_threads => 6
  6. topics => "nginx_access"
  7. codec => "json"
  8. }
  9. }
  10. filter {
  11. mutate {
  12. split => ["request_uri" , "?"]
  13. add_field => {
  14. "uri_path" => "%{request_uri[0]}"
  15. "uri_query" => "%{request_uri[1]}"
  16. }
  17. remove_field => ["request_uri"]
  18. convert => {
  19. "response" => "integer"
  20. "body_bytes_sent" => "integer"
  21. "request_time" => "float"
  22. "upstream_response_time" => "float"
  23. }
  24. }
  25. }
  26. output {
  27. elasticsearch {
  28. hosts => ["http://192.168.0.2:9208"]
  29. index => "nginx_access-%{+YYYY.MM.dd}-1"
  30. }
  31. }

以针对日志当中一些特殊字段进行相应处理。

weinxin

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK