1.测试read旧集群input
input {
  elasticsearch {
    hosts => "172.16.1.121:9200"
    index => "bigdata"
    query => '{ "query": { "query_string": { "query": "*" } } }'
    size => 500
    scroll => "5m"
    docinfo => true
  }
}
output {
  stdout {}
}

2.测试写入新集群output
input {
  elasticsearch {
    hosts => "172.16.1.121:9200"
    index => "bigdata"
    query => '{ "query": { "query_string": { "query": "*" } } }'
    size => 500
    scroll => "5m"
    docinfo => true
  }
}
output {
  elasticsearch {
    hosts => "es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200"
    index => "bigdata"
    user => "elastic"
    password => "1zO^U6!zPxHr9vex"
  }
}

2.1失败信息
[2018-07-30T11:51:53,393][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"bigdata", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x233b403f>], :response=>{"index"=>{"_index"=>"bigdata", "_type"=>"doc", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index and [action.auto_create_index] contains [-*] which forbids automatic creation of the index", "index_uuid"=>"_na_", "index"=>"bigdata"}}}}
原因：新集群没创建索引，并且新集群禁止写入时自动创建索引

创建索引
es官方不建议使用transport client，建议使用rest client
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/setup-xpack-client.html

2.2写入新集群时发现type没有使用老集群的type
解决output增加一行
document_type => "%{[@metadata][_type]}"

input {
  elasticsearch {
    hosts => "172.16.1.121:9200"
    index => "bigdata"
    query => '{ "query": { "query_string": { "query": "*" } } }'
    size => 500
    scroll => "5m"
    docinfo => true
  }
}
output {
  elasticsearch {
    hosts => "es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200"
    index => "bigdata"
    document_type => "%{[@metadata][_type]}"
    user => "elastic"
    password => "1zO^U6!zPxHr9vex"
  }
}

3.测试大批量数据迁移
旧集群索引es_advert_slot_app_coupon_di
JestMigrateClient
{
	"2018-07-30": {
		"properties": {
			"ae": {
				"type": "keyword"
			},
			"consume_total": {
				"type": "long"
			},
			"launch_pv": {
				"type": "long"
			},
			"slot_source": {
				"type": "keyword"
			},
			"app_name": {
				"type": "keyword"
			},
			"ef_click_pv": {
				"type": "double"
			},
			"cur_date": {
				"type": "keyword"
			},
			"advert_name": {
				"type": "keyword"
			},
			"slot_id": {
				"type": "long"
			},
			"click_pv": {
				"type": "long"
			},
			"exposure_pv": {
				"type": "long"
			},
			"arpu": {
				"type": "double"
			},
			"slot_name": {
				"type": "keyword"
			},
			"app_id": {
				"type": "long"
			},
			"coupon_type": {
				"type": "keyword"
			},
			"advert_id": {
				"type": "long"
			},
			"ef_click_piece": {
				"type": "double"
			}
		}
	}
}

input {
  elasticsearch {
    hosts => "172.16.1.121:9200"
    index => "es_advert_slot_app_coupon_di"
    query => '{
              	"query": {
              		"match": {
              			"_type": "2018-07-30"
              		}
              	}
              }'
    docinfo => true
  }
}
output {
  elasticsearch {
    hosts => "es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200"
    index => "es_advert_slot_app_coupon_di"
    document_type => "%{[@metadata][_type]}"
    user => "elastic"
    password => "1zO^U6!zPxHr9vex"
  }
}

3.es-sql
3.1site-server未启动成功
3.2插件功能验证
验证查询，通过
http://es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200/_sql?sql=select * from es_advert_slot_app_coupon_di limit 10
dsl转换，通过
http://es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200/_sql/_explain?sql=select * from es_advert_slot_app_coupon_di limit 10

4.es-hadoop
4.1jar依赖问题
4.2脚本properties指定
TBLPROPERTIES(
'es.nodes'='es-cn-4590plkku0004vufk.elasticsearch.aliyuncs.com:9200',
'es.nodes.wan.only'='true'
'es.net.http.auth.user'='elastic',
'es.net.http.auth.pass'='1zO^U6!zPxHr9vex',
'es.resource' = 'es_tuia_app_di_test/2018-07-30'
);
'es.nodes.wan.only'='true' 云ES服务的话这个属性一定要加
'es.net.http.auth.user'='elastic' 用户名
'es.net.http.auth.pass'='1zO^U6!zPxHr9vex' 密码
