How to share ElasticSearch mappings in files on FreeBSD
March 2015.
Let's say you want to create multiple indexes sharing the same object types.
Using the API
You could specify the index mappings using the PUT mapping API, but you'd have to do that every time:
Create the index with the mapping:
curl -XPUT 'http://localhost:9200/awesome_index' -d '
{
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}
}'
Put something into the index:
curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '
{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'
Querying the mapping confirms that it's correct:
curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "boolean"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
}
}
}
}
}
}
Storing the mappings in files
The quest of the home folder
The documentation and people online say that JSON mapping files can be put in folder:
$ES_HOME/config/mappings/_default
But where on earth is that folder???
Maybe it's user
elasticsearch's home folder on the system.
# getent passwd elasticsearch
elasticsearch:*:965:965:elasticsearch user:/nonexistent:/usr/sbin/nologin
Nope, the user doesn't have a home.
On mailing lists, it appears that ES_HOME is where elasticsearch was built. Since I installed it using the ports, that does not make sense to me.
Maybe querying the nodes of the server will give me the info:
# curl 'localhost:9200/_nodes?pretty'
{
"cluster_name" : "elasticsearch",
"nodes" : {
"v0rj_yxBTW2XnbChNL__EA" : {
"name" : "Father Time",
"transport_address" : "inet[/203.0.113.42:9300]",
"host" : "dbs3.example.com",
"ip" : "203.0.113.42",
"version" : "1.4.2",
"build" : "927caff",
"http_address" : "inet[/203.0.113.42:9200]",
"settings" : {
"pidfile" : "/var/run/elasticsearch.pid",
"path" : {
"work" : "/var/tmp/elasticsearch",
"data" : "/var/db/elasticsearch",
"conf" : "/usr/local/etc/elasticsearch",
"logs" : "/var/log/elasticsearch",
"plugins" : "/usr/local/lib/elasticsearch/plugins"
},
"cluster" : {
"name" : "elasticsearch"
},
"config" : "/usr/local/etc/elasticsearch/elasticsearch.yml",
"client" : {
"type" : "node"
},
"name" : "Father Time"
},
[...]
}
The instance does not have a home configured. Hum.
Let's add one in
elasticsearch.yml:
path.home: /usr/home/elasticsearch
Restarting the server and querying the node again confirms that the parameter was understood.
Having the mapping file loaded
Let's add a mapping file in our configured path:
# cat /usr/home/elasticsearch/config/mappings/_default/awesome_doctype.json
{
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}'
Let's delete the index, add a doc and see if the correct mapping was loaded automatically:
# curl -XDELETE 'http://localhost:9200/awesome_index'
{"acknowledged":true}
# curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}
# curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "long"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "string"
},
"a_string" : {
"type" : "string"
}
}
}
}
}
}
It wasn't!
The only remaining solution is to trace the instance and see where it REALLY looks for that mysterious mapping folder.
# ktrace -d /usr/local/openjdk7/bin/java -Des.pidfile=/var/run/elasticsearch.pid -server -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOcc
upancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch -Des.config=/usr/local/etc/elasticsearch/elasticsearch.yml -cp /usr/local/li
b/elasticsearch/elasticsearch-1.4.2.jar:/usr/local/lib/elasticsearch/*:/usr/local/lib/elasticsearch/sigar/* org.elasticsearch.bootstrap.Elasticsearch
[...]
# kdump | grep mappings
[...]
3700 java CALL stat(0x8602998e0,0x7ffffe181668)
3700 java NAMI "/usr/local/etc/elasticsearch/mappings"
3700 java RET stat -1 errno 2 No such file or directory
[...]
Haha! It's in etc!
Let's move the mapping there:
# cat /usr/local/etc/elasticsearch/mappings/_default/awesome_doctype.json
{
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}
Let's try the experiment again:
# curl -XDELETE 'http://localhost:9200/awesome_index'
{"acknowledged":true}
# curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}
# curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "boolean"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
}
}
}
}
}
}
Success!
What needs to be remembered here is that the mappings aren't in the home folder. They're in the conf folder. The fact that the conf folder is in the home folder is just mere coincidence on many systems.