How to share ElasticSearch mappings in files on FreeBSD


March 2015.
Image The FreeBSD logo

Let's say you want to create multiple indexes sharing the same object types.

Using the API


You could specify the index mappings using the PUT mapping API, but you'd have to do that every time:

Create the index with the mapping:

curl -XPUT 'http://localhost:9200/awesome_index' -d '
{
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}
}'

Put something into the index:

curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '
{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'

Querying the mapping confirms that it's correct:

curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "boolean"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
}
}
}
}
}
}

Storing the mappings in files



The quest of the home folder


The documentation and people online say that JSON mapping files can be put in folder:

$ES_HOME/config/mappings/_default

But where on earth is that folder???

Maybe it's user elasticsearch's home folder on the system.

# getent passwd elasticsearch
elasticsearch:*:965:965:elasticsearch user:/nonexistent:/usr/sbin/nologin

Nope, the user doesn't have a home.

On mailing lists, it appears that ES_HOME is where elasticsearch was built. Since I installed it using the ports, that does not make sense to me.

Maybe querying the nodes of the server will give me the info:

# curl 'localhost:9200/_nodes?pretty'
{
"cluster_name" : "elasticsearch",
"nodes" : {
"v0rj_yxBTW2XnbChNL__EA" : {
"name" : "Father Time",
"transport_address" : "inet[/203.0.113.42:9300]",
"host" : "dbs3.example.com",
"ip" : "203.0.113.42",
"version" : "1.4.2",
"build" : "927caff",
"http_address" : "inet[/203.0.113.42:9200]",
"settings" : {
"pidfile" : "/var/run/elasticsearch.pid",
"path" : {
"work" : "/var/tmp/elasticsearch",
"data" : "/var/db/elasticsearch",
"conf" : "/usr/local/etc/elasticsearch",
"logs" : "/var/log/elasticsearch",
"plugins" : "/usr/local/lib/elasticsearch/plugins"
},
"cluster" : {
"name" : "elasticsearch"
},
"config" : "/usr/local/etc/elasticsearch/elasticsearch.yml",
"client" : {
"type" : "node"
},
"name" : "Father Time"
},
[...]
}

The instance does not have a home configured. Hum.

Let's add one in elasticsearch.yml:

path.home: /usr/home/elasticsearch

Restarting the server and querying the node again confirms that the parameter was understood.

Having the mapping file loaded



Let's add a mapping file in our configured path:

# cat /usr/home/elasticsearch/config/mappings/_default/awesome_doctype.json
{
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}'

Let's delete the index, add a doc and see if the correct mapping was loaded automatically:

# curl -XDELETE 'http://localhost:9200/awesome_index'
{"acknowledged":true}

# curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}

# curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "long"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "string"
},
"a_string" : {
"type" : "string"
}
}
}
}
}
}

It wasn't!

The only remaining solution is to trace the instance and see where it REALLY looks for that mysterious mapping folder.

# ktrace -d /usr/local/openjdk7/bin/java -Des.pidfile=/var/run/elasticsearch.pid -server -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOcc
upancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch -Des.config=/usr/local/etc/elasticsearch/elasticsearch.yml -cp /usr/local/li
b/elasticsearch/elasticsearch-1.4.2.jar:/usr/local/lib/elasticsearch/*:/usr/local/lib/elasticsearch/sigar/* org.elasticsearch.bootstrap.Elasticsearch
[...]

# kdump | grep mappings
[...]
3700 java CALL stat(0x8602998e0,0x7ffffe181668)
3700 java NAMI "/usr/local/etc/elasticsearch/mappings"
3700 java RET stat -1 errno 2 No such file or directory
[...]

Haha! It's in etc!

Let's move the mapping there:

# cat /usr/local/etc/elasticsearch/mappings/_default/awesome_doctype.json
{
"awesome_doctype" : {
"properties" : {
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
},
"a_long" : {
"type" : "long"
},
"a_boolean" : {
"type" : "boolean"
}
}
}
}

Let's try the experiment again:

# curl -XDELETE 'http://localhost:9200/awesome_index'
{"acknowledged":true}

# curl -XPUT 'http://localhost:9200/awesome_index/awesome_doctype/42' -d '{
"a_mysql_date": "2015-02-02 22:22:22",
"a_string": "Je suis une saucisse",
"a_long": 4242,
"a_boolean": 1
}'
{"_index":"awesome_index","_type":"awesome_doctype","_id":"42","_version":1,"created":true}

# curl -XGET 'http://localhost:9200/awesome_index/_mapping/awesome_doctype?pretty'
{
"awesome_index" : {
"mappings" : {
"awesome_doctype" : {
"properties" : {
"a_boolean" : {
"type" : "boolean"
},
"a_long" : {
"type" : "long"
},
"a_mysql_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"a_string" : {
"type" : "string",
"analyzer" : "french"
}
}
}
}
}
}

Success!

What needs to be remembered here is that the mappings aren't in the home folder. They're in the conf folder. The fact that the conf folder is in the home folder is just mere coincidence on many systems.