Hive Metastore HA with DBTokenStore: Failed to initialize master key

Hive Metastore HA with DBTokenStore: Failed to initialize master key

By David WORMS

Jul 21, 2016

This article describes my little adventure around a startup error with the Hive Metastore. It shall be reproducable with any secure installation, meaning with Kerberos, with high availability enabled and with the storage of the delegation token in a database. The version of Hive is the 1.2 packaged inside the Hortonworks 2.4.2 distribution.

Storage for delegation token is defined by the hive.cluster.delegation.token.store.class property. The available choices are Zookeeper, the Metastore database and memory. Both Cloudera and Hortonworks recommend using the database as org.apache.hadoop.hive.thrift.DBTokenStore.

The error in question occurs at the launch of the Metastore and bears the following signature:

java.io.IOException: Failed to initialize master key
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
    at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:96)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5945)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:211)
    ... 9 more
Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:158)
    at org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:42)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:193)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:335)
    ... 14 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at java.lang.Class.getMethod(Class.java:1670)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:148)
    ... 17 more
Exception in thread "main" java.io.IOException: Failed to initialize master key
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
    at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:96)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5945)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:211)
    ... 9 more
Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:158)
    at org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:42)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:193)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:335)
    ... 14 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at java.lang.Class.getMethod(Class.java:1670)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:148)
    ... 17 more

The message printed into stdout is quite clear:

java.io.IOException: Failed to initialize master key Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey

Let’s dive into the source code. Hortonworks publishes on GitHub the source code of all the components of its distribution. Each distribution version corresponds to an associated tag under Git. Hive for HDP 2.4.2 can be imported into our local workstation with the commands:

git clone https://github.com/hortonworks/hive-release/blob/HDP-2.4.2.0-tag
git checkout HDP-2.4.2.0-tag

Indeed, there is no function HiveMetaStore$HMSHandler.addMasterKey in version 2.4.2. We can find the exact same function inside the ObjectStore class of the same package. In Hive version 1.3 like in version 2.1, the code didn’t seem to have changed. In the logs, we confirm that it is the ObjectStore class which is mentioned:

metastore.HiveMetaStore - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

So from a configuration point of view, the parameter is well transmitted from the configuration and everything should be OK. The error is called in class DBTokenStore line 42 by the code:

return (Integer)invokeOnRawStore("addMasterKey", new Object[]{s},String.class);

Then again on line 156 with the code:

return rawStore.getClass().getMethod(methName, paramTypes).invoke(rawStore, params)

So, let’s find out how rawStore is built:

  • it arrives through the DBTokenStore.init method
  • not very sure where init is called
  • but after a quick search, objectStore is instantiated in HiveAuthFactory by rawStore = baseHandler.getMS (); where baseHandler is HiveMetaStore.HMSHandler

    • in HiveMetaStore.HMSHandler.getMS, the getMS method returns an instance of threadLocalMS.get() or newRawStore() if null
  • the latter, newRawStore , feeds on the rawStoreClassName property
  • this property is initialized from the configuration
    rawStoreClassName = hiveConf.getVar(HiveConf.ConfVars.METASTORE_RAW_STORE_IMPL);
  • he default value is declared in HiveConf METASTORE_RAW_STORE_IMPL (“hive.metastore.rawstore.impl”, “org.apache.hadoop.hive.metastore.ObjectStore”, …)
  • we return to the class ObjectStore, which contains our method getMasterKey, so what is happening?

By re-browsing the source code, we see that HiveAuthFactory, where rawstore is built and passed to the function startDelegationTokenSecretManager, applies to HiveServer2. Hmm, could there be an equivalent for Metastore?

A search with startDelegationTokenSecretManager leads us again to the HiveMetaStore. And once there, what can we read:

delegationTokenManager.startDelegationTokenSecretManager(
            conf, baseHandler, ServerMode.METASTORE);

With baseHandler being instantiated a little above by the code:

HMSHandler baseHandler = new HiveMetaStore.HMSHandler(
          "new db based metaserver", conf, false);

The signature of startDelegationTokenSecretManager is:

public void startDelegationTokenSecretManager(Configuration conf, Object hms, ServerMode smode) throws IOException

Any object can be passed as a second argument which is logical since Hive is using reflection.

Inspired by HiveAuthFactory, we can see how rawStore is itself obtained from baseHandler. So the parameter is not baseHandler but baseHandler.getMS ().

So, HiveMetaStore on line 6031 should look like:

delegationTokenManager.startDelegationTokenSecretManager(conf, baseHandler.getMS(), ServerMode.METASTORE);

Once the modification are applied, it is now time to compile all of this.

The command mvn clean package -DskipTests -Phadoop-2 is not working on the first attempt. It could actually not happen. Being interested only in the compilation of the jar associated with the Metastore, let us take the option of quickly correcting the compilation errors. Fortunately there will be only one. It consists of removing all references to the class CallerContext in Hadoop23Shims. After that, Maven does not compile all the projects but goes far enough to generate the MetaStore jar.

In terms of deployment, everything is not so simple either. Replacing the jar in question is not enough because another previously loaded jar also contains the impacted classes. I took the solution to rewrite the environment variable HADOOP_CLASSPATH by prefixing it with our newly imported jar. In the file “/usr/hdp/current/hive-metastore/bin/ext/metastore.sh”, add:

export HADOOP_CLASSPATH="/usr/hdp/2.4.2.0-258/hive/lib/hive-metastore.jar:$HADOOP_CLASSPATH"

before the line:

export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS"

You can now execute the command hive --service metastore and the Metastore shall start.

At the time of this writing, it seems that this problem is still present in the master branch of Hive Metastore. However, it was not present in the previous release HDP 2.4.0.

UPDATE: The links in the article might be broken.

Canada - Morocco - France

International locations

10 rue de la Kasbah
2393 Rabbat
Canada

We are a team of Open Source enthusiasts doing consulting in Big Data, Cloud, DevOps, Data Engineering, Data Science…

We provide our customers with accurate insights on how to leverage technologies to convert their use cases to projects in production, how to reduce their costs and increase the time to market.

If you enjoy reading our publications and have an interest in what we do, contact us and we will be thrilled to cooperate with you.