Category Archives: ambari

Stability of the Apache Ambari REST-API [on hold]

I am starting to work with Apache Ambari and I have to realize a System to get a short Overview over all Ambari Instances in one place. My first approach is to connect to the Ambari-REST API and extract the relevant metadata. My colleagues told me, that the datamodel can be changed in the upgrade process of Ambari to a next Version. Now my Querstion, can I use the REST-API for my use-Case even if Ambari will be upgraded periodically.

Before I want to start programming, I want to weigh up all possible approaches.

I see forward for your Feedback

Best regards Dominik

Overview multiple Ambari instances

I am starting to work with Apache Ambari and I have to realize a System to get a short Overview over all Ambari Instances in one place. Ambari has a Dashboard to view the relevant metadata of the cluster, but in my environment I have multiple Instances. My first approach is to connect to the Ambari-REST API and extract the relevant metadata. This Approach is possible, but wasteful, if an existing System or Framework had realized this use-Case jet.

Before I want to start programming, I want to weigh up all possible approaches.

In following Metadata I am interested in:

  • Cluster-Name
  • Connected Hosts
  • Installed Services
  • Permission and Roles
  • Users
  • Jobs/Workflows
  • Alerts
  • Maybe Configurations

I see forward for your feedback

Best regards Dominik

Gather information about hive table execution

I am working on Hadoop framework/technologies with Hive, Ambari, Ranger and planning to create a dashboard, which provides following information :

  • Execution time of a Hive query.
  • Size of data generated by the created table.
  • Frequency of using each hive schema, table and each column of respective table.
  • User/ Application name or ID firing any query.
  • Resources usage of each application /User.

For now, I am trying to use HiveMetastore and Ranger audit logs to access the above mentioned information.

Is there any other better way to fetch the information above ?

Kindly let me know, if I need to provide any more information.

Error installing Apache Ambari in 3 node cluster

INFO:root:BootStrapping hosts ['prthutech-l2'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu14 with user 'prthutech' sshKey File /var/run/ambari-server/bootstrap/12/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/12 ambari: prthutech-s.local; server_port: 8080; ambari version: 2.1.2; user_run_as: root INFO:root:Executing parallel bootstrap WARNING:root:Bootstrap at host prthutech-l2 timed out and will be interrupted INFO:root:Finished parallel bootstrap

27 Jun 2016 16:44:55,679 INFO [pool-16-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/12 27 Jun 2016 16:44:55,679 INFO [pool-16-thread-1] BSHostStatusCollector:62 - HostList for polling on [prthutech-l2] 27 Jun 2016 16:45:13,540 INFO [qtp-client-32] BootStrapImpl:108 - BootStrapping hosts prthutech-l2: 27 Jun 2016 16:45:13,552 INFO [Thread-192] BSRunner:160 - Kicking off the scheduler for polling on logs in /var/run/ambari-server/bootstrap/13 27 Jun 2016 16:45:13,552 INFO [Thread-192] BSRunner:220 - Host= prthutech-l2 bs=/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py requestDir=/var/run/ambari-server/bootstrap/13 user=prthutech keyfile=/var/run/ambari-server/bootstrap/13/sshKey passwordFile null server=prthutech-s.local version=2.1.2 serverPort=8080 userRunAs=root

27 Jun 2016 16:46:13,555 INFO [pool-17-thread-1] BSHostStatusCollector:62 - HostList for polling on [prthutech-l2] 27 Jun 2016 16:46:14,693 INFO [Thread-192] BSRunner:264 - Bootstrap output, log=/var/run/ambari-server/bootstrap/13/bootstrap.err /var/run/ambari-server/bootstrap/13/bootstrap.out at prthutech-s.local 27 Jun 2016 16:46:14,694 INFO [Thread-192] BSRunner:275 - Script log Mesg

INFO:root:BootStrapping hosts ['prthutech-l2'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu14 with user 'prthutech' sshKey File /var/run/ambari-server/bootstrap/13/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/13 ambari: prthutech-s.local; server_port: 8080; ambari version: 2.1.2; user_run_as: root INFO:root:Executing parallel bootstrap ERROR:root:ERROR: Bootstrap of host prthutech-l2 fails because previous action finished with non-zero exit code (255) ERROR MESSAGE: ssh: connect to host prthutech-l2 port 22: Connection timed out

STDOUT: ssh: connect to host prthutech-l2 port 22: Connection timed out

INFO:root:Finished parallel bootstrap

27 Jun 2016 16:46:14,694 INFO [pool-17-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/13 27 Jun 2016 16:46:14,694 INFO [pool-17-thread-1] BSHostStatusCollector:62 - HostList for polling on [prthutech-l2] 27 Jun 2016 16:50:48,839 INFO [qtp-client-32] BootStrapImpl:108 - BootStrapping hosts prthutech-l2: 27 Jun 2016 16:50:48,842 INFO [Thread-199] BSRunner:160 - Kicking off the scheduler for polling on logs in /var/run/ambari-server/bootstrap/14 27 Jun 2016 16:50:48,842 INFO [Thread-199] BSRunner:220 - Host= prthutech-l2 bs=/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py requestDir=/var/run/ambari-server/bootstrap/14 user=prthutech keyfile=/var/run/ambari-server/bootstrap/14/sshKey passwordFile null server=prthutech-s.local version=2.1.2 serverPort=8080 userRunAs=root 27 Jun 2016 16:50:48,849 INFO [pool-18-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/14 27 Jun 2016 16:50:48,849 INFO [pool-18-thread-1] BSHostStatusCollector:62 - HostList for polling on [prthutech-l2] .........................................

27 Jun 2016 16:55:48,862 INFO [pool-18-thread-1] BSHostStatusCollector:62 - HostList for polling on [prthutech-l2] 27 Jun 2016 16:55:50,217 INFO [Thread-199] BSRunner:264 - Bootstrap output, log=/var/run/ambari-server/bootstrap/14/bootstrap.err /var/run/ambari-server/bootstrap/14/bootstrap.out at prthutech-s.local 27 Jun 2016 16:55:50,218 INFO [Thread-199] BSRunner:275 - Script log Mesg

INFO:root:BootStrapping hosts ['prthutech-l2'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu14 with user 'prthutech' sshKey File /var/run/ambari-server/bootstrap/14/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/14 ambari: prthutech-s.local; server_port: 8080; ambari version: 2.1.2; user_run_as: root INFO:root:Executing parallel bootstrap WARNING:root:Bootstrap at host prthutech-l2 timed out and will be interrupted INFO:root:Finished parallel bootstrap

/var/lib/ambari-server/keys/keystore.p12 file not being generated while starting ambari-server

I've added a new stack in amabari-server and built it. now after installing the ambari-server rpm on new hosts, when i start ambari-server, it give following error:

20 Jun 2016 16:25:53,020 INFO [main] Configuration:1067 - Web App DIR test /usr/lib/ambari-server/web 20 Jun 2016 16:25:53,027 INFO [main] CertificateManager:68 - Initialization of root certificate 20 Jun 2016 16:25:53,027 INFO [main] CertificateManager:70 - Certificate exists:false 20 Jun 2016 16:25:53,027 INFO [main] CertificateManager:137 - Generation of server certificate 20 Jun 2016 16:25:55,627 INFO [main] ShellCommandUtil:44 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096 was finished with exit code: 0 - the operation was completely successfully. 20 Jun 2016 16:25:55,644 INFO [main] ShellCommandUtil:44 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the operation was completely successfully.

20 Jun 2016 16:25:55,654 WARN [main] ShellCommandUtil:46 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 1 - an error occurred parsing the command options.

20 Jun 2016 16:25:55,663 WARN [main] ShellCommandUtil:46 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:**** was finished with exit code: 1 - an error occurred parsing the command options. 20 Jun 2016 16:25:55,696 INFO [main] AmbariServer:611 - Jetty is configuring qtp-ambari-agent with 4 reserved acceptors/selectors and a total pool size of 25 for 4 processors. 20 Jun 2016 16:25:55,717 INFO [main] ViewRegistry:1538 - Reading view archive /var/lib/ambari-server/resources/views/ambari-admin-2.2.2.1.0.jar.

20 Jun 2016 16:26:02,549 WARN [main] AbstractLifeCycle:204 - FAILED [email protected](/var/lib/ambari-server/keys/keystore.p12,/var/lib/ambari-server/keys/keystore.p12): java.io.FileNotFoundException: /var/lib/ambari-server/keys/keystore.p12 (No such file or directory) java.io.FileNotFoundException: /var/lib/ambari-server/keys/keystore.p12 (No such file or directory)

Can anyone help with this? I'm using CentOS7 for ambari-installation.

Ambari – Ranger usersync

I installed Apache Ranger on my Hadoop cluster, including HDFS, Hive and HBase plugin - all of them are functional. However, I also integrated ranger-usersync, but somehow it does have a negative impact, which leads to the following error (while refreshing the client configs in Ambari / restarting a service):

conf-select set-conf-dir --package ranger-usersync --stack-version 1.0.0.0 --conf-version 0

error ​ranger-usersync not installed or incorrect package name

How can I configure Ambari so it recognises ranger-usersync? The usersync itself is working, but the it does have an implication on the other components, since Ambari aborts e.g. the starting process of services.

Thank you

Ambari – Ranger usersync

I installed Apache Ranger on my Hadoop cluster, including HDFS, Hive and HBase plugin - all of them are functional. However, I also integrated ranger-usersync, but somehow it does have a negative impact, which leads to the following error (while refreshing the client configs in Ambari / restarting a service):

conf-select set-conf-dir --package ranger-usersync --stack-version 1.0.0.0 --conf-version 0

error ​ranger-usersync not installed or incorrect package name

How can I configure Ambari so it recognises ranger-usersync? The usersync itself is working, but the it does have an implication on the other components, since Ambari aborts e.g. the starting process of services.

Thank you