Search our Blogs
Showing results for 
Search instead for 
Do you mean 
 

Automatic Vertica Scaling and Node Replacement on AWS

If you are using Amazon Web Services (or are considering using AWS) to host your HP Vertica cluster, then you may wonder if the AWS Auto Scaling service will work with an HP Vertica cluster. Well, wonder no more - the answer is yes!

 

The HP Vertica AWS Auto Scaling open-source package (available on Github) makes it easy. Using this package, you can:

 

  • Provision a new HP Vertica cluster with the number of nodes you want
  • Add horsepower by making your cluster bigger
  • Save money by making your cluster smaller
  • Keep your cluster resilient by automatically replacing failed nodes with new nodes (no need to pay for active standby nodes)
  • Subscribe to receive notifications when nodes are added or removed

The HP Vertica Auto Scaling package is provided to you free of charge (and also free of any warranty or liability). If you like it you can use it as it is, or you can make it better, and contribute your enhancements back to the HP Vertica community.

 

As you read through the sections below, you will learn how it all works, and you'll be able to try it for yourself.

Ready to get started?

 

Architectural overview

The Vertica Auto Scaling package is built around the concept of an auto-scaling group, which controls the creation and termination of all EC2 instances used as HP Vertica cluster nodes. You specify the desired size of the group; the group uses this value to establish and maintain the instance count. The EC2 images are built using the HP Vertica 7.1.2 Amazon Machine Image (AMI) - so the HP Vertica software package is pre-installed.

 

The HP Vertica Auto Scaling package provides the smarts for the AWS Auto Scaling service to integrate with HP Vertica to: a) make a cluster bigger, b) make a cluster smaller, c) replace one or more DOWN nodes.

 

aws_scaling_arch.png

 

 

Here's how it works.

 

To get started, launch a new auto scaling group with just one node, on which you create a bootstrap cluster and database.

 

To grow the cluster, expand the size of the group. This action prompts auto scaling to launch new EC2 instances. Each new node runs a custom launch script, allowing it to join the cluster by connecting (using vsql) to the database running on an existing node in the group, and invoking a script on that node via an HP Vertica external procedure call.

 

To reduce the size of the cluster, decrease the group size. In this case, auto scaling terminates as many EC2 instances as necessary to reduce the group to the new desired size. The auto scaling lifecycle hook mechanism publishes a Simple Queue Service (SQS) message. The SQS message allows the running nodes to automatically execute the necessary scripts to remove the terminating nodes and rebalance the cluster before the instances go offline.

 

You can configure a timeout threshold to specify the maximum amount of time a node is allowed to be down. When the threshold is met, AWS will terminate the EC2 instance associated with the down node. Auto Scaling then attempts to launch a new instance to replace the terminated instance. The new instance’s launch script determines if any nodes are down before it tries to join the cluster. If so, the instance 'adopts' the Private IP Address of the down node and joins the cluster masquerading as the node which failed, initiates node recovery, and so restores the cluster to health.

 

Setup

Download the HP Vertica Auto Scaling package from github or from the HP Haven Marketplace
git clone https://github.com/vertica/aws-autoscaling-vertica.git

 

Create the config file
Copy the template, autoscaling_vars.sh.template, to autoscaling_vars.sh, and edit it to provide valid settings for each variable (see Appendix in the README file). When you are done editing, check the configuration with ./validate_config.sh

 

Make the instance user data launch script
Run ./makeLaunchScript.sh to create launch.sh from the template and the config settings entered in the previous step. The launch.sh script will be run by each new EC2 instance on its first startup.

 

Install and configure the AWS client
Run ./installAWSClient.sh to download, install, and configure the AWS command line interface using the AWS account information you put in autoscaling_vars.sh.

 

Auto Scaling in action

Create a new HP Vertica cluster from scratch

 

Use these three simple steps:

# create autoscaling group with 1 instance

./setup_autoscaling.sh

 

# install Vertica 1-node cluster and set up database

./bootstrap.sh

 

# expand cluster to desired size (per config file)

./scale_cluster.sh

Configuring Autoscaling Group: BobsAutoscalingCluster1

Setting

- min instances:       3

- max instances:       9

- desired instances:   3

Done

 

After a few minutes your cluster will be ready. Now you can import it to your HP Vertica Management Console, create users, load data, hook up your application, etc.

 

To list the public and private IP addresses for each node in your cluster, use the handy cluster_ip_addresses.sh script.

 

Connect using ssh as dbadmin user to the public address of any node using your certificate (pem) file: ssh -i <pem> dbadmin@<publicIp>. Once securely logged in to any cluster node you can connect locally to the database as dbadmin, without a password.

 

You can also connect to the database from outside the cluster using vsql or one of the supported client libraries. E.g. vsql -h <publicIp> -U dbadmin -w <password>.

 

Here's the Management Console view of our new (3-node) cluster:

 

aws_scaling_Mc.png

 

Make your cluster bigger

Edit the autoscaling_vars.sh file, and change the setting desired to a larger number (not greater than max).


Then run ./scale_cluster.sh to configure the new setting, and see your cluster grow! Here we set the desired size to 4 nodes:

 

$ ./scale_cluster.sh

Configuring Autoscaling Group: BobsAutoscalingCluster1

Setting

- min instances:       3

- max instances:       9

- desired instances:   4

Done

 

You can query the table autoscale.launches to see the status as the new instances are added to the cluster, and then to the database.

 

Management Console will automatically detect and add the additional node(s) to its cluster/database view.

 

aws_scaling_Mc2.png

 

Make your cluster smaller

Edit the autoscaling_vars.sh file, and change the setting desired to a smaller number (not smaller than min).

 

Then run ./scale_cluster.sh to configure the new group setting. Here we set the desired size back to 3 nodes:

 

$ ./scale_cluster.sh

Configuring Autoscaling Group: BobsAutoscalingCluster1

Setting

- min instances:       3

- max instances:       9

- desired instances:   3

Done

 

You can query the table autoscale.terminations to see the updating status as the instances are removed from the database, and then from the cluster.

 

Management Console will automatically remove the terminated node(s) from its cluster/database view. Now the database looks just like it did when we started, with 3 nodes again.

 

aws_scaling_Mc3.png

 

Replace a DOWN node

A DOWN node will be automatically terminated after a configurable timeout period, and replaced by a new node which will be configured to adopt the private IP address of the old one.

 

Here, we see a failed node highlighted DOWN in MC:

 

aws_scaling_Mc4.png

 

After the timeout has elapsed (the default is 5 minutes), the AWS EC2 instance for the DOWN node will be terminated by one of the other cluster nodes. This action is logged in the table autoscale.downNodes.

 

The Auto Scaling service will then launch a new instance to restore the cluster to the desired node count. You can query the table autoscale.launches to see the updating status as the new instance is added to replace the failed one.

 

When it has finished, the cluster is restored again to perfect health.

 

aws_scaling_Mc5.png

 

Using the AWS console

The EC2 instances associated with your auto scaling cluster can be seen from the AWS EC2 Console, identified by name tags showing the group name that you specified in the configuration.

 

aws_scaling_console.png

 

You can also view, and even edit, the auto scaling group settings from the EC2 console. If you prefer, you can initiate scale up or scale down actions by changing the desired group from the console, instead of editing the config file and running scale_cluster.sh.

 

aws_scaling_console2.png

 

Dynamic cluster scaling

AWS makes it possible to initiate auto scaling actions based on metrics from Cloudwatch, using auto scaling policies. By adding policies, you could, for example, configure your auto scaling group to add nodes to your cluster if, say, the average CPU utilization of the existing instances exceeds some threshold for some sustained time window, and to reduce the cluster size later, if the CPU utilization average is sustained under some lower threshold. This process can provide true 'hands off' scaling.

 

You can also schedule auto scaling changes via the AWS CLI. For example, if you know that you need a bigger cluster at certain times in your business cycle (for example, end of month, end of quarter, during special events, and so forth.) you could schedule a cluster size increase in advance, and then scale down again when you no longer need the additional nodes.

 

NOTE: Every time nodes are added or removed, the cluster will be rebalanced automatically. Normal database operations can continue during rebalancing, although performance will be degraded until the rebalancing has completed. The HP Vertica 'Elastic Cluster' feature (with local segmentation) will be enabled by default; this helps to minimize the overhead of rebalancing under the right conditions. See HP Vertica docs on Elastic Cluster to learn more about this feature and how to make best use of it.

 

You should be conservative with your auto scaling policies. Cluster rebalancing can be an expensive operation so you may want to experiment with policies that limit the frequency of changes, and/or manage the changes to run during off-peak hours. If you do experiment with dynamic auto scaling policies, please share your examples, and let us know what works well and what doesn't. Perhaps, together as a community, we can establish some auto scaling policy best practices.

 

Placement groups

By default, an AWS EC2 Placement Group will be created for you during setup. All instances in your auto scaling group will be launched within this placement group to maximize the performance and reliability of your cluster, by providing the fastest and most reliable 10Gbps networking possible in AWS.

However, using a placement group makes it possible that an attempt to launch new instances may fail due to insufficient capacity. If that happens, the Auto Scaling service will notify you (via SNS) of a failed launch, and it will retry (and hopefully succeed) a little later.

 

Subscribing for SNS email notifications

After setting up the auto scaling group, go to the AWS SNS console, and select the topic that was created for your group (GroupName_Scale_Event). Use the Action menu to subscribe to the topic and specify your delivery method of choice.

 

Additional information

For additional detail on troubleshooting, and on setting up the configuration file, please refer to the README file on github.

 

To see a demo of the Auto Scaling package in action, please come visit us at the HP Vertica Professional Services booth at the HP Big Data Conference in August.

 

________________________________________________________________________________________

 

This HP Vertica AWS Auto Scale package is free for you to use and redistribute subject to the terms in the included license.txt file. You are welcome (even encouraged) to make it better and contribute your enhancements back to the community.

 

Note that Hewlett Packard does not provide technical support on this or any other package on GitHub or the Marketplace

 

Copyright (c) 2011-2015 by Vertica, an HP Company. All rights reserved.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

 

 

 

Social Media
† The opinions expressed above are the personal opinions of the authors, not of HPE. By using this site, you accept the Terms of Use and Rules of Participation