alarmageddon.validations package

Submodules

alarmageddon.validations.kafka module

Convenience Validations for working with Kafka

class alarmageddon.validations.kafka.KafkaStatusValidation(ssh_context, zookeeper_nodes, kafka_list_topic_command='/opt/kafka/bin/kafka-list-topic.sh', priority=2, timeout=None, hosts=None)

Bases: alarmageddon.validations.ssh.SshValidation

Validate that the Kafka cluster has all of it’s partitions distributed across the cluster.

Parameters:

• ssh_contex – An SshContext class, for accessing the hosts.
• zookeeper_nodes – Kafka zookeeper hosts and ports in CSV. e.g. “host1:2181,host2:2181,host3:2181”
• kafka_list_topic_command – Kafka command to list topics (defaults to “/opt/kafka/bin/kafka-list-topic.sh”)
• priority – The Priority level of this validation.
• timeout – How long to attempt to connect to the host.
• hosts – The hosts to connect to.

perform_on_host(host)

Runs kafka list topic command on host

alarmageddon.validations.cassandra module

Convenience Validations for working with Cassandra

class alarmageddon.validations.cassandra.CassandraStatusValidation(ssh_context, service_state='UN', number_nodes=5, owns_threshold=40, priority=2, timeout=None, hosts=None)

Bases: alarmageddon.validations.ssh.SshValidation

Validate that the Cassandra ring is within expected parameters.

Check that the specified Cassandra ring is in the specified state and that the ring ownership of the nodes is within a certain threshold.

Parameters:

• ssh_contex – An SshContext class, for accessing the hosts.
• service_state – The expected service state value (defaults to “UN”).
• number_nodes – The expected number of cassandra nodes in the ring.
• owns_threshold – The maximum percentage of the ring owned by a node.
• priority – The Priority level of this validation.
• timeout – How long to attempt to connect to the host.
• hosts – The hosts to connect to.

check(host, nodes)

Compares the results of nodetool status to the expected results.

perform_on_host(host)

Runs nodetool status and parses the output.

class alarmageddon.validations.cassandra.Node(ip_address, status=0, state=0, load=None, tokens=None, owns=None, host_id=None, rack=None)

Bases: object

Information about a Cassandra node including its load, what percent of the ring it owns, its state, etc.

class alarmageddon.validations.cassandra.NodetoolStatusParser

Bases: object

Parses the output of the Cassandra nodetool status command and tries to make sense of it despite changes made to the format.

parse(status_output)

class alarmageddon.validations.cassandra.State

Bases: object

An enum-like object that represents the state of a Cassandra Node

JOINING = 3

LEAVING = 2

MOVING = 4

NORMAL = 1

UNKNOWN = 0

static from_text(text)

static to_text(value)

Convert State to String

class alarmageddon.validations.cassandra.Status

Bases: object

An enum-like object that represents the status of a Cassandra Node

DOWN = 2

UNKNOWN = 0

UP = 1

static from_text(text)

static to_text(value)

Convert Status to String

alarmageddon.validations.graphite module

Classes that support validation of metrics collected by Graphite

class alarmageddon.validations.graphite.GraphiteContext(graphite_host)

Bases: object

Create one of these and then pass it to all of the GraphiteValidation objects you create.

get_graphite_host()

returns the Graphite host name

class alarmageddon.validations.graphite.GraphiteValidation(context, name, metric_name, time_range=datetime.timedelta(0, 3600), **kwargs)

Bases: alarmageddon.validations.validation.Validation

A Validation that queries Graphite for data and then validates any defined expecations against that data.

expect_average_greater_than(lower_bound)

The average reading of the specified time range should fall above the lower bound

expect_average_in_range(lower_bound, upper_bound)

The average reading of the specified time range should fall between the upper and lower bound

expect_average_less_than(upper_bound)

The average reading of the specified time range should fall below the upper bound

expect_greater_than(lower_bound)

All readings in the specified time range should fall above the lower bound

expect_in_range(lower_bound, upper_bound)

All readings in the specified time range should fall between the upper and lower bound

expect_less_than(upper_bound)

All readings in the specified time range should fall below the upper bound

fail(reason)

Causes this GraphiteValidation to fail with the given reason.

perform(group_failures)

Perform the validation and propagate any failures to reporters

alarmageddon.validations.graphite_expectations module

Expectations that can be held against metrics collected in Graphite

class alarmageddon.validations.graphite_expectations.AverageGreaterThanExpectation(validation, lower_bound)

Bases: alarmageddon.validations.graphite_expectations.GraphiteExpectation

Expect that the average of a graphite metric is greater than a specified number

validate(readings, time_range)

class alarmageddon.validations.graphite_expectations.AverageLessThanExpectation(validation, upper_bound)

Bases: alarmageddon.validations.graphite_expectations.GraphiteExpectation

Expect that the average of a graphite metric is less than a specified number

validate(readings, time_range)

class alarmageddon.validations.graphite_expectations.GraphiteExpectation(validation, name)

Bases: object

An expectation placed on a list of Graphte readings

validate(readings, time_range)

make sure the expectation is met

class alarmageddon.validations.graphite_expectations.GreaterThanExpectation(validation, lower_bound)

Bases: alarmageddon.validations.graphite_expectations.GraphiteExpectation

Expect that a graphite metric is greater than a specified number

validate(readings, time_range)

class alarmageddon.validations.graphite_expectations.LessThanExpectation(validation, upper_bound)

Bases: alarmageddon.validations.graphite_expectations.GraphiteExpectation

Expect that a graphite metric is less than than a specified number

validate(readings, time_range)

alarmageddon.validations.http module

HTTP Validation

class alarmageddon.validations.http.HttpValidation(method, url, data=None, headers=None, priority=2, timeout=None, group=None, retries=1, ignore_ssl_cert_errors=False, auth=None)

Bases: alarmageddon.validations.validation.Validation

A Validation that executes an HTTP request and then performs zero or more checks on the response.

add_expectation(expectation)

Add a custom expecation to the Validation

duplicate_with_hosts(host_names, port=None)

Returns a list of new HttpValidation that are identical to this HttpValidation except with the host name replaced by the elements of host_names.

expect_contains_text(text)

Add an expectation that the HTTP response will contain a particular string.

expect_content_type(content_type)

Add an expectation that the HTTP response’s content type will be equal to the specified content_type.

expect_header(name, value)

Add an expectation that the HTTP response will contain a header with the specified name and value.

expect_json_property_value(json_property_path, expected_value)

Add an expectation that the HTTP response will be JSON and contain a property (found by traversing json_property_path) with the specified value.

expect_json_property_value_greater_than(json_property_path, greater_than)

Add an expectation that the HTTP response will be JSON and contain a numeric property (found by traversing json_property_path) greater than greater_than.

expect_json_property_value_less_than(json_property_path, less_than)

Add an expectation that the HTTP response will be JSON and contain a numeric property (found by traversing json_property_path) less than less_than.

expect_status_codes(status_codes)

Add an expectation that the HTTP response will have one of the specified status_codes.

fail(reason)

Causes this HttpValidation to fail with the given reason.

static get(url, **kwargs)

Create an HttpValidation that will GET to the specified url passing specified headers.

headers - a dictionary where each key is a header name and the value that corresponds to the key is the header value.

priority - the priority of the call; this determines how failures are routed.

timeout - the number of seconds the HTTP request is allowed to take.

group - the group to include this Validation in

get_elapsed_time()

static head(url, **kwargs)

Create an HttpValidation that will retrieve the HEAD of the specified url passing specified headers.

headers - a dictionary where each key is a header name and the value that corresponds to the key is the header value.

priority - the priority of the call; this determines how failures are routed.

timeout - the number of seconds the HTTP request is allowed to take.

group - the group to include this Validation in

static options(url, **kwargs)

Create an HttpValidation that will retrieve OPTIONS for the specified url passing specified headers.

headers - a dictionary where each key is a header name and the value that corresponds to the key is the header value.

priority - the priority of the call; this determines how failures are routed.

timeout - the number of seconds the HTTP request is allowed to take.

group - the group to include this Validation in

perform(group_failures)

Perform the HTTP request and validate the response.

static post(url, **kwargs)

Create an HttpValidation that will POST to the specified url passing specified headers and payload.

headers - a dictionary where each key is a header name and the value that corresponds to the key is the header value.

data - data that is sent along with the request

priority - the priority of the call; this determines how failures are routed.

timeout - the number of seconds the HTTP request is allowed to take.

group - the group to include this Validation in

static put(url, **kwargs)

Create an HttpValidation that will PUT to the specified url passing specified headers and payload.

headers - a dictionary where each key is a header name and the value that corresponds to the key is the header value.

data - data that is sent along with the request

priority - the priority of the call; this determines how failures are routed.

timeout - the number of seconds the HTTP request is allowed to take.

group - the group to include this Validation in

send_header(name, value)

adds an HTTP header with the specified name and value to the request when it’s sent

timer_name()

alarmageddon.validations.http_expectations module

Expectations that can be placed on an HTTP request

class alarmageddon.validations.http_expectations.ExpectContainsText(text)

Bases: alarmageddon.validations.http_expectations.ResponseExpectation

An expectation that an HTTP response will include some text.

validate(validation, response)

class alarmageddon.validations.http_expectations.ExpectedContentType(content_type)

Bases: alarmageddon.validations.http_expectations.ExpectedHeader

An expectation that an HTTP response will have a particular content type

class alarmageddon.validations.http_expectations.ExpectedHeader(name, value)

Bases: alarmageddon.validations.http_expectations.ResponseExpectation

An expectation that an HTTP response will include a header with a specific name and value.

validate(validation, response)

class alarmageddon.validations.http_expectations.ResponseExpectation

Bases: object

An expectation placed on an HTTP response.

validate(validation, response)

If the expectation is met, do nothing. If the expectation is not met, call validation.fail(...)

alarmageddon.validations.json_expectations module

Expectations that can be held against some JSON text

class alarmageddon.validations.json_expectations.ExpectedJsonEquality(json_property_path, value)

Bases: alarmageddon.validations.json_expectations.ExpectedJsonPredicate

expects that a JSON value is equal to a specified value

validate_value(validation, expected_value, actual_value)

class alarmageddon.validations.json_expectations.ExpectedJsonPredicate(json_property_path, value)

Bases: alarmageddon.validations.http_expectations.ResponseExpectation

An expectation that an HTTP response will be JSON and have a property with a specified value.

validate(validation, response)

Validates that the HTTP response is JSON and that it contains a property (found by traversing self.json_property_path) equal to self.value

validate_value(validation, expected_value, actual_value)

validates a JSON value

class alarmageddon.validations.json_expectations.ExpectedJsonValueGreaterThan(json_property_path, value)

Bases: alarmageddon.validations.json_expectations.ExpectedJsonPredicate

Expects that a numeric JSON value is greater than a specified value

validate_value(validation, expected_value, actual_value)

class alarmageddon.validations.json_expectations.ExpectedJsonValueLessThan(json_property_path, value)

Bases: alarmageddon.validations.json_expectations.ExpectedJsonPredicate

Expects that a numeric JSON value is less than a specified value

validate_value(validation, expected_value, actual_value)

alarmageddon.validations.rabbitmq module

Validation for RabbitMQ

class alarmageddon.validations.rabbitmq.RabbitMqContext(host, port, user_name, password)

Bases: object

information needed to connect and interact with RabbitMQ

get_connection(timeout=None)

Connects to RabbitMQ and returns the connection object

Third Party (pika) Bug: https://github.com/pika/pika/issues/354 - Once this bug is fixed we can take out our own retrying logic and use pika’s retry logic. In the mean time, connection failure messages will be inaccurate; they’ll say that only one connection attempt was made.

get_credentials()

get “plain” credentials based on this object’s user name and password

class alarmageddon.validations.rabbitmq.RabbitMqValidation(rabbitmq_context, name, queue_name, max_queue_size, priority=2, timeout=None, num_attempts=4, seconds_between_attempts=2, group=None, ignore_connection_failure=False)

Bases: alarmageddon.validations.validation.Validation

A Validation that can be held against a RabbitMQ server

perform(group_failures)

Perform the validation. If the validation fails, call self.fail passing it the reason for the failure.

alarmageddon.validations.ssh module

Validations that are performed by executing commands remotely on other servers using SSH

We’re using fabric for easy SSH command execution.

class alarmageddon.validations.ssh.LoadAverageValidation(ssh_context, priority=2, timeout=None, group=None, hosts=None)

Bases: alarmageddon.validations.ssh.SshValidation

Validates that a server’s load average falls within a set of parameters

add_expectation(expectation)

check(host, minutes, load)

Make sure that the n-minute load average for the given host is within the allowed range.

expect_exit_code(exit_code)

expect_max_15_minute_load(max_load)

expect a maximum 15 minute load

expect_max_1_minute_load(max_load)

expect a maximum 1 minute load

expect_max_5_minute_load(max_load)

expect a maximum 5 minute load

expect_min_15_minute_load(min_load)

expect a minimum 15 minute load

expect_min_1_minute_load(min_load)

expect a minimum 1 minute load

expect_min_5_minute_load(min_load)

expect a minimum 5 minute load

perform_on_host(host)

Runs the SSH Command on a host and checks to see if all expectations are met.

class alarmageddon.validations.ssh.OutputContains(validation, text)

Bases: alarmageddon.validations.ssh.SshCommandExpectation

Expects that the output of an SSH command is contains specified text

validate(validation, host, command_output, exit_code)

class alarmageddon.validations.ssh.OutputDoesNotContain(validation, text)

Bases: alarmageddon.validations.ssh.SshCommandExpectation

Expects that the output of an SSH command does not contain specified text

validate(validation, host, command_output, exit_code)

class alarmageddon.validations.ssh.OutputGreaterThan(validation, value)

Bases: alarmageddon.validations.ssh.SshCommandExpectation

Expects that the output of an SSH command is greater than the specified value. This method casts the command_output string to a float to do the comparison.

validate(validation, host, command_output, exit_code)

class alarmageddon.validations.ssh.OutputLessThan(validation, value)

Bases: alarmageddon.validations.ssh.SshCommandExpectation

Expects that the output of an SSH command is less than the specified value. This method casts the command_output string to a float to do the comparison.

validate(validation, host, command_output, exit_code)

class alarmageddon.validations.ssh.SshCommandExpectation(validation)

Bases: object

Base class for expectations that can be placed on an SshValidation

fail_on_host(host, reason)

Report a failure and the host the failure occurred on

validate(validation, host, command_output, exit_code)

Defined by derived classes

class alarmageddon.validations.ssh.SshCommandValidation(ssh_context, name, command, working_directory=None, environment=None, priority=2, use_sudo=False, timeout=None, connection_retries=0, group=None, hosts=None)

Bases: alarmageddon.validations.ssh.SshValidation

A validation that runs a command and checks zero or more expectations against its exit code and/or output.

perform_on_host(host)

Runs the SSH Command on a host and checks to see if all expectations are met.

class alarmageddon.validations.ssh.SshCommands

Bases: object

Some commands that might be helpful

static get_cpu_count()

return the number of processors on the server

static get_uptime()

return the system uptime

class alarmageddon.validations.ssh.SshContext(user, key_file)

Bases: object

Context that SSH commands execute in: the user and the user’s key file.

Note that the list of hosts is not part of the SshContext because it changes at a very high rate compared to the user name and their key file.

class alarmageddon.validations.ssh.SshValidation(ssh_context, name, priority=2, timeout=None, group=None, connection_retries=0, hosts=None)

Bases: alarmageddon.validations.validation.Validation

A Validation that is performed using SSH (more specifically, fabric)

add_expectation(expectation)

Adds an expectation deriving from SshCommandExpectation to the list of expectations to be performed as part of the validation.

add_hosts(hosts)

Add additional hosts to run validations against

expect_exit_code(exit_code)

Add the expectation that the SSH command’s exit code is equal to exit_code

expect_output_contains(text)

Add the expectation that the SSH command’s output contains text

expect_output_does_not_contain(text)

Add the expectation that the SSH command’s output does not contain text

fail_on_host(host, reason)

signal failure the test on a particular host

perform(group_failures)

Perform validation against all of this object’s hosts

perform_on_host(host)

perform a validation against a particular host

class alarmageddon.validations.ssh.UpstartServiceValidation(ssh_context, service_name, service_state='running', priority=2, timeout=None, group=None, hosts=None)

Bases: alarmageddon.validations.ssh.SshCommandValidation

Validates that the specified upstart process is in the specified state (e.g. running)

Module contents

Validations that Alarmageddon can perform