[yagogarea]

Description

riak_core_node_watcher:service_down(Id, true) sends {service_down, Id, health_check}, but handle_call/3 had no clause for it. Every call crashed the node watcher with function_clause. On restart it lost every service and health check registered on the node, not only Id.

Changes

• Add a handle_call({service_down, Id, health_check}, ...) clause. It marks the service down and keeps its health check, with the failure count set to at least max_health_failures. The next successful check brings the service back up.
• If a health check is already running when the service is forced down, discard its result and schedule a new check. Without this, a result computed before the call could put the service straight back up.
• If the service has no health check, it is just marked down.

Tests

• service recovers after a successful check
• failure count above the max is kept
• service without a health check is dropped cleanly
• in-flight check result is ignored

Checked manually on a node:

1> ok = riak_core_node_watcher:service_up(my_svc, self(), {erlang, is_process_alive, []}, [{check_interval, infinity}]).
ok
2> ok = riak_core_node_watcher:service_up(my_svc2, self(),{erlang, is_process_alive, []},[{check_interval, infinity}]).
ok
3> riak_core_node_watcher:services(node()).
[my_svc2,my_svc]
4> riak_core_node_watcher:service_down(my_svc, true).
ok
5> riak_core_node_watcher:services(node()).
[my_svc2]

Closes #1016