Dependency Management [2]

Transitional dependencies, groups of services, applications between states.
By Kostas Koutsogiannopoulos

We discussed in a previous article about the need of medium to large companies to report efectively the status of their IT ecosystem. We also built an example application doing just that.
We are coming back now to move that solution on the next level, so it can be more practical for developers, administrators and users.

The developer’s perspective

In a service oriented architecture software design, developers are encouraged to serve every functionality "as a service" along with user interfaces. In order to implement functionalities they probably using other external or internal services. So they have:

Services they serve (probably documented)
Services they consume (end points)

Developers are not obligated to know if the services they consume have dependencies of their own.

The administrator’s perspective

Administrators know about applications installed on servers. They do not know about services. Applications for them are sets of services that have dependencies like:

application servers
databases
file servers
HTTP servers
server clusters
etc

In order to adjust our dependency management system to these perspectives, we needed to change our basic entity from "Application" to "Service". Services depending on other services. We only have to know about direct dependencies and calculate indirect dependencies, because the latter are considered unknown. We still need "Application" entity as group of services that have a name, a homepage and run on a logical group of servers.

To visualize this, the new picture is like the following diagram:

So when the problem occurs, admin can mark the relevant "Application" group as failed. Of course this action will mark every single service of the group as failed and other groups as partialy failed based on what services of them depend on the first application services.

Implementation

In order to implement the logic and persist the status of all services, we used Django's . Every instance.save() signal triggers search for parent objects and -maybe- other saves.

This is our new model:

~/operations_env/operations/appstatus/models.py

from django.db import models
from django.db.models.signals import post_save
from django.dispatch import receiver


class Application(models.Model):
    name = models.CharField(max_length=50, unique=True)
    date_modified = models.DateTimeField(auto_now=True)
    description = models.TextField(blank=True, max_length=1024)
    app_url = models.CharField(max_length=100, unique=True, null=True)
    appok = models.BooleanField(default=True)
    failreason = models.TextField(blank=True, max_length=1024)

    def __str__(self):
        return '%s' % self.name

    def appok_rel(self):
        """
        App status depending on its services
        """
        return self.appok and not any(self.services.filter(serviceok=False))

    def app_updated(self):
        """
        Modified when the latest service modification
        """
        return self.services.latest('date_modified').date_modified or self.date_modified


class Service(models.Model):
    name = models.CharField(max_length=50, unique=False)
    application = models.ForeignKey(Application, related_name='services', null=True)
    date_created = models.DateTimeField( auto_now_add=True )
    date_modified = models.DateTimeField(auto_now=True)
    service_url = models.CharField(max_length=100, unique=True)
    description = models.TextField(blank=True, max_length=1024)
    dependencies = models.ManyToManyField('self', symmetrical=False, blank=True, related_name="services")
    serviceok = models.BooleanField(default=True)
    failreason = models.TextField(blank=True, max_length=1024)

    def __str__(self):
        return '%s' % self.name + ' (' + str(self.application) + ')'

    def serviceok_rel(self):
        return self.application.appok and self.serviceok

    def get_all_deps(self, include_self=False):
        dep_list = []
        if include_self:
            dep_list.append(self)
        for service in Service.objects.filter(dependencies=self):
            _dep_list = service.get_all_deps(include_self=True)
            if 0 < len(_dep_list):
                dep_list.extend(_dep_list)
        return dep_list


@receiver(post_save, sender=Service, dispatch_uid="update_serviceok")
def update_status(sender, instance, **kwargs):
    if not Service.objects.filter(dependencies=instance):
        """
        The service has no dependencies, do nothing
        """
        pass
    for service in Service.objects.filter(dependencies=instance):
        if service == instance or service.serviceok == instance.serviceok:
            """
            The service is self depended or already updated, do nothing
            """
            pass
        else:
            """
            Update the service and triger a new post_save signal
            """
            service.serviceok = instance.serviceok
            service.failreason = 'Dependency on %s' % instance.name + ' (' + instance.application.name + ')'
            service.save()

@receiver(post_save, sender=Application, dispatch_uid="update_all_services")
def update_services(sender, instance, **kwards):
    print(instance)
    for service in instance.services.all():
        print(service.name)
        service.serviceok = instance.appok
        service.save()

Views & Templates

We have two views. One for listing the Apps/Services to the users and one for the admins to toggle Application's status(authentication required):

~/operations_env/operations/appstatus/views.py

from appstatus.models import Service, Application
from django.views.generic import ListView
from django.http import HttpResponseRedirect


class ServiceList(ListView):
    model = Application
    template_name = 'service_list.html'
    queryset = Application.objects.order_by('appok')


def StatusUpdate(request):
    app_object = Application.objects.get(id = request.POST.get("application"))
    app_object.failreason = request.POST.get("reason", "")
    app_object.appok = not app_object.appok
    app_object.save()
    return HttpResponseRedirect('/dashboard/')

class ServiceInfo(DetailView):
    model = Application
    template_name = 'service_detail.html'

    def get_context_data(self, **kwargs):
        context = super(ServiceInfo, self).get_context_data(**kwargs)
        return context

The template uses bootstrap panels for the applications and -if there are failed services- uses a list-group to report the services.

If you are authenticated there is also a button on every panel that opens a modal with a form to fill up the reason for status toggle:

~/operations_env/operations/appstatus/templates/service_list.html

{% block app_list %}
    {% for application in object_list|dictsort:"appok_rel" %}
       <div class="col-xs-12 col-sm-4 col-md-3">
       {% if not application.appok %}
          <div class="panel panel-danger">
              <div class="panel-heading">
              {% if request.user.is_authenticated %}
                  <button type="button panel-title" class="btn btn-success pull-right" data-toggle="modal" data-target="#statusModal" data-application="{{ application }}" data-application_id="{{ application.id }}"><span class="glyphicon glyphicon-ok" aria-hidden="true"></span>
                  </button>
              {% endif %}
                  <a href='/applications/{{ application.id }}'><h3 class="panel-title">{{ application.name }}</h3></a>
                  Updated {{ application.app_updated|naturaltime }}
              </div>
              <div class="panel-body">
                  <div>All services down.</div>
                  <div>Reason: {{ application.failreason }}</div>
              </div>
          </div>
       {% elif not application.appok_rel %}
          <div class="panel panel-warning">
              <div class="panel-heading">
              {% if request.user.is_authenticated %}
                  <button type="button panel-title" class="btn btn-warning pull-right" data-toggle="modal" data-target="#statusModal" data-application="{{ application }}" data-application_id="{{ application.id }}"><span class="glyphicon glyphicon-wrench" aria-hidden="true"></span>
                  </button>
              {% endif %}
                  <a href='/applications/{{ application.id }}'><h3 class="panel-title">{{ application.name }}</h3></a>
                  Updated {{ application.app_updated|naturaltime }}
              </div>
              <div class="list-group">
                  {% for service in application.services.all %}
                    {% if service.serviceok_rel %}
                      <p class="list-group-item list-group-item-success">{{ service.name }}</p>
                    {% else %}
                      <li class="list-group-item list-group-item-danger">{{ service.name }}<br>Reason: {{ service.failreason }}</li>
                    {% endif %}
                  {% endfor %}
              </div>
          </div>
       {% else %}
          <div class="panel panel-success">
              <div class="panel-heading">
              {% if request.user.is_authenticated %}
                  <button type="button panel-title" class="btn btn-warning pull-right" data-toggle="modal" data-target="#statusModal" data-application="{{ application }}" data-application_id="{{ application.id }}"><span class="glyphicon glyphicon-wrench" aria-hidden="true"></span>
                  </button>
              {% endif %}
                  <a href='/applications/{{ application.id }}'><h3 class="panel-title">{{ application.name }}</h3></a>
                  Updated {{ application.app_updated|naturaltime }}
              </div>
              <div class="panel-body">
                  <div>All services are up 'n running.</div>
              </div>
          </div>
       {% endif %}
       </div>
    {% endfor %}
<div class="clearfix"></div>
<div class="modal fade" id="statusModal">
  <div class="modal-dialog" role="document">
    <div class="modal-content">
      <div class="modal-header">
        <button type="button" class="close" data-dismiss="modal" aria-label="Close">
          <span aria-hidden="true">&times;</span>
        </button>
        <h4 class="modal-title">Modal title</h4>
      </div>
      <div class="modal-body">
      <form class="form-horizontal" method="post" action="/status_update/">
        <input class="form-control application hidden" id="id_application" name="application" placeholder="Application*" type="text" required />
        <br>
        <input class="form-control" id="id_reason" name="reason" placeholder="Reason*" type="text" required />
            </div>
            <div class="modal-footer">
              <button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
              <input type="submit" class="btn btn-danger" name="action">{% csrf_token %}
            </div>
      </form>
    </div><!-- /.modal-content -->
  </div><!-- /.modal-dialog -->
</div><!-- /.modal -->
{% endblock %}

~/operations_env/operations/appstatus/templates/service_detail.html

<div class="container">
<div>Information for application: <h1>{{ object.name }}</h1></div>
<div>Application is ok? {{ object.appok_rel }}</div>
<br>
<div>
<u>{{ object.name }} is serving the following services:</u>
</div>
<br>
{% for service in object.services.all %}
<div>{{ service.name }}  is ok: {{ service.serviceok_rel }}{% if service.get_all_deps  %} &bull; Consumers: {{ service.get_all_deps }} {% endif  %}</div>
{% endfor %}
</div>

You also need the script for modal opening somewhere in your templates:

Of course you need the admin registration for easy data entry:

~/operations_env/operations/appstatus/admin.py

from django.contrib import admin
from .models import Service, Application


@admin.register(Service)
class ServiceAdmin(admin.ModelAdmin):
    list_display = ('name', 'application', 'serviceok')
    pass

@admin.register(Application)
class ApplicationAdmin(admin.ModelAdmin):
    list_display = ('name', 'date_modified', 'appok')
    pass

At last, you need the three url patterns bellow in your urls.py

~/operations_env/operations/operations/urls.py

url(r'^dashboard/$', ServiceList.as_view()),
url(r'^status_update/$', StatusUpdate, name='status_update'),
url(r'^applications/(?P<pk>[-\w]+)/$', ServiceInfo.as_view(), name='service-detail'),