Krzysztof Żuraw

What is ansible and how's is different from puppet

Ansible is a tool that helps automate boring tasks. These tasks are connected with setting up Linux machines, installing proper software on them and moving code from repositories to machines. Ansible has a different way of accomplishing these tasks than puppet. It is using push system - in short ansible connects to your machine via ssh and push changes. No need for masters and agents etc. Puppet, on the other hand, is using pull system which allows every machine to pull changes from master. Ansible is using the same principles as puppet so you declare how should host look like after running ansible.

Provisioning django application using ansible

I will be provisioning geodjango-leaflet. I assume that you know basic concepts of ansible like play, playbook or role. This is how a structure of my ansible repo looks like:

.
├── ansible.cfg
├── inventory
│   └── vagrant
│       └── hosts.ini
├── playbooks
│   ├── roles -> ../roles/
│   └── vagrant.yaml
├── roles
│   ├── db
│   │   └── tasks
│   │       └── main.yml
│   ├── geodjango
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── tasks
│   │   │   └── main.yml
│   │   └── templates
│   │       ├── nginx.conf.j2
│   │       └── supervisord.conf.j2
│   ├── redis
│   │   └── tasks
│   │       └── main.yml
│   └── roles -> roles
└── Vagrantfile

Let's start from the bottom - Vagrantfile. I will be using vagrant as a playground. Configuration file a.k.a vagrantfile looks as follows:

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "trusty64"
config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
config.ssh.insert_key = false
config.vm.hostname = "vagrant-ansible"
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provision "ansible" do |ansible|
 ansible.playbook = "playbooks/vagrant.yaml"
 ansible.inventory_path = "inventory/vagrant/hosts.ini"
 ansible.sudo = true
 ansible.verbose = "v"
 ansible.limit = "all"
end
end

I setup basic private_network with ip of a vagrant box. In config.vm.provision I specified playbook which should be run in vagrant and inventory where the configuration of my hosts lay. This inventory presents itself below:

vagrant-ansible ansible_ssh_host=192.168.33.10 ansible_ssh_port=22

My ansible playbook don't have tasks inside it but I delegate it to roles:

---
- hosts: vagrant-ansible
  become: yes
  roles:
    - db
    - geodjango
    - redis

Let's start with the first role: db. In folder with this role, I have tasks folder with main.yml:

---
- name: ensure apt cache is up to date
  apt: update_cache=yes
- name: ensure packages are installed
  apt:
    name: "{{item}}"
  with_items:
    - postgresql
    - libpq-dev
    - python-psycopg2
    - postgresql-9.3-postgis-2.1
    - python3-dev
    - python-dev
- name: ensure database is created
  become_user: postgres
  postgresql_db:
    name: geodjango
- name: ensure user has access to database
  become_user: postgres
  postgresql_user:
    db: geodjango
    name: geodjango
    password: geodjango
    priv: ALL
- name: enable postgis for database
  become_user: postgres
  postgresql_ext:
    name: postgis
    db: geodjango

In this task, I run apt-get update at the top then I install a couple of packages so I can setup Postgres. Right below that I create db, grant user access to that db and create PostGIS extension. As this role completes ansible will execute geodjango role:

---
- name: ensure packages are installed
  apt:
    name: "{{item}}"
  with_items:
    - binutils
    - libproj-dev
    - gdal-bin
    - git
    - python-virtualenv
    - build-essential
    - postgresql-server-dev-all
    - supervisor
    - nginx
- name: ensure git repo is present
  git:
    repo: https://github.com/krzysztofzuraw/geodjango-leaflet.git
    dest: /opt/geodjango
- name: create virtualenv
  command: virtualenv /opt/venv -p python3.4 creates="/opt/venv"
- name: install requirements
  pip:
    requirements: /opt/geodjango/requirements.txt
    executable: /opt/venv/bin/pip
- name: migrate django application
  django_manage:
    command: migrate
    virtualenv: /opt/venv
    app_path: /opt/geodjango
- name: load django initial data
  django_manage:
    command: load_inital_voivodeships
    virtualenv: /opt/venv
    app_path: /opt/geodjango
- name: collect static files
  django_manage:
    command: collectstatic
    virtualenv: /opt/venv
    app_path: /opt/geodjango
- name: ensure config dir for supervisor extists
  file:
    path: /etc/supervisor/conf.d
    state: directory
- name: ensure supervisor config is present
  template:
    src: templates/supervisord.conf.j2
    dest: /etc/supervisor/conf.d/geodjango.conf
  notify: reread supervisor
- name: remove default nginx configuration
  file:
    name: /etc/nginx/sites-enabled/default
    state: absent
- name: ensure nginx config is present
  template:
    src: templates/nginx.conf.j2
    dest: /etc/nginx/sites-enabled/geodjango.conf
  notify: restart nginx

This code above is self-explanatory but I will write closely about task called create virtualenv. Normally you can write this and next one task in one like:

pip:
  requirements: /opt/geodjango/requirements.txt
  virtualenv: /opt/venv

And if this virtualenv is not present it will be created. But there is a bug in ansible that is causing these requirements to be installed in system wide python, not virtualenv one. Reference is here. I use fix provided by one of the guys in this issue discussion - I break this task into two separate: one for creating virtualenv and second one for installing requirements.

What is different in this task is that I'm also using templates for supervisor and Nginx. They have j2 ending as ansible is using jinja2 template system. During the ansible run, they will be copied to given dest. At the end of tasks with templates I have notify keyword which tells ansible to look for handlers folder with tasks for restarting services. In my case they look as follows:

---
- name: reread supervisor
  supervisorctl:
    name: geodjango_leaflet
    state: present
- name: restart nginx
  service:
    name: nginx
    state: restarted

The last role is redis. This code installs redis-server and starts it:

---
- name: ensure redis packages are installed
  apt:
    name: "{{item}}"
  with_items:
    - redis-server
- name: ensure redis is started
  become: true
  service:
    name: redis-server
    state: started
    enabled: yes

My thoughts and feelings about ansible

I have to say I'm really impressed on how easy is to write ansible tasks. With puppet, I have this problem that I need to look for modules in puppet forge or write my own. Here everything is included. You want to use django commands - use django_manage, need to reread supervisor config use present in supervisorctl task. Really easy and fun to work with. I can quickly get a job done and move to another stuff.

Yet, I don't know how ansible will behave when it comes to provisioning a large amount of machines. Here I have only one host and it's going smoothly, but for sure when I will have a need for provisioning my private machines I will choose ansible.

That's all for this week blog post! Feel free to comment - I really appreciate that.

Repo with this code is available on github.

Cover image Unsplash under CC0

How message application should work

From previous post I know that the last point of my application flow is to inform user that file is transcoded and ready to download. To do such thing I have to display message on every webpage that current user is. This message should have information about which file was processed. First I wanted to do this with existing django messaging framework but as it turns out is works only with request. As I decided to show message for different users as long as they dismiss this information I had to write my own small application.

Implementation in django

In my newly created application I created following model:

from django.db import models
from django.contrib.auth.models import User
class Message(models.Model):
    text = models.CharField(max_length=250)
    read = models.BooleanField(default=False)
    def __str__(self):
        return self.text

I decided to display my message only when it wasn't read. Based on that right now I can use it in endpoint that works with AWS (audio_transcode/views.py):

@csrf_exempt
def transcode_complete(request):
    # rest of code is in previous blog post
    if json_body['Message']['state'] == 'COMPLETED':
        audio_file = AudioFile.objects.get(
            mp3_file=json_body['Message']['input']['key'][6:]
        )
        Message.objects.create(
            text='Your file {} has been processed'.format(audio_file.name)
         )
    return HttpResponse('OK')

As my message is created right now comes time for displaying it to the user. To do that I have to add a message to template context. It can be done via creating your own context manager:

from .models import Message
def message_context_processor(request):
    if request.user.is_anonymous():
        return {'messages': []}
    return {'messages': Message.objects.filter(read=False)}

And registering it:

TEMPLATES = [
    {
        # rest of options
        'OPTIONS': {
            'context_processors': [
                # rest of context processors
                'transcode_messages.context_processors.message_context_processor'
            ],
        },
     },
 ]

And adding a message as django template tag:

{% if messages %}
  {% for message in messages %}
    <div class="alert alert-success alert-dismissible" data-message-id="{{ message.id }}" data-message-url="{% url 'messages:read-message' %}"role="alert">
      <button type="button" class="close" data-dismiss="alert" aria-label="Close">
      <span aria-hidden="true">x</span>
      </button>
      {{ message.text }}
    </div>
  {% endfor %}
{% endif %}

Which renders as follows:

In the previous screenshot, there is an X that dismiss the message and make it read. To communicate with the backend I wrote quick jQuery script:

var csrftoken = Cookies.get('csrftoken');
function csrfSafeMethod(method) {
    // these HTTP methods do not require CSRF protection
    return (/^(GET|HEAD|OPTIONS|TRACE)$/.test(method));
}
$.ajaxSetup({
    beforeSend: function(xhr, settings) {
        if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
            xhr.setRequestHeader("X-CSRFToken", csrftoken);
        }
    }
});
$('.alert').on('closed.bs.alert', function(event) {
  $.ajax({
    url: event.target.dataset.messageUrl,
    method: 'POST',
    data: {'message_id': event.target.dataset.messageId}
  });
});

Going from the top - django by default uses csrftoken so I have to get it that my request passes the authentication. I'm using here library called js-cookie. In ajaxSetup I tell jQuery to always send csrftokens while using ajax request. Below I add the event listener to an element that has .alert class. This event - closed.bs.alert is provided by bootstrap. On triggering this event I send ajax POST to url from data attribute in alert element - data-message-url. Data that I send is taken from data-message-id attribute on alerts div. How endpoint for receiving such messages looks like? See below:

from .models import Message
from django.http import HttpResponse
def read_message(request):
     message = Message.objects.get(id=request.POST['message_id'])
     message.read = True
     message.save()
     return HttpResponse('OK')

Here I take message_id and set read to True and save message.

That's all for this blog post and blog series! I know that in this design are particular flaws like: what is there will be more users than one? Everybody will see everyone messages. If you have idea how to fix that please write in comments below.

Other blog posts in this series

• Transcoding with AWS- part one
• Transcoding with AWS- part two
• Transcoding with AWS- part three
• Transcoding with AWS- part four

The code that I have made so far is available on github. Stay tuned for next blog post from this series.

Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons

Setting up AWS SNS to work with AWS Transcoder

After logging to AWS console and selecting SNS I have to create a topic:

Topic is endpoint for other application in AWS to send their notifications. For my case I have to change it in AWS Transcoder pipeline settings:

Last thing I have to do was to create subscription for topic created above. They are a lot of types of subscription that you can find in SNS settings but I will be using HTTP request.

Receiving notifications from SNS service in Django

The flow of application will look like this:

1. User upload a file
2. File is sent to S3
3. Transcode job is fired after uploading form view
4. After transcode completion AWS transcoder sends SNS notification
5. This notification is taken by SNS subscription and send to my endpoint
6. After validating notification endpoint inform user that his or her files are transcoded

To receive HTTP notifications I have to create a endpoint in my Django application. First I add url in audio_transcoder/urls.py:

url(
      regex=r'^transcode-complete/$',
      view=views.transcode_complete,
      name='transcode-complete'
  )

Code for this endpoint looks as follows (audio_transcoder/views.py):

from django.views.decorators.csrf import csrf_exempt
from .utlis import convert_sns_str_to_json
from django.http import (
  HttpResponse,
  HttpResponseNotAllowed,
  HttpResponseForbidden
)
@csrf_exempt
def transcode_complete(request):
    if request.method != 'POST':
        return HttpResponseNotAllowed(request.method)
    if request.META['HTTP_X_AMZ_SNS_TOPIC_ARN'] != settings.SNS_TOPIC_ARN:
        return HttpResponseForbidden('Not vaild SNS topic ARN')
    json_body = json.loads(request.body.decode('utf-8'), object_hook=convert_sns_str_to_json)
    if json_body['Message']['state'] == 'COMPLETED':
        # do something
        pass
    return HttpResponse('OK')

What is happening there? The first 2 ifs in transcode_complete are for checking if user sends POST request and as a SNS documentation says I have to make sure that message received are valid as everyone can send request to this endpoint.

In line with json_body I have to use helper that I pass to object_hook:

import json
def convert_sns_str_to_json(obj):
  value = obj.get('Message')
  if value and isinstance(value, str):
      obj['Message'] = json.loads(value)
  return obj

This small function is for converting nested strings received from SNS to python dicts. I know that every notification will have Message key so based on that I can load string and convert it to python dictionary.

The last if will be completed in next blog post.

Right now I have my endpoint up and running. But there is a problem - Amazon SNS needs to have access to that endpoint and I'm developing this application on my localhost. How to overcome such issue? I used ngrok which allows me to tunnel to my localhost from internet. How to use it? After downloading and unpacking you first run:

$ python transcoder/manage.py runserver 0.0.0.0:9000

And in other window:

$ ./ngrok http 9000

Ngrok will start and you can use url shown in console - for me: http://fba8f218.ngrok.io/.

With this url I go to AWS SNS subscription tab and add new subscription:

After setting this up you will receive SNS message with link that you need to paste in browser to confirm subscription.

That's all for today! In the next blog post I will take care about how to inform user that transcode job has completed. Feel free to comment - your feedback is always welcome.

Other blog posts in this series

• Transcoding with AWS- part one
• Transcoding with AWS- part two
• Transcoding with AWS- part three
• Transcoding with AWS- part five

The code that I have made so far is available on github. Stay tuned for next blog post from this series.

Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons

Projects that I did

I really like doing things and based on that learning form it. Recently I read a couple of good pieces about why this is important from people that I personally admire: piece one & piece two.

So I did a couple of projects that I wouldn't have done if I hadn't had this blog, you can find them under this github repo. I also created a small prototype of an application that works with docker python api: tdd-app-prototype. What is more, in my work I started using puppet so I did a simple puppet & vagrant project for provisioning django application using this tool: vagrant-puppet. I wrote a small project that is using reddit api and I learnt about ports & adapters: reddit-stars. The last project that I did last year was simple web crawler: histmag-to-kindle.

I also started one project: poznaj that I'm working on in my free time.

Conferences, events that I went & speeches that I had

I happened to be at 2 conferences previous year: PyConPl and Django Under The Hood. I also did become a mentor in Django Girls Kraków.

I did one speech about microservices during wrocpy.

Open source contributions

This is the area that I can work on more - I made a contribution to python-libarchive-c. It was my first open source contribution! I was so happy that I helped this project. But there are more things that I can do. I realize that most of the people who are involved in open source are doing it in only a couple of projects. It can be connected with that, every project that you are working with you need to know a little bit more and after that you can start contributing more - it takes time and dedication. That's why I want to work a little bit more on mozilla addons-server. I assigned myself to one of the issues there but I haven't had time to work on them. It's high time to fix that. I also joined Django Under The Hood to contribute more to Django but I was able only to work on some documentation fix.

That all for today! It was my short summary of the previous year! Thanks for reading and see you next week.

Special thanks to Kasia for being an editor for this post. Thank you.

Cover image by AngMoKio under CC BY-SA 2.5, via Wikimedia Commons

What is AWS Lambda

AWS Lambda is a service that allows you to run code against some event. What event may you say? For instance uploading a file to S3 bucket. In my example, I use this service to start transcode jobs. User upload file to media/music_file and then instead of firing up event from django application I trigger AWS Lambda function that does the same job.

Right now it's time to jump into the code.

Setting up AWS Lambda for transcoder

When you want to create an AWS Lambda functions you can use a couple of predefined functions a.k.a blueprints. As a base, I used one called: s3-get-object-python. As you chosen your function now it's time to add trigger so the function can run.

And AWS Lambda function is created! But by default, it only gets content type of the object that is put in the S3 bucket. If I want to start to transcode job I can use following code:

import os
import boto3
def lambda_handler(event, context):
  transcoder = boto3.client('elastictranscoder', 'eu-west-1')
  pipeline_id = get_pipeline(transcoder, 'Audio Files')
  base_filename = os.path.basename(event['Records'][0]['s3']['object']['key'])
  output = transcoder.create_job(
      PipelineId=pipeline_id,
      Input={
          'Key': create_aws_filename('media', base_filename, ''),
          'FrameRate': 'auto',
          'Resolution': 'auto',
          'AspectRatio': 'auto',
          'Interlaced': 'auto',
          'Container' : 'auto'
      },
      Outputs=[{
          'Key': create_aws_filename('transcoded', base_filename, '.wav'),
          'PresetId': '1351620000001-300300'
          }, {
          'Key': create_aws_filename('transcoded', base_filename, '.flac'),
          'PresetId': '1351620000001-300110'
          }, {
          'Key': create_aws_filename('transcoded', base_filename, '.mp4'),
          'PresetId': '1351620000001-100110'
          }
      ]
  )
  return output
def get_pipeline(transcoder, pipeline_name):
      paginator = transcoder.get_paginator('list_pipelines')
      for page in paginator.paginate():
          for pipeline in page['Pipelines']:
              if pipeline['Name'] == pipeline_name:
                  return pipeline['Id']
def create_aws_filename(folder, filename, extension):
      aws_filename = os.path.join(
          folder, filename + extension
      )
      return aws_filename

It's code from my previous posts but modified in a few places so it can work in AWS Lambda. In this service, you can use python 2.7. The main function is called lambda_handler and takes an event from S3 in form of JSON and context which is python object. As you can see creating transcoder and pipeline_id are the same as previously. base_filename is taken from event JSON. Then I create transcode job and return its output.

As you may noticed I specified a different folder for outputs than for inputs. Why? Because this function has trigger for put in media. Then it starts transcoder jobs that are creating files on S3 bucket. If I specified the same location for output I can start recursion and AWS Lambda start triggering itself. It's not a good idea and doesn't try this at home unless you have a lot of money. That's why it is so important to test your code before you run it. It's possible - while saving your code you can add the event to test so your AWS Lambda the function will run against this test event:

{
"Records": [
  {
    "eventVersion": "2.0",
    "eventTime": "2016-12-15T21:20:44.231Z",
    "requestParameters": {
      "sourceIPAddress": "IP_ADDRESS"
    },
    "s3": {
      "configurationId": "configurationId",
      "object": {
        "eTag": "eTag",
        "sequencer": "sequencer",
        "key": "media/5981d6e9-8e88-44a9-bd7b-f8dce886877b",
        "size": 571258
      },
      "bucket": {
        "arn": "arn:aws:s3:::YOUR_BUCKET_NAME",
        "name": "YOUR_BUCKET_NAME",
        "ownerIdentity": {
          "principalId": "YOUR_BUCKET_ID"
        }
      },
      "s3SchemaVersion": "1.0"
    },
    "responseElements": {
      "x-amz-id-2": "x-amz-id-2",
      "x-amz-request-id": "x-amz-request-id"
    },
    "awsRegion": "eu-west-1",
    "eventName": "ObjectCreated:Put",
    "userIdentity": {
      "principalId": "AWS:USER_ID"
    },
    "eventSource": "aws:s3"
  }
]
}

Right now clicking test you can know if your function is behaving correctly:

That's all! Your function is working and creating transcode jobs. This is another way of accomplishing the same result - transcoding the files uploaded from Django. If you have any questions don't hesitate to comment!

It's the last blog post in this year - Merry Christmas and Happy New Year!

Special thanks to Kasia for being an editor for this post. Thank you.

Other blog posts in this series

• Transcoding with AWS- part one
• Transcoding with AWS- part two
• Transcoding with AWS- part four
• Transcoding with AWS- part five

Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons