As I recently have opportunity of having a workshop about ansible in my work and I decided to write a blog post on how to provision django application using this tool. In this blog post I am using the same application as in puppet post.
Table of Contents:
Ansible is a tool that helps automate boring tasks. These tasks are connected with setting up Linux machines, installing proper software on them and moving code from repositories to machines. Ansible has a different way of accomplishing these tasks than puppet. It is using push system - in short ansible connects to your machine via ssh and push changes. No need for masters and agents etc. Puppet, on the other hand, is using pull system which allows every machine to pull changes from master. Ansible is using the same principles as puppet so you declare how should host look like after running ansible.
I will be provisioning geodjango-leaflet. I assume that you know basic concepts of ansible like play, playbook or role. This is how a structure of my ansible repo looks like:
.
├── ansible.cfg
├── inventory
│ └── vagrant
│ └── hosts.ini
├── playbooks
│ ├── roles -> ../roles/
│ └── vagrant.yaml
├── roles
│ ├── db
│ │ └── tasks
│ │ └── main.yml
│ ├── geodjango
│ │ ├── handlers
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── nginx.conf.j2
│ │ └── supervisord.conf.j2
│ ├── redis
│ │ └── tasks
│ │ └── main.yml
│ └── roles -> roles
└── Vagrantfile
Let's start from the bottom - Vagrantfile. I will be using vagrant as a playground. Configuration file a.k.a vagrantfile looks as follows:
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "trusty64"
config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
config.ssh.insert_key = false
config.vm.hostname = "vagrant-ansible"
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provision "ansible" do |ansible|
ansible.playbook = "playbooks/vagrant.yaml"
ansible.inventory_path = "inventory/vagrant/hosts.ini"
ansible.sudo = true
ansible.verbose = "v"
ansible.limit = "all"
end
end
I setup basic private_network with ip of a vagrant box. In config.vm.provision I specified playbook which should be run in vagrant and inventory where the configuration of my hosts lay. This inventory presents itself below:
vagrant-ansible ansible_ssh_host=192.168.33.10 ansible_ssh_port=22
My ansible playbook don't have tasks inside it but I delegate it to roles:
---
- hosts: vagrant-ansible
become: yes
roles:
- db
- geodjango
- redis
Let's start with the first role: db. In folder with this role, I have tasks folder with main.yml:
---
- name: ensure apt cache is up to date
apt: update_cache=yes
- name: ensure packages are installed
apt:
name: "{{item}}"
with_items:
- postgresql
- libpq-dev
- python-psycopg2
- postgresql-9.3-postgis-2.1
- python3-dev
- python-dev
- name: ensure database is created
become_user: postgres
postgresql_db:
name: geodjango
- name: ensure user has access to database
become_user: postgres
postgresql_user:
db: geodjango
name: geodjango
password: geodjango
priv: ALL
- name: enable postgis for database
become_user: postgres
postgresql_ext:
name: postgis
db: geodjango
In this task, I run apt-get update at the top then I install a couple of packages so I can setup Postgres. Right below that I create db, grant user access to that db and create PostGIS extension. As this role completes ansible will execute geodjango role:
---
- name: ensure packages are installed
apt:
name: "{{item}}"
with_items:
- binutils
- libproj-dev
- gdal-bin
- git
- python-virtualenv
- build-essential
- postgresql-server-dev-all
- supervisor
- nginx
- name: ensure git repo is present
git:
repo: https://github.com/krzysztofzuraw/geodjango-leaflet.git
dest: /opt/geodjango
- name: create virtualenv
command: virtualenv /opt/venv -p python3.4 creates="/opt/venv"
- name: install requirements
pip:
requirements: /opt/geodjango/requirements.txt
executable: /opt/venv/bin/pip
- name: migrate django application
django_manage:
command: migrate
virtualenv: /opt/venv
app_path: /opt/geodjango
- name: load django initial data
django_manage:
command: load_inital_voivodeships
virtualenv: /opt/venv
app_path: /opt/geodjango
- name: collect static files
django_manage:
command: collectstatic
virtualenv: /opt/venv
app_path: /opt/geodjango
- name: ensure config dir for supervisor extists
file:
path: /etc/supervisor/conf.d
state: directory
- name: ensure supervisor config is present
template:
src: templates/supervisord.conf.j2
dest: /etc/supervisor/conf.d/geodjango.conf
notify: reread supervisor
- name: remove default nginx configuration
file:
name: /etc/nginx/sites-enabled/default
state: absent
- name: ensure nginx config is present
template:
src: templates/nginx.conf.j2
dest: /etc/nginx/sites-enabled/geodjango.conf
notify: restart nginx
This code above is self-explanatory but I will write closely about task called create virtualenv. Normally you can write this and next one task in one like:
pip:
requirements: /opt/geodjango/requirements.txt
virtualenv: /opt/venv
And if this virtualenv is not present it will be created. But there is a bug in ansible that is causing these requirements to be installed in system wide python, not virtualenv one. Reference is here. I use fix provided by one of the guys in this issue discussion - I break this task into two separate: one for creating virtualenv and second one for installing requirements.
What is different in this task is that I'm also using templates for supervisor and Nginx. They have j2 ending as ansible is using jinja2 template system. During the ansible run, they will be copied to given dest. At the end of tasks with templates I have notify keyword which tells ansible to look for handlers folder with tasks for restarting services. In my case they look as follows:
---
- name: reread supervisor
supervisorctl:
name: geodjango_leaflet
state: present
- name: restart nginx
service:
name: nginx
state: restarted
The last role is redis. This code installs redis-server and starts it:
---
- name: ensure redis packages are installed
apt:
name: "{{item}}"
with_items:
- redis-server
- name: ensure redis is started
become: true
service:
name: redis-server
state: started
enabled: yes
I have to say I'm really impressed on how easy is to write ansible tasks. With puppet, I have this problem that I need to look for modules in puppet forge or write my own. Here everything is included. You want to use django commands - use django_manage, need to reread supervisor config use present in supervisorctl task. Really easy and fun to work with. I can quickly get a job done and move to another stuff.
Yet, I don't know how ansible will behave when it comes to provisioning a large amount of machines. Here I have only one host and it's going smoothly, but for sure when I will have a need for provisioning my private machines I will choose ansible.
That's all for this week blog post! Feel free to comment - I really appreciate that.
Repo with this code is available on github.
This is the last blog post in this series - the only thing that has to be done is telling the user that file he or she uploads is processed. It will be done by writing custom message application.
Tabel of Contents:
From previous post I know that the last point of my application flow is to inform user that file is transcoded and ready to download. To do such thing I have to display message on every webpage that current user is. This message should have information about which file was processed. First I wanted to do this with existing django messaging framework but as it turns out is works only with request. As I decided to show message for different users as long as they dismiss this information I had to write my own small application.
In my newly created application I created following model:
from django.db import models
from django.contrib.auth.models import User
class Message(models.Model):
text = models.CharField(max_length=250)
read = models.BooleanField(default=False)
def __str__(self):
return self.text
I decided to display my message only when it wasn't read. Based on that right now I can use it in endpoint that works with AWS (audio_transcode/views.py):
@csrf_exempt
def transcode_complete(request):
# rest of code is in previous blog post
if json_body['Message']['state'] == 'COMPLETED':
audio_file = AudioFile.objects.get(
mp3_file=json_body['Message']['input']['key'][6:]
)
Message.objects.create(
text='Your file {} has been processed'.format(audio_file.name)
)
return HttpResponse('OK')
As my message is created right now comes time for displaying it to the user. To do that I have to add a message to template context. It can be done via creating your own context manager:
from .models import Message
def message_context_processor(request):
if request.user.is_anonymous():
return {'messages': []}
return {'messages': Message.objects.filter(read=False)}
And registering it:
TEMPLATES = [
{
# rest of options
'OPTIONS': {
'context_processors': [
# rest of context processors
'transcode_messages.context_processors.message_context_processor'
],
},
},
]
And adding a message as django template tag:
{% if messages %}
{% for message in messages %}
<div class="alert alert-success alert-dismissible" data-message-id="{{ message.id }}" data-message-url="{% url 'messages:read-message' %}"role="alert">
<button type="button" class="close" data-dismiss="alert" aria-label="Close">
<span aria-hidden="true">x</span>
</button>
{{ message.text }}
</div>
{% endfor %}
{% endif %}
Which renders as follows:
In the previous screenshot, there is an X that dismiss the message and make it read. To communicate with the backend I wrote quick jQuery script:
var csrftoken = Cookies.get('csrftoken');
function csrfSafeMethod(method) {
// these HTTP methods do not require CSRF protection
return (/^(GET|HEAD|OPTIONS|TRACE)$/.test(method));
}
$.ajaxSetup({
beforeSend: function(xhr, settings) {
if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
xhr.setRequestHeader("X-CSRFToken", csrftoken);
}
}
});
$('.alert').on('closed.bs.alert', function(event) {
$.ajax({
url: event.target.dataset.messageUrl,
method: 'POST',
data: {'message_id': event.target.dataset.messageId}
});
});
Going from the top - django by default uses csrftoken so I have to get it that my request passes the authentication. I'm using here library called js-cookie. In ajaxSetup I tell jQuery to always send csrftokens while using ajax request. Below I add the event listener to an element that has .alert class. This event - closed.bs.alert is provided by bootstrap. On triggering this event I send ajax POST to url from data attribute in alert element - data-message-url. Data that I send is taken from data-message-id attribute on alerts div. How endpoint for receiving such messages looks like? See below:
from .models import Message
from django.http import HttpResponse
def read_message(request):
message = Message.objects.get(id=request.POST['message_id'])
message.read = True
message.save()
return HttpResponse('OK')
Here I take message_id and set read to True and save message.
That's all for this blog post and blog series! I know that in this design are particular flaws like: what is there will be more users than one? Everybody will see everyone messages. If you have idea how to fix that please write in comments below.
The code that I have made so far is available on github. Stay tuned for next blog post from this series.
Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons
As I have my transcoder up and running now it's time to let user know that their uploaded files were transcoded. To this occasion I will use AWS SNS service which allows me to send notification about completion of transcode job.
Table of Contents:
After logging to AWS console and selecting SNS I have to create a topic:
Topic is endpoint for other application in AWS to send their notifications. For my case I have to change it in AWS Transcoder pipeline settings:
Last thing I have to do was to create subscription for topic created above. They are a lot of types of subscription that you can find in SNS settings but I will be using HTTP request.
The flow of application will look like this:
To receive HTTP notifications I have to create a endpoint in my Django application. First I add url in audio_transcoder/urls.py:
url(
regex=r'^transcode-complete/$',
view=views.transcode_complete,
name='transcode-complete'
)
Code for this endpoint looks as follows (audio_transcoder/views.py):
from django.views.decorators.csrf import csrf_exempt
from .utlis import convert_sns_str_to_json
from django.http import (
HttpResponse,
HttpResponseNotAllowed,
HttpResponseForbidden
)
@csrf_exempt
def transcode_complete(request):
if request.method != 'POST':
return HttpResponseNotAllowed(request.method)
if request.META['HTTP_X_AMZ_SNS_TOPIC_ARN'] != settings.SNS_TOPIC_ARN:
return HttpResponseForbidden('Not vaild SNS topic ARN')
json_body = json.loads(request.body.decode('utf-8'), object_hook=convert_sns_str_to_json)
if json_body['Message']['state'] == 'COMPLETED':
# do something
pass
return HttpResponse('OK')
What is happening there? The first 2 ifs in transcode_complete are for checking if user sends POST request and as a SNS documentation says I have to make sure that message received are valid as everyone can send request to this endpoint.
In line with json_body I have to use helper that I pass to object_hook:
import json
def convert_sns_str_to_json(obj):
value = obj.get('Message')
if value and isinstance(value, str):
obj['Message'] = json.loads(value)
return obj
This small function is for converting nested strings received from SNS to python dicts. I know that every notification will have Message key so based on that I can load string and convert it to python dictionary.
The last if will be completed in next blog post.
Right now I have my endpoint up and running. But there is a problem - Amazon SNS needs to have access to that endpoint and I'm developing this application on my localhost. How to overcome such issue? I used ngrok which allows me to tunnel to my localhost from internet. How to use it? After downloading and unpacking you first run:
$ python transcoder/manage.py runserver 0.0.0.0:9000
And in other window:
$ ./ngrok http 9000
Ngrok will start and you can use url shown in console - for me: http://fba8f218.ngrok.io/.
With this url I go to AWS SNS subscription tab and add new subscription:
After setting this up you will receive SNS message with link that you need to paste in browser to confirm subscription.
That's all for today! In the next blog post I will take care about how to inform user that transcode job has completed. Feel free to comment - your feedback is always welcome.
The code that I have made so far is available on github. Stay tuned for next blog post from this series.
Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons
As I have static and media files integrated with AWS now it's time to transcode them. In this post, I will write a short example of how to integrate AWS ElasticTranscoder with Django application.
Table of Contents:
ElasticTranscoder allows you to transcode files from your S3 bucket to various formats. To set this service up first you have to create a pipeline. What pipeline is? Basically, it's a workflow- how your transcoder should work. You can create a different pipeline for long content and different for short one. In my application I created the following pipeline:
As I have my pipeline configured next step is to create jobs. Jobs are tasks for a transcoder that say which file I want to transcode, to what format or codec I want to do this:
PresetID is user created or already existing configuration that defines the format of transcoder output: is it mp4 or maybe flac? What resolution should video files have? All of this is set up in present.
As we know basic terms used in AWS Elastic Transcoder let's jump into the code.
AWS has very good python API called boto3. Using that API and few examples from the internet I was able to create a simple class to create transcode job:
import os
from django.conf import settings
import boto3
class AudioTranscoder(object):
def __init__(self, region_name='eu-west-1', pipeline_name='Audio Files'):
self.region_name = region_name
self.pipeline_name = pipeline_name
self.transcoder = boto3.client('elastictranscoder', self.region_name)
self.pipeline_id = self.get_pipeline()
def get_pipeline(self):
paginator = self.transcoder.get_paginator('list_pipelines')
for page in paginator.paginate():
for pipeline in page['Pipelines']:
if pipeline['Name'] == self.pipeline_name:
return pipeline['Id']
def start_transcode(self, filename):
base_filename, _ = self.create_aws_filename(filename, '')
wav_aws_filename, wav_filename = self.create_aws_filename(
filename, extension='.wav'
)
flac_aws_filename, flac_filename = self.create_aws_filename(
filename, extension='.flac'
)
mp4_aws_filename, mp4_filename = self.create_aws_filename(
filename, extension='.mp4'
)
self.transcoder.create_job(
PipelineId=self.pipeline_id,
Input={
'Key': base_filename,
'FrameRate': 'auto',
'Resolution': 'auto',
'AspectRatio': 'auto',
'Interlaced': 'auto',
'Container': 'auto'
},
Outputs=[{
'Key': wav_aws_filename,
'PresetId': '1351620000001-300300'
}, {
'Key': flac_aws_filename,
'PresetId': '1351620000001-300110'
}, {
'Key': mp4_aws_filename,
'PresetId': '1351620000001-100110'
}
]
)
return (wav_filename, flac_filename, mp4_filename)
@staticmethod
def create_aws_filename(filename, extension):
aws_filename = os.path.join(
settings.MEDIAFILES_LOCATION, filename + extension
)
return aws_filename, os.path.basename(aws_filename)
transcoder = AudioTranscoder()
Going from the top - I specified my region_name as well as pipeline_name for boto3 to know to which region it should connect. In method get_pipeline I iterate through all available pipelines and return that has the same name as pipeline_name. In this function paginator is an object which holds on portion of data so user don't have to wait until all available pipelines are fetched.
The main logic is in start_transcode method. At the beginning, I used helper function create_aws_filename that's creating proper AWS file name like media/my_mp3.mp3 and returns that whole path with filename. After I created filenames for all of my files I'm calling create_job that creates a job basing on pipeline_id and base_filename. The job can have multiple outputs so I specified one for wav, flac and mp4 files. How is it used in code? Let's go to view:
class UploadAudioFileView(FormView):
# some code
def form_valid(self, form):
audio_file = AudioFile(
name=self.get_form_kwargs().get('data')['name'],
mp3_file=self.get_form_kwargs().get('files')['mp3_file']
)
audio_file.save()
wav_filename, flac_filename, mp4_filename = transcoder.start_transcode(
filename=audio_file.mp3_file.name
)
audio_file.mp4_file = mp4_filename
audio_file.flac_file = flac_filename
audio_file.wav_file = wav_filename
audio_file.save()
return HttpResponseRedirect(
reverse('audio:detail', kwargs={'pk': audio_file.pk})
)
In form_valid first I'm calling save() on AudioFile object which is uploading the file to S3 bucket. Then I'm using transcoder.start_transcode and basing on output from this function I match filenames to their respective fields. I know that this solution is not the best one as I have to call save twice and if you have a better way to do this I'm glad to hear it from you.
That's all for today! Transcoding works fine but there is a problem with what when files are big? Transcoding such files will take lots of time and user don't want to wait for a response. The solution will be revealed in next post.
The code that I have made so far is available on github. Stay tuned for next blog post from this series.
Special thanks to Kasia for being an editor for this post. Thank you.
While creating this blog post I used an code from offcial boto github account.
Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons
Nowadays moving everything to the cloud becomes more and more popular. A lot of software companies move their technology stack to such infrastructure. One of the biggest players in this field is Amazon Web Services - AWS. That's why I decided decided to adapt existing code from my previous project and move transcoding to write blog posts about that. In this series I process to cloud.
Table of Contents:
I decided to adapt code from my previous blog series about celery and rabbit-mq. I did that because code from this django application actually transcodes mp3 files to other formats. This series will be divided into these parts:
AWS transcoder operates only on files that are inside S3 bucket so first I need to change how these files are served in django.
Let's say that I already had my account on AWS. Next step is to generate specific account using IAM. While creating a user I want to give him access to AWS S3:
and after I download its credentials I can create S3 container - I have chosen Ireland because with Frankfurt I have a problem with uploading files to S3. After bucket creation, it's time to add policy. The policy is basically JSON that tells AWS which user can access given bucket. More information about that can be found here. Adding policy is quite simple from S3 management view:
This policy looks like this:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "PublicReadForGetBucketObjects",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "AWS_RESOURCE"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "AWS_PRINCIPAL"
},
"Action": "s3:*",
"Resource": [
"AWS_RESOURCE",
]
}
]
}
Where AWS_PRINCIPAL is your IAM user in format of ARN resource: "arn:aws:iam::AMAZON_ID:user/IAM_USER" and AWS_RESOURCE: arn:aws:s3:::S3_BUCKET_NAME/*.
As I have this set up right now I can make changes in the application itself. To use S3 as a static and media files container I used django-storages. To make django-storages to work I have to add couple things in settings.py:
import environ
env = environ.Env()
INSTALLED_APPS = [
# other applicaitons
'storages',
]
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME')
AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY')
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_HEADERS = {
'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT',
'Cache-Control': 'max-age=86400',
}
AWS_S3_HOST = 's3-eu-west-1.amazonaws.com'
I'm using here another package called django-environ. It allows me to get certain settings from environmental variables. I'm setting them in my virtualenvwrapper script inside $ENV_PATH/bin/postactivate:
export AWS_STORAGE_BUCKET_NAME='name'
export AWS_ACCESS_KEY_ID='key'
export AWS_SECRET_ACCESS_KEY='acces_id'
The last line with AWS_S3_HOST is really important here as boto - client that django-storages use underneath to connect to AWS doesn't have default region set up. If this is not specified I upload files with redirection which don't allow to transfer static files or upload any large media file.
As I have AWS settings set up there is time to change static files settings in settings.py:
STATICFILES_LOCATION = 'static'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'audio_transcoder.storages.StaticStorage'
STATICFILES_DIRS = (
os.path.join(BASE_DIR.root, 'static'),
)
I add custom StaticStorage as I want my static files to be under static in S3 bucket:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
To upload my static files I simply run python manage.py collectstatic. After a while I can see that my files are in a bucket:
Right now when I run my server I can see the location of my static files:
As static files are working it's high time to use AWS for media files. Right now it's simple - in settings I add:
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'audio_transcoder.storages.MediaStorage'
with custom storage:
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
Now when I upload my mp3 file it's sent directly to S3 bucket under media location:
That's all for today! In the next blog post, I will write about how to set up AWS transcoder.
The code that I have made so far is available on github. Stay tuned for next blog post from this series.
Special thanks to Kasia for being an editor for this post. Thank you.
While creating this blog post I used an excellent tutorial from cactus group.
Cover image by Harald Hoyer under CC BY-SA 2.0, via Wikimedia Commons