Krzysztof Żuraw

Ports and Adapters in python - part three

2016-06-12T10:20:00+02:00

Next part of my application will be module for saving links to read them later.

In the last post, I made a reddit search view for the specific keyword that display results to the user. To save them to read later I need database representation of link from reddit:

from django.db import models

class RedditLink(models.Model):
    title = models.CharField(max_length=250)
    is_favourite = models.BooleanField(default=False)

    def save(self, *args, **kwargs):
        if self.is_favourite:
            super(RedditLink, self).save(*args, **kwargs)

I made my own save because I only need links that are favorite in my database. In addition, I have multiple reddit links on my search page to save. So how to handle multiple forms of the same model in django? The answer is to use Fromset. What is it? It is module provided by django for creation multiple forms. How to use it? Look at forms.py:

from django import forms
from .models import RedditLink

RedditAddToFavouritesFormset = forms.modelformset_factory(
    RedditLink,
    fields=('title', 'is_favourite'),
    extra=5
)

I used something called forms.modelformset_factory which is a function to produce fromset from model. So I provided model name with fields to calling this function. What is more, I add additional argument extra for creating more than one form in formset. How to use newly created RedditAddToFavouritesFormset? In views:

from django.views.generic import CreateView
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse_lazy
from .forms import RedditAddToFavouritesFormset


class RedditAddToFavourites(CreateView):
    template_name = 'search/index.html'
    success_url = reverse_lazy('main_page')

    def post(self, request, *args, **kwags):
        reddit_links_formset = RedditAddToFavouritesFormset(request.POST)
        if reddit_links_formset.is_valid():
            reddit_links_formset.save()
            return HttpResponseRedirect(success_url)
        else:
            return self.render_to_response(
                'search/index.html',
                self.get_context_data(
                    reddit_links_formset=reddit_links_formset
                )
            )

I write RedditAddToFavourites which is a subclass of CreateView. The main point for this view class is to create RedditLink instances from formset. So I override post method which is responsible for handling POST requests. At first I a create new instance of formset from the request. After validation if everything was filled in correctly by the user. If so I save formset and create entries in database. Then HttpResponseRedirect redirect user to main page. If validation was incorrect I rerender template with form errors. Thanks to that my search/index.html looks as follows:

{% if sucess %}
    <form method="post" action="{% url 'add_to_favourites' %}">{% csrf_token %}
        <table>
            {{ reddit_links_formset }}
        </table>
            <input type="submit" class="btn btn-primary" value="Favourite">
    </form>
{% else %}

To insert values that are from search I have to instantiate formset with argument initial in search/views.py under get_context_data method:

reddit_links_formset = RedditAddToFavouritesFormset(
    initial=[{'title': title} for title in self.search_result[:5]]
)

And that all! Right now when user type query to search bar and click search he or she is redirected to page with 5 forms that have initial title set. After that user select favorite links and saves them to database. But I see a problem here: first, I only display for user 5 forms with data from search results and I want it more, but it is what I will be taking care of in next blog post.

I really appreciate every comment that you have! You can reach me in any way- just click icons at the bottom of this very page. Thank you for reading! Code for this you can find under this repo.

Cover image by Creative Magic under CC0.

Ports and Adapters in python - part two

2016-06-05T10:20:00+02:00

Last time I wrote about how to do simple port & adapter in python. In this post, I will show to actually use them.

I briefly remind you what is purpose of application build in this series: user will log in, then search with keyword so he can save any search result to database for read it later.

I decided to first implement search mechanism for Reddit. This is what I will write today. Search request will be sent via GET. First, I need some form to handle this:

 1  from django import forms
 2  from django.conf import settings
 3 
 4  from external_api.external_api_port import instantiated_port
 5 
 6  class RedditSearchForm(forms.Form):
 7      query = forms.CharField(label='search query', max_length=100)
 8 
 9      def perform_search(self):
10          search_result = instantiated_port.search(self.cleaned_data['query'])
11          return search_result

I defined simple form that has only one field: query which is CharField field with label. My form has one method perform_search. In this method, I import instantiated reddit port that takes instance of reddit adapter with settings from django settings module. Idealy this adapter should be singleton class. This is how it looks in reddit_adapter:

from django.conf import settings

# reddit adapter class here ...

instantiated_adapter = RedditAdapter(
    settings.REDDIT_CLIENT_ID,
    settings.REDDIT_CLIENT_SECRET,
    settings.REDDIT_USERNAME,
    settings.REDDIT_PASSWORD
)

and in external_api_port:

from .reddit_adapter import instantiated_adapter

# port class here ...

instantiated_port = ExternalAPIPort(instantiated_adapter)

Lastly, I perform the search using the port and cleaned_data['query']. I have access to cleaned_data attribute after form validation which will be shown in the view. At the end of perform_search I return search results. These results are processed further in view:

 1 from django.views.generic.edit import FormView
 2 from django.http import HttpResponse
 3 from django.shortcuts import render
 4 from .forms import RedditSearchForm
 5 
 6 class RedditSearchView(FormView):
 7     template_name = 'search/index.html'
 8     form_class = RedditSearchForm
 9     success_url = 'add-to-favourites'
10     search_result = None
11 
12     def get(self, request, *args, **kwargs):
13         form = self.form_class(self.request.GET or None)
14         if form.is_valid():
15             self.search_result = form.perform_search()
16         return self.render_to_response(self.get_context_data(form=form))
17 
18     def get_context_data(self, **kwargs):
19         context = super(RedditSearchView, self).get_context_data(**kwargs)
20         if self.search_result:
21             context.update({
22                 'search_result': self.search_result,
23                 'sucess': True
24                 }
25             )
26         return context

Let begin from get method: this method is called every time get request is performed by the user. How to ensure that? I used method parameter in html:

<form method="get" class="form" role="form">
    {{ form }}
    <input type="submit" class="btn btn-primary" value="Search">
</form>

In get method I get the form for given request.GET. On this form I call form.is_valid() to get access to cleaned_data. After that I have search results so I can insert them to html. It is done via get_context_data method when I get my basic context calling super. And if there was search performed I update context with search results and I tell my html to render them in one template.

Such updated context is taken by django and rendered to full html. Key success is present because I got if statement in html template which allows me to render results on the same page that search was performed:

{% if sucess %}
    {% for item in search_result %}
        <li>{{ item }}</li>
    {% endfor %}
{% else %}
<!--- form here ---!>

And that basically all for search view. In next post I will take care of saving results to database. Code for this you can find under this repo.

Changes from 07.06.16:

Moving port & adapter to it's own module
Having only one instance of port & adapter

(Special thanks for pointing this to Mariusz)

Cover image by Creative Magic under CC0.

Ports and Adapters in python - part one

2016-05-22T10:20:00+02:00

Welcome! Today I'm going to start series about how to use port and adapter design pattern in simple django application.

Let me explain a little bit what exactly ports and adapters design pattern is. According to this article (which by the way I strongly recommend to read) it is a way to separate business logic from user code.

What I mean by that? Let pretend that you want to create simple django application which connects to reddit using its API. Then app retrieves the content of search query provided by the user. After that user can save for later founded link. In this blog post, I will focus only on reddit API part. Normally you will write some module using request for retrieving search results from reddit. But what when it comes to testing such code? You just mock requests calls or use responses library.

How do you do it in ports and adapters way? You will have one thing called port for all external connections. Throught this all requests to external APIs will be done because who knows if the reddit will not change to duckduckgo? In such case you just add DuckDuckGo Adapter and you are all set. Because port don't care if there is Reddit adapter or DuckDuckGo adapter as long as it provides necessary methods. As I mentioned before, port is communicating only with adapters. And what is adapter? It is part of code designed only for calling in this case Reddit API and passing results. To test port you need fake adapter with all methods that original has. But how you test adapter? You will have to write integration tests.

Such design pattern is also called hexagonal architecture.

This is a picture presenting adapters and ports in my application.

As you can see in above picture all connections to external APIs are made using ExternalAPIPort so this class in python knows only about the adapter and that it should have some search method for searching. How is it translating to code?

File external_api_ports.py:

class ExternalAPIPort(object):

    def __init__(self, adapter):
        self.adapter = adapter

    def search(self, query, *args, **kwargs):
        return self.adapter.search(query, *args, **kwargs)

As you can see port takes adapter in __init__. Then in search it uses adapter method for searching and passing results. I only needed the title of a post that comes from search so I generate them using generator expression. Moreover here we have contract that tell us that adapter has to have such method as search that uses query arguments (at least).

And how adapter look like?

reddit_adapter.py:

 1 import requests
 2 import requests.auth
 3 
 4 
 5 class RedditAdapter(object):
 6     def __init__(
 7         self, reddit_client_id, reddit_client_secret,
 8         reddit_username, reddit_password
 9     ):
10         self.reddit_client_id = reddit_client_id
11         self.reddit_client_secret = reddit_client_secret
12         self.reddit_username = reddit_username
13         self.reddit_password = reddit_password
14         self.reddit_token = None
15 
16     def authorize(self):
17         client_auth = requests.auth.HTTPBasicAuth(
18             self.reddit_client_id,
19             self.reddit_client_secret
20          )
21          post_data = {
22               "grant_type": "password",
23               "username": self.reddit_username,
24               "password": self.reddit_password
25          }
26          headers = {"User-Agent": "RedditAdapter/0.1 by Krzysztof Zuraw"}
27          response = requests.post(
28              "https://www.reddit.com/api/v1/access_token",
29              auth=client_auth,
30              data=post_data,
31              headers=headers
32          )
33          self.reddit_token = response.json()['access_token']
34 
35      def search(self, query, subreddit=None):
36          self.authorize()
37          headers = {
38              "Authorization": "bearer {token}".format(token=self.reddit_token),
39              "User-Agent": "RedditAdapter/0.1 by Krzysztof Zuraw"
40          }
41          response = requests.get(
42              "https://oauth.reddit.com/r/{subreddit}/"
43              "search.json?q={query}&restrict_sr={restrict}".format(
44                  subreddit=subreddit,
45                  query=query,
46                  restrict='on' if subreddit else 'off'
47               ),
48               headers=headers
49          )
50          return response.json()
51 
52          search_result = []
53          for result in raw_response['data']['children']:
54              search_result.append(result['data']['title'])
55 
56          return search_result

What is happening here? Start from init (line 6) which takes reddit_client_id and reddit_client_secret arguments. There are created by going to apps tab under preferences:

After that click on create new application on the end of the page and you will see something like this:

By clicking create app you will see that reddit_client_secret is secret and reddit_client_id is string under personal use script.

After initialization, there is method called authorize (line 16) which takes care of proper authorization via Oauth2.

Lastly, there is search (line 35) which retrieves JSON response from reddit API from given subreddit or globally from all subreddits.

So how to test it?

First by creating FakeRedditAdapter:

REDDIT_RESPONSE = {
         "kind": "Listing",
         "data": {
             "facets": {},
             "modhash": "",
             "children": [
                 {
                     "kind": "t3",
                     "data": {
                         "domain": "domain",
                         "banned_by": None,
                         "media_embed": {},
                         "subreddit": "django",
                         "selftext_html": None,
                         "selftext": "",
                         "likes": None,
                         "suggested_sort": None,
                         "user_reports": [],
                         "secure_media": None,
                         "link_flair_text": None,
                         "id": "id123",
                         "from_kind": None,
                         "gilded": 0,
                         "archived": False,
                         "clicked": False,
                         "report_reasons": None,
                         "author": "author",
                         "media": None,
                         "score": 20,
                         "approved_by": None,
                         "over_18": False,
                         "hidden": False,
                         "num_comments": 4,
                         "thumbnail": "",
                         "subreddit_id": "id_sub",
                         "hide_score": False,
                         "edited": False,
                         "link_flair_css_class": None,
                         "author_flair_css_class": None,
                         "downs": 0,
                         "secure_media_embed": {},
                         "saved": False,
                         "removal_reason": None,
                         "stickied": False,
                         "from": None,
                         "is_self": False,
                         "from_id": None,
                         "permalink": "/r/django/comments/link",
                         "locked": False,
                         "name": "t3_4b7lzf",
                         "created": 1458511233,
                         "url": "http://url.com",
                         "author_flair_text": None,
                         "quarantine": False,
                         "title": "Post title",
                         "created_utc": 1458482433,
                         "distinguished": None,
                         "mod_reports": [],
                         "visited": False,
                         "num_reports": None,
                         "ups": 20
                     }
                 }
             ],
         "after": None,
         "before": None
         }
     }


class FakeRedditAdapter(object):
    def authorize(self):
        return 'oauth2-authorized-key'

    def search(self, query, subreddit=None):
        search_result = []
        for result in REDDIT_RESPONSE['data']['children']:
            search_result.append(result['data']['title'])
        return search_result

As you can see FakeRedditAdapter returns hardcoded response from reddit API that can be used in test:

import pytest

from tests.utils import FakeRedditAdapter

from reddit_stars.external_api_port import ExternalAPIPort


@pytest.fixture(scope='function')
def reddit_port():
    port = ExternalAPIPort(adapter=FakeRedditAdapter())
    return port


def test_reddit_search(reddit_port):
    assert list(reddit_port.search('test_search')) == ['Post title']

That's all for today. In the next post, I will show how to combine these ports and adapters with django application. Code for this you can find under this repo.

Changes from 23.05.16:

Removing coupling from ExternalAPIPort
Adding new test
Adding word about contracts

Cover image by Creative Magic under CC0.

My first personal project

2016-05-14T10:20:00+02:00

Hi, today I wanted to share my thoughts about project that I made called histmag to kindle.

Why?

I really enjoy reading historical articles over the internet. Very good source of such writings in polish is histmag. One problem is that articles are divided into sub-pages which don't work well while I save them to read later. Then when I'm onto reading I send my read it later articles to kindle. To bypass such inconvenience I started adding every subpage manually to my queue in pocket. As a step into programming path I decided to code my own crawler to crawl through the website and then send the neatly prepared mobi file to my kindle.

What I learned?

After I made my mind I have to choose between many crawling tools available in python like: scrapy, beautifulsoup or lxml.

I have chosen lxml because it was new to me and I really enjoy working with new things. So after reading documentation about lxml I found out that most of scraping is done via XPATH.

Syntax and how to use XPATH is for another post, but I end up with very long queries like:

for elem in parsed_page.xpath('{root}//child::p[not(contains(@class, "article-tags")) '
                                   'and not(contains(@class, "article-info"))] '
                                   '| {root}//a[contains(@href, "author")]'
                                   '| {root}//em '
                                   '| {root}//img '
                                   '| {root}//span'.format(root=self.xpath_root)):

As you can see I take elements that are not paragraphs with class article-tags and article-info. Moreover I pull out the author and em and img tags with contents.

What did I learn about lxml? I really enjoyed using XPATH which is easily provided by this library. I also was able to get python representation of html object retrieved by XPATH. What I don't like is that sometimes installing lxml can cause memory problems as well as you need to install some packages first (system wide).

Another thing that I try was TDD (Test Driven Development). I have used in on code dojo before but no in real project. The first thing that I learned was to use TDD only for production code- I was trying to test code that supposes to be playground and I wasn't able to do this. But after designing how it suppose to work, TDD actually helps. Thanks to that method I have to create such small functions or classes that they will be easily testable. Moreover, I have a lot html in test because it is a web crawler so I created a mocked web pages to be the same as real one. What is more test allows me to change my code and see if there is any place that my program is broken.

After development, I use my test in tox. For using tox I change a bit imports and some pieces of code to let the test be run in a proper way. Learning how to organize imports to let package be installable from setup.py was another new experience.

After parsing html I wanted to generate my own html document with all subpages into. At first, I used dominate library but I then I switched to pyxml. Mainly because I need something easy and simple to just generate html. Pyxml API is very simple:

doc = html.html(
    html.head(
        # python don't allow keyword args with hypens
        html.meta(content="text/html; charset=utf-8", **{"http-equiv": "Content-Type"}),
        _get_tile(html, pages)
        ),
    html.body(
        html.div(_generate_body(html, pages, tempdir), id='article'),
    )
)

This piece of code actually will generate the whole html.

Then thanks to amazon I can just send html to my kindle and it will be automatically converted to mobi. The one problem with this approach is that I cannot send the whole html with images so I have to generate mobi locally and send it to the ebook reader. I did this by calling kindlegen. It works really good but the license stays that I can't redistribute it so enduser have to download it manually. Whole html file with pictures was in the same directory, then I called kindlegen on html file:

proc = subprocess.Popen(
    [
        os.environ.get('KINDLEGEN'),
        os.path.join(html_dir_path, 'histmag.html'),
        '-o',
        output
    ],
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)

To call external binay I used subprocess library.

Lastly, I send a simple email via mailgun. It could be done in better way with my own domain, because right now enduser has to provide mailgun_api_key and email_server for sending messages to kindle.

What can be done differenlty?

I believe that whole crawling mechanism should be redesigned due to being synchronous. Moreover, next time I will try beautifulsoup for getting particular elements from html tree. I would also like to try generating epub file instead of mobi.

Cover picture by Unsplash under CC0 Public Domain.

Mocks and monkeypatching in python

2016-04-24T10:20:00+02:00

Hello, in today's post I will look onto essential part of testing- mocks.

First of all, what I want to accomplish here is to give you basic examples of how to mock data using two tools: mock and pytest monkeypatch.

Why bother mocking?

Some of the parts of our application may have dependencies for other libraries or objects. To isolate behaviour of our parts we need to substitue external dependencies. Here comes the mocking. We mock external API to have certain behaviours such as proper return values that we previously defined.

Mocking function

Let's say we have module called function.py:

def square(value):
    return value ** 2

def cube(value):
    return value ** 3

def main(value):
    return square(value) + cube(value)

Then let's look how these functions are mocked using mock library:

 1  try:
 2      import mock
 3  except ImportError:
 4      from unittest import mock
 5 
 6  import unittest
 7 
 8  from function import square, main
 9 
10 
11  class TestNotMockedFunction(unittest.TestCase):
12 
13      @mock.patch('__main__.square', return_value=1)
14      def test_function(self, mocked_square):
15          # because you need to patch in exact place where function that has to be mocked is called
16          self.assertEquals(square(5), 1)
17 
18      @mock.patch('function.square')
19      @mock.patch('function.cube')
20      def test_main_function(self, mocked_square, mocked_cube):
21          # underling function are mocks so calling main(5) will return mock
22          mocked_square.return_value = 1
23          mocked_cube.return_value = 0
24          self.assertEquals(main(5), 1)
25          mocked_square.assert_called_once_with(5)
26          mocked_cube.assert_called_once_with(5)
27 
28 
29  if __name__ == '__main__':
30      unittest.main()

What is happening here? Lines 1-4 are for making this code compatible between python 2 and 3. In python 3 mock is part of standard library whereas in python 2 you need to install by pip install mock.

In line 13 I patched the square function. But you have to remember to patch it in the same place you use it. For instance, I'm calling square(5) in test itself so I need to patch it in __main__. This is the case if I'm running this by python tests/test_function.py. If I'm using pytest for that I need to patch it like test_function.square.

In lines 18-19, I patch square and cube functions in their module because they are used in main function. The last two asserts come from mock library and are for making sure that mock was called with proper values.

The same can be accomplished using mokeypatching for py.test:

from function import square, main


def test_function(monkeypatch):
    monkeypatch.setattr("test_function_pytest.square", lambda x: 1)
    assert square(5) == 1

def test_main_function(monkeypatch):
    monkeypatch.setattr('function.square', lambda x: 1)
    monkeypatch.setattr('function.cube', lambda x: 0)
    assert main(5) == 1

As you can see I'm using monkeypatch.setattr for setting up return value for given functions. I'm still need to monkeypatch it in proper place: test_function_pytest and function.

Mocking classes

I have module called square:

import math

class Square(object):
     def __init__(radius):
         self.radius = radius

     def calculate_area(self):
         return math.sqrt(self.radius) * math.pi

And mocks using standard lib:

 1 try:
 2     import mock
 3 except ImportError:
 4     from unittest import mock
 5 
 6 import unittest
 7 
 8 from square import Square
 9 
10 
11 class TestClass(unittest.TestCase):
12 
13     @mock.patch('__main__.Square') # depends in witch from is run
14     def test_mocking_instance(self, mocked_instance):
15         mocked_instance = mocked_instance.return_value
16         mocked_instance.calculate_area.return_value = 1
17         sq = Square(100)
18         self.assertEquals(sq.calculate_area(), 1)
19 
20 
21     def test_mocking_classes(self):
22         sq = Square
23         sq.calculate_area = mock.MagicMock(return_value=1)
24         self.assertEquals(sq.calculate_area(), 1)
25 
26     @mock.patch.object(Square, 'calculate_area')
27     def test_mocking_class_methods(self, mocked_method):
28         mocked_method.return_value = 20
29         self.assertEquals(Square.calculate_area(), 20)
30 
31 if __name__ == '__main__':
32     unittest.main()

At line 13 I patch class Square (again be aware if you run this test using pytest or standard way). Lines 15 and 16 presents mocking instance; at first mocked_instance is mock object which by default returns another mock and to these mock.calculate_area I add return_value 1. In line 23 I'm using MagicMock which is normal mock class except it also retrieves magic methods from given object. Lastly I use patch.object to mock method in Square class.

The same using pytest:

try:
    from mock import MagicMock
except ImportError:
    from unittest.mock import MagicMock

from square import Square


def test_mocking_class_methods(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square.calculate_area', lambda: 1)
    assert Square.calculate_area() ==  1


def test_mocking_classes(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square', MagicMock(Square))
    sq = Square
    sq.calculate_area.return_value = 1
    assert sq.calculate_area() ==  1

The issue here is with test_mocking_class_methods which works well in python 3 but not in python 2. Right now I don't have clear answer to this so if you can help I appreciate this!

All examples can be found under this repo.

References:

Cover picture by Chiltepinster under CC BY-SA 3.0.

My first workshop experience

2016-04-17T10:20:00+02:00

Hello ! In today's blog post I present my experiences about some workshop that I have the pleasure to conduct. Later as I promised I will present exercises and their answers.

When I joined STX Next I saw an opportunity to teach people how to write code. I am not experienced guy yet but I believe that teaching others can have great benefits for me. So when I heard that 2016 edition of workshops called PHP in Wrocław was announced I decided to take part. By the way, PHP means Python has Power, not any affiliations with elephants.

Obligatory selfie from workshop.

What is exactly this workshop about? In about 7 hours my task was to present basics of python. I know that it sounds a little bit overwhelmed but I believe that is a nice start to have.

So what is exactly on agenda?

loops, data types, declaration of functions, module importing, common operations on data, debugging
list/dict comprehensions, generators and iterators
pep8, pylint, unittests
writing algorithm

As you can see almost all necessary topics are covered. What I really enjoyed was questions from the audience. People ask me a variety of questions from how to do something to how I started programming. Moreover, every 1,5h there was the break on network coffee and one for pizzas!

Handwritten agenda with snakes.

For me, it was a great experience. I was able to teach others and learn from it!

Attendees at work.

At the end of the workshop, I have question about why is that possible in python

>>> def func(a,b,c):
...     print(a,b,c)
>>> func(c=1,b=2, *(13,))
(13, 1, 2)

After some investigation, I found out that positional arguments are processed before keyword ones. So, in this case, unpacking tuple will assign 13 to a. More on this can be found in PEP 3102.

Update 18.04.16

I almost forget to include source code from this workshop. You can find code and presentation under this github repo.

What is more, below I present my solutions for tasks that were on PHP.

Solutions for part one:

Part two:

Last part:

Cover picture by Petar Milošević under CC BY-SA 4.0.

Automatic blog deployment with TravisCI

2016-04-09T10:20:00+02:00

Hello! Today I will write about how to automate blog deployment using TravisCI with bash scripts.

When I first started writing blog using pelican my workflow look as follows:

I open one terminal window and go under output just for serving static files using python -m SimpleHTTPServer
Then I open another terminal and if I wanted to generate my changes in the post I have to do pelican content -s pelicanconf.py -t pelican-clean-blog
After this I open my browser window and go to 127.0.0.1:8000 to see my changes
To publish content I push it to responsible branch

After some time, I know that there must be the way to automate such boring stuff. So I started looking around my pelican folder and I found Makefile.

This makefile is used to manage all these things that I am doing manually into bash script so it is perfect!

Let's look into content of makefile, what is worth seeing is command devserver:

devserver:
ifdef PORT
        $(BASEDIR)/develop_server.sh restart $(PORT)
else
        $(BASEDIR)/develop_server.sh restart
endif

So whenever you specify $(PORT) or not the same command will be used: develop_server.sh restart. So what we have in restart?

elif [[ $1 == "restart" ]]; then
 shut_down
 start_up $port

and start_up:

 1 function start_up(){
 2   local port=$1
 3   echo "Starting up Pelican and HTTP server"
 4   shift
 5   echo $PELICANOPTS
 6   $PELICAN --debug --autoreload -r $INPUTDIR -o $OUTPUTDIR -s $CONFFILE $PELICANOPTS &
 7   pelican_pid=$!
 8   echo $pelican_pid > $PELICAN_PID
 9   cd $OUTPUTDIR
10   $PY -m pelican.server $port &
11   srv_pid=$!
12   echo $srv_pid > $SRV_PID
13   cd $BASEDIR
14   sleep 1
15   if ! alive $pelican_pid ; then
16     echo "Pelican didn't start. Is the Pelican package installed?"
17     return 1
18   elif ! alive $srv_pid ; then
19     echo "The HTTP server didn't start. Is there another service using port" $port "?"
20     return 1
21   fi
22   echo 'Pelican and HTTP server processes now running in background.'
23 }

In line 6 is normal pelican command like:

pelican --debug --autoreload -r personal_blog/content -o personal_blog/output -s personal_blog/pelicanconf.py

Is is running pelican in debug mode with auto reload taking post content from personal_blog/content. It generate output to personal_blog/output with settings from personal_blog/pelicanconf.py.

After that script enters the $OUTPUTDIR and starts the pelican server by $PY -m pelican.server $port.

But what if I wanted to publish my work into GitHub Pages? There is command for that in Makefile

github: publish
       ghp-import -m "Generate Pelican site" -b $(GITHUB_PAGES_BRANCH) $(OUTPUTDIR)
       git push origin $(GITHUB_PAGES_BRANCH)

It is using ghp-import. Ghp-import is python application that allows pushing content to GitHub branches. Why is that important? The GitHub pages are organized in that way:

Type of site	Page address	Publishing branch
User Pages site	`username.github.io`	master
Project Pages site owned by a user account	`username.github.io/projectname`	gh-pages

Table taken from GitHub Pages help.

As you can see to publish content on GitHub Pages you have to push either to master and you can see is under krzysztofzuraw.github.io or to gh-pages and go to krzysztofzuraw.github.io/personal-blog. As I wanted building process of my blog to be automated I choose second option: on master I will have all settings and content that is be used to render HTML will be on gh-pages branch.

So to automate I need TravisCI. After connecting service to your GitHub account, let's enable it for given project:

And on the next screen enable repository for what you want to trigger Travis. After this click on wheel next to the name of the repo. In next page setup two environmental variables:

GH_TOKEN is your token generated from this page. Click there on Generate new token and in next screen make sure you add token description and scope for public_repo:

After generating token add it to Travis. The another variable in Travis TRAVIS_REPO_SLUG is in the form of username/project-name so in my case it was krzysztofzuraw/personal-blog. After setting up these variables add the following file .travis.yml to the main root of your pelican blog project with following content:

language: python
branches:
  only:
  - master
install:
- pip install pelican
- pip install ghp-import
- pip install pelican_gist
script:
- make publish github

It tells Travis to install pelican, ghp-import and pelican_gist (plugin for gists in pelican) and then it runs make publish github.

To make it work I need to change one thing in Makefile:

github: publish
        ghp-import -m "Generate Pelican site" -b $(GITHUB_PAGES_BRANCH) $(OUTPUTDIR)
        git push -fq https://${GH_TOKEN}@github.com/$(TRAVIS_REPO_SLUG).git $(GITHUB_PAGES_BRANCH)

The change is that I push to repo using GH_TOKEN not username so Travis is able to do it.

So right now when I push my commits to master branch the Travis starts building my blog!

References:

Publish your Pelican blog on Github pages via Travis-CI.

Cover image by Manjith Kainickara under CC BY-SA 2.0.

Moving blog to pelican

2016-04-03T10:20:00+02:00

Hello! Spring has come so I decided that I need to changes- so I change my blog engine to pelican.

This blog will be about small compassion between three static site generators and why I choose pelican.

What are static site generators?

Basically they are engines that generates web sites using only static content. So there is no need for any backend or database at all. Most of static site generators allow writing posts in markdown or rst formats.

Nikola

When I started writing this blog I used a Nikola. What is it? It python powered static site generator. I has a lot of features 'out of the box':

Incremental builds. That means only modified content is regenerated to html.
Support for comments. All you need to do is to set up option.
Support for IPython Notebooks.
Automatic deployment to Github pages.
Extensions to RST format.

I really enjoyed working with Nikola, but I changed it to another engine because I was overwhelmed by a lot of options that I don't need. Also community is not huge but it is very helpful.

Jekyll

Then I moved to Jekyll. This is ruby static site generator. What I like about jekyll is support for github pages. Also is the most popular static site generator with a lot of community and themes. What I don't like is it doesn't have support for tags out of the box. I know that it can be done by some ruby code, but I like this feature to be build in. Moreover while you 'upload' your entire catalog with jekyll configuration your custom plugins don't work. You have to stick to these provided in github.

Pelican

So I decided to move to pelican. What I like so far is that I have some more freedom to customize engine than in nikola and I have out of the box support for tags. But there is thing: pelican only generates content. So you have to serve it somehow. To do this I run simple python server. I don't like this approach- I want it to be automatic. I will write next blog post about how to do this. Stay tuned!

Cover image by Manjith Kainickara under CC BY-SA 2.0.

Django + Celery & Rabbit - part four

2016-03-19T10:20:00+01:00

This is the fourth part of Celery and RabbitMQ in Django series. Today I will fix minor bugs and sum up this series.

Audio File detail view

This is the first bug that I wanted to tackle. Did you remember my last post ? At the end there was a bug show below:

The problem was after successful upload django redirect to detail view of uploaded file. And in HTML template of this view, it expects that ac3_file will be there but FFmpeg still is transcoding it. So I came up with solution:

1.First, I added new field to AudioFile model called was_processed to indicate whenever file has been processed:

class AudioFile(models.Model):
    # ....
    was_processed = models.BooleanField(default=False)

By default, this field has value False.

2.Then in my task I added signal handler that ran after every task:

from celery.signals import task_postrun

@task_postrun.connect
def task_executed_handler(sender=None, body=None, **kwargs):
    audio_file = AudioFile.objects.get(id=kwargs['args'][0])
    audio_file.was_processed = True
    audio_file.save()

In this handler id of AudioFile object is taken from kwargs. After retrieving certain file from the database, the flag was_processed is set.

3.Lastly, in my audiofile_detail.html I added this code:

{% if object.was_processed %}
    <ul><a href="{{ object.ac3_file.url}}">Ac3 File</a></ul>
    <ul><a href="{{ object.ogg_file.url}}">Ogg File</a></ul>
    <ul><a href="{{ object.wav_file.url}}">Wav File</a><ul>
{% endif %}

So right now when I upload file I get:

Your browser does not support the video tag.

Logging

Right now everything works great but what if something goes wrong? To make sure that I will be able to find the issue I need logging. This is especially valuable for celery because Django doesn't show output from Celery as it is a different application. So to setup basic logging I need to add only a few things in tasks.py:

from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)


@app.task
def transcode_mp3(mp3_id):
    # ...

    logger.debug(
        'Created output files: %s, %s, %s.',
        ogg_output_file_path,
        ac3_output_file_path,
        wav_output_file_path
    )

    logger.info('Started transcoding.')
    # transcoding here
    logger.info('End of transcoding.')

    # rest of program ...

Thanks to that I can see in my console:

[2016-03-19 09:55:07,184: INFO/Worker-4] taskapp.tasks.transcode_mp3[b6ca93d4-e58c-496f-b8e5-4ba493b8a92a]: Started transcoding.
# transcoding ...
[2016-03-19 09:55:11,837: INFO/Worker-4] taskapp.tasks.transcode_mp3[b6ca93d4-e58c-496f-b8e5-4ba493b8a92a]: End of transcoding.

Summary

This was the last post of Django Celery Rabbit series. I made basic transcoder application that uses FFmpeg, Django, Celery and RabbitMQ. I learned quite a bit about how celery works with rabbitmq and django. Thanks to that I stumbled upon some useful blog posts. I also see some issues with my solutions. For instance todays AudioFile detail view. I'm thinking about other ways to solve this problem because right now I need 2 operations on database for one file. Maybe you know solution to this? I'm really keen to hear about your view on this issue or other comments so feel free to write comments or send me an email! Thanks to all people who give me feedback- I really appreciate this! Code for this series can be found on github.

Cover image that presents queue by Aykapog under CC0.

Django + Celery & Rabbit - part three

2016-03-12T10:20:00+01:00

This is a third part of Celery and RabbitMQ in Django series. Today I will be building the Celery and RabbitMQ stack.

In previous 2 posts: Django + Celery & RabbitMQ part one and Django + Celery & RabbitMQ part two I have made simple application and played with ffmpeg.

First: why we need Celery? Imagine that user upload mp3 file to the application and then in form validation the file is transcoded to other formats. The problem is that user will have to wait for the end of a task. So user sends a request. Then he waits for ffmpeg to transcode uploaded file to different format and then sends the response back. At first glance, it may look correct. But imagine that there are big files to transcode or there is a lot of formats to transcode. The user will have to wait a lot of time. To avoid this I will use celery task with rabbitmq broker to provide transcoding in the background.

So what is exactly celery? From the docs:

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

What does it mean? It is task manager that handle messages (tasks) in whenever form you like. Task is a function that could calculate something or handle some logic or maybe transcode files. But celery without message broker is useless. Celery support a lot of message brokers, but RabbitMq is supported out of the box so I will use this service. You may ask what is RabbitMq? It is broker- translates a message from the sender (django application) to reciever (celery).

So go to code:

First install rabbitmq with plugin that displays its status:

$ sudo apt-get install rabbitmq-server
$ sudo rabbitmq-plugins enable rabbitmq_management
$ sudo rabbitmqctl stop
$ sudo invoke-rc.d rabbitmq-server start

Add admin user to rabbitmq-server:

$ sudo rabbitmqctl add_user admin admin
$ sudo rabbitmqctl set_user_tags admin administrator
$ sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

Right now you can go to localhost:15672 to see:

Now it's time to get celery working. First I will create new folder called taskapp where I will be putting my celery configuration:

├── audio_transcoder
├── taskapp
│   ├── celery.py
│   ├── __init__.py
│   └── tasks.py
│ # some other files ...

In celery.py are configs:

from __future__ import absolute_import

from celery import Celery

app = Celery('transcoder',
             broker='amqp://admin:admin@localhost//',
             include=['taskapp.tasks'])

if __name__ == '__main__':
    app.start()

The first line is for backward compatibility with python2. In app configuration I specify: name of application- transcoder, url where broker will be listening with credentials- broker='amqp://admin:admin@localhost//' and files containing tasks- include=['taskapp.tasks'].

Then in tasks.py I added task itself:

from __future__ import absolute_import
import os
import os.path
import subprocess

from taskapp.celery import app
from audio_transcoder.models import AudioFile
import config.settings as settings

@app.task
def transcode_mp3(mp3_id):
    audio_file = AudioFile.objects.get(id=mp3_id)
    input_file_path = audio_file.mp3_file.path
    filename = os.path.basename(input_file_path)

    ogg_output_file_name = os.path.join('transcoded', '{}.ogg'.format(filename))
    ogg_output_file_path = os.path.join(settings.MEDIA_ROOT, ogg_output_file_name)

    ac3_output_file_name = os.path.join('transcoded', '{}.ac3'.format(filename))
    ac3_output_file_path = os.path.join(settings.MEDIA_ROOT, ac3_output_file_name)

    wav_output_file_name = os.path.join('transcoded', '{}.wav'.format(filename))
    wav_output_file_path = os.path.join(settings.MEDIA_ROOT, wav_output_file_name)

    if not os.path.isdir(os.path.dirname(ogg_output_file_path)):
        os.makedirs(os.path.dirname(ogg_output_file_path))

    subprocess.call([
            settings.FFMPEG_PATH,
            '-i',
            input_file_path,
            ogg_output_file_path,
            ac3_output_file_path,
            wav_output_file_path
        ]
    )

    audio_file.ogg_file = ogg_output_file_name
    audio_file.ac3_file = ac3_output_file_name
    audio_file.wav_file = wav_output_file_name
    audio_file.save()

What I got there? Let's start with transcode_mp3 function. It has @app.task decorator to indicate for celery that it is its task. The argument is mp3_id. After getting the id of newly uploaded file this task gets audio_file model from the database and retrieve path to the uploaded mp3 file. Then it generated file names and paths for every format: ogg, wav and ac3. Right after it checks whenever folder transcoded under media is present. Calling subprocess is basically the same as calling ffmpeg -i mp3_file.mp3 ogg_file.ogg ac3_file.ac3 wav_file.wav. At the end task saves the paths to outputs to database.

The tasks itself is called in views:

from taskapp.tasks import transcode_mp3

class UploadAudioFileView(FormView):
    # ...
    def form_valid(self, form):
    # ...
    audio_file.save()
    transcode_mp3.delay(audio_file.id)
    # ...

Everything works as expected and I added redirection to detail view of audio file after successful upload. The problem is that transcode is not so fast as is redirection. Because of that similar error is shown:

This and other bugs and small improvements will be fixed and added in last post of this series in next week. Thanks for reading! I really appreciate your feedback so please comment or write email. The code is available on this github repo.

Cover image that presents queue by Aykapog under CC0.

Django + Celery & Rabbit - part two

2016-03-05T10:20:00+01:00

Hello, today post is the second one in series about Celery in Django. Today I write about ffmpeg.

In my application transcoding will be performed by ffmpeg. Why is that?

First of all, it's free & open source. So I don't have to pay to use it and I like using the open source tools. Second thing is that ffmpeg is more than enough for my task. It has a lot of features including recording, streaming and transcoding both video and audio.

So how to use it? For this project, the ffmpeg needs to be compiled with libmp3lame for mp3 files encoding, libvorbis for ogg format and enabled non-free for ac3. There is guide on ffmpeg site how to compile ffmpeg build. Also, make sure that you have cookies and tea with you while installation can take even 30 minutes.

After everything is setup all you need to do is:

$ ffmpeg -i mp3_file.mp3 wav_file.wav
$ ffmpeg -i mp3_file.mp3 ogg_file.ogg
$ ffmpeg -i mp3_file.mp3 ac3_file.ac3

The code above will transcode mp3 files into provided format. In my application one celery task will be called with subprocess, then another celery task will save the output of the first task to django model. This is all for today post but next week I will be building the celery tasks and integrating them with django.

Cover image that presents queue by Aykapog under CC0.

Django + Celery & Rabbit - part one

2016-02-26T10:20:00+01:00

Hello, today post is first one in series about Celery in Django application and how to use it.

What you gain after reading such series?

How to integrate celery + rabbitmq in basic Django application.

So what will be our basic application?

It is planned to be web service where you can upload mp3 file and then have them transcoded into ogg, wav and ac3 files. I wanted to make easy and in the same time sophisticated project.

Recently I started reading an excellent book called Two Scoops of Django: Best Practices for Django 1.8. The book is about what to do and what not to do for your Django project. I feel some opportunity to try advice from the book in real project. I will start from project layout: normally you have something like this:

$ tree trancoder

trancoder
├── audiotranscoder
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── trancoder
    ├── manage.py
    └── trancoder
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

But in the book authors suggest that it can be changed to something like this:

$ tree transcoder

transcoder
├── README.rst
├── requirements.txt
└── transcoder
    ├── audio_transcoder
    │   ├── admin.py
    │   ├── apps.py
    │   ├── forms.py
    │   ├── __init__.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── config
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── manage.py
    ├── media
    └── templates
        ├── base.html
        ├── home.html
        └── upload.html

After working some time with this layout I have to say that it's very responsive and good structured in my opinion. But there is need for changes in 2 files for Django to be able recognizing where to look up for settings and WSGI modules:

manage.py:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

settings.py:

# rest of settings file
ROOT_URLCONF = 'config.urls'

WSGI_APPLICATION = 'config.wsgi.application'

MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

MEDIA_URL = '/media/'

The transcoder need file to transcode so I created a model, form and view for mp3 file.

audio_transcoder/models.py:

import uuid

from django.db import models


def unique_file_path(instance, filename):
    new_file_name = uuid.uuid4()
    return str(new_file_name)

class AudioFile(models.Model):
    name = models.CharField(max_length=100, blank=True)
    mp3_file = models.FileField(upload_to=unique_file_path)
    ogg_file = models.FileField(blank=True, upload_to=unique_file_path)
    wav_file = models.FileField(blank=True, upload_to=unique_file_path)
    ac3_file = models.FileField(blank=True, upload_to=unique_file_path)

    def __str__(self):
        return self.name

To avoid filename duplication of uploaded files I changed their names to be unique. the upload_to argument takes function unique_file_path which will generate unique name. This function has to take 2 arguments: instance and filename.

audio_transcoder/forms.py:

from django.forms import ModelForm
from .models import AudioFile


class AudioFileFrom(ModelForm):
    class Meta:
        model = AudioFile
        fields = ['name', 'mp3_file']

Here I have used ModelForm which is the easiest way to generate the form for given model.

audio_transcoder/views.py:

from django.views.generic.edit import FormView
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse

from .forms import AudioFileFrom
from .models import AudioFile


class UploadAudioFileView(FormView):
    template_name = 'upload.html'
    form_class = AudioFileFrom


    def form_valid(self, form):
        audio_file = AudioFile(
            name=self.get_form_kwargs().get('data')['name'],
            mp3_file=self.get_form_kwargs().get('files')['mp3_file']
        )
        audio_file.save()

        return HttpResponseRedirect(self.get_success_url())

    def get_success_url(self):
        return reverse('home')

And in UploadAudioFileView in form validation I took name and mp3_file from form submitted by user and save then in corresponding model. I have hard time figuring out where to put saving model. At first I wanted to work as generic CBV so no form_valid I just pass additional argument to class: success_url but it didn't save audioFile. Also moving form_valid to AudioFileFrom didn't help. After some more research I found this GoDjango tutorial. If you have better way to do this please let me know!

The code that I have made so far is available on github. Stay tuned for next blog post from this series.

Cover image that presents queue by Aykapog under CC0.

Django cookiecutter

2016-02-19T10:20:00+01:00

Have you ever wanted to automate all these boring things that you have to do while setting up new django project? Like writing proper settings, setting up whole folder structure, adding docs, readmes, setup.py etc? There is app that can do these mundane actions for you: Cookiecutter Django.

Q: What you get after reading this post?

A: Basic understanding how you can automate boring tasks and get exciting ones done!

Let's start with cookiecutter. What exactly is it? It's CLI tool for automatically generating projects from given template. What do I mean by that?

Suppose that when you make new python project you want certain things to be setup the same way every time. For instance: setup.py or docs folders. Thanks to cookiecutter it's possible to automate these things so every time you make the new project the boring stuff is already made for you.

Django Cookiecutter is made especially for django applications. Based on GitHub README it provides:

For Django 1.9

Renders Django projects with 100% test coverage

Twitter Bootstrap v4.0.0 - alpha

End-to-end via Hitch

AngularJS

12-Factor based settings via django-environ

Optimized development and production settings

Registration via django-allauth

Comes with custom user model ready to go.

Grunt build for compass and livereload

Basic e-mail configurations for sending emails via Mailgun

Media storage using Amazon S3

Docker support using docker-compose for development and production

Procfile for deploying to Heroku

And some additional features:

Serve static files from Amazon S3 or Whitenoise

Configuration for Celery

Integration with MailHog for local email testing

Integration with Sentry for error logging

Integration with NewRelic for performance monitoring

Integration with Opbeat for performance monitoring

So it can make a huge improvement in initial project configuration. Let's try this out!

After installation via pip by:

$ pip install cookiecutter

Then after typing:

$ cookiecutter https://github.com/pydanny/cookiecutter-django.git

And answering all these questions:

project_name [project_name]: blog_cookiecutter
repo_name [blog_cookiecutter]: blog-cookiecutter
author_name [Your Name]: Krzysztof Żuraw
email [Your email]: kz@example.com
description [A short description of the project.]: This is example of cookiecutter django usage
domain_name [example.com]: example.com
version [0.1.0]: 0.0.1
timezone [UTC]: UTC
now [2016/01/29]: 2016/02/18
year [2016]:
use_whitenoise [y]: n
use_celery [n]: y
use_mailhog [n]: n
use_sentry [n]: y
use_newrelic [n]: n
use_opbeat [n]: n
windows [n]: n
use_python2 [n]: n
Select open_source_license:
1 - MIT
2 - BSD
3 - Not open source
Choose from 1, 2, 3 [1]: 1

You got the simple django application with celery, sentry, tests, grunt, less and docker.

Folder structure looks as follows:

$ tree . -a -L 3
.
└── blog-cookiecutter
    ├── app.json
    ├── blog-cookiecutter
    │   ├── contrib
    │   ├── __init__.py
    │   ├── static
    │   ├── taskapp
    │   ├── templates
    │   └── users
    ├── compose
    │   ├── django
    │   └── nginx
    ├── config
    │   ├── __init__.py
    │   ├── settings
    │   ├── urls.py
    │   └── wsgi.py
    ├── CONTRIBUTORS.txt
    ├── .coveragerc
    ├── dev.yml
    ├── docker-compose.yml
    ├── Dockerfile
    ├── Dockerfile-dev
    ├── .dockerignore
    ├── docs
    │   ├── conf.py
    │   ├── deploy.rst
    │   ├── docker_ec2.rst
    │   ├── index.rst
    │   ├── __init__.py
    │   ├── install.rst
    │   ├── make.bat
    │   └── Makefile
    ├── .editorconfig
    ├── env.example
    ├── .gitattributes
    ├── .gitignore
    ├── Gruntfile.js
    ├── install_os_dependencies.sh
    ├── install_python_dependencies.sh
    ├── LICENSE
    ├── manage.py
    ├── package.json
    ├── Procfile
    ├── .pylintrc
    ├── README.rst
    ├── requirements
    │   ├── base.txt
    │   ├── local.txt
    │   ├── production.txt
    │   └── test.txt
    ├── requirements.apt
    ├── requirements.txt
    ├── runtime.txt
    ├── setup.cfg
    ├── tests
    │   ├── all.settings
    │   ├── base.yml
    │   ├── ci.settings
    │   ├── engine.py
    │   ├── hitchreqs.txt
    │   ├── register-and-log-in.test
    │   ├── system.packages
    │   └── tdd.settings
    └── .travis.yml

So as this application follows the 12-Factor application guidelines most of django settings variables are set in an environment so it's good to run this in docker.

To install docker on ubuntu type:

$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates
$ sudo sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

Now it's time to add entry in /etc/apt/sources.list.d/docker.list for Ubuntu 14.04

deb https://apt.dockerproject.org/repo ubuntu-trusty main

And run:

$ sudo apt-get update
$ sudo apt-get install linux-image-extra-$(uname -r)
$ sudo apt-get install docker-engine

Then verify if everything is installed:

$ sudo service docker start $ sudo docker run hello-world

Now you have Docker! But to be able to use docker in cookiecutter-django project there is need for installing additional package called docker-compose. It allows to run multiple docker containers.

Installation is simple:

$ sudo pip install docker-compose
$ docker-compose --version
docker-compose version 1.6.0, build d99cad6

So it's time to fire up the installation:

$ sudo docker-compose -f dev.yml build
$ sudo docker-compose -f dev.yml up -d

-f flag means that we specify which yml file is taken for configuration (by default it is docker-compose.yml) and -d flag in the second command is for detached mode.

Now it's time to run basic django commands to migrate data to database and to create superuser:

$ sudo docker-compose -f dev.yml run django python manage.py makemigrations
$ sudo docker-compose -f dev.yml run django python manage.py migrate
$ sudo docker-compose -f dev.yml run django python manage.py createsuperuser

Then go to the localhost:8000 and you can see that Django application works!:

And that's it! You got the working project made in few minutes. If you found this post helpful please share it with your friends.

Resources:

Cover image by Kimberly Vardeman under CC BY 2.0.

Django Haystack and Elasticsearch- part two

2016-02-12T10:20:00+01:00

Hello! This is the second part of Django Haystack and Elasticsearch series. First you can find here.

Now it's time to install and elasticsearch. On ubuntu you can do it as follows:

1.First install java-8

$ sudo apt-get install python-software-properties -y
$ sudo add-apt-repository ppa:webupd8team/java -y
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer -y

2.Verify if it's properly installed

$ java -version
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

3.Now install elasticsearch itself

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb http://packages.elastic.co/elasticsearch/1.7.5/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elk.list
$ sudo apt-get update && sudo apt-get install elasticsearch -y
$ sudo service elasticsearch start

4.Verify if elasticsearch is running

$ curl http://localhost:9200
{
  "status" : 200,
  "name" : "May \"Mayday\" Parker",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.5",
    "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",
    "build_timestamp" : "2016-02-02T09:55:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Now it's time to install to more python packages:

$ pip install django-haystack==2.4.1
$ pip install elasticsearch

After adding them to INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'haystack',
    'persons'
]

and setup up connection in settings.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': 'localhost:9200',
        'INDEX_NAME': 'haystack',
    },
}

Create file called search_indexes.py in your django application folder (django_project/django_app/search_indexes.py):

from haystack import indexes

from .models import Person


class PersonIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    first_name = indexes.CharField(model_attr='first_name')
    last_name = indexes.CharField(model_attr='last_name')
    gender = indexes.CharField(model_attr='gender')
    email = indexes.CharField(model_attr='email')
    ip_address = indexes.CharField(model_attr='ip_address')

    def get_model(self):
        return Person

In this file we declare indexes with will be created in elasticsearch. The first field text indicates which field is primary to be searched within. This field can be named wherever you wanted but the convention is to name it text. There is only one field in each index with document=True argument. Another argument use_template=True tells haystack to use a template for building document for an index. This document is usually located under django_project/templates/search/indexes/django_app/index_name.txt. And for this data looks like this:

{{ object.title }}
{{ object.first_name }}
{{ object.last_name }}
{{ object.gender }}
{{ object.email }}
{{ object.ip_address }}

Don't forget to add this django_project/templates/ to TEMPLATES in settings.py:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR,'templates/'),],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

After this add haystack.urls to urls.py:

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^search/', include('haystack.urls'))
]

Now it's time to create search.html in django_project/templates/search/search.html:

{% extends 'base.html' %}

{% block content %}

<h2>Person search</h2>

    <form method="get" action="." class="form" role="form">
        {{ form.non_field_errors }}
        <div class="form-group">
                {{ form.as_p }}
        </div>
        <div class="form-group">
            <input type="submit" class="btn btn-primary" value="Search">
        </div>

        {% if query %}
            <h3>Results</h3>
            <div>
            <table class="table table-striped table-bordered" cellspacing="0" id='result_table'>
                <thead>
                    <tr>
                        <th>First name</th>
                        <th>Last name</th>
                        <th>Gender</th>
                        <th>Email</th>
                        <th>IP address</th>
                </thead>
                <tbody>
                    {% for result in page.object_list %}
                        <tr>
                            <td>{{ result.first_name }}</td>
                            <td>{{ result.last_name }}</td>
                            <td>{{ result.gender }}</td>
                            <td>{{ result.email }}</td>
                            <td>{{ result.ip_address}}</td>
                        </tr>
                    {% empty %}
                        <tr>No results found.</tr>
                    {% endfor %}
                </tbody>
            <table>
            </div>
        {% endif %}
    </form>
{% endblock content %}

{% block extrajs %}
<script>
$(document).ready(function() {
    $('#result_table').DataTable({
        "searching": false
    });
} );
</script>
{% endblock %}

This is basic template for searching. I added DataTable just for better appearance.

Before we can search let's rebuild the indexes by:

$ ./manage.py rebuild_index

Now try to search something in elasticsearch itself to see if the documents are there:

$ curl -XGET http://localhost:9200/haystack/_search?pretty=true&q=first_name:Scott
{
"took" : 3,
 "timed_out" : false,
 "_shards" : {
   "total" : 5,
   "successful" : 5,
   "failed" : 0
 },
 "hits" : {
   "total" : 1000,
   "max_score" : 1.0,
   "hits" : [ {
     "_index" : "haystack",
     "_type" : "modelresult",
     "_id" : "persons.person.1",
     "_score" : 1.0,
     "_source":{"django_ct": "persons.person", "last_name": "Harrison", "ip_address": "38.84.45.160", "email": "rharrison0@linkedin.com", "first_name": "Russell", "gender": "Male", "text": "\nRussell\nHarrison\nMale\nrharrison0@linkedin.com\n38.84.45.160\n", "id": "persons.person.1", "django_id": "1"}
   },
   # rest of results here...
}

Then running server and going to /search will be shown as follows:

Your browser does not support the video tag.

And that's all. We got working search! You can find the repo on github. If you feel that this post was valuable please comment or send me email. Thanks!

Resources:

Cover image by Lets Go Out Bournemouth and Poole under CC BY 2.0.

Django Haystack and Elasticsearch- part one

2016-02-06T10:20:00+01:00

Hello! Today blog post is about Django Haystack and how to integrate it quickly with Elasticsearch.

First after creating django project (At beginning of 2016 django-haystack don't work properly with django 1.9 so I used 1.8.9 version) and making new app let's add models:

from django.db import models

GENDER_CHOICES = (
    ('Male', 'Male'),
    ('Female', 'Female')
)

class Person(models.Model):
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    gender = models.CharField(max_length=10, choices=GENDER_CHOICES)
    email = models.CharField(max_length=100)
    ip_address = models.CharField(max_length=100)

    def __str__(self):
        return '{first_name} {last_name}'.format(first_name=self.first_name, last_name=self.last_name)

And register model to the admin site. Don't forget about adding created app to settings.py and making manage.py makemigrations and manage.py migrate after it:

from django.contrib import admin

from .models import Person

admin.site.register(Person)

Then create simple script wich will load a data from JSON to the database. This JSON data is randomly generated data from this webpage. Call it load.py and place in your django application folder.

# coding=utf-8
import os
import json

from .models import Person

DATA_FILE = os.path.join(
os.path.dirname(
    os.path.dirname(
    os.path.dirname(__file__))),
    'MOCK_DATA.json'
)


def run(verbose=True):
with open(DATA_FILE) as data_file:
    data = json.load(data_file)
    for record in data:
        Person.objects.create(
            first_name=record['first_name'],
            last_name=record['last_name'],
            gender=record['gender'],
            email=record['email'],
            ip_address=record['ip_address'])
        print(record)

This script looks for file MOCK_DATA.json. Then based on fields on this JSON loads data to the django application. You can run this by manage.py shell and then:

>>> from django_app import load
>>> load.run()
{'ip_address': '86.24.99.139', 'gender': 'Female', 'first_name': 'Christine', 'last_name': 'Cunningham', 'email': 'ccunninghamrq@howstuffworks.com'}
{'ip_address': '250.20.255.181', 'gender': 'Male', 'first_name': 'Scott', 'last_name': 'Hanson', 'email': 'shansonrr@utexas.edu'}
# rest of the records

That's all for this week. In next post, I will concentrate on how to setup Elasticsearch and django-haystack.

Cover image by Lets Go Out Bournemouth and Poole under CC BY 2.0.

Python class @decorators

2016-01-29T10:20:00+01:00

Today post will be about syntactic sugar of python language- decorators.I will concentrate on class decorators.

Let's start with basic example of decorator defined by class in example_1:

class decorator(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args):
        print('Called {func} with args: {args}'.format(func=self.func.func_name,
                                                       args=args))
        return self.func(*args)

@decorator
def func(x,y):
    return x,y

if __name__ == '__main__':
    func(1,2)

So running it results in:

$ python example_1.py
Called func with args: (1, 2)

But there is another special method that can be used in such cases: __get__. This is used for example in implementation of cached_property decorator in django.

Let's look onto example_2:

class property_(object):
    def __init__(self, func):
        self.func = func
        self.name = func.__name__

    def __get__(self, instance, cls):
        print(
            'Called property from {instance} ',
            'of {klass}'.format(instance=instance, klass=cls)
        )
        return self.func(instance)

    def __set__(self, obj, value):
        print(
            'Setting up {value} '
            'for {obj}'.format(value=value, obj=obj)
        )
        [setattr(obj, k, v) for k, v in value.items()]


class Apple(object):

    @property_
    def get_color(self):
        print('Accessing get_color property')
        return 'red'

if __name__ == '__main__':
    apple = Apple()
    print(apple.get_color)
    apple.get_color = {'shape':'triangle'}
    print(apple.shape)

What is happening here? Instead of implementing __call__ we got access to get the certain attribute. It's useful when we want to implement logic to e.g properties. Here I implemented full descriptor. Running this example results is this output:

$ python example_2.py
Called property from <__main__.Apple object at 0x7ff05de056d0> of <class '__main__.Apple'>
Accessing get_color property
red
Setting up {'shape': 'triangle'} for <__main__.Apple object at 0x7ff05de056d0>
triangle
Deleting <__main__.Apple object at 0x7ff05de056d0>

You can also decorate classes and functions at the same time. Consider example_3.py:

def decorator(F):
    def wrapper(*args):
        print('Called {}'.format(args))
    return wrapper

@decorator
def func(x, y):
    print(x,y)

class C(object):
    @decorator
    def method(self, x, y):
        print(x,y)

if __name__ == '__main__':
    c = C()
    c.method(1,2)
    func(3,4)

Running this:

$ python example_3.py
Called (<__main__.C object at 0x7f28ce438590>, 1, 2)
Called (3, 4)

Here the decorator wraps either class or function. In the first case tuple with args contains only variables passed to the unction. In the class call in args, there is also an instance of C class.

It's also possible to decorate whole classes, like in example_4:

def decorator(cls):
    class Wrapper(object):
        def __init__(self, *args):
            self.wrapped = cls(*args)

        def __getattr__(self, name):
            print('Getting the {} of {}'.format(name, self.wrapped))
            return getattr(self.wrapped, name)

    return Wrapper

@decorator
class C(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y


if __name__ == '__main__':
    x = C(1,2)
    print(x.x)

Output of example_4:

$ python example_4.py
Getting the x of <__main__.C object at 0x7fed2468f750>
1

In this example, the class Wrapper on __init__ calls the class with args and store it under self.wrapped. So cls(*args) is the same as C(1,2).

Most of this examples are taken from book Learning Python 5th Edition by Mark Lutz.

Cover image by Unsplash under CC0.

Python slots

2016-01-23T10:20:00+01:00

Hello everyone in new layout of blog. Today I will write more about python ``__slots__``.

What exactly __slots__ do? Imagine that you have a two python classes- one with __slots__ and other without:

class Slots(object):
    __slots__ = ['arg1', 'arg2']

    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2

class NoSlots(object):
    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2

Then instantiate them:

>>> slots = Slots(arg1='first', arg2='second')
>>> no_slots = NoSlots(arg1='third', arg3='fourth')

At first there is no much difference between slots and no_slots instances, but when we use dir():

>>> len(dir(slots))
26
>>> len(dir(no_slots))
27

So what is different? The answer is slots got __slots__ and no_slots: __dict__ and __weakref__. The implication of this is as follows:

>>> slots.arg3 = 'fifth'
AttributeError: 'Slots' object has no attribute 'arg3'
>>> no_slots.arg3 = 'sixth'
>>> no_slots.arg3
'sixth'

Thanks to not having __dict__. Slots class is a bit faster, but it's impossible to add attribute that is not in __slots__. Moreover there is no __weakref__ which means that it is not possible to cache this object.

Cover image by JohnsonMartin under CC0.

Http and websockets logging handlers

2016-01-16T10:20:00+01:00

Hello, this posts will be about 3 specific logging handlers: HTTPHandler, SocketHandler and DatagramHandler.

HTTPHandler

Let's start with HTTPHandler: reading python docs about HTTPHandler we can see that:

The HTTPHandler class, located in the logging.handlers module, supports sending logging messages to a Web server, using either GET or POST semantics.

So this will be useful to have such handler in case of many different modules in different machines that sends logs to one central server.

As an example, I will build simple flask application which prints out the logging message from the client.

To install Flask:

$ pip install Flask

Then make server.py:

from flask import Flask, request

app = Flask(__name__)

@app.route("/", methods=['POST', 'GET'])
def hello():
for key, value in request.args.items():
    print(key,value)
return 'response' # it has to return something


if __name__ == "__main__":
app.run(debug=True)

To send some data, create script called send_log.py:

import logging
import logging.handlers

logger = logging.getLogger(__name__)

server = '127.0.0.1:5000'
path = '/'
method = 'GET'

sh = logging.handlers.HTTPHandler(server, path, method=method)

logger.addHandler(sh)
logger.setLevel(logging.DEBUG)

logger.debug("Test message.")

Running server.py and send_log.py will result in this output:

Your browser does not support the video tag.

SocketHandler

Now let's move to the SocketHandler: this is what python docs say about it

The SocketHandler class, located in the logging.handlers module, sends logging output to a network socket. The base class uses a TCP socket.

Based on this we can now guess that web socket will receive logging message.Then we can process it further. It will be useful when there is a lot of logs to be sent to the server. So opening HTTP connection every time is not a good solution.

So first we need some TCP server:

class LogRecordSocketReceiver(socketserver.ThreadingTCPServer):
    allow_reuse_address = True

    def __init__(self, host='localhost',
                 port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
                 handler=LogRecordStreamHandler):
        socketserver.ThreadingTCPServer.__init__(self, (host, port), handler)
        self.abort = 0
        self.timeout = 1
        self.logname = None

    def serve_until_stopped(self):
        import select
        abort = 0
        while not abort:
            rd, wr, ex = select.select([self.socket.fileno()],
                                       [], [],
                                       self.timeout)
            if rd:
                self.handle_request()
            abort = self.abort

def main():
    tcpserver = LogRecordSocketReceiver()
    print('About to start TCP server...')
    tcpserver.serve_until_stopped()

if __name__ == '__main__':
    main()

What is going on here? In the main function we instantiate threading TCP server and we serve it until we don't hit Ctrl+C. In the serve_until_stopped method of LogRecordSocketReceiver we are waiting for the key combination to the stop server and if this not happening the we retrieve information about the socket by self.socket.fileno() which is a descriptor of a socket. Then we pass it to another function call: this time select(). Select is system call for examining the status of file descriptors of open input/output channels which in this case is information from the socket. If there is anything ready to be read we handle the request and process it.

To process it we need handler:

class LogRecordStreamHandler(socketserver.StreamRequestHandler):

    def handle(self):
        while True:
            chunk = self.connection.recv(4)
            if len(chunk) < 4:
                break
            slen = struct.unpack('>L', chunk)[0]
            chunk = self.connection.recv(slen)
            while len(chunk) < slen:
                chunk = chunk + self.connection.recv(slen - len(chunk))
            obj = pickle.loads(chunk)
            print(obj)

In method handle we read chunks of information from sent logging message. The chunk is byte type so then we need to translate it to python object by calling pickle.loads().

So running our TCP server and sending log looks like this:

Your browser does not support the video tag.

Full gist with socket logger looks like this:

And with socket sender:

DatagramHandler

Lastly, there is DatagramHandler which supports sending logging messages over UDP.

The actual code is very similar to SocketHandler:

class MyUDPHandler(socketserver.BaseRequestHandler):
    def __init__(self, request, client_address, server):
        socketserver.BaseRequestHandler.__init__(self, request,
                                                 client_address,
                                                 server)

    def handle(self):
        msg, socket = self.request
        print("{} wrote:".format(self.client_address[0]))
        print(pickle.loads(msg[4:]))
        socket.sendto(msg.upper(), self.client_address)

Thanks to RooTer answer on stackoverflow I got this working by omitting first 4 bytes of data because they contain length of dumped object.

So the full upd_server looks as follows:

Which works like this:

Your browser does not support the video tag.

Resources

Socket server and log sender based on python logging cookbook

Updates

23.01.16 Thanks to RooTer answer I added UDP log handler

Cover image by Quinn Dombrowski under CC BY 2.0.

GeoDjango and Leaflet.js- part two

2016-01-08T10:20:00+01:00

This is the second post from GeoDjango i Leaflet.js series. You can find the previous post under this link.

After loading data to GeoDjango application now, it's time to present it to the user. You can use django template tag like {{object}} but I think it's better to provide api endpoints. I will be using GeoDjango builtin GeoJSON serializer. To do this declare new views in views.py:

from django.http import HttpResponse
from django.core.serializers import serialize
from .models import Point, Voivodeship

def points_view(request):
    points_as_geojson = serialize('geojson', Point.objects.all())
    return HttpResponse(points_as_geojson, content_type='json')

def wojewodztwa_view(request):
    voivodeships_as_geojson = serialize('geojson', Voivodeship.objects.all())
    return HttpResponse(voivodeships_as_geojson, content_type='json')

GeoJSON is open format for encoding geographical data. It's based on JSON.

Then add lines to urls.py:

from django.conf.urls import include, url
from django.contrib import admin
from voivodeships.views import points_view, voivodeships_view

urlpatterns = [
    url(r'^admin/', include(admin.site.urls)),
    url(r'^points.data/', points_view, name='points'),
    url(r'^voivodeships.data/', voivodeships_view, name='voivodeships'),
]

As you can see GeoDjango displays data from database in GeoJSON:

It's nice but end user need to see results on the map not in JSON format so I use Leaflet.js.

You can download leaflet.js from the web page but there is a better way: django-leaflet. It's django application with allows you embed leaflet to django project. Install it by:

$ pip install django-leaflet

Then make sure that leaflet is added to INSTALLED_APPS in settings.py:

INSTALLED_APPS = (
  # other applications
  leaflet
)

Let's add main page view to GeoDjango application in views.py:

from django.views.generic import TemplateView

class MainPageView(TemplateView): template\_name =
'voivodeships/index.html'

And to urls.py:

from voivodeships.views import MainPageView

urlpatterns = [# rest of urls
               url(r'^$', MainPageView.as\_view()),]

After this add new index.html under voivodeships/templates/voivodeships/index.html with this content:

<html>
{% load leaflet_tags %}
    <head>
        {% leaflet_js %}
        {% leaflet_css %}
    </head>
    <body>
        {% leaflet_map "poland" %}
    </body>
</html>

And going to the web page with running GeoDjango application you can see map:

Thanks to django-leaflet you can control behavior of all maps. Let add the following content to end of settings.py:

LEAFLET_CONFIG = {
  'DEFAULT_CENTER': (52.00,20.00),
  'DEFAULT_ZOOM': 6,
  'MIN_ZOOM': 1,
  'MAX_ZOOM': 20,
}

But still map is not taking full space in the web page so let's add more CSS lines to fix that in index.html:

<head>
   <style media="screen">
     #poland { width:100%; height:100% }
   </style>
   <!-- Rest of html -->

One of the Leaflet.js strong points is huge extensions database. In this project I will use few of them including: leaflet-ajax, leaflet-spin, markercluster. It's up to you how you want to install it. I will use bower for that:

$ bower install leaflet-ajax leaflet-spin leaflet.markerculster

Add STATICFILES_DIRS to settings.py:

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

After installation got to index.html and use these plugins:

{% load static %}
   <head>
      <!-- style tag and django-leaflet tag here -->
      <script src="{% static 'leaflet-ajax/dist/leaflet.ajax.min.js' %}"></script>
      <script src="{% static 'spin.js/spin.min.js' %}"></script>
      <script src="{% static 'leaflet-spin/leaflet.spin.js' %}"></script>
   </head>
   <body>
      <script type="text/javascript">
         function map_init_basic (map, options) {
               var geojsonPointLayer = new L.GeoJSON.AJAX("{% url 'points' %}", {
                   onEachFeature:function(feature, layer) {
                        layer.bindPopup(feature.properties.name.toString());
                    }
                });
               geojsonPointLayer.addTo(map);

               var geojsonVoivodeshipsLayer = new L.GeoJSON.AJAX("{% url 'voivodeships' %}", {
                   onEachFeature:function(feature, layer) {
                       layer.bindPopup(feature.properties.jpt_nazwa_field.toString());
                   }
               });
               geojsonVoivodeshipsLayer.addTo(map);
           }
      </script>
      {% leaflet_map "poland" callback="window.map_init_basic" %}
   </body>

I added new function map_init_basic which is a callback for django-leaflet tag. Then thanks to leaflet-ajax I get points and voivodeships GeoJSONs from GeoDjango. Moreover, I use function from leaflet.js: onEachFeature. This function add popup with the name of point or voivodeship.

After this changes main page looks like this:

Your browser does not support the video tag.

There is one problem. GeoJSON with voivodeship is so accurate that deserializing takes a lot of time (about 41 sec). So one of the solution is to dump GeoJSON to cache, I will use Redis as a cache database.

First, install and check if Redis is working by:

$ sudo apt-get install redis-server $ redis-cli ping PONG

Then it's time to install python bindings:

$ pip install redis $ pip install django-redis-cache

After this adjust some settings in settings.py:

MIDDLEWARE_CLASSES = [
    'django.middleware.cache.UpdateCacheMiddleware',
    # ... another middlewares
    'django.middleware.common.CommonMiddleware',
    # ... rest of middlewares
    'django.middleware.cache.FetchFromCacheMiddleware',
]

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': '127.0.0.1:6379',
    },
}

What is important in MIDDLEWARE_CLASSES is order: UpdateCacheMiddleware should go before CommonMiddleware and FetchFromCacheMiddleware is supposed to be last.

Lastly, add cache to voivodeships_view in views.py:

from django.core.cache import cache

def voivodeships_view(request):
    redis_key = 'voivodeships'
    voivodeships = cache.get(redis_key)  # getting value for given key from redis
    if not voivodeships:
       voivodeships = serialize('geojson', Voivodeship.objects.all())
       cache.set(redis_key, voivodeships)  # if not GeoJSON is not in cache set it
    return HttpResponse(voivodeships, content_type='json')

Right now GeoJSON will be loaded from the database. After reloading the web page, django will get results from cache.

That's all: you have working GeoDjango application. The github repo is under this link

Cover image present 1570 Abraham Ortelius map.

GeoDjango and Leaflet.js- part one

2016-01-01T10:20:00+01:00

Today is the day, when I will start my blog. I would like to publish new posts once a week. The comments and feedback will be pretty much appreciated as I have just started blogging in English.

I assume that the Reader knows how to run basic Django application. However, if this is not the case, I encourage you to start here.

So, let's start:

First of all, what is GeoDjango?

From official geodjango documentation:

GeoDjango intends to be a world-class geographic Web framework. Its goal is to make it as easy as possible to build GIS Web applications and harness the power of spatially enabled data.

To run GeoDjango you have to install some additional packages. In ubuntu you can easily do this way:

$ sudo apt-get install binutils libproj-dev gdal-bin

Gdal-bin package is not necessary but it's very helpful so I encourage to install it.

Then you need to choose what database you will use with your GeoDjango project. I personally choosen PostgreSQL with PostGIS because it support all of the GeoDjango features according to this table.

After starting project, and making new application add few lines to your settings.py:

DATABASES = {
'default': {
     'ENGINE': 'django.contrib.gis.db.backends.postgis',
     'NAME': 'your_database_name',
     'USER': 'your_user',
     'PASSWORD': 'your_password',
     'HOST': 'localhost',
     'PORT': '5432',
    }
}

and modify INSTALLED_APPS:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.gis',
    'name_of_your_applications' # in my case voivodeships
 )

To actually show some data we need it first. In this project I'll use shp files and points with coordinates.

There is a lot of sites with free shp files available but I will choose one for Poland. The shapefiles are in zip file. So go and grab them:

$ wget ftp://91.223.135.109/prg/jednostki_administracyjne.zip
$ mkdir data && unzip jednostki_administracyjne.zip -d data

I'm going to use only 1 shapefile from this zip called 'województwa.shp' (voivodeships). After unziping you want to examine contents of shapefiles. You can do it in tool like QuantumGIS or use GDAL ogrinfo.

In QGIS this shapefile presents as follows:

Using ogrinfo:

$ ogrinfo -so data/PRG_jednostki_administracyjne_v10/województwa.shp województwa

INFO: Open of `data/PRG_jednostki_administracyjne_v10/województwa.shp'
 using driver `ESRI Shapefile' successful.

Layer name: województwa
Geometry: Polygon
Feature Count: 16
Extent: (171677.555190, 133223.725152) - (861895.746988, 774923.747473)
Layer SRS WKT:
PROJCS["ETRS89 / Poland CS92",
   GEOGCS["ETRS89",
       DATUM["European Terrestrial Reference System 1989",
           SPHEROID["GRS 1980",6378137.0,298.257222101],
           TOWGS84[0,0,0]],
       PRIMEM["Greenwich",0.0],
       UNIT["Decimal Degree",0.017453292519943295]],
   PROJECTION["Transverse_Mercator"],
   PARAMETER["latitude_of_origin",0.0],
   PARAMETER["central_meridian",18.999999999999982],
   PARAMETER["scale_factor",0.9993],
   PARAMETER["false_easting",500000.0],
   PARAMETER["false_northing",-5300000.0],
   UNIT["Meter",1.0],
   AUTHORITY["EPSG","2180"]]
   iip_przest: String (255.0)
   iip_identy: String (255.0)
   iip_wersja: String (255.0)
   jpt_sjr_ko: String (255.0)
   jpt_kod_je: String (255.0)
   jpt_nazwa_: String (255.0)
   jpt_nazw01: String (255.0)
   jpt_organ_: String (255.0)
   jpt_orga01: String (255.0)
   jpt_jor_id: Real (32.0)
   wazny_od: Date (10.0)
   wazny_do: Date (10.0)
   jpt_wazna_: String (255.0)
   wersja_od: Date (10.0)
   wersja_do: Date (10.0)
   jpt_powier: Real (33.16)
   jpt_kj_iip: String (255.0)
   jpt_kj_i01: String (255.0)
   jpt_kj_i02: String (255.0)
   jpt_kod_01: String (255.0)
   id_bufora_: Real (32.0)
   id_bufor01: Real (32.0)
   id_technic: Real (32.0)
   jpt_opis: String (255.0)
   jpt_sps_ko: String (255.0)
   gra_ids: String (255.0)
   status_obi: String (255.0)
   opis_bledu: String (255.0)
   typ_bledu: String (255.0)

We can make model representing this data in our application by hand but it's django so you can type:

$ ./manage.py ogrinspect data/PRG_jednostki_administracyjne_v10/województwa.shp Voivodeship --mapping --srid 2180 --multi >> voivodeships/models.py

Where --mapping tells you to generate mapping used to load data from shapefile, --srid sets the SRID for the geographic field and --multi sets geographic field to MultiPolygonField

Our voivodeships/models.py will look like this:

from __future__ import unicode_literals
# This is an auto-generated Django model module created by ogrinspect.
from django.contrib.gis.db import models

class Voivodeship(models.Model):
    iip_przest = models.CharField(max_length=255)
    iip_identy = models.CharField(max_length=255)
    iip_wersja = models.CharField(max_length=255)
    jpt_sjr_ko = models.CharField(max_length=255)
    jpt_kod_je = models.CharField(max_length=255)
    jpt_nazwa_field = models.CharField(max_length=255)
    jpt_nazw01 = models.CharField(max_length=255)
    jpt_organ_field = models.CharField(max_length=255)
    jpt_orga01 = models.CharField(max_length=255)
    jpt_jor_id = models.FloatField()
    wazny_od = models.DateField()
    wazny_do = models.DateField()
    jpt_wazna_field = models.CharField(max_length=255)
    wersja_od = models.DateField()
    wersja_do = models.DateField()
    jpt_powier = models.FloatField()
    jpt_kj_iip = models.CharField(max_length=255)
    jpt_kj_i01 = models.CharField(max_length=255)
    jpt_kj_i02 = models.CharField(max_length=255)
    jpt_kod_01 = models.CharField(max_length=255)
    id_bufora_field = models.FloatField()
    id_bufor01 = models.FloatField()
    id_technic = models.FloatField()
    jpt_opis = models.CharField(max_length=255)
    jpt_sps_ko = models.CharField(max_length=255)
    gra_ids = models.CharField(max_length=255)
    status_obi = models.CharField(max_length=255)
    opis_bledu = models.CharField(max_length=255)
    typ_bledu = models.CharField(max_length=255)
    geom = models.MultiPolygonField(srid=2180)

    def __str__(self):
        return '{}'.format(jpt_nazwa_field)

    def __unicode__(self):
        return '{}'.format(jpt_nazwa_field)

# Auto-generated `LayerMapping` dictionary for Voivodeship model
voivodeship_mapping = {
    'iip_przest' : 'iip_przest',
    'iip_identy' : 'iip_identy',
    'iip_wersja' : 'iip_wersja',
    'jpt_sjr_ko' : 'jpt_sjr_ko',
    'jpt_kod_je' : 'jpt_kod_je',
    'jpt_nazwa_field' : 'jpt_nazwa_',
    'jpt_nazw01' : 'jpt_nazw01',
    'jpt_organ_field' : 'jpt_organ_',
    'jpt_orga01' : 'jpt_orga01',
    'jpt_jor_id' : 'jpt_jor_id',
    'wazny_od' : 'wazny_od',
    'wazny_do' : 'wazny_do',
    'jpt_wazna_field' : 'jpt_wazna_',
    'wersja_od' : 'wersja_od',
    'wersja_do' : 'wersja_do',
    'jpt_powier' : 'jpt_powier',
    'jpt_kj_iip' : 'jpt_kj_iip',
    'jpt_kj_i01' : 'jpt_kj_i01',
    'jpt_kj_i02' : 'jpt_kj_i02',
    'jpt_kod_01' : 'jpt_kod_01',
    'id_bufora_field' : 'id_bufora_',
    'id_bufor01' : 'id_bufor01',
    'id_technic' : 'id_technic',
    'jpt_opis' : 'jpt_opis',
    'jpt_sps_ko' : 'jpt_sps_ko',
    'gra_ids' : 'gra_ids',
    'status_obi' : 'status_obi',
    'opis_bledu' : 'opis_bledu',
    'typ_bledu' : 'typ_bledu',
    'geom' : 'MULTIPOLYGON',
}

Where under Voivodeship model we have all fields from shp file and in voivodeship_mapping we got proper mapping for loading data. I added __str__ and __unicode__ just for convenience.

Let's add an admin for our voivodeship application by editing admin.py:

from django.contrib.gis import admin
from .models import Voivodeship

admin.site.register(Voivodeship, admin.OSMGeoAdmin)

After making and executing migrations to your database we can load shapes with voivodeships. To do this let's create load.py with following contents:

import os
from django.contrib.gis.utils import LayerMapping
from .models import Voivodeship

voivodeship_mapping = {
    'iip_przest' : 'iip_przest',
    'iip_identy' : 'iip_identy',
    'iip_wersja' : 'iip_wersja',
    'jpt_sjr_ko' : 'jpt_sjr_ko',
    'jpt_kod_je' : 'jpt_kod_je',
    'jpt_nazwa_field' : 'jpt_nazwa_',
    'jpt_nazw01' : 'jpt_nazw01',
    'jpt_organ_field' : 'jpt_organ_',
    'jpt_orga01' : 'jpt_orga01',
    'jpt_jor_id' : 'jpt_jor_id',
    'wazny_od' : 'wazny_od',
    'wazny_do' : 'wazny_do',
    'jpt_wazna_field' : 'jpt_wazna_',
    'wersja_od' : 'wersja_od',
    'wersja_do' : 'wersja_do',
    'jpt_powier' : 'jpt_powier',
    'jpt_kj_iip' : 'jpt_kj_iip',
    'jpt_kj_i01' : 'jpt_kj_i01',
    'jpt_kj_i02' : 'jpt_kj_i02',
    'jpt_kod_01' : 'jpt_kod_01',
    'id_bufora_field' : 'id_bufora_',
    'id_bufor01' : 'id_bufor01',
    'id_technic' : 'id_technic',
    'jpt_opis' : 'jpt_opis',
    'jpt_sps_ko' : 'jpt_sps_ko',
    'gra_ids' : 'gra_ids',
    'status_obi' : 'status_obi',
    'opis_bledu' : 'opis_bledu',
    'typ_bledu' : 'typ_bledu',
    'geom' : 'MULTIPOLYGON',
}

voivodeship_shp = os.path.abspath(os.path.join('data', 'PRG_jednostki_administracyjne_v10', 'województwa.shp'))

def run(verbose=True):
    lm = LayerMapping(voivodeship, voivodeship_shp, voivodeship_mapping,
                      transform=False, encoding='iso-8859-1')

    lm.save(strict=True, verbose=verbose)

We don't need to provide any transform in LayerMapping call because a correct srid is already there. After saving the file run load.py from django shell:

>>> from voivodeships import load
>>> load.run()
# a lot of output here...
# IntegrityError

We got this IntegrityError because our models fields does not allow empty values in wazny_od field and this shp has some empty values. To avoid such errors we can edit Voivodeship model fields:

# rest of code here ...

class Voivodeship(models.Model):
    #...
    wazny_od = models.DateField(null=True, blank=True)
    wazny_do = models.DateField(null=True, blank=True)
    #...
    wersja_od = models.DateField(null=True, blank=True)
    wersja_do = models.DateField(null=True, blank=True)
    #...

# rest of code here...

Rerun ./manage.py makemigrations and ./manage.py migrate then try one more time to run load.py

from voivodeships import load
load.run()
# Saved: Voivodeship object
# 15 times more

When you run ./manage.py runserver and go to the admin site you can see that geometric field is displayed in form of a map:

To make our map more robust let's add additional data (points). There are 16 points in capitals of polish voivodeships in CSV file.

"Rzeszów","50.04015435","22.006124806535"
"Kraków","50.0591381","19.9372696"
"Katowice","50.244967474000475","19.02679182200046"
"Opole","50.66800573900048","17.91971986900046"
"Wrocław","51.1071531","17.0230635"
"Zielona Góra","51.9391761","15.5120624461033"
"Poznań","52.4299047","16.9352252"
"Łódź","51.7471563","19.5665896"
"Kielce","50.8961522","20.5798348952528"
"Lublin","51.250969","22.5547430471451"
"Warszawa","52.2458285","21.2234804"
"Białystok","53.13074825","23.1708377413392"
"Olsztyn","53.783197","20.4918318"
"Toruń","53.0129269","18.6040463"
"Gdańsk","54.37834175","18.5866973556132"
"Szczecin","53.43813","14.5423387"

Let's make additional model for points:

class Point(models.Model):
    name = models.CharField(max_length=200)
    geom = models.PointField('longitude/latitude', blank=True, null=True)
    objects = models.GeoManager()

def __str__(self):
    return self.name

def __unicode__(self):
    return self.name

Then to admin.py:

from .models import Point
admin.site.register(Point, admin.OSMGeoAdmin)

And to add them to GeoDjango lets add few lines to load.py:

from django.contrib.gis import geos
from .models import Point

point_csv = os.path.abspath(os.path.join('data', 'points.csv'))
def point_load():
    with open(point_csv) as point_file:
        for line in point_file:
            name, lon, lat = line.split(',')
            point = "POINT(%s %s)" % (lat.strip(), lon.strip())
            Point.objects.create(name=name, geom=geos.fromstr(point))

Then import load.py in django shell and run point_load().

Ok, that's all for this post. Stay tuned for the next part where I show how display data on actual map.

Cover image present 1570 Abraham Ortelius map.

Update 08.01.16:

Fixing grammar errors. Thank you Adrianna!

Krzysztof Żuraw

Ports and Adapters in python - part three

Ports and Adapters in python - part two

Changes from 07.06.16:

Ports and Adapters in python - part one

Changes from 23.05.16:

My first personal project

Why?

What I learned?

What can be done differenlty?

Mocks and monkeypatching in python

Why bother mocking?

Mocking function

Mocking classes

References:

My first workshop experience

Update 18.04.16

Automatic blog deployment with TravisCI

References:

Moving blog to pelican

Nikola

Jekyll

Pelican

Django + Celery & Rabbit - part four

Audio File detail view

Logging

Summary

Django + Celery & Rabbit - part three

Django + Celery & Rabbit - part two

Django + Celery & Rabbit - part one

Django cookiecutter

Resources:

Django Haystack and Elasticsearch- part two

Resources:

Django Haystack and Elasticsearch- part one

Python class @decorators

Python __slots__

Http and websockets logging handlers

HTTPHandler

SocketHandler

DatagramHandler

Resources

Updates

GeoDjango and Leaflet.js- part two

GeoDjango and Leaflet.js- part one

Update 08.01.16:

Python slots