Tag python

What is LRU cache?

When you cache is starting to grow more and more you have to remove something so new values can be stored in a cache. One of the algorithms that are used to accomplish this task is called Least Recently Used (LRU). When you performing LRU caching you always throw out the data that was least recently used.

Imagine you have in cache five elements: A,B,C,D,E. You access element A which is in cache - nothing changes. Right after that, you want to add a new element to cache - F. At this moment the least recently used item is B so you throw it and replace with F. The same mechanism goes for other items. That's how LRU cache works.

Gunicorn & LRU pitfall

In python 3 you can use decorator @lru_cache from functools module. It stores a result of decorated function inside the cache. Imagine that you have simple flask application:

from flask import Flask, jsonify
from functools import lru_cache
app = Flask(__name__)
@app.route('/')
def main():
  store_to_cache()
  return jsonify({'message': 'Stored'})
@lru_cache(maxsize=2)
def store_to_cache():
  return {'this_goes_to_cache': 'and_this_too'}

You enter the root URL and you store dictionary to cache. Cache is setup to have only 2 elements inside. Then you have a helper function for getting info about an object that is inside that cache:

@app.route('/get_cache_info')
def get_cache_info():
  cache_info = store_to_cache.cache_info()
  return jsonify({
      'Hits': cache_info.hits,
      'Misses': cache_info.misses,
      'Maxsize': cache_info.maxsize,
      'Currsize': cache_info.currsize
  })

When you run this application in development mode - without gunicorn everything works as expected - you store to cache and receive proper information:

$ curl -X GET http://127.0.0.1:8000
{
"message": "Stored"
}
$ curl -X GET http://127.0.0.1:8000/get_cache_info
{
  "Currsize": 1,
  "Hits": 0,
  "Maxsize": 2,
  "Misses": 1
}

Let's run the same code but with using gunicorn with two workers:

$ gunicorn --workers=2 application:app
$ curl -X GET http://127.0.0.1:8000
$ curl -X GET http://127.0.0.1:8000/get_cache_info
{
  "Currsize": 1,
  "Hits": 0,
  "Maxsize": 2,
  "Misses": 1
}
curl -X GET http://127.0.0.1:8000/get_cache_info
{
  "Currsize": 0,
  "Hits": 0,
  "Maxsize": 2,
  "Misses": 0
}

Sometimes request returns that there is one item inside cache and other times that there are no items in the cache. Why is that? Because LRU cache is using cache per worker. It means that when user enters your site cache is stored but it is stored only on this worker! The same user enters another time and his request is handled by the second worker which doesn't have anything stored in the cache!

For this reason, it's not a good idea to use cache per worker in your web application. What can you use instead? Use centrally stored cache like Memcached. You will thank yourself in the future.

That's all for today! Feel free to comment - maybe you have a better idea which cache use to avoid pitfalls?

Example of how LRU cache works is based upon this article.

The code that I have made so far is available on github. Stay tuned for next blog post from this series.

Update 13-02-16:

Side note from my friend from work: Cache per worker is good for data that doesn't changex like archival exchange rate. But this type of cache is not good for data that can change.

Thank you Paweł for this note.

Cover image by Tim Green under CC BY-SA 2.0.

Assumptions

I need an application that gets a code from the user, executes it and gives output back. As many people before me said output from user cannot be trusted so I need to use some kind of container for user input. To do that I used Docker python API- docker.py. Using that and Flask I created Tdd-app-prototype. Under the hood, this application will work like this: user writes a code on a website, clicks submit. Then Docker creates a container based on python docker image and executes code. I take the output from the container and destroy it afterwards.

As we know what application should do, let's jump into the code.

Code

The first problem that I have is that I don't want to write a code provided by the user to a disk, then read it from the disk and it execute by Docker. I want to store it in memory - perfect case for StringIO. Code that does this looks as follows:

@app.route("/send_code", methods=['POST'])
def execute_code():
  data = request.form['source_code']
  code = io.StringIO(data)
  create_container(code)
  output = get_code_from_docker()
  return output

Here beside specifying routes in Flask I take data from the form, cast it to StringIO and create a container from that code. Function that does that is below:

def create_container(code):
  cli.create_container(
       image='python:3',
       command=['python','-c', code.getvalue()],
       name='tdd_app_prototype',
  )

What is cli here? I can use docker.py with Docker from other than my own computer location so before I can use any of these functions I need to specify Client:

cli = Client(base_url='unix://var/run/docker.sock')

It tells docker.py to use my local Docker. Let's go back to create_container. I tell docker.py to use official python 3 images. Then I specify a command to run: python -c and my code from StringIO. If you want to run standalone python script you can use this:

def create_container(code):
  cli.create_container(
       image='python:3',
       command=['python','-c', 'my_code.py'],
       volumes=['/opt'],
       host_config=cli.create_host_config(
           binds={ os.getcwd(): {
               'bind': '/opt',
               'mode': 'rw',
               }
           }
       ),
       name='tdd_app_prototype',
       working_dir='/opt'
  )

volumes and host_config keywords are for telling Docker to mount volumes. It is the same as running docker run -v "$PWD":/opt. Finally I set up working_dir so I don't need to provide a full path to my_code.py. As we have a container created now it is time to start it:

def get_code_from_docker():
  cli.start('tdd_app_prototype')
  cli.wait('tdd_app_prototype')
  output = cli.logs('tdd_app_prototype')
  cli.remove_container('tdd_app_prototype', force=True)
  return "From docker: {}".format(output.strip())

I used here wait so I wait for the container to stop. Then I take output in form of lists and remove the container. Right now it looks as follows:

That's all for today! If you want to see full code grab it here. Do you know other ways of using docker.py? Please leave a comment.

Special thanks to Kasia for being editor for this post. Thank you.

Cover image by Gabriel Barathieu under CC BY-SA 2.0, via Wikimedia Commons

Day 1

I was on a presentation by Marcin Bardź PostgreSQL i SQLAlchemy Duet (nie)doskonały. It was a talk about the tandem of two tools that are commonly used in Python world: PostreSQL and SQLAlchemy. In this talk, Marcin presented a basic description of what exactly a database is, how postgresql is meeting the requirements of a good database. Then he talked about SQLAlchemy basic stuff like quering, creating tables or making relations. I would have liked this talk more if it would been for more experienced people. But in general it was a good talk.

Day 2

It was a hard day. Only talks that I had time to attend were lightning talks where I found a range of topics starting from python tests to golf coding contest. As I am on this topic I was one of the makers of my company Code Golf game - golf.stxnext.pl. It was a simple code golf game in which contestants had to solve one of the following tasks: game of life, tic tac toe and reverse polish notation. What I have learned from this is that there always have to be tests. ALWAYS. I don't have ones and as it turns out there was a problem that my test cases that were supposed to test code were bugged and solving this issue on spot isn't pleasant.

Day 3

In the morning I was attending talk by Michał Lowas Rzecionek about Async.io: Async pod maską. It was an interesting talk about what exactly is going on under the hood in ansync.io. Next presentation that I happened to be at was a keynote by Jessica McKellar about systems. It was an interesting talk mainly about current social systems and their relation to programming.

Day 4

During the conference, there was a very interesting capture the flag competition. The theme of this CTF was the Lord of the Rings. As it turns out there was about 60 teams of 2 people. Tasks were not so easy and many people did not do all 10 tasks. Right after that there was a talk by Kasia Jachim about how to write code that is not testable. To be honest I really don't know what this talk was about.

This was my short relation from this year PyConPl 2016. What are your feelings? Please tell me in comments.

Special thanks to Kasia for being editor for this post. Thank you.

Cover picture taken from Hotel Ossa.

Libarchive & python-libarchive-c

If you happen to learn more about how to create archives in various formats like tar, iso or zip I bet you heard about libarchive. It is widely used archive library written in C.

To use it within python you can choose from a few libraries but one that is currently maintained is called python-libarchive-c. When in my work I was to implement the feature of adding entries to archive from memory I decided to use existing module and give something back to a community in form of an open source contribution.

Add entry from memory

To make such a feature I have to reread carefully code examples in libarchive c itself. I also get familiar with few archive formats and their limitations. But enough talking lets jump to the code:

import requests
import libarchive
def create_archive_from_memory_file():
    response = requests.get('link', stream=True)
    with libarchive.file_writer('archive.zip', 'zip') as archive:
        archive.add_file_from_memory(
            entry_path='filename',
            entry_size=int(response.headers['Content-Length']),
            entry_data=response.iter_content(chunk_size=1024)
        )
if __name__ == '__main__':
    create_archive_from_memory_file()

My changes in code have not been released so make sure that you install python-libarchive-c from github like this (to run this script you also need requests library):

$ pip install git+https://github.com/Changaco/python-libarchive-c

In this snippet, I use request feature that doesn't require loading the whole content of the response to memory but instead I add the argument: stream=True and then I use response.iter_content(chunk_size=1024). Rest of the code is simply calling add_file_from_memory with a path (entry_path) and size of the entry in an archive (entry_size).

Under the hood, python-libarchive-c is using c_types with ffi to call libarchive functions. At first, it setup path to entry then sets its size, filetype and permission which file will be saved in the archive. Then write the header and start iterating through the entry_data by chunks and write them. At the end, header is set and archive is ready for user.

To see it in action have snippet above as example.py and run this script:

$ python example.py
$ ls -la
-rw-r--r--. 1 kzuraw kzuraw 11M 09-24 13:04 archive.zip
-rw-rw-r--. 1 kzuraw kzuraw 511 09-24 12:59 example.py

That's all for this week. Feel free to comment and if you have any questions don't hesitate to ask them.

Special thanks to Kasia for being editor for this post. Thank you.

Cover image by Archivo-FSP under CC BY-SA 3.0.

Python WebDAV client - easywebdav

I was searching for good python library to work with WebDAV for a long time. I finally found it- easywebdav. It works nicely but the problem is that doesn't have support for python 3. Let's jump quickly to my simple project for cli tool- webdav editor.

WebDAV editor

I decided to create cli tool to work with WebDAV server- webdav editor. Right now it supports only basic commands like login, listing the content of directories, uploading and downloading files.

I started from creating file webdav_utility.py:

from urlparse import urlparse
import easywebdav
class Client(object):
    def login(self, *args):
        argparse_namespace = args[0]
        url_components = urlparse(argparse_namespace.server)
        host, port = url_components.netloc.split(':')
        webdav_client = easywebdav.connect(
            host=host,
            port=port,
            path=url_components.path,
            username=argparse_namespace.user,
            password=argparse_namespace.password
        )
        pickle.dump(webdav_client, open('webdav_login', 'wb'))
    def list_content(self, *args):
        argparse_namespace = args[0]
        print [i.name for i in webdav_client.ls(argparse_namespace.path)]
    def upload_file(self, *args):
        argparse_namespace = args[0]
        webdav_client.upload(
            argparse_namespace.from_path, argparse_namespace.to_path
        )
    def download_file(self, *args):
        argparse_namespace = args[0]
        webdav_client.download(
            argparse_namespace.from_path, argparse_namespace.to_path
        )

In class Client, I write simple functions that are wrappers around easywebdav API. In login I parse provided URL in form like localhost:8888/owncloud/remote.php/webdav to get host, port and path for easywebdav.connect to establish a proper connection.

Another method that is worth mentioning is list_content where I retrieve names of files under a directory on WebDAV server. In every method I provide *args argument and argparse_namespace which leads to another component of application- module cli.py:

import argparse
from webdav_utility import Client
client = Client()
parser = argparse.ArgumentParser(description='Simple command line utility for WebDAV')
subparsers = parser.add_subparsers(help='Commands')
login_parser = subparsers.add_parser('login', help='Authenticate with WebDAV')
login_parser.add_argument('-s', '--server', required=True)
login_parser.add_argument('-u', '--user', required=True)
login_parser.add_argument('-p', '--password', required=True)
login_parser.set_defaults(func=client.login)
ls_parser = subparsers.add_parser('ls', help='List content of directory under WebDAV')
ls_parser.add_argument('-p', '--path', required=True)
ls_parser.set_defaults(func=client.list_content)
upload_parser = subparsers.add_parser('upload', help='Upload files to WebDAV')
upload_parser.add_argument('-f', '--from', metavar='PATH')
upload_parser.add_argument('-t', '--to', metavar='PATH')
upload_parser.set_defaults(func=client.upload_file)
download_parser = subparsers.add_parser('download', help='Download files from WebDAV')
download_parser.add_argument('-f', '--from', metavar='PATH')
download_parser.add_argument('-t', '--to', metavar='PATH')
download_parser.set_defaults(func=client.download_file)
if __name__ == '__main__':
    args = parser.parse_args()
    args.func(args)

There I use argparse. I create the main parser with four additionals subparsers for login, ls, upload and download. Thanks to that I have different namespace for every one of previously mentioned subparsers.

Problem is that this solution is not generic enough because after running my command with login parameter I can get: Namespace(server='localhost:8888', user='admin', password='admin') and running the same command but with ls I will receive: Namespace(path='path_to_file'). To handle that I used set_defaults for every subparser. I tell argparse to invoke function specified by func keyword (which is different for every command). Thanks to that I only need to call this code once:

if __name__ == '__main__':
    args = parser.parse_args()
    args.func(args)

That's the reason I introduce argparse_namespaces in Client.

OK, tool right now works nicely, but there is no place to store information if I am logged or not. So calling python cli.py login -s localhost -u admin -p admin works but python cli.py ls -p / not. To overcome that I came up with an idea to pickle webdav_client like this:

class Client(object):
  def login(self, *args):
    # login user etc
    pickle.dump(webdav_client, open('webdav_login', 'wb'))
  def list_content(self, *args):
    webdav_client = pickle.load(open('webdav_login', 'rb'))
    # rest of the code

Then I can easily run:

$ python cli.py login --server example.org/owncloud/remote.php/webdav --user admin --password admin
$ python cli.py ls --path '/'
['/owncloud/remote.php/webdav/', '/owncloud/remote.php/webdav/Documents/', '/owncloud/remote.php/webdav/Photos/', '/owncloud/remote.php/webdav/ownCloud%20Manual.pdf']

Conclusion

In this series, I setup an owncloud server and write simple tool just to show capabilities of WebDAV. I believe that some work, especially for webdav editor cli can still be done: the better way to handle user auth than pickle, separate Client class from argparse dependencies. If you have additional comments or thoughts please write a comment! Thank you for reading.

Other blog posts in this series:

• Python & WebDAV- part one

Github repo for this blog post: link.

Special thanks to Kasia for being editor for this post. Thank you.

Cover image by kleuske under CC BY-SA 2.0.