Krzysztof Żuraw


Personal site

Python & WebDAV- part two

In the last post, I set up owncloud with WebDAV server. Now it's time to use it.

Python WebDAV client - easywebdav

I was searching for good python library to work with WebDAV for a long time. I finally found it- easywebdav. It works nicely but the problem is that doesn't have support for python 3. Let's jump quickly to my simple project for cli tool- webdav editor.

WebDAV editor

I decided to create cli tool to work with WebDAV server- webdav editor. Right now it supports only basic commands like login, listing the content of directories, uploading and downloading files.

I started from creating file webdav_utility.py:

from urlparse import urlparse
import easywebdav


class Client(object):

    def login(self, *args):
        argparse_namespace = args[0]
        url_components = urlparse(argparse_namespace.server)
        host, port = url_components.netloc.split(':')
        webdav_client = easywebdav.connect(
            host=host,
            port=port,
            path=url_components.path,
            username=argparse_namespace.user,
            password=argparse_namespace.password
        )
        pickle.dump(webdav_client, open('webdav_login', 'wb'))

    def list_content(self, *args):
        argparse_namespace = args[0]
        print [i.name for i in webdav_client.ls(argparse_namespace.path)]

    def upload_file(self, *args):
        argparse_namespace = args[0]
        webdav_client.upload(
            argparse_namespace.from_path, argparse_namespace.to_path
        )

    def download_file(self, *args):
        argparse_namespace = args[0]
        webdav_client.download(
            argparse_namespace.from_path, argparse_namespace.to_path
        )

In class Client, I write simple functions that are wrappers around easywebdav API. In login I parse provided URL in form like localhost:8888/owncloud/remote.php/webdav to get host, port and path for easywebdav.connect to establish a proper connection.

Another method that is worth mentioning is list_content where I retrieve names of files under a directory on WebDAV server. In every method I provide *args argument and argparse_namespace which leads to another component of application- module cli.py:

import argparse

from webdav_utility import Client

client = Client()

parser = argparse.ArgumentParser(description='Simple command line utility for WebDAV')
subparsers = parser.add_subparsers(help='Commands')

login_parser = subparsers.add_parser('login', help='Authenticate with WebDAV')
login_parser.add_argument('-s', '--server', required=True)
login_parser.add_argument('-u', '--user', required=True)
login_parser.add_argument('-p', '--password', required=True)
login_parser.set_defaults(func=client.login)

ls_parser = subparsers.add_parser('ls', help='List content of directory under WebDAV')
ls_parser.add_argument('-p', '--path', required=True)
ls_parser.set_defaults(func=client.list_content)

upload_parser = subparsers.add_parser('upload', help='Upload files to WebDAV')
upload_parser.add_argument('-f', '--from', metavar='PATH')
upload_parser.add_argument('-t', '--to', metavar='PATH')
upload_parser.set_defaults(func=client.upload_file)

download_parser = subparsers.add_parser('download', help='Download files from WebDAV')
download_parser.add_argument('-f', '--from', metavar='PATH')
download_parser.add_argument('-t', '--to', metavar='PATH')
download_parser.set_defaults(func=client.download_file)

if __name__ == '__main__':
    args = parser.parse_args()
    args.func(args)

There I use argparse. I create the main parser with four additionals subparsers for login, ls, upload and download. Thanks to that I have different namespace for every one of previously mentioned subparsers.

Problem is that this solution is not generic enough because after running my command with login parameter I can get: Namespace(server='localhost:8888', user='admin', password='admin') and running the same command but with ls I will receive: Namespace(path='path_to_file'). To handle that I used set_defaults for every subparser. I tell argparse to invoke function specified by func keyword (which is different for every command). Thanks to that I only need to call this code once:

if __name__ == '__main__':
    args = parser.parse_args()
    args.func(args)

That's the reason I introduce argparse_namespaces in Client.

OK, tool right now works nicely, but there is no place to store information if I am logged or not. So calling python cli.py login -s localhost -u admin -p admin works but python cli.py ls -p / not. To overcome that I came up with an idea to pickle webdav_client like this:

class Client(object):

  def login(self, *args):
    # login user etc
    pickle.dump(webdav_client, open('webdav_login', 'wb'))

  def list_content(self, *args):
    webdav_client = pickle.load(open('webdav_login', 'rb'))
    # rest of the code

Then I can easily run:

$ python cli.py login --server example.org/owncloud/remote.php/webdav --user admin --password admin
$ python cli.py ls --path '/'
['/owncloud/remote.php/webdav/', '/owncloud/remote.php/webdav/Documents/', '/owncloud/remote.php/webdav/Photos/', '/owncloud/remote.php/webdav/ownCloud%20Manual.pdf']

Conclusion

In this series, I setup an owncloud server and write simple tool just to show capabilities of WebDAV. I believe that some work, especially for webdav editor cli can still be done: the better way to handle user auth than pickle, separate Client class from argparse dependencies. If you have additional comments or thoughts please write a comment! Thank you for reading.

Other blog posts in this series:

Github repo for this blog post: link.

Special thanks to Kasia for being editor for this post. Thank you.

Cover image by kleuske under CC BY-SA 2.0.

To see comments and full article enter: Python & WebDAV- part two

Python & WebDAV- part one

I must confess I was ignorant. Recently I saw an application done by my colleague at work and it was using python WebDAV client. I was intrigued because for me WebDAV was connected with calendars (CalDAV) and contacts (CardDAV). And there I have WebDAV which is filesystem.

I decided to write this blog post about it for better understanding what it is. This is the first part of blog series about this extension to HTTP.

What is WebDAV and what is it useful for?

From wikipedia:

Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that allows clients to perform remote Web content authoring operations. (...)

The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share.

In other words users that work with the server that implements WebDAV can share, move and edit files through a web server. There is also a way to lock file or get a specific revision of it. WebDAV is supported by many client applications like Windows Explorer or Nautilus. It is similar to another protocol- FTP. FTP is faster, but it doesn't run on top of HTTP. WebDAV also support SSL and authentication.

CardDAV and CalDAV are extensions to WebDAV that enable client/server address book and to accessing the calendar on the remote server.

Setting up your own WebDAV server

I will set up basic WebDAV server using owncloud. Owncloud is a self-hosted solution for the cloud. In addition to this owncloud provides WebDAV server that I will use in the next blog post.

On owncloud download page, there is a lot of options to choose from but I choose appliances tab with OVA (open virtual application) image for VirtualBox. Installation is really simple just follow this manual.

After a while, you will have working owncloud served from VirtualBox. Now it's time to play with WebDAV server. To check if this is working I will use CURL:

$ curl --user user:password 'http://localhost:8888/owncloud/remote.php/webdav/'
This is the WebDAV interface. It can only be accessed by WebDAV clients such as the ownCloud desktop sync client.⏎

To get properties about root folder:

$ curl --user user:password --include --request PROPFIND --header "Depth: 1" 'http://localhost:8888/owncloud/remote.php/webdav'
<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
 <d:response>
  <d:href>/owncloud/remote.php/webdav/</d:href>
  <d:propstat>
   <d:prop>
    <d:getlastmodified>Thu, 08 Sep 2016 04:22:23 GMT</d:getlastmodified>
    <d:resourcetype>
     <d:collection/>
    </d:resourcetype>
    <d:quota-used-bytes>4756701</d:quota-used-bytes>
    <d:quota-available-bytes>-3</d:quota-available-bytes>
    <d:getetag>&quot;57d0e77f723e4&quot;</d:getetag>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
  </d:propstat>
 </d:response>
 <d:response>
  <d:href>/owncloud/remote.php/webdav/Documents/</d:href>
  <d:propstat>
   <d:prop>
    <d:getlastmodified>Thu, 08 Sep 2016 04:22:23 GMT</d:getlastmodified>
    <d:resourcetype>
     <d:collection/>
    </d:resourcetype>
    <d:quota-used-bytes>36227</d:quota-used-bytes>
    <d:quota-available-bytes>-3</d:quota-available-bytes>
    <d:getetag>&quot;57d0e77f4b534&quot;</d:getetag>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
  </d:propstat>
 </d:response>
 <d:response>
  <d:href>/owncloud/remote.php/webdav/Photos/</d:href>
  <d:propstat>
   <d:prop>
    <d:getlastmodified>Thu, 08 Sep 2016 04:22:23 GMT</d:getlastmodified>
    <d:resourcetype>
     <d:collection/>
    </d:resourcetype>
    <d:quota-used-bytes>678556</d:quota-used-bytes>
    <d:quota-available-bytes>-3</d:quota-available-bytes>
    <d:getetag>&quot;57d0e77f69116&quot;</d:getetag>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
  </d:propstat>
 </d:response>
 <d:response>
  <d:href>/owncloud/remote.php/webdav/ownCloud%20Manual.pdf</d:href>
  <d:propstat>
   <d:prop>
    <d:getlastmodified>Thu, 08 Sep 2016 04:22:23 GMT</d:getlastmodified>
    <d:getcontentlength>4041918</d:getcontentlength>
    <d:resourcetype/>
    <d:getetag>&quot;1951114eecb977f35fb154c06dcfc4e0&quot;</d:getetag>
    <d:getcontenttype>application/pdf</d:getcontenttype>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
  </d:propstat>
 </d:response>
</d:multistatus>

That's all for this post! Next week as I got my server running I will be looking into python library for WebDAV. Feel free to comment- I really appreciate your feedback.

Other blog posts in this series:

Special thanks to Kasia for being editor for this post. Thank you.

Cover image by kleuske under CC BY-SA 2.0.

To see comments and full article enter: Python & WebDAV- part one

Makefiles in python projects

When I join my current company I saw in their git repos strange file. It wasn't used by any python code. It just sits in the main directory of the project. I asked my colleagues what is this file for? They told me- to make your life easier. That's why today I write about this file- Makefile.

What is makefile and what is it typical use

From this tutorial:

Makefiles are a simple way to organize code compilation.

Typically they are used in writing C programs to ease all stuff that needs to be done before code can be used as a program. You can specify rules to tell make how to compile your programs. Simple makefile for C code can be as follows:

helloword: helloword.c
    gcc -o hellword hellword.c -I.

Then running:

$ make helloword

you compile C code using gcc.

How is this even connected with python? This programming language is compiled itself while invoked so it doesn't need any makefiles to work. As I said in the beginning in python projects you can ease your life and save a lot of keystrokes using makefile.

What specific to python you can put in makefile

Have you ever wanted to clean up .pyc files from your project or remove artifacts after building python packages? Or maybe you want to run tests with coverage? Use pep8, lint or isort? Maybe run the application in docker container and end up writing commands that are too long for your screen?

This is where makefile comes. You can have everything kept in one place and use only make clean to clean up unnecessary files or make tests to test your application.

Let start with some examples from makefile that I am using:

HOST=127.0.0.1
TEST_PATH=./

clean-pyc:
    find . -name '*.pyc' -exec rm --force {} +
    find . -name '*.pyo' -exec rm --force {} +
    find . -name '*~' -exec rm --force  {} +

clean-build:
    rm --force --recursive build/
    rm --force --recursive dist/
    rm --force --recursive *.egg-info

isort:
    sh -c "isort --skip-glob=.tox --recursive . "

lint:
    flake8 --exclude=.tox

test: clean-pyc
    py.test --verbose --color=yes $(TEST_PATH)

run:
    python manage.py runserver

docker-run:
    docker build \
      --file=./Dockerfile \
      --tag=my_project ./
    docker run \
      --detach=false \
      --name=my_project \
      --publish=$(HOST):8080 \
      my_project

At the beginning, I add two variables HOST and TEST_PATH for every command to use them. Rule clean-pyc finds all files that end with *.pyc, *.pyo or *~ and delete them. The plus sign at the end of the command is for -exec command {} which means that the total number of invocations of the command will be much less than the number of matched files.

Next one clean-build is for removing build artifacts. In isort shell is executing isort command with proper attributes, -c flag is for reading commands from a string rather than from standard input. lint and run works on the same pattern. In test I added the additional rule to execute before actual tests- clean-pyc. Last docker-run rule builds and runs docker.

Additional things that you want to add is something called PHONY. By default, makefile operates on files so if there will be a file called clean-pyc it will try to use it instead of a command. To avoid this use PHONY at the beginning of your makefile:

.PHONY: clean-pyc clean-build

I also like to have help function for my makefile so I put this somewhere inside:

help:
    @echo "    clean-pyc"
    @echo "        Remove python artifacts."
    @echo "    clean-build"
    @echo "        Remove build artifacts."
    @echo "    isort"
    @echo "        Sort import statements."
    @echo "    lint"
    @echo "        Check style with flake8."
    @echo "    test"
    @echo "        Run py.test"
    @echo '    run'
    @echo '        Run the `my_project` service on your local machine.'
    @echo '    docker-run'
    @echo '        Build and run the `my_project` service in a Docker container.'

There is @ before each echo because by default make prints every line to the console before it's executed. At sign is to suppress this and @ is discarded before line is passed to the shell.

But what if I wanted to run my application on different host and port using makefile? It's simple just add:

run:
    python manage.py runserver --host $(HOST) --port $(PORT)

Then you can simply run:

$ make run HOST=127.0.0.1 PORT=8000

Lastly be aware that indentation in makefile has to be made using TAB, not spaces.

What benefit you can have by using makefile in python projects

As you can see using makefile in python projects can bring many good things. If you are tired of writing complicated shell commands- put them under a rule in the makefile. Want other people easily run tests against your project? Put pytest calls in makefile. Ideas are endless.

Do you use makefile in your project? Do you find it usefull or maybe not? What else you put inside? Please write it in comments!

Cover image by JohnsonMartin under CC0.

To see comments and full article enter: Makefiles in python projects

My thoughts on blogging

This week I decided to write something more about what are reasons I wrote this blog, what I am taking from it so if you want some more technical post please check other articles.

What I want from this blog

I started this blog as a reference to myself about things in software development that I learn along the way. Moreover, this blog is a place to present my own opinions but I really welcome any comments especially ones that expand my view.

Why blogging?

I see couple benefits of blogging. First of all, you are learning. And I mean a lot- every blog post I write teach me something- either how to format my code in code snippets or implementation specific things. People who giving comments either via reddit or disqus teach me something. And I really mean that. Before I started writing content I knew that there will be people who are just haters and while I was thinking how to be invulnerable to such persons I came to the conclusion that every guy who left the comment on my blog wants to teach me something- I try not to take things personally.

Another thing is also connected with learning- your are learning not even how to code but you also learn how to write, to write in English. This language isn't my native one so I still have trouble with writing long passages of text in English but with every post, I make myself better.

The last thing with learning is my belief that the best way to learn something is to teach this thing to another human being. You have to know a big deal of subject if you want to write a good blog post and during writing, you are thinking how to present this topic in simple and consistent manner.

Conclusion and why I write this post

I made this quick blog post for the remainder to myself why I made this blog as well as for readers who may be interested in figuring out for what reason I write. But the most important thing is: I made this post because I don't have time this week to prepare proper blog post and I added some features to ease sharing content wrote by me.

Cover image by StockSnap under CC0.

To see comments and full article enter: My thoughts on blogging

Factory pattern in python

What is factory pattern all about? It introduces abstraction. In other words: helps software developers with the problem of creating objects without knowing exact class of the object that will be created.

Why use it? From programmers stack exchange:

(...) they allow the project to follow the SOLID principles more closely. In particular, the interface segregation and dependency inversion principles.

Factories and interfaces allow for a lot more long term flexibility. It allows for a more decoupled - and therefore more testable - design. (...)

  • It allows you to introduce an IoC container easily
  • It makes your code more testable as you can mock interfaces
  • It gives you a lot more flexibility when it comes time to change the application (i.e. you can create new implementations without changing the dependent code)

As we know why to use it let's jump into the code:

class BaseArchive(object):
    EXTENSION = None

    def __init__(self, location_path, files_to_pack):
        self.location_path = location_path
        self.files_to_pack = files_to_pack

    def generate(self):
        raise NotImplementedError()

At first, I created a simple base class to have a common interface for all archive classes. There is nothing sophisticated here: only setting up all necessary arguments in __init__ and telling that all children of BaseArchive have to implement generate method. Let's look how these children look like:

from zipfile import ZipFile
import tarfile


class ZIPArchive(BaseArchive):

    def generate(self):
        with ZipFile(self.location_path, 'w') as zip_file:
            for file_ in self.files_to_pack:
                zip_file.write(file_)


class TARArchive(BaseArchive):

    def generate(self):
        with tarfile.open(self.location_path, 'w') as tar_file:
            for file_ in self.files_to_pack:
                tar_file.add(file_)

Basically, ZIPArchive and TARArchive classes have generate method to create archives in given format using ZipFile or tarfile modules from standardlib as follows:

zip_archive = ZIPArchive(os.path.join(os.getcwd(), 'zip.zip'), ['for_zip'])
zip_archive.generate()
tar_archive = TARArchive(os.path.join(os.getcwd(), 'tar.tar'), ['for_tar.txt'])
tar_archive.generate()

Then the real fun begins. Say I want to generate archive just only by providing path and files to pack into the archive. For this purpose I use something called ArchiveManager which is factory:

class ArchiveManager(object):
    ARCHIVE_ENGINES = [ZIPArchive, TARArchive]

    def __init__(self, location_path, files_to_pack):
        self.location_path, self.extension = os.path.splitext(location_path)
        self.files_to_pack = files_to_pack
        self.archive_engine = self.choose_archive_engine()

    def choose_archive_engine(self):
        for engine in self.ARCHIVE_ENGINES:
            if engine.check_extenstion(self.extension):
                return engine(self.location_path, self.files_to_pack)

    def create_archive(self):
        self.archive_engine.generate()

Here I have ARCHIVE_ENGINES class attribute for every archive engine that I want to use. Below in __init__ I setup all necessary attributes with extension to use in choose_archive_engine. Here is the place where factory pattern starts to work: I iterate through all engines to check_extenstion and if there is a match I simply return the correct engine to self.archive_engine. To use that I need to write some more code in BaseArchive:

class BaseArchive(object):

  # rest of the code

  @classmethod
  def check_extenstion(cls,extension):
      return extension == cls.EXTENSION:

This check_extenstion that is classmethod helps me in figuring out which underlying archive class should I use. Thanks to that in ArchiveManager.create_archive I only need to provide self.archive_engine.generate(). This method doesnt't know if there is ZIPArchive or TARArchive class being used.

This was quick introduction how to use factory pattern in python. The code for this is available in this repo. Do you know more usages? Or maybe you don't agree with what I write- feel free to comment.

Edits (12.08.2016):

  • Refactor of check_extenstion method

Cover image by Tpsdave released into public domain.

To see comments and full article enter: Factory pattern in python


Page 1 / 7