Ian Lewis' Blog

Using the Django sitemap framework is so easy it's almost no work at all. Just make a sitemap object and add it to the sitemap in urls.py. The sitemap framework calls items() in your Sitemap to get the list of objects to put in the sitemap and then calls get_absolute_url() on each object.

models.py

sitemap.py

urls.py

You can even generate sitemap indexes and it will pagenate the indexes on Google's limit of 50,000 urls so that you don't have a problem with it crawling your indexes.

For my new project dlife, I went about implementing a simple comments interface that would allow users to make comments on imported feed items. I wanted to support this in the admin in the typical manner such that when you click on an item in the admin, you can see all the comments and edit them from the item's page.

I found that you can use inline forms in the admin but it seems to show a bunch of forms (3 by default) even though I don't have any comments for the item yet. I'll mess with this a bit more later to try to get the behavior I want.

models.py

admin.py

Over the weekend at Python Onsen I worked on a lifestream web application using Django and feedparser. I was really impressed with how simple feedparser is to use and how easy it is to get unified results from atom or rss feeds. You simply import feedparser and call feedparser.parse to parse a feed from a url.

You can check out feeds.py here.

The interesting bit comes with how I had to parse the dates which sometimes include timezone info and other goodies. In my search for a solution to the problem of how to deal with dates in various formats I turned came across this blog entry which describes the problem and some possible solutions. The solution I used was the simplest and most robust (please skip the comments talking about taking a slice of the date string). I used mikael's suggestion from the comments and used the dateutil.parser to parse the date string into a proper datetime object.

This however leaves timezone info on the timestamp which isn't supported by mysql so I hand rolled some code convert the timestamp to utc and remove the timezone info.

I'm not sure this works in all situations yet so I might go with something like how another commenter solved the problem by converting feedparsers parsed date to a utc timestamp before converting to a datetime object. I think either way would work but which is cleaner and less prone to breakage, I'm not sure.

Last weekend I went to my second Python Onsen[jp] organized by Nakai-san(id:voluntas). I talked about Python Onsen in my first blog post here. Python Onsen is a 3 day event (Fri, Sat, Sun) but as before I only participated on Saturday and Sunday. This time I opted to work on creating a lifestream web app using feedparser and Django. feedparser is a snappy little parser for reading RSS and Atom feeds. The result was dlife which so far can parse a set of feeds and show them on a user's lifestream though it's not in any way user friendly yet (you have to update the feeds in the django shell ).

I also got to know my soon to be co-worker, Okano-san (id:tokibito[jp]), by talking about jQuery Internals' data() function and web/Django development.

Here's a recap:

• Worked on a sweetcron lifestream replacement in Django (dlife)
• Onsen is pretty lonely by yourself.
  
• Introduced id:tokibito[jp] to the jQuery Internals' data() function
• Since I come on Satruday, I always miss introductions so I never know who is who.
  
• feedparser is really simple and easy to use. Though I'm not sure what I'll do about pictures and video yet.
• No one mentioned my blog or linked to it in their posts   (Maybe because I never write anything? )
• In a raffle、I got a cool Python shirt from Accense Technologies'[jp] Masuda-san(id:whosaysni[jp]).
  
• It was lonely going home by myself from Kinomiya station.

I just released a new version of jsonschema 0.2 alpha over at http://code.google.com/p/jsonschema

The source can be downloaded here: jsonschema-0.2a.tar.gz
The documentation can be found here: jsonschema (version 0.2a) documentation

The new release includes the following notable changes.

• The additionalProperties attribute is now validated.
• Using schemas in the type attribute now works properly.
• Changed support for unique attribute to the "identity" attribute (Note: this is not a backwards compatible change)
• Fixed a bug where the original schema object/dictionary was modified by the validator
• Added a new "interactive mode" which will add default values to objects if not specified as readonly by the schema
• Made error messages a bit more friendly.
• Fixed bugs with validating Unicode strings

The additionalProperties attribute is used to define the format of additional properties that aren't explicitly specified in the properties attribute. This is useful for json like

where you have some things like game scores and the name of the attribute is someone's name which can't be defined in schema. You can use it like so:

The type field was also fixed so that it handles adding schemas as types, so now you can define,

This can let you define more complex types for use in schema.

Kris Zyp (the author of the JSONSchema proposal) mentioned jsonschema on his blog at json.com. Thanks Kris!!

I just released the first version for a project that I've been working on since the Python Onsen. It's a validator for JSON Schema written in Python. It's based on the JSON Schema Proposal Second Draft.

The source can be downloaded here: jsonschema-0.1a.tar.gz
The documentation can be found here: jsonschema (version 0.1a) documentation

JSON Schema's purpose is to allow validation of JSON documents much like XML Schema, DTD. You can use it to define what kind of data should be present in the document as well as the structure of the data. You might have some JSON for a contact like so:

And you could describe this in JSON Schema with the following:

This can be validated with something like the following Python code:

It can be easily extended to include support for new properties or to override the default validation for standard properties so I think it could be used for a wide range of applications. I plan to use it for a Form Maker application (code) on GAE. Let me know what you think!

I was thinking about using Django for one of my projects on GAE because it seems like a popular project and somewhat easy to use, but I'm not quite understanding yet why it's better to have helper functions rather than controller/handler classes like Pylons or GAE's normal WSGI handling has. With handler classes my controller might look like:

Whereas the django helper function might look like

While the Django method might have the potential to have be a bit less verbose it feels like it would be harder to do things correctly, like factor code etc. I also don't really like the conditional checks to see what kind of HTTP method was used. So either I would need to split GETs and POSTs to separate urls or just live with the conditional checks.

Personally I feel better with the Pylons-ish controller/handler approach. Anyone have an opinion?

A few days ago Protocol Buffers was released by Google as an open source project. Protocol Buffers is a way to generate code for objects that can be serialized to and de-serialized from the protocol buffers binary format. An implementation of the protocol buffers compiler which reads a "proto" and can output Java, Python, and C++ code. Because the format is a binary format and the compiler can output in several languages, this would allow for fast message passing between applications that may or may not be implemented in the same language.

I went ahead and created two simple Java and Python applications that test protocol buffers by simply creating a simple User object, prompting the user for a nickname and e-mail and then serializing the result. I also added a command to de-serialize the output and show the contents of the reconstituted object.

Here I ran the Java program to write the protocol buffers data to a file and then I run the python program to read it.

ian@laptop:~/src/protocol-buffers-test$ make javawrite
mkdir -p java
protoc --java_out=java/ prototest.proto
javac -classpath "java/:protobuf-java-2.0.0beta.jar" java/test.java java/prototest/Prototest.java
java -cp ".:java/:protobuf-java-2.0.0beta.jar" test write test.out
Nickname: Ian <- this is input
Email: test@test.com <- this is input
土 7月 12 19:09:28
ian@laptop:~/src/protocol-buffers-test$ make javaread
java -cp ".:java/:protobuf-java-2.0.0beta.jar" test read test.out
User
Nickname: Ian <- this is output
Email: test@test.com <- this is output
土 7月 12 19:09:33
ian@laptop:~/src/protocol-buffers-test$

Code available here. Code listing follows.

Read more »

This weekend I went to the Python Onsen (Japanese) organized by voluntas. Python Onsen is an event where people who like or are interested in python get together at a Japanese Ryokan/Onsen and program/mingle/study together. The event started Friday but I had to work so I joined everyone yesterday. If you aren't famaliar with the Ryokan experience check out the Ryokan link. Essentially you have a traditional style room and traditional meals are served twice a day (with generous proportions).

In between meals there was a lot of programming and talk about programming. I was recieved pretty well considering that I was the only non-Japanese in the group of 28 or so people. I spent the time here working on a form maker project for google app engine which will be using JSON quite a bit for server communication and API interfaces. It is programmed on the client side using the google web toolkit and it during the course of development it became clear that there would be a need for a way to validate incoming JSON on the client and server (for error checking and security) as well as making the interface easier to deal with. Currently the typing of the JSON data makes dealing with it in Java a real pain.

We realized this could be done with a schema, kind of like XML Schema. Something that could be used as a way to define the structure of the JSON and thus allow programs to validate it. So after searching a bit we found the JSON Schema proposal. JSON schema is maintained in JSON and can be maintained inline so if it is, it doesn't solve any security issues, but it looks like a good way to validate and do error checking on JSON data that might be coming from an external (or internal) source. So one programmer whipped up a simple validator in python which I will hopefully be working on and using on the server side of my application while I'll be going ahead and creating a clent side schema and JSON parsing library over top of, or separate from, the existing JSON library for the google web toolkit.

Pretty good for a two day hackathon.

Google Developer Day Japan 2008 is being held on June 10th at Google's offices in Shibuya and I've registered to attend this year. There were a number of sessions that people could take part in but I decided to register for a Google appengine hackathon. I'm pretty curious about appengine since I've been working at becoming more familiar with really newly evolving technologies and not necessarily ones that have been around a while. Newly evolving technologies is something I've always felt I've had to catch up on since starting programming in high school. Going to high school with folks like Bob Ippolito (Mochikit, simplejson) and Konrad Rokicki who started coding stuff when they were in early middle school didn't help my self esteem.

Anyway, in the spirit of learning about Appengine I took a dive into the documentation and learned a few of appengines silly limitations but I came up with a simple application that utilizes the simple python library I created for prefix back in college. I put it up in my mercurial repository under prefix-appengine if you care to take a look.

The main work is done in two handlers which are essentially the controller part of the MVC pattern. One simply renders the page as a template, which is really simple since there isn't any template code, and the other implements a simple rest API that I use for an AJAX call to evaluate an expression given by the user. Using JSON seemed like a waste since there was only one returned value.

The rest of the code is in the javascript which I just wrote strait into the template file because I was lazy. The javascript uses jquery to do an AJAX call when the button is pressed and update the HTML DOM.

For what it's worth I started a new project on sourceforge for my past project prefix and I released a new version. It's a prefix (polish notation) command line calculator. It's kind of an exercise to learn how to develop and distribute software in python. It's also a useful little utility. I use it to do quick calculations when I'm feeling averse to pushing buttons. Anyway, I should hit the sack.

It's late and I've got more work in the morning.