Ian Lewis' Blog

I went to Google Developer Day 2008 in Yokohama Japan yesterday. The keynote speech was pretty much the exact same info as was given at the keynote at Google I/O where Google announced their direction, moving forward the web as a platform.

Keynote

As with the Google I/O keynote it was mentioned how Google feels that Computing power and accessability have kind of flip-flopped over the years. In the mainframe era you had computing power but no accessability, in the PC era you had accessability but lost relative computing power, and now in the web era we are getting back computing power in the form of cloud computing but we are loosing accessablity to those resources. They plan on fixing this with the, so called, three Cs. Client, Connectablity and Cloud.

The first refers to the browser, so Google wants to make the browser richer in order to give us accessability to the computing power that they can provide. They are doing this with Google Gears and some other handy browser plugins.

Connectability refers to allowing everyone equal access to resources and making sure every one can connect. This means making sure that internet lines are fast, airwaves are open etc. They see mobiles as big in the future so they hope to help the connectability problem with Android, their free, open operating system for mobiles.

Cloud refers to their vast data centers. They hope to give access to these resources through products like Appengine where developers can access the vast resources and scalability that Google's data centers provide.

Appengine Hackathon

In the afternoon I attended the Appengine Hackathon which was presided over by Brett Slatkin, who is none other than the guy in the Appengine demo video. It was interesting because from the e-mails I recieved about the event, I figured it would be in Japanese but it ended up being entirely in English. Many of the Japanese folks had trouble following along so I tried to help where I could.

In the beginning, Brett talked about Appengine and used an example wiki as a demo app. Then we went into coding our projects. At the end some folks showed off their applications. Despite the language barriers many folks came up with some really original, and cool ideas. The first was created by a Google engineer, who said he would set the bar low but ended up with one of the better applications. His app read calandar events from RSS and allowed users to add comments to it. He also implemented memcache support. There was an application with the idea to attach pictures based on the hostility/mood level of a chat message or Twitter tweet. There was a social bookmarking app, and an app to allow live translating of a django application.

For what it's worth I presented my application which I hope to make into a workable form application builder. I haven't uploaded it yet so you'll have to make do with my first Appengine application, a prefix calculator with a simple rest api.

Dinner

Afterwards I went out to dinner with a number of folks who participated in the Hackathon. It turned out to be a lot of fun and I made a lot of new friends many of whom are now in my twitter contacts All in all a hugely satisfying experience.

Google Developer Day Japan 2008 is being held on June 10th at Google's offices in Shibuya and I've registered to attend this year. There were a number of sessions that people could take part in but I decided to register for a Google appengine hackathon. I'm pretty curious about appengine since I've been working at becoming more familiar with really newly evolving technologies and not necessarily ones that have been around a while. Newly evolving technologies is something I've always felt I've had to catch up on since starting programming in high school. Going to high school with folks like Bob Ippolito (Mochikit, simplejson) and Konrad Rokicki who started coding stuff when they were in early middle school didn't help my self esteem.

Anyway, in the spirit of learning about Appengine I took a dive into the documentation and learned a few of appengines silly limitations but I came up with a simple application that utilizes the simple python library I created for prefix back in college. I put it up in my mercurial repository under prefix-appengine if you care to take a look.

The main work is done in two handlers which are essentially the controller part of the MVC pattern. One simply renders the page as a template, which is really simple since there isn't any template code, and the other implements a simple rest API that I use for an AJAX call to evaluate an expression given by the user. Using JSON seemed like a waste since there was only one returned value.

The rest of the code is in the javascript which I just wrote strait into the template file because I was lazy. The javascript uses jquery to do an AJAX call when the button is pressed and update the HTML DOM.

I implemented tracking using Google Analytics for my company's mobile sites using a technique described by Peter van der Graff on his site. The technique involves performing a GET to to an image on Google's server and passing it a bunch of options. Incidentally this is because Javascript can perform gets of images but not gets for any other kinds of content (as an aside, this kind of protection seems usless since the server could return any kind of content in wants to the javascript even though the GET has an image in the url. Maybe someone could enlighten me).

Peter originally came up with the idea because he wanted to track hits to a RSS xml url (which also seemed strange to me since the rss aggregator could read it as many times as it wants and doesn't give much insight into the number of readers, but I digress), or to another type of file download (image, pdf, etc) which wouldn't trigger the javascript that Google uses for Analytics.

One important difference between his motives and mine were that I'm tracking hits to a mobile site. Doing analytics on the server side are important since most phones (in Japan at least) don't support javascript. I also, because of the differences in what I was doing, needed to make some changes to how his script worked. Since I'm not tracking downloads or rss hits, I care about things like sessions, language, and user agent (why Peter didn't also care about this I'm not sure).

So I modified his code as follows. I forward the language and user agent of the client to Google Analytics so that I can track these things properly. I also pass my own cookie number so that Google Analytics can aggregate page hits from the same user into a session. I also make use of the user var to track hits to different customer's web pages. The example is in PHP but it could be easily translated into another language.

Note that, because of the use of stream contexts, this code will require a version of PHP >= 4.3.0.

I read today on the Official Google Blog that Google documents has offline support using Google gears but unfortunately it's perhaps not released to me yet. Unfortunately it sometimes takes time for new versions to get out to everyone and it takes even longer before they are supported in different languages other than English.

One thing that many people might not realize is that features often take a good amount of time to be released in other languages other than English. My Japanese version of Gmail still doesn't support greasemonkey scripts. I think that feature was released many months ago. Google Reader in Japanese still doesn't support showing your the shared items of your contacts or Google gears. Another feature that was released many months ago. Google docs tends to be pretty decent about getting the newer versions released in Japanese but Gmail has been particularly lacking. I hope they continue that with this new version.

I took a look at the Google summer of code mentoring organizations for 2008 and was surprised to see BBC research listed. They were probably previous participants but it seems that BBC maintains a number of open source projects on their open source page. The most notable ones seem to be the ones that they are going to use the interns for, Kamaelia, which is apparently a test system for large scale media distribution, and Dirac, an open source video codec. All in all, some pretty cool stuff.

I just set up my e-mail settings with Mercurial so that I can e-mail patches via my Gmail account. I have Debian installed on my machine which has exim installed by default so it was pretty easy to set up. I'm not terribly versed at setting up mailing agents so I basically followed these instructions on the Debian Wiki. After getting that set up it's easy to set up Mercurial to use exim4 since it's a drop in replacement for sendmail.

To set up Mercurial to use exim I followed the instructions on the Mercurial Wiki:

email::
...
method;;
    Optional.  Method to use to send email messages.  If value is
    "smtp" (default), use SMTP (see section "[smtp]" for
    configuration).  Otherwise, use as name of program to run that
    acts like sendmail (takes "-f" option for sender, list of
    recipients on command line, message on stdin).  Normally, setting
    this to "sendmail" or "/usr/sbin/sendmail" is enough to use
    sendmail to send messages.

  Email example:

    [email]
    from = Joseph User <joe.user@example.com>
    method = /usr/sbin/sendmail

So here is my very simple ~/.hgrc file:

[ui]
username = Ian Lewis <IxxMLxxxx@gmail.com>
[email]
from = Ian Lewis <IxxMLxxxx@gmail.com>
method = /usr/sbin/exim4

Simple. Now I just enable POP for my gmail and I can use hg email and it will go through my gmail account. Now only if the Mercurial guys would fix this issue so I can send the patch email with the correct encoding.

A couple of days ago Google released some new themes for iGoogle. They also mentioned that they have a new API for creating themes for iGoogle and that anyone can create themes. This is a bit of a misnomer because you can basically create a theme using the API and test it using the testing interface but it won't be added to iGoogle unless it's good and you did you homework and used the API properly.

I'm partial to the John Maeda "Simple is Complex" theme. He's pretty well known. I saw his talk on Ted (which is full of inspiring talks. PLEASE check it out) and I like his work, though some of his work seems a bit trendy and he did come of as a bit of an unorganized speaker (not that I have anything on him though ). He also needs to fix his javascript so it works in Firefox (click the dropdown).

I also liked the Yves Behar's "Earth-light" theme. Though it seemed to not work properly for me in Japan. It's supposed to show you where the sun is but strangely it says that it should be high noon when it's 7:00 at night here in Japan (check the bottom right corner in the screenshot). It almost seemed like it was showing the sun as if I was in California! Maybe it's because the author and the Google headquarters are both there? Didn't they bother to have someone test it in another timezone?

 

I came across this website called Zoho which has a lot of interesting web apps like Google's web apps. Google has Google docs, spreadsheets, and presentations. Zoho has docs, spreadsheets, and presentations. Google has Google notebook. Zoho has a notebook. Google has gmail. Zoho has Zoho mail. Google has... well you get the idea.

All that stuff is well and good but the thing that caught my eye was the Zoho creator and the Zoho DB & Reports. Granted, Google has it's own things like a neat little chart api(on the right) but nothing like Zoho's DB and integrated report creator that I know of. Google Base seems built for an entirely different purpose. Granted that the Zoho DB and Reports are not going to rival any enterprise reporting software but would provide you with an easy to use reporting tool for small to <the step above small (not medium)> sized databases.

Anyway, the possibilities seemed interesting for an online set of integrated applications like Zoho has. Especially the DB, App creator, and Spreadsheets but I'm sure you could think of powerful ways to integrate the other things like TODO and e-mail. The problem is that Zoho hasn't integrated these (yet?). The Zoho creator, though billed as a way to "Create Database Applications", is not integrated with the DB. When you create an app, and with it, a form and a view in the creator, the database and tables don't show up in the database app. *scratches head* Isn't integrating these two obvious? It would be cool to have an application with a database that you could then load in Zoho Database and create reports etc. But alas, the creator is a stand alone.

The integration issues combined with bugs in the creator when saving data, when saving text other than english, and when accessing using a Japanese phone, all are pretty much show-stoppers for me using it. It seems like it's so close, but missing some critical pieces. They may be adhering to the release early, release often philosophy and they are definitely sticking to the "release early" part. I just hope they stick to the "release often" part.

Recently I've noticed that my website has been getting slow recently and, sure enough, looking at Google Webmaster Tools the access time for my site has gotten pretty chaotic and much slower in the last couple months. Currently I've used Hasweb for 4 years for hosting and I've never had trouble with them in the past and I'll raise the issue with them, but if the current problems continue I might need to look for another hosting company.

I recently found the 60 minutes podcast on Google Reader. The postcast contains full length episodes of 60 minutes and I think it should be a good resource particularly for me in Japan.

http://www.cbsnews.com/common/includes/podcast/podcast_60min_1.rss

Recently I added a small applet which displays news stories that I am sharing from Google Reader. With Google Reader you can share news stories and blog entries that you think are important publicly. I thought it was pretty cool so I added it to the page here.

The news stories that I tend to share will probably concern three topics, Environmentalism, Technology, and Japan related news. If you have a chance, check it out.

Somewhat recently, Google released their online version of Microsoft Word and Excel. Essentially, it is a document editor that can import and export various formats and allow editing of documents on the web. I originally thought that the AJAX approach of Google Docs could never touch the feature set, speed, and usability of a native application. After all, documents can be big, and native applications like Word, and Open Office have huge feature sets that couldn't possibly be replicated easily and efficiently on the web using AJAX.

So I thought. But in reality Google has replicated many of the important features in Word and Excel already with surprisingly good results. But even still that's missing the point. The point is being able to manage all of the documents for you and/or your organization collaboratively online. And then once finished being able to publish the results to the web. The power of such an idea only hit me fully today. Once you publish it on the web, your users and customers can view it. If you were able to integrate this functionality into your organization, namely writing documents online collaboratively and publishing the results to the web, think of the things you could do with such functionality. You could release financial data and reports this way. You could release and manage revisions of software documentation this way. Many things are possible with this approach.

Managing revisions of anything is a hard task. Maintaining what versions of the software are installed where and what versions are published is an annoying and hard task to manage. This sort of approach would make it easy. Publishing documents on the internet this way could allow customers and/or users to view new documentation or new versions of documentation from your organization's website instantly. I feel that, more and more, integrating standards based applications to create working systems that help people manage their businesses (including publishing) and access data from anywhere via the internet is where the future will be heading.

I'm not sure how much of all the great new things Google is putting out were the result of Google's own inventiveness or whether Google simply read the writing on the wall (which is not such an easy task itself) but it's exciting to see the new companies direction first hand as many of the applications are available for free for essentially non-commercial use.

Google of course wishes to remain an innovator but history has shown that companies, just like people grow up and gain responsibilities (to employees and customers) like people gain responsibilities to their families and that the innovation really rests in the hands of those who are young and free to explore their whims. I just hope that Google can maintain it's innovation well into the future. It's quite amazing to watch.

I've used Gmail for my personal mail for almost 2 years now. But I had never had a phone where I could access the internet from the cell phone until I came to Japan. So I was delighted to know that Gmail has a mobile feature that allows me to read e-mails from the cell phone.

However, it's not all Samurai's and Green Tea here in Japan because Gmail doesn't work on Japanese mobiles. In fact almost none of Google's services work on the cell phone. Google has a news reader that you can access from your mobile phone as well. But that doesn't work either. The simple search seems to work ok but any service where you need to log in fails and Google doesn't support Japanese phones.

Each service complains that you need cookies enabled in your browser in order log in. Though cookies are enabled in the browser and other Japanese sites like Mixi have no problems logging you in.

I tried searching the internet for possible causes and solutions to the problem and I came across a few websites but nothing seemed to work. This site seems to suggest that at first there is a possibility that it would fail but if you login again it would work. But nothing seemed to work for me and gmail seems to spend you to a page that isn't for mobiles.

I suppose all of this could be my phone's lousy browser implementation but lots of phones in Japan seem to have the same problem. I suppose the lack of resources in Japanese, not supporting Japanese phones and not having services useful to Japanese (like train schedules etc.) is why people in Japan tend to use Yahoo and not Google. Yahoo just serves Japanese people better. And besides the messenger has cuter smilies

Just recently Google Calendar has allowed calendar users to display their calendars publicly. This has been something I've been looking forward to since now people I know can see when I'm busy. You could do this previously with Yahoo Calendar and was lacking in Google's version.

Though Yahoo allowed users to publicly publish calendars, Google went a step farther and allowed users to imbed calendars in webpages, so I went ahead and imbeded it here. It uses a >iframe< tag to imbed the calendar in the webpage. Which I'm having trouble getting to render well in Internet Explorer. In fact I'm having trouble trying to get Internet explorer to not lock up when browsing the imbeded calendar. Strangely enough, if you press the forward arrow on the calendar to move forward a month, it seems to do ok. But, if you hit the back arrow, it locks up Internet Explorer. Internet Explorer is also not rendering the width of the calendar correctly. It seems that it's making it too wide within the box that it's contained in. Firefox of course renders it perfectly.

Anyway, the imbeded calendar is pretty cool. If you are logged into google calendar it even shows it with the correct permissions.

Today I'm messing with Google Calendar. I've been using Yahoo's Calendar which is nice and has features for everything I use and ties nicely into Yahoo Groups. It has the capability of providing 2 reminders, which I like. These reminders can come to you as e-mail or go to your cell phone. You can provide a location, along with other data such as a phone number and location name. You can schedule repeating events using odd schedules like "The second Friday of the Month" (which works really well for our Japanese Meetup Group because it ensures that the repeating event happens on the same day of the week every month), or 'Every other day' or 'every third day'. You can enter multi-day events, and it handles them ok, creating a new box for each day. So a three day event looks like 3 identical one day events.

Google Calendar doesn't allow you to put in as much data as Yahoo groups. You can put things like directions, information, or a phone number in a description but it doesn't show unless you look at the event details. It allows you to put in the location of the event which is normal. But I've found that if I put in the location name in with the address of the location, sometimes it doesn't open up the location correctly in google maps. It also doesn't support 2 reminders, and the reminders seem to come up as a popup by default when I have the calendar open rather than as an e-mail. It's also not integrated with Google groups, which is in-and-of-it-self designed as just a mailing list, and doesn't hold a candle to Yahoo groups which integrates a calendar and photo album along with the mailing list for your group. Sometimes it's also a bit unresponsive and gives you no indication that it's doing anything because it's all AJAX.

That said, when comparing the two Google Calendar looks much better. There are no ads. It handles multi-day events well, displaying them as only one box, showing the title only once. It also allows you to repeat a meeting on Mon/Wed/Fri, or Tues/Thurs which Yahoo doesn't do. This is good for me as I have Tues/Thurs events. It also would be easy for students I imagine. Though it does have a weekly option with checkboxes to select the days of the week to repeat on. It also allows you to change the time and date of the event easily using drag-and-drop and normal gui (however, I'd add that events for me don't change that much and it's no problem to go to the event to change it. Drag-and-drop is more likely to be done accidentally than on purpose for me). Another thing that it does that Yahoo doesn't do is display Japanese/UTF-8. That's a big plus for me.

Overall, there are things I like about both but Google wins for it's good looks and ability to do everything I want. But even though I like Google Calendar, it seems like a work in progress and I'm looking forward to new versions of the calendar.

So, I figured out what the problem was with my Google Talk build. I did a bit of debugging and found that the nss-ssl plugin wasn't loading properly because it was getting a link error:

libsoftokn3.so: undefined reference to `PR_GetLibraryFilePathname'

A little looking revealed that this function is included in libnspr4.so. Running ldd on my nss-ssl.so in /usr/local/lib/gaim revealed the following:

libglib-2.0.so.0 => /opt/gnome/lib/libglib-2.0.so.0
libnss3.so => not found
libsmime3.so => not found
libssl3.so => not found
libsoftokn3.so => not found
libplds4.so => /opt/gnome/lib/libplds4.so
libplc4.so => /opt/gnome/lib/libplc4.so
libnspr4.so => /opt/gnome/libnspr4.so
libpthread.so.0 => /lib/libpthread.so.0
libdl.so.2 => /lib/libdl.so.2
libnsl.so.1 => /lib/libnsl.so.1
libc.so.6 => /lib/libc.so.6
/lib/ld-linux.so.2 => /lib/ld-linux.so.2

Yikes! Some libraries aren't found and somehow I've got an old version of libplds4.so, libplc4.so, and libnspr4.so and it's linking to them! So I add my /usr/local/lib/mozilla to ld.so.conf and removed the 3 libraries in /opt/gnome/lib, run ldconfig, and poof!! Google Talk works on gaim!! I didn't even need to recompile. How's that!?!

After going back and looking at the debug log for my Windows build of gaim at work I noticed that google does send a set of mechanisms that is the same but the response that the Windows build sends is different. The response causes google to re-send a set of mechanisms which has the PLAIN mechanism added.


Sending:
<?xml version='1.0' ?>

<stream:stream to='gmail.com' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
Recv:

<?xml version="1.0" encoding="UTF-8"?><stream:stream from="gmail.com" id="EC3C8EBE" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features>

  <starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>
  <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">

    <mechanism>X-GOOGLE-TOKEN</mechanism>
  </mechanisms>

</stream:features>
Sending:

<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
Recv:

<proceed xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>
Sending (ssl):

<stream:stream to='gmail.com' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
Recv (ssl):

<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="EC3C8EC4" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">

<stream:features>
  <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">

    <mechanism>PLAIN</mechanism>
    <mechanism>X-GOOGLE-TOKEN</mechanism>

  </mechanisms>
</stream:features>

Sending (ssl):
<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='PLAIN'>

***auth string edited out***</auth>
Recv (ssl):

<success xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/>


So it appears that something is wrong with the tls and ssl in my build at home since my build at home doesn't send the starttls tag back to the server alhough I have both ssl support and mozilla nss support compiled in. In fact the about window, which shows what support was compiled in, for the windows build shows almost identical information as my build at home. So I'm still a bit clueless as to why it's being wierd.

I tried to get Google Talk to work with my build of gaim 1.5.0 yesterday but it didn't really play nice. I get a cool error message, "Server does not use any supported authentication method"

I was able to connect at work so I thought that not being able to connect from home was a bit strange. After ensureing that I compiled in support for SSL and correctly linked the mozilla NSS libraries I took at look at the debug dialog in gaim. Since Google Talk is based on Jabber I could look at the protocol and I noticed that at home Google Talk's server sends a stream tag that only contains one authentication mechanism called X-GOOGLE-TOKEN but at work it had two, X-GOOGLE-TOKEN, the method the standard Google Talk client uses, and PLAIN, which is what gaim expects.


<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="EC4FFD5B" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">

<stream:features>
  <starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>

  <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
    <mechanism>X-GOOGLE-TOKEN</mechanism>

  </mechanisms>
</stream:features>



instead of


<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="ED36A600" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">

<stream:features>
  <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">

    <mechanism>PLAIN</mechanism>
    <mechanism>X-GOOGLE-TOKEN</mechanism>

  </mechanisms>
</stream:features>



I currently have no idea why it's doing that but hopefully I'll figure it out soon. It's not a high priority or anything I just have 1 friend so far that's using it. :p