Deviant Abstraction

I have recently relocated from a big city to a small town in the countryside. This lifestyle change pointed out several limits of the web application paradigm since I have now only a very limited Internet connectivity.

At first I tried to use web applications extensively for my private life: agenda, text processor, todo and so on…. But it turns out this is a far from ideal situation.

For instance:

• No confidentiality. I don’t want to expose all my data to outsiders especially on a wifi network. I send and receive sensitive data like my credit card number.
• Only online access. OK this one is obvious but my Internet access is limited and irregular. This is is the first time in more than ten years. It is destabilizing. How do I access my calendar even when there is no Internet around? Web application do not provide this.
• Silo effect. I usually process subset of my data with custom made scripts. Those scripts are not web aware. There are as far as I know no find or grep for the Web. Not even a universal and easy way to parse any HTML page from any computer language.
• Lack of customization. I have tailored my desktop to my needs and this is a huge value add for me. For instance, Evolution, Skype and Gaim are automatically launched at startup times, my SVN repositories are updated… Web application does not offer any way to configure them this easily. What I need is kind of NetVibes on steroids
• Integration with current application. Desktop applications are heavily integrated. For instance, I can load with my desktop computer my Ipod. I can not from web applications.
• Interactivity. I need speed and efficiency. This is the basis of interactive applications. Web applications (even Gmail) are slow and clumsy for obvious latency reasons. This is OK for email not for code editing.

One silly example that happened yesterday. I need to fill online my tax declaration. For this French’s government is generating a certificate. But no web applications allow me to store this certificate for me (and for good reasons).

You might object that I could have built a web application to take care of those needs, this Netvibes on steroid. Actually I started but I found in between a much more elegant solution: the use of a remote desktop system. One issue remains though. Its cost: a dedicated server is expensive.

(USB systems have also their own limit mainly “no background mode”.)

A virtual machine paid by the hour such as EC2 is perfect and nearly free (it could even be financed by advertisement if a company wanted to operate such a service). It is my Personal Virtual Machine (PVM). Some companies have started offering them for free (ie Desktop On Demand) but their offer is unreliable, slow and you cannot run all the applications you want/need.

In the end I installed KDE and NXE on my dedicated server (NXE is a great WAN remote desktop tool. Truly impressive). It solved all my problems really fast although it is costly (more than 30 euros per month). Marketing hype is on web applications but now we should start to explore alternatives especially if they empower users and are a cheaper alternatives (I can demonstrate it if needed). I can access it from my corporate PC, my cellphone or a cybercafe.

As a final note I am not saying that WebApp are bad. Just that they are not the universal panacea. Especially for lonely, interactive and heavily used applications. I will discuss this in more depth later. The PVM vision is not either the perfect solution but for heavy computer users such as myself it offers real advantages: no need for backup ever,r power consumption alway the lowest possible, you access your machine at your will without leaving it on.

The next steps is to be able to tie a virtual machine to a physical computer and then sends it back in the cloud. VMware system allows such trick. I will discuss this later in more depth too and I will tell you how it was to use this prototype for a month.

What do you think of this idea? Would you like me to explore those ideas more in depth?

WidgetAccelerator is the best way to accelerate the displays of your widgets in a webpage.

My blog uses it. It has two accelerated widgets. The display of the page has been increased of more than 10 times actually.

It caches the JS file and prints its content directly in the HTML page. Therefore, your visitors do not need anymore to gather a lot of different javascript files from a lot of various hosts. The main consequence is a speedier website.

It does not check for cache consistency before displaying the page. This is why it is fast (Most HEAD requests are as slow as a GET.)

This is also why you have to specify which widget to cache. Of course you should not use it for widget with highly dynamic content like a Javascript webstat tracker.

Installation
To use this script, you will need to.

1. Download the script (copy/paste this link and rename the extension of the file to .php)
2. Upload it to your server.
3. Create a writable directory to store the cached widgets.
4. Configure the script: two constants must be defined (at the top of the script). $CACHE_DIR is the path to the cache (ie: $CACHE_DIR="/var/www/cache/";) and $CACHE_TTL is the lifetime in seconds of the cached widget (ie: $CACHE_TTL = 3600;)
5. Include the script in your template or in the page you want to accelerate: $lt;?php include( "/PATH/TO/widget_accelerator.php" ) ?>
6. Accelerate the widgets you like. You have to change for instance this widget
   
<script defer="defer" id="snap_preview_anywhere" type="text/javascript" src="http://spa.snap.com/snap_preview_anywhere.js?ap=0&etc" ></script>

   to
   
<?php accelerate('<script defer="defer" id="snap_preview_anywhere" type="text/javascript" src=http://spa.snap.com/snap_preview_anywhere.js?ap=0&etc" ' ></script>');?>

7. Optional but nice for me add this quote to your template: powered by WidgetAccelerator
   which in html is: <a href="http://www.deviant-abstraction.net/index.php/2007/01/30/widget-accelerator/"> powered by WidgetAccelerator</a>

A little bit further
WidgetAccelerator looks for src=” or src=’ inside accelerate(). Without proper enclosing of this attribute, the accelerate function will print an error. Finally, all URL inside src must be in absolute to be accelerated.

In case of problem, features ideas or contribution, do not hesitate to mail me or leave a comment here.

Legal
Of course, this script is offered AS IS and I withhold all responsability for any damage that might come due to its use.

Next steps
If you like it, I will add support for images and predictive cache (to automatically set up the cache TTL).

I am building a small Ruby script to teach myself a few things about the language and solve a current issue I have. Here are the notes about the application I am building. All comments are appreciated.

My Use Case
My company has provided me with a very cool laptop and let me use Ubuntu Linux. However, it is really heavy (or I am not strong enough) and I don’t want to carry it back home and forward (or I would prefer to use my brain instead of developing my muscular masses).

I do not want both computers to be always on (waste of energy). I would like instead when I shut down one of them computer to send all my updated data on Internet and when I boot one to fetch all the new data from Internet.

The obvious solution is use SSH+Rsync but since I work in corporate trenches I cannot count on anything except my faithful 80 port.

My data daily update rate is pretty small. So S3 and its cheap hosting rate for low data ($0.15 per Go per month + $0.20 per Go transferred), might do the trick if and only S3 is used as a pivot storage medium (a bus actually). Only updated will be put on S3 and are deleted after consumption (yes it is a bus).

The initial sync might be expensive but it is a one time expense.

How to build it?
I don’t want to put in S3 all my data (mainly for cost reasons) but only the updated data. They will be inside two different computers. Besides, since it contains all my personal data, I would rather not overwrite some new data with the old one so extra-care must be taken for those. My 2 PCs will not be used at the same time. If two different updates set are sent at the same time, the application will quit.

We cannot count on clock since both computers can be in different time zone and could be out of sync. We could use S3’s time or a time’s server but using a logical clock is simpler and more efficient.

The grain of this application is file. Some optimization could be applied later there to send only the updated part(s) of the file. All updated data will be archived in one zip file.

S3syncer has two main actions: put and get

• Get will get all new updated data and set them up on the local computer.
• Put will take all data updated after the last get operations and will put them on S3.

Manifest
The manifest contains all the metadata of the zip archive. Since storage space is not an issue, it will be written in XML and put on S3 with the archive.

<manifest>
<version></version>
<logical_clock></logical_clock>
<emitter>MAC address or any UUID</emitter>
<new>
<file name=”…” path=”…”/>
</new>
<updated>
…
</updated>
<deleted>
<file name=”…” path=”…” archivedName=”can be different if file same name”/>
</deleted>
</manifest>

NB 3 classes of actions => space economy

File naming in S3
manifest<logical_clock>.xml
archive<logical_clock>.zip

Every field is self explanatory. The logical clock is a way to know for a machine if some updates have been missed. Basically it is updated with each put. We have n -> n+1 with n and n+1 logical timestamp and -> a relation for happened before.

The logical clock is present on each local computer and updated by a call to get. (It allows also to retrieve several different updates.)

It goes from 0 to 8 and then to 0 (mod 8).

Configuration file
Since we are in Ruby, we could use YAML serialization layer, but a simple Ruby class might be smarter although less user friendly than a XML file.

For the first version, a Ruby class will be fine.

Get in more depth
Get works that way:

1. Get the manifest file self.logical_clock + 1 (if do not exist: quit).
2. Check emitter != self (else quit since no update needs to be done)
3. Get archives
4. store it in a temp directory
5. apply updates on the local FS.
6. delete local archives. Keep manifest + store local times (to handle change of time)
7. delete files on S3
8. self.logical_clock ++
9. check for next manifest file

NB If logical clock do not exist => 0

Put in more depth
Put works that way:

1. Find all data updated using the timestamp stored. How? Using time… But error prone
2. check for manifest existence on s3. If yes: quit b/C error.
3. Create manifest
4. Send it to S3

NB Issues can exist if you change the timestamp after the get. In this case some files can be forgotten. This will be detected by checking no file has been updated 2 hours before the last get. If so, we will resend all the files.

Solve conflicts
The conflicts will be detected through the logical clock. In this case, the application would quit and the manifest file should be deleted.

The language used will be Ruby mainly because I like it and I am still learning it) with s3.rb. If I am courageous enough I will provide a Debian and an Ubuntu package.

Other uses

We can easily extend the system to support multiple computers and use it to sync a common directory (a little bit as Coda does).

References
http://townx.org/blog/elliot/thoughts_on_rsync_and_s3
http://www.amazon.com/gp/browse.html?node=16427261
http://search.cpan.org/dist/File-RsyncP/FileList/FileList.pm
http://samba.anu.edu.au/rsync/documentation.html

I have been selected for the Google Summer of Code application!

Yes. Yes. Yes.

I will be working on JackRabbit, a really interesting repository project based on the JSR 170 (the Java Content Repository API).

Here is the project: Jackrabbit Backup.

“Implement a tool for backing up and restoring content in an Apache Jackrabbit content repository. In addition to the basic content hierarchies, the tool should be able to efficiently manage binary content, node version histories, custom node types, and namespace mappings. Incremental or selective backups would be a nice addition, but not strictly necessary.”

I assume I will have a busy summer and a brand new Apple laptop (:D) at the end of the summer.

I will post here regular updates on how everything going… This will be useful at least for next year contestant.

Link to my application
Google Summer of Code

You have all heard about the famous homebrew computer club. The Silicon Valley is doing it again with cellphones this times. I don’t know if this initiative will be successful; from what I can tell I would like to attend. However it shows how much Silicon Valley is taking cellphones. A technology where for once Europe is in advance (for how long yet?).

Link to Wired’s article

As a student, I am applying this year to the Google Summer of Code. Best luck to all of us.