Google Reader Synchronization

As you might know I've been working on Google Reader Synchronization in Liferea for a Google SoC project.

Let me go through what has been done, and what has not.

What's new

A noticeable change has been the ability to items "Mark as Important", or in the Google Reader lingo, "Mark as Starred". Liferea will synchronize this flag to, and from, Google Reader.

Another cool feature that has been added is an efficient feed updater (Let's call this fast-update). So here's how it works: Every 10 minutes, Liferea will make a request to Google Reader asking for a list of modified feeds. The response is very small, and hence this does not affect your bandwidth. But from this small request, we can determine a list of feeds that have been updated, and so we can download exactly these feeds. This means that when a new post is available, you will get the update within 10 minutes! Every 24hrs, a full-update is done to complete the synchronization, because there are some situations where fast-update can miss out subscriptions.

What does not work

Comments do not work (as I have pointed out in the past, this is something I cannot fix). I decided not to implement Labels as Folders, simply because some users (who might have been more generous at tagging their subscriptions!) would not like it.

Synchronization with other Web-based aggregators

True --- I wanted to work on this. However, I couldn't get myself motivated enough, because I didn't think I will be using it. :-) I would have had to learn the intricacies of the new API. (Btw, the Google Reader API is *nasty*!) And plus, since I won't be using it, I won't be able to maintain it properly. (oh, excuses!) If you need help implementing synchronization for another web-based aggregator, I will be ready to help. As long as they provide a clean API, it should not be too hard.

That's all folks!

Although SoC is almost over, I will continue developing and maintaining the Google Reader code in Liferea. So you are most welcome to give me suggestions at any time.

Video Bug Reports

Today I found this video a Liferea user created and posted in his YouTube video channel.:



It describes two problems with the current handling of the 'updated' item state:

1. Changes are not persistent. There is a bug preventing correct saving of the state change into the DB.
2. Many users confuse this state with the item read state and wonder why "Mark All Read" cannot be used to reset the 'updated' state.

While the first problem is a functional one and could be fixed I still decided to solve everything by removing the current 'updated' state UI (the icon in the item list). The reason is that I believe it to be of low value to most of the users, to be not inituitive when distinguishing it from the 'read' state and to be visually disturbing if you do see it in the itemlist. Just too many disadvantages and it removing it will save code, documentation and support efforts.

If you got ever confused by this feature starting with 1.4.19 this will be fixed.

How to run VACUUM

As explained in the last post I see no way to automatically run the "VACUUM" command of sqlite which more or less defragments the DB structure. Nonetheless for everyone who wants to run it manually here is how to do it:

1. Shutdown Liferea
2. Start the sqlite client by running: "sqlite3 ~/.liferea_1.4/liferea.db"
3. At the prompt enter: "VACUUM;"
4. Wait until the prompt reappears.
5. Restart Liferea

Situations when you might want to VACUUM

• When the DB file (~/.liferea_1.4/liferea.db) is very large (e.g. >50MB)
• When you have only a few feeds with a low cache setting (e.g. 30 feeds and 100 items) and believe Liferea to be unreasonably slow.
• When you have run Liferea for ages.

If you don't know what this is all about: please do not worry about it. In many cases you might not need to do anything.

Why auto-VACUUM is no good...

During the various performance discussions during the last time here and there people suggested to run "VACUUM" on the Liferea database once it gets slow. This is in line with the sqlite documentation which says:

When an object (table, index, or trigger) is dropped from the database, it leaves behind empty space. This makes the database file larger than it needs to be, but can speed up inserts. In time inserts and deletes can leave the database file structure fragmented, which slows down disk access to the database contents.

The VACUUM command cleans the main database by copying its contents to a temporary database file and reloading the original database file from the copy. This eliminates free pages, aligns table data to be contiguous, and otherwise cleans up the database file structure.

The problem with it is that it also takes very long. With a 50MB DB file I experienced a runtime of over 1 minute. This is why this can be only a tool for experienced users that know how to do it manually knowing what to expect. For executing such a long term operation automatically on runtime would surely be unacceptable to the unsuspecting user. Also there is no good way how to decide when to do a VACUUM to save disk space and improve performance.