Custom URL support for Upstream Tracker Project

One of the features available on the Upstream Tracker project is the ability to process custom URLs to track upstream package versions. This allows URLs to contain regular expressions which are processed using the re module in python. To demonstrate this feature, let me post a few sample inputs and outputs that were generated using this module.

Custom URL : http://coherence.beebits.net/download/Coherence-(.*)\.tar\.gz
Coherence-0.6.6.2.tar.gz http://coherence.beebits.net/Coherence-0.6.6.2.tar.gz
Custom URL : ftp://ftp.gimp.org/pub/gimp/help/gimp-help-(.*)\.tar\.gz
gimp-help-2-0.11.tar.gz ftp://ftp.gimp.org/gimp-help-2-0.11.tar.gz
Custom URL : http://code.google.com/p/giver/downloads/list/giver-(.*)\.tar\.gz
giver-0.1.8.tar.gz http://code.google.com///giver.googlecode.com/files/giver-0.1.8.tar.gz
Custom URL : http://sourceforge.net/api/file/index/project-name/libyui/rss/libyui-(.*)\.tar\.gz
libyui-gtk-2.21.96.tar.gz http://sourceforge.net/projects/libyui/files/libyui-gtk-2.21.96.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-(.*)\.tar\.gz
dee-1.0.10.tar.gz https://launchpad.net/https://launchpad.net/dee/1.0/1.0.10/+download/dee-1.0.10.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz
dee-0.5.22.tar.gz https://launchpad.net/https://launchpad.net/dee/0.5/0.5.22/+download/dee-0.5.22.tar.gz

From the above examples, we can see that the module supports projects hosted on sourceforge as well as projects where the download pages are accessible via HTTP and FTP. It is also possible to narrow down and search for only 0.x branch of a package using suitable regex as shown in the last example.  The module tries to implement complete regex support and does this with the help of the python re module. However, to identify if a term has to be evaluated using re, we look for parenthesis. Thus, regular URLs with parenthesis would be treated as regex terms and would be processed accordingly. For example, consider the URL below :

https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz

Here, we list all the links present at https://launchpad.net/dee/+download/ and apply the regex dee-0\.(.*)\.tar\.gz on each of them. The links that conform to the format are processed to obtain the latest version and the location or the absolute URL of the tarball/source code. Here, dee-0\.(.*)\.tar\.gz is considered as the regex term as it contains parenthesis. Hence, currently, all regex terms should contain parenthesis  ‘( )’ and should NOT contain ‘/’.

As work continues, there would be ways to escape the parenthesis which would enable users to work with URLs which contain parenthesis.

The Upstream Tracker Project

Well into GSoC 2012, Its time I wrote a bit about my project this year. I am working with the openSUSE project again this year on the Upstream Tracker project under the mentorship of Vincent Untz. In this post, I will try to give a brief overview of the tool and it’s working. I will focus on the details in subsequent posts.

The ‘Upstream Tracker’ project aims to be a central hub where the upstream versions of open source softwares can be continually monitored for new version releases. One problem with open source softwares is that it is completely de-centralized. Every project has a place of it’s own. Some are hosted at Sourceforge while others at Google Code and a variety of other places. This de-centralized approach has it’s own advantages, however, it also has it’s downsides. For packagers and package maintainers of communities, it is quite a tedious task to keep note of all the packages they maintain and manually look up for new releases. Also, for minor releases, the code change is minimal enough that there is almost no change in the packaging. However, currently, even for minor releases, package maintainers have to be involved at some scale. The project, at a later stage, would also be able to give a comparison table of the latest version available upstream against the latest version currently available on a particular version of a linux distro. This comparison could be of great value to developers/packagers as well as users.

With the upstream tracker project, we crowd source data from users which help us in looking for upstream versions of a given package. The project is divided into two distinct parts – The Rails Frontend, where the users can submit data and look up at the results and The Python Backend, which does all the heavy lifting using the data from the Rails DB. A user is required to input three key parameters – A package name, the download URL and the method used to process the record/package.

Once the user inputs the parameters, the python backend swings into action by collecting records from the Rails DB and processing them one by one. The download URL is opened and all the links on the page are processed. The links are then filtered and only those that satisfy a few basic requirements (ex: file name should be of the format -.tar.) are kept. Finally, the version strings from the file names are extracted, sorted and the latest version is returned along with the complete location of the file. The record on the Rails DB is then updated to reflect the latest version and the URL of the file.

Currently, these methods are supported :

  • HTTPLS – Download files listed as links on a HTTP page
  • FTPLS    –  Download files on FTP
  • DualHTTPLS – Download files listed as links on two HTTP pages (One for release and one for testing)
  • LP – For projects hosted on Launchpad
  • SF – For projects hosted on Sourceforge
  • Google – For projects hosted on Google Code
  • Trac – Download files hosted using Trac
  • SVNLS – Download files on SVN with Web Access

The user can look up the results of the processing on the web interface where packages with errors on them are highlighted in red while those with no errors are highlighted in green. Packages yet to be processed are not highlighted. Also, a separate error page is used to look at erroneous records so that it is easier for people to suggest corrections and update the record.

The form also supports a separate method called Custom, where in the user can enter a custom URL with REGEX, similar to Debian watch files. This forms the base of DEHS imports where in Debian watch files can be directly imported.

The python backend is threaded and would process only X files every few minutes. This allows the load to be spread evenly and hence would require less resources.

The project is being actively worked upon and more features will be added over the course of the next few weeks. So far, the basics are tested and they work fine. Here are a few screenshots to show the workflow.

ElementaryOS – Luna

Get ready for quite a long post!

Ever since the elementaryOS project began, it has been slowly gaining popularity for it’s simple, yet stunning interfaces and it’s lightning fast responsiveness. Naturally, being curious and intrigued, I decided to try out the Jupiter release on my laptop nearly a year ago. It was wonderful! The laptop was a delight to use. It was slick, fast, highly responsive and more importantly, stable. Now, I am not the kind of guy who really cares a lot about stability. I like to experiment with new softwares and am almost always running alpha/beta applications/distros. But that does not mean I like to deal with application crashes every other second as well. And for production computers, stability is a huge deal. And hence, it seemed like elementary was the Perfect Distro.

But later on, when I changed my laptop, i was shocked to find that Jupiter wont run well on my Dell Vostro 3550. It contained an older kernel which did not have built in support for Centrino N-1030 wireless cards that my Dell came with. And since wifi was my only source of internet, i had to be content with using other distros and wait for the new release to come by. And thus, I waited on and on… It never came.

Now, the elementary experience is all about perfection and simplicity. There is no point in using elementary if things randomly crash or the UI has missing graphics and icons. It’s just not elementary. Initially, the luna release was scheduled to be based on Ubuntu 11.10, but due to various stability issues, it was pushed to the 12.04 release. Now while all this may be for the greater good, the enormous wait tends to make people lose interest. Atleast I can say personally, that the lack of an alpha or beta is rather irksome. Not only do pre-release versions of distro’s provide the developers an option to test their software out in the world, it also provides enthusiastic testers and other curious developers to get a sneak peak of the new technology and improvements that the next release would bring. And for a distro like elementary, where perfection matters a lot, I am really surprised to find that there has been no official builds yet.

So, after scouring the web for the best ways to test luna, I came across a few posts. This one particular post : http://blane.me/2012/03/test-drive-elementary-os-luna/. I liked this post because it makes use of netinstall and helps you install elementary from the ground up. Other methods involve installing elementary using a script after installing ubuntu which will remove half of the files installed with ubuntu. Also, I did find an ISO of elementary Luna for those who dont want to mess with command lines and terminals. You can get it from here : http://noiaggiorniamo.blogspot.in/2012/02/lets-test-elementary-os-luna-02-daily.html#more.

Now, regardless of which method you chose to test, or how adventurous you are, here are a few things to remember.

  • Elementary Luna – Is NOT READY! It’s a work in progress. And needless to say, is not fit for production machines.
  • It is not advisable to test by installing to hard drive at this time if you are not ready to deal with the problems and the glitches that may occur.

Now, since that’s out of the way, let’s focus a bit on the problems/glitches I have faced in the last 2 hours or so (since i installed Luna).

  • The wallpaper. There is no straightforward way to change the wallpaper in Luna. And by default, it is in centered mode. This left me with two grey bands on the sides of the wallpaper. To change this, install gconf-editor and change the parameter in /desktop/gnome/background/picture_options and set it to “stretched”. You can change the backgroud by changing the /desktop/gnome/background/picture_filename value.
  • The wallpaper-plug package does not work if installed from the repositories. It seems to be a packaging issue, as compiling by hand makes it work flawlessly.
  • Plank, the elementaryOS dock, seemed to have 4 icons for the file manager marlin. And there seemed to be no way to drag the icons out to remove from the dock. To fix this, remove all files from ~/.config/plank/dock1/launchers. This will remove all the applications from the dock. To add, open the application, right click the icon on the dock and select “keep in dock”.
  • It is not possible yet to change the order in which the icons are placed on the dock. Not by dragging the icons atleast. To change the order, open the corresponding dockitem file from ~/.config/plank/dock1/launchers/ and edit the sort value. The dock items are aligned in ascending order of their sort values.
  • A small glitch with feedly was that the first time you import an OPML file, clicking on a feed on the left side will not list the feed items on the right pane. It works, however when you change the view by clicking on the buttons after the search box. This issue is one-time-only and does not occur once feedly is reopened.
  • Footnote, a new note taking utility, is extremely fragile and crashes with almost every click.
  • Ctrl+Enter does not complete the URL by adding .com at the end. This is more of a missing handy feature than a bug
  • Postler, Merlin and Scratch seems to work without any issues so far.
  • The “issues” listed above are for reference only. It is by no means to scare you off in case you were interested to try out luna, but you should be aware of the issues you might face in the endeavor. Also, please understand that the above issues are not really major. Pre-release softwares are expected to have errors. And that’s one of the reasons there is no alpha or beta releases so far by the elementary team. However hard we might try to get luna up and running, i think everyone will have to wait for the official ISO to get that true elementary experience.

    Phew. This was a rather long post (told ya!), but i think the topic deserves one. The elementary devs are truly doing a great job here.

GNOME Shell Themes

As a follow up to my previous post about customizing your GNOME Shell, here are a few themes I had recently come across. http://www.omgubuntu.co.uk/2011/10/five-pretty-awesome-gnome-shell-themes/ (opens in a new window). Other than smooth inset, which has been doing the rounds on the internet since GNOME 3 first arrived, the other themes look pretty new to me.

Have fun customizing!

Customizing GNOME3

GNOME3 and the GNOME Shell are no doubt, major improvements. They bring in usability and quite a bit of eye candy (in a different way compared to compiz) while at the same time, a few features went missing in 3.0 release – like Emblems, for instance. Nevertheless, GNOME3 is a great release and has been extremely stable so far (Yes, i have been using it since it’s release in April.).

One major complaint that I hear from people is that GNOME3 is not as easy to customize as 2.x. A lot of people seem to think that they are “stuck” with the blue-black theme that comes by default. Ofcourse this is not the case. In this post, I ll explain a few ways in which we can customize GNOME3 and the GNOME Shell.

But before proceeding any further, we have to make a clear distinction which was not necessary before. In GNOME3, with the advent of the shell, there are two sets of themes – one for the shell and the other for the windows (The GTK Theme). To get a smooth and complete makeover, one has to change both the themes. By default, we have the GNOME3 shell theme and the Adwaita Metacity theme for the windows.

1. GNOME Shell Themes
This must be one of the biggest issues I have with he way GNOME3 looks. I cant bear the big, black stripe on the top. It just does not look good. I am sure there are many of you out there who would agree with me on that. But luckily, there are quite a few good options out there that can replace it for us. Also, just to make it easier for us, let’s install the gnome-shell-theme-selector plugin so that we can swtich between themes faster. This plugin adds a Theme button on the top along with windows and application buttons in the Activities mode so that you can change you theme on the fly! So, let’s get started.

Open up a terminal, and type in the following commands to install GNOME Shell Extensions.

git clone http://git.gnome.org/browse/gnome-shell-extensions

cd gnome-shell-extensions

./autogen.sh –prefix=/usr –enable-extensions=user-theme

make && sudo make install

Now, let’s install the Themeselector extension. Fire up another terminal and follow me.

wget http://www.fpmurphy.com/gnome-shell-extensions/themeselector-0.9.tar.gz

tar xvf themeselector-0.9.tar.gz

mv themeselector-0.9/extension.js ~/.local/share/gnome-shell/extensions/themeselector@fpmurphy.com/

mv themeselector-0.9/metadata.json ~/.local/share/gnome-shell/extensions/themeselector@fpmurphy.com/

mv themeselector-0.9/* ~/.themes/

Now, restart GNOME-Shell by pressing Alt+F2 and typing ‘r’ (Without quotes followed by enter). Now, in the activities window, you should have a themes tab with 6 themes to choose from!

To add any new themes you may come across later on, simply extract and add the folders to ~/.themes/ directory. Simple right?

2. GTK Themes
Now, to complete the new look for your desktop, grab a suitable GTK theme to match with your Shell theme from GNOME-Looks.org or from any other website. To install, simply extract the theme to /usr/share/themes.

Great, Not the last thing is to actually tweak the GNOME settings and make it use the GTK theme we just installed. Let’s see how to do that next.

3. GNOME-Tweak-Tool
The GNOME-Tweak-Tool is an aplication that is used to tweak a few values that changes the way your desktop looks and feels which, otherwise, is hard to change. The UI is clean and intuitive, which is a key aspect of a configuration tool.

To obtain the tweak tool, open another terminal and :

git clone http://git.gnome.org/browse/gnome-tweak-tool

cd gnome-tweak-tool

./gnome-tweak-tool

NOTE: Compiled binary is available in the git repository. Hence no need to recompile. Works as it is.

That should fire up a window as shown above. On the left, choose interface from the list and on the right, select the GTK theme you just installed using the drop down menu. And that’s it!

You can mess around and change a few other things like the font, font size and quite a few other settings. So be sure to check out all the options…

Here is how I have set up my GNOME3. I am not artistically inclined and I am sure a lot of you out there can do much better. So let me know what you guys end up with.

libYUI Ported

As most of you reading this post would know, separating libYUI was one of the topics for GSoC 2011 by openSUSE. Over the last three months, a lot of work has gone into the project. As the deadline approaches, we am happy to announce that libYUI is an independent framework!

Let me summarize the changes that have taken place over the last three months. For starters, libYUI is available over a range of platforms. Major platforms like Fedora, Ubuntu and Debian have ready made binaries available at OBS - http://download.opensuse.org/repositories/home:/nbprashanth/. Recently, Michal Hrušecký even ported the library to Gentoo! – michal.hrusecky.net/2011/08/libyui-in-gentoo/. The source has been completely detached from SUSE-centric technologies. So if binaries are not available for your distro, don’t fret. You can always download the source and install from – http://sourceforge.net/projects/libyui/files/.

Apart from porting the library to various platforms, another recent improvement was the upgrade of the GTK plugin from GTK2 to GTK3. In many aspects, this is a very important update.

As the major idea behind the porting effort is to make the library truly independent, it is necessary for libYUI to get a place for itself. For this, we have made a SourceForge page for libyui – sourceforge.net/projects/libyui/. This is an important step that allows us to concentrate on the development of libYUI as a separate framework not influenced much by any other project. We have a separate mailing list, forum and bug tracker set up as well.

Despite the SourceForge page, libYUI will not be using the SVN repository provided by SourceForge. Instead, the source would be moved from the openSUSE SVN to git (github) along with YaST.

There are a lot of projects that currently make us of this wonderful library. YaST is one major application that many geekos out there would be familiar with. Another upcoming application is the sought after SaX3 – An Advanced Graphical X configuration tool by Manu Gupta – http://sysbytes.wordpress.com/. Due to the port of libYUI, SaX3 can be compiled on various platforms! Now, is that not wonderful?

Apart from the above, several small developments have occurred, which one may be familiar with having read my previous, scattered posts. The documentation, for one, has been uploaded. It can be found at – doc.opensuse.org/projects/libyui/HEAD/. Also, we have a libYUI SourceForge page that will serve as the starting point for everything YUI-related – http://libyui.sourceforge.net/. Ofcourse, the page is pretty bland at the moment, but it does satisfy the basics. I ll work on that aspect a while later.

For those who want to contribute or take a look at the development version of libYUI, you can find the sources at – gitorious.org/libyui. Be warned that this URL is bound to change soon when YaST migrates to GitHub.

We are on the lookout for daring and bold testers who can give the new libYUI a spin on various platforms and report bugs, if any. Bugs can be reported on the SourceForge page. Feel free to give feedback as comments (as long as it is not about a bug). And if you are a developer who is curious or just not convinced about YUI’s capability, then give it a spin and see for yourself why it is better! Trust me, you wont regret it.

GSoC Progress

Last week has been quite dull after the mid term evaluations. The week started off with me trying to improve the documentation and a home page for it. While the latter went pretty smooth, the documentation is quite baffling. Most of the source code was quite well documented and small hacks with the Doxyfile seemed to produce the proper results. Nevertheless, there were cases (quite a few) where the results seemed out of place. While struggling with the docs, I also started working on the libyui-gtk-pkg package. This package provides the libzypp plugin for the libyui-gtk module. This has been separated from libyui-gtk package so as to make it platform independent. I am nearly there on getting this package to compile. A day or two should be sufficient, I guess.

By the end of last week, I switched over to writing examples for YUI in the hope that it would give me further insight into what a developer might expect from the docs. After a few not-so-great ideas, I settled on an IRC client written with YUI. The library to manage the IRC protocol is self-implemented and is complete. At this point in time, it handles only conversations and a single channel. This is very small compared to the span of the IRC protocol. But since the main aspect of the example is to showcase YUI as a UI library, I guess the features are sufficient.

YUI moves to SourceForge

Over the last month, libYUI has significantly expanded. It is an independent framework and is avalable outside the SUSE framework making it accessible to a wide audience. This has resulted in the project having distributed content on the web. For example, the source code is maintained at gitorious.org (currently svn.opensuse.org) and the documentation at docs.opensuse.org etc. YUI does not have it’s own mailing list and the issues are still being dealt with on the yast ML. Considering all this, we thought that it would be good to have a central place where from all the resources can be accessed. Thus, we bring to you, sourceforge.net/projects/libyui/.

This means that :
1. YUI has it’s own bug tracker. Any bugs/feature requests should be reported at the sourceforge page.
2. YUI will not maintain it’s source code at SF as we are already porting the code to gitorious.org.
3. Current release tarballs can be found at the SF downloads page.
4. Mailing List for YUI can be found at libyui-devel@lists.sourceforge.net. This can be used for all YUI (not YaST) related discussions.
5. Webpage for YUI along with examples and links to external resources will be up soon.
6. Forum for YUI related topics is available at SF.

So, stay tuned to sourceforge to keep track of YUI!

Build Update

libYUI along with the ncurses, Qt and Gtk plugins are now available for non-SUSE platforms. The builds are hosted at OBS and the build targets are Fedora (14 & 15) , Ubuntu (10.10 & 11.04) and Debian 6. The download repository can be found at : http://download.opensuse.org/repositories/home:/nbprashanth/.
Bugs in packaging can be reported in the comments section below.

Screenshots

It’s time for some screenshots! (Finally). The programs are standard YUI examples found at the openSUSE svn : svn.opensuse.org/svn/yast/libyui.

System : Fedora 15; GNOME 3 Desktop

ComboBox1.cc – GTK

ComboBox1-GTK-1    ComboBox1-GTK-2

ComboBox1.cc – Ncurses

ComboBox1-ncurses

ComboBox1.cc – Qt

ComboBox1-Qt

HelloWorld.cc – GTK(Left) & Qt (Right)

HelloWorld-GTK  HelloWorld-Qt

HelloWorld.cc – Ncurses

HelloWorld-ncurses

All plugins in action

YUI-All-Plugins

Follow

Get every new post delivered to your Inbox.