Custom URL support for Upstream Tracker Project

One of the features available on the Upstream Tracker project is the ability to process custom URLs to track upstream package versions. This allows URLs to contain regular expressions which are processed using the re module in python. To demonstrate this feature, let me post a few sample inputs and outputs that were generated using this module.

Custom URL : http://coherence.beebits.net/download/Coherence-(.*)\.tar\.gz
Coherence-0.6.6.2.tar.gz http://coherence.beebits.net/Coherence-0.6.6.2.tar.gz
Custom URL : ftp://ftp.gimp.org/pub/gimp/help/gimp-help-(.*)\.tar\.gz
gimp-help-2-0.11.tar.gz ftp://ftp.gimp.org/gimp-help-2-0.11.tar.gz
Custom URL : http://code.google.com/p/giver/downloads/list/giver-(.*)\.tar\.gz
giver-0.1.8.tar.gz http://code.google.com///giver.googlecode.com/files/giver-0.1.8.tar.gz
Custom URL : http://sourceforge.net/api/file/index/project-name/libyui/rss/libyui-(.*)\.tar\.gz
libyui-gtk-2.21.96.tar.gz http://sourceforge.net/projects/libyui/files/libyui-gtk-2.21.96.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-(.*)\.tar\.gz
dee-1.0.10.tar.gz https://launchpad.net/https://launchpad.net/dee/1.0/1.0.10/+download/dee-1.0.10.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz
dee-0.5.22.tar.gz https://launchpad.net/https://launchpad.net/dee/0.5/0.5.22/+download/dee-0.5.22.tar.gz

From the above examples, we can see that the module supports projects hosted on sourceforge as well as projects where the download pages are accessible via HTTP and FTP. It is also possible to narrow down and search for only 0.x branch of a package using suitable regex as shown in the last example.  The module tries to implement complete regex support and does this with the help of the python re module. However, to identify if a term has to be evaluated using re, we look for parenthesis. Thus, regular URLs with parenthesis would be treated as regex terms and would be processed accordingly. For example, consider the URL below :

https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz

Here, we list all the links present at https://launchpad.net/dee/+download/ and apply the regex dee-0\.(.*)\.tar\.gz on each of them. The links that conform to the format are processed to obtain the latest version and the location or the absolute URL of the tarball/source code. Here, dee-0\.(.*)\.tar\.gz is considered as the regex term as it contains parenthesis. Hence, currently, all regex terms should contain parenthesis  ‘( )’ and should NOT contain ‘/’.

As work continues, there would be ways to escape the parenthesis which would enable users to work with URLs which contain parenthesis.

The Upstream Tracker Project

Well into GSoC 2012, Its time I wrote a bit about my project this year. I am working with the openSUSE project again this year on the Upstream Tracker project under the mentorship of Vincent Untz. In this post, I will try to give a brief overview of the tool and it’s working. I will focus on the details in subsequent posts.

The ‘Upstream Tracker’ project aims to be a central hub where the upstream versions of open source softwares can be continually monitored for new version releases. One problem with open source softwares is that it is completely de-centralized. Every project has a place of it’s own. Some are hosted at Sourceforge while others at Google Code and a variety of other places. This de-centralized approach has it’s own advantages, however, it also has it’s downsides. For packagers and package maintainers of communities, it is quite a tedious task to keep note of all the packages they maintain and manually look up for new releases. Also, for minor releases, the code change is minimal enough that there is almost no change in the packaging. However, currently, even for minor releases, package maintainers have to be involved at some scale. The project, at a later stage, would also be able to give a comparison table of the latest version available upstream against the latest version currently available on a particular version of a linux distro. This comparison could be of great value to developers/packagers as well as users.

With the upstream tracker project, we crowd source data from users which help us in looking for upstream versions of a given package. The project is divided into two distinct parts – The Rails Frontend, where the users can submit data and look up at the results and The Python Backend, which does all the heavy lifting using the data from the Rails DB. A user is required to input three key parameters – A package name, the download URL and the method used to process the record/package.

Once the user inputs the parameters, the python backend swings into action by collecting records from the Rails DB and processing them one by one. The download URL is opened and all the links on the page are processed. The links are then filtered and only those that satisfy a few basic requirements (ex: file name should be of the format -.tar.) are kept. Finally, the version strings from the file names are extracted, sorted and the latest version is returned along with the complete location of the file. The record on the Rails DB is then updated to reflect the latest version and the URL of the file.

Currently, these methods are supported :

• HTTPLS – Download files listed as links on a HTTP page
• FTPLS    –  Download files on FTP
• DualHTTPLS – Download files listed as links on two HTTP pages (One for release and one for testing)
• LP – For projects hosted on Launchpad
• SF – For projects hosted on Sourceforge
• Google – For projects hosted on Google Code
• Trac – Download files hosted using Trac
• SVNLS – Download files on SVN with Web Access

The user can look up the results of the processing on the web interface where packages with errors on them are highlighted in red while those with no errors are highlighted in green. Packages yet to be processed are not highlighted. Also, a separate error page is used to look at erroneous records so that it is easier for people to suggest corrections and update the record.

The form also supports a separate method called Custom, where in the user can enter a custom URL with REGEX, similar to Debian watch files. This forms the base of DEHS imports where in Debian watch files can be directly imported.

The python backend is threaded and would process only X files every few minutes. This allows the load to be spread evenly and hence would require less resources.

The project is being actively worked upon and more features will be added over the course of the next few weeks. So far, the basics are tested and they work fine. Here are a few screenshots to show the workflow.

The Rails based Web Interface for The Upstream Tracker project.

Rails View listing available records

Record Details before processing

Python Backend running on Windows 7

Records listed on the Rails web page.

Record Details

Record Listing page on the Rails Web Application with an error

Custom URL making use of the Custom method to process URL’s with REGEX

Custom URL Result showing latest version and URL of the file

libYUI Ported

As most of you reading this post would know, separating libYUI was one of the topics for GSoC 2011 by openSUSE. Over the last three months, a lot of work has gone into the project. As the deadline approaches, we am happy to announce that libYUI is an independent framework!

Let me summarize the changes that have taken place over the last three months. For starters, libYUI is available over a range of platforms. Major platforms like Fedora, Ubuntu and Debian have ready made binaries available at OBS - http://download.opensuse.org/repositories/home:/nbprashanth/. Recently, Michal Hrušecký even ported the library to Gentoo! – michal.hrusecky.net/2011/08/libyui-in-gentoo/. The source has been completely detached from SUSE-centric technologies. So if binaries are not available for your distro, don’t fret. You can always download the source and install from – http://sourceforge.net/projects/libyui/files/.

Apart from porting the library to various platforms, another recent improvement was the upgrade of the GTK plugin from GTK2 to GTK3. In many aspects, this is a very important update.

As the major idea behind the porting effort is to make the library truly independent, it is necessary for libYUI to get a place for itself. For this, we have made a SourceForge page for libyui – sourceforge.net/projects/libyui/. This is an important step that allows us to concentrate on the development of libYUI as a separate framework not influenced much by any other project. We have a separate mailing list, forum and bug tracker set up as well.

Despite the SourceForge page, libYUI will not be using the SVN repository provided by SourceForge. Instead, the source would be moved from the openSUSE SVN to git (github) along with YaST.

There are a lot of projects that currently make us of this wonderful library. YaST is one major application that many geekos out there would be familiar with. Another upcoming application is the sought after SaX3 – An Advanced Graphical X configuration tool by Manu Gupta – http://sysbytes.wordpress.com/. Due to the port of libYUI, SaX3 can be compiled on various platforms! Now, is that not wonderful?

Apart from the above, several small developments have occurred, which one may be familiar with having read my previous, scattered posts. The documentation, for one, has been uploaded. It can be found at – doc.opensuse.org/projects/libyui/HEAD/. Also, we have a libYUI SourceForge page that will serve as the starting point for everything YUI-related – http://libyui.sourceforge.net/. Ofcourse, the page is pretty bland at the moment, but it does satisfy the basics. I ll work on that aspect a while later.

For those who want to contribute or take a look at the development version of libYUI, you can find the sources at – gitorious.org/libyui. Be warned that this URL is bound to change soon when YaST migrates to GitHub.

We are on the lookout for daring and bold testers who can give the new libYUI a spin on various platforms and report bugs, if any. Bugs can be reported on the SourceForge page. Feel free to give feedback as comments (as long as it is not about a bug). And if you are a developer who is curious or just not convinced about YUI’s capability, then give it a spin and see for yourself why it is better! Trust me, you wont regret it.

GSoC Progress

Last week has been quite dull after the mid term evaluations. The week started off with me trying to improve the documentation and a home page for it. While the latter went pretty smooth, the documentation is quite baffling. Most of the source code was quite well documented and small hacks with the Doxyfile seemed to produce the proper results. Nevertheless, there were cases (quite a few) where the results seemed out of place. While struggling with the docs, I also started working on the libyui-gtk-pkg package. This package provides the libzypp plugin for the libyui-gtk module. This has been separated from libyui-gtk package so as to make it platform independent. I am nearly there on getting this package to compile. A day or two should be sufficient, I guess.

By the end of last week, I switched over to writing examples for YUI in the hope that it would give me further insight into what a developer might expect from the docs. After a few not-so-great ideas, I settled on an IRC client written with YUI. The library to manage the IRC protocol is self-implemented and is complete. At this point in time, it handles only conversations and a single channel. This is very small compared to the span of the IRC protocol. But since the main aspect of the example is to showcase YUI as a UI library, I guess the features are sufficient.

YUI moves to SourceForge

Over the last month, libYUI has significantly expanded. It is an independent framework and is avalable outside the SUSE framework making it accessible to a wide audience. This has resulted in the project having distributed content on the web. For example, the source code is maintained at gitorious.org (currently svn.opensuse.org) and the documentation at docs.opensuse.org etc. YUI does not have it’s own mailing list and the issues are still being dealt with on the yast ML. Considering all this, we thought that it would be good to have a central place where from all the resources can be accessed. Thus, we bring to you, sourceforge.net/projects/libyui/.

This means that :
1. YUI has it’s own bug tracker. Any bugs/feature requests should be reported at the sourceforge page.
2. YUI will not maintain it’s source code at SF as we are already porting the code to gitorious.org.
3. Current release tarballs can be found at the SF downloads page.
4. Mailing List for YUI can be found at libyui-devel@lists.sourceforge.net. This can be used for all YUI (not YaST) related discussions.
5. Webpage for YUI along with examples and links to external resources will be up soon.
6. Forum for YUI related topics is available at SF.

So, stay tuned to sourceforge to keep track of YUI!

Build Update

libYUI along with the ncurses, Qt and Gtk plugins are now available for non-SUSE platforms. The builds are hosted at OBS and the build targets are Fedora (14 & 15) , Ubuntu (10.10 & 11.04) and Debian 6. The download repository can be found at : http://download.opensuse.org/repositories/home:/nbprashanth/.
Bugs in packaging can be reported in the comments section below.

Screenshots

It’s time for some screenshots! (Finally). The programs are standard YUI examples found at the openSUSE svn : svn.opensuse.org/svn/yast/libyui.

System : Fedora 15; GNOME 3 Desktop

ComboBox1.cc – GTK

  

ComboBox1.cc – Ncurses

ComboBox1.cc – Qt

HelloWorld.cc – GTK(Left) & Qt (Right)

 

HelloWorld.cc – Ncurses

All plugins in action

Another YUI example

Hi again! If you have read the last few blog posts, you would have (hopefully) understood the basics of YUI and how to handle events in YUI. Now, it’s time to move on to a more practical and slightly complex example which can show how YUI can be used to make small applications.

Today’s example is a quiz (demo) application written by manugupt1 for his presentation on YUI at the GNOME.Asia Summit 2011. The code was written at short notice, but it is still a very good example. You can find the code at : https://github.com/manugupt1/playground/tree/master/quiz.

The code is similar in structure to the radiobutton and event handling examples that were dealt with before, so we will not be discussing it here today. Please feel free to try out the code and experiment. I hope you find this example interesting.

libYUI is now available for other operating systems as well!
Fedora 15 and Ubuntu 11.04 users can grab the initial binary packages from: gitorious.org/libyui to test the examples.

Packages Available

The deb and rpm packages for libyui and it’s plugins (ncurses and qt) are available for testing. They are hosted at the git repository : gitorious.org/libyui and can be obtained using git clone or manually by browsing the source tree. Please give your feedbacks/bug reports in the form of comments below.

Note: These are initial, test packages compiled on Fedora 15 and Ubuntu 11.04.

Watch out for the GTK plugin packages soon!

Happy coding.

Radio Buttons, Combo Boxes and Event Handling

Welcome back! Today we will walk through a slightly bigger example to understand the way event handling works in YUI. We will also learn use two new widgets that we see in almost all the applications we use : Radio Buttons and Combo Boxes.

We will create a program that contains a combo box, two radio buttons, a label and a push button. The radio buttons are called immediate and event1. If immediate is selected, the label reflects the selected value of the combo box without clicking the push button. If event1 radio button is selected, the push button has to be clicked for the label to reflect the value of the selected element of the combo box. Simple concept, right?

Before we proceed, please note that all the examples found in this blog are available at : https://gitorious.org/libyui. So feel free to compile and see the output for yourself! (ofcourse, only on openSUSE at the moment.)

Also, I will be assuming that you know the basics of YUI and that you have read my previous post : http://nbprashanth.wordpress.com/2011/04/22/libyui-an-introduction/. If not, it is highly recommended that you read that first as we will build on that foundation.

So, lets start with the example.


#include "YUI.h"
#include "YWidgetFactory.h"
#include "YDialog.h"
#include "YLayoutBox.h"
#include "YComboBox.h"
#include "YEvent.h"
#include "YLabel.h"
#include "YPushButton.h"
#include "YRadioButton.h"
#include "YRadioButtonGroup.h"

int main( int argc, char **argv )
{
YDialog * dialog = YUI::widgetFactory()->createPopupDialog();

YLayoutBox * vbox = YUI::widgetFactory()->createVBox( dialog );
YLayoutBox * radiobox = YUI::widgetFactory()->createHBox( vbox );

YRadioButtonGroup *radiogrp = YUI::widgetFactory()->createRadioButtonGroup(radiobox);
YRadioButton *immediate = YUI::widgetFactory()->createRadioButton(radiobox,"Immediate", false);
YRadioButton *event1 = YUI::widgetFactory()->createRadioButton(radiobox,"Not Immediate", true);
immediate->setNotify(true);
event1->setNotify(true);

radiogrp->addRadioButton(immediate);
radiogrp->addRadioButton(event1);

YLayoutBox * hbox = YUI::widgetFactory()->createHBox( vbox );
YComboBox *cbox = YUI::widgetFactory()->createComboBox(hbox,"Choose Item",false);
YLabel *label = YUI::widgetFactory()->createLabel(hbox,"",false,false);

YLayoutBox * hbtnbox = YUI::widgetFactory()->createHBox( vbox );
YPushButton *button = YUI::widgetFactory()->createPushButton(hbtnbox,"&Update");

cbox->addItem("English",true);
cbox->addItem("Spanish",false);
cbox->addItem("French",false);
cbox->addItem("Hindi",false);
cbox->addItem("Chinese",false);
cbox->addItem("Russian",false);

do
{
YEvent *event = dialog->waitForEvent();

if(event)
{
if(event->eventType() == YEvent::CancelEvent)
break;

if(event->widget() == cbox || event->widget() == button)
{
label->setText(cbox->selectedItem()->label());
}

if(event->eventType() == YEvent::WidgetEvent && (event->widget() == immediate || event->widget() == event1))
{
radiogrp->uncheckOtherButtons((YRadioButton*)event->widget());

if(event->widget()==immediate)
cbox->setNotify(true);
else
cbox->setNotify(false);
}

}
}while(1);

dialog->destroy();
}


Lets go through the code a few lines at a time now.


YDialog * dialog = YUI::widgetFactory()->createPopupDialog();


As in hello world example, the line above is used to create the window, which would contain all our widgets (buttons, radio buttons etc).


YLayoutBox * vbox = YUI::widgetFactory()->createVBox( dialog );
YLayoutBox * radiobox = YUI::widgetFactory()->createHBox( vbox );


The next two lines, are used to set the layout. We add a VBox layout (used to arrange the widgets in a vertical stack) to the window and add a HBox layout (used to arrange the widgets in a horizontal stack) to the vbox layout created before. The layouts, as such do not show up on the screen. They are used as containers to place widgets on the screen in an easy way. It is always preferred to place components/widgets in such containers as they take care of events like window resizing.


YRadioButtonGroup *radiogrp = YUI::widgetFactory()->createRadioButtonGroup(radiobox);
YRadioButton *immediate = YUI::widgetFactory()->createRadioButton(radiobox,"Immediate", false);
YRadioButton *event1 = YUI::widgetFactory()->createRadioButton(radiobox,"Not Immediate", true);


The next three lines are used to create a radio button group, radiogrp and two radio buttons immediate and event1. Radio buttons are used when the user has to select exactly one of the given choices. Hence, it becomes necessary to group these radio buttons. The radio button group helps us do just that.

Please note that just by adding radio buttons to a radio button group, the other radio buttons are not deselected by default. This is handled in the event loop.

While creating the radio buttons, the first parameter is the parent widget, inside which the radio button is placed. The second parameter is the label value or the value of the text displayed next to the radio button. And finally, the last value represents the default selected state. In our example, the immediate radio button is disabled by default whereas the event1 radiobutton is enabled by default.


immediate->setNotify(true);
event1->setNotify(true);


In the last two lines, we call the set notify property of both the radio buttons and set them to true. If this is not done, an event is not generated by the radio button itself and we will have to manually check using some other strategy, like using a push button.

Let’s take a minute to understand this. There are two ways in which we can handle events. The first is when any change to the widget generates an event or by using another widget to generate an event and use it to check for any change. In the second case, the most commonly used widget would be the push button. You would probably have noticed google instant, where the search page gets updated as you change the search phrase. While not similar in technology (they use ajax), that is the an example for the first type of event notification. If you remember google in the olden days, where we had to press the search button after we change the search phrase everytime. That is an example for the second type. Let’s call the first type of events as “immediate” and the second type as “event-driven”.

So, since we want any clicks/changes to the radio button to reflect on it’s own without any button presses, we set their setNotify methods to true! Easy right? Let’s move on now.


radiogrp->addRadioButton(immediate);
radiogrp->addRadioButton(event1);


In the first two lines, we add the radio button to the radio button group we created. This is done because immediate mode and event driven modes are opposite and cannot be enabled at the same time. Hence, we group them so that only one can be enabled at a time. We disable the other radio buttons in the event loop and is not done automatically. This is a very important point to note as it is not very intuitive.


YLayoutBox * hbox = YUI::widgetFactory()->createHBox( vbox );
YComboBox *cbox = YUI::widgetFactory()->createComboBox(hbox,"Choose Item",false);
YLabel *label = YUI::widgetFactory()->createLabel(hbox,"",false,false);


The next set of lines define a HBox for the combo box and the label widget. The combo box is simply another name for the drop down box which we use to select an item from a list of items. In our case, while creating the combo box, we use three parameters. The first, is obviously the parent widget. the second, is a label that is used to describe the combo box. In our case it is “choose item”(Maybe “Choose language” is more appropriate?). The third parameter is a boolean value that is used to set the editable property of the combo box. If set to true, the user is allowed to enter a value that is not a part of the list. He can type it out as if the combo box was a text field or choose from the list. Since we want the user to select only from the list, we set this to false.


YLayoutBox * hbtnbox = YUI::widgetFactory()->createHBox( vbox );
YPushButton *button = YUI::widgetFactory()->createPushButton(hbtnbox,"&Update");


The next two lines are used to create a HBox and add a push button to it. Since we have seen the parameters used to create the push button in the last post, i am not elaborating it here again.


cbox->addItem("English",true);
cbox->addItem("Spanish",false);
cbox->addItem("French",false);
cbox->addItem("Hindi",false);
cbox->addItem("Chinese",false);
cbox->addItem("Russian",false);


The last set of lines are used to add elements to the combo box. The second boolean parameter is the default selected element. If “Spanish” was set to true, the first and default element in the combo box would be Spanish.

Please note that there are more efficient ways to add large number of items to a combo box. In this example, we neglect that as we have very few items.

A point to be noted here is that when layouts are used, please pay attention to their parent widget parameters. Giving incorrect parameters can cause skewed layouts. Also, like in our case, when we nest 3 HBox elements in a VBox, the Hbox elements are stacked one over another with the first HBox on the top. But the elements are arranged linearly within each HBox. I hope this distinction is very clear.

And that’s the end of the user interface! Take a break and lets move to the event handling part.


do
{
YEvent *event = dialog->waitForEvent();

if(event)
{
if(event->eventType() == YEvent::CancelEvent)
break;

if(event->widget() == cbox || event->widget() == button)
{
label->setText(cbox->selectedItem()->label());
}

if(event->eventType() == YEvent::WidgetEvent && (event->widget() == immediate || event->widget() == event1))
{
radiogrp->uncheckOtherButtons((YRadioButton*)event->widget());

if(event->widget()==immediate)
cbox->setNotify(true);
else
cbox->setNotify(false);
}

}
}while(1);


These set of lines control all the events and actions of our program. It is very important to understand the working of the event loop in order to program effectively in YUI. Let’s go line by line.

As you can see, the entire set of code is placed in an infinite do while loop. The loop need not be do while. It can be a while loop or a for loop (your pick). The reason for the loop being infinite is obvious. We want to keep polling for events as long as the program is running.

The first line in the loop is is used to create an instance of YEvent which is equated to dialog->waitForEvent(). Here, dialog is the main window. We use this as it is the top level container for out program i.e, all widgets are under this. The dialog->waitForEvent() function is used to poll for events. With no parameters, it waits indefinitely for an event to occur. In some cases, we would like to set a time out and carry on with the loop in case no event occurs. In such a case, we can use an over loaded method which takes an integer which is the timeout in milli seconds. Thus, dialog->waitForEvent(5) waits for 5 ms for an event to occur. If an event occurs, the event is stored in the YEvent variable and the the loop continues. If no event occurs, then the program execution continues after the timeout, 5ms.

It is important to note that the waitForEvent is an infinite loop which is truncated by the timeout time or by an event.

In the next line, we check if event is not empty. This is not needed if a timeout has not been specified since this line is executed only when an event has occurred.

The main functions associated with YEvent variables are :
1. eventType() – Returns the event type. The types are defined in the YEvent class.
2. widget() – Returns the widget that raised the event.

The above two functions are frequently used and is handy to know about their function before we move further.


if(event->eventType() == YEvent::CancelEvent)
break;


In the above line, we check if the event is of the type “CancelEvent”. This event is triggered when the user closes the window. In such a case, we want to break out of the loop and destroy the widgets. Hence, we issue the break command.


if(event->widget() == cbox || event->widget() == button)
{
label->setText(cbox->selectedItem()->label());
}


In the above set of lines, we check if either the combo box or the button has raised an event using the widget() function. If true, we change the label text by using the setText() function as shown above. cbox->selectedItem()->label() is used to retrieve the text of the currently selected element.


if(event->eventType() == YEvent::WidgetEvent && (event->widget() == immediate || event->widget() == event1))
{
radiogrp->uncheckOtherButtons((YRadioButton*)event->widget());

if(event->widget()==immediate)
cbox->setNotify(true);
else
cbox->setNotify(false);
}


After the last two blocks of code, i guess this is easier to understand.Here we check for the event type “WidgetEvent”. Radio buttons, when setNotify is set to true, raise such events in case of a change. Also, we check if the widget that raised this event is either immediate or event1, our two radio buttons. This is to ensure that the event is being raised by specific widgets though in our case, it may be trivial and not necessary.

In the next line, we unset the other radio buttons in the same group. Again, note that we do this explicitly!! The uncheckOtherButtons takes a radio button as a parameter and unchecks all other radio buttons that are a part of the same group. Note that the uncheckOtherButtons function belongs to the RadioButtonGroup class and not the RadioButton class.

In the last few lines, we toggle the setNotify of the combo box depending on the radio button selected.


dialog->destroy();


This is the last line of code and is placed outside the event loop. It is used to destroy all widgets and the main window as well. Execution of this command truncates the program.

And that’s it! Phew… long post.

I hope this post helped a few of you out there to understand a the basic functioning of the YUI library. Now, you are ready to start writing a few programs of your own! In case you need some help, feel free to contact me through the comments section below.