Category Archives: GsoC

Custom URL support for Upstream Tracker Project

One of the features available on the Upstream Tracker project is the ability to process custom URLs to track upstream package versions. This allows URLs to contain regular expressions which are processed using the re module in python. To demonstrate this feature, let me post a few sample inputs and outputs that were generated using this module.

Custom URL : http://coherence.beebits.net/download/Coherence-(.*)\.tar\.gz
Coherence-0.6.6.2.tar.gz http://coherence.beebits.net/Coherence-0.6.6.2.tar.gz
Custom URL : ftp://ftp.gimp.org/pub/gimp/help/gimp-help-(.*)\.tar\.gz
gimp-help-2-0.11.tar.gz ftp://ftp.gimp.org/gimp-help-2-0.11.tar.gz
Custom URL : http://code.google.com/p/giver/downloads/list/giver-(.*)\.tar\.gz
giver-0.1.8.tar.gz http://code.google.com///giver.googlecode.com/files/giver-0.1.8.tar.gz
Custom URL : http://sourceforge.net/api/file/index/project-name/libyui/rss/libyui-(.*)\.tar\.gz
libyui-gtk-2.21.96.tar.gz http://sourceforge.net/projects/libyui/files/libyui-gtk-2.21.96.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-(.*)\.tar\.gz
dee-1.0.10.tar.gz https://launchpad.net/https://launchpad.net/dee/1.0/1.0.10/+download/dee-1.0.10.tar.gz
Custom URL : https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz
dee-0.5.22.tar.gz https://launchpad.net/https://launchpad.net/dee/0.5/0.5.22/+download/dee-0.5.22.tar.gz

From the above examples, we can see that the module supports projects hosted on sourceforge as well as projects where the download pages are accessible via HTTP and FTP. It is also possible to narrow down and search for only 0.x branch of a package using suitable regex as shown in the last example.  The module tries to implement complete regex support and does this with the help of the python re module. However, to identify if a term has to be evaluated using re, we look for parenthesis. Thus, regular URLs with parenthesis would be treated as regex terms and would be processed accordingly. For example, consider the URL below :

https://launchpad.net/dee/+download/dee-0\.(.*)\.tar\.gz

Here, we list all the links present at https://launchpad.net/dee/+download/ and apply the regex dee-0\.(.*)\.tar\.gz on each of them. The links that conform to the format are processed to obtain the latest version and the location or the absolute URL of the tarball/source code. Here, dee-0\.(.*)\.tar\.gz is considered as the regex term as it contains parenthesis. Hence, currently, all regex terms should contain parenthesis  ‘( )’ and should NOT contain ‘/’.

As work continues, there would be ways to escape the parenthesis which would enable users to work with URLs which contain parenthesis.

The Upstream Tracker Project

Well into GSoC 2012, Its time I wrote a bit about my project this year. I am working with the openSUSE project again this year on the Upstream Tracker project under the mentorship of Vincent Untz. In this post, I will try to give a brief overview of the tool and it’s working. I will focus on the details in subsequent posts.

The ‘Upstream Tracker’ project aims to be a central hub where the upstream versions of open source softwares can be continually monitored for new version releases. One problem with open source softwares is that it is completely de-centralized. Every project has a place of it’s own. Some are hosted at Sourceforge while others at Google Code and a variety of other places. This de-centralized approach has it’s own advantages, however, it also has it’s downsides. For packagers and package maintainers of communities, it is quite a tedious task to keep note of all the packages they maintain and manually look up for new releases. Also, for minor releases, the code change is minimal enough that there is almost no change in the packaging. However, currently, even for minor releases, package maintainers have to be involved at some scale. The project, at a later stage, would also be able to give a comparison table of the latest version available upstream against the latest version currently available on a particular version of a linux distro. This comparison could be of great value to developers/packagers as well as users.

With the upstream tracker project, we crowd source data from users which help us in looking for upstream versions of a given package. The project is divided into two distinct parts – The Rails Frontend, where the users can submit data and look up at the results and The Python Backend, which does all the heavy lifting using the data from the Rails DB. A user is required to input three key parameters – A package name, the download URL and the method used to process the record/package.

Once the user inputs the parameters, the python backend swings into action by collecting records from the Rails DB and processing them one by one. The download URL is opened and all the links on the page are processed. The links are then filtered and only those that satisfy a few basic requirements (ex: file name should be of the format -.tar.) are kept. Finally, the version strings from the file names are extracted, sorted and the latest version is returned along with the complete location of the file. The record on the Rails DB is then updated to reflect the latest version and the URL of the file.

Currently, these methods are supported :

  • HTTPLS – Download files listed as links on a HTTP page
  • FTPLS    –  Download files on FTP
  • DualHTTPLS – Download files listed as links on two HTTP pages (One for release and one for testing)
  • LP – For projects hosted on Launchpad
  • SF – For projects hosted on Sourceforge
  • Google – For projects hosted on Google Code
  • Trac – Download files hosted using Trac
  • SVNLS – Download files on SVN with Web Access

The user can look up the results of the processing on the web interface where packages with errors on them are highlighted in red while those with no errors are highlighted in green. Packages yet to be processed are not highlighted. Also, a separate error page is used to look at erroneous records so that it is easier for people to suggest corrections and update the record.

The form also supports a separate method called Custom, where in the user can enter a custom URL with REGEX, similar to Debian watch files. This forms the base of DEHS imports where in Debian watch files can be directly imported.

The python backend is threaded and would process only X files every few minutes. This allows the load to be spread evenly and hence would require less resources.

The project is being actively worked upon and more features will be added over the course of the next few weeks. So far, the basics are tested and they work fine. Here are a few screenshots to show the workflow.

libYUI Ported

As most of you reading this post would know, separating libYUI was one of the topics for GSoC 2011 by openSUSE. Over the last three months, a lot of work has gone into the project. As the deadline approaches, we am happy to announce that libYUI is an independent framework!

Let me summarize the changes that have taken place over the last three months. For starters, libYUI is available over a range of platforms. Major platforms like Fedora, Ubuntu and Debian have ready made binaries available at OBS - http://download.opensuse.org/repositories/home:/nbprashanth/. Recently, Michal Hrušecký even ported the library to Gentoo! – michal.hrusecky.net/2011/08/libyui-in-gentoo/. The source has been completely detached from SUSE-centric technologies. So if binaries are not available for your distro, don’t fret. You can always download the source and install from – http://sourceforge.net/projects/libyui/files/.

Apart from porting the library to various platforms, another recent improvement was the upgrade of the GTK plugin from GTK2 to GTK3. In many aspects, this is a very important update.

As the major idea behind the porting effort is to make the library truly independent, it is necessary for libYUI to get a place for itself. For this, we have made a SourceForge page for libyui – sourceforge.net/projects/libyui/. This is an important step that allows us to concentrate on the development of libYUI as a separate framework not influenced much by any other project. We have a separate mailing list, forum and bug tracker set up as well.

Despite the SourceForge page, libYUI will not be using the SVN repository provided by SourceForge. Instead, the source would be moved from the openSUSE SVN to git (github) along with YaST.

There are a lot of projects that currently make us of this wonderful library. YaST is one major application that many geekos out there would be familiar with. Another upcoming application is the sought after SaX3 – An Advanced Graphical X configuration tool by Manu Gupta – http://sysbytes.wordpress.com/. Due to the port of libYUI, SaX3 can be compiled on various platforms! Now, is that not wonderful?

Apart from the above, several small developments have occurred, which one may be familiar with having read my previous, scattered posts. The documentation, for one, has been uploaded. It can be found at – doc.opensuse.org/projects/libyui/HEAD/. Also, we have a libYUI SourceForge page that will serve as the starting point for everything YUI-related – http://libyui.sourceforge.net/. Ofcourse, the page is pretty bland at the moment, but it does satisfy the basics. I ll work on that aspect a while later.

For those who want to contribute or take a look at the development version of libYUI, you can find the sources at – gitorious.org/libyui. Be warned that this URL is bound to change soon when YaST migrates to GitHub.

We are on the lookout for daring and bold testers who can give the new libYUI a spin on various platforms and report bugs, if any. Bugs can be reported on the SourceForge page. Feel free to give feedback as comments (as long as it is not about a bug). And if you are a developer who is curious or just not convinced about YUI’s capability, then give it a spin and see for yourself why it is better! Trust me, you wont regret it.

GSoC Progress

Last week has been quite dull after the mid term evaluations. The week started off with me trying to improve the documentation and a home page for it. While the latter went pretty smooth, the documentation is quite baffling. Most of the source code was quite well documented and small hacks with the Doxyfile seemed to produce the proper results. Nevertheless, there were cases (quite a few) where the results seemed out of place. While struggling with the docs, I also started working on the libyui-gtk-pkg package. This package provides the libzypp plugin for the libyui-gtk module. This has been separated from libyui-gtk package so as to make it platform independent. I am nearly there on getting this package to compile. A day or two should be sufficient, I guess.

By the end of last week, I switched over to writing examples for YUI in the hope that it would give me further insight into what a developer might expect from the docs. After a few not-so-great ideas, I settled on an IRC client written with YUI. The library to manage the IRC protocol is self-implemented and is complete. At this point in time, it handles only conversations and a single channel. This is very small compared to the span of the IRC protocol. But since the main aspect of the example is to showcase YUI as a UI library, I guess the features are sufficient.

GsoC Report – 3

Yes. Sorry. It is another report. But i ll try my best to keep this one short though. :P

Last week was all about the GTK plugin. This plugin was different from the other plugins and was dependent on libzypp. And libzypp was dependent on libsatsolver and that was dependent on…. well, you get the point. The libzypp dependecy had to be removed and that was my job.

Most of the time was spent in analysing how (and why) libzypp was used by the plugin. Once i figured that out and got the plugin to compile without libzypp, I was faced with more problems. Initial builds were fine on my Fedora box (F15). But when I built it on OBS, the F14 build failed. After reading the logs, i found out that gdk-pixbuf package by default installs itself in a different location in F14. This broke the build on F14.

The Deb builds were more problematic. They failed on my build system (ubuntu 11.04). CMake seemed to have difficulties in finding GTK includes and libs. After trying to tweak the code and check for alternatives using pkg-config, i finally stumbled on to a link where i found out that this was a known issue with CMake on Ubuntu 11.04.

Well, after this it seemed to be a waste to carry on with CMake when 50% of the builds failed. So i started off on a quest to convert the packages to autotools. Now, the GTK plugin completely uses autotools and works fine on Ubuntu 11.04 and F15. I am in the process of converting the other packages to use autotools as well for the sake of uniformity and maintainability. But that would ofc, depend on the approval of the respective package maintainers.

Another change this week is the removal of the git repo at gitorious.org/libyui/libyui. This space will be filled by the official libyui repository once the svn-git conversion is complete. The old repo can be found at gitorious.org/libyui/gsoc2011. I ll be maintaining all project related files there including the source packages that have been converted to autotools and examples. I am no expert at autotools. Infact, i had to do quite a bit of studying to replace cmake. So if anyone with a good autotools background finds any flaws with the packages, please leave a comment below.

The documentation would be the next task. At present, the documentation is generated by a makefile using doxygen. This is not linked to the documentation servers at openSUSE and hence, there are no weekly updates yet. This would be implemented once the YUI repo is shifted to git.

The GTK package will be added to OBS soon.

Less than 500 words. :)

GsoC Report – 2

Hi. The last week was full of changes for libYUI. To start off, the packages are no longer being built locally and the build process is now handled by OBS. This makes libYUI available on a wide range of operating systems and architectures. Currently, the build at OBS is being done for Fedora 14/15 as well as Ubuntu 10.10/11.04 for both x86 and x86_64 architectures. Interested people can find the OBS repository at http://download.opensuse.org/repositories/home:/nbprashanth/. This means that the binaries will no longer be availalbe at the gitorious repository(gitorious.org/libyui).

Secondly, the initial documentation for the library is hosted at http://doc.opensuse.org/projects/libyui/HEAD/. This documentation is subjected to frequent updates and will change regularly (weekly updates planned). Again, to avoid duplications, the gitorious repository will not contain the documentation anymore.

People who test the new packages are requested to submit bug reports (here or at OBS) in order to help improve libYUI.

GsoC Report – 1

This is my GsoC report for the first two weeks (May 23 – June 04). I
will also in brief state the goals of the project and the future plans.

My project for this years GsoC is “Separating libYUI” for the OpenSUSE
organization. My mentor for the project is Martin Vidner and the
abstract can be found at : http://www.google-melange.com/gsoc/project/google/gsoc2011/nbprashanth/22003

Let me begin by laying out the main goals of this project. The project
aims to :

  1. Separate YUI from the YaST framework.
  2. Port YUI to other major Operating Systems such as Ubuntu and
    Fedora.
  3. Remove the libzypp dependency for the gtk pluging and hence making
    YUI SUSE-independent.
  4. Improve documentation with doxygen.
  5. Provide small tutorials.
  6. To provide a large and non-trivial example to show the capability of
    YUI.

The work done so far is listed below :

  1. libYUI has been successfully compiled on Ubuntu 11.04 and Fedora
    15.
  2. Initial packages for libYUI along with ncurses and Qt plugins have
    been made. They are available in my gitorious account :
    gitorious.org/libyui
  3. Small scale examples are being added to this blog. I will try to
    maintain a weekly schedule.
  4. Currently, I am trying to eliminate the libzypp dependency for the
    Gtk plugin. This is important as libzypp is highly OpenSUSE
    centric.
  5. The packages are currently in the testing phase. So all interested
    people, please feel free to test and post any bugs you may find in the
    comments section. Please note, this is an initial version.
  6. Patches have been made for small bugs like “long long” warnings and
    enumeration ending with a ‘,’. More serious bugs, if reported will be
    tended to when encountered.

Another important change that I would like to implement is the use of a
uniform build system. Currently, some parts use CMake while others use
autotools. Though this does not make much of a difference to the outside
world, I think it will be easier for developers and porters in the long
run. But this will be done later on unless there is an urgent need for
it.

Since my end semester exams ended on May 27th, no work was done during
the first week of GsoC. Hence, this can be regarded as a summary of my
work for the second week (May 29 – June 04). Also, please note that I
will maintain weekly reports on the blog henceforth.

Please feel free to add comments and suggestions below.

Quick Update on YUI Port

After one and a half weeks of hunting down missing libraries and header files (dependency hell), a lot of compilation, and a lot more of SVN checkouts, it’s finally done. All three interfaces of YUI run well on Ubuntu 10.10. I have just finished porting and have tested with the standard examples provided along with YUI.

I ll post more details soon. But now, I have an exam coming up in an hour!!

GsoC 2011

After being an enthusiast about computing (well programming) for so many years, I finally got an opportunity to go beyond the realm of small time programming into the world of open source. Though I am not exactly new to open source, I must say that i have not done anything significant. After nearly 3 years of being a hardcore linux fan, i finally decided to start contributing to open source – through GsoC 2011.

I attended GNOME.Asia summit 2011 in Bangalore earlier this month where i attended a talk by one of my friends (Manu Gupta) on libYUI. Impressed by it’s abilities and it’s potential, I thought that it would make an ideal project to start out with as it was also a GsoC topic (http://www.google-melange.com/gsoc/homepage/google/gsoc2011). And hence, here we are!

I am applying for GSoC, with the http://en.opensuse.org/openSUSE:GSOC_2011_Ideas#Separate_libyui project.

For those of you who do not know what libYUI is, here is a quick intro. libYUI can be thought as an abstraction layer for Qt, GTK and ncurses UI’s. So, a single program written using YUI can produce outputs in 3 different frameworks. Seems great, doesn’t it? But there is a downside to it as well. It was written for the YaST tool (openSUSE) and is heavily coupled along with it. As a consequence, it is available only on the openSUSE platform.

The goal of the project is to separate libYUI from it’s YaST bindings so that it can be an independent framework on it’s own. Moreover, this would allow it to be easily ported to other linux based distributions so that more people can use and benefit from this brilliant framework!

Google has done a great job with GsoC. It is an ideal platform to start off with if open source is your passion. The results would be out later this month, but i have already started working! The yast community is small, but very helpful. Along with help from the community and Martin Vidner (Mentor for libYUI project), quite a bit of work has been done and a lot has been learnt. Over the next few posts, I will post the challenges faced, the hurdles overcome and the progress made.

Current status : Qt and ncurses run on ubuntu!

Follow

Get every new post delivered to your Inbox.