15 Temmuz 2010 Perşembe

Preparing a Pisi Package (introduction)


    Preparing a Pisi package is a step-by-step procedure which is easy to learn but difficult to apply and solve possible problems. I have followed how to manuals and examples on Pardus Wiki about preparing a Pisi package. To see the document on making a package for the beginners, you can visit “http://tr.pardus-wiki.org/NASIL:Yeni_ba%C5%9Flayanlar_i%C3%A7in_paket_yap%C4%B1m_rehberi”. Adding package to pisi system is more than just making a Pisi package. It is a process including gathering information about the package you want to make Pisi package of, testing and debugging your Pisi package, keeping up with newer versions. I also learned the underlying mechanism to move on. Packaging mechanism makes binary packages that Pisi uses by using source codes. The process is like following: Source codes get downloaded by processing source part of the pspec.xml file and after it gets verified, it gets extracted to /var/pisi/package_name/work. Then, to obtain binary data action.py gets processes and installation is made to /var/pisi/package_name/install using source code. If there is any additional file they get coppied to /install directory as it defined at pspec.xml. After that, Pisi makes index file for the files in the /install directory. This index is compatible with pisi database and in XML format. Then, index and binary get combined with datas obtained from pspec.xml and archived as .pisi package. As we can see, pisi installs package to /var/pisi/paket_adi/install while it produces .pisi file. By examining this directory, we can see which files go to which directory. Understanding this mechanism was important for me before I start making a package.
    In order to make a Pisi package, I have to prepare at least two files: pspec.xml and actions.py. As I mentioned before, pspec.xml file is about definition and content of package and packaging progress. The general structure is shown here: "http://en.pardus-wiki.org/Making_Pisi_Packages#pspec.xml"
    The source part gives basic information about package to Pisi. Also, which category and part that package belongs in Pardus system structure, which packages need to exist while compiling, which patches need to applied before compiling process are all stated in the source part. The general structure is shown here: "http://en.pardus-wiki.org/Making_Pisi_Packages#Source_part".
    IsA tag is required to determine which category that package belongs and this tag can be defined more than once. It can take values as app:gui, app:console, library, daemon, service etc. I will explain how to fill each part by examples later. Before moving further, the concept of dependency is important to learn. As I have learned, dependecy means the packages that are required for a package to be built and installed. There are two types of dependencies as build dependency (required for a package to be built by pisi bi...) and run-time dependency (required for a package to be installed later by pisi it...). 
    Package part keeps definitions about pisi files that will be created after packaging. If it is necessary to create more than one pisi file from a package, every one of them should be written in the same pspec.xml file, but into different Package part for each. The general struture is shown here: "http://en.pardus-wiki.org/Making_Pisi_Packages#Package_part"
    Files node here is important, since source file is installed to /var/pisi/package_name/install. Directories that Path tag uses should be these directories. Contents of directories will be added to pisi file by pisi. While these files are added to directories, they are added according to their "fileType" property.
    Lastly, History part keeps the information about updates and updaters by update nodes that every updater adds. The general structure is shown here: "http://en.pardus-wiki.org/Making_Pisi_Packages#History_part"
    You can check the most recent and correct specification of pspec.xml file from here: “http://svn.pardus.org.tr/uludag/trunk/pisi/pisi-spec.rng”. After preparing pspec.xml file, it is time to prepare actions.py file written in Python. This file uses different Actions API and has a general or example structure as follows:

#!/usr/bin/python     
...
def setup():
    #Preparation processes
def build():
    #Compile processes
def install():
    #Installation processes
    We know that when we install a program from the source code, we use configure + make + install commands in this order. Similarly in acitons.py, when setup() is run by pisi we need to do configure, when build() is run we need to do make, and when install() is run we need to do install processes. However, it is not that much simple for most of the time. We need to use correct actions APIs with proper parameters. To learn how to write actions.py file, we need to learn configure and make parameters from the documents of the program on which we want to prepare a package. Then we need to look for proper Actions API functions for necessary processes we learned from these documents. From that point on, trial-error will be our guide. We can add new functions to fix the problems and check the result by viewing /var/pisi/package_name/install directory. Thus, there is no fixed action.py file anc each package may have its own setup(), build() and install() parts. For the beginners like me, it was a good idea to examine many actions.py files in SVN repository. Moreover, “http://tr.pardus-wiki.org/Pardus:ActionsAPI” will always be my guide to find the necessary API usage and parameters that will do my intended task.
    In addition to pspec.xml and action.py files, we can wirte a translations.xml file. We can write multi-lingual description and summary sections at this file. This file must be located at same folder with pspec.xml file. In this file, I generally provide Turkish versions of the description and summary parts in pspec.xml file. In addition, we can also add files/ directory while making a package. If there is a patch for source file, it must be placed at this directory. Also some files which source package does not contain but application needs for running are put here, like icons etc. Patches could be needed to solve some external problems without changing the actual source code. I will explain how to prepare each of these files by examples later.
    Finally I have prepared my first-example package called merhaba-pisi (Pisi Hello World). It is a simple program that when you type “merhaba-pisi” from console, it responds “sana da merhaba username/root”. The source code is written in Python and put into an online source page in a compressed format. This is the starting condition for preparing a new package that has a source code available online. You can find how to fill in pspec.xml and actions.py files for this example from Pardus Wiki manual: “http://tr.pardus-wiki.org/NASIL:PiSi_Hello_World”. It was a good example for me to learn how to fill these files, by searching program's/application's website and documents that come out from tar.gz source files etc. By following this tutorial, finally I prepared a new package, then built it by pisi bi command, and finally installed by pisi it command. After these steps, now it was available in my system and whenever I type “merhaba-pisi” from console, it writes back “sana da merhaba root/pars”, thus it works fine.

Choosing a Project and Pre-works


    In Pardus, a project is assigned for each intern to work on during his/her internship period each summer. The list of available projects are announced on Pardus Wiki, see “http://tr.pardus-wiki.org/Staj2010#Projeler”. Interns can choose one of these projects, or developers of Pardus – as a mentor- can assign a project to them, or even an intern can come up with an idea/project that he/she wants to work on (if it is acceptable). After my orientation period is over, I gave a break on searching and learning the general system, and I started to decide on my project. Our internship coordinator gave me some ideas about each project, and I read their descriptions, requirements, difficulty levels and supplementary materials on Pardus Wiki. After this research, we decided to chose the project called “Engelsiz Pardus” (i.e. unimpeded or abled Pardus). The aim of this project is providing some improvements on Pardus for the benefits of users with certain disabilities. Some applications like screen readers should be integrated with Pardus and KDE in order to make Pardus easier to use for disabled people. The tasks can be listed as researching some screen reader applications and speech synthesizers like Orca, Fire Vox, Linux Screen Reader, Suse BLinux, KTTSD/KTTSmgr; and then choosing the most appropriate ones that can be suitable to integrate with Pardus, and finally adding these properties to Pardus. The requirements for this project is the knowledge of PiSi packaging, since many new packages will be prepared and possible problems will be solved during this integration. Moreover, I should learn at least the general syntax and structure of Python language since it will be used while preparing actions.py python script file for each package. Thus, my working schedule is set like this: first of all, pre-research on similar and useful screen reader applications, collecting necessary informations/links/sources/documents about them, examining these materials together with examining current Pardus packages, optimizing the availability and integration possibility etc. While doing these, I should also start learning how to prepare Pisi packages and how to solve possible problems, in addition to learning Python.
     The pre-work period of the project took long time than we expected. The reason is that what we can do is not so obvious and research would be a possible starting point. Thus, after first days of general orientation and education, I have spent nearly 2 days on searching and reading about different screen readers, speech synthesizers and dispatchers, examining their documentation, how to manuals, installation and system requirements details etc. I have collected links and sources that will be helpful later. I also try to kept in mind that we are looking for the ones that can be applicable to Pardus OS. For instance, if an application requires some desktop or library dependencies that are not found in Pardus repository, we should look if these pre-requirements can be matched easily. If they are difficult or nearly impossible to be matched, we should eliminate these applications. Moreover, while comparing more than one application that does more or less the same, we should focus on the one that has less dependencies or the one with dependencies that are already found in Pardus. The reason is that, if we cannot meet these dependencies, we need to prepare a package for each of these dependencies first, and then try to prepare a package for the general application. Unfortunately, if this requirement package also has a dependency that is not present, we need to prepare another package for that too etc. As you see, it is a recursive process and choosing the easiest but powerful option is hard to pick. Thus it deserves some research time before jumping into the first option we have.
     With this motivation, I have started searching on different applications. The list of possible screen readers and comparisons can be listed from here: “http://en.wikipedia.org/wiki/Comparison_of_screen_readers” or speech synthesizers from here: “http://en.wikipedia.org/wiki/Comparison_of_speech_synthesizers”. The distinction between screen readers and speech synthesizers is important. Speech synthesis is the artificial production of human speech and a computer system used for this purpose is called a speech synthesizer. A text-to-speech (TTS) system converts normal language text into speech. An intelligible text-to-speech program allows people with visual impairments or reading disabilities to listen to written works on a home computer. Some example speech synthesizers that work on unix/Linux systems are eSpeak, Festival, Flite, FreeTTS, OpenTTS. However, screen reader is more general and different. A screen reader is a software application that attempts to identify and interpret what is being displayed on the screen. This interpretation is then represented to the user with text-to-speech. Thus, we can say that screen readers use underlying speech synthesizers to work. Some example screen readers that work on unix/Linux platforms are Brltty, Fire Vox, Linux Screen Reader (LSR), Orca, Suse Blinux, Emacspeak. I have examined each of them form their websites and forums, read their documentation to have an idea and to report back when necessary. I cannot summarize all the technical details about each of them here, but I had general idea about each of them. Furthermore, since Pardus uses KDE (a desktop environment provided as the default working environment on many Linux distributions), we should focus on applications suitable for KDE. Since most of them are prepared for GNOME desktop, even if we know that applications for KDE or GNOME can work for each other using their libraries, choosing the ones for KDE is better. For that purpose, KDE has KTTS which stands for KDE Text-to-Speech. It is a subsystem within the KDE desktop for conversion of text to audible speech. This is a part of KDE accessibility project and you can visit KTTS from: “http://accessibility.kde.org/developer/kttsd/”. It has main parts as KTTSD (The KDE Text-to-Speech Deamon, a non-gui application that runs in the background, providing TTS support to KDE applications) and KTTSMGR (An application for configuring and controlling KTTSD). After working on its road map and general documentation, we discussed with project mentors and decided to focus on KTTS. Since this project can be so huge and many things could be added over time, it will never be finished; especially during the limited internship period. Thus, we decided to start from kde accessibility and continue as much as we can. The “to do” list is prepared that involves what packages should be prepared at first, by looking at requirements and dependencies that I have collected and reported. Kde accessibility needs speech dispatcher or OpenTTS, then OpenTTS needs Dot.conf library, eSpeak, Flite, Festival-freebsoft-utilsas speech synthesizers, and at-spi2-core plus at-spi2-atk etc. Moreover, for Fire Vox we need Orca or FreeTTS, Brltty, python-pyatspi etc. My first package to be prepared is Dot.conf library, but before that I need to learn how to prepare a Pisi package from the beginning.

Orientation

    First days of the internship period at Pardus can be defined as an orientation. I had my own desk and PC, and I installed the last version of Pardus, Pardus 2009.2 Geronticus eremita. During these first days, the activities can be listed as meeting with the staff and other interns, observing the specific responsibilities of each Pardus developers, observing the general working environment of software/kernel developers. I reviewed my operating systems and especially Linux usage knowledge, bash commands, usage of Linux text editors like Vim, Vi, Emacs etc. Moreover, I have started learning the general system architecture of Pardus. They prepared a handbook for interns, which involves the list of things to examine during the orientation and I followed this handbook as a tutorial. 
     In this handbook/tutorial, the necessary information was given about most frequently used Pardus components such as Yalı, Çomar, Pisi, Tasma, Müdür, Kaptan etc. For more information, I searched them on Pardus Wiki which involves the documentation, how to manuals and all necessary information about Pardus. You can visit Pardus Wiki from “http://tr.pardus-wiki.org”, or the English version from “http://en.pardus-wiki.org/Main_Page”. In short, we can say that Yalı is a system component responsible for the setup and installation of Pardus OS, Kaptan helps to guide the user for the personalization of desktop and connection settings, Müdür is responsible for automatically recognizing the hardware devices connected, Çomar is the Configuration Manager of Pardus and Tasma is the graphical interface that transmits the user settings to Çomar architecture. The most important system component of Pardus is PiSi, which stands for Packages Installed Successfully as Intended.
    Before understanding the structure of PiSi and PiSi packages, I reviewed the concepts like packages, scripts, APIs and Actions API. In general, a software package refers to computer software packaged in an archive format to be installed by a package manager system. Linux distributions are generally segmented into packages. Each package contains a specific application or service and each package contains meta-information such as a package description, version, and dependencies. The package management system can evaluate this meta-information to allow package searches, to perform an automatic upgrade to a newer version, to check that all dependencies of a package are fulfilled and sometimes to fulfill them automatically. Furthermore, "scripts" are distinct from the core code of the application, as they are usually written in a different language and are often created or at least modified by the end users. Scripts are often interpreted from source code, whereas the applications they control are traditionally compiled to native machine code. In addition, an application programming interface (API) is an interface implemented by a software program which enables it to interact with other software. An API is implemented by applications, libraries and OS to determine their vocabularies and calling conversions, and is used to access their services. Actions APIs are used in PiSi packages, which I will mention later in detail.
     After gaining these basic knowledge, I was ready to examine PiSi package structure. As I mentioned before, PiSi is a binary package management system which have been developed within Pardus project. It is the tool that installs, upgrades and removes the software packages successfully. PiSi stores and handles the dependencies for the other packages, libraries and tasks. You can have more information about PiSi from “http://tr.pardus-wiki.org/Pisi”. PiSi is implemented in Python, a programming language that I did not know, and thus learning Python would be one of my internship missions. By the way, Pardus graphical designs and codes for GUIs are written using Qt, fortunately which we have learned in Cmpe230 Systems Porgramming course. However, I would need to review my Qt knowledge during my internship too.
     After that, I have learned how to use PiSi commands which are invoked in the command-line trough the `pisi` executable. You can see the document that provides an overview of Pisi usage on the command line (pisi-cli) from “ http://en.pardus-wiki.org/Pisi_CLI_Usage ”. PiSi package sources are written in XML and Python. The structure of a Pisi source package consists of an XML file named “pspec.xml” which is for description, a Python script named “actions.py” which is for construction (setup, build, install), another optional XML file named “translations.xml” which is for language supports, and a directiory named files/ for optional additional files (like pathces).
     I have examined the general structure of these two basic files for a Pisi package, “pspec.xml” and “actions.py”. Pspec.xml is an XML file containing at least 3 child nodes: Source, Package, History. There can be multiple Package nodes in here, which means that one source package may generate multiple binary packages. Source node contains general package and packager information, sha1sum, type and location of source archive, Patches and BuildDependencies. Package node contains RuntimeDependencies and locations of different file types, AdditionalFiles and Comar scripts if needed. History node simply contains information about package history. The Package may contain the AdditionalFiles tag, which can be used to copy files from the files subdirectory of your source tree into the .pisi. After gaining this general information on pspec.xml file, I moved into the actions.py file. Actions.py file contains Python codes that would compile and install the source package into a specific install directory. In this actions.py file, we use Actions API that comes with Pisi. Actions API has all functions for us to compile and install our package. You can see all Actions API definitions and usages from “http://tr.pardus-wiki.org/Pardus:ActionsAPI”. 
     After preparing pspec.xml and actions.py properly, we can easily build a Pisi package by typing “sudo pisi build pspec.xml” command in the console. If the package is successfully built, then a file with .pisi extension is formed in the current directory. If we want to install this package, we should type “sudo pisi install mypackage-x-y-z.pisi” command in the console. I will explain the details of how to prepare a package by preparing pspec.xml and actions.py later, since this was the second weeks job.
     Before moving forward, the concept of subversion (SVN) is needed to be learned in order to understand the Pardus source repository structure and usage. The development process in Pardus is maintained via a Subversion version control system. Subversion is an open source version tracking system. Its a development infrastructure which makes it possible for more than one application developer to work together not worried about destroying each others' changes. By SVN, any single software's development process can be tracked backwards, the changes made gradually can be watched and can be easily returned to any version of a particular time. After understanding the SVN system, the concept of repository came next. A repository is a disk area on which the last version, all the previous versions and the changes between versions of the software packages every developer works on, information including their user, date and cause are stored which can be reached. We can view Pardus repositories using svn form “http://svn.pardus.org.tr”. I have examined Pardus official packages repository, Contrib packages repository, Uludag repository and Websvn (for svn logs). There are also different subdirectories within them like devel/, stable/, trunk/, branches/ etc. with different meanings and usages. After surfing around these repositories, I learned how to use SVN from console using basic SVN commands. For instance, to get a directory/file from a repository, you should type >svn checkout http://...source link.... You can olsa type >svn add ..., >svn update, >svn diff (for diff output of what you have changed). All these changes are not reflected in the repositories until you commit them with >svn ci -m “commit comment” command. You can learn about svn, repository and all from “http://tr.pardus-wiki.org/Pardus:Yeni_geli%C5%9Ftirici_k%C4%B1lavuzu”.
     My next mission in the handbook for interns was preparing a patch and applying it to the Pardus sources. The aim was to change the text on one of the buttons used in Kaptan's GUI. I wanted to change the text “next” on the moving forward button of Kaptan and make it to “Eda”. To do this, first of all I needed to checkout Kaptan's sources from Uludag repository with “svn co http://svn.pardus.org.tr/uludag/trunk/kde/kaptan/” command. Then I moved into src/gui/ and opened the file kaptanMain.ui with Qt Designer. I changed the text on the button from “next” to “Eda” and saved. I could see the changes I have made by typing “svn diff”. Then, I saved these changes into a file by typing “svn diff > change-button-text-to-eda.patch”. At this step, my patch was ready. To apply this patch into the system, I needed to checkout the kaptan package from Pardus repository into my workspace with “svn co http://svn.pardus.org.tr/pardus/2009/devel/desktop/kde/addon/kaptan/”. In this kaptan/ directory we have pspec.xml, action.py, translations.xml and files/.Since all patches should be located into files/, I copied the patch that I have created into files/. Then I needed to define this patch to the package, and I opened pspec.xml for that purpose. I needed to add “change-button-text-to-eda.patch” line into pspec.xml. This time, my patched package was ready to be built and I typed “sudo pisi bi pspec.xml” command. This command creates a kaptan-x-y-z.pisi (x-y-z for any versions) which is a Pisi package ready to be installed. Finally, I installed it by typing “sudo pisi it kaptan-x-y-z.pisi”command. After doing all these, I have tried to run Kaptan and saw that “Eda” is written on the next button! It was nice to see that I can change nearly anything in the system, play with Gui's and source codes in whatever way I like. If anyone wants to send his/her patch or any changes to the developer of this package, he/she can use svn Contrib repository which is an unofficial repository that contains packages provided by the users. As an intern, we had our own svn accounts activated so that we can commit anything into playground directory of Uludag repository where source codes and gui designs of Pardus developers are found. The exact place where internship projects are located is http://svn.pardus.org.tr/uludag/trunk/playground/intern/.


14 Temmuz 2010 Çarşamba

Introduction

This blog is dedicated for my internship period at Pardus, Tübitak-UEKAE.

What is Tubitak, UEKAE?

The National Research Institute of Electronics and Cryptology (UEKAE), an affiliate of the Scientific and Technological Research Council of Turkey (Tübitak), produces and applies technological and scientific solutions in the areas of information security, communications and advanced electronics.


What is Pardus?

National Operating System (PARDUS) is being developed under the National Distribution Project in the UEKAE. Pardus is a GNU/Linux distribution and it targets the basic desktop needs of the computer users.

For more information about vision, aims and history of Pardus; you can visit http://www.pardus.org.tr/eng/about/

Recent product:

Pardus 2009.2 Geronticus eremita (03.06.2010)

You can download Pardus 2009.2 Geronticus eremita from here, read the release notes from here.