Andrew's SBR600 Blog

0.3 Release - Part 1

2010-12-17T06:49:00.000-08:00

Unfortunately, I did not fully accomplish my goal for release 0.3 - I was able to make progress but in the end I did not turn out the way I had hoped. I tried to get JSON output working as exampled in the BuildAPI existing scripts but could not get it going. However, I did learn a tremendous amount of information which can be seen by browsing through my blog.

In addition, I have created a quick and concise How-to document for BuildAPI in hopes that the next student that possibly takes on BuildAPI will have a head start into the project and a better understanding of its inner workings without wasting too much time figuring it out.
Furthermore, there will also be a second document that provides more information on the controllers, models and template files that will be posted later.

Documentation

Here is a link to the documentation: http://asdesigned.ca/sbr/BuildAPI-How-To-Setup.pdf
It is in PDF form (clickable links included).

Below is my Getting Started documentation in HTML format, although I'd recommend viewing the PDF version instead:

Seneca College

BuildAPI

How to Get Started with BuildAPI

Andrew Singh

12/17/2010

Table of Contents

Introduction

Purpose

Tasks Summary

I. Install MySQL and Import Database Snapshots

II. Install Python and Python SetupTools (Easy_install)

III. Install Google Python Visualizations Library

IV. Install MySQL-Python (MySQLdb for Python)

V. Install BuildAPI Files

VI. Configure and Run BuildAPI

VII. Optional: Set up sample project files

VIII. Resources

BuildAPI

“BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.”

Introduction

This project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying databases for information on the BuildBot jobs, which will then be used to generate the necessary reports.

Purpose

To install, configure and run BuildAPI

Tasks Summary

Install MySQL and import database snapshots
Install Python, Python SetupTools (Easy_install) and Pylons
Install Google Python Visualizations Library
Install MySQL-Python (MySQLdb for Python)
Install BuildAPI Files
Configure and Run BuildAPI
Optional: Set up sample project files
Resources

I. Install MySQL and Import Database Snapshots

Install MySQL

Install Mysql and Mysql Server:

yum –y install mysql mysql-server

Enable the service at startup:

/sbin/chkconfig mysqld on

Start the MySQL server

/sbin/service mysqld start

Set the MySQL root password

mysqladmin -u root password 'new-password'

The quotes around the new password are required

Root can be used to access MySQL or a new MySQL user can be created for the project.

For testing purposes, root can be used.

Install Snapshots

Download the database snapshot files, which are in bunzip2 format. Please check https://wiki.mozilla.org/ReleaseEngineering/BuildAPI for the latest snapshot releases, under “Project Requirements”.

If bunzip2 does not exist on the system, install it using: yum install bzip2

Extract the databases

bunzip2 schedulerdb-2010-10-22.sql.bz2

bunzip2 statusdb-2010-10-22.sql.bz2

Create the respective databases; Example: statusdb and schedulerdb in MySQL

$ mysql –uroot -ppassword

mysql>

mysql> create database statusdb;

mysql> create database schedulerdb;

mysql> exit

Import the databases into mysql server

mysql –uroot -ppassword -hlocalhost statusdb < statusdb.sql

mysql –uroot -ppassword -hlocalhost statusdb < schedulerdb.sql

II. Install Python and Python SetupTools (Easy_install)

Install Python

yum install python

Install Setup Tools

yum install python-setuptools

Install Pylons

yum install pylons or easy_install Pylons

yum install python-pylons

Virtual Environment

We can also set up a virtual environment to work in, whereby it allows us to have multiple isolated environments for Python so it is possible to run different applications on the same system, but using different sets of installed Python packages. This is recommended by the Mozilla Release Engineering team.

Install virtual environment using easy_install and read the readme file to activate

easy_install virtualenv

Manually installing virtual environment

Use wget to get the latest package from http://pypi.python.org/packages/source/v/virtualenv/

Untar the downloaded file, create a directory and activate the environment

$ tar zxfv filename.tar.gz

$ virtualenv.py ~/venv/sandbox

$ source ~/venv/sandbox/bin/activate

Once in the sandbox, install packages as necessary, example:

(sandbox)$ easy_install Pylons==0.9.6.1

If you will be working within a virtual environment, it is important that you install the necessary packages below in the same environment as well.

III. Install Google Python Visualizations Library

Download the library from http://google-visualization-python.googlecode.com/ and extract it.

To install the library:

cd into the extracted directory and run the command:

python ./setup.py install

Test the library by running python ./setup.py test to make sure it is installed correctly

IV. Install MySQL-Python (MySQLdb for Python)

Install Dependencies

This package requires the python-devel dependency or it will not install properly.

yum install python-devel

Install Package

Once that is complete, we can install MySQLdb.

You can stop the mysql server before installing MySQL-Python and restart it once complete.

yum install mysql-python

In the event that it fails, you can also try a manual installation:

Download the source file from http://download.sourceforge.net/sourceforge/mysql-python/

Extract the downloaded file and change to its directory

Run the commands:

python setup.py build

python setup.py install

Optional: run the test to see if it is installed properly: python setup.py test

V. Install BuildAPI Files

The easiest way to install BuildAPI is to use Python’s setup tools:

easy_install buildapi

This will also install SQLAlchemy which is required; otherwise it can be installed using easy_install SQLAlchemy

For manual installation, you can download the source files from http://hg.mozilla.org/build/buildapi and extract it to a directory called buildapi

This will create a directory structure similar to the following:

------buildapi/

-------MANIFEST.in

-------README.txt

-------docs

-------ez_setup.py

-------buildapi (This is a nested directory)

-------buildapi.egg-info

-------setup.cfg

-------setup.py (used for application setup)

-------test.ini (used for testing, i.e. when you run python setup.py test)

Run the command:

python setup.py install

VI. Configure and Run BuildAPI

Next, BuildAPI must be configured to host content on your system, either through localhost or on the internet. Moreover, it uses the Paster server for hosting and requires creating and editing a configuration file.

We will create our own configuration file using the command:

paster make-config buildapi config.ini

This command produces a different configuration file with sensible options for production use

Now we can edit the created config.ini configuration file and make changes as below.

[server:main]

use = egg:Paste#http

host = 127.0.0.1

port = 5000

# SQLAlchemy database URL

sqlalchemy.scheduler_db.url = mysql://root:root@localhost/schedulerdb

sqlalchemy.status_db.url = mysql://root:root@localhost/statusdb

sqlalchemy.pool_recycle = 3600

Note: host can be set to any value such as iraq.proximity.on.ca to make it accessible on the internet provided configurations such as firewalls allow it. The sqlalchemy database url and username:password fields as they may need editing accordingly if you have used different names or different databases.

Run BuildAPI

To run, stop or reload BuildAPI, make sure you are in the buildapi directory (first level, where the config.ini file resides) and use the following commands

paster serve --start --daemon config.ini

paster serve --stop --daemon config.ini

paster serve --reload --daemon config.ini

Whenever a change is made to the configuration file, or major changes made to controller files, the paster server must be reloaded.

Check the current directory (buildapi), there should be two files that were created:

paster.pid – contains the PID number for the Paster process

paster.log – All messages are contained in this file. Extremely useful file for troubleshooting!

Test it out!

Depending on your config.ini configuration settings, direct the browser to

http://localhost:5000 or http://domain:5000

And that’s it! BuildAPI should now be live on your system!

VII. Optional: Set up sample project files

These are files that were part of the BuildAPI project. They can be used to get a better understanding of BuildAPI and to understand of how it works, along with the blog posts (on http://andrewasdesigned.blogspot.com/).

Additionally, see the https://wiki.mozilla.org/ReleaseEngineering/BuildAPI page for a quick beginner tutorial as well.

File	Description	Download Location
config.ini	Paster Configuration file	http://asdesigned.ca/SBR/config.ini
hello.py	Hello World controller file – a Getting Started controller	http://asdesigned.ca/SBR/hello.py
project.py	Project Controller file	http://asdesigned.ca/SBR/project.py
query.py	Project Query file, which is the BuildAPI query file with Project queries appended	http://asdesigned.ca/SBR/query.py
project.mako	Project template file	http://asdesigned.ca/SBR/project.mako
routing.py	Project routing.py file	http://asdesigned.ca/SBR/routing.py

Directory Structure showing where files should be placed:

buildapi/

+-- buildapi

¦ +-- config #routing.py, placed in config

¦ +-- controllers #project.py, hello.py placed in controllers

¦ +-- model #query.py, placed in model

¦ +-- templates #project.mako, placed in templates

Note that other directories and files have been omitted

The Python library called Routes handles mapping URLs to controllers and their methods, or their action as Routes refers to them. By default, Pylons sets up the following routes (found in config/routing.py):

map.connect('/{controller}/{action}')

Therefore, you can also edit the buildapi/buildapi/config/routing.py file and add the following in manually, instead of using the routing.py provided file:

map.connect('/project', controller='project', action='index')

This file controls how the controller or project site is accessed. The above means that the site is accessed, for example, by www.domain.com/project which will automatically redirect it to the index as defined in the controller file.

See blog posts on http://andrewasdesigned.blogspot.com/ regarding how the model works.

A Word on Database and queries:

It is important to understand the database structure and use the resources to figure out which queries and tables would provide the best indication of certain values, such as CPU usage. The queries are also done using SQLAlchemy and it can get complicated with complex queries, therefore, seek out help (See Resources section).

VIII. Resources

It is extremely important to reach out to the community for help in any area. I cannot stress this enough. There are many resources available, such as IRC, official documentation and discussion groups that you can take advantage of. Some are listed below.

IRC

Server: irc.freenode.net

channels: #seneca #mozilla

Blogs and Wikis:

http://anamariamoz.wordpress.com/

https://wiki.mozilla.org/ReleaseEngineering/BuildAPI

http://andrewasdesigned.blogspot.com/

Discussion Groups

http://groups.google.com/group/pylons-discuss (Google Group)

Official Pylons Documentation

http://pylonshq.com/docs/en/0.9.7/ (Pylons Documentation)

http://pylonsbook.com/ (Free Online Pylons Book and amazing resource)

SBR600 (Seneca) BuildAPI Project Page:

http://zenit.senecac.on.ca/wiki/index.php/BuildAPI

SBR600: Reflection

2010-12-17T01:54:00.000-08:00

Course Reflections

For this to be my final semester, I had to take 7 courses, and so I did. I figured it would be challenging and it definitely was – especially also working part-time plus other responsibilities, but sometimes you 'just gotta do what you gotta do'.

I know I’m probably not the only one as there were others that were in similar positions; and to you all out there who’s graduating: Kudos! Cheers! This chapter is done, complete or whatever your word is, either way, run and tell that, run and tell that, run and tell that, home home home homeboy (if you don’t get the reference, click here and enjoy)

Okay, on to what is important though: My reflections on the course.
The first half was interesting, and I liked the fact that we had a tentative schedule that allowed us divide and devote our time appropriately to the course.

It was good to learn about Fedora’s build and release and I enjoyed creating RPMs and going through the process of creating our own SPEC files, testing build times, signing our RPMS and testing packages on different platforms.

I have to say that was definitely one of my favorite parts of this course. If I had to do this course again (and not that I want to, lol) and choose a project I would have chosen one to rebuild packages or test builds and build times, but no worries, there was only love for BuildAPI...and some frustration. I’ll leave my thoughts on that for last.

As I mentioned in my GIT blog post I’ve used version control clients (SVN) before but never GIT. However, after playing around with it for this course, I actually used it for another course, INT620, which involved creating a website using PERL and TT2. I used GIT to keep track of changes to the website files and it worked out quite well…for the period of time I used it as we eventually switched to using Mercurial. I’m kind of ranting off topic here. Let’s move on to the projects.

It was my fault for missing the class where Chris Tyler explained each project and the respective general expectations. However, we were given a wide choice of projects to choose from that were well thought out and incorporated a little bit of everything; it definitely gains points on my recommendation scale for this course! In regards to that, if there any students who have not taken SBR600 yet, I would highly suggest that you do and also see my “Advice” section later on this post.

The second half of the course focused on developing my chosen project - BuildAPI. I must say there were difficulties along the way and these projects require our own initiative to progress. There were the tasks of learning Python, Pylons, SQLAlchemy queries, MVC model (see previous blog posts) and integrating it all. Although, I may have went too deep into it and should have been more focused on specific goals rather than trying to figure out everything in such a short time.

BuildAPI was especially challenging at the beginning, but I pushed along albeit slowly to gain results and ultimately new knowledge. However, as I worked my way through and progressed, there was a satisfying sense of accomplishment from researching, learning and putting the knowledge to practical use.

Even while working on my INT620 website project using Perl and TT2, I often reflected on how our project can also be done in pylons since it encompassed similar processes as my BuildAPI project: querying databases, displaying/managing data and using a model like the MVC to separate the application code from business.

But I also believe I learned something much greater than learning the ropes on my project. And that is the value of the resources available - the large community of able-bodied and willingly helpful souls. SBR600 opened my eyes to the “community”.

What about my advice for those who plan on taking this course?
-    Do it, take it
-    Choose your project wisely and one that you will thoroughly enjoy
      o    Some projects involve scripting – such as learning Python (which you ultimately do by yourself)
-    Get involved with the community – seek help, and do not be afraid to ask ANY question regardless of how “ridiculous” you may think it sounds
-    Get INVOLVED with the COMMUNITY, success in the course, project and perhaps your career depend on it

Lastly, Cheers and Thanks to: Chris Tyler, Armen Zambrano, Mike Orr, fellow students, IRC users cthomas and aki, blogs, Mozilla Release Engineering, others and the Google group - Pylons-Discuss for all the help!

Oh wait....I have to post 0.3 for the project! I will do that shortly!

BuildAPI: 0.2

2010-11-26T03:44:00.000-08:00

Quick note:

Last week I posted a series of blog posts that I did not have time to post during the past week or more. They were all sort of backed up and hence all posted at once.

BuildAPI 0.2

On to the BuildAPI project: Let me just say that I have been doing this project in a step-by-step approach in order to learn what I need to learn, understand what I am working with, and apply it practically.
Also, the BuildAPI wiki page has been updated: http://zenit.senecac.on.ca/wiki/index.php/BuildAPI

To see it running: http://iraq.proximity.on.ca:5000/
Please click on Project 0.2 under BuildAPI Project Test, or http://iraq.proximity.on.ca:5000/project

0.1x - Between 0.1 and 0.2

My 0.1 release included getting BuildAPI set up and taking a peek at the extensively used MVC model.
However, since BuildAPI uses Pylons framework, I took some time to read The Definitive Guide to Pylons book and skipped to the chapters that were necessary. It's a good thing the book is available online, it is such a great resource! It helped me to learn some of the language and concepts. Next was to create very simple controllers for working with model and view templates and that was the beginning of wrapping my head around the MVC model concepts.

0.2 - Summary

For 0.2, I feel very comfortable working with the MVC model and was able to create my own controller, add to the existing BuildAPI universal model script for querying (query.py, orginal seen here), and create my own MAKO template to output my results. (See MVC scripts). In addition, I appended a link to my project to the main BuildAPI index page, which I have set up on http://iraq.proximity.on.ca:5000.

In regards to the model file, I had to read some of the documentation on SQLAlchemy and how queries work in order to pull data from the database, and from that, I now understand how to formulate basic queries using SQLAlchemy. However, I am still having a bit of trouble with complex queries such as using joins and multiple where clauses (See blog post on SQLAlchemy and Basic Queries).

I also stripped special formatting from the output and template as I did not fully understand how it works and how it is integrated with Pylons and BuildAPI - it seemed confusing. By special formatting, I am referring to Javascript/JSON and Json tables and charts.
However, after a bit of fiddling and reading on google visualizations documentation, I get the gist of how it works and I have implemented simple json using google viz api. Although, I still need to seek out help and apply it practically - which will be in 0.3.

MVC – Revisited

From my 0.1 post, I'd like to revisit the MVC model in terms of the BuildAPI project instead of a generic breakdown. I will also reference the MVC scripts which can be seen later in the post.
Here goes

The model (query.py) knows about the databases and queries, or in other terms, business objects/data
The template (project.mako) knows about HTML, Javascript, CSS, and the tmpl_context or "c" variables (example: c_output in controller projecy.py script) that are passed to it by the controller.
The controller knows about both the model and the template. Therefore, it will query or modify the model to obtain data, modifies the values of the "c" variables and then passes it to the template.
Also, the model can invoke other models, and the controller other controllers etc

After working with it, I can definitely see how each component can be updated separately without affecting the other.

MVC Scripts

The Controller - project.py

import logging

from pylons import request, response, session, tmpl_context as c, url

from pylons.controllers.util import abort, redirect

from pylons.decorators import jsonify

from buildapi.lib.base import BaseController, render

from buildapi.model.query import GetProjectQuery

from buildapi.model.query import GetQueryTest

log = logging.getLogger(__name__)

class ProjectController(BaseController):

    def index(self):

        output = GetProjectQuery()

        results = GetQueryTest()

        c.output = output

        c.results = results

        return render("/project.mako")

        return self.jsonify(output)

        # What the lines below do is to gather the format of the request using request.GET method 

        #  which contains the variables in a query string

        # These variables are set in the template through javascript or through the form request

        # This is commented out for reasons (See JSON/Javascript section)

        #if 'format' in request.GET:

        #    format = request.GET.getone('format')

        #else:

        #    format = 'html'

        #if format not in ('html', 'json'):

        #    abort(400, detail='Unsupported format: %s' % format)

        #if format == "html":

            # assign to c_output

            # return template render

        #else:

           # jsonify the data                    

           # return self.jsonify(results)

The Model - query.py (My test queries appended to existing file)

def GetProjectQuery():

    rr = meta.status_db_meta.tables['builders']

    q = select([rr.c.id, rr.c.name])

    q = q.limit(50)

    query_results = q.execute()

    output = []

    for r in query_results:

        this_result = {}

        for key, value in r.items():

            this_result[str(key)] = value

        output.append(this_result)

    return output

def GetQueryTest():

    rr = meta.status_db_meta.tables['builders']

    bb = meta.status_db_meta.tables['builds']

    q = select([rr.c.name, bb.c.starttime, bb.c.endtime])

    #q = q.join(rr, bb.c.builder_id = rr.c.id)

    q = q.where(and_(rr.c.id == bb.c.builder_id))

    q = q.where(and_(rr.c.name.like('%moz%')))

    q = q.limit(50)

    query_results = q.execute()

    results = []

    for r in query_results:

        this_result = {}

        for key, value in r.items():

            this_result[str(key)] = value

results.append(this_result)

    return results

The Template - project.mako

Only the important parts shown. This creates the table using the results from the query that are passed in as variables.

This is for the first table:

<tr>

% for key in ('Builder ID','Builder Name'):

<td><h3>${key}</h3></td>

% endfor

</tr>

<tbody>

<%

  from pytz import timezone

  from datetime import datetime

  eastern = timezone('US/Eastern')

  now = datetime.now().replace(microsecond=0)

%>

% for key in c.output:

        <tr>

        % for x in ('id', 'name'):

        <td>${key[x]}</td>

        % endfor

</tr>

% endfor

</tbody>

This is for the second table:

% for key in ('Builder Name','Start Time', 'End Time', 'Duration'):

<td><h3>${key}</h3></td>

% endfor

</tr>

% for key in c.results:

<%

        key['duration'] = key['endtime'] - key['starttime']

%>

        <tr>

        % for x in ('name', 'starttime', 'endtime', 'duration'):

        <td> ${key[x]} </td>

        % endfor

        </tr>

% endfor

The below is simply using Google's visualization to create the table, it can also be used to create charts. I just don't know how to loop the data into a chart/table as yet properly.

<script type="text/javascript" src="http://www.google.com/jsapi"></script>

<script type="text/javascript">

        google.load('visualization', '1', {packages: ['table']});

</script>

% for key in c.output:

        rows: [{c:[{v: '${key['id']}'}, {v: '${key['name']}'}]}]};

% endfor

<script type="text/javascript">

    function drawVisualization() {

      // Create and populate the data table.

    var JSONObject = {

      cols: [{id: 'task', label: 'Builder ID', type: 'string'},

              {id: 'hours', label: 'Builder Name', type: 'string'}],

      rows: [{c:[{v: '${key['id']}'}, {v: '${key['name']}'}]}]};

      var data = new google.visualization.DataTable(JSONObject, 0.5);

      // Create and draw the visualization.

      visualization = new google.visualization.Table(document.getElementById('table'));

      visualization.draw(data, {'allowHtml': true});

}

    google.setOnLoadCallback(drawVisualization);

</script>

After the above script, this is all that's needed in the HTML file that displays the table:

<div id="table"></div>

Javascript, JSON and Outputting data in special formats

That being said, JSON is just another data format structure, which is used for formatting the tables and charts. However, I am trying to get JSON and javascript to work so I can integrate with existing BuildAPI structure, but honestly, it is quite a lot to learn in a short time.

Whatever it is that I am doing or learning, I like to make sure I understand why I am doing it and how it works.

I've researched that the controller can produce the json on request using several methods, one of which is by passing the format specifier. In addition, this can be done either by passing it from a template using javascript to the controller or by using what is called "RESTful services". In Python documentation for Routes and RESTful services, it shows "Several routes are paired with an identical route containing the format variable. The intention is to allow users to obtain different formats by means of filename suffixes; e.g., "/messages/1.xml".

This ultimately means that you could put a test in the controller to see if it JSON or HTML data:

def view(self, id, format='html'):

     if format == 'json':

         return self.jsonify(data)

     else:

         return ('templates/project.mako')

Then, the output will depend on how the data is called, for example:
http://examplepath/controller/page.html - the format will be html thus displaying an HTML page
or
http://examplepath/controller/page.json - using json format and application/json content type

However, BuildAPI takes the approach of using javascript to set the format and furthermore, this javascript is inserted into another template file (call it B) that is invoked by the main template (call it A), so A invokes values from B. Maybe I am just confusing things and I have it all wrong? PLEASE correct me if I am wrong! (I've sent some emails out to request for help on the matter)

In BuildAPI, I understand that the controllers will check the 'format' of the request by using Pylon's "request.GET" and I understand that for the "Reports" this format (such as charts) is set in the respective mako template files. If it is an html request, it shows the HTML page, if it is a json request then it shows the json data in the form of a table or chart in the html page. However, some controllers (namely recent.py, running.py and pending.py) - where is the format set? I don't see it set anywhere in the controller, model or template file and it is not inheriting it from anywhere, yet the format seems to be set and it uses html + json to create those nice looking tables! How?

Moreover, I also received errors when trying to 'jsonify' output with more than one key/value pairs such as a pylons list that contains:
{'name': 'mozilla-1.9.1-win32-unittest', 'starttime': '2009-09-27 14:06:30', 'endtime': '2009-09-27 14:27:50'}
It gives me the error:

Error - <type 'exceptions.TypeError'>: datetime.datetime(2009, 9, 27, 14, 27, 50) is not JSON serializable

Aaahh, the frustration!

Although, I've thought about a simpler solution:

Dump the data from the query into a python list
Then dump that data into json objects and pass it as a c_context variables to the MAKO template
n the template, iterate through the results to display it using google visualizations to create table or charts

However, I've read that this may not be as secure as it perhaps uses the javascript eval() function – but I'm not sure.

I will update this in the near future with a clear path to 0.3.

Update: I recently exchanged emails with Armen Zambrano regarding the objectives and will post another blog in the coming days of what I've decided to do.

Getting BuildAPI online + Previous Errors - Fixed

2010-11-18T20:12:00.000-08:00

Preamble

I had tried to get BuildAPI set up on iraq.proximity.on.ca server before milestone 0.1 release however, I ran into problems that was posted on this linked blog post. I was receiving errors running the command paster serve --reload --daemon config.ini inside or outside of a python virtual environment.

I went back onto the iraq server to take a second look at it.

Fixing Errors

When I ran the paster command by itself it gave me an error. The paster.log provided a lot of information on what could be done to possibly fix it and mentioned to make sure that PasteScript was installed properly.
I ran the command:

easy_install PasteScript==1.7.3

And what do you know, that got the Paster error resolved. So easy eh?

However, I was still having errors with BuildAPI but that was quickly solved after running the following command in the BuildAPI directory:

python setup.py develop

And that was it, just two commands to solve all my problems...I feel like kicking myself (lol).

Serving Content
I set Paster server to host on http://iraq.proximity.on.ca:5000 and credits to Chris Tyler for pointing out that I needed to add an IPtables firewall rule to allow incoming connections to port 5000.
This is so much better than working off of my laptop!

SQLAlchemy MySQL error

2010-11-18T19:55:00.000-08:00

When setting up the Paster server for my BuildAPI project, I used Sqlalchemy and MySQL to connect to the databases. I noticed that if I accessed a page on the server that connects to a database and then several hours later try to access the page again, I received a Server Error on the page.

Additionally, checking the paster.log file shows:

Error - <class 'sqlalchemy.exc.OperationalError'>: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT builders.id, builders.name \nFROM builders \n LIMIT 50' ()

This error is a result of the database connections being specified to stay open forever and the server closing the connection after a period of time. There is an option called pool_recycle that should be set for MySQL connections, as noted in The Definitive Guide to Pylons book:

pool_recycle:
The length of time to keep connections open before recycling them. If not specified, the connections will stay open forever. This should be specified for MySQL in particular because servers typically close connections after eight hours, resulting in a “MySQL server has gone away” error.

Therefore, for example, in the project configuration I specified the pool_recyle as shown below:

sqlalchemy.your_db1.url = mysql://user:pass@localhost/your_db1

sqlalchemy.your_db2.url = mysql://user:pass@localhost/your_db2

sqlalchemy.pool_recycle = 3600

Pylons/SQLAlchemy Simple Queries

2010-11-18T19:30:00.000-08:00

Working with Simple Queries

Recently I've been working using the MVC model (See 0.1 post here) and feel much more comfortable with it. I have to say, it's awesome! Anyway, I am still have some trouble with complex queries and to be able to complete my 0.3 objective I will have to get some help with it, but for now, I understand and can create simple queries just fine.

/// Behind the Scenes information not included - such as creating metadata for the tables or binding it to the engine (as can be seen here and here). In addition, the query.py model contains the queries and also imports data from other model files ///

Here is an example from one of the BuildAPI queries:

ss = meta.scheduler_db_meta.tables['sourcestamps']

q = select([ss.c.branch]).distinct()

q = q.where(not_(ss.c.branch.like("%unittest")))

results = q.execute() 

The database and table information is stored in the ss variable.
In a more readable format, it would be: table sourcestamps from scheduler_db database
The q variable is where the query begins. It is first assigned a select statement and reuses the ss variable.
The ss.c.branch simply means the branch field/column(.c) in the sourcestamps table.
The distinct is added at the end to pull out distinct values.
Then, the q variable is reused again and a where clause is added.

If this was to translated to a MySQL query it would look like this:

select distinct(branch) from sourcestamps where branch not like '%unittest';

Another simple example is:

rr = meta.status_db_meta.tables['builders']

q = select([rr.c.id, rr.c.name])

q = q.limit(50)

query_results = q.execute()

Here the query would be:

select id, name from builders limit 50;

To put this in context of the MVC model, the results from the query could be assigned to an array variable such as c.results_output that the 'View' template can use to display it. This will be shown in another blog post, perhaps in the 0.2 release.

Using Vim with Python

2010-11-18T18:20:00.000-08:00

Getting Vim to work with Python
I'm sure there are better editors out there but I've been using vim for creating/editing python scripts. Also I should mention that since I am accustomed to typing vi on the command line, I have vi aliased to vim (alias vi=vim).
However, using vim with the default configuration is not ideal for python, especially with its 8 character tab spaces! After doing some research, I came across a few changes you can make to vim's configuration to optimize it for use with python:

You will need to edit the /etc/vimrc file (on Fedora), and append the following four lines:

autocmd BufRead,BufNewFile *.py syntax on

autocmd BufRead,BufNewFile *.py set ai

autocmd BufRead *.py set smartindent cinwords=if,elif,else,for,while,with,try,except,finally,def,class

au FileType python setl autoindent tabstop=4 expandtab shiftwidth=4 softtabstop=4 

This will automatically enable syntax highlighting, and do automatic indentation for Python code (indicated by the "smartindent cinwords=..."). In addition, once a python file is loaded, tabs will be expanded to spaces and be 4 characters long instead of the 8 character default tab space.
That's it.

Credits to http://wiki.python.org/moin/Vim

Git 'er done

2010-11-15T19:11:00.000-08:00

[Apologies for the lame title? lol]

Git?
I’ve heard the word Git thrown around here and there, and in these past years the only thing I knew of Git was that it was some sort of repository. However, after further reading, it seemed very much like Subversion, which I’ve used for Papervision (Adobe Flash) projects. I guess it was time to get acquainted with Git.

Git is a version control system, like subversion, however each user has their own repository and these repositories can push changes to a central repository.

Getting Started - Testing
To start off testing git, first change to the directory that you want to work with (project directory).
$ cd /public/svn/

I added two files in here: file.txt and test.txt

The first thing that needs to be done is to initialize your repository.
$ git init

Output:

Initialized empty Git repository in /public/svn/.git/

Notice that the .git directory is hidden.

If you would like to name the repository or to add a description, there is a file appropriately named description in the .git/ directory that you can edit.

Adding Files
To add files to the repository, we would use the command:
$ git add .
This recursively adds all files in the current directory. Of course, we can specify only to add specific files or a group of files rather than all files by simply specifying the files as arguments after git add.

To commit importing the files and thereby adding them to the repository index, we use:
$ git commit
At which git will open the .git/COMMIT_EDITMSG file and prompt you to enter a “commit message” or a description of the changes or what has been done.
This can be done quicker by simply using the –m “description” argument:
$ git commit –m “Initial repository setup”

Output:

[master (root-commit) a980722] Initial repository setup
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 file.txt
create mode 100644 test.txt

Making Changes to Files
Now if we were to make any changes to our files:
1. We first have to use the $ git add command again as this will incrementally add these changes to the repository index
2. Then we use the $ git commit command once more to commit these changes.

To simplify this two step process, we can simply use the following:
$ git commit –a
Any files that are already in Git’s repository index that have been changed will be updated in the repository. This one command will stage and commit in one step.

However, it is important to note that these files have to be already known by git, that is, in its index or else it would not commit any changes. If the file is not already in the index, just use the $ git add command to add it.

Renaming and Removing Files
To remove or rename files it is just the same as doing it on the command line except we add git in front of the command, such as:
$ git mv file.txt file2.txt
$ git rm test.txt

Note: There is the git log command that’s awesome, it shows you a list of recent commits (and also their hashes).
Also use the git status command to check the status of the local repository (it will show you if files have been changed/committed). If you use the git status command and notice “nothing added to commit but untracked files present” that means you either need to tell git to ignore those files as you perhaps don’t want it to track them, or if you do, use git add to start tracking.

Branches
I noticed all of these changes were being done under the “master” branch. It is a good idea to create different branches for different parts or features of your project. This is done by:
$ git branch branchname
And you can create as many as you need. It is also very easy to switch between branches by using
$ git checkout branchname
Branches can be checked out individually or merged with the master branch or other branches.

There is a lot of information on Git available, such as on:
Git Documentation or the Git Community Book

Plans
So far I’m liking Git, and I want to use it for my BuildAPI project to track changes – maybe after release 0.2 I’ll try to set it up.

All I gotta say is I just need to Git ‘er done!

BuildAPI: Errors

2010-11-09T19:57:00.000-08:00

Problems Getting BuildAPI to Work on Seneca's Iraq Host

From my previous post, it is apparent that I had BuildAPI running on a virtual machine on my laptop. However, I tried running and installing it on one of our Seneca host systems, iraq.proximity.on.ca, but it only gave me errors and I could not figure out why (see errors below).

I followed the same steps as I did on my virtual machine, and checked online for similar issues/errors but was left with little. I even tried working with it in a python virtual environment and also set SELinux to permissive to test but it was the same issue. I ended up doing most of the work on my VM but I would still like to get it working on the Iraq machine.

Running in a Virtual Environment
The following error was encountered when running paster serve --reload --daemon config.ini in a virtual environment:

--From the paster.log file--
File "/usr/lib/python2.6/site-packages/paste/config.py", line 76, in _current_obj
"No configuration has been registered for this process "
AttributeError: No configuration has been registered for this process or thread
Removing PID file paster.pid

I checked the /var/log/messages file and it showed:

Nov 9 22:39:00 iraq python: abrt: detected unhandled Python exception in /home/asingh114/newbuild/venv/bin/paster
Nov 9 22:39:00 iraq abrtd: dumpsocket: New client connected
Nov 9 22:39:00 iraq abrtd: dumpsocket: Saved Python crash dump of pid 10494 to /var/spool/abrt/pyhook-1289360340-10494
Nov 9 22:39:00 iraq abrtd: dumpsocket: Socket client disconnected
Nov 9 22:39:00 iraq abrtd: Directory 'pyhook-1289360340-10494' creation detected
Nov 9 22:39:00 iraq abrtd: Executable '/home/asingh114/newbuild/venv/bin/paster' doesn't belong to any package
Nov 9 22:39:00 iraq abrtd: Corrupted or bad crash /var/spool/abrt/pyhook-1289360340-10494 (res:4), deleting
Nov 9 22:39:17 iraq python: abrt: detected unhandled Python exception in ../venv/bin/paster

If I run yum info python, it shows python 2.6.4 installed, but if I do python -V, it shows python 2.6.6 installed. In addition, python-setuptools are installed.

Running Outside Virtual Environment
I also tried running paster serve --reload --daemon config.ini outside of the virtual environment (after all dependencies and packages were installed) and I also get an error:

Traceback (most recent call last):
File "/usr/bin/paster", line 5, in <module>
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

And in the /var/log/messages file:

Nov 9 22:39:00 iraq abrtd: Executable '/home/asingh114/newbuild/venv/bin/paster' doesn't belong to any package
Nov 9 22:39:00 iraq abrtd: Corrupted or bad crash /var/spool/abrt/pyhook-1289360340-10494 (res:4), deleting
Nov 9 22:39:17 iraq python: abrt: detected unhandled Python exception in ../venv/bin/paster

I will check into it further, and in the meantime I will continue to work on BuildAPI on my virtual machine and get my scripts going.

I would remove and reinstall the packages but don't want to screw up anyone's work on the Iraq system.

Additionally, I don't want to spend all my time trying to get it to work on Iraq when I have it working on my virtual machine anyway but if anyone has any ideas or input, please let me know!

BuildAPI: 0.1

2010-11-05T20:55:00.000-07:00

Getting StartedTo run BuildAPI locally, there were a few things that needed to be done, as outlined in the ReleaseEngineering/BuildAPI wiki (< Many thanks for that!!).

These were the steps:

Install MySQL and import database snapshots
Install Python and Python SetupTools (Easy_install)
Install Pylons (or use a virtual environment)
Install Google Python Visualizations Library
Install MySQL-Python (MySQLdb for Python)
Configure BuildAPI

1. Installing MySQLInstalling MySQL is easy:

$ yum install mysql-server

Then create a new user and password for MySQL (optional), create the two databases (schedulerdb and statusdb) in MySQL and import the two snapshots into the appropriate databases. As noted in my previous post, I did run into problems getting the snapshots into MySQL, see here. However, that was resolved.

2. Python and Python SetupTools
There were a few version of Python that I could have installed, however I chose to install version 2.6.6 as I had noted that the newest version 3.x was (maybe) not compatible with some of the components such as python mysqldb. I had used yum install to install it but it installed version 2.4 through yum so I downloaded the source and compiled it.
The python-setuptools package is necessary as it contains easy_install, which definitely lives up to its name and makes our lives soooo much easier (it resolves dependencies for python packages, just like yum)!

3. Install Pylons
What do you know, yum install pylons could work as well but instead I did easy_install pylons.

4. Install Google Python Visualizations Library
The visualizations library is used for, you guessed it, creating graphical output such as graphs and charts for BuildAPI reports. After downloading the source code, I used python setup.py install to install it.

5. Install MySQL-Python (MySQLdb for Python)
This one took me a while to get installed and working. There were several dependencies that I needed to have installed before it could build, such as python-devel.
Smooth-sailing after that though.

6. BuildAPI
Once the BuildAPI source has been downloaded (buildapi source), I had to extract it to a directory called buildapi/ (by default it did not do this). Then run easy_install buildapi - it seems that easy_install goes into the folder and runs the python setup.py.

Once that is done, a default config.ini file was created (using paster make-config buildapi config.ini while in the buildapi directory) which would serve as the runtime configuration file. Changes needed to be made in here to allow BuildAPI to be accessible on the localhost and to be able to access the MySQL databases (SQLAlchemy is used with MySQL).

Thus, the following changes were made to the default config.ini file that was created using paster:

[server:main]

use = egg:Paste#http

host = 127.0.0.1

port = 5000

# SQLAlchemy database URL

sqlalchemy.scheduler_db.url = mysql://root:root@localhost/schedulerdb

sqlalchemy.status_db.url = mysql://root:root@localhost/statusdb

sqlalchemy.pool_recycle = 3600

That's it, that sets up BuildAPI to be accessible on http://127.0.0.1:5000/. Accessing it on the browser produces the following:

Each of the links work, as the database is set up as well. Here are a few example outputs:

MVC: Model–View–Controller

This is the concept that I need to understand in order to create the necessary code for the project. I noticed it is very similar to Perl's Templating Toolkit, where it allows you to separate the logic and view.

From what I understand so far, the controller contains the logic, the model has the data and the view is basically the template that the end user will read the data from, as on a webpage for example.

Controller
When I am creating the scripts, I understand that the controller will call on the models or the views and return data to the model (please correct if this is wrong). It was also possible to use just a controller to show data on a webpage, however, it was not formatted. Using the command paster controller name created a default template controller that I could play around with.

Model

The model is where I can put the database connection and queries and store it into values that the template would then also use. I did read that for passing arguments from the model to the template, BuildAPI can use a function called "tmpl_context as c". Variables are called using the ${c.name} and this allows us to use the objects assigned as attributes of c anywhere else in the application - therefore, like global variables.

View
The model passes the data to the view/template which will display that information on a webpage with graphs and charts. These templates use a library set called Mako, which again, is similar to Perl's Templating Toolkit. It has many functions, expressions and tags that can be used. The template files are saved as name.mako

Here is a diagram of it (from wikipedia):

Well, this is how I understand the code will be created using MVC and its quite a lot to learn!

Testing

One of the difficulties is knowing how all the components fit together, but after playing around with BuildAPI and taking a look at the structure of the pre-existing controllers, mako templates and model files, I have a fairly better understanding to progress into milestone 0.2.

I managed to create a few controllers myself, along with models and views using documentation on the internet (such as on PylonsHQ)

However, they were pretty simple and I need to learn more about the syntax and formats.
Example:

As an example, I was able to pull data (only used one column and a simple select query) and print it out on the screen using just a controller. I ran the command paster controller datatest, which created the template controller file and I modified that.
Note: To print to console, I had to edit the config.ini file and set

sqlalchemy.echo = True.

Simple Controller: datatest.py

from sqlalchemy import *

engine = create_engine('mysql://root:root@localhost/schedulerdb charset=utf8&use_unicode=0')
connection = engine.connect()

result = engine.execute("select name from schedulers")
for row in result:
        print "name:", row['name']
result.close()

This printed out all the names from the schedulers table from the schedulerdb database:

I believe I can get this to display on a webpage, but I need a little help. I can use a controller file, datatest.py, such as this:

import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.getmydata import functionnname

log = logging.getLogger(__name__)

class DatatestController(BaseController):

    def index(self):
        # Return a rendered template
        c.message = GetMessage()
        return render('/displaydata.mako')

The return render will call on the displaydata.mako View/Template.
In the line

"from buildapi.model.getmydata import functionnname"

It will call the model getmydata.py (indicated by buildapi.model.getmydata - i.e. directory/directory/file) and in this file is where I could define the functionname.
For example, if the function name is GetMessage, then it would be defined as:

def GetMessage():
    return "Gimme my data fool!"

In the template, I simply use ${c.message}to get the data. If it is defined as above, then in the webpage (view) that is displayed, it will replace ${c.message} with "Gimme my data fool!".
Note: as shown in the controller file, datatest.py, c.message was defined as:

c.message = GetMessage()

Expectations

I would like to be able to print the data out to a webpage but will need to understand MVC and pylons syntax more. I will use the resources as listed on the BuildAPI wiki to help me with this
Once that is done, then I will try to put that data into a graph or chart
Next would be to pull the correct data and generate graphs based on the project objectives
And finally, get it all documented!

Quick Note about Python Virtual Environment

A python virtual environment was highly recommended by Release Engineering, and therefore I also set up one for testing BuildAPI code before I brought it over to my actual installation on Fedora. It was extremely easy to set up, and literally took a few minutes to get it running.
Using easy_install, you can simply use easy_install virtualenv and it will install the virtualenv script to your path. Then you just need to create a directory, run the script wit the directory as your argument and it will set up a sandbox environment. Inside the sandbox/bin folder is an activate script, which as the name implies, activates the virtual environment.

Here are the commands if you are interested:
$ easy_install virtualenv

$ mkdir ~/venv

$ virtualenv ~/venv/sandbox

$ source ~/venv/sandbox/bin/activate

(sandbox)$ easy_install [packagename]

Once you're in the sandbox, you can install whatever else you need to install

I kept detailed instructions on how to do each step, and I will leave that for the final documentation.

BuildAPI: Pre 0.1 - free disk space problem

2010-11-05T19:20:00.000-07:00

MySQL Database in my space!

For BuildAPI, I needed to install MySQL and import snapshots of two databases that I would be using. So it was installed on my Fedora virtual machine (with a 25GB raw disk (yes, it should have been LVM))... and that eventually ran out of space. The database snapshots 'statusdb' and 'schedulerdb' were almost 10GB of space combined - something I did not expect, and did not account for.

I was close to cloning my virtual machine into another with a larger disk but after consulting with Chris Tyler, there were much simpler options - such as simply adding another raw disk, and this is what I did.

Moreover, I edited my MySQL configuration file (/etc/my.cnf) and pointed the data directory to be on the new virtual raw disk and worked on it from there. Although my VM was terribly slow at importing the (very large) sql dumpfile, and even failed once, it worked out worked out fine in the end.

Lesson: use LVM, or better yet, do this all on an actual host and not a virtual machine

BuildAPI: Milestone 0.1 Release (To be blogged)

2010-11-05T05:28:00.000-07:00

Three cups of coffee later...
Instead of posting everything for my BuildAPI 0.1 release now, I am going to get some sleep and post it later on today. However, the BuildAPI Project Page has been updated with what has been done for 0.1 so far and what needs to be done between now and milestone 0.2.

Here's a quick summary of 0.1:

Completed a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
Set up BuildAPI and ran the Paste server to use the pre-existing BuildAPI controllers
Created several test controllers, models and templates for testing and gaining a practical understanding how Pylons work
Created my own script using SQLAlchemy and MySQL engine to pull information from the SchedulerDB and StatusDB database.
However, it displays only text (one line - refreshing page shows next line etc) and I need it to loop the values on the page to display all (should not be hard at all)
Before milestone 0.2, I would like to be able to generate a simple graph from the data I pulled so far.

--and would you believe it, I actually had fun working with BuildAPI all night...until the caffeine was gone and dawn broke.

Oh, and save the procrastination lecture for later. I have just been extremely busy (if you want to know - drop me a comment and I'll tell ya) and did not find enough time to work on the project. However, this is what all-nighters are for, because I'm a crazy workhorse!!

zzzzzzzzzzz......

BuildAPI: A First Look

2010-10-26T12:11:00.000-07:00

The Beginning
Previously we had a conference call with Armen Zambrano from Mozilla regarding Mozharness, BuildAPI and Simple Release Bugs (Thanks Armen!). He had introduced us to these topics and got us rolling on projects that we could tackle. Consequently, I took an interest in BuildAPI which is a Pylons project used by RelEng (Release Engineering) to surface information collected from two databases updated through Buildbot (http://buildbot.net/trac) masters as they run jobs.

The BuildAPI project wiki page can be found here.

A Little About BuildAPI
This project involves generating analytic reports by querying databases, which can be used for a multitude of purposes such as performance or usage reports or discovering rogue Buildbot slave machines. Furthermore, it uses Pylons which combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.

Getting Started
However, there is a great deal of information that needs to be learned and it would definitely require me to take a strong iniative. The first step is figuring out where exactly to start and how to progress.
Fortunately, Armen had posted more information about BuildAPI on his blog (http://armenzg.blogspot.com/search/label/mozharness), which provided an objective:

What I need students to do is one of the two:
1) generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts like this
a) this is very useful and could move us forward towards having this information being published publicly for consumption
b) I highly encourage this one as understanding the mental model behind it is easier
2) write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)

He also provided us with snapshots from the database and extremely helpful links such as How to get started with BuildAPI and Google Chart API (more on that later).

Challenges (I love 'em, bring 'em on!):

Need to wrap my head around BuildAPI's concepts
Absolutely no experience with Python (although it is similar to BASH, which I am fine with) and just barely touched Ruby
Need to learn how to integrate Google Chart API / how it works
Figure out the database structure and which information needs to be pulled to generate the required reports
How to integrate it all and create a tool that can accomplish the latter

What I have done so far:
- I have looked at the links Armen had posted, especially the ReleaseEngineering/BuildAPI as it provides instructions on how to get started with BuildAPI.
- Went over the database and its structure, which can be seen here and here or below:

and

- Started reading on Pylons framework from the following sources:
http://pylonshq.com/docs/en/0.9.7/gettingstarted/
http://pylonshq.com/docs/en/0.9.7/tutorials/
- Also, briefly went over Google Chart API (It's awesome!)

A Note about Google Chart API:
The Google Chart API allows you to dynamically generate charts using a URL string.
It is the perfect tool as the charts can be embedded on web pages which the advantage is there are no files to save or serve. Although, the API does provide the option to download the image for local or offline use.

Here is an example:
http://chart.apis.google.com/chart?cht=p3&chd=t:60,40&chs=250x100&chl=Work|Play

Isn't that awesome?

Signing RPM Packages and Creating Your Own Repository

2010-10-25T21:02:00.000-07:00

So much for "break" week. With so much to do, I'll be spending most of it just doing work.
Anyway, aside from that, let's look at another lab which consists of two phases:
- signing RPM packages, and then
- creating our own yum repository from where we will serve our signed packages.

First Phase - Signing Packages

The first phase involved signing rpm packages and I had chosen to sign two rpms that I created using spec files: nled and snort.
This was an easy task which is accomplished by:
- generating a GPG key (using gpg --gen-key),
- editing the .rpmmacros file and adding my email address (adding %_gpg_name "asingh114@learn.senecac.on.ca"), and finally,
- signing the desired rpm packages using rpm --addsign packagefilename

Instructions can be found on our SBR600 Weekly Schedule wiki under "Week 5 (October 4) - Repositories/Distributing)"

Second Phase - Creating a Yum Repository

Again, instructions can be found on our SBR600 Weekly Schedule wiki.

Since I am running my host system (Fedora 12) in a virtual machine, I decided to create a local or internal yum repository and test it using another fedora virtual machine that I already had installed. The test virtual machine is running Fedora 13 64-bit edition. Furthermore, I will be using HTTP as the protocol to serve my repository directories and of course, Apache Web Server was already installed and running on the system.

The repository directories were created in the public HTTP directory, and will be served out of /var/www/html/fedora/12/. However, in order to organize content I created two additional directories:
- i386, for 32 bit Fedora editions, and
- x86_64 for 64 bit.

The following command was used:

mkdir -p /var/www/html/fedora/{i386,x86_64}

Then, I copied my signed repositories from Phase 1 over to their respective directory (either i386 or x86_64). Next is creating the repository metadata for both of the directories, which can be accomplished through the command:

createrepo /var/www/html/fedora/i386
and then do the same for x86_64 directory.

Or through a script such as this:

#!/bin/bash

destdir="/var/www/html/repo/fedora"

for repo in i386 x86_64

do

    pushd ${destdir}/${repo}

    createrepo .

done

Modified from http://blogs.techrepublic.com.com/opensource/?p=609

Either way, this will create a repodata directory containing the repository metadata in both of those directories.

Testing the Repository

We will be using a GPG key for our repository, so I also had to create a GPG key file. This is simply done by using the command:

gpg --export --armour asingh114@learn.senecac.on.ca > RPM-GPG-KEY-asingh114

This created the gpg key output which is saved into the RPM-GPG-KEY-asingh114 file. This file is placed into the /etc/pki/rpm-gpg/ directory and will be used by the repository.

As mentioned before, this repository will be tested by serving to internal clients. Therefore, I created a new repository file in the /etc/yum.repos.d directory called asingh114.repo, which contained the following:

[asingh114 repo]

name=Asingh114 Repository

failovermethod=priority

baseurl=http://localhost/repo/fedora/$basearch

enabled=1

metadata_expire=7d

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-asingh114

Notes:
- The gpgkey is pointing to the file I generated earlier.
- Notice the baseurl is pointing to my localhost. The yum built-in $basearch variable denotes the architecture such as i386 or x86_64, etc, hence using $basearch at the end allows it to be able to find the proper repository directories for the host depending on the host's architecure.
- Additionally, I could have used another yum variable, $releasever, which is the release version of the system (e.g. Fedora 12, 13 etc) to further organize content, but this is just a simple repository test and I figured it is not needed.

When I tested this on my host machine, it worked fine.

Moreover, these two files were packaged into the RPM below that was created using a SPEC file:
http://asdesigned.ca/asingh114-repo-1-1.fc12.i686.rpm

I wanted to try this on the test virtual machine (Fedora 13 64 bit) so I temporarily moved all other repository files from the test machines' /etc/yum.repos.d directory and installed the rpm I created. This placed my repository file and my GPG key file into the proper locations.
Then I ran yum to test it, which I was then prompted to import the keys - after which, it was able to pull from my own repository. Cool stuff!

By the way, we can also use rpm --import GRPM-GPG-KEY-asingh114 to import the GPG key files manually.

To summarize, the setup was as follows:
Host machine: Fedora 12 (32 bit)
- Repository Serving Protocol: HTTP, Web Server
- Repository Directories: /var/www/html/fedora/i386 and /var/www/html/fedora/x86_64

Test Virtual machine: Fedora 13 64 bit
- Moved all other repository files out of the /etc/yum.repos.d directory temporarily
- Repository configuration file installed: /etc/yum.repos.d/asingh114.repo via my RPM package
- GPG Key file installed: /etc/pki/rpm-gpg/RPM-GPG-KEY-asingh114 via my RPM package
- Ran yum

Mock and Koji testing with NLED (simple)

2010-10-11T15:08:00.000-07:00

Mock:
After recreating my own NLED spec file, I test it with Mock using:

mock -r fedora-12-i386 --rebuild nled-252-6.fc12.src.rpm

The results were successful. I checked the log files in /var/lib/mock/fedora-12-i386/results/, mainly the root.log file and went through it. There were no errors and I did see the ncurses-devel dependency that I had originally placed into my spec file was tested. NLED was really simple so I didn't expect any errors anyway.
I expect to run into issues when I try my own SNORT and Irrsi spec file (when I get around to finishing them).

Koji:
Next, I installed Koji - the instructions are on http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji
The basic steps are running yum install fedora-packager and then running the fedora-packager-setup, at which point you will be prompted to enter your FAS username and password.

In LAB 1 (Communication Lab), we created our FAS accounts, so I used those credentials successfully. It then creates the necessary certificates (see the fedoraproject.org link above for more information) and allows you to access the Koji Fedora Build system where you can see your builds and tests.

Here is a link to mine: Koji Fedora Builds
If you click on my "Tasks", you will see that I have tested NLED on Fedora 12 platforms and all were successful.

Below is the command and ouput for Koji on my Fedora system:
# koji build dist-f12 --scratch nled-2.52-6.fc12.src.rpm
2528731 build (dist-f12, nled-2.52-6.fc12.src.rpm) completed successfully

I have to say that Koji is excellent and was very easy to set up! It's good to have such a resource available.

Well, I am going to test SNORT and Irssi source RPMs here later and will post results.

P.S. Happy Thanksgiving! Mmm Turkey....and if you don't get any work done, blame it on the Tryptophan!!!

NLED Spec file revisited

2010-10-11T14:49:00.000-07:00

Revisited:
In one of my previous posts, Creating an NLED Spec file, I had posted my spec file and RPM. However, I might not have had ran rpmlint on it so therefore there were some Warnings (no errors) that needed to be fixed.

Rpmlint Warnings:

When I ran rpmlint on the spec file, I received warnings that I had macros in the %changelog section, which I did have.

Warning messages:
"W: macro-in-%changelog %files"
"W: macro-in-%changelog %Install"

That was an easy fix, and I learned that no macro should be placed in the %changelog section at all (I initially thought it did not matter).

Running rpmlint on the nled-2.52-6.fc12.i686.rpm file, gave me two warnings:

nled.i686: W: no-documentation
nled.i686: W: no-manual-page-for-binary nled

After checking the SOURCE/nled-2.52/ directory under ~/rpmbuild, there is a nled.txt file there. So it was simple as adding %doc nled.txt to the %files section. Now the original source rpm does not have a manual page for the NLED binary so that is a warning that we can safely ignore (am I assuming right?)

Update:
I decided, for testing purposes, to add a man page to the NLED rpm. I used the output from the command "?" within NLED that gives the list of hotkeys and shortcuts to create a man page file called nled.1.
(To create your own man page, see here).

Then I edited the nled.spec file and added to the sections below.
Please note that comments were added in for explanations.

# I used Source1 instead of using PATCH 

# because it is just a straight file 

# that doesn't need to be uncompressed 

Source1:  nled.1       #This was put in the SOURCES directory

%prep
%setup -q

 #This copies the nled.1 to the BUILD directory
cp -p %SOURCE1 %{_builddir}/%{name}-%{version}/       

%install

rm -rf %{buildroot}
mkdir -p %{buildroot}%{_bindir}/
cp -p  nled %{buildroot}%{_bindir}/

# This creates the man directory in the BUILDROOT 

mkdir -p %{buildroot}%{_mandir}/             

# and this copies the nled.1 from the BUILD to the BUILDROOT man directory
cp -p nled.1 %{buildroot}%{_mandir}/         

%files
%defattr(-,root,root,-)
%doc nled.txt  #This adds the nled documentation
%{_bindir}/*

# This tells rpm that the nled.1 man file is part of the package
%{_mandir}/*

I will have to reupload the SPEC file and the RPM as they have changed, and the previous blog entry's versions were not tested.

Revisited: Testing RPM build times - Part 2

2010-10-04T22:14:00.000-07:00

In one of my previous blog posts, Revisited: Testing RPM build times - Part 1, I had indicated that I would be testing the build time of SNORT using an automated script. Well that didn't turn out too well. I had left it running overnight on my laptop and when I checked in the morning, my VirtualBox was not responding - it crashed. Of course, I lost all the data.

Anyway, I redid this test using aircrack-ng which is much faster to build because I honestly do not have much time to try SNORT again and wait with 7 courses + work + web design work. Besides, I need to use my laptop tonight and don't want to skewer the build time results.

Summary of my results:

The first, and very first time I ran rpmbuild -ba aircrack-ng, it took approximately 1 minute and 31 seconds. I ran it again after it had "warmed up" and I was surprised to see that it now took 43 seconds to complete.

From the results, using a -j1 value proved to be the fastest which correlates with the fact that my virtual machine is only using 1 CPU.

The Results:

-j1

real      0m43.798s

user     0m9.046s

sys       0m27.381s

-j2

real      0m45.369s

user     0m9.536s

sys       0m28.681s

-j3

real      0m46.145s

user     0m9.595s

sys       0m29.114s

-j4

real      0m46.829s

user     0m9.774s

sys       0m29.842

Quick Notes
Quote on time output: real, user, sys from a site on the internet:
Real refers to actual elapsed time; User and Sys refer to CPU time used only by the process.
Real is wall clock time - time from start to finish of the call. This is all elapsed time including time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete).
 User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU time used in executing the process. Other processes and time the process spends blocked do not count towards this figure.
 Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like 'user', this is only CPU time used by the process. See below for a brief description of kernel mode (also known as 'supervisor' mode) and the system call mechanism.

Success! Creating my own spec file for NLED

2010-09-30T23:53:00.000-07:00

Success!

I finally got around to finishing my own spec file for NLED. I can tell you this - I learned quite a lot going through each one of the errors and trying to research and fix it. I'll admit it was quite frustrating at first. Getting it to finally work was good, but the best part was gaining a MUCH better understanding of how the .spec file works.

The Fedora How to pointed me in all the right directions, and of course, can't forget man's new best friend - Google. I just needed to spend some time going through the sections and understand what each is supposed to do.

I will attach my RPM file here as well as the SPEC file for those of you who are interested. Please see below:

Spec file: http://asdesigned.ca/nled.spec

Please Right-click and use "Save as" to download the nled.spec file or just simply click on it to view it in the browser

RPM File: http://asdesigned.ca/nled-2.52-8.fc12.i686.rpm

A Few Notes

NLED does not require any ./configure or make install commands, instead it is simply compiled using the make command, where it will create the nled binary that can be placed into /usr/bin/ and executed anywhere on the command line.
Therefore, you will have to remove the ./configure option from the %build section of the spec file and also remove the make install option from the %install section
I took a look at the original NLED Makefile and at the top it showed that the make options were CCOPTIONS=–O –c and LDOPTIONS=–lcurse .

# AIX: CCOPTIONS=-qnoro -O -c
# some Linuxes: CCOPTIONS=-O -c -I /usr/include/ncurses
# other Linuxes (try this first):
CCOPTIONS=-O -c

# AIX: LDOPTIONS=-lcurses
# some Linuxes: LDOPTIONS=-lncurses
# other Linuxes:
LDOPTIONS=-lcurses

So I added to the %build section:
make %{?_smp_mflags} CCOPTIONS="-O -c" LDOPTIONS="-lcurses"

Next, the installation needs to be done "by hand" and the files need to be copied to the correct locations. Therefore, we need to create the directories (using mkdir -p) and then copy the files from inside the build directory to the buildroot directory (using cp -p). So following the Fedora How To resource, I added:

%install
rm -rf %{buildroot}
mkdir -p %{buildroot}%{_bindir}/
cp -p nled %{buildroot}%{_bindir}/

After doing that, I got an error when running rpmbuild:

error: Installed (but unpackaged) file(s) found:
/usr/bin/nled

This just means that I did not count those files under the %files section, as the %files section identifies what files and directories were added by the package - and thus, which files and directories are owned by the package.

To fix it, I just needed to add %{_bindir}/* under the %files section, and the rest is history.

Revisited: Testing RPM build times - Part 1

2010-09-29T22:30:00.000-07:00

In my last post, Testing RPM Build times, I had used different -j values in the ~/.rpmmacros file to find the best value that resulted in the fastest build time for NLED, which is a very small package and the build time was relatively fast at around 6 seconds. However, after reading Chris Tyler's email today and chatting with him on IRC (irc.freenode.net, #seneca channel), it came to my attention that

"most of that time is spent in unpacking the tarball and packing the RPMs, which are unaffected by the -j value"

In addition, Chris informed us of a build log file that contained a list of packages and build times that we could use to find a solid package to rebuild and test the -j value properly. This file can be found at http://scotland.proximity.on.ca/sbr600/buildall.log

Therefore, I am rebuilding SNORT, which I had compiled from source here, to finally get the proper results of the -j values.

Oh and lucky me! According to the build log file, SNORT typically takes approximately 433 seconds to build!

Regardless, I am using a script to let it run overnight and I will post the results tomorrow evening.

Testing RPM build times

2010-09-24T20:57:00.000-07:00

The rpmmacros file
This week we were experimenting with the rpmbuild macros configuration file: ~/.rpmmacros.
The –j value of the smp_mflags in the ~/.rpmmacros file controls how many simultaneous jobs make starts and thus we can change it to see which value would give us the fastest build time.

The Setup
For the first test, I used my laptop, which is running Fedora12 in an Oracle VM VirtualBox virtual machine with 1GB of RAM allocated.
My laptop specifications are shown below:
CPU: Intel Core 2 Duo Mobile T5800 @ 2.00GHz
Ram: 4.00 GB

The package I chose to build was NLED.

The Test
The following command was used to gather the results:
time rpmbuild -ba nled.spec

The Results

Values: -j1, -j3 (default), -j4, -j5

This resulted in an elapsed time of 15.8 seconds. Keeping in mind that this test is on Fedora running in a virtual machine, I ran it again. This time the result was16.9 seconds which I attributed to factors such as the CPU being used for other processes on the host machine. Values 1, 4 and yielded similar results of times between 15.8 - 16.9 seconds and value 5 yielded close to 18 seconds.

Best Value: -j2

However, using a value of j2 yielded an elapsed build time of 14.3 seconds, so it seems that this is the optimal value to use, at least in a virtual machine.

I re-did the test a second time after restarting the machine, and all the times dropped about 40% for each of the respective values!

Unsatisfied

I couldn't just have my cake and eat it too. I wanted seconds. So I pulled out the big guns and used my desktop with Fedora 13 installed.

The Setup Revisited

CPU: Intel i7 920 Overclocked

Ram: 6GB

The Results Revisited

Blazing.

No, really. The best value was a -j4 which yielded 3 seconds to build NLED.

I would definitely say that the benefits of changing the -j values of the smp_mflags would be more noticeable when compiling larger builds, but it was still interesting to test out the build times.

A Prequel: Rebuilding an Existing Fedora SRPM

2010-09-24T19:50:00.000-07:00

Rebuilding an Existing Fedora SRPM:
I chose to rebuild the NLED package on my machine which was quite simple.

Steps:

Install the necessary packages: rpm-build, rpmdevtools, rpmlint, yum-utils
Setup your system, see here
Download the source file wget http://cdot.senecac.on.ca/software/nled/nled_2_52_src.tgz
Install it, rpm -ivh nled_2_52_src.tgz
Go in the: cd ~/rpmbuild/SPECS directory
Run the command rpmbuild -ba packagename.spec

The .spec file was fairly straightforward. This took about 15 seconds to complete (also see Testing Build Times), and was done without breaking a sweat - unlike the next section below.

Creating my own .spec file
In the second week of the course, I had built Snort from source using ./configure, make and make install, which can be found here.

A few nights ago, I tried building my own spec file for Snort but ran into a few problems. Thus I took a look at the original Snort spec file after installing the source (To see how, click here), and it was filled with information that I am unfamiliar with (for now – can’t emphasize that enough – for now). However, I did take some time to go over it to try to understand out what each line was doing.

Next, I figured I can write my own (much) simpler snort spec file and get it to build into an rpm package – but I’ll do that at a later time. Since this was my first attempt at creating my own spec file, I wanted to try something much easier at first and get practice, understand it more.

I will post my experience of creating my rpm build via a spec file in the coming days along with a link to download the respective spec file, source and package for your perusal or curiosity.

Heck, I’m still going through SourceForge looking at all the goodies available.

Compiling NLED and Snort from source

2010-09-12T22:44:00.000-07:00

The instructions were to build two software packages from their source files, of which one must be NLED and any other software of your choice.

First let’s begin wiith NLED:

Compiling NLED from Source

Download and extract the Tarball:

$ wget http://cdot.senecac.on.ca/software/nled/nled_2_52_src.tgz

$ tar -xvz nled_2_52_src.tgz

Next, run make and make install to compile and install:

$ make

$ make install

Incidentally, this installation ran without issues and I am successfully using NLED.

For the second software, I decided to install Snort.

"Snort® is an open source network intrusion prevention and detection system (IDS/IPS) developed by Sourcefire. Combining the benefits of signature, protocol and anomaly-based inspection, Snort is the most widely deployed IDS/IPS technology worldwide."

Compiling Snort from Source

You can download the Source snort package from:

http://www.snort.org/snort-downloads

Untar the Source file:

$ tar -xvf snort-2.8.6.1.tar.gz
$ cd snort-2.8.6.1/

The Snort website (www.snort.org) showed that before installing Snort you need have a number of software packages installed such as Libpcap, PCRE, Libnet and Barnyard.

I ran the following commands to see if these packages are installed:

rpm -q libpcap             - Yes, installed
rpm -q pcre                      - Yes, installed
rpm -q libnet                 - No, not installed (or at least rpm did not find it)
rpm -q barnyard            - No, not installed (or at least rpm did not find it)

However, I decided to run the configure file to see what would happen.

$ ./configure

But alas, I got the following error:

Snort Error:

ERROR! Libpcap library/headers (libpcap.a (or .so)/pcap.h)not found, go get it from http://www.tcpdump.org or use the --with-libpcap-* options, if you have it installed in unusual place. Also check if your libpcap depends on another shared library that may be installed in an unusual place

But wait, the latest lipbcap package WAS installed according to rpm (as I had run rpm –q libpcap above).

Perhaps, I was missing something else. I checked the www.tcpdump.org website but it had the same libcap I had installed. I ran the command:

$ yum search libpcap

Loaded plugins: presto, refresh-packagekit
=================Matched: libpcap ================= libpcap.i686 : A system-independent interface for user-level packet capture
libpcap-devel.i686 : Libraries and header files for the libpcap library
…

Since libpcap was installed, I figured I needed the development libpcap library as well so I installed it using:

$ yum install libpcap-devel

Then I ran ./configure again, as it was going through I noticed:

./configure: line 13616: pcre-config: command not found
./configure: line 13622: pcre-config: command not found
checking pcre.h usability... no
checking pcre.h presence... no
checking for pcre.h... no

I knew what was coming and then BAM! Another error:

ERROR! Libpcre header not found.
Get it from http://www.pcre.org

So I checked www.pcre.org and found the same pcre library I have installed already. Now I know there must be a pcre development library like libpcap so I ran yum search pcre and it came back with:

Loaded plugins: presto, refresh-packagekit
==========Matched: pcre ============
…

pcre.i686 : Perl-compatible regular expression library
pcre-devel.i686 : Development files for pcre

Of course, I then ran yum install prce-devel

Finally, I ran ./configure again, and voila! It completed (although I didn’t seem to have barnyard or libnet according to rpm –q)

If you need to run ./configure with special options, then please see this resource I found on the configuration options you can use:
http://204.152.191.100:8080/wiki/index.php/Configure_options_-_snort

Next I ran make as my current user, followed by make install as root

Snort is successfully installed. Time to start snortin’…or not (insert fail comment here).

SBR600 Communication Lab

2010-09-10T00:25:00.000-07:00

Hello,
My name is Andrew Singh. I am currently a 6th (and final) semester student at Seneca College in the Computer Systems Technology course. I plan to graduate with high honours so here's to the start of another to-be-great semester (and with that: raises a glass of beer)!

This blog was created for two purposes:

For my current Software Build and Release course SBR600, and
as a blog for my web and graphic designs.

A few thing about me:

I love programming (PERL, PHP and learning JavaScript and AJAX. I am willing to learn any programming language!)
My flavour of Linux would definitely be Fedora
I create web and graphic designs, currently for my (own) company ASDesigned (Site will be up shortly)

There's more but let's get started with SBR600. Our first assignment was to complete the Communication Lab, and without further ado:

Here is my Seneca Wiki: Seneca Wiki

And here you will find me on Fedora Wiki: Fedora Wiki

For IRC, I've decided to use the firefox plugin called ChatZilla (which can be found here: ChatZilla Plugin for Firefox). It is definitely a lightweight and easy-to-use extension with similar capabilities as usual IRC clients as noted from the official description:

ChatZilla provides all the usual IRC client features: multiple servers, a built-in list of standard networks, easy searching and sorting of available channels, logging, and DCC chat and file transfers, plus easy customization with JavaScript plug-ins and CSS styling.

You may find me on IRC under the freenode network (irc://irc.freenode.net) under the #seneca channel.

IRC username: asingh

Additionally, here is a screenshot of ChatZilla in the #seneca channel on the freenode network: