BuildAPI: 0.2

Quick note:

Last week I posted a series of blog posts that I did not have time to post during the past week or more. They were all sort of backed up and hence all posted at once.

BuildAPI 0.2

On to the BuildAPI project: Let me just say that I have been doing this project in a step-by-step approach in order to learn what I need to learn, understand what I am working with, and apply it practically.
Also, the BuildAPI wiki page has been updated: http://zenit.senecac.on.ca/wiki/index.php/BuildAPI

To see it running: http://iraq.proximity.on.ca:5000/
Please click on Project 0.2 under BuildAPI Project Test, or http://iraq.proximity.on.ca:5000/project

0.1x - Between 0.1 and 0.2

My 0.1 release included getting BuildAPI set up and taking a peek at the extensively used MVC model.
However, since BuildAPI uses Pylons framework, I took some time to read The Definitive Guide to Pylons book and skipped to the chapters that were necessary. It's a good thing the book is available online, it is such a great resource! It helped me to learn some of the language and concepts. Next was to create very simple controllers for working with model and view templates and that was the beginning of wrapping my head around the MVC model concepts.

0.2 - Summary

For 0.2, I feel very comfortable working with the MVC model and was able to create my own controller, add to the existing BuildAPI universal model script for querying (query.py, orginal seen here), and create my own MAKO template to output my results. (See MVC scripts). In addition, I appended a link to my project to the main BuildAPI index page, which I have set up on http://iraq.proximity.on.ca:5000.

In regards to the model file, I had to read some of the documentation on SQLAlchemy and how queries work in order to pull data from the database, and from that, I now understand how to formulate basic queries using SQLAlchemy. However, I am still having a bit of trouble with complex queries such as using joins and multiple where clauses (See blog post on SQLAlchemy and Basic Queries).

I also stripped special formatting from the output and template as I did not fully understand how it works and how it is integrated with Pylons and BuildAPI - it seemed confusing. By special formatting, I am referring to Javascript/JSON and Json tables and charts.
However, after a bit of fiddling and reading on google visualizations documentation, I get the gist of how it works and I have implemented simple json using google viz api. Although, I still need to seek out help and apply it practically - which will be in 0.3.

MVC – Revisited

From my 0.1 post, I'd like to revisit the MVC model in terms of the BuildAPI project instead of a generic breakdown. I will also reference the MVC scripts which can be seen later in the post.
Here goes

• The model (query.py) knows about the databases and queries, or in other terms, business objects/data
• The template (project.mako) knows about HTML, Javascript, CSS, and the tmpl_context or "c" variables (example: c_output in controller projecy.py script) that are passed to it by the controller.
• The controller knows about both the model and the template. Therefore, it will query or modify the model to obtain data, modifies the values of the "c" variables and then passes it to the template.
• Also, the model can invoke other models, and the controller other controllers etc

After working with it, I can definitely see how each component can be updated separately without affecting the other.

MVC Scripts

The Controller - project.py

import logging

from pylons import request, response, session, tmpl_context as c, url

from pylons.controllers.util import abort, redirect

from pylons.decorators import jsonify

from buildapi.lib.base import BaseController, render

from buildapi.model.query import GetProjectQuery

from buildapi.model.query import GetQueryTest

log = logging.getLogger(__name__)

class ProjectController(BaseController):

    def index(self):

        output = GetProjectQuery()

        results = GetQueryTest()

        c.output = output

        c.results = results

        return render("/project.mako")

        return self.jsonify(output)

        # What the lines below do is to gather the format of the request using request.GET method

        #  which contains the variables in a query string

        # These variables are set in the template through javascript or through the form request

        # This is commented out for reasons (See JSON/Javascript section)

        #if 'format' in request.GET:

        #    format = request.GET.getone('format')

        #else:

        #    format = 'html'

        #if format not in ('html', 'json'):

        #    abort(400, detail='Unsupported format: %s' % format)

        #if format == "html":

            # assign to c_output

            # return template render

        #else:

           # jsonify the data                   

           # return self.jsonify(results)

The Model - query.py (My test queries appended to existing file)

def GetProjectQuery():

    rr = meta.status_db_meta.tables['builders']

    q = select([rr.c.id, rr.c.name])

    q = q.limit(50)

    query_results = q.execute()

    output = []

    for r in query_results:

        this_result = {}

        for key, value in r.items():

            this_result[str(key)] = value

        output.append(this_result)

    return output

def GetQueryTest():

    rr = meta.status_db_meta.tables['builders']

    bb = meta.status_db_meta.tables['builds']

    q = select([rr.c.name, bb.c.starttime, bb.c.endtime])

    #q = q.join(rr, bb.c.builder_id = rr.c.id)

    q = q.where(and_(rr.c.id == bb.c.builder_id))

    q = q.where(and_(rr.c.name.like('%moz%')))

    q = q.limit(50)

    query_results = q.execute()

    results = []

    for r in query_results:

        this_result = {}

        for key, value in r.items():

            this_result[str(key)] = value

        results.append(this_result)

    return results

The Template - project.mako

Only the important parts shown. This creates the table using the results from the query that are passed in as variables.

This is for the first table:

<tr>

% for key in ('Builder ID','Builder Name'):

<td><h3>${key}</h3></td>

% endfor

</tr>

<tbody>

<%

  from pytz import timezone

  from datetime import datetime

  eastern = timezone('US/Eastern')

  now = datetime.now().replace(microsecond=0)

%>

% for key in c.output:

        <tr>

        % for x in ('id', 'name'):

        <td>${key[x]}</td>

        % endfor

</tr>

% endfor

</tbody>

This is for the second table:

% for key in ('Builder Name','Start Time', 'End Time', 'Duration'):

<td><h3>${key}</h3></td>

% endfor

</tr>

% for key in c.results:

        <%

        key['duration'] = key['endtime'] - key['starttime']

        %>

        <tr>

        % for x in ('name', 'starttime', 'endtime', 'duration'):

        <td> ${key[x]} </td>

        % endfor

        </tr>

% endfor

The below is simply using Google's visualization to create the table, it can also be used to create charts. I just don't know how to loop the data into a chart/table as yet properly.

<script type="text/javascript" src="http://www.google.com/jsapi"></script>

<script type="text/javascript">

        google.load('visualization', '1', {packages: ['table']});

</script>

% for key in c.output:

        rows: [{c:[{v: '${key['id']}'}, {v: '${key['name']}'}]}]};

% endfor

<script type="text/javascript">

    function drawVisualization() {

      // Create and populate the data table.

    var JSONObject = {

      cols: [{id: 'task', label: 'Builder ID', type: 'string'},

              {id: 'hours', label: 'Builder Name', type: 'string'}],

      rows: [{c:[{v: '${key['id']}'}, {v: '${key['name']}'}]}]};

      var data = new google.visualization.DataTable(JSONObject, 0.5);

      // Create and draw the visualization.

      visualization = new google.visualization.Table(document.getElementById('table'));

      visualization.draw(data, {'allowHtml': true});

    }

    google.setOnLoadCallback(drawVisualization);

</script>

After the above script, this is all that's needed in the HTML file that displays the table:

<div id="table"></div>

Javascript, JSON and Outputting data in special formats

That being said, JSON is just another data format structure, which is used for formatting the tables and charts. However, I am trying to get JSON and javascript to work so I can integrate with existing BuildAPI structure, but honestly, it is quite a lot to learn in a short time.

Whatever it is that I am doing or learning, I like to make sure I understand why I am doing it and how it works.

I've researched that the controller can produce the json on request using several methods, one of which is by passing the format specifier. In addition, this can be done either by passing it from a template using javascript to the controller or by using what is called "RESTful services". In Python documentation for Routes and RESTful services, it shows "Several routes are paired with an identical route containing the format variable. The intention is to allow users to obtain different formats by means of filename suffixes; e.g., "/messages/1.xml".

This ultimately means that you could put a test in the controller to see if it JSON or HTML data:

def view(self, id, format='html'):

     if format == 'json':

         return self.jsonify(data)

     else:

         return ('templates/project.mako')

Then, the output will depend on how the data is called, for example:
http://examplepath/controller/page.html - the format will be html thus displaying an HTML page
or
http://examplepath/controller/page.json - using json format and application/json content type

However, BuildAPI takes the approach of using javascript to set the format and furthermore, this javascript is inserted into another template file (call it B) that is invoked by the main template (call it A), so A invokes values from B. Maybe I am just confusing things and I have it all wrong? PLEASE correct me if I am wrong! (I've sent some emails out to request for help on the matter)

In BuildAPI, I understand that the controllers will check the 'format' of the request by using Pylon's "request.GET" and I understand that for the "Reports" this format (such as charts) is set in the respective mako template files. If it is an html request, it shows the HTML page, if it is a json request then it shows the json data in the form of a table or chart in the html page. However, some controllers (namely recent.py, running.py and pending.py) - where is the format set? I don't see it set anywhere in the controller, model or template file and it is not inheriting it from anywhere, yet the format seems to be set and it uses html + json to create those nice looking tables! How?

Moreover, I also received errors when trying to 'jsonify' output with more than one key/value pairs such as a pylons list that contains:
 {'name': 'mozilla-1.9.1-win32-unittest', 'starttime': '2009-09-27 14:06:30',  'endtime': '2009-09-27 14:27:50'}
It gives me the error:

Error - <type 'exceptions.TypeError'>: datetime.datetime(2009, 9, 27, 14, 27, 50) is not JSON serializable

Aaahh, the frustration!

Although, I've thought about a simpler solution:

1. Dump the data from the query into a python list
2. Then dump that data into json objects and pass it as a c_context variables to the MAKO template
3. n the template, iterate through the results to display it using google visualizations to create table or charts

However, I've read that this may not be as secure as it perhaps uses the javascript eval() function – but I'm not sure.

I will update this in the near future with a clear path to 0.3.

Update: I recently exchanged emails with Armen Zambrano regarding the objectives and will post another blog in the coming days of what I've decided to do.

Getting BuildAPI online + Previous Errors - Fixed

Preamble

I had tried to get BuildAPI set up on iraq.proximity.on.ca server before milestone 0.1 release however, I ran into problems that was posted on this linked blog post. I was receiving errors running the command paster serve --reload --daemon config.ini inside or outside of a python virtual environment.

I went back onto the iraq server to take a second look at it.

Fixing Errors

When I ran the paster command by itself it gave me an error. The paster.log provided a lot of information on what could be done to possibly fix it and mentioned to make sure that PasteScript was installed properly.
I ran the command:

easy_install PasteScript==1.7.3

And what do you know, that got the Paster error resolved. So easy eh?

However, I was still having errors with BuildAPI but that was quickly solved after running the following command  in the BuildAPI directory:

python setup.py develop

And that was it, just two commands to solve all my problems...I feel like kicking myself (lol).


Serving Content
I set Paster server to host on http://iraq.proximity.on.ca:5000 and credits to Chris Tyler for pointing out that I needed to add an IPtables firewall rule to allow incoming connections to port 5000.
This is so much better than working off of my laptop!

SQLAlchemy MySQL error

When setting up the Paster server for my BuildAPI project, I used Sqlalchemy and MySQL to connect to the databases. I noticed that if I accessed a page on the server that connects to a database and then several hours later try to access the page again, I received a Server Error on the page.

Additionally, checking the paster.log file shows:

Error - <class 'sqlalchemy.exc.OperationalError'>: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT builders.id, builders.name \nFROM builders \n LIMIT 50' ()

This error is a result of the database connections being specified to stay open forever and the server closing the connection after a period of time. There is an option called pool_recycle that should be set for MySQL connections, as noted in The Definitive Guide to Pylons book:

pool_recycle:

The length of time to keep connections open before recycling them. If not specified, the connections will stay open forever. This should be specified for MySQL in particular because servers typically close connections after eight hours, resulting in a “MySQL server has gone away” error.

Therefore, for example, in the project configuration I specified the pool_recyle as shown below:

sqlalchemy.your_db1.url = mysql://user:pass@localhost/your_db1

sqlalchemy.your_db2.url = mysql://user:pass@localhost/your_db2

sqlalchemy.pool_recycle = 3600

Pylons/SQLAlchemy Simple Queries

Working with Simple Queries

Recently I've been working using the MVC model (See 0.1 post here) and feel much more comfortable with it. I have to say, it's awesome! Anyway, I am still have some trouble with complex queries and to be able to complete my 0.3 objective I will have to get some help with it, but for now, I understand and can create simple queries just fine.

/// Behind the Scenes information not included - such as creating metadata for the tables or binding it to the engine (as can be seen here and here). In addition, the query.py model contains the queries and also imports data from other model files ///

Here is an example from one of the BuildAPI queries:

ss = meta.scheduler_db_meta.tables['sourcestamps']

q = select([ss.c.branch]).distinct()

q = q.where(not_(ss.c.branch.like("%unittest")))

results = q.execute()

• The database and table information is stored in the ss variable.
• In a more readable format, it would be: table sourcestamps from scheduler_db database
• The q variable is where the query begins. It is first assigned a select statement and reuses the ss variable. 
• The ss.c.branch simply means the branch field/column(.c) in the sourcestamps table. 
• The distinct is added at the end to pull out distinct values. 
• Then, the q variable is reused again and a where clause is added.

If this was to translated to a MySQL query it would look like this:

select distinct(branch) from sourcestamps where branch not like '%unittest';

---

Another simple example is:

rr = meta.status_db_meta.tables['builders']

q = select([rr.c.id, rr.c.name])

q = q.limit(50)

query_results = q.execute()

Here the query would be:

select id, name from builders limit 50;

To put this in context of the MVC model, the results from the query could be assigned to an array variable such as c.results_output that the 'View' template can use to display it. This will be shown in another blog post, perhaps in the 0.2 release.

Using Vim with Python

Getting Vim to work with Python
I'm sure there are better editors out there but I've been using vim for creating/editing python scripts. Also I should mention that since I am accustomed to typing vi on the command line, I have vi aliased to vim (alias vi=vim).
However, using vim with the default configuration is not ideal for python, especially with its 8 character tab spaces! After doing some research, I came across a few changes you can make to vim's configuration to optimize it for use with python:

You will need to edit the /etc/vimrc file (on Fedora), and append the following four lines:

autocmd BufRead,BufNewFile *.py syntax on

autocmd BufRead,BufNewFile *.py set ai

autocmd BufRead *.py set smartindent cinwords=if,elif,else,for,while,with,try,except,finally,def,class

au FileType python setl autoindent tabstop=4 expandtab shiftwidth=4 softtabstop=4

This will automatically enable syntax highlighting, and do automatic indentation for Python code (indicated by the "smartindent cinwords=..."). In addition, once a python file is loaded, tabs will be expanded to spaces and be 4 characters long instead of the 8 character default tab space.
That's it.

Credits to http://wiki.python.org/moin/Vim

Git 'er done

[Apologies for the lame title? lol]

Git?
I’ve heard the word Git thrown around here and there, and in these past years the only thing I knew of Git was that it was some sort of repository. However, after further reading, it seemed very much like Subversion, which I’ve used for Papervision (Adobe Flash) projects. I guess it was time to get acquainted with Git.

Git is a version control system, like subversion, however each user has their own repository and these repositories can push changes to a central repository.


Getting Started - Testing
To start off testing git, first change to the directory that you want to work with (project directory).
$ cd /public/svn/

I added two files in here: file.txt and test.txt

The first thing that needs to be done is to initialize your repository.
$ git init

Output:

Initialized empty Git repository in /public/svn/.git/

Notice that the .git directory is hidden.

If you would like to name the repository or to add a description, there is a file appropriately named description in the .git/ directory that you can edit.


Adding Files
To add files to the repository, we would use the command:
$ git add .
This recursively adds all files in the current directory. Of course, we can specify only to add specific files or a group of files rather than all files by simply specifying the files as arguments after git add.

To commit importing the files and thereby adding them to the repository index, we use:
$ git commit
At which git will open the .git/COMMIT_EDITMSG file and prompt you to enter a “commit message” or a description of the changes or what has been done.
This can be done quicker by simply using the –m “description” argument:
$ git commit –m “Initial repository setup”

Output:

 [master (root-commit) a980722] Initial repository setup
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 file.txt
create mode 100644 test.txt

Making Changes to Files
Now if we were to make any changes to our files:
1. We first have to use the $ git add command again as this will incrementally add these changes to the repository index
2. Then we use the $ git commit command once more to commit these changes.

To simplify this two step process, we can simply use the following:
$ git commit –a
Any files that are already in Git’s repository index that have been changed will be updated in the repository. This one command will stage and commit in one step.

However, it is important to note that these files have to be already known by git, that is, in its index or else it would not commit any changes. If the file is not already in the index, just use the $ git add command to add it.


Renaming and Removing Files
To remove or rename files it is just the same as doing it on the command line except we add git in front of the command, such as:
$ git mv file.txt file2.txt
$ git rm test.txt

Note: There is the git log command that’s awesome, it shows you a list of recent commits (and also their hashes).
Also use the git status command to check the status of the local repository (it will show you if files have been changed/committed). If you use the git status command and notice “nothing added to commit but untracked files present” that means you either need to tell git to ignore those files as you perhaps don’t want it to track them, or if you do, use git add to start tracking.


Branches
I noticed all of these changes were being done under the “master” branch. It is a good idea to create different branches for different parts or features of your project. This is done by:
$ git branch branchname
And you can create as many as you need. It is also very easy to switch between branches by using
$ git checkout branchname
Branches can be checked out individually or merged with the master branch or other branches.

There is a lot of information on Git available, such as on:
Git Documentation or the Git Community Book


Plans
So far I’m liking Git, and I want to use it for my BuildAPI project to track changes – maybe after release 0.2 I’ll try to set it up.

All I gotta say is I just need to Git ‘er done!

BuildAPI: Errors

Problems Getting BuildAPI to Work on Seneca's Iraq Host

From my previous post, it is apparent that I had BuildAPI running on a virtual machine on my laptop. However, I tried running and installing it on one of our Seneca host systems, iraq.proximity.on.ca,  but it only gave me errors and I could not figure out why (see errors below).

I followed the same steps as I did on my virtual machine, and checked online for similar issues/errors but was left with little. I even tried working with it in a python virtual environment and also set SELinux to permissive to test but it was the same issue. I ended up doing most of the work on my VM but I would still like to get it working on the Iraq machine.

Running in a Virtual Environment
The following error was encountered when running paster serve --reload --daemon config.ini in a virtual environment:

--From the paster.log file--

File "/usr/lib/python2.6/site-packages/paste/config.py", line 76, in _current_obj
    "No configuration has been registered for this process "
AttributeError: No configuration has been registered for this process or thread
Removing PID file paster.pid

I checked the /var/log/messages file and it showed:

Nov  9 22:39:00 iraq python: abrt: detected unhandled Python exception in /home/asingh114/newbuild/venv/bin/paster
Nov  9 22:39:00 iraq abrtd: dumpsocket: New client connected
Nov  9 22:39:00 iraq abrtd: dumpsocket: Saved Python crash dump of pid 10494 to /var/spool/abrt/pyhook-1289360340-10494
Nov  9 22:39:00 iraq abrtd: dumpsocket: Socket client disconnected
Nov  9 22:39:00 iraq abrtd: Directory 'pyhook-1289360340-10494' creation detected
Nov  9 22:39:00 iraq abrtd: Executable '/home/asingh114/newbuild/venv/bin/paster' doesn't belong to any package
Nov  9 22:39:00 iraq abrtd: Corrupted or bad crash /var/spool/abrt/pyhook-1289360340-10494 (res:4), deleting
Nov  9 22:39:17 iraq python: abrt: detected unhandled Python exception in ../venv/bin/paster

If I run yum info python, it shows python 2.6.4 installed, but if I do python -V, it shows python 2.6.6 installed. In addition, python-setuptools are installed.


Running Outside Virtual Environment
I also tried running paster serve --reload --daemon config.ini outside of the virtual environment (after all dependencies and packages were installed) and I also get an error:

Traceback (most recent call last):
  File "/usr/bin/paster", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

And in the /var/log/messages file:

Nov  9 22:39:00 iraq abrtd: Executable '/home/asingh114/newbuild/venv/bin/paster' doesn't belong to any package
Nov  9 22:39:00 iraq abrtd: Corrupted or bad crash /var/spool/abrt/pyhook-1289360340-10494 (res:4), deleting
Nov  9 22:39:17 iraq python: abrt: detected unhandled Python exception in ../venv/bin/paster

I will check into it further, and in the meantime I will continue to work on BuildAPI on my virtual machine and get my scripts going.

I would remove and reinstall the packages but don't want to screw up anyone's work on the Iraq system.

Additionally, I don't want to spend all my time trying to get it to work on Iraq when I have it working on my virtual machine anyway but if anyone has any ideas or input, please let me know!

BuildAPI: 0.1

Getting StartedTo run BuildAPI locally, there were a few things that needed to be done, as outlined in the ReleaseEngineering/BuildAPI wiki (< Many thanks for that!!).

These were the steps:

1. Install MySQL and import database snapshots
2. Install Python and Python SetupTools (Easy_install)
3. Install Pylons (or use a virtual environment)
4. Install Google Python Visualizations Library
5. Install MySQL-Python (MySQLdb for Python)
6. Configure BuildAPI

 1. Installing MySQLInstalling MySQL is easy:

$ yum install mysql-server

Then create a new user and password for MySQL (optional), create the two databases (schedulerdb and statusdb) in MySQL and import the two snapshots into the appropriate databases. As noted in my previous post, I did run into problems getting the snapshots into MySQL, see here. However, that was resolved.


2.  Python and Python SetupTools
There were a few version of Python that I could have installed, however I chose to install version 2.6.6 as I had noted that the newest version 3.x was (maybe) not compatible with some of the components such as python mysqldb. I had used yum install to install it but it installed version 2.4 through yum so I downloaded the source and compiled it.
The python-setuptools package is necessary as it contains easy_install, which definitely lives up to its name and makes our lives soooo much easier (it resolves dependencies for python packages, just like yum)!

3. Install Pylons
What do you know, yum install pylons could work as well but instead I did easy_install pylons.


4. Install Google Python Visualizations Library
The visualizations library is used for, you guessed it, creating  graphical output such as graphs and charts for BuildAPI reports. After downloading the source code, I used python setup.py install to install it.

5. Install MySQL-Python (MySQLdb for Python)
This one took me a while to get installed and working. There were several dependencies that I needed to have installed before it could build, such as python-devel.
Smooth-sailing after that though.


6. BuildAPI
Once the BuildAPI source has been downloaded (buildapi source), I had to extract it to a directory called buildapi/ (by default it did not do this). Then run easy_install buildapi - it seems that easy_install goes into the folder and runs the python setup.py.

Once that is done, a default config.ini file was created (using paster make-config buildapi config.ini while in the buildapi directory) which would serve as the runtime configuration file. Changes needed to be made in here to allow BuildAPI to be accessible on the localhost and to be able to access the MySQL databases (SQLAlchemy is used with MySQL).

Thus, the following changes were made to the default config.ini file that was created using paster:
 

[server:main]

use = egg:Paste#http

host = 127.0.0.1

port = 5000

# SQLAlchemy database URL

sqlalchemy.scheduler_db.url = mysql://root:root@localhost/schedulerdb

sqlalchemy.status_db.url = mysql://root:root@localhost/statusdb

sqlalchemy.pool_recycle = 3600

That's it, that sets up BuildAPI to be accessible on http://127.0.0.1:5000/. Accessing it on the browser produces the following:

Each of the links work, as the database is set up as well. Here are a few example outputs:

MVC: Model–View–Controller

This is the concept that I need to understand in order to create the necessary code for the project. I noticed it is very similar to Perl's Templating Toolkit, where it allows you to separate the logic and view. 

From what I understand so far, the controller contains the logic, the model has the data and the view is basically the template that the end user will read the data from, as on a webpage for example.

Controller
When I am creating the scripts, I understand that the controller will call on the models or the views and return data to the model (please correct if this is wrong). It was also possible to use just a controller to show data on a webpage, however, it was not formatted. Using the command paster controller name created a default template controller that I could play around with.

Model

The model is where I can put the database connection and queries and store it into values that the template would then also use. I did read that for passing arguments from the model to the template, BuildAPI can use a function called "tmpl_context as c". Variables are called using the ${c.name} and this allows us to use the objects assigned as attributes of c anywhere else in the application - therefore, like global variables.

View
The model passes the data to the view/template which will display that information on a webpage with graphs and charts. These templates use a library set called Mako, which again, is similar to Perl's Templating Toolkit. It has many functions, expressions and tags that can be used. The template files are saved as name.mako

Here is a diagram of it (from wikipedia):

Well, this is how I understand the code will be created using MVC and its quite a lot to learn!

Testing 

One of the difficulties is knowing how all the components fit together, but after playing around with BuildAPI and taking a look at the structure of the pre-existing controllers, mako templates and model files, I have a fairly better understanding to progress into milestone 0.2.

I managed to create a few controllers myself, along with models and views using documentation on the internet (such as on PylonsHQ)

However, they were pretty simple and I need to learn more about the syntax and formats.
Example:

As an example, I was able to pull data (only used one column and a simple select query) and print it out on the screen using just a controller. I ran the command paster controller datatest, which created the template controller file and I modified that.
Note: To print to console, I had to edit the config.ini file and set

sqlalchemy.echo = True.

Simple Controller: datatest.py

from sqlalchemy import *

engine = create_engine('mysql://root:root@localhost/schedulerdb charset=utf8&use_unicode=0')
connection = engine.connect()

result = engine.execute("select name from schedulers")
for row in result:
        print "name:", row['name']
result.close()

This printed out all the names from the schedulers table from the schedulerdb database:

I believe I can get this to display on a webpage, but I need a little help. I can use a controller file, datatest.py, such as this:

import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.getmydata import functionnname

log = logging.getLogger(__name__)

class DatatestController(BaseController):

    def index(self):
        # Return a rendered template
        c.message = GetMessage()
        return render('/displaydata.mako')

The return render will call on the displaydata.mako View/Template.
In the line

"from buildapi.model.getmydata import functionnname"

It will call the model getmydata.py (indicated by buildapi.model.getmydata - i.e. directory/directory/file) and in this file is where I could define the functionname.
For example, if the function name is GetMessage, then it would be defined as:

def GetMessage():
    return "Gimme my data fool!"

In the template, I simply use ${c.message}to get the data. If it is defined as above, then in the webpage (view) that is displayed, it will replace ${c.message} with "Gimme my data fool!".
Note: as shown in the controller file, datatest.py, c.message was defined as:

c.message = GetMessage()

Expectations

• I would like to be able to print the data out to a webpage but will need to understand MVC and pylons syntax more. I will use the resources as listed on the BuildAPI wiki to help me with this
• Once that is done, then I will try to put that data into a graph or chart
• Next would be to pull the correct data and generate graphs based on the project objectives
• And finally, get it all documented!

Quick Note about Python Virtual Environment

A python virtual environment was highly recommended by  Release Engineering, and therefore I also set up one for testing BuildAPI code before I brought it over to my actual installation on Fedora. It was extremely easy to set up, and literally took a few minutes to get it running.
Using easy_install, you can simply use easy_install virtualenv and it will install the virtualenv script to your path. Then you just need to create a directory, run the script wit the directory as your argument and it will set up a sandbox environment. Inside the sandbox/bin folder is an activate script, which as the name implies, activates the virtual environment.

Here are the commands if you are interested:
$ easy_install virtualenv

$ mkdir ~/venv

$ virtualenv ~/venv/sandbox

$ source ~/venv/sandbox/bin/activate

(sandbox)$ easy_install [packagename]

Once you're in the sandbox, you can install whatever else you need to install 

I kept detailed instructions on how to do each step, and I will leave that for the final documentation. 

BuildAPI: Pre 0.1 - free disk space problem

MySQL Database in my space!

For BuildAPI, I needed to install MySQL and import snapshots of two databases that I would be using. So it was installed on my Fedora virtual machine (with a 25GB raw disk (yes, it should have been LVM))... and that eventually ran out of space. The database snapshots 'statusdb' and 'schedulerdb' were almost 10GB of space combined - something I did not expect, and did not account for.

I was close to cloning my virtual machine into another with a larger disk but after consulting with Chris Tyler, there were much simpler options - such as simply adding another raw disk, and this is what I did.

Moreover, I edited my MySQL configuration file (/etc/my.cnf) and pointed the data directory to be on the new virtual raw disk and worked on it from there. Although my VM was terribly slow at importing the (very large) sql dumpfile, and even failed once, it worked out worked out fine in the end.

Lesson: use LVM, or better yet, do this all on an actual host and not a virtual machine

BuildAPI: Milestone 0.1 Release (To be blogged)

Three cups of coffee later...
Instead of posting everything for my BuildAPI 0.1 release now, I am going to get some sleep and post it later on today. However, the BuildAPI Project Page has been updated with what has been done for 0.1 so far and what needs to be done between now and milestone 0.2.

Here's a quick summary of 0.1:

• Completed a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
• Set up BuildAPI and ran the Paste server to use the pre-existing BuildAPI controllers 
• Created several test controllers, models and templates for testing and gaining a practical understanding how Pylons work
• Created my own script using SQLAlchemy and MySQL engine to pull information from the SchedulerDB and StatusDB database. 
• However, it displays only text (one line - refreshing page shows next line etc) and I need it to loop the values on the page to display all (should not be hard at all)
• Before milestone 0.2, I would like to be able to generate a simple graph from the data I pulled so far.

--and would you believe it, I actually had fun working with BuildAPI all night...until the caffeine was gone and dawn broke.

Oh, and save the procrastination lecture for later. I have just been extremely busy (if you want to know - drop me a comment and I'll tell ya) and did not find enough time to work on the project. However, this is what all-nighters are for, because I'm a crazy workhorse!!


zzzzzzzzzzz......