Unfortunately, I did not fully accomplish my goal for release 0.3 - I was able to make progress but in the end I did not turn out the way I had hoped. I tried to get JSON output working as exampled in the BuildAPI existing scripts but could not get it going. However, I did learn a tremendous amount of information which can be seen by browsing through my blog.
In addition, I have created a quick and concise How-to document for BuildAPI in hopes that the next student that possibly takes on BuildAPI will have a head start into the project and a better understanding of its inner workings without wasting too much time figuring it out.
Furthermore, there will also be a second document that provides more information on the controllers, models and template files that will be posted later.
Documentation
Here is a link to the documentation: http://asdesigned.ca/sbr/BuildAPI-How-To-Setup.pdfIt is in PDF form (clickable links included).
Below is my Getting Started documentation in HTML format, although I'd recommend viewing the PDF version instead:
Seneca College |
BuildAPI |
How to Get Started with BuildAPI |
Andrew Singh 12/17/2010 |
Table of Contents
| Introduction |
| Purpose |
| Tasks Summary |
| I. Install MySQL and Import Database Snapshots |
| II. Install Python and Python SetupTools (Easy_install) |
| III. Install Google Python Visualizations Library |
| IV. Install MySQL-Python (MySQLdb for Python) |
| V. Install BuildAPI Files |
| VI. Configure and Run BuildAPI |
| VII. Optional: Set up sample project files |
| VIII. Resources |
BuildAPI
“BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.”
Introduction
This project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying databases for information on the BuildBot jobs, which will then be used to generate the necessary reports.
Purpose
To install, configure and run BuildAPI
Tasks Summary
- Install MySQL and import database snapshots
- Install Python, Python SetupTools (Easy_install) and Pylons
- Install Google Python Visualizations Library
- Install MySQL-Python (MySQLdb for Python)
- Install BuildAPI Files
- Configure and Run BuildAPI
- Optional: Set up sample project files
- Resources
I. Install MySQL and Import Database Snapshots
Install MySQL
Install Mysql and Mysql Server:
yum –y install mysql mysql-server
Enable the service at startup:
/sbin/chkconfig mysqld on
Start the MySQL server
/sbin/service mysqld start
Set the MySQL root password
mysqladmin -u root password 'new-password'
The quotes around the new password are required
Root can be used to access MySQL or a new MySQL user can be created for the project.
For testing purposes, root can be used.
Install Snapshots
Download the database snapshot files, which are in bunzip2 format. Please check https://wiki.mozilla.org/ReleaseEngineering/BuildAPI for the latest snapshot releases, under “Project Requirements”.
If bunzip2 does not exist on the system, install it using: yum install bzip2
Extract the databases
bunzip2 schedulerdb-2010-10-22.sql.bz2
bunzip2 statusdb-2010-10-22.sql.bz2
Create the respective databases; Example: statusdb and schedulerdb in MySQL
$ mysql –uroot -ppassword
mysql>
mysql> create database statusdb;
mysql> create database schedulerdb;
mysql> exit
Import the databases into mysql server
mysql –uroot -ppassword -hlocalhost statusdb < statusdb.sql
mysql –uroot -ppassword -hlocalhost statusdb < schedulerdb.sql
II. Install Python and Python SetupTools (Easy_install)
Install Python
yum install python
Install Setup Tools
yum install python-setuptools
Install Pylons
yum install pylons or easy_install Pylons
yum install python-pylons
Virtual Environment
We can also set up a virtual environment to work in, whereby it allows us to have multiple isolated environments for Python so it is possible to run different applications on the same system, but using different sets of installed Python packages. This is recommended by the Mozilla Release Engineering team.
Install virtual environment using easy_install and read the readme file to activate
easy_install virtualenv
Manually installing virtual environment
Use wget to get the latest package from http://pypi.python.org/packages/source/v/virtualenv/
Untar the downloaded file, create a directory and activate the environment
$ tar zxfv filename.tar.gz
$ virtualenv.py ~/venv/sandbox
$ source ~/venv/sandbox/bin/activate
Once in the sandbox, install packages as necessary, example:
(sandbox)$ easy_install Pylons==0.9.6.1
If you will be working within a virtual environment, it is important that you install the necessary packages below in the same environment as well.
III. Install Google Python Visualizations Library
Download the library from http://google-visualization-python.googlecode.com/ and extract it.
To install the library:
cd into the extracted directory and run the command:
python ./setup.py install
Test the library by running python ./setup.py test to make sure it is installed correctly
IV. Install MySQL-Python (MySQLdb for Python)
Install Dependencies
This package requires the python-devel dependency or it will not install properly.
yum install python-devel
Install Package
Once that is complete, we can install MySQLdb.
You can stop the mysql server before installing MySQL-Python and restart it once complete.
yum install mysql-python
In the event that it fails, you can also try a manual installation:
Download the source file from http://download.sourceforge.net/sourceforge/mysql-python/
Extract the downloaded file and change to its directory
Run the commands:
python setup.py build
python setup.py install
Optional: run the test to see if it is installed properly: python setup.py test
V. Install BuildAPI Files
The easiest way to install BuildAPI is to use Python’s setup tools:
easy_install buildapi
This will also install SQLAlchemy which is required; otherwise it can be installed using easy_install SQLAlchemy
For manual installation, you can download the source files from http://hg.mozilla.org/build/buildapi and extract it to a directory called buildapi
This will create a directory structure similar to the following:
------buildapi/
-------MANIFEST.in
-------README.txt
-------docs
-------ez_setup.py
-------buildapi (This is a nested directory)
-------buildapi.egg-info
-------setup.cfg
-------setup.py (used for application setup)
-------test.ini (used for testing, i.e. when you run python setup.py test)
Run the command:
python setup.py install
VI. Configure and Run BuildAPI
Next, BuildAPI must be configured to host content on your system, either through localhost or on the internet. Moreover, it uses the Paster server for hosting and requires creating and editing a configuration file.
We will create our own configuration file using the command:
paster make-config buildapi config.ini
This command produces a different configuration file with sensible options for production use
Now we can edit the created config.ini configuration file and make changes as below.
[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000
# SQLAlchemy database URL
sqlalchemy.scheduler_db.url = mysql://root:root@localhost/schedulerdb
sqlalchemy.status_db.url = mysql://root:root@localhost/statusdb
sqlalchemy.pool_recycle = 3600
Note: host can be set to any value such as iraq.proximity.on.ca to make it accessible on the internet provided configurations such as firewalls allow it. The sqlalchemy database url and username:password fields as they may need editing accordingly if you have used different names or different databases.
Run BuildAPI
To run, stop or reload BuildAPI, make sure you are in the buildapi directory (first level, where the config.ini file resides) and use the following commands
paster serve --start --daemon config.ini
paster serve --stop --daemon config.ini
paster serve --reload --daemon config.ini
Whenever a change is made to the configuration file, or major changes made to controller files, the paster server must be reloaded.
Check the current directory (buildapi), there should be two files that were created:
paster.pid – contains the PID number for the Paster process
paster.log – All messages are contained in this file. Extremely useful file for troubleshooting!
Test it out!
Depending on your config.ini configuration settings, direct the browser to
And that’s it! BuildAPI should now be live on your system!
VII. Optional: Set up sample project files
These are files that were part of the BuildAPI project. They can be used to get a better understanding of BuildAPI and to understand of how it works, along with the blog posts (on http://andrewasdesigned.blogspot.com/).
Additionally, see the https://wiki.mozilla.org/ReleaseEngineering/BuildAPI page for a quick beginner tutorial as well.
File | Description | Download Location |
config.ini | Paster Configuration file | |
hello.py | Hello World controller file – a Getting Started controller | |
project.py | Project Controller file | |
query.py | Project Query file, which is the BuildAPI query file with Project queries appended | |
project.mako | Project template file | |
routing.py | Project routing.py file |
Directory Structure showing where files should be placed:
buildapi/
+-- buildapi
¦ +-- config #routing.py, placed in config
¦ +-- controllers #project.py, hello.py placed in controllers
¦ +-- model #query.py, placed in model
¦ +-- templates #project.mako, placed in templates
Note that other directories and files have been omitted
The Python library called Routes handles mapping URLs to controllers and their methods, or their action as Routes refers to them. By default, Pylons sets up the following routes (found in config/routing.py):
map.connect('/{controller}/{action}')
Therefore, you can also edit the buildapi/buildapi/config/routing.py file and add the following in manually, instead of using the routing.py provided file:
map.connect('/project', controller='project', action='index')
This file controls how the controller or project site is accessed. The above means that the site is accessed, for example, by www.domain.com/project which will automatically redirect it to the index as defined in the controller file.
A Word on Database and queries:
It is important to understand the database structure and use the resources to figure out which queries and tables would provide the best indication of certain values, such as CPU usage. The queries are also done using SQLAlchemy and it can get complicated with complex queries, therefore, seek out help (See Resources section).
VIII. Resources
It is extremely important to reach out to the community for help in any area. I cannot stress this enough. There are many resources available, such as IRC, official documentation and discussion groups that you can take advantage of. Some are listed below.
IRC
Server: irc.freenode.net
channels: #seneca #mozilla
Blogs and Wikis:
Discussion Groups
http://groups.google.com/group/pylons-discuss (Google Group)
Official Pylons Documentation
http://pylonshq.com/docs/en/0.9.7/ (Pylons Documentation)
http://pylonsbook.com/ (Free Online Pylons Book and amazing resource)
SBR600 (Seneca) BuildAPI Project Page:
No comments:
Post a Comment