Archive for the ‘Bugdar’ category

Bugdar2 Update

It’s been a while since I last wrote about Bugdar2 development, so I thought I’d post an update.

Over this summer, I managed to get a lot of work done for Bugdar2 and started working through the roadmap to the Alpha 1 milestone. The purpose of the Alpha 1 milestone was to do as much of the core backend as possible, without writing much templated UI. While I did end up writing some skeleton templates to test basic functionality, most of the backend work was tested through unit tests.

When I got to the point where I couldn’t do much more backend work without having interfaces to add and modify data, Alpha 1 was complete. The goal of the next milestone Alpha 2, which I’m currently working, is to get the basic UI shell up. I’ve started building the CRUD templates for entities like bugs, settings, usergroups, and attributes. In this milestone, the initial look and feel will come together and a round of string extraction (the first step towards localization) will occur.

Here’s two screen shots of Bugdar2 as it currently exists. These are very rudimentary and will evolve as more time is put into them.

Unit Testing in phalanx

In a previous post, I stated that Bugdar2 is powered by a brand-new PHP framework called phalanx that I had written from scratch. In this post, I’m going to talk a little about what’s been going on with Bugdar2. The past month or so have been very busy, but I’ve managed to start writing unit tests and do some work on MacGDBp 1.4.

Phalanx is not a typical MVC framework. While it aims to keep that style of separation, it does so in a very different way than most web frameworks. Zend Framework, for instance, uses a Controller class that has various action methods. In phalanx, the controller unit is replaced by an Event object.

What is an event? An event is essentially a three-stage function with explicit input and output values. When defining an event, clients must explicitly list the input values the event requires and the output values it will produce. This is important because it makes unit testing extremely easy: if you know what should happen, it is not difficult to test it. The three stages an Event goes through are WillFire(), Fire(), and Cleanup(). The only required method is Fire(), which is where the event does the work of transforming the inputs to the outputs. WillFire() provides an opportunity to cancel an event if preconditions are not met, and Cleanup() is called regardless of if an event was successfully fired or not.

Here is an example. This is the event that creates new bugs. It takes in a title and the body of the first comment (description). On return, it provides the bug and comment IDs. From here, it’s easy to verify the results: create an input dictionary, create the event, and post it to the pump to run it.

In the code, references are made to the EventPump. This is the class that is responsible for firing events. The following is the description of control flow in a web app: In the index.php page, a HTTPDispatcher is created; this class parses HTTP requests and synthesizes the Event object for that request. The resulting event is then passed to the EventPump via PostEvent(). Since that is the first event in the pump, it begins processing immediately. If another event is sent to the pump via PostEvent() or RaiseEvent() [preempting the current event], it will be processed, too. Once all events finish processing, the OutputHandler is notified to take the list of events and figure out which one(s) it needs to display output from. The ViewOutputHandler simply takes the last successfully completed event, gathers the output from the event, and passes that information to the template. When unit testing, we can do away with the Dispatcher and OutputHandler. Just synthesize events, send them to the pump, and test the outputs.

Currently, work on Bugdar2 consists mostly of writing these unit test cases. Currently I’m blocked by engineering the API for bug attributes. Because attributes are at the heart of Bugdar2, I want to make sure I get this part right. Attributes and usergroups are the last two features before Alpha 1 is complete.

Public Repos and Bug Attributes

Bugdar 2 is now sufficiently far along that I’m comfortable posting the git repository. The Phalanx repository has been public from the beginning (albeit unnanounced) at Github: http://github.com/rsesek/phalanx. Bugdar 2’s repository is now also available there: http://github.com/rsesek/Bugdar2. At the same time, they will be availiable at their canonical home in https://bluestatic.org/git/. The Github mirrors are just easier for most people.

Development of Bugdar 2 is still moving fast. In the past few days I’ve laid down the core of the search system and added support for resubmitting a POST request after login (don’t lose your work if you’re not logged in). There’s two more major milestones before I consider us past the ‘first alpha’ stage: usergroup permissions and bug attributes. What are bug attributes?

In Bugdar 1, there were a set of fields that Bugdar compelled you to use: product, version, status, resolution, priority, severity, and assignment. If you didn’t want to use one of these fields, you couldn’t remove it — you just had to ignore it. There were also custom fields, but the architecture under them was always shaky. By having all these required fields, we were defining a bug as so:

CREATE TABLE bug (
	bugid int unsigned NOT NULL AUTO_INCREMENT
	userid int unsigned NOT NULL
	dateline bigint unsigned NOT NULL
	product int unsigned NOT NULL
	component int unsigned NOT NULL
	version int unsigned NOT NULL
	summary varchar(255) NOT NULL
	priority int unsigned NOT NULL
	severity int unsigned NOT NULL
	status int unsigned NOT NULL
	resolution int unsigned NOT NULL
	assignedto int unsigned NOT NULL
	duplicateof int unsigned NOT NULL
	dependency text NOT NULL
	hidden smallint unsigned NOT NULL
	initialreport int unsigned NOT NULL
	-- [A lot of other cache information here]
);

That’s a lot of data! If a bug isn’t going to use one of those fields, it’s still going to carry around all that weight. In Bugdar 2, this is how we define a bug:

CREATE TABLE bugs (
	bug_id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
	title varchar(250) NOT NULL DEFAULT '',
	reporting_user_id int NOT NULL DEFAULT 0,
	reporting_date int NOT NULL DEFAULT 0,
	hidden boolean NOT NULL DEFAULT 0,
	first_comment_id int NOT NULL DEFAULT 0
);

That’s a huge difference. You’ll notice all the attributes — the metadata — are not stored on the bug itself anymore. Instead, that’s done through a tuple:

CREATE TABLE bug_attributes (
	bug_id int NOT NULL DEFAULT 0,
	attribute_title varchar(50) NOT NULL DEFAULT '',
	value varchar(250),
	PRIMARY KEY (bug_id, attribute_title)
);

The implications of this are that an single (unique), arbitrary attribute can be attached to a bug at the user’s will. The requirement of data is now imposed by the administrator rather than the developer. It also helps to normalize the database schema. So how do attributes work?

Attributes can be defined formally in an administrative panel, or they can be defined ad-hoc while editing bugs. Attributes can be titled to create a field, or they can just be values, which will be represented as tags. The goal with this system is to create a highly flexible data model in which all the stock Bugdar 1 fields could be represented as attributes.

Formally defined attributes can be one of five types: text, boolean, list, date, or user. Text fields are self explanatory. Booleans are interesting because they don’t necessarily have binary value; they have ternary value: true, false, and unset — but that’s more of a semantic detail. Lists are a set of selectable, pre-defined options. The last two types are new since Bugdar 1. Date fields will have a calendar selection widget. Finally, user fields allow the bug to be attached to some user in Bugdar’s system; this is how assignments work.

This is a rendering of how attributes will be arranged on the bug display:

That same information takes up more than half the page in Bugdar 1. That’s all for now.

Long-overdue update

So I know I promised at Thanksgiving an update on the state of things. Well I lied/got busy. Sorry; these things happen. The past few months have been a whirlwind. So here’s a brief personal update before I talk about software.

In May 2009 the Mac port of Chromium (the open source project that Google uses for their Chrome browser) was starting to gear up. I’d been interested in working on a Mac browser project for a while; but Firefox’s code seemed beastly and I never have been a fan of XUL. Safari is great, but it isn’t open source. Enter Chromium: an open-source, truly native, WebKit-based browser. Consider it love at first sight. Fast forward 12 patches and four months later to August when I am nominated and made a committer. My work has all involved bridging the cross-platform C++ model (as in MVC) to Objective-C/Cocoa land. I wrote the Mac the page info window, history menu, cookie manager, font and language settings, download settings, and added favicons throughout various parts of the UI. Now, present day. Last week Google offered me an internship position in Manhattan. I’m thrilled at the opportunity and am excited to start work there in May with such smart, interesting, and talented people. I expect to learn a lot!

So now let’s talk software.

Bugdar

Last time I wrote about the future of Bugdar 2, I posted a link to a list of phases development that would occur. The original plan was to create 1.3 as a stepping stone to 2.0. That line of development has since been abandoned. Instead, I decided to greenfield the project. There were many reasons for doing this; but here are the two big ones:

1. I have a better, more focused vision of what I want Bugdar 2 to be. When writing Bugdar 1, I didn’t have a clear goal of what I wanted — there was no philosophy to the design, I was merely thinking “copy Bugzilla, but do it better”. Overall, I think I was successful with this. But this is not Bugdar 2, at all. Bugdar 2 will be a refined, incredibly flexible system that I am excited to share with you. But because the two projects will be so different, trying to create a large, temporary scaffolding system (version 1.3) was extremely time consuming and kludgey.

2. ISSO. Originally dubbed the “Iris Studios Shared Objects” framework. This thing is ancient. The first version dates back to when I was a rather n00b PHPer. The first commit was from 12 January 2005! Since that time I’ve had a lot more development experience and completed a Minor in Computer Science. A better, more-efficient architecture could be devised.

I wrote an entirely new PHP framework (that will require PHP 5.3+) that is a bare-minimum, no-frills framework. I call this Phalanx and the code is currently available on Github; it is licensed under the GPL version 3. Phalanx is event-driven. Rather than writing another MVC framework, I decided to look at the problem of web frameworks through the lens of functional programming. The problem with MVC on the web is the lack of state; HTTP is not persistent. Functional programming focuses more on inputs and outputs (and the transformations between them), which is how Phalanx works. Each Event class defines a set of inputs that it accepts and the outputs it will return. This also makes unit testing much more clear-cut. I won’t get into further details here, though. That’s for another post.

So before I could start moving again on Bugdar, I had to write this new framework. That took a matter of months (July – January). But I’m now back on Bugdar. Phalanx makes development really fast, and in the past two days I’ve implemented the basic user system, a preliminary Auth2 API, and initial submission and listing of bug reports. I’ve posted a new roadmap for Bugdar 2.

MacGDBp

I’ve been refactoring and cleaning up MacGDBp’s code over the past few months. It’s also been accumulating various fixes for non-critical bugs. I’d also like to re-tackle the socket/network layer for 1.4, as I tried and failed to improve this in 1.3. You can probably expect this update before Spring.

Site Redesign

I’m also working on redesign the Blue Static site. It really hasn’t changed since 2006 (insert the ‘learned a lot’ bit here) and it’s time. I’m not very far on this, yet.

Wow that’s a lot. More in the next few days. UI mocks to come.

Happy Thanksgiving!

Happy Thanksgiving to all you US citizens!

Today I’m releasing a security update to Bugdar, version 1.2.4. This version addresses a possible SQL injection vector in search.php. The details of the report can be found here. Thanks to Adam DiCarlo for disclosing the issue.

Please update to 1.2.4 as soon as you can. The release links can found on the normal Bugdar webpage. Alternatively, here are the direct ZIP and Tar links.

I apologize for the lack of posts, but I’d rather be coding rather than talking about coding ;). I’ll post an update on the state of Bugdar 2.0 in a few days.

Bugdar 1.2.3 and the Future

Today I released Bugdar 1.2.3, exactly 364 days since the last release of Bugdar. I regret that this release was so long-coming and was far overdue. This release fixes a bunch of bugs that have been accumulating over the course of the past year.

In the future, I will not drag my feet when releasing minor updates like this. The problem is that the current code base for Bugdar (currently the 2.0 Alpha) is quite incompatible with the 1.2.x line. This means, all fixes for bugs affecting 1.2.x need to be redone for 2.0 — which is a lot of duplication of effort. So, rather than continuing to do that, I came up with a better solution.

Bugdar 2.0’s development has been broken up into distinct phases. Unfortunately, due to my own lack of time, progress is slower than I would like. Therefore, I’ve decided that there will be a 1.3 branch to “bridge the gap” between the far-off 2.0 and the current, aging 1.2 branch. Phases 1-3 of the 2.0 development cycle are really underlying architecture changes that just bring the code base up-to-date. After those three phases are complete, Bugdar 2.0 will look and work essentially like version 1.2, but will be more robust and (most importantly) will have a significantly improved language system. It is after Phase 3 that I intend to branch the 2.0 development line into two (the maintenance branch 1.3.x and the current development head 2.0). This would effectively kill the 1.2.x line. And because the two branches would have a more recent ancestor, migrating patches should be easier.

This branching strategy requires little work on my part and will reduce the time I spent migrating patches between branches. Also, it will put in place a language system that will work across 1.3 and 2.0 which is crucial. I have been wanting to create a language pack exchange system so that users can collaborate on and exchange translations of Bugdar, but at present the MO/Gettext system is not good for that. This new language system will be designed to do that. And this branch strategy will get that deployed sooner rather than later.

I wish I could have some sort of timeline for when 1.3 will come out, but I can only really say “as soon as I can.”

Bugdar 2.0 Admin CP

The new Admin Control Panel in Bugdar 2.0.

The navigation system has changed from the tabbed navigation in the 1.1 and 1.2 series. The navigation sections are openable via JavaScript, which will allow navigation of the entire Admin CP from any page, rather than limiting navigation to the current tab section.

Bugdar 2.0 β Phase 1 Complete

I’m pleased to announce today that Bugdar 2.0’s first development phase has been completed. During this phase, all of the Bugdar code was ported to the ISSO3 framework, allowing it take full advantage of property access control, static functions, and the latest ISSO3 conventions.

The point of this phase was to upgrade the ISSO method calls to make refactoring code a simpler process. Had this first phase not upgraded ISSO-based code, the refactoring phase (phase 4) would have been twice as long and twice as much effort because the underlying ISSO code would be out-of-date.

Next in Bugdar’s development is the admin control panel rewrite. Before this can happen, however, a new ISSO3 module must be developed to make creating the admin interface easier. This will include removing the BSPrinter classes (which currently use PHP functions to generate HTML) and replacing it with well-designed CSS style sheets that will be used in templates in the Admin CP.

Separation of the view-controller logic in the Admin CP will make the code more agile and will allow for a more intuitive interface because the HTML will not be generated via on-the-fly code, but rather by designed HTML.

Localization in Bugdar 1.2.2

Some people have been running into issues with the PHP Gettext extension. The problem is that to use the extension right, it requires that a lot of server environmental variables to be tinkered just right. And I’m not even sure it works on Windows (to be honest, I don’t test on Windows servers at all).

However, I was looking at the MO format specification and it is pretty easy to parse, so I wrote a custom implementation of Gettext in PHP. This custom implementation does not rely on the Gettext extension at all, so this will also solve the problem for people who need to localize and don’t have the extension installed.

Because I didn’t want to introduce such a sweeping change in a point-release, this is currently disabled. With a hidden toggle and a simple query, you can try out the new localization feature if Gettext is giving you trouble.

Steps to enable the new MOReader localization system:

1. Open includes/config.php and add this line anywhere in the file:

$debug = 1;

2. Go to Administration –> Bugdar Settings and find the setting labeled “Custom Gettext Localizer” and enable it.

3. Run this SQL query to clear the template cache (and thus all the calls to the old localization system):

TRUNCATE TABLE template

4. Back in includes/config.php, remove the line you added above. Debug mode should not be used in a production environment.

This hidden toggle will disappear in 2.0 because it will be the default (and only) translation system. But like I said, it was too big to make the default in a maintenance release. You will only have to enable this once (it will stick across upgrades, etc.). If you want to disable it, just follow the above steps, but just flip the switch to “No.”

The Bug Back-Log

If you’ve taken a look at the bug tracker recently, you’ll see quite a few bugs back-logged. I’m not ignoring them (I still check out each individual report when it’s posted or updated), and if the bug is critical I’ll, of course, fix it. But at the moment I’m focusing completely on finishing the rewrite of ISSO. My time is extraordinarily limited so things are taking a lot longer than expected, but I’m really excited to start work on Bugdar 2.0. Before I can get there, though, ISSO needs to be finished. Here’s a little update on what’s left to do for ISSO:

API

The API module is what I’m currently focusing on. The error system is getting an overhaul to use PHP5’s exception system instead of trigger_error(). Furthermore, the validation system is being rewritten to work with the new error system.

Db

Both MySQL and MySQLi modules are done and have complete unit tests. I still need to update the PostgreSQL module, however, because I don’t have Postgres on my laptop yet.

Graph

The graphing module, which has yet to be used in an application, is being removed. Instead, I have started a new project called WebFreeChart to take the place of it.

Pagination

I have yet to unit test and update the pagination module. This will pretty much stay the same, though instead of using templates (whose names are hard-coded), I’ll switch to hooks and callbacks.

Printer

The printer classes are used to generate the green-styled interface for admin control panels and the like. I have already rewritten the classes but I am unhappy with them because they’re so verbose to use. I’m now thinking of ditching it for XML-based (maybe even XSLT) generators. There’d be various XML tags for the existing classes and methods to generate form elements with the proper markup. This would separate the presentation from the logic, while still not having to worry about styling and direct HTML.

I hope to finish these things off soon to get working on Bugdar 2.0, but because my time is so limited, I really can’t give an ETA. I’ll try to keep posting updates so you’re aware of what’s going on.