Wrong Notes

Now can we please raise the debt ceiling?!

2011-07-22T23:38:00.016-04:00

I wanted to look at how bad it's gotten just by looking at the numbers we're up against. What we are arguing over is money the US takes in vs. pays out in obligations. At present time the government spends $3.834 trillion, and takes in $2.567 trillion. You should already see the problem. We're spending $1.267 trillion that we don't have. So where do we get that from without raising taxes?

By issuing more bonds, but we can't do that until this debt ceiling is raised. See we've been doing this since the 1980s. We spend more than we take in, and to get money we sell US Treasure bonds to people to keep operating. However, the debt ceiling is a law on the books that states the US Government won't borrow more than X, and every time we reach X Congress votes to raise it to Y, sells more bonds to cover the deficit, and we keep going. And, people are perfectly happy to buy them because the USA has NEVER defaulted on those obligations.

Now of that $3.834 trillion in spending some of it is allocated by law. By law we have to spend it. If we wanted to change it Congress would have to create a new law that cuts that spending. These are things like Social Security, Medicare/Medicaid, National Debt Interest, Income Security, and Veterans Benefits. This doesn't get discussed much because passing a law to cut these is really difficult, and politicians, on both sides, don't want to be the one that slashes these because they will be voted out. Some of these you can't do anything about like National Debt Interest. You don't pay that and that spells default, USA gets it's AAA rating slashed, interest rates rise up, babies die, and Jesus weeps. The $250 billion in National Debt interest is interest on all that borrowing we keep doing. For the remaining items Social Security, and Income Security are funded by specific taxes. If you cut those programs it doesn't help because those special taxes can't be used to pay for other spending. That is illegal. So what does that leave? Medicare, Medicaid, and the discretionary budget as places you can cut. I'm leaving Veteran's Benefits out of it because it's $68 billion which even you completely cut it to zero it would contribute squat, and persons who cut that thing would make Casey Anthony look like Mother Teresa.

What we're really talking about is the Discretionary Budget which in 2011 is $1.415 trillion dollars of which 63% ($895 billion) is spent on Military spending, and 37% ($520 billion) is spent on non-Military spending. In 2004 the Discretionary Budget was $782 billion and 51% ($399 billion) for Military Spending and 49% ($383 billion for non-Military Spending). That's a 58% increase in the budget in 7 years. You'll also notice how much the military percentage of the pie has increased. That means it's rising at roughly 6.8% per year. More than twice the typical 3% inflation rate. But, the more disturbing trend is military spending has increased 12.3% per year while non-military spending rose only 5% per year. Why is that important? Because Discretionary Military spending is single largest expense the American government pays out, hence if we really want to make serious cuts it has to start with the military spending.

If we didn't want to raise the debt ceiling we need to come up with $1.267 trillion by cutting spending or raising taxes. If we didn't want to raise taxes and you don't want to cut the non-Discretionary items, then we'd need to cut $1.267 trillion from the $1.415 trillion Discretionary budget. That would leave $148 billion for the government (both military and non-military) to run on. Our government couldn't function no matter how much the Tea Party wishes that were true.

What if we consider the full budget for cutting funding. In order to cut spending enough, so we don't have to raise taxes, we'd need to cut 58% from Discretionary Military spending, Discretionary Non-Military spending, Medicare, and Medicaid. If we included Income Security in those cuts we can get it down to 46% cuts across the board. And if we included Social Security it'd be around 36% cuts across the board.

Ok so let's look at it from what we'd need to do to raise taxes to cover it. In order to get $1.267 trillion more we'd need to increase taxes by 50%! 50% tax increase would cover the deficit without cutting any spending. Now if you thought cutting spending to cover it was insane. Raising taxes by 50% is bonkers. I can't afford a 50% tax hike as I bet neither can you, and corporations would get a shock so bad Wall Street would absolutely freak their shit. And, send their K Street soliders to figure out a way to shirk their responsibility. Yep same song different verse. So even if you could pass the bill I bet they couldn't collect on those taxes.

Those are the two extremes of the argument. You can't cut your way to a balanced budget, and you can't tax your way one either. However, getting really serious about fixing those problems means serious cuts and serious tax hikes. Looking at closing loops holes to raise revenue, and cutting spending is the only way you could reasonably do it. But again, there's no perfect answer given the constraints. It will still require serious cuts, and tax hikes. Even raising taxes 10% you'll need to cut $1 Trillion in spending across the board. That is going to be very hard. What about the Bush tax cuts? Even rolling those back will only add $300 Billion-ish in revenue.

The easiest way out is to raise the debt ceiling because defaulting will have tremendous consequences. And, to think it will get worked out if we miss the Aug 2nd deadline is a farse because we're already on borrowed time. This thing was supposed to get wrapped up 6 months ago, and the Treasury did some funny accounting to get more time. They've been in a stalemate since then. So if they can't figure it out in 6 months what makes you think they'll figure it out in another 6 months when the Treasury is out of money? They've been living on life support for 6 months.

So given all of the facts can we please just raise the debt ceiling? My 401K doesn't need 3rd shot to the junk in 10 years.

How Failing Fast allows you to reframe the problem

2011-03-28T20:42:00.005-04:00

Just read an article on Fast Company on how human powered flight was solved by Paul MacCready. It's really cool because it's not a software story, but it has so many similarities with software. Success centers around creating an environment where you can iterate on your idea. I like stories like this because the motto of "fail fast" gets hollow as it is over used. After a while It's hard to remember what it originally meant. Stories help re-affirm it's meaning.

In so many ways this is really what agile software development is trying to get you to. Agile demands a lot from your team, and the only way you can live up to the promises of agile development is to create this environment. Without it you'll just fail, or worse just survive on far less productivity.

No more big design up front. It failed people for human powered flight, it failed for cars, and it failed for software.

http://www.fastcodesign.com/1663488/wanna-solve-impossible-problems-find-ways-to-fail-quicker

View Source and SVG on the iPad

2011-03-19T17:21:00.011-04:00

I'm playing around with SVG on the iPad, and I find it's hard to really debug even the smallest thing on it. Apple is a lot of things, but calling them a developer of great development environments would be a grandiose lie. Before I say something I'll have to do a lot of explaining about I wanted to share a script for view the source on your iPad for SVG documents.


javascript:var%20sourceWindow%20%3D%20window.open('about%3Ablank')%3B%20%0Avar%20newDoc%20%3D%20sourceWindow.document%3B%20%0AnewDoc.open()%3B%20%0AnewDoc.write('%3Chtml%3E%3Chead%3E%3Ctitle%3ESource%20of%20'%20%2B%20document.location.href%20%2B%20'%3C%2Ftitle%3E%3C%2Fhead%3E%3Cbody%3E%3C%2Fbody%3E%3C%2Fhtml%3E')%3B%20%0AnewDoc.close()%3B%20%0Avar%20pre%20%3D%20newDoc.body.appendChild(newDoc.createElement(%22pre%22))%3B%20%0Avar%20src%20%3D%20''%3B%0Aif(%20document.documentElement.innerHTML%20)%20%7B%0A%20%20%20src%20%3D%20document.documentElement.innerHTML%3B%0A%7D%20else%20%7B%0A%20%20%20var%20div%20%3D%20newDoc.createElement(%22div%22)%3B%0A%20%20%20div.appendChild(%20document.documentElement.cloneNode(true)%20)%3B%0A%20%20%20src%20%3D%20div.innerHTML%3B%0A%7D%0Apre.appendChild(newDoc.createTextNode(src))%3B

To get this on the iPad follow these steps.

Open this page on the iPad.

Select all of the text from the prior paragraph

Add a bookmark for this page.

Edit the 2nd field and past the copied text in there

Now open a SVG document and click your new bookmark

This is a modified version of source code from Rob's Blog. The only problem with Rob's version is the use of innerHTML. Unfortunately, SVG doesn't have innerHTML. This code will handle document nodes that don't have innerHTML property by cloning them and placing the clone in a DIV element. That way we can properly get the innerHTML from there. Using this code will allow you to see the SVG and HTML source.

Here's the source code for this bookmarklet for easy debugging if you have trouble:


var sourceWindow = window.open('about:blank'); 
var newDoc = sourceWindow.document; 
newDoc.open(); 
newDoc.write('<html><head><title>Source of ' + document.location.href + '</title></head><body></body></html>'); 
newDoc.close(); 
var pre = newDoc.body.appendChild(newDoc.createElement("pre")); 
var src = '';
if( document.documentElement.innerHTML ) {
   src = document.documentElement.innerHTML;
} else {
   var div = newDoc.createElement("div");
   div.appendChild( document.documentElement.cloneNode(true) );
   src = div.innerHTML;
}
pre.appendChild(newDoc.createTextNode(src));

When you're doing it wrong...

2010-11-18T10:03:00.010-05:00

How do you know when you're doing it right? Most of the time I know I'm doing it right is when it feels like I'm always hitting my goals, and it's getting easier than it was yesterday. Although that might be a little lie I tell myself because it might just be because I know when I'm doing it wrong, and how bad that feels. If I don't have those bad feelings I know I must be doing something right.

Here's a great example of doing it wrong. I'm at a place that loves to branch code. Most of the time they are branching because the business demands a release, but they have such a large team in order to keep everyone "busy-ish" they have to branch. They have an idea they're doing it wrong, but they don't really have a clue as to how to do it right so they just do what they know. The developer's don't like branching, but the business doesn't give them much choice.

Problem is refactoring is important because the code base is pretty hard to work with. Now when they add multiple branches + refactoring + big team = double black diamond level of difficultly in the merges. So that's bad, but another side effect is when a merge is going on it prevents people from modifying the repository. Nobody can use the source control system while this is happening. It's an all stop. One of these merges is going on it's 5th day. That 5 days where no one has integrated their changes, built all of the code, or synchronized with other people's changes. Now all of a sudden the choices to use SCM, continuous integration, refactoring, and small agile practices is really loosing it's benefit. One developer suggested we send around patches to each other while the merge is going on. We specifically picked a SCM system so we don't do that. Once the merge is done the SCM system is going to hit with tons of changes, and when something breaks functionality they won't be able to easily resolve it because of the volume of changes. Now quality is suffering directly because of relentless branching.

Funny thing is I can't think of anyone out there that suggests branching as a technique for achieving quality. However, there are countless examples from experts that generally agree using SCM, continuous integration, refactoring, and small changes help overall quality. Why are we doing something that sacrifices those best of breed practices? Now the guy with the "big picture" view seems to believe it's the actual code quality is to blame for productivity problems, and quality issues. He thinks more code reviews, and education is in order about how to write "good" code will right the ship. At some level business is just throwing crap over the wall without any real conversations.

This is what I call an "everything is arduous and ridiculous" environment. Everything about this place feels over the top hard. Why don't people around me seems to realize this is the ridiculous way to operate? Haven't they ever had that effortless feeling of productivity? How you're always the man, and it's just right? Sure this works in that we are producing a product very slowly, but it doesn't feel like a success. Is it luck? Is it innate to the problem you're trying to solve? Well...maybe.

Sure some problems are harder than others. Building Google mail is harder than building an android app. But, I've been on some pretty nasty android apps. Which tells me there is a way to make an easy problem hard, and hard problem easy. So what are we doing that makes this problem so hard?

Sometimes it's not being smart enough. We all love algorithms, and finding that simple algorithm that just makes the problem go away is sublime. That's what we all fell in love with if we have any formal training. But, those types of problems are far and few between. Mostly what we do is slog crap from one database, slap it on the glass, then slog the new crap back into the database. Rinse and repeat 1000x and you've got a product. There is no algorithm that makes that easier. If there's no algorithm then what is it?

Technique. There's a difference between algorithm and technique, and what types of problems they are best suited for. Technique isn't going to come up with map reduce. That's algorithm. Technique is your choices for what you're going to use, and how you're going to use it so the problem is easier. Technique is also about how you choose to define the problem which means technique comes before algorithm. How can you choose an algorithm if you don't know what your problem is?

Technique breaks down into two parts. Choosing a set of tools and processes, and how you apply those tools or processes. Technique extends past the end product into the support systems that nurture how that end product is created with bug tracking systems, source control management systems, continuous integration, user forums, etc. And, those choices can have a greater effect on the end product than what you put into the product. Just reread the example above for justification.

To some degree, we place too much emphasis on tool choice because how you apply it can undermine the choice of using that tool. If your technique doesn't match the tool the tool will never matter. Have two bug tracking systems because one group doesn't want to give up their existing one. Been there, real story, doesn't work, definitely doing it wrong. (Actually same place as the example probably could have a book of "doing it wrong" ideas from this place). As in the example at some point application of those tools made the choice of SCM moot.

In the end we need to discuss technique more passionately than specific technologies. The two do go hand in hand, but it's the technique in the end that makes the difference. So how do you know you're doing it right? When technique matters more than technology.

Flexjson meet Android

2010-09-23T00:05:00.017-04:00

Flexjson 2.1 now supports running Flexjson on Android. So I thought I'd show a quick example of using Flexjson in an Android application. Hopefully this will spark some ideas about what you can use Flexjson for in your own application. I'm going to start simple creating a quick Android app that pulls recipes from Puppy Recipe, parses it using Flexjson, and displays it in a list. Let's get started.

Recipe Puppy has a very simple REST API, almost too simple, that returns responses in JSON. Recipe puppy allows you to search recipes by the ingredients contained within by using a URL parameter i. Individual ingredients are separated by a comma, and URL encoded. Here is a simple example:

http://www.recipepuppy.com/api/?&i=banana,chicken&p=1

Exciting isn't it? If you click that link you'll see the JSON response. It's a little hard to read like that so here is a simple break down with a little formatting:


{
   "title":"Recipe Puppy",
   "version":0.1,
   "href":"http:\/\/www.recipepuppy.com\/",
   "results":[
      {
         "title":"Chicken Barbados \r\n\r\n",
         "href":"http:\/\/www.kraftfoods.com\/kf\/recipes\/chicken-barbados-53082.aspx",
         "ingredients":"chicken, orange zest, chicken, banana, orange juice, brown sugar, flaked coconut",
         "thumbnail":"http:\/\/img.recipepuppy.com\/602538.jpg"
      },
      ...
   ]
}

This is pretty straight forward. We have a little header and what we really are interested in results property which is an array of recipe objects. So we'll create two simple Java classes to map those data members. RecipeResponse for the header portion, and Recipe which is the object contained within "results" property.

Here are those objects:


public class RecipeResponse {
    public String title;
    public Double version;
    public String href;
    public List<Recipe> results;

    public RecipeResponse() {
    }
}

public class Recipe {

    private String title;
    private String href;
    private String ingredients;
    private String thumbnail;
    private Drawable thumbnailDrawable;

    public Recipe() {
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title.trim();
    }

}

In the Recipe object I actually created a Java Bean with getter/setter, but I didn't include most of those methods. I did make a point to show the setter for the title property. Turns out some of the data coming out of recipe puppy contains extra newlines characters in the title. To get rid of those I'm doing a trim() in the setter. Flexjson is smart enough to call the setter method if you have defined it instead of setting values directly into the instance variables. However, if you use public instance variables it will set values directly into those too. This was a fix made in 2.1 with respect to using public instance variables during deserialization process. You'll be happy to know it works now.

So let's jump to the usage of Flexjson in the android code. So we create a RecipeActivity that contains a List to display the recipes. We're going to look at the AsyncTask that loads the data using Flexjson. Here is the full code for that:


        new AsyncTask=<String, Integer, List<Recipe>>() {

            private final ProgressDialog dialog = new ProgressDialog(RecipeActivity.this);

            @Override
            protected void onPreExecute() {
                dialog.setMessage("Loading Recipes...");
                dialog.show();
            }

            @Override
            protected List<Recipe> doInBackground(String... strings) {
                try {
                    return getRecipe( null, 1, "banana", "chicken" );
                } catch( IOException ex ) {
                    Log.e( RECIPES, ex.getMessage(), ex );
                    return Collections.emptyList();
                }
            }

            @Override
            protected void onPostExecute(List<Recipe> results) {
                if( dialog.isShowing() ) {
                    dialog.dismiss();
                }
                Log.d( RECIPES, "Loading " + results.size() + " Recipes" );
                recipes.setList( results );
                new ThumbnailLoader( recipes ).execute( recipes.toArray( new Recipe[ recipes.size() ]) );
                Log.d( RECIPES, "Loaded " + recipes.size() + " Recipes" );
            }

            protected List<Recipe> getRecipe( String query, int page, String... ingredients ) throws IOException {
                String json = HttpClient.getUrlContent( String.format( "http://www.recipepuppy.com/api/?q=%s&i=%s&p=%d",
                        query != null ? URLEncoder.encode(query) : "",
                        ingredients.length > 0 ? URLEncoder.encode(join(ingredients,",")) : "",
                        page ) );
                RecipeResponse response = new JSONDeserializer<RecipeResponse>().deserialize(json, RecipeResponse.class );
                return response.results;
            }
        }.execute();

The method your probably most interested in is getRecipe(). This method formats the URL we're going to load. It then loads that URL and passes the results returned as a JSON block to the JSONDeserializer. JSONDeserializer will take a JSON formatted String and bind that into a Java object. In this example, we're binding into a RecipeResponse object. Here is how that is done:


RecipeResponse response = new JSONDeserializer<RecipeResponse>().deserialize(json, RecipeResponse.class );

A single line of code does that. The deserialize() method performs the deserialization and binding. The first argument is the JSON String, and the second is the top level class we want to bind into. Notice we didn't have to mention anything about Recipe. Flexjson is smart enough to use the data types from the top level object to figure out any other data types contained within. So if you refer to the RecipeResponse.results instance variable you can see the List data type with a generic type. Flexjson will use generics whenever possible to figure out concrete types to instantiate. Of course polymorphism, interfaces, abstract classes, and the like causes issues with this, but we're not going into that right now. See the Flexjson home page to find out more.

You'll notice the RecipeResponse object is returned fully populated with the JSON data, but we're really only interested in response.results so we just return that. It'd be nice if Recipe Puppy returns how many total pages there were in the header (hint, hint) so that it was more interesting. Anyway it is beta. That array is then added to the ListAdapter and displayed on the screen.

Other things Flexjson could be used for is saving state by serializing objects to JSON, and then deserializing when Activities are reconstituted. This can be easier than writing ContentProviders to dump stuff into the database. One of my biggest gripes with Android is how between pages objects can be reliably sent because Intent's require you break everything down to primitives. With Flexjson we can just simply serialize an object put that in the Intent, and then deserialize it on the other side. So no more boilerplate code to flatten your objects.

Here's a simple example serializing our recipes to the disk:


        File f = app.getFilesDir();
        Writer writer = new BufferedWriter( new FileWriter( new File( f, "recipes.json") ) );
        try {
             new JSONSerializer().deepSerialize(favorites, writer);
             writer.flush();
         } finally {
             writer.close();
         }

Now I know there are people worried about performance, but timing the following code this ran on device in less than 40ms which is within the acceptable bounds for UI performance. If you need more performance you can cache the JSONSerializer/JSONDeserializer instance which optimizes data type mappings so it doesn't recompute those when serializes and deserializes. As always measure, measure, measure.

You've gotten an introduction about how Flexjson can make it easier to work with JSON data with Android.

Caringorm is Architectural Poison

2010-05-09T12:28:00.009-04:00

No one has ever accused me of shying away from sensational titles, and now's not the time to get timid. I must confess I've never been a fan of Caringorm. My first impression of Caringorm was it's over engineered. Why are there so many layers? Isn't that going to just slow you down having to develop a View, Command, Delegate, Service, etc for every round trip I make to the backend? Now that I'm working on a project that has gone horribly wrong I see how Caringorm architecture directly contributed to the problems. We've been called in to straighten out the mess and put down a more suitable architecture. After understanding what the team has done I begin to see the techniques, that Caringorm purports as best practices, create more work for yourself the longer you use them.

Software architecture should organize your work so you can work at a higher level related to your problem you are trying to solve. It does this by fostering reuse in your code. It should allow you to reuse what you did yesterday to apply to today's problem reducing the work required get work done. As your project grows the only way you can move quickly is through reuse. Without reuse the work grows exponentially to the point where value can't be delivered. How long this takes before your code base becomes unproductive? Ten releases? 10 Years? I've seen it happen in 1 release, and less than 1 year.

Key signs this has happened in your project is talks about rewriting your application or major refactoring. Other signs come in when your customer says that should be easy, and then developing it takes significantly more time than you'd expect. Bad architecture robs your team's performance to deliver value. If this goes on too long your project will get scrapped and if you're lucky you'll be allowed to start another project. Most likely you won't because the business will be putting your project in maintenance mode while they spin up the "solution". Good architecture is quite the opposite. Easy things are easy and hard things are possible. At the core of this is the level of reuse in your project.

Caringorm doesn't foster reuse. It stalks it, attacks, leaves it dead, and poisons the earth to keep it from ever fostering. At the heart of this is the age old singleton problem. Singleton's are seriously bad technique, and I wish every developer out there understood this. Using singletons to limit an instance to a single instance is not all together bad, but using it as a locator pattern is where the serious issues began to make your code single use. Unfortunately you can't limit a singleton to the good parts without accepting the very serious downsides, and this is the reasons I try and void them at all costs because the downsides are that damaging. Cargingorm has no problem using the ModelLocator (which is a singleton) in your views (mxml). And there is no amount of other techniques you can introduce to overcome the problems that come with this. I don't care if you're using Code Behind, Presentation Model, or whatever. If you use a singleton in your views you can't reuse them. Anything that directly references singletons becomes single purpose in its use as well, including anything referencing those objects, and so on and so on.

Why is that a problem? Well consider if we wrote DataGrid with the same techniques Caringorm purports as acceptable practices. Let's say DataGrid.dataProvider was hard coded to look in ModelLocator.getInstance().dataProvider. Now how can you have two instances of DataGrid in your program pointing at two different dataProviders? You can't. And this is precisely the problem that leads to serious architecture problems with Caringorm programs. Now throw in calls to getController().eventManager.addEventListener() in your views and you have a serious recipe for disaster.

You might find my example contrived so let me describe a more real world scenario. Say you have a signup process that people fill out on your site and you have a view that represents the information you want to gather. In that view you're using the ModelLocator. Now the customer wants to add a new way to sign up because their doing an email campaign, and they'd like to pre-populate that form from details like the email address and ad campaign number to track it into the view. Unfortunately, ModelLocator makes it difficult to put two different models into your view because it's hard coded to one. What would have been an easy task by instantiating another instance of your view has turned into creating another view from scratch. So let's say you need to do this fast and you copy the view and makes the changes to create two different views. Then the customer wants to add a field to both views. Now you need to update two places in your application. This is precisely what I mean when I say Caringorm creates more work for you. Over time if enough of these exist in your application your productivity will drift to zero because maintaining all of it too much work.

Now hopefully I convinced you that the patterns Caringorm suggests are not helping you. And, you decide to banish ModelLocator from the views. However, the problem of geting something from the model and bound into the view still exists. So what part of the Caringorm architecture will interact with the ModelLocator and the view? Normally this would be the Controller in a traditional MVC pattern. In Caringorm the Command is suppose to be this part, but it doesn't have access to the view. Therefore, how will it set the data properties on the view? You could do some gymnastics by passing references through the FrontController into the Commands, but at this point I'm taking some serious liberties to modify Caringorm's architecture design to make it work. If the architects of Caringorm had realized this then their examples would have shown how to do this.

I'd like to think that as an industry we're working towards a shared understanding about the dangers of singletons, and if that were true I'd expect to see a drop in the number of projects using them. I'd expect to see a reduction in the number of frameworks employing singletons as an instance locator pattern. However, it's quite the opposite. Most developers don't see a singleton and get that tingling sensation that something bad is about to happen. But, singletons have all the same problems that global variables do and by in large most developers realize global variables are poison if not very carefully used.

Caringorm is like EJB of Flex. Over engineered. Expect there to be serious changes to Caringorm in the future to save the marketing that Adobe has done with clients. Just like Sun did with EJB3 for EJB. Sun had over emphasized the benefits (if there were really ever any) to using EJB, and once the community realized EJB was over engineered and more trouble than it was worth. Sun had no choice but to hire Hibernate's creator to design EJB 3.0 and started begging for forgiveness. Adobe will have to do the same.

GreenThread: Problems with Recursive Functions

2010-03-08T22:21:00.014-05:00

In previous blog posts on GreenThreads I mentioned that the downsides of using GreenThreads meant you couldn't write recursive functions. In one of the comments I was asked to expand on this idea, and after the comment got so long I figured a blog article might be a better forum for this topic. I'm going to discuss the issues with regular recursive functions, then we'll explore the differences between two types of recursive functions, and potential changes that could be made to help make it easier to write recursive GreenThreads.

Let's start by examining the following function:


public function factorial( i : int ) : int {
   if( i == 0 ) return 1;
   return i * factorial( i - 1 );
}

It's like the "hello world" of recursive functions. What makes a function recursive is the fact factorial() function calls itself in evaluating the value of the function. If we were to run it in a GreenThread there's no way for the system to interrupt the function calls should this function take longer than the length of a frame. For example, say you ran factorial(5) in the GreenThread. The call stack will look like the following:


factorial(5) -> factorial(4) -> factorial(3) -> factorial(2) -> factorial(1) -> factorial(0)

There's no way to let Flash insert a paint in between factorial(3) calling factorial(2) because factorial(3) calls factorial(2) directly.

The way GreenThread framework works is that it handles repeatedly calling your GreenThread until the time has elapsed for a single frame. At that point it let's Flash have control again and then resumes on the next frame repeating this process until your GreenThread says it's finished. This is actually implemented using a big loop outside your GreenThread.

It gets even harder for recursive functions. Look back at factorial() function. Notice that factorial(5) has to compute factorial(4) before multiplying by 5 so it can return it's value. Therefore, it's not possible break out of the function call, allow Flash to paint, then resume within a function so it can multiply by 5 to finish the computation. (Not unless Flash supported continuations, but that's a whole another topic). So now recursive functions can't be interrupted because the function directly calls itself, and depending on how you write your recursive function it's not possible to put a break because of operations that might come after the recursive call finishes.

There are other issues with recursive functions, but they are not possible to use in a GreenThread because of the dependencies between stack frames. However, there is another type of recursion that can help eliminate dependencies. Let's write our recursive function to remove the dependency on local operations:


public function factorial( i : int, accumlator : int = 1 ) : int {
   if( i == 0 ) return accumlator;
   return factorial( i - 1, i * accumlator );
}

Now notice that factorial(5) doesn't have extra work to run after factorial(4,5) runs like we did before. This means factorial(5) could be replaced by the return from factorial( 4, 5 ). In fact factorial(5) == factorial(4,5)! This technique is called tail recursion, and in certain languages it helps make recursion work without growing the stack frames so large iterations don't overflow the stack. Now Actionscript doesn't benefit from this, but this will allow us to work around the second problem we have. We still have our original problem so we'll have to tackle that before we're done.

Now we still have factorial(5) directly calling factorial( 4, 5 ) so Flash can't interrupt the function calls so it can paint. However, what if we had a special call that would delay calling factorial(4,5) so we could do whatever Flash wanted, then it would resume our recursive function.

Well there exists such a function: callLater(). callLater() can be used to schedule a function to be called in the next frame, and in fact from all of the testing done it's safe to use callLater() as a technique for implementing GreenThreads. However, directly using it will suffer from poor performance because of long waits between function calls. So, let's assume there is a new function in GreenThread that acts like callLater(), but achieves better performance. Now our recursive function could look like:


public function factorial( i : int, accumlator : int = 1 ) : Boolean {
   if( i == 0 ) return false;
   return invokeOnThread( factorial, i - 1, i * accumlator );
}

Now invokeOnThread() doesn't exist in the current code base, but it could be written. Actually the accumlator would probably be best served as an instance variable within your GreenThread, and we'd need to change some more features to fit within the framework. Assuming that it is we could support recursive functions given these constraints:

You must write your recursive calls so they conform to tail recursion.

You must use invokeOnThread() to recursively call your function.

You must conform to the contracts of the GreenThread framework.

The upside is the ability to think recursively. While every algorithm can be expressed either in iteration or recursion it's not easy to convert between each form. Some algorithms are easier to express using recursion and can be very hard to write iteratively. The downside is the requirement to write tail recursive algorithms which can be difficult for the uninitiated, but it's a skill that can be honed.

What if HTML wasn't Top Dog?

2009-09-11T16:17:00.013-04:00

What if HTML wasn't the top node in our web pages? Sounds strange right? But, what if it was just another node inside a larger structure? HTML is great for defining textual documents where you want page flow layout as your choice. However, it really is painful to use a general purpose UI layout language. Which I would argue is the more common practice these days. Even the simplest blogs, forums, or search pages have some form of application layout involved. Why is it so hard with HTML? Lots of that derives from page flow layout and legacy support of this concept. But, if we embed HTML in a larger structure we could do whatever we wanted.

What sucks most about HTML? I would argue it's all the time I waste trying to get the layout I wanted. All the time I spent learning CSS and HTML 4 was probably 6 months or more before I felt comfortable with it. I could come close to the layout I had in my mind. However, as soon as I switched browsers my beautiful layout went to crap, and I had to dig into arcane browser hacks to make it work. Who enjoys that?

How much of that is the complexity of CSS rules and HTML? Ever tried or thought about creating a browser? It's NOT easy. I find understanding the interaction between CSS and HTML arcane as a web designer. If I find it hard then it's really hard for the browser developer to get it right. And that's precisely what we've seen. Lots of inconsistency in how they interpret the meaning of things. Leading to browser inconsistencies. If it's simple to understand then it's simple to implement. If it's simple to implement it's easier for two people to come to a common expectation.

Let's get specific. Say I wanted my node to be position relative to it's parent. I want to set the top left corner of an element to be 50 pixels from the left and 50 pixels from the top. In HTML I can set the left and right of my component, but I also have to set the layout to absolute on the child, and set the parent to relative. This is a common practice in other UI toolkits, but it's complex in HTML. What if all I did was this:


<application>
   <box top="50" left="50" width="200" height="200"></box>
</application>

Pretty simple right? Although this isn't that far from HTML/CSS, there are other things that aren't so easy. What if I wanted to horizontally align that box relative to the parent's center.


<application>
   <box width="800" height="600" horizontalCenter="0"></box>
</application>

Simple. HTML/CSS you would use margin: auto? WTF!? Doesn't horizontalCenter make more sense? Of course it does. Try vertical centering on for size:


<application>
   <box width="800" height="600" verticalCenter="0"></box>
</application>

Try that with HTML and you'll come up short or at best bizarre.

What about defining boxes that grow when the window is changed? That can easy too:


<application>
   <box id="banner" left="0" right="0" top="0" height="50"></box>
   <box id="leftmenubar" left="0" width="250" top="50" bottom="0"></box>
   <box id="content" left="250" right="0" top="50" bottom="0"></box>
</application>

Simple. The content area sets his left and right relative to the parent's edges. When the parent grows so does the child. The banner and leftmenubar are fixed in position. However, the banner grows its width as the parent's width grows.

Even supporting legacy HTML documents could be simple.


<application>
   <box width="800" height="600" horizontalCenter="0" verticalCenter="0">
      <HTML width="100%" height="100%">
      </HTML>
   </box>
</application>

HTML just becomes another possible node within the super document. It would create yet another box that can display text documents using what you want for text documents which is page flow layout. HTML nodes could occur as many times as we need in our over all application.

Furthermore, legacy HTML documents (e.g. those starting with HTML) could be converted into our application tag just by surrounding the application tag around the legacy HTML only document. Hence making all HTML documents forward compatible with application documents.

It's a simple idea to fix the constant layout problems with the web.

On the Importance of being Synchronous: Asynchronous + Actionscript

2009-09-03T23:13:00.011-04:00

Dealing with Asynchronous Events Part-2

Damn Blogspot sucks. What the hell? Why haven't they added the 1st new feature in like 5 years? Trackbacks hello???? WTF? I'm not up for cobbling a solution together with greasemonkey, yada, yada, yada. I need a new blog platform. Enough about that let's get to code.

Anyway I wanted to add my fuel to the fire on asynchronous programming. This is a topic I'm very interested in because Actionscript isn't the only language suffering from this. It's very much rooted in classic Computer Science so it's a deep topic. That blog post is old, but it's still something that doesn't have a satisfactory answer yet. Computer scientists have been discussing this topic since the 1970s in one form or another.

I'm very satisfied with my solution to the single request problem. By that I mean making a single round trip to the server and back. Here is roughly how I do asynchronous calls in actionscript.


var tag : String = "Archive";
var loader : URLLoader = defaultLoader();
loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
   var json : Object = JSON.decode(loader.data);
   var mail : Array = json.map( funcion( json : Object, index : int, arr : Array ) : Email { 
      return new Email( json ) 
   } );
   var event : DynamicEvent = new DynamicEvent('mail.loaded');
   event.mail = mail;
   dispatch( event );
} );
loader.load( session.httpGet( '/home/email/', { tag: tag } ) );

Really tight code, and it doesn't feel like the infrastructure for doing the calls are in the way of understanding what's going on. Now I'm using several factory methods to encapsulate common error handling, host name, authentication tokens, etc. Of course all of this can be overridden, but having defaults keeps that code out of the flow of how you work.

The difficult part comes when you need synchronous flow control over asynchronous calls. This only starts to show up with more than one trip to the server. Say for example, server call 1 must complete before server call 2. You can chain them like so:


var loader : URLLoader = defaultLoader();
loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
   var json1 : Object = JSON.decode(loader.data);

   // do something with json1

   var nextLoader : URLLoader = defaultLoader();
   loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
      var json2 : Object = JSON.decode( nextLoader.data );

      // do something else with json2, and maybe json1

   });
   loader.load( session.httpPost( '/home/update', { arg1: json1.arg1 } ) );
} );
loader.load( session.httpGet( '/home/synchronize/', { hashkey: hashkey } ) );

It's doable, but it's getting messy. And quite frankly a little hard to understand. Is that all we need? If so, then we can stop here and be ok. Sadly, no the rabbit hole can get worse and twisted. Say we want to do server call 1, server call 2, or both based on some conditions! And we want to maintain the order call 1 precedes call 2 if call 1 is done. Kinda of like:


var result : Object = null;
if( someExpression ) {
   result = executeServerCall1();
}

var result2 : Object = null;
if( someOtherExpression ) {
   result2 = executeServerCall2( result.arg1 );
}

Now I want to stop right here and say. Look how easy that was to specify in synchronous code. Jr. programmers can understand that code. All things like conditional logic, control flow, data flow, and more importantly re-usability are all effortless. Just doing simple control flow between asynchronous code is a real challenge.

One thing that I really have trouble with is refactoring logic into a re-usable method that I can call from multiple locations. In synchronous land I can wrap behavior around it doing logic before and after that method. I can easily pass data between in and out. All of these properties lead to reuse and powerful constructs for hiding details. The basis of easy to follow and maintain algorithms.

Adding logic before and after is very hard when the method uses asynchronous calls. I've decided that adding callback objects into the calls it the best route. For example,


public function updateUser( user : User, callback : Function ) : void {
   var loader : URLLoader = defaultLoader();
   loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
      var json : Object = JSON.decode(loader.data);
      var user : User = new User( json );
      callback( user );
   });
   loader.load( session.httpPost( '/user/update/', { id: user.id, email: user.email } ) );
}

I prefer callbacks to using event listener. The main reason for that is event listeners are more long living, i.e. longer than a single method call. If you use event listeners you have to register and unregister between calls, i.e. more mess. If this method is apart of a longer living instance, as I typically do, you could get more than one callback happening. Callbacks are isolated between method calls so they can independent from one another. (There's a lot to discuss here too, but I'll save that for later).

I'm working on my next evolution of this idea to try and build up an architecture to help aid in making multiple round trips to the server in order without adding fuss, and hopefully allowing an outside person to read my code without needing a lobotomy to put my brain in their head. We will have to step away from our friend closure for this to work. But, I want to leave you with this thought.

All of the tools we use today are aided by synchronous control flow. When we remove synchronous flow our tools fall apart. We have very little tools at our disposal to help specify complex flow using asynchronous semantics. Closures are about it, but they aren't enough and fall apart quickly. We need new constructs that aid in asynchronous control flow. Possibly a way to restore synchronous flow, but asynchronous underneath. If we had these constructs we could do this type of work independent from things we typically think of like threads, processes, message passing, etc. Those constructs could be underneath it, but we as programmers would be less involved with their presence.

Agile Methods are Disruptive

2009-05-21T17:28:00.013-04:00

Now I might actually start a fight with that title, but at least it got you here. I recently just finished reading the "Inventor's Dilemma" by Clayton Christensen. It's an amazing book that focuses on the difference between disruptive technologies and sustaining technologies. This distinction is important because it changes how your organization should attempt to develop and manage these types of technologies. Within the book he discusses some of the social forces at play that make developing disruptive technologies different from sustaining ones. I believe these same dynamics are at work with adopting agile development processes.

In order to understand this you have to understand the difference between what is disruptive and what is sustaining. Sustaining technologies are complimentary to existing technologies your customers use. Sustaining technologies will be easily accepted by your existing customer base. That property makes it very easy for you to develop within product line using your existing resources and process. Disruptive technologies are the opposite. They most likely won't be accepted by your customers at first and your organization will find it extremely hard to develop them in house. Christensen's argument is that disruptive technologies only work if they are spun off into a separate organization independent from your own. They must be quarantined away from the core or else your organization will kill them at all costs.

It's a fascinating phenomenon, but at the end of the book he gives hints as to why this is so. He starts talking about the three components that make up any organization: resources, processes, and values. This is where the book starts to sound more like a sociology study on business structure. He points out that resources are portable: people, assets, software, money, etc. They can be fired, hired, moved, procured, sold, bought, etc. They don't care where they are, and they can be applied anywhere you want. This the key difference between the other two because processes and values are NOT portable. It's much harder to move processes and values to between organizations. These are very important properties because without them the organization would disappear. Remember resources move in and out of an organization but it's these processes and values that stay behind and keep it alive. This also means that changing these processes and values is next to impossible. Why? Well because they are what define the organization if you change them then the organization dies, and a new one comes into being.

This got me thinking about agile environments and how they try to affect the later two components. Agile is a process, but mainly targeted at software development. And with it comes a certain set of values you must adopt or else you're going to find it very hard to follow the process. If you don't accept the idea that high levels of communication and collaboration are much better than comprehensive documentation then you'll find agile methods very hostile.

The other day a group of developers were all talking about agile development. Eventually we drifted towards the difficultly we were all having trying to convert an organization into an agile one. Almost all of us felt like it was somewhere between limited success to impossible. It finally hit me. We're trying to do the exact thing Christensen says you can't do. Change a company's process AND values! Not so much a company, but a development team which like a company has processes and values.

We all had anecdotal evidence of a lack of success in doing so. In fact of all the organizations I know that have successfully adopted agile development were green field starts, or they were able to convert everyone in their organization to it all at once. This normally meant small shops or isolated teams. And, in fact my only successful attempt was when I was on a team that was separate from the rest of the development organization that had virtually no dependencies on non-agile groups. My other attempts were very large groups, or groups that had lots of dependencies on other non-agile groups. No surprise those all failed to reap the benefits of agile development.

Why do groups with dependencies fail? It seemed obvious at the time, but I think another idea Christiansen mentions is to blame. And, that is Resource Dependence Theory. In the book Christiansen explains a theory of management that says something like the following. Employees (e.g. CEO, the board, VPs, managers, etc) aren't in control of the decisions in a company. The customers they serve are. I would actually add to this that it's not just the customers, but suppliers, partners, etc. For example, think of the car dealers for GM and how they have crippled GM's ability to cut costs by closing dealerships through the years. The CEO could do nothing to change this until they almost went bankrupt. That's how entrenched customers can make an organization.

This same idea of resource dependence comes into play with agile teams. If you have a lot of dependency between you another team then you will find it increasingly difficult to be agile yourself. Why? Just like company heads aren't in charge of their companies you aren't in charge of your group.

Just like when you develop a new disruptive technology you can't stop at the product development teams. You have to break everything off: sales, marketing, etc. Agile is much the same as it requires you to adopt a new set of processes and values in how you build your products.

Agile processes' values are in conflict with traditional development values. For one, agile can't predict what will be in the release and when it will be done. It can only predict one or the other, but not both. Traditional development thinks it can do both, but really it can't. However, this points out a key difference in values between the two. Traditional shops like both pieces, or they like the idea that they might know both pieces. Traditional development believes you can reliably predict outcomes and plan for long term success. Agile shops believe predictions are unreliable and reject the idea of long term planning in terms of project management. Traditional shops call for loads of documentation and check points. Agile groups reject documentation as wasteful and unproductive in favor of collaboration and high level of communication. These differences in values makes traditional shops find reasons to reject agile, or at best neuter it into submission.

I remember one such conversation that illuminated how ferocious this difference in values can be. We were in a meeting trying to explain agile development practices work. It turned into a huge argument with one of the vice presidents about why agile practices would never work for product development. He fixated on the lack of "robust" agile procedures. He claimed that they might work in the consulting world, but don't apply to product development because product development need more "robust" procedures. His main evidence was this problem of predictability. Agile development could not predict both features and delivery schedule. The VP insisted agile development would not deliver quality software that a product company needs. In many ways it sounded exactly like an existing customer might come after a disruptive technology. Disruptive technologies typically don't have the same level of performance the entrenched technology does at first. The VP was making an argument over quality much in the same way Christensen says existing customers will make over disruptive technologies. Disruptive technologies, at first, don't perform as well, scale as well, or meet the high end needs of the existing customer base. So they usually find their foothold in smaller markets with lower margins. The incumbents are often all to happy to let the disruptive technology provider enter them because they don't make much money from these lower level markets anyway. I think it's interesting how agile development found it's foothold in consulting and small teams first.

If you want to succeed with agile development in your organization don't try and change your existing development process into an agile one. It won't work or if you are able to do it it will be a very frustrating and tiresome process. Better is to start an agile organization. Separate them from other non-agile groups, give them autonomy to affect the processes outside your development organization. Don't see agile as just something your engineers do.

GridGain, GigaSpaces, Windows HPC on EC2

2009-03-12T19:01:00.003-04:00

For those of you interested in grid computing I found an older, but great post about scalability of ec2 for grid based applications. The thing that caught my eye was the final test using Windows HPC and Velocity. The tests were not comparable to each other, but the final test shows how much degradation you suffer when you're data is stored away from your computations. In there tests 31x reduction in performance when your data is stored "out of the cloud". I think this really shows the importance for good redundant storage at the point of computation.

http://highscalability.com/your-cloud-scalable-you-think-it

The good news is for GridGain is the near linear scalability up to 512 nodes in pure CPU tests. Not as high as 2000 nodes for Hadoop, but that's the only real numbers I've seen anywhere on it. Does hint that GridGain's network overhead is really pretty light.

Grid Computing: Intro To GridGain Talk is Online

2009-03-12T14:32:00.005-04:00

I finally got some time to put up the slides, and source code for my talk I gave at the Devnexus conference in Atlanta. Here is the link to the slides, and the source code is here.

Actionscript and Concurrency (III of III)

2009-02-21T01:32:00.017-05:00

In the previous article we covered techniques for breaking up our long running job, but the performance was 40x slower than if we just ran our algorithm straight out. The problem is our algorithm spends very little time doing work, and a lot of time waiting for the next frame. Actionscript's performance is really quite high. We need to increase the time spent running our algorithm and minimize the time we spend doing nothing. We can do that by doing many iterations per frame instead of just one. Using getTimer() we measure how much time we spent looping and back off right before the next frame. Let's look at the code:


public function start() : void {
    Application.application.addEventListener( Event.FRAME_ENTER, onCycle );
}

private function onCycle( event : Event ) : void {
   var cycle : Boolean = true;
   var start : Number = getTimer();
   var milliseconds = 1000 / Application.application.stage.frameRate - DELTA;
   while( cycle && (getTimer() - start) < milliseconds ) {
       cycle = doLongWork();
   }

   if( cycle == false ) {
       Application.application.removeEventListener( Event.FRAME_ENTER, doLongWork );
   }
}
    
public function doLongWork() : Boolean {
    // do some work
    i++;
    return i < total;
}

Now we broken up our algorithm into an extra method. First the start() method which we're already seen. The new method is the onCycle which calculates how long a frame is in milliseconds. The loop continues until either the doLongWork method returns false, or we run out of time. Notice the DELTA constant is some constant that keeps us from eating up the entire frame. We need to give a little breathing room for Flash to drain the queue. Notice how our doLongWork method is just the code pertaining to our job. This makes it's easier to build a general purpose solution that we can reuse.

Green Threads

We can't use true OS threads in Actionscript, but any language can emulate threads. This technique is often called Green Threads. Lots of languages have used this in the past. Threads in Ruby are still green, and early versions of Java were green as well. Now Actionscript can too. I should pause and give credit to Drew Cummins who implemented a version of this for Flash player 10. I've rewritten this to remove the dependency of Flash 10, and changed some of the API so event dispatch is more natural, added easy progress events, and optional progress tracking. Let's see how our Mandelbrot algorithm changes when we use this.

In order to use GreenThreads create a subclass of GreenThread, override run method, and optionally override initialize method to add code that runs at the start. Here is an example:


public class Mandelbrot extends GreenThread {
 private var _bitmap : BitmapData;
 private var _maxIteration : uint = 100;
 private var _realMin : Number = -2.0;
 private var _realMax : Number = 1.0;
 private var _imaginaryMin : Number = -1.0;
 private var _imaginaryMax : Number = 1.0;
 private var _shader : Shader;
 
 private var _realStep : Number;
 private var _imaginaryStep : Number;
 private var screenx : int = 0;
 private var screeny : int = 0;

 override protected function initialize( ) : void {
  _bitmap = new BitmapData( width, height, false, 0x020202 );
  screenx = screeny = 0;
  _realStep = (_realMax - _realMin) / Number(_bitmap.width);
  _imaginaryStep = ( _imaginaryMax - _imaginaryMin ) / Number( _bitmap.height );
 }

 override protected function run():Boolean {
  if( screenx > _bitmap.width ) {
   screenx = 0;
   screeny++;
  }
  if( screeny < _bitmap.height ) {
   var x : Number = screenx * _realStep + _realMin;
   var y : Number = screeny * _imaginaryStep + _imaginaryMin;
   var x0 : Number = x;
   var y0 : Number = y;
   var iteration : int = 0;
   while( x * x + y * y <= (2 * 2) && iteration < _maxIteration ) {
    var xtemp : Number = x * x - y * y + x0;
    y = 2 * x * y + y0;
    x = xtemp;
    iteration = iteration + 1;
   }
   
   if( iteration == _maxIteration ) {
    _bitmap.setPixel( screenx, screeny, 0x000000 );
   } else {
    _bitmap.setPixel( screenx, screeny, shader.lookup( Number(iteration) / Number(maxIteration) ) );
   }
   screenx++;
   return true;
  } else {
   return false;
  }
 }
}

The run() method is the body our of loop. The intialize() method is called once after the user calls the start() method. After that run() method is called repeatedly until it returns false. It's perfectly acceptable to call start() more than once to kick off the thread again after it's finished. That means you can calculate the Mandelbrot set from different zoom levels without needing to recreate new instances. The initialize() method will be called every time start() is called. Check out the results here.

You can also add optional progress tracking by setting maximum, and progress members. This will automatically dispatch ProgressEvents so that your instance can be a source to a ProgressBar. It makes tracking your job easy. GreenThread also subclasses EventDispatcher so you can dispatch events from within the run method.

By in large we've solved the performance problems or we've gotten very close. What's holding us back is the resolution of getTimer(). Since we only have precision of millisecond we really can't run the risk of going smaller than 1 millisecond for our DELTA. That costs us a few iterations on our run() method which can make a difference over 1000 cycles. We could be a full second behind Actionscript that just ran the job straight through. There are a few things we can do to squeeze a little more performance out of GreenThreads.

Frame rate governs everything we do, and by default Flex applications run at 24 frames/s, but really most Flex applications don't do that much animation so if we dropped the frame rate in half to 12 frames/s we would be able to run for longer periods uninterrupted. The fewer interruptions we have, the faster we'll go.

GreenThreads also allows you to configure how much of the frame's time you dedicate to running your thread. By default it's set at 0.99 that roughly leaves 1 ms to update the UI. Under some experimentation this has proven to work quite well without creating lots of timeouts, but if you want to tweak it just provide a new value in the start method like so:


public function go() : void {
   start( 0.5 );
}

If the delta is less than 1 then it means a percentage of the length of a frame. If it's >=1 then it means the number of milliseconds to subtract from the length of a frame. Some more thought needs to go into this so that as you run your application on different machines with different CPUs so the pause is appropriate for the CPU. In the future it might need to be dynamically adjusted as the algorithm runs.

Thread Statistics

GreenThreads supports runtime statistics for tracking your job. To turn on thread statistics pass true to the GreenThread constructor. Thread statistics collects total time the job took, number of timeouts, min and max iteration times, average time a single iteration took, how many cycles it took, etc. There is a fair amount of information that can be gathered to help tune your thread. You can access that information by doing the following:


public class SomeJob extends GreenThread {

   public function SomeJob() {
      super( true );   // turn on debug statisitics

      addEventListener( Event.COMPLETE, function( event : Event ) : void {
         trace( statistics.print() );
      });
   }
}

Conclusion

There are some drawbacks to doing concurrency this way. One is algorithm have to be cooperative, and stop processing in the middle to let Flash do its thing. That means your algorithm normally have to be rewritten to conform with this approach. That can be particularly difficult for recursive algorithms. There needs to be more research done into how you might fix this with the callLater() technique. The biggest draw back is that we cannot take advantage of multi-processors. For all the code you write Flash runs on a single OS thread. This is a serious disadvantage for us going forward because as Actionscript developers we cannot access boosts in hardware performance as cores are added.

It's been a lot of information but hopefully you now understand the theory behind concurrency in Actionscript, and you have a new library that helps you optimize your code. You can access the source code here, and download the GreenThread's library here. I look forward to hearing about what sorts of long running jobs you create.

Full source code of the Mandelbrot set is here.

Download GreenThread's library here.

Actionscript and Concurrency (II of III)

2009-02-19T14:21:00.023-05:00

Now that we understand more about how Flash works internally we can begin to talk about strategies to chop up our large running job into smaller pieces. We'll render the Mandelbrot set. It's a time consuming algorithm, and it's fun when your demos make pretty pictures too. Let's look at a simple implementation.


public function calculate( width : int, height : int ) : void {
 _bitmap = new BitmapData( width, height, false, 0x020202 );

 var realStep : Number = (_realMax - _realMin) / Number(_bitmap.width);
 var imaginaryStep : Number = ( _imaginaryMax - _imaginaryMin ) / Number( _bitmap.height );
 
 for( var screeny : int = 0; screeny < _bitmap.height; screeny++ ) {
  for( var screenx : int = 0; screenx < _bitmap.width; screenx++ ) {
   var x : Number = screenx * realStep + _realMin;
   var y : Number = screeny * imaginaryStep + _imaginaryMin;
   var x0 : Number = x;
   var y0 : Number = y;
   var iteration : int = 0;
   while( x * x + y * y <= (2 * 2) && iteration < _maxIteration ) {
    var xtemp : Number = x * x - y * y + x0;
    y = 2 * x * y + y0;
    x = xtemp;
    iteration = iteration + 1;
   }
   
   if( iteration == _maxIteration ) {
    _bitmap.setPixel( screenx, screeny, 0x000000 );
   } else {
    _bitmap.setPixel( screenx, screeny, shader.lookup( Number(iteration) / Number(maxIteration) ) );
   }
  }
 }
 var evt : Event = new Event( Event.COMPLETE );
 dispatchEvent( evt );
}

Click here to see it in action. Notice how there was a pause before it actually drew the Mandelbrot set. Maybe you got the pinwheel of death or not responding. That is what happens when you hold up the Event Queue.

At a high level this algorithm calculates whether or not a pixel at (screenx,screeny) is inside the Mandelbrot set (i.e. stays below 4). If it stays below 4 after maxIterations of the loop then it colors the pixel black. If not it's color is based on how many times the inner while loop ran before going past 4. It's not as important you understand what the algorithm is doing as so much the parts of the algorithm. The parts that make this a long job are the three loops inside.

Breaking Apart Algorithms

In order to split this job up and run across many frames we'll need to break up those top two loops. Before we jump into that let's talk in a little more general terms. Say we want to bust up a general purpose loop something like:


public function doLongWork( arg1 : String ) : void {
   for( var i : int = 0; i < 1000000; i++ ) {
      // do some work
   }
}

callLater() Technique

We could use the UIComponent.callLater() method to trigger our loop that might look like the following:


public function doLongWork( arg1 : String, i : int, total : int ) : void {
   // do work
   if( i < total ) {
      uicomponent.callLater( doLongWork, [ arg1, i + 1, total ] );
   }
}

This is nice. We've removed the for loop and replaced it with what looks like a recursive call, but it's not. Actually what we're doing is doing a single iteration of the loop, and then scheduling the Event Queue to call us back later to do the next iteration of our loop. We do this until i == total, and that point we stop.

Timer Technique

Another way we could restructure our code is to us a timer. Here is another way to do this:


public function start() : void {
   i = 0; total = 1000000;
   var milliseconds = 1000 / Application.application.stage.frameRate;
   _runner = new Timer( milliseconds );
   _runner.addEventListener( TimerEvent.TIMER, doLongWork );
}
    
public function doLongWork() : void {
   // do some work
   i++;
   if( i >= total ) {
      _runner.stop();
      _runner.removeEventListener( TimerEvent.TIMER, doLongWork );
   }
}

A little more code, but this works too. Now we're scheduling a timer to call us at an interval of a single frame. That's the first line where we calculate in milliseconds the length of a single frame. Then we register our doLongWork method to get called back by the timer. We then remove the listener and stop the timer once i reaches total. Notice that in this method we have to move i and total into instance variables which means we have to initialize those in some sort of start method.

FRAME_ENTER Technique

The final option we could use is FRAME_ENTER event. That looks like this:


public function start() : void {
   i = 0; total = 1000000;
   Application.application.addEventListener( Event.FRAME_ENTER, doLongWork );
}
    
public function doLongWork( event : Event ) : void {
   // do some work
   i++;
   if( i >= total ) {
      Application.application.removeEventListener( Event.FRAME_ENTER, doLongWork );
   }
}

This is nice because we don't have to fiddle with math in order to call us back at the frame rate. We just register a listener, when we're done we unregister our listener. Not as clean as callLater, but this works in both Flash and Flex. What's important to remember is that all of these techniques are equivalent. There is no discernible difference in performance between them.

Restructuring Our Demo

So now we can restructure our Mandelbrot algorithm to match one of these patterns. In our case we'll move screenx, screeny, _realStep, _imaginaryStep, and our BitmapData outside of our calculate() method, and put them inside calculateAsync(). Here's our Mandelbrot algorithm restructured:


public function calculateAsync( width : int, height : int ) : void {
 _bitmap = new BitmapData( width, height, false, 0x020202 );
 screenx = screeny = 0;
 _realStep = (_realMax - _realMin) / Number(_bitmap.width);
 _imaginaryStep = ( _imaginaryMax - _imaginaryMin ) / Number( _bitmap.height );
       Application.application.addEventListener( Event.FRAME_ENTER, calculate );
}

private function calculate( event : Event ) : void {
 if( screenx > _bitmap.width ) {
  screenx = 0;
  screeny++;
 }
 if( screeny < _bitmap.height ) {
  var x : Number = screenx * _realStep + _realMin;
  var y : Number = screeny * _imaginaryStep + _imaginaryMin;
  var x0 : Number = x;
  var y0 : Number = y;
  var iteration : int = 0;
  while( x * x + y * y <= (2 * 2) && iteration < _maxIteration ) {
   var xtemp : Number = x * x - y * y + x0;
   y = 2 * x * y + y0;
   x = xtemp;
   iteration = iteration + 1;
  }
  
  if( iteration == _maxIteration ) {
   _bitmap.setPixel( screenx, screeny, 0x000000 );
  } else {
   _bitmap.setPixel( screenx, screeny, shader.lookup( Number(iteration) / Number(maxIteration) ) );
  }
  screenx++;
 } else {
         Application.application.removeEventListener( Event.FRAME_ENTER, calculate );
        }
}

Now if we ran this version of it you'd see it's 40x slower! Why is that? Well we're executing a single iteration every start of a frame (~40ms). If you timed the running of the calculate method. You'd see that it probably takes less than a millisecond to complete a single iteration. So we've just take a single iteration that ran in <1ms and now we're running it in 40ms! Yikes!

Remember when I said Flash is all about animation? Running jobs at the frame rate is great for animation, but horrible for general purpose concurrency. We need a better way. So our final part we'll discuss optimizing our technique and demonstrate a general purpose solution so that you don't have recreate the wheel when you need to run jobs for long periods of time. See you in the next installment.

Actionscript and Concurrency (Part I of III)

2009-02-19T12:51:00.023-05:00

This is a three part series from a talk I gave at the Atlanta Flex and Flash User Group meeting in February. Original slides are here.

Part I Intro to the Event Queue
Part II Techniques for Long running Jobs
Part III Increasing Performance and a General Purpose Solution

I can hear it now Actionscript can't do that. And some of you might say what is concurrency? Concurrency goes by a lot of names, but it simply means doing two things at once. More accurately stated doing two things simultaneously. Now I can hear what you might say "But, Flash already does that? I mean I can animate two objects on the screen at once." While that's true, Flash is optimized for animation not for general purpose concurrency. In fact the design of animation is so foundational it governs everything that happens in Flash.

Timeline

If you're a Flash developer I'm sure you're familiar with the timeline. The timeline is divided into a series of frames, and Flash executes those frames at a particular rate one after the other. For flex applications the rate is 24 frames / second. If you do the math that means each frame lasts a little more than 40ms. The timeline is very natural for designers. But, for developers this concept is a little strange because it's hard to understand where my code is executed?

Event Queue

For developers I think a thought experiment helps clarify the timeline. Say you were hired to implement the timeline concept. What data structure would help you in doing this task? The answer is surprisingly simple and bears a lot of resemblance to most UI toolkits out there. At the heart of the Flash platform there exists a queue of events. The Event Queue, as it's known, is where all code is triggered. Every piece of code in Flex and Flash is related back to some event being triggered. So when you move your mouse, click a button, type on the keyboard, set timers they all go onto the queue. Flash then pops off those events from the queue and executes them one by one. When one event is done processing the next event is processed. There's no way any two events can be processed at the same time. It all happens one at a time.

Now in Flash even those frames from the timeline are modeled as events. The difference between them and other events is that frame events must occur at certain points in time. Unlike mouse or keyboard which can wait till the next frame to be processed. Frame events must be processed every ~40ms (or 1000 ms / 24).

That means if one event takes too long to process Flash can't update the screen, process other events, or doing anything. What happens is Flash locks up and you get a pinwheel of death or a Not Responding next to your application. In fact Flash punishes such acts and stops processing mouse, keyboard, or button clicks till it catches up. Why? Well that because if you block the queue from processing it won't update that nice animation, and Flash prioritizes that over anything else.

Rule number one is that any code you write must execute within the time span of a frame. If not you run the risk of having Flash coming down on you. This leads us to the second rule which is Flash is not concurrent. There is no way for Flash to update the UI and run your job at the same time.

Timeslicing

So what do you do if you have a long running job? Well the answer lies in how animation works in Flash. Animating several objects at once requires a series of many smaller steps. Say you want to move a ball from one side of the screen to the other. That means you need to move it a little, redraw, move it some more, redraw, etc until the ball is at the other side of the screen. In a way animation is chopping up a long running job into several smaller jobs that run once per frame. We can do that too by chopping up our job into many smaller jobs, and executing them a little at time until we're done!

So how might you do this? Well there are several tricks you can use, and they all perform the same. It's more a matter of taste in which one you choose, but remember technically there is no real difference between these. Our next part we'll look at each of these techniques and discuss unique problems when doing concurrent actions in Flash.

Fun with Fluent Interfaces and Java

2009-01-07T11:33:00.008-05:00

I've written about fluent interfaces before, but I thought I'd share this one I use quite a bit. You never know how much you like something until it's gone. I never thought I really liked Java's InputStream and OutputStream that much until I had to do a lot of streaming work in Actionscript. They have no abstraction for doing stream manipulations. But, let's be honest Java's io first settler's haven't changed much since their introduction. In fact their interfaces have not changed one bit. Sad really because they are so ubiquitous. I find I'm always copying data from one stream the other, dealing with IOExceptions, remembering to close streams, etc. And I got really tired of doing it over and over. What started out as static methods has evolved into a very simple object called ExtendInputStream. Extended as in extending the interface to add more rich functionality rather than the use of inheritance.

The greatest single thing about InputStream and OutputStream is that it's the quintessential example of a decorator. Decorator is one of the foundational software patterns. What I love about decorators is the ability to encapsulate related classes behind a new interface while still retaining interoperability with other decorators.

ExtendedInputStream is a InputStream so it can interact just as plain old InputStream would, but it adds methods like copy, closeQuietly, copyAndClose, and integration with File objects which has always been a pet peeve of mine with InputStream. Let's look at some examples:

Here is copying a file to a directory.


new ExtendedInputStream( someFile ).copyAndClose( dir );

One liner! It's amazing how File object doesn't have these methods already implemented, but then again this approach is much more flexible because we can copy this file to any OutputStream. Here is copying a set of files to a zip.


ZipOutputStream zout = new ZipOutputStream( out );
for( File myfile : files ) {
   ZipEntry entry = new ZipEntry( myfile.getName() );
   zout.putNextEntry( entry );
   new ExtendedInputStream( myFile ).copyAndClose( zout );
}

Five lines of code! Not bad given that 4 of those lines is just to work with ZipOutputStream. Notice how I'm not saving the reference to the ExtendedInputStream here. The copyAndClose() method copies the contents of the file to the OutputStream and closes the InputStream. Closing the OutputStream is your responsibility.

And the more general case of copying an plain old InputStream to any OutputStream.



   URLConnection remote = new URL("...").openConnection();
   new ExtendedInputStream( new URL("...").openStream() ).copyAndClose( remote.openOutputStream() );

Here is a more advanced version. Say we want to pull down a URL and save it to a file on our local filesystem.


   File someDirectory = ...;
   new ExtendedInputStream( new URL("...").openStream() ).name( "SavedUrl.txt" ).copyAndClose( someDirectory );

Here we use the optional method name() to set the name of the stream so when we save something to a directory it will use this name as the filename. You could have just as easily done new File( someDirectory, "SaveUrl.txt" ), but it's not always convenient.

You can use a similar pattern for increasing the buffer size used when copying as well.


   new ExtendedInputStream( new URL("...").openStream() ).bufferSize( 8096 * 2 ).copyAndClose( someDir );

While I have enjoyed writing this simple class I think I've enjoyed using it more so. I really can't start a new Java project without it now. It's a lot of fun to use. I'd be interested in hearing other features people might want to see added.


package com.wrongnotes.util;

import java.io.*;

public class ExtendedInputStream extends InputStream {

    private InputStream delegate;
    private String name;
    private int bufferSize = 8096;

    public ExtendedInputStream( InputStream stream ) {
        this( "no_name_file", stream );
    }

    public ExtendedInputStream(String name, InputStream delegate) {
        this.name = name;
        this.delegate = delegate;
    }

    public ExtendedInputStream( File src ) throws FileNotFoundException {
        name = src.getName();
        delegate = new BufferedInputStream( new FileInputStream( src ) );
    }

    public int read() throws IOException {
        return delegate.read();
    }

    public int read(byte b[]) throws IOException {
        return delegate.read(b);
    }

    public int read(byte b[], int off, int len) throws IOException {
        return delegate.read(b,off,len);
    }

    public long skip(long n) throws IOException {
        return delegate.skip(n);
    }

    public int available() throws IOException {
        return delegate.available();
    }

    public void close() throws IOException {
        delegate.close();
    }

    public synchronized void mark(int readlimit) {
        delegate.mark(readlimit);
    }

    public synchronized void reset() throws IOException {
        delegate.reset();
    }

    public long copy(File dest) throws IOException {
        if( dest.isDirectory() ) {
            dest = new File( dest, name );
        }
        FileOutputStream out = new FileOutputStream(dest);
        try {
            return copy( out );
        } finally {
            out.close();
        }
    }

    public long copy(OutputStream out) throws IOException {
        long total = 0;
        byte[] buffer = new byte[bufferSize];
        int len;
        while ((len = this.read(buffer)) >= 0) {
            out.write(buffer, 0, len);
            total += len;
        }
        out.flush();
        return total;
    }

    public void closeQuietly() {
        try {
            close();
        } catch( IOException ioe ) {
            // ignore
        }
    }

    public void copyAndClose( File file ) throws IOException {
        try {
            copy( file );
        } finally {
            close();
        }
    }

    public void copyAndClose(OutputStream out) throws IOException {
        try {
            copy( out );
        } finally {
            close();
        }
    }

    public ExtendedInputStream bufferSize( int size ) {
        bufferSize = size;
        return this;
    }

    public ExtendedInputStream name( String newName ) {
        name = newName;
        return this;
    }
}

AC/DC Black Ice Tour: Atlanta

2008-12-17T17:13:00.007-05:00

So my ears are still ringing even as I rock out to AC/DC at work. Last night AC/DC rocked Atlanta to a sold out crowd. The set list was the typical set list they've been playing in other cities. The concert started with a mini-movie segment of Angus driving a train, and two very slutty women come into the cab to derail the train. It's chocked full of not-so subtle innuendo and general male humor. Of course I'm not so sure why the women felt it necessary to try and stop the train. They looked like groupies so why they were stopping the train is unexplained. The overt sexual references and general absurdity of the piece had me laughing. Of course it ended in a huge on stage explosion and pyrotechnics display, a huge train smashes through the stage and out pops AC/DC playing their latest "Rock 'N Roll Train". It was an awesome entrance.

Then they followed it up a Bon Scott original "Hell Ain't A Bad Place To Be". While those are some my favorite AC/DC tunes. I really wanted them to play "It's a Long Way To the Top (If you wanna Rock 'N' Roll)", and Rock 'N' Roll Singer which are probably my favorites. Well that's not true I always have trouble choosing my favorites, but I do play those a lot.

Then they broke out "Back in Black" which I thought they played too early. This is their anthem get your damn hands up, but it definitely got the crowd going. It's hard to pick what they could have shuffled around because it's a solid set, but "Back in Black" is just such a powerful song it's got to be deeper in the set list.

Then "Big Jack" off their new album. Followed by "Dirty Deeds Done Dirt Cheap" followed by "Thunderstruck" which was my wife's favorite. I do have a warm spot in my heart for "Thunderstruck". Thank god her favorite wasn't "Dirty Deeds".

Then it was back to the new album for "Black Ice", and at this point I thought I'd have to go to the bathroom. But, I stuck it out, and was treated to another movie with "Warmachine". My favorite part was the parachuting women walking on the tank treads. Hilariously.

Then it was back to the hits with "Anything Goes" and "You Shook Me All Night Long" with a flaming model dancing on the screens. Cue the 5 ladies in the audience to jump up and start dancing. While a lot of AC/DC songs are about women I can tell you there aren't many who listen to it. Then it was into T.N.T. which rocked.

Other highlights of the night were Angus' guitar solo a top an elevated stage, and the gigantic inflatable Rosie that road the train during "Whole Lotta Rosie". Finally it was real cannon fire with "For Those About to Rock We Salute You". There was one long encore ending with "Highway to Hell" finishing the nite. I don't know if my ears could take anymore. It rocked them off.

Rails on IntelliJ

2008-08-09T12:39:00.008-04:00

As I promised I'll give my verdict on Ruby on Rails with IntelliJ. First and foremost I haven't gone back to RadRails/Aptana Studio. I've been using IntelliJ's Ruby plugin now for 2 weeks I guess, and I still really like it. It's stable and very peppy. Much more so than Java editing I hate to say, but there aren't as many features in Rails plugin as main stay Java. Just the editor alone is so much nicer than Eclipse like hitting home goes to the beginning of your code on a line not the start of the first column. And, it doesn't just crash periodically like Eclipse does. Aptana Studio was really flaky. I've only shutdown IntelliJ to reboot my machine for Apple updates so it's very stable.

By in large it covers most of rails and ruby development. You get all the comforts of home with generator scripts. Just right click on your project > New > Controller and viola it runs the Rails generator creating all of the parts you'd expect: controller, it's helper, tests, controller's view and such. You have access to all the same generators you'd expect: Controller, Model, View, Migration, plain old ruby, etc. And it works just like you expect with IntelliJ.

One draw back is undo. It looks like they tried their best to support undo'ing a generator script, but all to often it gives up saying "Undo is too complex" or something like that.

Syntax highlighting is a go. Nothing really unexpected there. Controllers, Models, Views, and even YAML editor.

Code completion. Well it doesn't work that well in Aptana, and it's not too good in IntelliJ. I've gotten to the point where I just don't use it. I think if they were to try and make this work it would have to be some sort of smart guessing going on about what you're doing. But, in the end code completion just isn't going to work for dynamic languages. Oh well.

Refactor is going to be the same verdict as Code completion. It does support move, copy, rename, and migrate. But, these aren't the fire and forgot refactorings you love with Java. Always preview...always. Sorry that's just what you have to give up when working with agile dynamic languages.

Navigating between files is just as you'd expect it with Shift-N and Ctrl-Shift-N (Splat-N and Shift Splat N for mac users). They both work well, and I go back and forth between the class version, and the file version. The class version is going to work for Controllers, Models, Tests, and Migrations. The file version will work for views, and pretty everything else. It's a little weird having these different ways you have to use, but if you're an average IntelliJ user you won't even notice. It also still carries the "Include Java Files" check box when searching in Files. That should probably be removed.

There are two views I usually go between the normal Project view, and a special Rails view. When I do Java I never switched from Project view, but the special Rails view is really quite nice. It pulls all of your code together showing your controllers, model, and tests. Under models you can access your migrations right there which is very nice. It's always a pain to navigate to the DB folder to find your migrations. It also pulls in your public folder for direct access to static resources. But, it's strangely absent of views, and this is probably why I find myself switching back to the project view. You can't see your view files (rhtml,rxml) from the Rails view. This is something I'd really wish they'd fix. It just doesn't even make sense why they left that out.

You also have access to Rake tasks as well. Just right click on your project view > Rake. From there is a bunch of fold out menus to run Rake tasks. It's really nice to see all of the options you have. I'm always finding new tasks I didn't know I got for free. The ability to view all of the Rake tasks is very important given that the Rails geeks don't document very well. But, trying to right click then navigating all of those fold out menus is a futile exercise in mouse dexterity. I'd rather see a Rake view similar to the Ant and Maven docking views. That would be really sweet. Then I can just type portions of my rake tasks and quickly run them with the find feature just like I do with Ant.

I normally use the script/server inside my own terminal to run the rails server, but IntelliJ supports running the server inside the IDE. You have all your choices of Mongrel, WEBrick, and Lighttpd. You can run your server in the various environments, but you have to specify that in the server arguments field. You can save various versions (development, production, etc) as separate programs to run, and just swap between them in the drop down box. You can run Ruby scripts, Ruby tests, Rails server, and RSpec.

The thing I miss is the ability to run the script/console from within the IDE. I think you could make IntelliJ do this as it will run normal Ruby scripts. I tried, but it failed with an exception related to readline. It probably has something to do with the fact that I rebuilt ruby to include readline library, and IntelliJ just can't load it.

I think they could make this experience a little easier by pre-populating your project with Run configurations for Developer, Test, Production, and console. That would be a nice enhancement.

Finally, the coup de grace for RadRails is that the Rails Plugin comes packed with inline template completion (remember sout?). Think TextMate like editing! The best feature of TextMate is it's fast abbreviations for common tasks in Rails. Well IntelliJ users have nothing to covet. Type vc Ctrl-J and you get a validates_confirmation_of. Type rec = redirect_to with controller. Type rf = render file. There are tons of these. Although most are targeted at Ruby, but there are some rails specifics. I'd like to see more migration templates, but the good news is you can add them.

Overall IntelliJ's rails plugin covers everything you need. I'm not switching back to Aptana/RadRails period. I think there are somethings they can improve, but overall it's the same quality you have come to love from Java development with IntelliJ. It's really one of the best Rails environments available to you.

Location of the Ruby SDK and IntelliJ on Mac

2008-07-26T13:16:00.004-04:00

I've been getting back into Rails development again, and I've been trying to find the right IDE to work in. I'm an IntelliJ fan, and I hate Eclipse. But, last time I was using Rails I used RadRails, and it was ok. However, now IntelliJ 7 has support for Rails development, and I thought I'd try it out.

First thing I ran into when I was configuring my project is setting up the Ruby SDK. Much like a Java project needs to know where the JDK home directory lives Ruby projects need to know where Ruby SDK lives. Only problem is on a Mac or Linux box those aren't so simple to find. I also had rebuilt Ruby so I could get readline support and installed it under /usr/local.

I tried /usr/local/lib/ruby that didn't work. I tried /usr/local/lib/ruby/1.8 that didn't work. Finally after stumbling around without any success I tried /usr/local and viola it worked.

If you didn't rebuild Ruby and are using the default Mac install. It would be /usr. Make sense? No not at all, but hopefully this blog post will help some peeps in the future.

Objects as Functions Part II: A lightweight web app validation utility

2008-04-07T11:28:00.010-04:00

Binding and validation is something almost all web frameworks must have a solution for. Some are more elegant than others, but the problem with most of them is that they are tied specifically to the use of that framework. I've yet to see a reusable utility that is framework agnostic for handling this. This is another take on Objects as Functions post I did a while back. Only now I'm applying it to binding and valiation for the web. The results are usable by any Java developer using any framework they want. Creating yet another idea "From Frameworks to Object Oriented Utilities."

I had some code that I had written a while back where I coded the binding and validation by hand. In other words it was just a bunch of if else statement ladders. It resembled something like the following:


setEmail( request.getParameter("email") );

List<String> errors = new ArrayList<String>();

if( isNotSpecified( getEmail() ) ) {
  errors.add("Email is missing.");
} else if( isNotSpecified( confirmEmail ) ) {
  errors.add("Confirm email is missing.");
} else if( validateEmailFormat( getEmail() ) ) {
  errors.add("Email address provided is not a valid.");
} else if( validateEmailFormat( confirmEmail ) ) {
  errors.add("Confirm Email address provided is not a valid.");
} else if( !confirmEmail.equals( getEmail() ) ) {
  errors.add("Email and Confirm Email did not match.");
}

return errors;

Ok so yikes! I just like jumped back 10 years by writing code like that! But, I did it because I was in a situation where the "architect" hadn't really thought about these problems, and hadn't picked a framework that gave us that ability. So most people weren't doing any validation, and very poor binding. Think Vietnam of web apps here.

So after I wrote this code once I knew I needed something better, but it wasn't until I was about to write it again that I decided to go back and try to refactor out the common code into a utility to make my job easier that I came up with a general solution. If you'll notice in that code above there are some handy instance methods I created in this class to help specifying the validation language. So I started pulling those common methods out into a separate class. I'll spare you the details of the refactoring for another blog post. I'll start with some simple examples:


public List<String> validateAndBind( RequestValidater validater ) {
   setFirstName( validater.param("firstName").require().toValue() );
   return validater.getErrors();
}

This first example simply validates that the parameter "firstName" was specified in the request, fetches that value, and binds it into the instance object using a setter method. You'll notice there is no reflection taking place here. I'm a huge fan of reflection, but I think you'll see that this is so easy you actually don't need it. Remember that even in reflective frameworks you have to specify the validation rules, and specifying the binding (i.e. making you call the setter method by hand) isn't really where all the hard work is.

Going into detail on what this does. The first step requests the "firstName" parameter from the validater object. Then it calls the require() method on it. This methods checks to see if the parameter is present if not it adds a default error message. The RequestValidater object keeps track of all the errors it encounters while executing the validation rules. Finally toValue() method returns that parameter's value as a String passing it to the object's setter method. If the value isn't present it simply returns null.

In this simple example, if firstName parameter was missing then it would create an error message like: "First Name is missing". Because the parameter used camel case (i.e. firstName) the validater can infer a display name from that by breaking apart the parameter's name on the capital letter boundaries. So "firstName" would become "First Name". You can override this by supplying a second parameter to the param() method. Like:


  setFirstName( validater.param("firstName", "First name").require().toValue() );

It will also infer using underscores as well (i.e. "first_name" = "First Name"). It's better to accept the default since it's less work, but realize that you can customize it if you so wish. The second way is to supply an actual error message in the require() method as a parameter. While this might be necessary sometimes, particularly with the matches() method, it's usually best to accept the defaults.

Here's another example that validates and binds a date object.


public List<String> validateBindings( RequestValidater validater ) {
   setBirthDate( validater.param("birthDate").require().toDate() );

   return validater.getErrors();
}

The key difference in this example is the call to toDate() rather than toValue(). The to*() methods convert strings into other values like integer, dates, etc. These methods will usually end the validation rule methods. You can also pass a default value into the to*() methods to provide a default date, integer, etc. Of course you wouldn't do that with a require() validation rule provided.

Here is a couple more examples:


public List<String> validateBindings( RequestValidater validater ) {
   setUsername( validater.param("username").require().between( 5, 30).toValue() );
   setEmail( validater.param("email").require().validateEmail().equals( validater.param("confirmEmail") );

   return validater.getErrors();
}

This example we see some more methods for performing validations. The between() method is used to validate a parameter's length is between the two values. If not it adds an error message. You can see in the email example. Two methods validateEmail() which makes sure the value conforms to an email address, and the equals method which tests to see if the value matches some other value. In this example you can see how the validater.param("confirmEmail") can be used again to refer to another parameter in the request.

Finally, there's a matches() method for making sure parameters conform to a regular expression. Here is an example of that:


public List<String> validateAndBind( RequestValidater validater ) {

   Pattern phoneNumber = Pattern.compile("\\(?\\d\\d\\d\\)?(-|\\s)\\d\\d\\d(-|\\s)\\d\\d\\d\\d");

   setPhonNumber( validater
      .param("phoneNumber")
      .require().matches( phoneNumber, "Phone Number provided does not look like a phone number.")
      .toValue() );

   return validater.getErrors();
}

Here's how you can use the RequestValidater in your controllers:


  MyObject obj = new MyObject();
  List<Errors> errors = obj.validateBindings( new RequestValidater( request ) );
  if( errors.isEmpty() ) {
     // no errors means the user's request was valid
  } else {
     // we have some errors so send them back with the form data.
  }

After I finished writing this utility I went back and refactored by code to use it. I had something like 100-150 lines of validation code that I reduced to a simple 8 lines of code. Actually I added some new lines and formatting between some of the chained method calls which inflated it to like 25 lines, but still that's an amazing amount of code reduction. And, that doesn't include the lines I would've written to do validation in the second object.

This is yet another example of how you can use Objects in Java as functions to really change how you can reuse code. Notice that I didn't create some static method utility class to do this because there was state being kept and managed inside RequestValidater for me. If I had a static method I'd have to keep track of the state for me. Even the error messages can be standardized across my entire app by using this class. Also notice that the fact I'm using binding and validating against HttpServletRequest is hidden from my model objects. This is another great example of how encapsulation hides the details of the system from my model objects. Something static utilties can't do for me. Why is this important? Well I didn't go into it, but I also made changes to the RequestValidater such that it's easy to use in unit tests by just instantiating it with a HashMap of parameters. That makes it really easy to automate your validation testing because your model object's aren't bound to the HttpServletRequest interface. Without using encapsulation it wouldn't have been that easy to reuse RequestValidater in a different context. You can see an example in the main method included in the source of how to reuse it in unit tests.

Finally, my last thought on this is that it's a single class. There is no framework you have to adopt to use this code. Java has many choices when it comes to web frameworks. It's both a blessing and a curse, but the reality of the matter is most people are using Struts! Yuck! Why continue to build code that no one else can use? This marks the time when we need to move away from frameworks and to utilities.

Now here's the source:


package com.cci.web.validation;

import javax.servlet.http.HttpServletRequest;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.regex.Pattern;

public class RequestValidater {
    private HttpServletRequest request;
    private Map<String,String> params;
    private List<String> errors = new ArrayList<String>();

    public RequestValidater(HttpServletRequest request) {
        this.request = request;
    }

    public RequestValidater( Map<String,String> params ) {
        this.params = params;
    }

    public boolean hasErrors() {
        return !errors.isEmpty();
    }

    public List<String> getErrors() {
        return errors;
    }

    public Parameter param( String name ) {
        return new Parameter( name );
    }

    public Parameter param( String name, String displayName ) {
        return new Parameter( name, displayName );
    }

    protected String getParameter( String name ) {
        if( request != null ) {
            return request.getParameter(name);
        } else {
            return params.get(name);
        }
    }

    public class Parameter {
        private String displayName;
        private String name;
        private String value;

        public Parameter(String name) {
            this.name = name;
            this.displayName = convertToDisplay( name );
            this.value = getParameter(name);
        }

        public Parameter(String name, String displayName) {
            this.name = name;
            this.displayName = displayName;
            this.value = getParameter(name);
        }

        private String convertToDisplay(String camelCase) {
            StringBuilder builder = new StringBuilder();
            if( !camelCase.contains("_") ) {
                builder.append( Character.toTitleCase( camelCase.charAt(0) ) );
                for( int i = 1; i < camelCase.length(); i++ ) {
                    char next = camelCase.charAt(i);

                    if(Character.isUpperCase( next ) ) {
                        builder.append( ' ' );
                    }
                    builder.append( next );
                }
            } else {
                String[] words = camelCase.split("_");
                for( String word : words ) {
                    builder.append( Character.toUpperCase( word.charAt(0) ) );
                    builder.append( word.subSequence( 1, word.length() ) );
                }
            }
            return builder.toString();
        }

        public Parameter require() {
            return require( displayName + " is missing." );
        }

        public Parameter require(String error ) {
            if( name == null || name.length() < 1 ) {
                errors.add( error );
            }
            return this;
        }

        public Parameter between( int minSize, int maxSize ) {
            return between( minSize, maxSize, displayName + " must be at least " + minSize + " characters, but no more than " + maxSize + " characters.");
        }

        public Parameter between( int minSize, int maxSize, String error ) {
            if( value == null ) return this;

            if( value.length() < minSize || value.length() > maxSize ) {
                errors.add( error );
            }
            return this;
        }

        public Parameter matches( Pattern pattern, String error ) {
            if( value == null ) return this;

            if( !pattern.matcher( value ).matches() ) {
                errors.add( error );
            }
            return this;
        }

        public Parameter validateAsEmail() {
            if( value == null ) return this;

            if( !value.matches("(\\w|\\.)+@\\w+\\.\\w+(\\.\\w+)*") ) {
                errors.add( value + " is not a valid email address.");
            }
            return this;
        }

        public Parameter equals( Parameter param ) {
            return equals( param.value, displayName + " does not match " + param.displayName + "." );
        }

        public Parameter equals( String aValue, String error ) {
            if( value == null ) return this;

            if( !value.equals( aValue ) ) {
                errors.add( error );
            }
            return this;
        }

        public Date toDate() {
            return toDate( "MM/dd/yyyy");
        }

        public Date toDate( String datePattern ) {
            return toDate( datePattern, value + " is not a valid date. (" + datePattern + ")" );
        }

        public Date toDate( String datePattern, String error ) {
            if( value == null ) return null;

            try {
                SimpleDateFormat dateFormat = new SimpleDateFormat( datePattern );
                return dateFormat.parse( value );
            } catch( ParseException pex ) {
                errors.add( error );
                return null;
            }
        }

        public Integer toInt() {
            return toInt( (Integer)null );
        }

        public Integer toInt( Integer defaultVal ) {
            return toInt( displayName + " must be a number without a decimal point.", defaultVal );
        }

        public Integer toInt( String error ) {
            return toInt( error, null );
        }

        public Integer toInt( String error, Integer defaultValue ) {
            if( value == null ) return defaultValue;

            try {
                return Integer.parseInt( value );
            } catch( NumberFormatException nex ) {
                errors.add( error );
                return null;
            }
        }

        public String toValue() {
            return value;
        }

        public String toValue( String defaultValue ) {
            return value != null ? value : defaultValue;
        }
    }

    public static void main(String[] args) {
        Map<String,String> params = new HashMap<String,String>();
        params.put("email", "jep1957@mindspring.com" );
        params.put("email1", "this.email@bad" );
        params.put("email2", "my address@bad.com" );

        validateThese( "jep1957@mindspring.com", "charlie.hubbard@coreconcept.com", "this.email@bad", "my address@bad.com", "bad@bad@bad@bad", "hiya", "foo.bar" );
    }

    private static void validateThese( String... emails ) {
        for( String email : emails ) {
            Map<String,String> params = new HashMap<String,String>();
            params.put("email", email );

            RequestValidater validater = new RequestValidater( params );
            String val = validater.param("email").validateAsEmail().toValue();
            System.out.println( val + " was " + ( validater.hasErrors() ? "not valid!" : "valid" ) );
        }
    }
}

Life without Getters and Setters

2008-02-02T22:37:00.001-05:00

Over at Stephan's Blog he mentioned an idea that I had the other day as well. Back when Java Beans came around the getter and setter notation was a convention to help frameworks determine what the properties of an object were. But, now that we have annotations why, other than legacy reasons, would we continue to use them? Just annotate the method with @Get, @Set, or @Property.

The cool thing is that if legacy frameworks used java.bean.Inspector class, you could encapsulate the new fangled annotation style properties inside it, and old frameworks wouldn't need to know that it changed.

This would get us closer to something I'd like in Java 7. Uniform Access Principle. No more would we need to write getter and setters. It's actually a feature that wouldn't harm the language, and you could turn it off with annotations if you had a need. The cool thing about it is that it's completely compiler driven. No language changes. Maybe if we can't get agreement on closures how about this for Java 7?

Objects as Functions

2008-01-31T15:33:00.001-05:00

I've been wanting to write this post for sometime now. It just takes my so long to write a post, usually because of their length, that I don't do it as frequently as I should. I promised more articles in my last post so here we go.

Over the last couple of years I've been learning other languages particularly Javascript and Ruby. I really like Javascript as a language, API could use some help. I love Ruby's style of meta-programming and the fact that everything is an object. What I'm really jealous is how powerful a single function in these languages can be. For example in javascript it's easy to create widgets with Ext.js:


var button = new Ext.Button( 'okButton', { 
                                text: "OK", 
                                click: function( evt ) {
                                   alert("Hey!");
                                }, 
                                "class": "myCssClass" 
                              } );

In Swing this would be 25 lines of setter methods to customize components (OK maybe I'm exaggerating a little). Or in Ruby active record is just so simple and elegant with things like:


   books = Book.find( :all, :condition => [ "where pubdate > ?", pubDate ], :include => [ :tags, :comments ] )

The ability to hand in optional arguments regardless of position turns simple functions into really expressive mini languages. Dare I say DSLs. I really miss these features when I have to use Java. Since Java opted for the C++ route of data structures. Data structures live in the API, and not the language, it's difficult for us to mimic this type of power.

I remember reading the Javascript Bible several years ago, and in there is a chapter titled "Functions as Data". And, the author is trying to show how functions can be assigned to variables, and passed around the system such one part of the program is unaware of which exact function it's calling. He's trying break the stereotype of functions being just actions, and show you how functions can be data as well. It's a good chapter for any developer.

This got me thinking about stereotypes we have about other languages. In Java we, all too often, get stuck thinking of our objects as largely data with some marginal functions mashed in to manipulate or manage that data. We usually create huge single instance, infinite lifetime objects that then do all the heavy lifting. However, what if we created objects that lived really only for a small period of time just to perform small functions? In fact what if we took what we would normally write as a function and turned it into an object. How would that change what we could express?

Now we need a problem to practice our idea on. Let's say we want to build a simple email utility because I think everyone who has worked on a web application in the past has had to send out email. For example, say we want to send an email from the site when people register to welcome them to the system, or when they forget their password. And, as you know Javamail, while a pretty good general purpose API, is very verbose to use. So let's say we create some utility method for sending email:


public MailHelper {
   public static MimeMessage createEmail( String to, String from, String subject, String template ) {
   }
}

The idea of this method is that we give it the to, from, subject, and a template file to use for the body, and it returns a MimeMessage, which is a Javamail class. It's basically an object representing an email. The template is either a Velocity or Freemarker template. I'm going to use Velocity in this example. If you're not familiar with these libraries they're really simple. They take in text like:


Hi $name,

Welcome to my cool new website.  Thanks for taking the time to register with us.  We're so glad to call you a member.  Here is your password: $password in case you forgot, and here are some great places to get started $interestingPlace.

Thanks

The Man

And replace all of the "$" references with something you assign to them. So in your program you might say "name"="Charlie", and "password"="wouldnt you like to know", and "interestingPlace"="Tahiti". If you've used properties in Ant it's really a similar concept. It's not rocket science.

Now back to our utility. The problem with our utility is that we need a way to set all of those interesting variables that we'll substitute into our template. But, how do we do that with our single static method?

In Javascript or Ruby they might just pass a hash map like:


function createEmail( to, from, subject, template, bindings ) {
   bindings = bindings || {};  // these notation is used for optional arguments.
}

What's cool about the function above is that if you don't have any bindings, just don't include it. Javascript doesn't require you pass all of the arguments to the function so the first line just happily sets bindings to empty object if they aren't specified.

So if we copy this in Java we'd add a HashMap onto our arguments:


public MailHelper {
   public static MimeMessage createEmail( String to, String from, String subject, String template, Map bindings ) {
   }
}

Then our usage would be something like:


public static void main( String args[] ) {

   Map bindings = new HashMap();
   bindings.put("name", "Unsuspecting Luser");
   bindings.put("password", "wouldnt you like to know" );
   bindings.put("interestingPlaces", "Tahiti" );

   MimeMessage message = MailHelper.createEmail( "ycnangerp1@hotmail.com", "kevben@coldmail.ca", "Hello New Member", "email.vm", bindings );
}

Now I don't know about you, but that kinda code drives me nuts. It's so verbose, and
boring! The Javascript and Ruby versions of this are so much more fun. Now what if we wanted to add some attachments, CC some people, or add HTML and text versions of the email? Our poor little static function just can't hang. What we need is something better.

If you're still not convinced that statics aren't up for this consider that I'm leaving out some very important architecture here. Particularly, I need a VelocityEngine instance inside the static function in order to evaluate the template. Now I have to make a choice. One I make a simplifying assumption that I only need one instance of VelcoityEngine and initialize it in the static utility. That would mean I give up on the ability to have multiple template locations, or mail servers, etc. This would make it hard to unit test. It also makes reusing this utility harder because everything is hardcoded structurally. And, singleton's don't help this! The other choice is force the user to manage of the data this static utility needs, but I think that would compromise the ease and usefulness of the utility. Kinda of like well geez if you're going to make me do all of that I might as well just do it myself. Both of them get us into ugly Java.

What if we take our function and turn it into an object? I'm going to approach this from the outside interface, and then show the implementation:


public static void main( String[] args ) {

   Emailer emailer = new Emailer();

   MimeMessage mime = emailer.email("ycnangerp1@hotmail.com", "Hello New Member", "email.vm" )
           .bind( "name", "Bad Spammer" )
           .bind("password", "wouldnt you like to know")
           .bind("interestingPlaces", "Hell" )
           .toMail();
}

Look at that! Much easier to use. Notice how I don't have to create a seperate HashMap. Our new object provides this for us so binding is completely optional. If the email you're sending doesn't have any bindings then just don't call the bind() method. Also notice that emailer instance has a email() method to create an email. It almost reads like English. It also scales nicely for other types of data like attachments, or CC'ing:


public static void main( String[] args ) {

   Emailer emailer = new Emailer();

   MimeMessage mime = emailer.email("ycnangerp1@hotmail.com", "Hello New Member", "email.vm" )
           .bind( "name", "Unsuspecting Luser" )
           .bind("password", "wouldnt you like to know")
           .bind("interestingPlaces", "Tahiti" )
           .cc( "manager@spambot.com" )
           .attach( stockHypeImage )
           .toMail();
}

I'm using email addresses of known spammers to increase the chances spam bots will pick them up and blast emails at them.

If you'll notice I moved one parameter out from the first function: the from parameter. I'll add this back in at a later point in time. You'll also notice the toMail() method. Since we're making multiple calls we'll need a function to end the transaction or kick off the action. I'm going to enhance the version of the code below to also send the email instead of just creating it. I'm not going to implement the toMail() method since sending is the ultimate goal. So now let's look at the code for implementing this kinda class:


public class Emailer {

    private static final Logger logger = Logger.getLogger(Emailer.class);

    private String from;
    private String url;
    private Properties mailProperties;
    private VelocityEngine engine;

    public Emailer( VelocityEngine anEngine, Properties mailProps, String from, String url ) {
        this.engine = anEngine;
        this.mailProperties = mailProps;
        this.from = from;
        this.url = url;
    }

    public Email email( String to, String subject, String mailTemplate ) {
        return new Email( to, subject, mailTemplate );
    }

    public Email email( String to, String subject, String mailTemplate, String htmlTemplate ) {
        return new Email( to, subject, mailTemplate, htmlTemplate );
    }

    public class Email {
        private String to;
        private String subject;
        private String textTemplate;
        private String htmlTemplate;
        private Map params;

        private Email() {
            this.params = new HashMap();
            if( url != null ) {
                this.params.put("website", url);
            }
        }

        public Email(String to, String subject, String textTemplate) {
            this();
            this.to = to;
            this.subject = subject;
            this.textTemplate = textTemplate;
        }

        public Email(String to, String subject, String textTemplate, String htmlTemplate) {
            this();
            this.to = to;
            this.subject = subject;
            this.textTemplate = textTemplate;
            this.htmlTemplate = htmlTemplate;
        }

        public Email bind( String key, Object obj ) {
            params.put( key, obj );
            return this;
        }

        public void send() {
            try {
                Session session = Session.getInstance( mailProperties );
                MimeMessage message = new MimeMessage( session );

                message.setRecipient( MimeMessage.RecipientType.TO, new InternetAddress( to ) );
                message.setFrom( new InternetAddress( from ) );
                message.setSubject( subject );

                if( htmlTemplate == null ) {
                    message.setText( renderTemplate( textTemplate ) );
                } else {
                    String text = renderTemplate( textTemplate );
                    String html = renderTemplate( htmlTemplate );

                    MimeMultipart part = new MimeMultipart("alternative");
                    part.addBodyPart( createBody( text,"text/plain") );
                    part.addBodyPart( createBody( html,"text/html") );

                    message.setContent( part );
                }
                message.saveChanges();

                Transport transport = session.getTransport("smtp");
                transport.connect();
                transport.sendMessage( message, message.getAllRecipients() );
                transport.close();
            } catch( Exception mex ) {
                logger.error( "There was an problem emailing: " + subject + " to: " + to, mex ); 
            }
        }

        private BodyPart createBody(String text, String mimetype) throws MessagingException {
            MimeBodyPart body = new MimeBodyPart();
            body.setContent( text, mimetype );
            return body;
        }

        private String renderTemplate( String template ) throws Exception {
            StringWriter writer = new StringWriter();

            Template tmpl = engine.getTemplate( template );
            VelocityContext context = new VelocityContext( params );
            tmpl.merge( context, writer );
            return writer.toString();
        }
    }
}

The two interesting methods are the Emailer.email() methods. The two versions are for sending text only emails, or multipart HTML emails. At the start it's really quite simple. Invoking email() creates an Email instance which is an inner class for the Emailer. At that point you can call bind() method for every object you want to bind into the template. Notice how bind() returns the Email instance allowing you to chain method calls. Finally once you're done call send() to actually send the email. This was something that you couldn't do with the static method. For one it was too complex to configure through a single method call, and you're static method would have to have access to java properties, velocity engine, etc. By limiting our interaction to statics we force our users to take on more responsibilities. In the static case the user has to use some other utility to send the email. Either we have to force user to manage the architecture by gathering up the velocity engine instance, java mail properties, etc. Or we hard code it with statics in such that our tool will only work for a single velocity engine, from, and java mail properties. Neither is very helpful. One doesn't do enough to make it helpful, and the other limits our architecture and impacts how we can reuse it.

The Emailer class is also split into two parts. The outer class which holds all of the architectural/configuration items (velocity engine, java mail properties, etc). By splitting the code this way those portions can be configured at some place else in the code, say at application initialization, and the parts of the code that need to send email can be blissfully unaware of the architecture used to do so. This outer class acts as the glue between the java mail settings, velocity engine, and simple data like the email address of the author. It's really similar to functional language technique of currying functions.

The second part is the inner class Email. Using an inner classes allows us to create an instance that holds temporary state (i.e. bindings, the template for a specific email, to, subject, etc), but still has access to long living state (i.e. mail host, properties, velocity engine, etc). If we mixed the two inside the Emailer class we couldn't share that across multiple threads. Notice there's no setter methods on Emailer. That means once the Emailer is created other threads can't modify the internal state of it. It's immutable so it's safe to share the Emailer across multiple threads. This pattern allows Emailer to be shared, while Email instances only live on the stack of each Thread. There by separating data being written to (i.e. specifics for sending a single email) from the data that's not changing (i.e. the architecture of sending emails). Safety without synchronizing!

Notice how what started out as our verb function, createEmail(), and turned into a Noun, Email, that acted like a verb. That's usually what happens when you move from static utility methods to active function Object. For lack of a better term.

What's cool about this approach is our Emailer class acts more like a complex function like we saw in Javascript and Ruby. In a single line we easily send an email that requires potentially complex optional data. In addition it allows clients of this class to focus only on sending email, and not worry about the details of how that email is actually sent. Since we're in Java our function Object can tell the user all the options you can call on it with our IDE's code completion feature. Something Ruby and Javascript can't do reliably enough to be useful. With those scripting languages you're relying on the documentation to tell you the specific options of the function.

This pattern in Java looks more like Smalltalk to me. Your functions can end up looking very english-like. By using method chaining and inner classes we can get the same power without having to sacrifice the compact, expressiveness of functions we enjoyed with Javascript and Ruby. Not only did we improve the code's reusability, but we delivered more features by switching from static method to function object. We went from just being able to create the email to actually creating and sending it using objects as functions. In the end, we can now create very powerful single line "functions" just like those scripting languages by using the one construct Java does well. Objects.

Now as I promised da code. This example create a sub directory called templates under your current working directory, and put a test.vm template in there. Copy the template text above, and it will work:


package mail;

import org.apache.log4j.Logger;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.Template;

import javax.mail.*;
import javax.mail.internet.MimeBodyPart;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import javax.mail.internet.InternetAddress;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.io.StringWriter;

public class Emailer {

    private static final Logger logger = Logger.getLogger(Emailer.class);

    private String from;
    private Properties mailProperties;
    private VelocityEngine engine;

    protected Emailer() {
    }

    public Emailer(VelocityEngine engine, Properties mailProperties, String from) {
        this.engine = engine;
        this.mailProperties = mailProperties;
        this.from = from;
    }

    public Email email( String to, String subject, String mailTemplate ) {
        return new Email( to, subject, mailTemplate );
    }

    public Email email( String to, String subject, String mailTemplate, String htmlTemplate ) {
        return new Email( to, subject, mailTemplate, htmlTemplate );
    }

    public class Email {
        private String to;
        private String subject;
        private String textTemplate;
        private String htmlTemplate;
        private Map params;

        private Email() {
            this.params = new HashMap();
        }

        public Email(String to, String subject, String textTemplate) {
            this();
            this.to = to;
            this.subject = subject;
            this.textTemplate = textTemplate;
        }

        public Email(String to, String subject, String textTemplate, String htmlTemplate) {
            this();
            this.to = to;
            this.subject = subject;
            this.textTemplate = textTemplate;
            this.htmlTemplate = htmlTemplate;
        }

        public Email bind( String key, Object obj ) {
            params.put( key, obj );
            return this;
        }

        public void send() {
            try {
                Session session = Session.getInstance( mailProperties );
                MimeMessage message = new MimeMessage( session );

                message.setRecipient( MimeMessage.RecipientType.TO, new InternetAddress( to ) );
                message.setFrom( new InternetAddress( from ) );
                message.setSubject( subject );

                if( htmlTemplate == null ) {
                    message.setText( renderTemplate( textTemplate ) );
                } else {
                    String text = renderTemplate( textTemplate );
                    String html = renderTemplate( htmlTemplate );

                    MimeMultipart part = new MimeMultipart("alternative");
                    part.addBodyPart( createBody( text,"text/plain") );
                    part.addBodyPart( createBody( html,"text/html") );

                    message.setContent( part );
                }
                message.saveChanges();

                Transport transport = session.getTransport("smtp");
                transport.connect();
                transport.sendMessage( message, message.getAllRecipients() );
                transport.close();
            } catch( Exception mex ) {
                logger.error( "There was an problem emailing: " + subject + " to: " + to, mex ); 
            }
        }

        private BodyPart createBody(String text, String mimetype) throws MessagingException {
            MimeBodyPart body = new MimeBodyPart();
            body.setContent( text, mimetype );
            return body;
        }

        private String renderTemplate( String template ) throws Exception {
            StringWriter writer = new StringWriter();

            Template tmpl = engine.getTemplate( template );
            VelocityContext context = new VelocityContext( params );
            tmpl.merge( context, writer );
            return writer.toString();
        }
    }

    public static void main(String[] args) throws Exception {
        Properties p = new Properties();
        p.setProperty("file.resource.loader.path", "./templates");

        VelocityEngine engine = new VelocityEngine( p );
        engine.init();

        String from = "author@yourcompany.com";

        Properties javamailProps = new Properties();
        javamailProps.setProperty("mail.host", "!YOUR_MAIL_HOST");
        javamailProps.setProperty("mail.user", "!YOUR_MAIL_USER");
        javamailProps.setProperty("mail.password", "!YOUR_MAIL_PASSWORD");
        javamailProps.setProperty("mail.port","25");

        Emailer emailer = new Emailer( engine, javamailProps, from );

        emailer.email("person@somedomain.com", "This is a test", "test.vm")
                .bind("name", "Chuck")
                .bind("password","KutzMutz")
                .bind("interestingPlaces","Tahiti")
                .send();
        System.out.println("Email sent!");
    }
}

You don't know Jack about Objects!

2008-01-11T11:46:00.001-05:00

I have been conducting phone interviews over the last few days, and I feel less positive about the computer programmers out there. Every candidate out there has Object Oriented Design on their resume in spite of the fact they can't define polymorphism or worse encapsulation! If you can't define polymorphism or encapsulation without resorting to examples then you can't write Object Oriented Design as one your skills. You don't even know what object oriented is!

The world is full of what I term imperatives. Essentially, it's the style of code you in most imperative languages like Fortran, C/C++, and sometimes Perl (well that's if you actually understand someone else's Perl). Object oriented programming is a total shift in the way you think. It's not just a language, and to effectively use it well, you have to see others using it effectively. And, that's so hard because so much code out there is imperative at its core.

Loads of static utilities classes, singletons, and objects with only getters/setters on it litter the landscape of the mainstream. If you've ever written an entity object with only getter/setters then each entity has a companion manager class voila that's how imperatives roll. What's the big deal? The big deal is you're not getting the benefits of true OO. I recently re-read the white XP book, and in there Kent Beck talks about the cost of change, and while you don't need object orient languages to lower the cost of change. "However, in my experience the cost of change rises more steeply without objects than with objects." This has been my experience too. Object oriented languages keep the cost of change lower than other imperative equivalents. But, only if you use them correctly. Now why would Kent say that?

He's referring to encapsulation, particularly the fact that certain parts of your program should not interact with each other. That's precisely what happens in object oriented programs. What makes changing programs hard, or worse scary, is the dependencies between parts of your program. It's easier to modify your program if your changes aren't going to affect some other unrelated part. Most people think of encapsulation as effective use of access modifiers like public, protected, and private. Access modifiers are the micro-level idea of encapsulation. There exists a macro-level version of encapsulation in how your objects communicate with each other. This macro level is more important than the micro level.

Object oriented languages typically work by calling methods on instances. That means if you don't have a reference to an instance then you can't invoke methods on that object. This fact forms the basis for the macro-level concept of encapsulation. It's all about these references between objects. This macro level encapsulation more commonly referred to as architecture. A great example of this form of encapsulation is the MVC pattern.

How does the MVC model help with maintenance? Well I think it comes from understanding how to make effective reuse of your code. According to the MVC pattern Views should be empty shells waiting to be filled. Void of any business logic. Controllers responsibilities are interacting with the model, choosing which view should be rendered, and binding the model to the view. The interesting part of this pattern is that model objects are used by the view, and the controller. It's this property that makes the model so important because it's the one portion of the system that can be accessed from any other part of the program without increasing your cost of change.

Here is an example that I recently had. I had two Controllers that were serving up thumbnails of movies, and pictures. Interestingly, I needed to stream the bytes of those images over the response. The one thing that's common between these two controllers is that they both use the Thumbnail object. Imperative systems you'll find some sort of ThumbnailUtil.send() method that will share between the Controllers to do this. Ugggh. So there is the duplication and separation that begins to creep into your system when you start programming in an imperative style in an object oriented language. It would be trivial to reuse that logic if it belonged to my Thumbnail object:

public ModelAndView handleRequest( HttpServletRequest request, HttpServletResponse response ) {
Thumbnail thumbnail = thubmbnailService.loadThumbnail( thumbnailId );
thumbnail.send( resposne.getOutputStream() );
return null;
}

It's that simple! For one you're using fewer classes. The send() method is accessible to any controller that can access a thumbnail. Your model objects should have behavior, and this really goes along with any language. I started getting this idea in Java, and I started to really crystallize when I was doing Ruby/Rails. You notice two things. It's easy to reuse logic, and quick to build functionality because after a while you build up a language that models your business logic. Things are now very high level uses of these basic pieces of business logic. This is exactly like Paul Graham's discusses in Programming Bottom-Up.

Another side effect of this is that it's easy to test. In Rails they went great lengths to try and test your model, controller and model, and full integration view, controller, and model. Notice how Rails doesn't try to separate Controller from model which I think is very practical approach. But, the first level of unit testing is testing the model alone. If you're business logic is pushed down into that level it's much easier to test it than if it's inside the controller. You don't need the model and the controller so there's less dependencies you have to initialize and setup in your test environment. The more logic you can test in your model, the more coverage your getting, and the more robust your application will be.

Another example I recently ran into of more static loving imperative code was called ViewUtil. Our application is based on SpringMVC so someone added a ViewUtil class for retrieving the ModelAndView object required by SpringMVC. There was several portions of logic included in there for retrieving XML based views vs. HTML depending on properties in the HTTP request. Here are what this might look like:

public ModelAndView handlRequest( HttpServletRequest request, HttpServletResponse ) {
Model model = modelService.getModel( ... );

ModelAndView mv = ViewUtil.getView( request, "modelview" ).addObject("model", model).addObject("someParam", request.getParam("someParam");
mv.addObject( "model", model );
mv.addObject( "someParam", request.getParameter("someParam");
mv.addObject("someOtherParam", request.getParameter("someOtherParam");
return mv;
}

However, there are other problems that pertain to Views that end up inside our controllers because of the choice of using statics. For example, quite often when errors on our forms occur we have to copy parameters from the request coming in onto the ModelAndView leaving. Particularly, hidden fields on the form that don't belong inside objects. One of the problems with statics is that you can call them from anywhere. Using getView() method really only belongs in the Controller since that's the responsibility of the Controller, and not anywhere else.

Here is a more object oriented approach. One we move this method to retrieve views into our Controllers in a common super class. That way only our controllers can call it.

public class ApplicationController extends AbstractController {

protected View getView( String viewName, HttpServletRequest request, HttpServletResponse response ) {
... // all of our existing logic inside ViewUtil goes here.

ModelAndView mv = new ModelAndView(...);
return new View( request, mv );
}
}

Notice how in this example I'm not actually returning the ModelAndView class as the ViewUtil does. The reason for this is we're going to add our behavior to that object thus giving it the illusion of behavior on our ModelAndView class. So here is an example of our View class:

public class View {
private ModelAndView view;
private HttpServletRequest request;

public View addObject( String key, Object obj ) {
view.addObject( key, obj );
return this;
}

public View copyParams( String... params ) {
for( String key : params ) {
view.addObject( key, request.getParameter( key ) );
}
return this;
}
}

So here we're exposing just two simple methods. One is a delegated method to the addObject() on ModelAndView. Notice that I didn't include getters or setters for the HttpServletRequest or the ModelAndView. That's because they aren't really needed if you have a View object. Whatever sorts of code I need to do that operates on either of those objects I will put in my View object, and simply call that from my Controllers. I could even go further and add a new method to my ApplicationController.doRequest() that returns a View object instead of ModelAndView then that way my subclasses don't need a View.getView() to satisfy the Spring MVC method, and leave that to my ApplicationController.

Now I've created a easier way to handle things like copy parameters from my request to my view, and I don't need yet another static method on ViewUtil or something like that I just have smart View object. This technique is really the same has doing currying in functional languages. Instead of creating a scoped function we're created a scoped object. What's amazing about this technique is it allows one part of the program to transfer knowledge of a section of subsystems without another part of your program having to understand details about that subsystem. It's like a specialized context for another part of the program that's highly tailored to the needs at a point of time for only a short time.

Now on some level you can argue that ModelAndView object should have the copyParams method on it. But, it's not practical to think the needs of every program could be anticipated by our framework developers. It would be too cumbersome and difficult to understand because you'd need to separate out what you need from what you don't.

This illustrates that when you really stop thinking in statics and start thinking in terms of objects opportunities, that would be too cumbersome and repetitive in imperative land, open up in object oriented land. Also notice how I didn't use inheritance to accomplish this. This is the power of composition, and delegation. Inheritance is useful, but most often abused. Composition is an easy technique to learn, but seldom do imperatives ever use it effectively.

Flexjson 1.5 is live!

2007-09-26T22:27:00.001-04:00

After a while I've finally released Flexjson 1.5 to the world. I've been running it for quite sometime trying to see how I liked some of the changes. Since I haven't wanted to change any of my decisions I thought that meant it was time to let other people try it out. There were some big additions to the library. The primary was adding wildcard support for including and excluding fields from your objects. This means you can now exclude one or more fields by using the star notation. The biggest example would be to omit the class attribute on JSON objects. For example:


  new JSONSerializer().exclude("*.class").serialize( obj );

Pretty simple huh? You can even use wildcards more than once so expressions like:


  new JSONSerializer().exclude("foo.*.bar.*").prettyPrint( obj );

Using a plain star ('*') will cause a deep serialization. In previous releases you used the deepSerialize() method to get a deep serialization. You can still use that method. The big thing that changed between releases is the evaluation order of includes and excludes. In prior releases includes were always processed before excludes. So that meant if you did the following:


  new JSONSerializer().exclude("*.class").include("my.hobbies").serialize( obj );

In previous release "my.hobbies" would always be processed before *.class, but now they are processed in the order in which you register them. So in 1.5 the order would be to evaluate each field against the "*.class" exclude then evaluate it against "my.hobbies". This enables you do things like:


  new JSONSerializer().exclude("foo.phoneNumbers").include("*").prettyPrint( obj );

You might notice a new method I'm using...prettyPrint(). Pretty print is fairly straight forward it formats your JSON output in a pretty to read format. Very nice for debugging.

The other big feature is Transformers. Transformers allow you to register classes that participate in serializing fields. It's best understood as described by a use case. Say we have an object that represents an email, and we want to send it to the browser over JSON. But, emails can have illegal characters for HTML like < or > in fields like the to, from, or cc portions. Before we would have to create a separate method that would HTML encode those characters, and exclude the method that returned those values in plain text. Now we can register a transformer for those fields on the fly. Here's how we can solve this problem:


  new JSONSerializer().transform( new HTMLEncoder(), "to", "from", "cc" ).prettyPrint( email );

So the transform() method allows us to register a Transformer called HTMLEncoder. The "to", "from", and "cc" are fields within the email object that we want to run this Transformer on. We can register this Transformer with one or more fields to make it easy on us. The transform() method supports dot notation, just like include and exclude, but doesn't support wildcards.

Transformers can do all sorts of things. Say transform Markdown into HTML from your objects, escape HTML tags and script tags to protect your application, or translating dates into non-numeral format (2007-08-12) or (1/1/2008). Flexjson ships with a HTMLEncoder that you can use out of the box. In the future I hope to add more, especially for security concerns.

This release also has several bug fixes, and performance enhancements. There are some very exciting features in this release. Grab it and see how you like it.

A little Erlang Tutorial

2007-09-21T01:13:00.000-04:00

So I've been very interested in learning a little more about Erlang. Who could ignore all this hype? Well I can't so I decided to check it out for myself. I've been relearning my functional programming so it hasn't been that difficult. It's taking some time to get used to the syntax. And, I miss spell my functions all the time. I really do miss auto-completion. I'm going to go fast so it helps if you've done some of the beginner tutorials that explained tuples, lists, and atoms.

My first function was to create a shopping cart like behavior. This was something that was in the Joe Armstrong book. Although Joe was probably envisioning a web cart. I was thinking more a game inventory system. I did something like the following:


-module(shopping).
-export([total/1]).
-export([cost/1]).

cost(longbow) -> 120;
cost(sword) -> 25;
cost(bow) -> 60;
cost(shield) -> 100;
cost(longsword) -> 250.

total( [ {Item, Count} | T ] ) -> cost(Item) * Count + total(T);
total( [] ) -> 0.

So this defines a couple of functions. The total function would take a list of items and total up the amount of gold it might be worth. Here is the erl command prompt on using this function:


1> c(shopping).
{ok,shopping}
2> shopping:total( [{ sword, 1 }, { shield, 2 }, { bow, 3 }] ).
405

Ok so what's going on? Erlang is very different than most languages in that a complete function is actually a series of smaller functions. Erlang uses pattern matching to figure out which one you meant to call. This is done based on the function name, and parameters you give it. In the total() function there are actually two functions. The first accepts a non-empty list. The second function is matched only when total() is called with an empty list which is worth zero.

There's more going on with the total() function. This syntax: [ {Item, Count} | T } ] is actually doing a lot more than just defining the parameters to total(). It's create three variables (Item, Count, and T). First thing it's doing is removing the first item in the list. In this case it's a tuple, and Item and Count are set to the tuple's two values. The variable T is set to the remainder of the list. Now to calculate the total for one tuple is fairly easy. cost(Item) * Count + total( T ). Calculate the value of this item and then recursively calculate the rest.

Notice that the cost() function really is matching data like swords to values like 25. In Java, Ruby, or Python you would represent this in a map or dictionary like


costs = { 'sword' => 25, 'longbow' => 120 }

~~In Erlang you don't have maps or dictionaries.~~ So how do we relate values together? Converting data to functions serves the same purpose. This really only works for fixed values, Erlang does have a map or dictionary like data structure see the comments for the docs. I'm going to use functions for the same purpose.

I want to shift gears a bit and write a new function along these same lines. Let's say our lonely hero has some gold, and he needs to purchase some supplies. Let's create a function that returns a list of items that he can afford. I'm going to write this function three different ways. Hopefully this will give you some more insight into how Erlang works. So for starters will want to iterate over the list, pull out the items we can afford, and add them to a new list. Sounds easy. Here is my first attempt:


find_possible_purchases( Gold, [{I,C} | Inventory] ) when C > 0 ->
  if
    Gold >= cost(I) -> [{I,C} | find_possible_purchases( Gold, Inventory )];
    Gold < cost(I) -> find_possible_purchases( Gold, Inventory )
  end;

You'd call this like:


1> find_possible_purchases( 50, [ {longbow, 1}, { sword, 2}, {shield, 4}, {bow, 3}, {longsword, 0} ).
[ {sword, 2} ]
2>

In Erlang the period (.) is the end of the statement delimiter. Similar to semicolons in Java or C. Remember to end your statements with a period (.) in the shell.

This isn't too hard to understand. The function starts off by pulling off the head of the inventory list just as our total() function did. It takes the tuple at the head of the list and creates variables I and C for the item and the number of items in the shop. Next it uses the if statement to compare the gold to the cost of that item. If the amount of gold is greater that the cost of that item. It's added to the head of a new list, and recursively processes the rest of inventory. The same code [ {I,C} | find_possible_purchases( Gold, Inventory )] actually is also used for adding items to a list as well. So this creates a tuple with I and C and adds to to the head of the list. The other branch of the if simply recursively calls the find_possible_purchases with the rest of the inventory. This effectively filters out the item our hero can't afford.

The last thing I want to point out is that this function won't even be called if the item is out of stock. By that I mean the number of items (i.e. C) is less than 1. The clause after the function parameters ( when C > 0 ) is a guard condition, and it helps further pattern match your functions. Guard conditions can ease your burden of writing big if or case statements just to filter out certain function calls. You can even specify multiple guard conditions in the when clause. There is one limitation and that's you can' call functions! Let's compile:


1> c(shopping).
./shopping.erl:24: illegal guard expression
./shopping.erl:25: illegal guard expression

Whoa! What happened?! It looks like our if statement is wrong! Well it has to do with what I mentioned before about guards. You can't call functions in guard conditions, and our if is calling our cost() function. If and case statements are specified in a series of guard conditions as well so you cannot use functions in them. Let me show you the corrected version:


find_possible_purchases( Gold, [{I,C} | Inventory] ) when C > 0 ->
  ItemCost = shopping:cost(I),
  if
    Gold >= ItemCost -> [{I,C} | find_possible_purchases( Gold, Inventory )];
    Gold < ItemCost -> find_possible_purchases( Gold, Inventory )
  end;

The solution is to call the function and store it in a variable before we use it in the guard conditions. This is a good example of a function with multiple statements. If you function is made up from multiple statements you use the comma to separate statements. I'm going to round out this function with the following:


find_possible_purchases( Gold, [{_,C} | Inventory] ) when C < 1 ->
  find_possible_purchases( Gold, Inventory );
find_possible_purchases( _, [] ) -> [].

I've also added two additional functions to handle when C < 1, and when there is an empty inventory. In each of these functions we see the underscore (_) used. The underscore is a special character used to represent anything or a wildcard match on this parameter. In these function's we aren't using I so we just tell Erlang to match anything.

All in all this is very verbose. The good news is there are easier ways using Erlang's built in functions, and other features of the language. So let's see if we can simplify.

The lists:filter function is a built in function for working with lists. Let's see how our function would change.


find_possible_purchases( Gold, Inventory ) ->
  Afford = fun( {I,C} ) -> Gold >= shopping:cost(I) andalso C > 0 end,
  lists:filter(Afford,Inventory).

Notice this is the only function we need. Our prior version we needed three functions to handle all of the cases. Using lists:filter() we can get it done to one. Much nicer. The lists:filter() method takes two arguments. The first argument is a function that returns true or false. If this function returns true the item is included in the output, otherwise it's filtered out. The second argument is the list we want to filter. I'm using another feature of Erlang which is an inline or anonymous function. The first line of the function is creating function that takes a single tuple, compares it's cost with our gold, and sees if it's in stock. If both of these conditions are matched it returns true otherwise false. Notice the anonymous function can access variables in our surrounding function like Gold just like closures in other languages. Let's move on.

We are going to simplify this even more using Erlang's List Comprehension feature. List comprehensions allow you to do both filtering and mapping at the same time. We actually only need to filter with them, but you can performing mapping at the same time. Let's take a look at using list comprehensions:


find_possible_purchases( Gold, Inventory ) ->
   [ {Item,Count} || {Item,Count} <- Inventory, Gold >= shopping:cost(Item) andalso Count > 0 ].

Now we've got a single statement. There's a lot going on here. Let's break it down. The || operator separates the mapping operations from the filtering. The stuff on the right is how we filter out items. The first statement pulls out the individual members of Inventory. It creates two variables Item and Count from the tuple. This does the same thing as our [ X | T ] notation. Those variables are used in the statements to the right to filter each member of the list (i.e. after the comma). If this item passes the filter it's passed to the left side of the || operator to map it into the resulting list. Notice we're doing something like we did in the inline function in our second attempt.

Let's compile and see our results.


40> c(shopping).
{ok,shopping}
41> shopping:find_possible_purchases( 150, [{ longbow, 3 }, { sword, 1 }, { bow, 2 }, { shield, 0 }, { longsword, 3 } ] ).
[{longbow,3},{sword,1},{bow,2}]

Alright it works. So there you have it three different ways to write the same function in Erlang. Our first implementation used only a very minimal capabilities of Erlang. It was lots of code, but hopefully it gave you the basis to understand how Erlang expresses flow control. Our next two examples built on that knowledge to simplify those raw skills into something that's easier to use. Hope you enjoyed this little lesson.

Javamail parsing made easy

2007-09-06T14:43:00.002-04:00

I've recently been working quite heavily with Javamail, and while it's fairly straight forward it does have lots of gotchas with little convenience factors built in. Part of it is just the pain of email (spam, lack of standards compliance, microsoft, outlook, bad servers, etc), but some of it is the lack real power in the API. Quite often people have to find the part of the email the user actually wrote so they can display it to the receiver. Say you're writing Java's Outlook killer, and you want to display HTML when it's available and fall back to plain text if the email didn't send it. To make it a little more complex let's also say we want to put the text portion of our email in Lucene so we can easily search for email just like google. In this case the search text would prefer plain text, but would settle for HTML if it had to. Just the opposite of display.

Here is a rough example of code I found out on the net to do this:


    private String findText( Part p ) throws MessagingException, IOException {
        if ( p.isMimeType("text/*") ) {
            String s = (String)p.getContent();
            if( p.isMimeType("text/html") ) {
                displayText = s;
                if( searchText == null ) {
                    searchText = displayText.replaceAll("<.*?>","");
                }
            } else {
                searchText = s;
                if( displayText == null ) {
                    displayText = "<pre>" + searchText + "</pre>";
                }
            }
            return s;
        } else if (p.isMimeType("multipart/*")) {
            Multipart mp = (Multipart)p.getContent();
            for (int i = 0; i < mp.getCount(); i++) {
                String text = initializeText(mp.getBodyPart(i));
                if (text != null)
                    return text;
            }
        }
        return null;
    }

So that sucks. And, yes I wrote it. Well I adapted some code I found off the net, but that only makes it a little less embarrassing. You know what? Don't even look at it. Stop it. It sucks, I know! Why can't you quit staring at it?! It's like a zit on my face. Stop!!!

It sucks because it's trying to do too much. It's trying to do two things in one algorithm. It's trying to find all the parts it's interested in, and it's trying to handle the precedence of HTML overrides plain text. But, that small fact makes it messy because we don't know which we'll encounter first. So, we have to handle both possibilities. It also sucks because sure it finds HTML or text, and it might even work, but it's brittle as all hell. Too much unsafe casting! The first "text/*" mime type will match "text/html" and "text/plain", but it will also match "text/calendar" or "text/xml" which doesn't result in a String when you call getContent(). It won't survive the messy hell that is email. Email is a mess with spammers, bad MTAs, crappy servers that don't reject illegal mail formats, etc. Lots of email uses illegal character sets like "printable-ascii" or "ISO-8859". These charsets aren't included in the JVM, because they aren't real character encodings, but Javamail chokes if it encounters these. So we need to handle the exceptions better.

So that codes sucks, and I know it sucks because I just got done watching it vomit all over a bunch of email I have. Let's simplify. How hard would it be if we just wrote a single algorithm for finding any parts with the given mime types we're interested in. Then we'll make the decision of which one we use. Let's just create a simple method that given a list of mime types it returns all the Part objects that represent those mime types. Something like:


Map mimeTypes = findMimeTypes( part, "text/html", "text/plain" );

That would make it pretty easy. After we have those then we'd just choose HTML over text. Something like the following would be easy to understand:


        if( contentTypes.containsKey( "text/plain" ) ) {
            try {
                Object content = contentTypes.get( "text/plain" ).getContent();
                searchText = content.toString();
                displayText = "<pre>" + searchText + "</pre>";
            } catch( UnsupportedEncodingException ex ) {
                logger.warn( ex );
            }
        }
        if( contentTypes.containsKey( "text/html" ) ) {
            try {
                Object content = contentTypes.get( "text/html" ).getContent();
                displayText = content.toString();
                if( searchText == null ) {
                    searchText = displayText.replaceAll("<.*?>","");
                }
            } catch( UnsupportedEncodingException ex ) {
                logger.warn( ex );
            }
        }
        if( searchText == null && displayText == null ) {
            searchText = displayText = "Unknown content type.  The content of this email was not able to be indexed or read.";
        }

That's pretty easy to understand. First we look for plain text. If we find it then set searchText to that value (i.e. text we want to place into Lucene). Next we surround that plain text with <pre> tags for display purposes. Next we look for HTML. If we find HTML then we use that as our displayText. Then if we didn't find any searchText already (i.e. null) then we'll try to strip out all the tags and use that text as our search text.

We also carefully surround each call to getContent() with exception handling in case someone used strange character encoding. Like all those russian emails I keep getting in my Yahoo account.

The last part says if we didn't find any searchable text and we didn't find any displable text, well that means we don't understand what kinda email this is, and we'll just display a message saying so.

That algorithm is pretty easy to follow. Now let's go deeper into the findMimeTypes() function.


   public Map findMimeTypes( Part p, String... mimeTypes ) {
      Map parts = new HashMap();
      findMimeTypesHelper( p, parts, mimeTypes );
      return parts;
   }

   // a little recursive helper function that actually does all the work.
   public void findMimeTypesHelper( Part p, Map parts, String... mimeTypes ) {
        try {
            if (p.isMimeType("multipart/*") ) {
                Multipart mp = (Multipart)p.getContent();
                for (int i = 0; i < mp.getCount(); i++) {
                    findContentTypesHelper( mp.getBodyPart(i), parts, mimeTypes );
                }
            } else {
                for( String mimeType : mimeTypes ) {
                    if( p.isMimeType( mimeType ) && !parts.containsKey( mimeType ) ) {
                        parts.put( mimeType, p );
                    }
                }
            }
        } catch( UnsupportedEncodingException ex ) {
            logger.warn( p.getContentType(), ex );
        }
   }

    private void findContentTypesHelper( Part p, Map contentTypes, String... mimeTypes ) throws MessagingException, IOException {
        try {
            if (p.isMimeType("multipart/*") ) {
                Multipart mp = (Multipart)p.getContent();
                for (int i = 0; mp != null && i < mp.getCount(); i++) {
                    findContentTypesHelper( mp.getBodyPart(i), contentTypes, mimeTypes );
                }
            } else {
                for( String mimeType : mimeTypes ) {
                    if( p.isMimeType( mimeType ) ) {
                        contentTypes.put( mimeType, p );
                    }
                }
            }
        } catch( UnsupportedEncodingException ex ) {
            logger.warn( p.getContentType(), ex );
        }
    }

Really not a lot of code. Probably easier than the first part huh? All this does is look at the parts mime type and compares it against the multipart mime type. This signifies if we have multiple parts in this email. It then tries to enumerate those emails and recursively calls itself on those parts as well. The next part tries to match that part's mime type against the mime types we're interested in. If we find something we save it off in our map. Notice it only takes the first mime type it encounters of the given mime types. We could look at the Content Disposition and only take Inline HTML and Inline Text, but I've seen problems with the Content Dispositions as well. It's optional so we'd want to generally take the first one we encounter, and that generally works well.

In conclusion I think it's little methods like the findMimeTypes() that make using Javamail so much easier. I wish Sun would add some fun little methods like these when it's building it's specs. Most JCP specs whittle everything down to bare minimum. No frills, no convenience methods, or simplified versions for easy access.

What makes a DSL a DSL?

2007-04-22T11:18:00.001-04:00

I recently released Flexjson to the world, and I've been thrilled by the response I've gotten. But, someone asked why I called Flexjson a Domain Specific Language. It's a good question so I thought I'd write down some ideas around what makes me say something is a DSL vs. when it's just an API.

DSL does get thrown around a lot, and it's hard to always pick out what's a DSL and what's just an API. Hibernate's HQL is definitely a DSL, SQL is a DSL, Phong Shader languages are DSLs, but it gets fuzzier when people say Rails is a DSL or something in LISP. LISP and Ruby are general purpose languages just like Java so how can a library written for a general purpose language be a language too. Isn't that just a well written API? I struggled to answer this question when I first encountered examples of these DSLs.

Martin Fowler wrote some thoughts of this subject and he divided DSLs into two types: external DSL and internal DSL. External DSLs examples are SQL, Hibernate's HQL, ant scripts, or phong shader languages. Internal DSLs would be certain aspects of Rails, or Javascript's Event.Behavior.

External DSLs are glaring examples of DSL. They're easy to spot. Internal DSLs are harder to spot as such. I've decided that Internal DSLs read like a sentence where an API is reads like an algorithm. It's a fuzzy definition, but no less fuzzy than spotting design patterns in code. Said another way DSL should feel more high level, like you're able to produce results in a very simple way.

So what makes an API an API? Well I guess API is not high level. Terrible definition I know, but I think it's pretty accurate. An example of a definite API is JDBC. There's no way to get things done in a simplified way. It's verbose as hell and low level as the day is long. Sorry I get that way about JDBC. ;-) But seriously, it makes me handle all the details. It just gives me access to the database, but I have to manage the connections, the statements, and results sets sure signs I'm dealing with an API.

Flexjson is an internal DSL. It's very simple DSL. There are really only three statements in this language: include, exclude, and serialize. However, while these are just simple methods include and exclude accept strings formatted in dot notation. Dot notation is a simple way of specifying the fields within the object graph. This is probably the part that is most language like. The overall style of Flexjson feels more high level almost language like. So we get something like:


new JSONSerializer().include("bids", "winner.winningBids").serialize(auction);

Let's say I had designed Flexjson like following:


JSONSerializer serializer = new JSONSerializer();
serializer.include( auction.getClass().getField("bids") );
serializer.include( auction.getClass().getField("winner") );
serializer.include( auction.getClass().getField("winner").getDeclaringClass().getField() );
serializer.serialize( auction );

I would say this is starting to look more like an API than a DSL. It doesn't really help me in specifying what I want to happen. It just does the serialization to JSON process. And not as easy to work with which means it's not fun either.

So while Flexjson purpose is to serialize Java objects to JSON. It also tackles the effort of walking the graph for you. Walking the graph is not central to it's purpose, but it's something you have to do as a client. So, Flexjson has a little language that let's you just say in a high-level way what you want to happen. This is what makes your job easier.

The difference between DSL and API sometimes boils down to just style. Java has had a tradition in writing APIs. Some better than others. However, there are a lot of good programmers out there doing things differently, and I think that's a very good thing. I've heard Rubynauts say that you can't create DSLs with Java because it doesn't support all the cool stuff Ruby has. And all that stuff is important to writing DSLs. I happen to disagree with that very much. Java is just as suitable for writing DSLs as any other general purpose language. You just have to work with what you got. I hope to add more on this subject later. DSL vs. API is root in a change in style, but it allows us to do things we couldn't do with our APIs.

It's really about reaching consensus if something is a DSL or not because these things are fuzzy. We're using these APIs to write mini-languages, and exposing our language as APIs so it can be confusing. But, just because it takes the form of an API doesn't mean it's an API. There are lot of benefits to this my favorite is code completion and use of our existing tools. Flexjson I think qualifies as a DSL, albeit a simple one.

Annoucing: Flexjson for Java 1.5

2007-04-18T14:11:00.000-04:00

I recently created a new library for sending Java objects over JSON. It's called Flexjson. Flexjson allows you to control the depth to which you serialize your objects. If you're using libraries like Hibernate for your objects you'll know that many libraries will try to serialize the entire object graph as JSON. This is be a serious performance bottle neck when your object form a connected graph as Hibernate requires. Other libraries force you to manually copy over fields into a seperate object thus creating lots of boiler plate code. Flexjson tries to give you a simple DSL implemented as a normal Java API in order to control with a minimal amount of code what gets translated to JSON.

Check it out: http://flexjson.sourceforge.net

Stopping the memory leaks with Prototype

2007-03-28T15:26:00.000-04:00

Lately, I've been trying to fix the memory leaks in my application. I found this cool little tool for Firefox: Leak Monitor. It's pretty nice at showing you when you have a leak, but it could be improved a lot more by showing you more details about the leaks. Right now it only tells you the creation point of the object that was leaked. Which if you're using a javascript library like Prototype then you get a lot of references to code you didn't write. It would be more helpful if it could print out the complete stack trace that caused the construction. And it would be nice if you could turn the plugin on and off for given pages. Right now it alerts you to every leak which is interesting to see what pages you visit the most cause your browser to leak *cough* Digg, gmail. But, ultimately it's annoying as all hell after a few visits. Bottom line it's still a nice tool.

What I found in my code was lots of problems using bind() and bindAsEventListener(). Massive amounts of memory being wasted by yours truly because I didn't clean up my event handlers. But, I thought Prototype handled this for me judge! Honest! I started reading the new prototypejs.org site for any hint of documenation mentioning cleaning up references. Prototype will install a listener on the unload event and cleanup all listeners registered through Event.observe(). But, reading the fine print it only does this if you're in IE! But, I'm seeing tons of memory leaked in Firefox. Why not do it for Firefox as well? It's all the same javascript, and if I have to go around and manage all of the listeners myself what's the point of having IE then do it for me? Only the Prototype developers know why. But, here is what I did to fix my problems:


Event.observe( document, 'unload', Event.unloadCache );

Put that in my javascript and viola I went from over 70-80 leaked objects down to 2. I'm so smart...hey wait a minute two! So I checked my leak monitor tool, and it's coming from the bind() method in prototype. Well I'm a heavy user of the bind() method so it could be anywhere. At first I thought maybe it was Behaviour so I removed Behaviour, but nope still 2 being leaked.

Fingers drumming, mouth frowning, what in the heck could it be?! I played around and noticed sometimes I leaked 1 object. Other times I leaked 4. Hmmm. Then I remembered something that I read on Jack Slocum's blog 3 Easy steps to avoid javascript memory leaks. Number 3 relates to using Ajax calls. So I commented out my Ajax call I was doing on page load and viola no leaks! Looks like my use of bind() method there was causing trouble. So now I know what it is, but what's is Prototype not doing?

After hunting through the Prototype source I see where it's setting the onreadystatechange to null. I've been working on it for a couple of days, and I still can't figure out why it's leaking. This is where it's unclear why my object is being leaked. Comparing this with how you look for leaks in Java this is a real pain. In order to implement garbage collection you have to have what's called a root set. This is a set of objects that is considered always live like global variables live forever, or the client side dom. The garbage collection algorithm starts recursively walking object references from the root set marking the objects as reachable. Once it's finished visiting all the reachable objects it can throw away the objects that aren't reachable. So essentially anything that is reachable from the root set will survive garbage collection. To clean up objects simply cut all the references from the live set. That means for a given object we ought to be able to see what objects have references to it, and what objects have reference to those, etc. Now I know that because IE uses ref counting it can leak objects not reachable from the root set, but every other browser out there implements a real GC algorithm. And remember I'm talking about tools that run in Firefox mainly.

Leak monitor just tells you if you're leaking memory, it doesn't tell you why. What we need is a tool that traces the objects back to the root set so it can show us where it's being referenced accidentally. Then we can cut that reference and move on. A good tool like this would make leak removal makes this process easy for anyone to do. Right now it's much more of a hunt and guess that takes forever.

Well I guess it's back to hunting.