The Metaverse is stoopid, and it will fail just like Facebook has.

So Facebook has died, and long live Meta which I think marks a huge historical milestone for humanity. The day social media died. Facebook is now going to try to morph itself into "The Metaverse" which is some sort of 3d virtual playground where nobody hates Mark Zuckerberg anymore? Honestly I can't make heads or tails of what the Metaverse is supposed to actually be beyond what has already come before as a persistent virtual reality social game. Just pick one and that's the "Metaverse": Second Life, Roblox, Minecraft, every MMOG game invited in the 21st century, practically. So why is Meta going to revolutionize this space? Because they have lots of money and an image to save? What furtile ground is actually left with this idea? And do you really want to live in a virtual reality based internet?

The "Metaverse" is basically trying to deliver on a promise as old as the internet. Most people cite Neil Stephenson's "Snowcrash" as the genesis of this idea, but it's probably only the most popular one. There had been attempts before to create it or something that could be possibly considered it. But, persistent social based VR based worlds or communities have been built before: most noteable Second Life. In fact I'd argue Second Life came closest to this idea because it wasn't trying to be anything beyond a excessively flexible virtual world. It also had the vision of being literally a 2nd life where you designed a new image of yourself within it. The economy of Second Life was larger than the GDP of Russia. I think it was the 27th largest economy of the world. It was quite popular for a time, but eventually it faded from the zeitgeist of the internet. While it failed, it did approximate this Metaverse vision. No matter how crude Second Life was it ultimately failed as an idea and that could be the precursor to fate waiting for "The Metaverse". I have serious doubts that a Facebook powered "Metaverse" is going to be radically different and transformative to make the concept stick.

I bet most Metaverse proponents would disagree with my comparison, and bristle at the fact that their project can be boiled down to a 4-5 year period in the mid 2000s that has been long forgotten. But there were many other attempts to bring about the "Metaverse" or technology that would support it. VRML was a promising idea that could democratize 3D worlds using a standards based approach, but it flamed out before it could even get going. Most promponents chalked up that failure to lacking implementations and compabilities, but there has been no real attempt to reboot that failure either. Mostly because the concept of the web as a 3D world was just not what people wanted. While we have better platforms like Unity or Unreal I don't think the crudeness of these inventions were really the problem.

Sure Mark Zukerberg and John Carmack can debate over whether the technology is ready or not, or if new technology will somehow improve the experience. However, the technology isn't really that much better than it was 10 or even 20 years ago. Maybe we have nicer graphics or higher resolutions, or even cheap VR headsets, but it hasn't really made VR more desirable. It's only really found a following inside a small pocket of gamers. Most of the time 2D monitors are the preferred method of expierence for gamers. And that really points to the fact of 3D expierences, which unless you are trying to build immersive gaming type worlds, most of us would rather just use 2D. We don't need to consume our instagrams and TikToks in 3D to get the point. Never have. That's why these 3D world platforms never last. VR interfaces are the video phone from the 1950s. Sounds really cool and sci-fi, but not a practical use case.

For a world that just spent roughly 18 months in doors and isolated with our only social outlet is zoom happy hours and conducting business over zoom calls I think what most people want right now is less computer time and more face time in the real life or what ever is left of it. That's what makes this pathetic attempt to revitalize a company's image by showcasing some bright new vision of the future as a transparent attempt to distract rather than deliver. It feels like the "Metaverse" is just some sort of fantasy escapism for Zuckerberg to run away from the problems he created because of his belief in excessively permissive 1st ammendment rights.

Modern social media is almost completely unregulated or moderated. Sure Facebook has standards of conduct and employs armies of people to moderate Facebok and Instagram, but it falls way short of effectively policing it. It took the almost overthrowing of the US government to force it to deplatform a President which shows every other bad actor on Facebook, Twitter, etc where the lines is which is very far away for almost all of them. These problems haven't gone away with a new platform and besides what army will moderate the "Metaverse"? Misinformation and hate speech is abound on the platform every minute of the day, and whatever Facebook is doing hasn't even come close to being a solution to the problem. Bot armies reign supreme ready to defame and spew nonsense 24-7 8 days a week. And it's largely those unchecked forces that spoiled Facebook and Twitter in the first place. Cyberbullying at scale runs rampant smear campaigns by a vocal minority and your neighbor believes in all sorts of pyschotic conspiracy theories that you are a liberal nazi who must be assinated.

The problem is that while Facebook is home for the cesspool of humanity so will it be in the "Metaverse". While we shake our finger at Mark Zuckerberg and his abomination of a creation he is only partly to blame because we're the ones at the end of the like and reshare button. And we'll be the exact same customer occupying "the Metaverse" if that becomes a thing. We're the ones that believe rumors, innuendos, and out right bat shit crazy conspiracy theories than pander to human's worst inner believes and fears. We are the ones that like and reshare content from the cesspool afraid to look away from the computer for fear we might miss something. In a way we are the ones that destroyed social media just as much as Zuckerberg or Jack Dorsey. The only real sin they committed was continuing to hold fast to their "no censorship" is a laudable goal. And that letting us "be our true selves" was a good idea. It wasn't. We're horrible people. In a way if the world was an ideal place inhabitted by rational and logical people you quite possibly could hold very open permissive 1st ammendment beliefs, but reality is we aren't rational nor ideal. And we don't deserve unfettered 1st amendment rights. We are quite easily manipulated by the dumbest schemes. Those faults require that public spaces censor out the ones that will pollute it, see also Paradox of Tolerance. And that hasn't really changed by peddling "The Metaverse".

This is all happening at just the time when globally we are waking up to how horrible we've been, and how awful Mark and other technofiles have been mismanaging their platforms. Now Mark Zuckerberg is trying to save what little he has left of his companies' credibility. It reminds me of the early 2000s when another company's image was falling apart because of rampant security issues. Microsoft. After Windows XP an inordinate storm of security vulnerabilities were threatening Micrsoft, and they couldn't patch it fast enough. The DOJ had slapped Microsoft down for violating Java's EULA with its "embrace and extend" philosophy, and basically was the all around 1 million lb. bad guy of tech. Then comes the promises of Vista. A new OS done right with a new language at its heart that would fix all of these pesky security issues in one fell swoop, and deliver us into a new world of discoverable services, the .NET platform, and nirvana. SQL database reimagined, hard disks and file systems were going to be "rethought" for "today's computing". Meanwhile the world moved on from Microsoft dominance, and old MS totally and completely missed the rise of web 2.0 and modern browsers. We began to break way from Internet Explorer 6.0 and softare as a service really started to become a reality like never before. Apple introduced it's OS 10, and people started to filter out into other operating systems away from the intel MS hegemony. Vista turned out to be a major failure, and it took a decade for MS to realize their mistake. And while that was happening they were moved out of the way. Vista distracted Microsoft and completely enveloped them into a world unto their own making. And the rest of computing just drove right past that exit to a happier and more exiciting industry.

I think "The Metaverse" will be Facebook's Vista. A new vision created when a company has reached a critical failure in its product. "The Metaverse" will be a project unto itself that Facebook will dedicate untold effort towards. A project that will become so big and all consuming Facebook will struggle to explain what it's doing and why. It will become the project that no one actually wants and will end up leading them to take their eye off the ball. Just like Microsoft did. They'll become less important in the fabric of the web economy, and will fade. They won't dissappear unless governments regulate them out of existence, but they will become less important in our lives. After all there still is a Myspace floating out there, and Facebook has enough gravity to float around too. A little more relevant than Myspace, but less relevant than now.

There are definite limits to any institutions we create even software based ones. We can never really perfect them or continue to build them ever larger. At some point the Tower of Babel always falls apart. Will Facebook have a 2nd act? Is Meta that second act or just the rebound girlfriend/boyfriend? I'm betting on the later.

Mysql and the Groovysh

I wanted to use mysql in groovy shell, and I found it a bit trickier than usual. Also there is literally nothing covering how to do this on the internet. First thing is to configure your graphConfig.xml file. Edit your grapeConfig.xml file (for *nix ~/.groovy/grapeConfig.xml). Then drop your config in there: I had to update mine because I guess the central repo changed to requiring https now. Fire up the groovy shell. Then execute this command:

groovy:000> :grab 'mysql:mysql-connector-java:8.0.18'

Now here is where things get a little weird:

groovy:000> sql = Sql.newInstance('jdbc:mysql://localhost:3306/someDB', 
    'person', 
    'none of your business!')
ERROR java.sql.SQLException:
No suitable driver found for jdbc:mysql://localhost:3306/someDB

So that sucks. Unfortunately the driver isn't known to the DriverManager so no workie. Too close for missiles, switching to guns for this one:

groovy:000> sql = Sql.newInstance('jdbc:mysql://localhost:3306/someDB', 
    'person', 
    'none of your business!', 
    "com.mysql.cj.jdbc.Driver")

Bam! Now we're ready to start slamming some SQL in groovysh (woot).

Jenkins, S3 Copy Artifact, Deploy Plugin, and ROOT Context Tricks

I've spent several frustrating days trying to get Jenkins to deploy my application remotely to my Tomcat 7.x app server. My build setup is pretty vanilla Java application built with Ant and Ivy. Overall I like Jenkins and I'm trying to learn how to use it for continuous deployment, but the lack of documentation explaining some of the plugins makes it extremely frustrating. This hopefully will explain some of the subtle configuration options better for these plugins. For this I've setup Jenkins with these plugins:

• GIT Client plugin
• GitBucket
• S3 Plugin

I have two jobs. One builds my application which uses a post build step to Publish artifacts to S3 bucket. The second job is used to remotely deploy the artifacts from the first job to the Tomcat 7.x server.

The 2nd job is a parameterized build with the following configuration:

Build selector for Copy Artifact
name = BUILD_SELECTOR

Execute Shell
Command = rm -rf $WORKSPACE/build

Copy S3 Artifact
Project Name = MyApp
Which Build = Specified by Build Parameter
Parameter Name = BUILD_SELECTOR
Artifact Copy = webappname-*.war
Target Directory = $WORKSPACE/build

Few things to note. BUILD_SELECTOR is the name of the environment variable that holds the user's selected build. The artifact to copy setting is not a path it's just a pattern used to select the artifact. I execute the rm command to clean up the artifacts between successive builds.

The first problem I encountered was the 2nd job kept failing because it said there were not any artifacts from the 1st job. It is NOT documented anywhere that I could find, but once I went back to the 1st Job and marked the "Publish artifacts to S3 bucket" step as "Manage Artifacts". Once that was checked it finally recognized the artifacts and I got the following!

Copied 1 artifact from "MyApp" build number 301 stored in S3

But the next problem was the deploy plugin kept failing with a very obtuse error.

java.io.IOException: Error writing request body to server

I found out that if I removed my application from the Tomcat 7.x webapps directory then it would actually deploy! But if I tried to redeploy it it failed with that obtuse error. I was deploying my app to Tomcat's ROOT context so my configuration looked like this:

WAR/EAR = build/fuseanalytics-*.war
Context = ROOT
Container = Tomcat7x
Manager User Name = none of your business
Password = also none of your business
Tomcat URL = http://somehost

So the clue was the following logging written out in the console output: "is not deployed."

Copied 1 artifact from "MyApp" build number 301 stored in S3
Deploying /var/lib/jenkins/workspace/MyApp/build/myapp-1.0-301.war to container Tomcat 7.x Remote
  [/var/lib/jenkins/workspace/DeployMyApp/build/myapp-1.0-301.war] is not deployed. Doing a fresh deployment.

So after looking through the Cargo code (what the deploy plugin is based on) I found out that Cargo has to use redeploy or undeploy if a webapp is already deployed. So why is it not working? It turns out Cargo cannot handle the Context of ROOT. The tomcat manager data doesn't specify the webapp deployed on ROOT! The work around is to change ROOT to '/' (without quotes) in Jenkins and viola! It works!

Simplest Explanation of Ivy Configurations

If you are here then you are probably trying to understand IVY's configuration concept. And quite frankly after getting comfortable with it I still can't understand their docs. They are freaking obtuse. Going on Stackoverflow proves frustrating as well. I'm going to try and explain this in a really straight forward example. One because the information out there isn't good, and two so people can throw stones at my explanation and improve my understanding. Here goes.

What does Ivy do?

Ivy downloads dependencies and put them into directories your Ant script will use when compiling, packaging, etc. The important part of that is Ivy downloads dependencies and organizes them. It's up to your Ant script to use them appropriately.
An ivy-module (ie ivy.xml file) has two main parts:

• What dependencies do you need?
• How do you want them organized?

The first part is configured under the <dependencies> element. The 2nd is controlled by the <configurations> element. For example:
This is fairly straightforward. We have three dependencies. If we have an Ant build file configured for Ivy this will download all three of these jar files and put them into the ${ivy.lib.dir}/compile/jar directory. That's great, but we when we go to package our application some of these aren't needed. For one we don't care to ship junit with our application so can we segment that out?
You could do this with filesets and excludes in Ant but that is tedious and error prone. Ivy will do this for you if you know how to ask to ask it. Ivy will put the dependencies in different directories based on if that dependency is needed for testing, compilation, or runtime. This is where configurations start to matter. So let's change what we have such that we divide up our dependencies using configurations. Let's create a 'test' configuration for this purpose.
Ok that was easy right? Well if you run this you'll find two directories under ${ivy.lib.dir}:

• ${ivy.lib.dir}/compile
• ${ivy.lib.dir}/test

However, all three dependencies in test, and the other two will be in compile! Doh! That's not what we wanted so what happened?! This comes from the fact that if you don't specify a conf attribute on each dependency it defaults to "*". Well sort of it's a bit more complicated, but you can think of it like match all configs. And because that dependency matches all configs mysql and log4j was copied to both test and compile directories. So let's fix that.
Alright now everything should be as we expect! But it's annoying to have to specify conf="compile" every time we add an dependency. This is where defaults come into play. Remember I said conf attribute defaults to "*" when nothing is specified? Well we can override that by setting the defaultconf on the dependencies tag.
Alright! Now we can just add dependencies and they will always be added to the compile configuration by default! Much easier.

Transitive Dependencies

Now there are some complexities about Ivy that I shielded you from thus far. And it has to do with the decisions Ivy has to make while trying to resolve dependencies. See when you declare you depend on A well A might also depend on B and C. Therefore you depend on not just A, but A, B, and C. B and C are called transitive dependencies. These are hidden from you because using Maven's POM files Maven (and Ivy) can figure those transitive dependencies. And there is where the information I've shielded from you lies in Maven's POM file.
See Maven has a different way to section out dependencies called scopes. And unlike Ivy they are fixed. But when Ivy is downloading these dependencies it needs to know what scopes to use when pulling these transitive dependencies (are we pulling this for testing, runtime, compilation, etc?). That should make your head spin a bit. But this is a real problem because we have to tell Ivy how to map our configurations to Maven scopes so it knows what to pull.
Without mapping our configurations they don't really work well so you have to understand this, but it's not this complicated once it's explained. So let's say we want to pull all of the dependencies JUnit has we'd do the following:
Whoa what the heck is test->default? This looks weird, but what we are saying is our configuration is test and we want to map that to the default scope in Maven. This will have the effect of pulling all junit's transitive dependencies. If we did the following:
That would only pull the dependencies junit directly declares, but not ITS transitive dependencies. You might do test->master if you wanted to compile against just junit, but not actually package it up in your application because it's optional. The user of your library must provide that library if they want to use that integration for example. Servlet API is a good example where you only need it for compilation, but you don't need to shipped with your WAR.
So here is the mystery of the -> operator in Ivy. It maps Ivy configurations onto Maven scopes when resolving dependencies so Ivy knows exactly what to pull down. It's that simple.
Back to our example now because we used defaultconf attribute to specify compile, but we didn't map it to scopes yet. So we can do that by doing the following:
We can go further and simply specify this at the configurations level so that we don't have to specify it every time we change a conf attribute.
Notice we didn't use test->default anymore? That's because we specified that at the configurations level and all our our configs are mapped to default scope in Maven for us.
There is a lot more to configurations that I don't fully understand, but I think this will demystify most things about configurations so you can start to structure your project appropriately using Ivy without trolling Stackoverflow and Ivy docs for vague answers.

Groovy Mixins and the undocumented features of this pointer

I've been using Groovy and Grails lately and I love the platform. It's a great productivity tool. However, the docs for Groovy the language are languishing and haven't been kept up to date as the platform has evolved. One of those evolutions that is poorly documented is Mixins and I'm specifically talking about dynamic Mixins. Compile time Mixins use the annotation and there are several versions using @Mixin and @Category, but essentially the are limited in their use because you can't add a mixin into a class you didn't author. That means you have to use a different mechanism to augment 3rd party classes. This leaves either modifying the metaClass property on the class or using the newer Dynamic Mixin feature.

For example let's say we want to add a zip method on java.util.File. This method would take this File instance and produce a zipped version of it. For files it simply compresses the file, and for directories it's compress the whole directory and return the resulting file. Using the metaClass property we could do the following to add this:

File.metaClass.zip = { String destination ->
   OutputStream result = new ZipOutputStream(new FileOutputStream(destination))
   result.withStream { ZipOutputStream zipOutStream ->
   delegate.eachFileRecurse { f ->
       if (!f.isDirectory()) {
            zipOutStream.putNextEntry(new ZipEntry(f.getPath()))
            new FileInputStream(f).withStream { stream ->
                zipOutStream << stream
                zipOutStream.closeEntry()
            }
       }
   }
}

This works well and now you can do something as simple as new File( 'some/directory').zip('some_directory.zip'), and boom it writes out a zipped copy of that directory! That's pretty awesome isn't it? I think you're seeing the reason for why we want to do this.

Now let's see if we can translate that into a dynamic Mixin. Here is the version in Mixin form:

class EnhancedFile {

    static {
        File.metaClass.mixin( EnhancedFile )
    }

    void zip( String destination ) {
        OutputStream result = new ZipOutputStream(new FileOutputStream(destination))
        result.withStream { ZipOutputStream zipOutStream ->
            eachFileRecurse { f ->
                if (!f.isDirectory()) {
                    zipOutStream.putNextEntry(new ZipEntry(f.getPath()))
                    new FileInputStream(f).withStream { stream ->
                        zipOutStream << stream
                        zipOutStream.closeEntry()
                    }
                }
            }
        }
    }
}

Some small changes were made to the code. One is the static block at the top now places the mixin into the File object when this class is loaded. This is where Mixins added to 3rd party could be better. Essentially I just want to add this to augment 3rd party libraries, and it could be added at compile time through a simple annotation that let's me annotate the Mixin instead of the target of the Mixin. For example if I could use @MixinTarget(File) on the Mixin to augment File it could register it at compile time, but sadly it doesn't exist. This is why were are using runtime mixins here.

The other change was removing the delegate member. In metaClass mixin land delegate is a magic keyword that points back to the target of the mixin, or the instance your code was mixed into. In Dynamic Mixin land delegate keyword doesn't exist. However, you can refer to methods in the target class by calling them as if they were instance methods on the Dynamic Mixin. Notice how File.eachFileRecurse() method is called within the mixin.

This is our first clue how Dynamic Mixins are different than metaClass mixins. In dynamic mixin land delegate is not defined so referring back to the target is undocumented! There is no discussion about how it works or how its suppose to work. This is the point of this blog post.

Now let's say we want to add an unzip method to our Mixin. Let's look at the metaClass version first:

In this example I have two overloaded versions of the unzip method. That's cool because Groovy honors Java's call differentiation by type, but the crux of this method is in the first one. It's pretty straight forward unzips this File instance into the destination File instance. See any issue with porting? That first line is passing the target of the mixin using delegate keyword to ZipFile! How can we implement that in a Dynamix Mixin!? This is the confusing part. In Dynamic Mixin land what does this pointer point to? Why it points to the instance of the Mixin. In this case its an instance of EnhancedFile. Well that doesn't do us much good does it? But what is the relationships between Mixin and Mixee? That gets a bit fuzzy. We could try casting this to a File after all it appears this is a File because we can simply call instance methods as if they were inside EnhancedFile too. Let's try that:

    ZipFile zf = new ZipFile( (File)this )

But that doesn't work and throws a ClassCastException. What about using the as keyword to convert it?

    ZipFile zf = new ZipFile( this as File )

That actually works! And here is a simple test you can try out:

    class MeMixin {
        def me() {
           return this
        }
    }

    class MeTarget {
    }

    MeTarget.mixin MeMixin

    target = new MeTarget()
    println( target.equals( target.me() as MeTarget ) )
    println( target.equals( target.me() )

The above code will print true then false. So the as keyword somehow changes the this pointer of the Mixin into the target class. It's the same reference as the original (that's important). Well it'd be pretty useless if it wasn't. Now why this works I can't explain that yet.

Here is the full code:

Now can we please raise the debt ceiling?!

I wanted to look at how bad it's gotten just by looking at the numbers we're up against. What we are arguing over is money the US takes in vs. pays out in obligations. At present time the government spends $3.834 trillion, and takes in $2.567 trillion. You should already see the problem. We're spending $1.267 trillion that we don't have. So where do we get that from without raising taxes?

By issuing more bonds, but we can't do that until this debt ceiling is raised. See we've been doing this since the 1980s. We spend more than we take in, and to get money we sell US Treasure bonds to people to keep operating. However, the debt ceiling is a law on the books that states the US Government won't borrow more than X, and every time we reach X Congress votes to raise it to Y, sells more bonds to cover the deficit, and we keep going. And, people are perfectly happy to buy them because the USA has NEVER defaulted on those obligations.

Now of that $3.834 trillion in spending some of it is allocated by law. By law we have to spend it. If we wanted to change it Congress would have to create a new law that cuts that spending. These are things like Social Security, Medicare/Medicaid, National Debt Interest, Income Security, and Veterans Benefits. This doesn't get discussed much because passing a law to cut these is really difficult, and politicians, on both sides, don't want to be the one that slashes these because they will be voted out. Some of these you can't do anything about like National Debt Interest. You don't pay that and that spells default, USA gets it's AAA rating slashed, interest rates rise up, babies die, and Jesus weeps. The $250 billion in National Debt interest is interest on all that borrowing we keep doing. For the remaining items Social Security, and Income Security are funded by specific taxes. If you cut those programs it doesn't help because those special taxes can't be used to pay for other spending. That is illegal. So what does that leave? Medicare, Medicaid, and the discretionary budget as places you can cut. I'm leaving Veteran's Benefits out of it because it's $68 billion which even you completely cut it to zero it would contribute squat, and persons who cut that thing would make Casey Anthony look like Mother Teresa.

What we're really talking about is the Discretionary Budget which in 2011 is $1.415 trillion dollars of which 63% ($895 billion) is spent on Military spending, and 37% ($520 billion) is spent on non-Military spending. In 2004 the Discretionary Budget was $782 billion and 51% ($399 billion) for Military Spending and 49% ($383 billion for non-Military Spending). That's a 58% increase in the budget in 7 years. You'll also notice how much the military percentage of the pie has increased. That means it's rising at roughly 6.8% per year. More than twice the typical 3% inflation rate. But, the more disturbing trend is military spending has increased 12.3% per year while non-military spending rose only 5% per year. Why is that important? Because Discretionary Military spending is single largest expense the American government pays out, hence if we really want to make serious cuts it has to start with the military spending.

If we didn't want to raise the debt ceiling we need to come up with $1.267 trillion by cutting spending or raising taxes. If we didn't want to raise taxes and you don't want to cut the non-Discretionary items, then we'd need to cut $1.267 trillion from the $1.415 trillion Discretionary budget. That would leave $148 billion for the government (both military and non-military) to run on. Our government couldn't function no matter how much the Tea Party wishes that were true.

What if we consider the full budget for cutting funding. In order to cut spending enough, so we don't have to raise taxes, we'd need to cut 58% from Discretionary Military spending, Discretionary Non-Military spending, Medicare, and Medicaid. If we included Income Security in those cuts we can get it down to 46% cuts across the board. And if we included Social Security it'd be around 36% cuts across the board.

Ok so let's look at it from what we'd need to do to raise taxes to cover it. In order to get $1.267 trillion more we'd need to increase taxes by 50%! 50% tax increase would cover the deficit without cutting any spending. Now if you thought cutting spending to cover it was insane. Raising taxes by 50% is bonkers. I can't afford a 50% tax hike as I bet neither can you, and corporations would get a shock so bad Wall Street would absolutely freak their shit. And, send their K Street soliders to figure out a way to shirk their responsibility. Yep same song different verse. So even if you could pass the bill I bet they couldn't collect on those taxes.

Those are the two extremes of the argument. You can't cut your way to a balanced budget, and you can't tax your way one either. However, getting really serious about fixing those problems means serious cuts and serious tax hikes. Looking at closing loops holes to raise revenue, and cutting spending is the only way you could reasonably do it. But again, there's no perfect answer given the constraints. It will still require serious cuts, and tax hikes. Even raising taxes 10% you'll need to cut $1 Trillion in spending across the board. That is going to be very hard. What about the Bush tax cuts? Even rolling those back will only add $300 Billion-ish in revenue.

The easiest way out is to raise the debt ceiling because defaulting will have tremendous consequences. And, to think it will get worked out if we miss the Aug 2nd deadline is a farse because we're already on borrowed time. This thing was supposed to get wrapped up 6 months ago, and the Treasury did some funny accounting to get more time. They've been in a stalemate since then. So if they can't figure it out in 6 months what makes you think they'll figure it out in another 6 months when the Treasury is out of money? They've been living on life support for 6 months.

So given all of the facts can we please just raise the debt ceiling? My 401K doesn't need 3rd shot to the junk in 10 years.

How Failing Fast allows you to reframe the problem

Just read an article on Fast Company on how human powered flight was solved by Paul MacCready. It's really cool because it's not a software story, but it has so many similarities with software. Success centers around creating an environment where you can iterate on your idea. I like stories like this because the motto of "fail fast" gets hollow as it is over used. After a while It's hard to remember what it originally meant. Stories help re-affirm it's meaning.

In so many ways this is really what agile software development is trying to get you to. Agile demands a lot from your team, and the only way you can live up to the promises of agile development is to create this environment. Without it you'll just fail, or worse just survive on far less productivity.

No more big design up front. It failed people for human powered flight, it failed for cars, and it failed for software.

http://www.fastcodesign.com/1663488/wanna-solve-impossible-problems-find-ways-to-fail-quicker

View Source and SVG on the iPad

I'm playing around with SVG on the iPad, and I find it's hard to really debug even the smallest thing on it. Apple is a lot of things, but calling them a developer of great development environments would be a grandiose lie. Before I say something I'll have to do a lot of explaining about I wanted to share a script for view the source on your iPad for SVG documents.

javascript:var%20sourceWindow%20%3D%20window.open('about%3Ablank')%3B%20%0Avar%20newDoc%20%3D%20sourceWindow.document%3B%20%0AnewDoc.open()%3B%20%0AnewDoc.write('%3Chtml%3E%3Chead%3E%3Ctitle%3ESource%20of%20'%20%2B%20document.location.href%20%2B%20'%3C%2Ftitle%3E%3C%2Fhead%3E%3Cbody%3E%3C%2Fbody%3E%3C%2Fhtml%3E')%3B%20%0AnewDoc.close()%3B%20%0Avar%20pre%20%3D%20newDoc.body.appendChild(newDoc.createElement(%22pre%22))%3B%20%0Avar%20src%20%3D%20''%3B%0Aif(%20document.documentElement.innerHTML%20)%20%7B%0A%20%20%20src%20%3D%20document.documentElement.innerHTML%3B%0A%7D%20else%20%7B%0A%20%20%20var%20div%20%3D%20newDoc.createElement(%22div%22)%3B%0A%20%20%20div.appendChild(%20document.documentElement.cloneNode(true)%20)%3B%0A%20%20%20src%20%3D%20div.innerHTML%3B%0A%7D%0Apre.appendChild(newDoc.createTextNode(src))%3B

To get this on the iPad follow these steps.

1. Open this page on the iPad.


2. Select all of the text from the prior paragraph


3. Add a bookmark for this page.


4. Edit the 2nd field and past the copied text in there


5. Now open a SVG document and click your new bookmark

This is a modified version of source code from Rob's Blog. The only problem with Rob's version is the use of innerHTML. Unfortunately, SVG doesn't have innerHTML. This code will handle document nodes that don't have innerHTML property by cloning them and placing the clone in a DIV element. That way we can properly get the innerHTML from there. Using this code will allow you to see the SVG and HTML source.

Here's the source code for this bookmarklet for easy debugging if you have trouble:

var sourceWindow = window.open('about:blank'); 
var newDoc = sourceWindow.document; 
newDoc.open(); 
newDoc.write('<html><head><title>Source of ' + document.location.href + '</title></head><body></body></html>'); 
newDoc.close(); 
var pre = newDoc.body.appendChild(newDoc.createElement("pre")); 
var src = '';
if( document.documentElement.innerHTML ) {
   src = document.documentElement.innerHTML;
} else {
   var div = newDoc.createElement("div");
   div.appendChild( document.documentElement.cloneNode(true) );
   src = div.innerHTML;
}
pre.appendChild(newDoc.createTextNode(src));

When you're doing it wrong...

How do you know when you're doing it right? Most of the time I know I'm doing it right is when it feels like I'm always hitting my goals, and it's getting easier than it was yesterday. Although that might be a little lie I tell myself because it might just be because I know when I'm doing it wrong, and how bad that feels. If I don't have those bad feelings I know I must be doing something right.

Here's a great example of doing it wrong. I'm at a place that loves to branch code. Most of the time they are branching because the business demands a release, but they have such a large team in order to keep everyone "busy-ish" they have to branch. They have an idea they're doing it wrong, but they don't really have a clue as to how to do it right so they just do what they know. The developer's don't like branching, but the business doesn't give them much choice.

Problem is refactoring is important because the code base is pretty hard to work with. Now when they add multiple branches + refactoring + big team = double black diamond level of difficultly in the merges. So that's bad, but another side effect is when a merge is going on it prevents people from modifying the repository. Nobody can use the source control system while this is happening. It's an all stop. One of these merges is going on it's 5th day. That 5 days where no one has integrated their changes, built all of the code, or synchronized with other people's changes. Now all of a sudden the choices to use SCM, continuous integration, refactoring, and small agile practices is really loosing it's benefit. One developer suggested we send around patches to each other while the merge is going on. We specifically picked a SCM system so we don't do that. Once the merge is done the SCM system is going to hit with tons of changes, and when something breaks functionality they won't be able to easily resolve it because of the volume of changes. Now quality is suffering directly because of relentless branching.

Funny thing is I can't think of anyone out there that suggests branching as a technique for achieving quality. However, there are countless examples from experts that generally agree using SCM, continuous integration, refactoring, and small changes help overall quality. Why are we doing something that sacrifices those best of breed practices? Now the guy with the "big picture" view seems to believe it's the actual code quality is to blame for productivity problems, and quality issues. He thinks more code reviews, and education is in order about how to write "good" code will right the ship. At some level business is just throwing crap over the wall without any real conversations.

This is what I call an "everything is arduous and ridiculous" environment. Everything about this place feels over the top hard. Why don't people around me seems to realize this is the ridiculous way to operate? Haven't they ever had that effortless feeling of productivity? How you're always the man, and it's just right? Sure this works in that we are producing a product very slowly, but it doesn't feel like a success. Is it luck? Is it innate to the problem you're trying to solve? Well...maybe.

Sure some problems are harder than others. Building Google mail is harder than building an android app. But, I've been on some pretty nasty android apps. Which tells me there is a way to make an easy problem hard, and hard problem easy. So what are we doing that makes this problem so hard?

Sometimes it's not being smart enough. We all love algorithms, and finding that simple algorithm that just makes the problem go away is sublime. That's what we all fell in love with if we have any formal training. But, those types of problems are far and few between. Mostly what we do is slog crap from one database, slap it on the glass, then slog the new crap back into the database. Rinse and repeat 1000x and you've got a product. There is no algorithm that makes that easier. If there's no algorithm then what is it?

Technique. There's a difference between algorithm and technique, and what types of problems they are best suited for. Technique isn't going to come up with map reduce. That's algorithm. Technique is your choices for what you're going to use, and how you're going to use it so the problem is easier. Technique is also about how you choose to define the problem which means technique comes before algorithm. How can you choose an algorithm if you don't know what your problem is?

Technique breaks down into two parts. Choosing a set of tools and processes, and how you apply those tools or processes. Technique extends past the end product into the support systems that nurture how that end product is created with bug tracking systems, source control management systems, continuous integration, user forums, etc. And, those choices can have a greater effect on the end product than what you put into the product. Just reread the example above for justification.

To some degree, we place too much emphasis on tool choice because how you apply it can undermine the choice of using that tool. If your technique doesn't match the tool the tool will never matter. Have two bug tracking systems because one group doesn't want to give up their existing one. Been there, real story, doesn't work, definitely doing it wrong. (Actually same place as the example probably could have a book of "doing it wrong" ideas from this place). As in the example at some point application of those tools made the choice of SCM moot.

In the end we need to discuss technique more passionately than specific technologies. The two do go hand in hand, but it's the technique in the end that makes the difference. So how do you know you're doing it right? When technique matters more than technology.

Flexjson meet Android

Flexjson 2.1 now supports running Flexjson on Android. So I thought I'd show a quick example of using Flexjson in an Android application. Hopefully this will spark some ideas about what you can use Flexjson for in your own application. I'm going to start simple creating a quick Android app that pulls recipes from Puppy Recipe, parses it using Flexjson, and displays it in a list. Let's get started.

Recipe Puppy has a very simple REST API, almost too simple, that returns responses in JSON. Recipe puppy allows you to search recipes by the ingredients contained within by using a URL parameter i. Individual ingredients are separated by a comma, and URL encoded. Here is a simple example:

http://www.recipepuppy.com/api/?&i=banana,chicken&p=1

Exciting isn't it? If you click that link you'll see the JSON response. It's a little hard to read like that so here is a simple break down with a little formatting:

{
   "title":"Recipe Puppy",
   "version":0.1,
   "href":"http:\/\/www.recipepuppy.com\/",
   "results":[
      {
         "title":"Chicken Barbados \r\n\r\n",
         "href":"http:\/\/www.kraftfoods.com\/kf\/recipes\/chicken-barbados-53082.aspx",
         "ingredients":"chicken, orange zest, chicken, banana, orange juice, brown sugar, flaked coconut",
         "thumbnail":"http:\/\/img.recipepuppy.com\/602538.jpg"
      },
      ...
   ]
}

This is pretty straight forward. We have a little header and what we really are interested in results property which is an array of recipe objects. So we'll create two simple Java classes to map those data members. RecipeResponse for the header portion, and Recipe which is the object contained within "results" property.

Here are those objects:

public class RecipeResponse {
    public String title;
    public Double version;
    public String href;
    public List<Recipe> results;

    public RecipeResponse() {
    }
}

public class Recipe {

    private String title;
    private String href;
    private String ingredients;
    private String thumbnail;
    private Drawable thumbnailDrawable;

    public Recipe() {
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title.trim();
    }

}

In the Recipe object I actually created a Java Bean with getter/setter, but I didn't include most of those methods. I did make a point to show the setter for the title property. Turns out some of the data coming out of recipe puppy contains extra newlines characters in the title. To get rid of those I'm doing a trim() in the setter. Flexjson is smart enough to call the setter method if you have defined it instead of setting values directly into the instance variables. However, if you use public instance variables it will set values directly into those too. This was a fix made in 2.1 with respect to using public instance variables during deserialization process. You'll be happy to know it works now.

So let's jump to the usage of Flexjson in the android code. So we create a RecipeActivity that contains a List to display the recipes. We're going to look at the AsyncTask that loads the data using Flexjson. Here is the full code for that:

        new AsyncTask=<String, Integer, List<Recipe>>() {

            private final ProgressDialog dialog = new ProgressDialog(RecipeActivity.this);

            @Override
            protected void onPreExecute() {
                dialog.setMessage("Loading Recipes...");
                dialog.show();
            }

            @Override
            protected List<Recipe> doInBackground(String... strings) {
                try {
                    return getRecipe( null, 1, "banana", "chicken" );
                } catch( IOException ex ) {
                    Log.e( RECIPES, ex.getMessage(), ex );
                    return Collections.emptyList();
                }
            }

            @Override
            protected void onPostExecute(List<Recipe> results) {
                if( dialog.isShowing() ) {
                    dialog.dismiss();
                }
                Log.d( RECIPES, "Loading " + results.size() + " Recipes" );
                recipes.setList( results );
                new ThumbnailLoader( recipes ).execute( recipes.toArray( new Recipe[ recipes.size() ]) );
                Log.d( RECIPES, "Loaded " + recipes.size() + " Recipes" );
            }

            protected List<Recipe> getRecipe( String query, int page, String... ingredients ) throws IOException {
                String json = HttpClient.getUrlContent( String.format( "http://www.recipepuppy.com/api/?q=%s&i=%s&p=%d",
                        query != null ? URLEncoder.encode(query) : "",
                        ingredients.length > 0 ? URLEncoder.encode(join(ingredients,",")) : "",
                        page ) );
                RecipeResponse response = new JSONDeserializer<RecipeResponse>().deserialize(json, RecipeResponse.class );
                return response.results;
            }
        }.execute();

The method your probably most interested in is getRecipe(). This method formats the URL we're going to load. It then loads that URL and passes the results returned as a JSON block to the JSONDeserializer. JSONDeserializer will take a JSON formatted String and bind that into a Java object. In this example, we're binding into a RecipeResponse object. Here is how that is done:

RecipeResponse response = new JSONDeserializer<RecipeResponse>().deserialize(json, RecipeResponse.class );

A single line of code does that. The deserialize() method performs the deserialization and binding. The first argument is the JSON String, and the second is the top level class we want to bind into. Notice we didn't have to mention anything about Recipe. Flexjson is smart enough to use the data types from the top level object to figure out any other data types contained within. So if you refer to the RecipeResponse.results instance variable you can see the List data type with a generic type. Flexjson will use generics whenever possible to figure out concrete types to instantiate. Of course polymorphism, interfaces, abstract classes, and the like causes issues with this, but we're not going into that right now. See the Flexjson home page to find out more.

You'll notice the RecipeResponse object is returned fully populated with the JSON data, but we're really only interested in response.results so we just return that. It'd be nice if Recipe Puppy returns how many total pages there were in the header (hint, hint) so that it was more interesting. Anyway it is beta. That array is then added to the ListAdapter and displayed on the screen.

Other things Flexjson could be used for is saving state by serializing objects to JSON, and then deserializing when Activities are reconstituted. This can be easier than writing ContentProviders to dump stuff into the database. One of my biggest gripes with Android is how between pages objects can be reliably sent because Intent's require you break everything down to primitives. With Flexjson we can just simply serialize an object put that in the Intent, and then deserialize it on the other side. So no more boilerplate code to flatten your objects.

Here's a simple example serializing our recipes to the disk:

        File f = app.getFilesDir();
        Writer writer = new BufferedWriter( new FileWriter( new File( f, "recipes.json") ) );
        try {
             new JSONSerializer().deepSerialize(favorites, writer);
             writer.flush();
         } finally {
             writer.close();
         }

Now I know there are people worried about performance, but timing the following code this ran on device in less than 40ms which is within the acceptable bounds for UI performance. If you need more performance you can cache the JSONSerializer/JSONDeserializer instance which optimizes data type mappings so it doesn't recompute those when serializes and deserializes. As always measure, measure, measure.

You've gotten an introduction about how Flexjson can make it easier to work with JSON data with Android.

Caringorm is Architectural Poison

No one has ever accused me of shying away from sensational titles, and now's not the time to get timid. I must confess I've never been a fan of Caringorm. My first impression of Caringorm was it's over engineered. Why are there so many layers? Isn't that going to just slow you down having to develop a View, Command, Delegate, Service, etc for every round trip I make to the backend? Now that I'm working on a project that has gone horribly wrong I see how Caringorm architecture directly contributed to the problems. We've been called in to straighten out the mess and put down a more suitable architecture. After understanding what the team has done I begin to see the techniques, that Caringorm purports as best practices, create more work for yourself the longer you use them.

Software architecture should organize your work so you can work at a higher level related to your problem you are trying to solve. It does this by fostering reuse in your code. It should allow you to reuse what you did yesterday to apply to today's problem reducing the work required get work done. As your project grows the only way you can move quickly is through reuse. Without reuse the work grows exponentially to the point where value can't be delivered. How long this takes before your code base becomes unproductive? Ten releases? 10 Years? I've seen it happen in 1 release, and less than 1 year.

Key signs this has happened in your project is talks about rewriting your application or major refactoring. Other signs come in when your customer says that should be easy, and then developing it takes significantly more time than you'd expect. Bad architecture robs your team's performance to deliver value. If this goes on too long your project will get scrapped and if you're lucky you'll be allowed to start another project. Most likely you won't because the business will be putting your project in maintenance mode while they spin up the "solution". Good architecture is quite the opposite. Easy things are easy and hard things are possible. At the core of this is the level of reuse in your project.

Caringorm doesn't foster reuse. It stalks it, attacks, leaves it dead, and poisons the earth to keep it from ever fostering. At the heart of this is the age old singleton problem. Singleton's are seriously bad technique, and I wish every developer out there understood this. Using singletons to limit an instance to a single instance is not all together bad, but using it as a locator pattern is where the serious issues began to make your code single use. Unfortunately you can't limit a singleton to the good parts without accepting the very serious downsides, and this is the reasons I try and void them at all costs because the downsides are that damaging. Cargingorm has no problem using the ModelLocator (which is a singleton) in your views (mxml). And there is no amount of other techniques you can introduce to overcome the problems that come with this. I don't care if you're using Code Behind, Presentation Model, or whatever. If you use a singleton in your views you can't reuse them. Anything that directly references singletons becomes single purpose in its use as well, including anything referencing those objects, and so on and so on.

Why is that a problem? Well consider if we wrote DataGrid with the same techniques Caringorm purports as acceptable practices. Let's say DataGrid.dataProvider was hard coded to look in ModelLocator.getInstance().dataProvider. Now how can you have two instances of DataGrid in your program pointing at two different dataProviders? You can't. And this is precisely the problem that leads to serious architecture problems with Caringorm programs. Now throw in calls to getController().eventManager.addEventListener() in your views and you have a serious recipe for disaster.

You might find my example contrived so let me describe a more real world scenario. Say you have a signup process that people fill out on your site and you have a view that represents the information you want to gather. In that view you're using the ModelLocator. Now the customer wants to add a new way to sign up because their doing an email campaign, and they'd like to pre-populate that form from details like the email address and ad campaign number to track it into the view. Unfortunately, ModelLocator makes it difficult to put two different models into your view because it's hard coded to one. What would have been an easy task by instantiating another instance of your view has turned into creating another view from scratch. So let's say you need to do this fast and you copy the view and makes the changes to create two different views. Then the customer wants to add a field to both views. Now you need to update two places in your application. This is precisely what I mean when I say Caringorm creates more work for you. Over time if enough of these exist in your application your productivity will drift to zero because maintaining all of it too much work.

Now hopefully I convinced you that the patterns Caringorm suggests are not helping you. And, you decide to banish ModelLocator from the views. However, the problem of geting something from the model and bound into the view still exists. So what part of the Caringorm architecture will interact with the ModelLocator and the view? Normally this would be the Controller in a traditional MVC pattern. In Caringorm the Command is suppose to be this part, but it doesn't have access to the view. Therefore, how will it set the data properties on the view? You could do some gymnastics by passing references through the FrontController into the Commands, but at this point I'm taking some serious liberties to modify Caringorm's architecture design to make it work. If the architects of Caringorm had realized this then their examples would have shown how to do this.

I'd like to think that as an industry we're working towards a shared understanding about the dangers of singletons, and if that were true I'd expect to see a drop in the number of projects using them. I'd expect to see a reduction in the number of frameworks employing singletons as an instance locator pattern. However, it's quite the opposite. Most developers don't see a singleton and get that tingling sensation that something bad is about to happen. But, singletons have all the same problems that global variables do and by in large most developers realize global variables are poison if not very carefully used.

Caringorm is like EJB of Flex. Over engineered. Expect there to be serious changes to Caringorm in the future to save the marketing that Adobe has done with clients. Just like Sun did with EJB3 for EJB. Sun had over emphasized the benefits (if there were really ever any) to using EJB, and once the community realized EJB was over engineered and more trouble than it was worth. Sun had no choice but to hire Hibernate's creator to design EJB 3.0 and started begging for forgiveness. Adobe will have to do the same.

GreenThread: Problems with Recursive Functions

In previous blog posts on GreenThreads I mentioned that the downsides of using GreenThreads meant you couldn't write recursive functions. In one of the comments I was asked to expand on this idea, and after the comment got so long I figured a blog article might be a better forum for this topic. I'm going to discuss the issues with regular recursive functions, then we'll explore the differences between two types of recursive functions, and potential changes that could be made to help make it easier to write recursive GreenThreads.

Let's start by examining the following function:

public function factorial( i : int ) : int {
   if( i == 0 ) return 1;
   return i * factorial( i - 1 );
}

It's like the "hello world" of recursive functions. What makes a function recursive is the fact factorial() function calls itself in evaluating the value of the function. If we were to run it in a GreenThread there's no way for the system to interrupt the function calls should this function take longer than the length of a frame. For example, say you ran factorial(5) in the GreenThread. The call stack will look like the following:

factorial(5) -> factorial(4) -> factorial(3) -> factorial(2) -> factorial(1) -> factorial(0)

There's no way to let Flash insert a paint in between factorial(3) calling factorial(2) because factorial(3) calls factorial(2) directly.

The way GreenThread framework works is that it handles repeatedly calling your GreenThread until the time has elapsed for a single frame. At that point it let's Flash have control again and then resumes on the next frame repeating this process until your GreenThread says it's finished. This is actually implemented using a big loop outside your GreenThread.

It gets even harder for recursive functions. Look back at factorial() function. Notice that factorial(5) has to compute factorial(4) before multiplying by 5 so it can return it's value. Therefore, it's not possible break out of the function call, allow Flash to paint, then resume within a function so it can multiply by 5 to finish the computation. (Not unless Flash supported continuations, but that's a whole another topic). So now recursive functions can't be interrupted because the function directly calls itself, and depending on how you write your recursive function it's not possible to put a break because of operations that might come after the recursive call finishes.

There are other issues with recursive functions, but they are not possible to use in a GreenThread because of the dependencies between stack frames. However, there is another type of recursion that can help eliminate dependencies. Let's write our recursive function to remove the dependency on local operations:

public function factorial( i : int, accumlator : int = 1 ) : int {
   if( i == 0 ) return accumlator;
   return factorial( i - 1, i * accumlator );
}

Now notice that factorial(5) doesn't have extra work to run after factorial(4,5) runs like we did before. This means factorial(5) could be replaced by the return from factorial( 4, 5 ). In fact factorial(5) == factorial(4,5)! This technique is called tail recursion, and in certain languages it helps make recursion work without growing the stack frames so large iterations don't overflow the stack. Now Actionscript doesn't benefit from this, but this will allow us to work around the second problem we have. We still have our original problem so we'll have to tackle that before we're done.

Now we still have factorial(5) directly calling factorial( 4, 5 ) so Flash can't interrupt the function calls so it can paint. However, what if we had a special call that would delay calling factorial(4,5) so we could do whatever Flash wanted, then it would resume our recursive function.

Well there exists such a function: callLater(). callLater() can be used to schedule a function to be called in the next frame, and in fact from all of the testing done it's safe to use callLater() as a technique for implementing GreenThreads. However, directly using it will suffer from poor performance because of long waits between function calls. So, let's assume there is a new function in GreenThread that acts like callLater(), but achieves better performance. Now our recursive function could look like:

public function factorial( i : int, accumlator : int = 1 ) : Boolean {
   if( i == 0 ) return false;
   return invokeOnThread( factorial, i - 1, i * accumlator );
}

Now invokeOnThread() doesn't exist in the current code base, but it could be written. Actually the accumlator would probably be best served as an instance variable within your GreenThread, and we'd need to change some more features to fit within the framework. Assuming that it is we could support recursive functions given these constraints:

• You must write your recursive calls so they conform to tail recursion.


• You must use invokeOnThread() to recursively call your function.


• You must conform to the contracts of the GreenThread framework.

The upside is the ability to think recursively. While every algorithm can be expressed either in iteration or recursion it's not easy to convert between each form. Some algorithms are easier to express using recursion and can be very hard to write iteratively. The downside is the requirement to write tail recursive algorithms which can be difficult for the uninitiated, but it's a skill that can be honed.

What if HTML wasn't Top Dog?

What if HTML wasn't the top node in our web pages? Sounds strange right? But, what if it was just another node inside a larger structure? HTML is great for defining textual documents where you want page flow layout as your choice. However, it really is painful to use a general purpose UI layout language. Which I would argue is the more common practice these days. Even the simplest blogs, forums, or search pages have some form of application layout involved. Why is it so hard with HTML? Lots of that derives from page flow layout and legacy support of this concept. But, if we embed HTML in a larger structure we could do whatever we wanted.

What sucks most about HTML? I would argue it's all the time I waste trying to get the layout I wanted. All the time I spent learning CSS and HTML 4 was probably 6 months or more before I felt comfortable with it. I could come close to the layout I had in my mind. However, as soon as I switched browsers my beautiful layout went to crap, and I had to dig into arcane browser hacks to make it work. Who enjoys that?

How much of that is the complexity of CSS rules and HTML? Ever tried or thought about creating a browser? It's NOT easy. I find understanding the interaction between CSS and HTML arcane as a web designer. If I find it hard then it's really hard for the browser developer to get it right. And that's precisely what we've seen. Lots of inconsistency in how they interpret the meaning of things. Leading to browser inconsistencies. If it's simple to understand then it's simple to implement. If it's simple to implement it's easier for two people to come to a common expectation.

Let's get specific. Say I wanted my node to be position relative to it's parent. I want to set the top left corner of an element to be 50 pixels from the left and 50 pixels from the top. In HTML I can set the left and right of my component, but I also have to set the layout to absolute on the child, and set the parent to relative. This is a common practice in other UI toolkits, but it's complex in HTML. What if all I did was this:

<application>
   <box top="50" left="50" width="200" height="200"></box>
</application>

Pretty simple right? Although this isn't that far from HTML/CSS, there are other things that aren't so easy. What if I wanted to horizontally align that box relative to the parent's center.

<application>
   <box width="800" height="600" horizontalCenter="0"></box>
</application>

Simple. HTML/CSS you would use margin: auto? WTF!? Doesn't horizontalCenter make more sense? Of course it does. Try vertical centering on for size:

<application>
   <box width="800" height="600" verticalCenter="0"></box>
</application>

Try that with HTML and you'll come up short or at best bizarre.

What about defining boxes that grow when the window is changed? That can easy too:

<application>
   <box id="banner" left="0" right="0" top="0" height="50"></box>
   <box id="leftmenubar" left="0" width="250" top="50" bottom="0"></box>
   <box id="content" left="250" right="0" top="50" bottom="0"></box>
</application>

Simple. The content area sets his left and right relative to the parent's edges. When the parent grows so does the child. The banner and leftmenubar are fixed in position. However, the banner grows its width as the parent's width grows.

Even supporting legacy HTML documents could be simple.

<application>
   <box width="800" height="600" horizontalCenter="0" verticalCenter="0">
      <HTML width="100%" height="100%">
      </HTML>
   </box>
</application>

HTML just becomes another possible node within the super document. It would create yet another box that can display text documents using what you want for text documents which is page flow layout. HTML nodes could occur as many times as we need in our over all application.

Furthermore, legacy HTML documents (e.g. those starting with HTML) could be converted into our application tag just by surrounding the application tag around the legacy HTML only document. Hence making all HTML documents forward compatible with application documents.

It's a simple idea to fix the constant layout problems with the web.

On the Importance of being Synchronous: Asynchronous + Actionscript

Dealing with Asynchronous Events Part-2

Damn Blogspot sucks. What the hell? Why haven't they added the 1st new feature in like 5 years? Trackbacks hello???? WTF? I'm not up for cobbling a solution together with greasemonkey, yada, yada, yada. I need a new blog platform. Enough about that let's get to code.

Anyway I wanted to add my fuel to the fire on asynchronous programming. This is a topic I'm very interested in because Actionscript isn't the only language suffering from this. It's very much rooted in classic Computer Science so it's a deep topic. That blog post is old, but it's still something that doesn't have a satisfactory answer yet. Computer scientists have been discussing this topic since the 1970s in one form or another.

I'm very satisfied with my solution to the single request problem. By that I mean making a single round trip to the server and back. Here is roughly how I do asynchronous calls in actionscript.

var tag : String = "Archive";
var loader : URLLoader = defaultLoader();
loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
   var json : Object = JSON.decode(loader.data);
   var mail : Array = json.map( funcion( json : Object, index : int, arr : Array ) : Email { 
      return new Email( json ) 
   } );
   var event : DynamicEvent = new DynamicEvent('mail.loaded');
   event.mail = mail;
   dispatch( event );
} );
loader.load( session.httpGet( '/home/email/', { tag: tag } ) );

Really tight code, and it doesn't feel like the infrastructure for doing the calls are in the way of understanding what's going on. Now I'm using several factory methods to encapsulate common error handling, host name, authentication tokens, etc. Of course all of this can be overridden, but having defaults keeps that code out of the flow of how you work.

The difficult part comes when you need synchronous flow control over asynchronous calls. This only starts to show up with more than one trip to the server. Say for example, server call 1 must complete before server call 2. You can chain them like so:

var loader : URLLoader = defaultLoader();
loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
   var json1 : Object = JSON.decode(loader.data);

   // do something with json1

   var nextLoader : URLLoader = defaultLoader();
   loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
      var json2 : Object = JSON.decode( nextLoader.data );

      // do something else with json2, and maybe json1

   });
   loader.load( session.httpPost( '/home/update', { arg1: json1.arg1 } ) );
} );
loader.load( session.httpGet( '/home/synchronize/', { hashkey: hashkey } ) );

It's doable, but it's getting messy. And quite frankly a little hard to understand. Is that all we need? If so, then we can stop here and be ok. Sadly, no the rabbit hole can get worse and twisted. Say we want to do server call 1, server call 2, or both based on some conditions! And we want to maintain the order call 1 precedes call 2 if call 1 is done. Kinda of like:

var result : Object = null;
if( someExpression ) {
   result = executeServerCall1();
}

var result2 : Object = null;
if( someOtherExpression ) {
   result2 = executeServerCall2( result.arg1 );
}

Now I want to stop right here and say. Look how easy that was to specify in synchronous code. Jr. programmers can understand that code. All things like conditional logic, control flow, data flow, and more importantly re-usability are all effortless. Just doing simple control flow between asynchronous code is a real challenge.

One thing that I really have trouble with is refactoring logic into a re-usable method that I can call from multiple locations. In synchronous land I can wrap behavior around it doing logic before and after that method. I can easily pass data between in and out. All of these properties lead to reuse and powerful constructs for hiding details. The basis of easy to follow and maintain algorithms.

Adding logic before and after is very hard when the method uses asynchronous calls. I've decided that adding callback objects into the calls it the best route. For example,

public function updateUser( user : User, callback : Function ) : void {
   var loader : URLLoader = defaultLoader();
   loader.addEventListener( Event.COMPLETE, function( event : Event ) : void {
      var json : Object = JSON.decode(loader.data);
      var user : User = new User( json );
      callback( user );
   });
   loader.load( session.httpPost( '/user/update/', { id: user.id, email: user.email } ) );
}

I prefer callbacks to using event listener. The main reason for that is event listeners are more long living, i.e. longer than a single method call. If you use event listeners you have to register and unregister between calls, i.e. more mess. If this method is apart of a longer living instance, as I typically do, you could get more than one callback happening. Callbacks are isolated between method calls so they can independent from one another. (There's a lot to discuss here too, but I'll save that for later).

I'm working on my next evolution of this idea to try and build up an architecture to help aid in making multiple round trips to the server in order without adding fuss, and hopefully allowing an outside person to read my code without needing a lobotomy to put my brain in their head. We will have to step away from our friend closure for this to work. But, I want to leave you with this thought.

All of the tools we use today are aided by synchronous control flow. When we remove synchronous flow our tools fall apart. We have very little tools at our disposal to help specify complex flow using asynchronous semantics. Closures are about it, but they aren't enough and fall apart quickly. We need new constructs that aid in asynchronous control flow. Possibly a way to restore synchronous flow, but asynchronous underneath. If we had these constructs we could do this type of work independent from things we typically think of like threads, processes, message passing, etc. Those constructs could be underneath it, but we as programmers would be less involved with their presence.

Agile Methods are Disruptive

Now I might actually start a fight with that title, but at least it got you here. I recently just finished reading the "Inventor's Dilemma" by Clayton Christensen. It's an amazing book that focuses on the difference between disruptive technologies and sustaining technologies. This distinction is important because it changes how your organization should attempt to develop and manage these types of technologies. Within the book he discusses some of the social forces at play that make developing disruptive technologies different from sustaining ones. I believe these same dynamics are at work with adopting agile development processes.

In order to understand this you have to understand the difference between what is disruptive and what is sustaining. Sustaining technologies are complimentary to existing technologies your customers use. Sustaining technologies will be easily accepted by your existing customer base. That property makes it very easy for you to develop within product line using your existing resources and process. Disruptive technologies are the opposite. They most likely won't be accepted by your customers at first and your organization will find it extremely hard to develop them in house. Christensen's argument is that disruptive technologies only work if they are spun off into a separate organization independent from your own. They must be quarantined away from the core or else your organization will kill them at all costs.

It's a fascinating phenomenon, but at the end of the book he gives hints as to why this is so. He starts talking about the three components that make up any organization: resources, processes, and values. This is where the book starts to sound more like a sociology study on business structure. He points out that resources are portable: people, assets, software, money, etc. They can be fired, hired, moved, procured, sold, bought, etc. They don't care where they are, and they can be applied anywhere you want. This the key difference between the other two because processes and values are NOT portable. It's much harder to move processes and values to between organizations. These are very important properties because without them the organization would disappear. Remember resources move in and out of an organization but it's these processes and values that stay behind and keep it alive. This also means that changing these processes and values is next to impossible. Why? Well because they are what define the organization if you change them then the organization dies, and a new one comes into being.

This got me thinking about agile environments and how they try to affect the later two components. Agile is a process, but mainly targeted at software development. And with it comes a certain set of values you must adopt or else you're going to find it very hard to follow the process. If you don't accept the idea that high levels of communication and collaboration are much better than comprehensive documentation then you'll find agile methods very hostile.

The other day a group of developers were all talking about agile development. Eventually we drifted towards the difficultly we were all having trying to convert an organization into an agile one. Almost all of us felt like it was somewhere between limited success to impossible. It finally hit me. We're trying to do the exact thing Christensen says you can't do. Change a company's process AND values! Not so much a company, but a development team which like a company has processes and values.

We all had anecdotal evidence of a lack of success in doing so. In fact of all the organizations I know that have successfully adopted agile development were green field starts, or they were able to convert everyone in their organization to it all at once. This normally meant small shops or isolated teams. And, in fact my only successful attempt was when I was on a team that was separate from the rest of the development organization that had virtually no dependencies on non-agile groups. My other attempts were very large groups, or groups that had lots of dependencies on other non-agile groups. No surprise those all failed to reap the benefits of agile development.

Why do groups with dependencies fail? It seemed obvious at the time, but I think another idea Christiansen mentions is to blame. And, that is Resource Dependence Theory. In the book Christiansen explains a theory of management that says something like the following. Employees (e.g. CEO, the board, VPs, managers, etc) aren't in control of the decisions in a company. The customers they serve are. I would actually add to this that it's not just the customers, but suppliers, partners, etc. For example, think of the car dealers for GM and how they have crippled GM's ability to cut costs by closing dealerships through the years. The CEO could do nothing to change this until they almost went bankrupt. That's how entrenched customers can make an organization.

This same idea of resource dependence comes into play with agile teams. If you have a lot of dependency between you another team then you will find it increasingly difficult to be agile yourself. Why? Just like company heads aren't in charge of their companies you aren't in charge of your group.

Just like when you develop a new disruptive technology you can't stop at the product development teams. You have to break everything off: sales, marketing, etc. Agile is much the same as it requires you to adopt a new set of processes and values in how you build your products.

Agile processes' values are in conflict with traditional development values. For one, agile can't predict what will be in the release and when it will be done. It can only predict one or the other, but not both. Traditional development thinks it can do both, but really it can't. However, this points out a key difference in values between the two. Traditional shops like both pieces, or they like the idea that they might know both pieces. Traditional development believes you can reliably predict outcomes and plan for long term success. Agile shops believe predictions are unreliable and reject the idea of long term planning in terms of project management. Traditional shops call for loads of documentation and check points. Agile groups reject documentation as wasteful and unproductive in favor of collaboration and high level of communication. These differences in values makes traditional shops find reasons to reject agile, or at best neuter it into submission.

I remember one such conversation that illuminated how ferocious this difference in values can be. We were in a meeting trying to explain agile development practices work. It turned into a huge argument with one of the vice presidents about why agile practices would never work for product development. He fixated on the lack of "robust" agile procedures. He claimed that they might work in the consulting world, but don't apply to product development because product development need more "robust" procedures. His main evidence was this problem of predictability. Agile development could not predict both features and delivery schedule. The VP insisted agile development would not deliver quality software that a product company needs. In many ways it sounded exactly like an existing customer might come after a disruptive technology. Disruptive technologies typically don't have the same level of performance the entrenched technology does at first. The VP was making an argument over quality much in the same way Christensen says existing customers will make over disruptive technologies. Disruptive technologies, at first, don't perform as well, scale as well, or meet the high end needs of the existing customer base. So they usually find their foothold in smaller markets with lower margins. The incumbents are often all to happy to let the disruptive technology provider enter them because they don't make much money from these lower level markets anyway. I think it's interesting how agile development found it's foothold in consulting and small teams first.

If you want to succeed with agile development in your organization don't try and change your existing development process into an agile one. It won't work or if you are able to do it it will be a very frustrating and tiresome process. Better is to start an agile organization. Separate them from other non-agile groups, give them autonomy to affect the processes outside your development organization. Don't see agile as just something your engineers do.

GridGain, GigaSpaces, Windows HPC on EC2

For those of you interested in grid computing I found an older, but great post about scalability of ec2 for grid based applications. The thing that caught my eye was the final test using Windows HPC and Velocity. The tests were not comparable to each other, but the final test shows how much degradation you suffer when you're data is stored away from your computations. In there tests 31x reduction in performance when your data is stored "out of the cloud". I think this really shows the importance for good redundant storage at the point of computation.

http://highscalability.com/your-cloud-scalable-you-think-it

The good news is for GridGain is the near linear scalability up to 512 nodes in pure CPU tests. Not as high as 2000 nodes for Hadoop, but that's the only real numbers I've seen anywhere on it. Does hint that GridGain's network overhead is really pretty light.

Grid Computing: Intro To GridGain Talk is Online

I finally got some time to put up the slides, and source code for my talk I gave at the Devnexus conference in Atlanta. Here is the link to the slides, and the source code is here.

Actionscript and Concurrency (III of III)

In the previous article we covered techniques for breaking up our long running job, but the performance was 40x slower than if we just ran our algorithm straight out. The problem is our algorithm spends very little time doing work, and a lot of time waiting for the next frame. Actionscript's performance is really quite high. We need to increase the time spent running our algorithm and minimize the time we spend doing nothing. We can do that by doing many iterations per frame instead of just one. Using getTimer() we measure how much time we spent looping and back off right before the next frame. Let's look at the code:

public function start() : void {
    Application.application.addEventListener( Event.FRAME_ENTER, onCycle );
}

private function onCycle( event : Event ) : void {
   var cycle : Boolean = true;
   var start : Number = getTimer();
   var milliseconds = 1000 / Application.application.stage.frameRate - DELTA;
   while( cycle && (getTimer() - start) < milliseconds ) {
       cycle = doLongWork();
   }

   if( cycle == false ) {
       Application.application.removeEventListener( Event.FRAME_ENTER, doLongWork );
   }
}
    
public function doLongWork() : Boolean {
    // do some work
    i++;
    return i < total;
}

Now we broken up our algorithm into an extra method. First the start() method which we're already seen. The new method is the onCycle which calculates how long a frame is in milliseconds. The loop continues until either the doLongWork method returns false, or we run out of time. Notice the DELTA constant is some constant that keeps us from eating up the entire frame. We need to give a little breathing room for Flash to drain the queue. Notice how our doLongWork method is just the code pertaining to our job. This makes it's easier to build a general purpose solution that we can reuse.

Green Threads

We can't use true OS threads in Actionscript, but any language can emulate threads. This technique is often called Green Threads. Lots of languages have used this in the past. Threads in Ruby are still green, and early versions of Java were green as well. Now Actionscript can too. I should pause and give credit to Drew Cummins who implemented a version of this for Flash player 10. I've rewritten this to remove the dependency of Flash 10, and changed some of the API so event dispatch is more natural, added easy progress events, and optional progress tracking. Let's see how our Mandelbrot algorithm changes when we use this.

In order to use GreenThreads create a subclass of GreenThread, override run method, and optionally override initialize method to add code that runs at the start. Here is an example:

public class Mandelbrot extends GreenThread {
 private var _bitmap : BitmapData;
 private var _maxIteration : uint = 100;
 private var _realMin : Number = -2.0;
 private var _realMax : Number = 1.0;
 private var _imaginaryMin : Number = -1.0;
 private var _imaginaryMax : Number = 1.0;
 private var _shader : Shader;
 
 private var _realStep : Number;
 private var _imaginaryStep : Number;
 private var screenx : int = 0;
 private var screeny : int = 0;

 override protected function initialize( ) : void {
  _bitmap = new BitmapData( width, height, false, 0x020202 );
  screenx = screeny = 0;
  _realStep = (_realMax - _realMin) / Number(_bitmap.width);
  _imaginaryStep = ( _imaginaryMax - _imaginaryMin ) / Number( _bitmap.height );
 }

 override protected function run():Boolean {
  if( screenx > _bitmap.width ) {
   screenx = 0;
   screeny++;
  }
  if( screeny < _bitmap.height ) {
   var x : Number = screenx * _realStep + _realMin;
   var y : Number = screeny * _imaginaryStep + _imaginaryMin;
   var x0 : Number = x;
   var y0 : Number = y;
   var iteration : int = 0;
   while( x * x + y * y <= (2 * 2) && iteration < _maxIteration ) {
    var xtemp : Number = x * x - y * y + x0;
    y = 2 * x * y + y0;
    x = xtemp;
    iteration = iteration + 1;
   }
   
   if( iteration == _maxIteration ) {
    _bitmap.setPixel( screenx, screeny, 0x000000 );
   } else {
    _bitmap.setPixel( screenx, screeny, shader.lookup( Number(iteration) / Number(maxIteration) ) );
   }
   screenx++;
   return true;
  } else {
   return false;
  }
 }
}

The run() method is the body our of loop. The intialize() method is called once after the user calls the start() method. After that run() method is called repeatedly until it returns false. It's perfectly acceptable to call start() more than once to kick off the thread again after it's finished. That means you can calculate the Mandelbrot set from different zoom levels without needing to recreate new instances. The initialize() method will be called every time start() is called. Check out the results here.

You can also add optional progress tracking by setting maximum, and progress members. This will automatically dispatch ProgressEvents so that your instance can be a source to a ProgressBar. It makes tracking your job easy. GreenThread also subclasses EventDispatcher so you can dispatch events from within the run method.

By in large we've solved the performance problems or we've gotten very close. What's holding us back is the resolution of getTimer(). Since we only have precision of millisecond we really can't run the risk of going smaller than 1 millisecond for our DELTA. That costs us a few iterations on our run() method which can make a difference over 1000 cycles. We could be a full second behind Actionscript that just ran the job straight through. There are a few things we can do to squeeze a little more performance out of GreenThreads.

Frame rate governs everything we do, and by default Flex applications run at 24 frames/s, but really most Flex applications don't do that much animation so if we dropped the frame rate in half to 12 frames/s we would be able to run for longer periods uninterrupted. The fewer interruptions we have, the faster we'll go.

GreenThreads also allows you to configure how much of the frame's time you dedicate to running your thread. By default it's set at 0.99 that roughly leaves 1 ms to update the UI. Under some experimentation this has proven to work quite well without creating lots of timeouts, but if you want to tweak it just provide a new value in the start method like so:

public function go() : void {
   start( 0.5 );
}

If the delta is less than 1 then it means a percentage of the length of a frame. If it's >=1 then it means the number of milliseconds to subtract from the length of a frame. Some more thought needs to go into this so that as you run your application on different machines with different CPUs so the pause is appropriate for the CPU. In the future it might need to be dynamically adjusted as the algorithm runs.

Thread Statistics

GreenThreads supports runtime statistics for tracking your job. To turn on thread statistics pass true to the GreenThread constructor. Thread statistics collects total time the job took, number of timeouts, min and max iteration times, average time a single iteration took, how many cycles it took, etc. There is a fair amount of information that can be gathered to help tune your thread. You can access that information by doing the following:

public class SomeJob extends GreenThread {

   public function SomeJob() {
      super( true );   // turn on debug statisitics

      addEventListener( Event.COMPLETE, function( event : Event ) : void {
         trace( statistics.print() );
      });
   }
}

Conclusion

There are some drawbacks to doing concurrency this way. One is algorithm have to be cooperative, and stop processing in the middle to let Flash do its thing. That means your algorithm normally have to be rewritten to conform with this approach. That can be particularly difficult for recursive algorithms. There needs to be more research done into how you might fix this with the callLater() technique. The biggest draw back is that we cannot take advantage of multi-processors. For all the code you write Flash runs on a single OS thread. This is a serious disadvantage for us going forward because as Actionscript developers we cannot access boosts in hardware performance as cores are added.

It's been a lot of information but hopefully you now understand the theory behind concurrency in Actionscript, and you have a new library that helps you optimize your code. You can access the source code here, and download the GreenThread's library here. I look forward to hearing about what sorts of long running jobs you create.

Full source code of the Mandelbrot set is here.

Download GreenThread's library here.