The Generative Programmer

Yet Another Ambitious Project (YAAP)

Lately I have been working in Erlang (forgive me, Tom), and in particular on a distributed web-based system that exposes much of its functionality over a RESTful interface. The web server has been Yaws and the database Mnesia, both of which are also implemented in Erlang, and are quite complex and interesting in their own right. Using Erlang's OTP (Open Telecom Platform) framework it's relatively easy to create and configure complex, robust client-server/state-machine applications, taking advantage of Erlang's native support for massive concurrency, high-availability and fault tolerance. Anyway, building such a system gives you a sense of where the grunt work is, and grunt work is a great candidate for code generation to mitigate the problem of time-to-market. This time-to-market problem is an issue that is felt quite poignantly when you are working in a startup on borrowed time, as I am at the moment. :) So currently I am devoting a little R&D time each day to the development of a code generation tool for RESTful applications in Erlang, which I call "shenkuo." The reason for the name is two fold: 1.) I plan to release it as open source via the Erlware project (www.erlware.org), and 2.) they like funky and exotic names based on real people that have some (obscure or obvious) relevance to the functionality of the project. The reason for Erlang is that it rocks for any problem involving concurrency and network communications--I mean really ROCKS. (More on that subject will be forthcoming in my Functional Programmer blog.) Shen Kuo was an ancient Chinese polymath who, among other things, came up with the concept of "due north" (a great aid to the compass, by the way), described the first ideas behind movable type, and invented several branches of what we in the west today call the social sciences. Whatever do these wacky and diverse things have to do with what I'm building? Well, RESTful applications are all the rage in social networking sites, and they are used to do all kinds of various and sundry things in real world "Web 2.0" architectures. REST has attained something of the aura of "due North" as a philosophy behind web application design. So the name "shenkuo" for a code generation framework for RESTful applications based on Erlang/OTP seems as good as any. And it just so happens that my first such distributed application is in the sphere of web publishing, so the reference to movable type in Shen Kuo's curriculum vitae seems to close the deal for me. I will be posting more about this YAAP in this space as I work on it. One of its key requirements will be to generate client proxy libraries to consume the generated RESTful interfaces in addition to the interfaces themselves, and I'll be focusing in this space on .NET clients (particularly WPF and Silverlight), for those of you here wondering why Tom tolerates my presence on his Microsoft SOA blog site. :)

Quick Update

While I've been putzing around creating a slew of new blogs on this site, I have not lost sight of my recent promise to explore creating new programming languages on this blog.

Actually, though I've been rather busy and must confess to dropping the ball a little bit, I have gotten a clearer vision for the language I'm going to be implementing and blogging about here--and more to the point, how I plan to do it.

In a nutshell I'm going to create a Basic variant called Concurrent Basic. It will be written using F#'s language parsing tools, which I've been learning since getting my copy of Expert F# by Don Syme et al. It will mimic Erlang more closely and leverage everything at its disposal--F#'s Mailboxes, the CCR, and whatever hook or crook I can concoct to implement the message passing primitives that make Erlang such a pleasure to work with for inter-process communication.

I still plan to create a Ruby variant of Erlang that compiles to core Erlang, but that's another project for another day. I have too many projects going on as it is and a little focus might help me get some of them done... :)

 

Concurrent Basic and Emerld Overview

I have now officially two compiler projects underway based on Erlang concepts. I catalog them here because compilers are code generators and this blog is about generative programming. :)

The first is project, to be written in F# and its compiler tools, is to create a functional variant of the BASIC language that I call, conspicuously enough, "Concurrent Basic," targeting the .NET Coordination and Concurrency Runtime (or CCR). The second project, written in Erlang, is to create a version of Ruby, which I call Emerld, that compiles to "core Erlang" and executes in the Erlang runtime. 

Both languages will bring to their "parents" functional programming features they currently lack, like pattern matching, single-assignment variables, and send-recieve message-passing primitives. They will offer some of the syntax sugar of both languages that Erlang currently lacks and be designed to take advantage of their respective platforms in unique ways.

I'm not so sure about the feasibility of efficient lightweight threading on .NET, so initially Concurrent Basic will only offer simplified access to .NET's threading capabilities and a runtime, which I call Muse, to facilitate the integration with the CCR. I will however spend some time on this blog trying to understand how, precisely, lightweight threading on .NET might be implemented---but not so much as to impede progress on other value-add feature of the language--namely the messaging primitives.

The fact that (incredible, amazing, awe-inspiring) F# developers didn't tackle the lightweight threading feature of OCaml (which is an alternative to the heavier-weight threading capability that it offers on Unix), and instead spin the lack of it as some kind of advantage to the language's interoperability story, I take to be a Big Clue.

At this stage of things I am in the process of defining the requirements and the specs. This is going to be a long process, since I'm not exactly Mr. Focus these days when it comes to my (too?) many "on the side" R&D efforts--but hopefully that will improve as my personal and professional situation evolves.
 

 

Introducing Muse and fxBASIC

Percolating in my mind for many a moon has been an idea of creating a language/runtime that I have called by various names; Muse has been its most recent identifier, named after the divine inspiration of Homer's great epic poem, The Odyssey. She is invoked in its first line:

"Tell me, Muse, of the man of many twists and turns, who wandered far and wide after he sacked Troy's holy citadel; he saw the cities of many men, and knew their mind."

Like any germ of an idea, it began with a high-level goal. The main idea was a middle-tier language that could "do" multiple paradigms, not unlike Mozart-Oz, but on a modern platform, in particular, .NET. In ancient Greek, the word for "man of many twists and turns" in the above translation, is polytropos... thus, Muse was to be the polytropos of .NET languages.

Lately, as I have studied trends in the industry related to concurrent development and learned a lot about how functional languages like Haskell and Erlang are impacting the .NET platform, a variation of the theme has appeared. I still plan to call the runtime Muse, but the language is going instead to be called Functional BASIC, or fxBASIC for short.

People learn what they don't know through what they do know, and considering the number of people who know BASIC in one form or another, I decided the language itself should look and feel as much like it as possible, to aid in adoption. Long term I want this new flavor of BASIC to become MultiBASIC, in recognition of its longer-term multi-paradigm ambitions. Initially I plan to focus fairly strictly on implementing the functional programming paradigm, so there's no point in false advertising at this point.

fxBASIC's goal will be to support all of the semantic capabilities of Erlang, a language that has more than proven its wares in industry--specifically, the very demanding telecommunications industry. Practically, this means that Muse is really all about building distributed applications with the following characteristics:

• They are highly available;
• They are widely scalable;
• They are inherently concurrent, using a message-passing approach built into the runtime rather than the inherently and dangerously complex shared-state approach baked into most languages;
  
• They are robust, exhibiting the highest levels of fault tolerance imaginable;
• They are service-oriented.

In other words, the language exists in the service of a platform that out-of-the-box already provides these software qualities. Just add water. Or rather, in this case, just add business logic code.

One question that I'm still trying to answer for myself is "Why .NET?" Why not Java? Why not compile to the Erlang runtime? All of these are viable options.

My main motivation for .NET is simply that I prefer it and know it better over the other alternatives. BASIC has a richer history on the Windows development platform, and language interop in .NET was something that was built into it from the beginning. All of this sounds like something a wise man might want to leverage. Moreover, new developments like the functional language features and compilers creeping into .NET and new SDK's like the Concurrency and Coordination Runtime (or CCR) make it a most attractive target.

Java is also a very attractive target given its ubiquity, and someday I may create a version for the JVM. But in the Java world, enterprise development is pretty much standardized around J2EE. My idea for Muse, if it has any merit, could do more good in the .NET world, it seems to me, since the story there is less standardized, and therefore more open to new approaches.

Why not make fxBASIC a front end to Erlang, generating BEAM files instead of .NET assemblies? Again, it's a very attractive idea, and one I haven't ruled out. I'm still considering that idea as a first step.

I have another motivation for this project, which is to piece together tidbits of strategies I've learned about compiler construction and use them in this effort. I am enamored of the ideas behind the GOLD parsing system, for example, which treats parse tables as data, and TreeCC, which allows for aspect-oriented construction of ASTs. Somehow I want to try to tie these ideas together into a generic compiler-compiler project. I have christened this one the "LILO DE", as in "Light-weight, Integrated, Language-Oriented Development Environment".

In future posts I'll be describing the spec for Muse and fxBASIC, as well as relating my adventures as I attempt to design and implement them. 

Delegates, Lambdas and Functors, Oh My!

As the .NET platform's Common Language Runtime evolves, it is acquiring quite a toolbox of baked-in features from which language implementors and systems designers can choose to achieve both elegance and efficiency--goals which are ordinarily at odds.

Two of those features--delegates and lambdas--have an interesting conceptual relationship to their low-tech brethren, plain ol' function objects (a.k.a., functors). Functors are more a pattern than a language feature, and lack the sex appeal (not to mention special compiler support) that lambdas and delegates enjoy on .NET. Nevertheless, they have a place in the designer's bag of tricks and their value in the context of generic programming should be well understood.

In the next several posts I'll explore these interesting beasts and show the power of each one and how to leverage it to create elegant and efficient software solutions. You'll come away understanding at a pretty low level how each one works, why one might want to use it, and what its trade-offs (that is, performance penalties or runtime constraints) are.
 

Adding Value, Not Overhead

As developers of frameworks, we know all too well that the fine line between adding value and adding overhead is drawn by user perceptions.

If we do our job well, and make their task easier without senselessly sacrificing performance, our customers view us as adding value. If not, we are viewed as an extra layer of overhead.

How do we assure them we are adding value?

Well, for one thing: we make it clear what we are doing for them--more precisely, we make it clear that what we are doing for them, code-wise, is nothing more than what they would otherwise be doing themselves. That is, we are saving them time, nothing more. This means being transparent.

The other thing we can do is publish our metrics. If they see we at least took the time to measure our performance, they'll be willing to see if next iteration we don't squeeze some more clock cycles out of our code. (This is clock-cycle-squeezing they don't have to do.)

I mention all of this because the perception of adding value vs. adding overhead is the key perception that needs to be managed if a framework project at a big company is to survive long-term.

There will always be a Performance Uber Alles crowd that will nitpick at your every attempt to add value, and what you need to do is accept their criticism and deal with it head on by being transparent about your approach, accepting other ideas for how to do whatever it is your framework does (this means in practice breaking solutions up into plug-n-play policies, so the act of testing which is really best is as clinical and emotionless as possible), and going the extra mile to show value and performance-consciousness in whatever overhead your framework adds.

Don't expect everyone to be rational about it, but most people will value your effort, and that's all that really matters. 

Technology Goals for 2008

Life has taught me, and my wife's wisdom has confirmed, that if you start early, you just might finish on time. Since I've never tried it before, but have been victimized many times by its converse, I'm taking this on faith, guided by experience. ;)

In that spirit I am thinking now about my goals for 2008. This is unprecedented: I usually give no thought to this matter until New Year's Eve, when we're more-or-less obligated to concoct some "resolutions" we fully intend -- or should I say pretend? -- to fulfill in the new year.

Since this is a technology blog, I'll try to keep my initial list to technical stuff.

1. Master parallel programming.

I have several motivations for this one. First, it's all the rage. With new, ever more powerful multi-core processors coming out with greater frequency, it seems like the unavoidable topic of the day.

Second, it's a huge differentiator: Threading is hard. It's an impressive differentiator on the job, and useful for many otherwise hard data- or computation-intensive problems, to boot.

Third: this means it's also interesting. People who say there's nothing left that's interesting in programming haven't thought deeply about parallel computing.

So, concretely, this general desire will be measured against the following specific goals for 2008:

1.1. Contribute something to Boost.Threads, and the overall discussion of standardizing parallel programming in C++.

1.2. Create a Boost-friendly library for constraint programming that uses the open source version of Intel's Threading Building Blocks (or Boost.Thread), and submit it for interest/review to Boost. Call it Boost.Constraints or something. I've found this subject and programming paradigm interesting ever since I stumbled on Mozart-Oz.

1.3. Use it to do something interesting with the new time_series library in Boost, like say a stock analysis and simulation system. Another possible application: something to do with tropical forecasting, another pet interest of mine. I have a strong desire to write simulation systems and finance and meteorology are domains that have always fascinated me. Unfortunately, I don't know enough about them so implicit in this goal is to learn enough about one (or both) of them to pull this goal off so that it passes as a serious attempt by somebody who actually knows what they're doing.

 (When it comes to learning new things I have always been immodest in this way...so far it hasn't hurt me.)

2. Solidify my ideas regarding good SOA design.

I'm throwing this in to mollify Tom Fuller, my good friend and colleague, who also happens to host this blog site, and is something of an apostle of the SOA creed. :)

Seriously: I actually do have a few ideas rumbling around in my head, mostly pertaining to leveraging generic programming to alter how we layer our architecture cake to overcome both conceptual and performance-related complexities that the current state of the art leaves open to criticism (as Tom and I know from experience at work).

Concretely, then:

2.1. Totally and completely grok .NET's WCF framework. Create a constraint service based on whatever comes out of goal 1.3 (above) that proves the concept in a way that leaves no doubt about the value or viability of SOA.

2.2. Commercialize and market whatever comes out of this in terms of framework code. OK, this might have to wait for 2009, but it's listed here as a nice-to-have for 2008.

3. Generalize my practice management domain experience.

I have been working on the side for years on a project with my father, who is a Certified Financial Planner, to create a software system for practice management specific to financial planners. This experience has given me a deep appreciation for the productivity issues faced by small practices and I wish to "de-verticalize" it and create a framework of software factories for the just-in-time re-verticalization of it to other kinds of practices. 

Again, concretely:

3.1. Create a new .NET framework for creating new practice management solutions. I call it the PMF (or "Practice Management Foundation", in the spirit of Redmond's current naming convention for frameworks). Make sure it uses all the patterns & practices stuff from Microsoft, including Enterprise Library, Guidance Automation Toolkit and Guidance Automation Extensions, and Software Factories.

3.2. Re-write my financial planning system on the PMF.

3.3. Show it off at Tech Ed or some other public venue next year.

That's probably enough technology goals for two or three or four years, but at least they're documented. :)

 

Testing, Testing, 1, 2, 3

This quick post is just to announce my new blog, "The Generative Programmer," which will examine generic and generative programming techniques and technologies from a framework developer perspective, with particular focus on how they affect enterprise architecture.