3.06.2014

Hacking Golang: Accessing Specific Value from Multi-Value Function Returns

In current version of Go (1.2), a multi-value returned from a function call can only be accessed by doing a multi-variable assignment:

b, c := F()

That works great.  However, there are times, instead of using a two-step approach (assignment, then access) to get to targeted values, I want to access the value inline using an expression, not an assignment.  Right now, however, Go does not support anything like that.  For instance
the following will blow up:

type Path struct{FileName string}
...
Path{FileName:path.Split("/a/b/c.d")}

The Go compiler will complain (paraphrasing here) about multi-value in a single-value context since path.Split() returns (string, string). This is as expected.  I started thinking, with a language as capable as Go, is there a way to express target a specific return value and mute others so the expression above would work?

A Workaround
So I started to research how the multi-value returned by a function is represented inside Go?  Is it an array, a slice, container, what?  And why does fmt.Println(path.Split(...)) works without any complaint from the Go compiler?  Looking for the answer to the ladder question in Go's source code for the fmt package revealed some interesting facts about the multi-values returned from a function call:
  • Multi-values returned from one function call can be passed in as multi-variable arguments to another function.  Meaning, func Split()(string,string) can be passed as argument to   fmt.Printf(a...interface{}) .  Inside, fmt.Printf(...) receives variable a as []interface{} from the compiler.
  • The compiler only does that trick (as far as I found) for function with varargs params.  For instance f.Foo(a []interface{}) cannot be called with b.Bar()(string, string) as argument.
  • In other contexts, you get big fat ugly failures:
    • a []interface{} := path.Split(...) // illegal, fails compilation 
    • var b []interface{} = path.Split(...) // fails with multi-value in single value context error
    • var c []interface{}; c = path.Split(...) // stil fails (as above)
So, that led me to come up with a function I call the mute function (represented by Greek letter mu, single character, simple, easy to use) as defined below.
func µ(a ...interface{}) []interface{} {
    return a
}
See code at http://play.golang.org/p/IwqmoKwVm-

The mu function exploits the automatic conversion from the compiler to convert a vararg parameters in the form of "x...interface{}" to a standard []interface{}.  The implication here is that I can use mu to wrap multi-values from a function call to access specific returned value by specifying an index as shown below.

Path{FileName:µ(path.Split("/a/b/c.d"))[0].(string)}

NOTE: since µ() returns []interface{}, you will need Type Assertion to get the actual value.

You can see an example of the mu function on the Go Playground at http://play.golang.org/p/IwqmoKwVm-

Thoughts
This is a hack by all means.  Go designers take pride in keeping things simple.  Having an extra assignment step to access multi-values returned form function calls is not that big of a deal.  On the other hand, while working with Go, you will inevitably run into cases where you wish there's a way to chain function calls as expressions.  But right now, when your function call returns multiple values, you have to do an assignment first to unpack that value.  Which feels like it is robbing Go of its functional qualities.

3.05.2014

Jumping to Go

This post is about my impression of Google's language Go (or Golang).  As the holidays approached last year (2013), I decided to pour my free time into Go.  I went through the Go documentation methodically to make sure I understand the motivation behind some of the decisions regarding the language.  That is very important since I carry the Java hammer around.

The result of all of my efforts culminated into the creation of an HDFS FileSystem client written in Go - https://github.com/vladimirvivien/gowfs.  That alone is a testament of the simplicity and approachability of the language.  Within 3 weekends (in Jan/Feb), I was able to put together a working WebHDFS client.

The followings are some points I wanted to share if you are thinking about jumping into Go yourself.

Jump into Go
Getting started in go is incredibly easy.  To get a taste of the language you can take the Go Tour (http://tour.golang.org/).  The tour gives you an online editor and server-side compilation service to explore several Go examples.  The tour covers all major language constructs and APIs to get you started and keep you interested.

Go!
When you are ready to do Go locally, the steps are just as easy.  First,  download and install Go for your platform (http://golang.org/doc/install).  Next, select your favorite text editor to start slinging Go code.

The Workspace
One important point that I enjoy (as a Java developer) is the notion of Go's workspace.  It is a location where Go keeps all source and binary packages organized in a specific directory structure.  This simplifies many aspects of working with Go including building, testing, and dependency resolution.  Because of this "Convention over Configuration" approach, one can build Go code without having to specify a long list path information (as is necessary using Java tools).  For instance, the following will resolve package dependencies declared in the code, compile, and run the specified Go file:

go run hello.go

If you are from the Java camp, this may be reminiscent of what the local repository does when using the Maven tool.  However, in Go, this approach is much simplistic and the workspace is a first-class citizen with the Go tools which understand how to deal with the code's organization.

For a full explanation of Go's code organization, check out 'How to Code in Go'.

No IDE Needed (Really)
If you are from the Java camp like myself, you may not believe it's possible to create complete projects without an IDE.  Well,  yes  really, no IDE.  I wrote the entire project using Sublime Text.  How is that possible you may ask?  Well, Go is a simple language with little surprises.  In addition, the Godoc (see below) tool provides easy access to the API reference documentation whether online or offline.

Non-Ceremonious Language
I won't go into too much about the language itself as it is well documented at the http://golang.org website.  Nevertheless,  Go takes a simplistic approach to language design by foregoing object-orientation.  Go programs are collections of first-class functions, simple  types, struct types, and Go's version of interfaces.  Functions can be graphed unto any types (function receivers) to further organize code as computational components.

By removing the need to declare a class,  a constructor, and main with program argument list, a compilable, functional Go code can be as simple as the following.

package main

import "fmt"

func main() {
 fmt.Println("Hello!")
}
Run the code at http://play.golang.org/p/duRF5gXJEP
Documentation! Documentation! Documentation!
Switching sweaty Steve Balmer's chant from "Developers! Developers! Developers!.." (http://www.youtube.com/watch?v=8To-6VIJZRE) to "Documentation! Documentation! Documentation! Docu..." would be an an apt way to explain the importance of documentation for the Go folks.  Here are a few documentation resources.

Go's Website (http://golang.org)
The website has all the docs you will ever need to get started with Go including API docs, videos, best-practice articles, etc.  I think the rarity of Go books has to do with the fact that the platform is so well documented on the site.  

The Godoc Tool
The Godoc tool further demonstrates the importance of documentation within Go all the while keeping things simple.  The following command will compile the entire golang.org website along with any source-level documentation in you own code and makes them available at http://localhost:6060/.

godoc -http=:6060
 
This makes all of Go's API documentation portable even when you don't have an Internet connection.

Godoc.Org (http://godoc.org)
This website aggregate Go projects documentation from major source code hosting services including GitHub, BitBucket, Google Project, and LaunchPad.  The content is indexed and made searchable.  It is a really neat tool.

Go Talks (http://talks.golang.org/)
Another fantastic source of information are the Go Tasks.  This section of the website is a collection of deck slides and accompanying source code. 

Fast Compile and Fast Tests
Again, being from the Java camp, I got used to my project taking minutes to compile and test.  I have used that time slice (between build start and build finish) to read entire books, catch up on twitter, or even pick up a new skillset.

The story is a bit different with Go.  I can compile and test my modestly-sized HDFS client project in milliseconds.  That also includes running about two dozen tests (and counting). "So what?" you may say.  Well, Go, it turns out, is the perfect platform for TDD.  Because testing is so fast, it almost feels like there is little to no cost to integrate unit testing into your code cycle.

Solid APIs
Within its short existence, Go has garnered an impressive list of APIs (http://golang.org/pkg/) that makes its completeness rival more established languages.  All major aspect of building systems are covered in one way or the other including
  • OS-level services, 
  • Cryptography,
  • IO
  • Network communication
  • HTTP
  • HTML
  • Database
  • Regex
  • Etc
Growing Adoption
The list of companies adopting Go is phenomenal.  Again, for such a young language, many companies are betting the future of vital components on a language that is still at version 1.x.  I won't enumerate any here, but a Google search will.  As the language improves over time, I am sure the trend of adoption will continue.

OK Annoyances
Yes, the Go platform is not without its annoyances. Some annoyances are due to my background from the Java world.  Other annoyances are due to the fact that the language is young and is still evolving with plenty of room for improvements.  However, none of the annoyances I have encountered are deterrents to keep me away.  The positives far outweigh the negatives.



4.22.2012

MockHttpServer for Testing Android HTTP Client

While developing an Android project, I needed to test an HTTP client service layer running in my application.  I wanted to ensure I unit tested features such as ability to send/receive headers, request http resources, and deal with HTTP response codes properly. However, I did not want to run my JUnit tests using an external HTTP server. This would create a dependency on yet another technology that I would have to launch with Maven. Furthermore, using an external server would make it difficult to validate unit test cases from the server side.

MockHttpServer

You may already know that Android's library comes with a version of Apache's HttpComponent. While Android promotes the use of HttpComponent for mostly client side development, the provided APIs include the server-side pieces as well. So, you can create your own HTTP server running on Android.
MockHttpServer is a wrapper around the HttpComponent's HttpService API. MockHttpServer provides a simple interaction point where developers simply provide an implementation of HttpRequestHandler.
The MockHttpServer handles the setup and thread management. The following shows how to use the MockHttpServer to enumerate and test the expected headers from the client:
        private final static int SVR_PORT = 8585;
        private final static String SVR_ADDR = "http://localhost:" + SVR_PORT;
...
	public void testHeaderEnumerations () throws Exception{
		// setup mock server and handler for given path
        MockHttpServer server = new MockHttpServer(SVR_PORT);
        server.start();

        server.addRequestHandler("/test1", new HttpRequestHandler() {

            @Override
            public void handle(HttpRequest req, HttpResponse rsp, HttpContext context) throws HttpException, IOException {
            	int headerCount = 0;
                String headers = "Accept Accept-Charset";
                HeaderIterator it = req.headerIterator();
                while (it.hasNext()){
                    Header h = (Header) it.next();
                    if(headers.contains(h.getName())){
                        headerCount++;
                    }
                }
                Assert.assertEquals(3, headerCount);
            }
        });

        // setup client connection
        URL url = new URL(SVR_ADDR + "/test1");
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.addRequestProperty("Accept", "text/xml");
        conn.addRequestProperty("Accept", "Audio/mpeg");
        conn.addRequestProperty("Accept-Charset", "utf-8");

        try{
        	conn.getInputStream();
        }finally{
        	conn.disconnect();
        }

        server.stop();
	}
The top portion of the code sets up the HttpRequestHandler implementation. This is where one specifies how the server would handle the incoming HTTP request. In the implementation above, we are enumerating and validating incoming HTTP headers. The second portion of the code sets up the client call to the MockHttpServer instance. Here, for simplicity, we are using an instance of HttpURLConnection to make the HTTP request to the MockHttpServer.

4.10.2012

Jmx-Cli: A Command-Line Console to JMX

One of the reasons I created the Clamshell-Cli Framework originally was to develop a text-based console for JMX.  Well, it is here. Introducing Jmx-Cli: Jmx-Cli a command-line interface console for JMX. It was developed using the Clamshell-Cli framework (http://code.google.com/p/clamshell-cli/) to prove that the framework was flexible enough to create a useful tool.

Installing Jmx-Cli

  1. Download the current distribution from https://github.com/vladimirvivien/jmx-cli/downloads.
  2. Unzip at the location of your choosing (jmx-cli-home).
  3. Cd to jmx-cli-home
  4.  From the command prompt, type > java -jar cli.jar
If all works ok, you should see:
    /#####                                /######    /##   /##
   |__  ##                               /##__  ##  | ##  |__/
      | ##   /######/####    /##   /##  | ##  \__/  | ##   /##
      | ##  | ##_  ##_  ##  |  ## /##/  | ##        | ##  | ##
 /##  | ##  | ## \ ## \ ##   \  ####/   | ##        | ##  | ##
| ##  | ##  | ## | ## | ##    >##  ##   | ##    ##  | ##  | ##
|  ######/  | ## | ## | ##   /##/\  ##  |  ######/  | ##  | ##
 \______/   |__/ |__/ |__/  |__/  \__/   \______/   |__/  |__/

A command-line tool for JMX
Powered by Clamshell-Cli framework 
http://code.google.com/p/clamshell-cli/

Getting Started

Using Jmx-Cli is straight forward. From the prompt type 'help' to get a list of available commands:
      exit       Exits ClamShell.
      help       Displays help information for available commands.
        ps       Displays a list of running JVM processes (similar to jps tool)
   connect       Connects to local or remote JVM MBean server.
     mbean       Creates a label for identifying an MBean
      desc       Prints description for specified mbean.
      list       Lists JMX MBeans.
      exec       Execute MBean operations and getter/setter attributes.

Command Hints

You can also get command hints from the prompt by pressing on the tab key. This will list all available combinations of input. You can narrow down the hints by typing a few letters of the command then press tab.

Connecting to a JVM

To connect to a running local JVM, do the followings:
1. Use the 'ps' command to get a list of JMV processes running locally:
> ps
1444	cli.jar
1740	org.apache.catalina.startup.Bootstrap
2. use the 'connect' command to attach to a running process
> connect pid:1740
The sequence above connects to a running instance of Tomcat server.

Listing MBeans

Once connected to a JVM, you can query available MBeans that are running in that JVM. Use the 'list' command to retrieve lists of MBeans.
> list filter:"Catalina:*"
The command above will list all MBeans running in the "Catalina" domain.  You can label the list for future reference by typing:
> list filter:"Catalina:*" label:true

MBean list [Catalina:*]

     [$0] Catalina:type=Server
     [$1] Catalina:realmPath=/realm0/realm0,type=Realm
...

Describing an MBean

Once you have a list, you can select a bean to describe. The following describes bean with label 'Catalina:port=8080,type=Connector':
> desc bean:"Catalina:port=8080,type=Connector"

MBean: Catalina:port=8080,type=Connector (org.apache.catalina.mbeans.ConnectorMBean)
Implementation of a Coyote connector

Attributes:
     port : int (rw) - The port number on which we listen for requests
     useIPVHosts : boolean (rw) - Should IP-based virtual hosting be used? 
     redirectPort : int (rw) - The redirect port for non-SSL to SSL redirects
     minSpareThreads : int (rw) - The number of request processing threads that will be created
     secure : boolean (rw) - Is this a secure (SSL) Connector?
     acceptCount : int (rw) - The accept count for this Connector
     maxThreads : int (rw) - The maximum number of request processing threads to be created
     URIEncoding : java.lang.String (rw) - Character encoding used to decode the URI
     modelerType : java.lang.String (r) - Type of the modeled resource. Can be set only once
     packetSize : int (rw) - The ajp packet size.
     processorCache : int (rw) - The processor cache size.
...
     xpoweredBy : boolean (rw) - Is generation of X-Powered-By response header enabled/disabled?
     stateName : java.lang.String (r) - The name of the LifecycleState that this component is currently in
     allowTrace : boolean (rw) - Allow disabling TRACE method
     useBodyEncodingForURI : boolean (rw) - Should the body encoding be used for URI query parameters
     secret : java.lang.String (w) - Authentication secret (I guess ... not in Javadocs)

Operations:
     destroy():void
     pause():void
     stop():void
     resume():void
     start():void
     init():void
The same command can be issued using the associated label of the bean from the list (see above):
> desc bean:$28

Execute an Operation

Jmx-cli lets you execute operation on your management beans as well. The next examples shuts down the Tomcat connector for port 8080:
> exec bean:"Catalina:port=8080,type=Connector" op:"stop"

References

1. https://github.com/vladimirvivien/jmx-cli - Jmx-Cli home page
2 http://code.google.com/p/clamshell-cli/ - Clamshell-Cli Framework web site

3.19.2012

Clamshell-Cli: A Framework for Creating Console-Based Shell Apps

Sometimes, you try to solve one problem and end up creating a solution for something completely different.  Such is the story of Clamshell-Cli.  On several occasions I came across the need of a purely console-based JMX shell.  Since I did some work in JMX before, I decided to create one that was usable and basically provided some of the more useful features of JConsole.  After looking around for a framework to build comand-line shells-based apps, it became clear that I would have to create one (yes, I considered OSGi and runtime implementations such as Felix,  but OSGi comes with its own set of constraints that solve problems other than what I wanted to solve).
So I created my own framework to build command-line shell tools. My requirements were simple: create a flexible, component-based, highly-extensible framework that would not get in developer's way.  The API had to be light, easy to learn, and the runtime had to be super simple to use.
Enter: Clamshell-Cli -  http://code.google.com/p/clamshell-cli/

Clamshell-Cli Features

  • Easy to get started
  • Small API footprint with low learning curve
  • Ability to build complex CLI tools such as REPL using plugin architecture
  • Simple component model that imposes little constraints on your design
  • The plugin architecture is designed for extensibility and feature-scalability:
    • If you don't like how the default implementation works, you can change it completely
    • Implement the components you want to change and your feature will be included next time the console is restarted
  • Each extension point is mapped to a Java type for easy implementation
  • Plugins are deployed as simple jar files
  • Support for input hints (tab-press at the console)
  • Support for input buffer history

The Design

The Clamshell-Cli API is kept simple on purpose.  It uses a plugin paradigm based on Java's ServiceLoader API.  The idea is not to make Clamshell-Cli a bloated piece of software trying to handle everything, but rather provide an extensible platform that lets developers build console-based tools by implementing pieces of functionality via plugins.
All major aspects of a working command-line shell are represented by statically defined interfaces. For instance, if you want to change the console prompt, you simply implement the Prompt interface to return the prompt you want displayed.  The Clamshell-Cli interfaces include :
  • SplashScreen - interface to render the first splash screen of console-app
  • ConsoleIO - interface to handle input and output streams
  • Prompt - interface to provide command prompt for the shell tool
  • InputController - Interface to handle text input at command prompt
  • Command - interface to handle action to be taken based on command input

The Default Runtime

When you download the default runtime, you get a basic shell environment with the following directory structure:
  • cli.config - Clamshell-Cli configuration file
  • cli.jar - the launcher jar file
  • clilib - lib files to boot Clamshell-Cli
  • lib - place your dependency jars here
  • plugins - location for Clamshell-Cli plugin jars
When you start the default runtime, you immediately encounter the SplashScreen plugin and the Pormpt plugin:
 
 .d8888b.  888                         .d8888b.  888               888 888
d88P  Y88b 888                        d88P  Y88b 888               888 888
888    888 888                        Y88b.      888               888 888
888        888  8888b.  88888b.d88b.   :Y888b.   88888b.   .d88b.  888 888
888        888     :88b 888 :888 :88b     :Y88b. 888 :88b d8P  Y8b 888 888
888    888 888 .d888888 888  888  888       :888 888  888 88888888 888 888
Y88b  d88P 888 888  888 888  888  888 Y88b  d88P 888  888 Y8b.     888 888
 :Y8888P:  888 :Y888888 888  888  888  :Y8888P:  888  888  :Y8888  888 888

Command-Line Interpreter

Java version: 1.6.0_22
Java Home: /usr/lib/jvm/java-6-openjdk/jre
OS: Linux, Version: 2.6.38-10-generic

prompt> _
The defaul runtime implements an input controller that interprets the input one line at time.  It assumes that the first word of the input is the command.   The controller then delegates the handling of the command to one of the registered Command instances.  For instance, if you type 'help' at the prompt, the InputController a) locates the Command mapped to command word 'help', then delegate handling of the command to the HelpCmd plugin:
prompt> help

Available Commands
------------------
      exit       Exits ClamShell.
      help       Displays help information for available commands.
   sysinfo       Displays current JVM runtime information.
      time       Prints current date/time

prompt> _

Using the API

To extend the default runtime and add your own command,  a developer would simply provide a plugin that implements the Command interface.  Package the class as jar that obeys the configuration rules of a Service Loader / Service Provider (see here), then place the jar in the plugins directory.  Next time the Clamshell-Cli console is restarted, the command would be available.  The following code shows how simple it is to create your own command using the Command interface. Class TimeCmd, shown below, implements the 'time' command as part of the default runtime::
public class TimeCmd implements Command {
    private static final String NAMESPACE = "syscmd";
    private static final String ACTION_NAME = "time";

    @Override
    public Object execute(Context ctx) {
        IOConsole console = ctx.getIoConsole();
        console.writeOutput(String.format("%n%s%n%n",new Date().toString()));
        return null;
    }

    @Override
    public void plug(Context plug) {
        // no load-time setup needed
    }

    @Override
    public Command.Descriptor getDescriptor(){
        return new Command.Descriptor() {
            @Override public String getNamespace() {return NAMESPACE;}

            @Override
            public String getName() {
                return ACTION_NAME;
            }

            @Override
            public String getDescription() {
               return "Prints current date/time";
            }

            @Override
            public String getUsage() {
                return "Type 'time'";
            }

            @Override
            public Map getArguments() {
                return Collections.emptyMap();
            }
        };
    }
}
A quick explanation of the code is in order:
  • Method execute() - invoked by the input controller instance when it detects the String time from the command-line. The method retrieves the IOConsole from the context object and use it to print the time. It returns null to the controller (indicating the command did not generate a result).
  • Method plug() - a lifecycle method that is invoked by the framework when the command is first initialized. For our example, there nothing to do.
  • Method getDescriptor() - returns an instance of interface Command.Descriptor which is used to describe the features and document the Command. For our example, the Descriptor interface is implemented anonymously with the following methods:
    • Method Descriptor.getNamespace() - returns a string identifying the command's namespace. This value can be used by input controllers to avoid command name collisions.
    • Method Descriptor.getName() - returns the string mapped to this command object. In our implementation, it returns "time".
    • Method Descriptor.getUsage() - intended to provide a descriptive way of using the command.
    • Method Descriptor.getArguments() - returns a Map containing the description for each arguments that may be attached to the command. This example uses none.

Clamshell-Cli Examples

The best way to learn how to use the Clamshell-Cli API is to download the source code and look at how the plugins are implemented. You can also check out:

References

https://github.com/vladimirvivien/jmx-cli - A JMX command-line tool (yes I did build it) - built using Clamshell-Cli (discussed in future post)
http://java.sun.com/developer/technicalArticles/javase/extensible/ - Article on creating extensible application using Java