Monday, April 27, 2009

Installing perl modules in OSX

This should not have been hard but it ended up being a big pain for me so I thought I would document the process for installing a perl module on Mac OSX. It ended up being tricky because there were two ways I found to do it. I will first list the way I ended up NOT using.
  1. Install Xcode - this is required for installing and running darwin ports (a.k.a macports)
    This is a disk image with a binary installer which is about 900 MBs in size and takes a few minutes to install, it also requires an apple developer connection membership before you can download it
  2. Install DarwinPorts - this is required for installing yum
    Download the disk image and run the binary installer, you have to fill in your name and email address to download
    Run this command as root once you finish running the installer:
    sudo port -d selfupdate
  3. Install Yum - this is required to install the perl module
    Run this command (takes a long long time) to as root to cause darwin ports to install yum:
    sudo port install yum
  4. Use yum to install the perl module
    yum -y install perl-Frontier-RPC
After doing all this and facing failure I searched around more and found the instructions here:
http://triopter.com/archive/how-to-install-perl-modules-on-mac-os-x-in-4-easy-steps/
For the same module (Frontier) the commands ended up being (after installing Xcode, much like linux):
sudo su
perl -MCPAN -e shell
install Frontier::Client
This worked out a lot better for me (though the process took about 15 minutes total). Make sure you run this as root since it will produce lots of fun failures otherwise.

Thursday, April 16, 2009

Configuring Jetty in the pax-web OSGi bundle

It took awhile for me to figure out how to configure Jetty as deployed in the pax-web web service bundle. It took even longer to figure out how to enable AJP in Jetty using the bundle fragment. Unfortunately, the steps on the pax-web site are incorrect (now fixed: 2009-04-17) for the current version of pax-web (0.6.0). Hopefully this will save someone else from having to go through this "fun".

Note that this will only work in pax-web version 0.5.2 or higher and requires an OSGi container that supports OSGi bundle fragments. I was doing this in Felix 1.6.0 (inside Sling).
  1. Checkout my bundle fragment source code
    https://source.caret.cam.ac.uk/camtools/trunk/sling/sling-jetty-config
  2. Edit the jetty.xml file to suit your taste
    (the one in there enables AJP)
  3. Build using Maven 2
    mvn clean install -Pajp
    (Leave off the -Pajp if you are not enabling/using ajp)
  4. Install the bundle fragment into your OSGi container using whatever mechanism you are used to. It tends to work best to install it with the same level as the bundle it is being used with.
  5. Restart the OSGi container
    (this is not always required but I find it tends to work a lot better if you do)
Unfortunately, I could not find a very good way to verify that this works other than putting a breakpoint at line 66 of /pax-web/bundle/src/main/java/org/ops4j/pax/web/service/internal/JettyServerImpl.java. You should see the jetty.xml file get loaded (resource will be non-null). You could also just check to see if Jetty is behaving as you would expect with the changes.

If there are problems (the fragment seems to have no effect) then here are a few debugging steps that may work:
  1. If you are using felix then you can run the resolve command from the felix command shell for your fragment. If you get nothing then you are probably good to go, if you get a failure then the Fragment-Host: org.ops4j.pax.web.pax-web-service value probably does not match the one in your container. Make sure the value is the same as the symbolic name of your installed pax-web-service-*.jar.
  2. Try making sure your jetty.xml actually works with Jetty. There are instructions on the Jetty website.

Wednesday, April 08, 2009

Filtering in FindBugs

Here is an example findbugs exclusion filter which is generally useful for most projects. I could not seem to find a good example of something general anywhere so hopefully this will be a helpful reference to others. If you placed an xml file like this into your project in eclipse you can configure findbugs to use it as an exclusion filter.

Here is the sample xml (escaped):
<FindBugsFilter>
<!-- filter out test classes with medium (2) or low (3) warnings -->
<Match>
<Or>
<Class name="~.*\.AbstractTest.+" />
<Class name="~.*Test" />
</Or>
<Bug category="PERFORMANCE,MALICIOUS_CODE,STYLE,SECURITY" />
</Match>

<!-- remove the rules which require all passed vars to be immutable -->
<Match>
<Bug code="EI,EI2" />
</Match>

<!-- Filter out certain categories of bugs -->
<Match>
<Bug category="STYLE" />
</Match>
</FindBugsFilter>

To make this work in eclipse (with findbugs plugin installed) just do the following:
  1. Create an xml file in your project with the contents from above
  2. Right click the project name and select Properties
  3. Choose FindBugs
  4. Check the box marked Run FindBugs automatically
  5. Click on the Filter files tab at the top
  6. Click the Add button next to Exclude filter files:
  7. Select the xml file you created
  8. Click OK
You can adjust the filter using the findbugs guide and the full list of bug types and categories is available.

Tuesday, April 07, 2009

Developing with OSGi and Apache Sling

These are notes I compiled while working on a project which uses OSGi (Apache Felix) and Apache Sling as the foundational framework. Sling is essentially Felix with JCR and templating bundles installed and a REST bundle. I will tell you how I got going with the basics of OSGi and Sling and some of the issues I ran into. First a few important links:
Before we really get started I want to mention something about OSGi in general. OSGi is basically a system for making Java code modular and handling isolation of dependencies (no slop allowed). This is done via a lot of really complex classloader graphing. A chunk of code in OSGi is called a module and a system will typically be made up of many modules. Inside modules there are packages which can be used internally (private), exported so others can use them (this is how services dependencies are shared), and imported (this is how a bundle would get the dependencies it needs to use an external service). OSGi completely manages the lifecycle of the the bundles in the system. Bundles define an activator (implements BundleActivator) which allows the developer to control startup and shutdown actions and register services. The modular nature of OSGi means services could go away at any time (or may not be started when your bundle is starting). Because of this, bundle code should be written to expect that a service might not be ready to use when it is starting. Waiting for services to be available is bad also because OSGi activators are expected to be quick so delaying them will cause failures. This generally means using the listeners and trackers provided by OSGi. The practice is called the Whiteboard Pattern (basically inverted listeners, those familiar with EntityBroker/EntityBus will recognize this as how providers work). These links are a good intro and you will want to be familiar with this conceptually before you attempt to create your first real bundle (helloworld not included):
http://www.theserverside.com/tt/articles/article.tss?l=WhiteboardForOSGi
http://www.knopflerfish.org/osgi_service_tutorial.html

Here are the steps I took to get Sling up and running (more details on the Sling site):
  1. Checkout the source code using subversion:
    svn co http://svn.apache.org/repos/asf/sling/trunk sling
    (I checked out revision 758703 since the current revision was not working)
  2. Build the source using maven 2:
    cd sling
    mvn clean install
  3. Run Sling using the built in Jetty webserver (as an executable jar):
    java -jar launchpad/app/target/org.apache.sling.launchpad.app-5-SNAPSHOT.jar
    (Note: the version of the jar will change over time)
    (you can also run drop a sling war into your own servlet container)
    (you can change the port by adding "-p #" (where # is the port number like 9090))
    (you can cause all logs to go to the console by adding "-f -")
    (you can change the logging level of sling by adding "-l #" (where # is 0-4 with 4=debug))
  4. Test out sling using your web browser:
    Go to http://localhost:8080/
    Click on the console link (login as admin/admin)
Now, if you are going to do anything in Sling (or Felix) at all you are likely to need to debug it, so here are the steps to use a debugger with Sling (generally applies to any java app really).
  1. Run Sling with debugging options enabled:
    java -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=9001,server=y,suspend=n -jar launchpad/app/target/org.apache.sling.launchpad.app-5-SNAPSHOT.jar
    (You should see this in the logs: Listening for transport dt_socket at address: 9001)
  2. Attach the debugger from eclipse (or use whatever debugger you like):
    Run -> Debug Configurations
    Right click Remote Java Application -> New
    Connect tab: Set Port to 9001
    Source tab: Add the imported Sling code and your OSGi bundle project
    Click the Debug button at the bottom
    (You should not get an error if everything connects up)
  3. Place a breakpoint in Runtime class on the gc method
  4. Click on System Information and then the Run button next to Garbage Collection
    (the debugger should pick up the call and pause the JVM at the breakpoint)
  5. Use the debugger to trace through if you like (but bear in mind that the admin console is not actually part of Sling, it is part of Apache Felix so you will need the source for Felix)
Now you probably want to deploy a bundle so here are the steps to make a really simple one you can build and deploy into sling. I use the maven-bundle-plugin for this example, see the links for more details about it:
Here is the part of the maven pom that configures the bundle manifest:
<dependencies>
<!-- OSGi -->
<dependency>
<groupId>org.osgi</groupId>
<artifactId>osgi_R4_core</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.osgi</groupId>
<artifactId>osgi_R4_compendium</artifactId>
<version>1.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Bundle-Activator>org.azeckoski.osgi.SampleActivator</Bundle-Activator>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
  1. Checkout the sample bundle code:
    svn co https://source.sakaiproject.org/contrib/caret/osgi-sample/tags/sample-1.1/ osgi-sample
  2. Build the bundle using maven 2:
    cd osgi-sample
    mvn clean install
  3. Access the Felix console for bundles (included in Sling):
    http://localhost:8080/system/console/bundles
  4. Browse and select the bundle (target/sample-1.1.jar)
  5. Click Install or Update and then Refresh Packages
  6. Scroll down to the Sample OSGi Bundle
    If it is listed then you have a properly installed bundle!
  7. Click the Start and then Stop buttons to the right of the bundle
  8. You should see something like this in the logs:
    Sample starting at: Mon Apr 13 13:15:27 BST 2009
    Sample stopping at: Mon Apr 13 13:15:29 BST 2009
  9. Now click Start to make sure the bundle is running
  10. Restart Sling (use the admin console or just kill it and rerun the command)
  11. You should see that the bundle is running (it was started automatically) and in the startup logs the Sample starting... should appear
Here is the (very) simple code from the activator class (comments removed). It just prints out a message when the bundle starts and another when it stops.
import java.util.Date;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

public class SampleActivator implements BundleActivator {
public void start(BundleContext context) throws Exception {
System.out.println("Sample starting at: " + new Date());
}

public void stop(BundleContext context) throws Exception {
System.out.println("Sample stopping at: " + new Date());
}
}
Congratulations. You have installed you first bundle. If you want to try out the debugging you can put breakpoints in the start and stop methods. It only took me a couple days to get to this point which I think is not great so I hope this tutorial has really sped up the process for you.

Now for the advanced bit. Creating your own bundle. I have a somewhat realistic set of bundles where one depends on another and one uses the http.service (and other bundles) so chances are you won't face things that are too much more complex than this.
There are a few things that I learned going through this that were not all that apparent so hopefully I can save you some digging with these pro-tips:
  • OSGi start order does not guarantee service start order - The order which bundles start does not guarantee that their services will also be started. In fact, relying on this is a mistake since OSGi is meant to be modular and services could go away at any time (see the note at the top about OSGi and whiteboard pattern). Code should be written to expect that a service might not be ready to use when the bundle starts. Waiting for services to be available is bad also because OSGi activators are expected to be quick so delaying them will cause failures.
  • Bundles should import the packages they export - This is not obvious but it makes sense when you think about it. Since there may be other bundles of higher rank that are exporting packages that match yours, you should use the highest priority, even in your own bundle. This behavior is actually taken care of my most bundle making tools automatically so just don't be surprised to see your packages in the imports.
  • javax packages will need to be imported - Only java.* is available to your bundle by default so all the dom,xml,net,wtc. stuff from javax has to imported. If you do not import it and happen to use it then your bundle will fail at runtime. Luckily, most OSGi installations have a core which already exports most of this stuff so just put it into your import-package as optional like so (just an example):
    org.w3c.dom;resolution:=optional,org.xml.*;resolution:=optional,javax.*;resolution:=optional,*
  • Services in the manifest are deprecated - As of OSGi R4, listing services (using export/import-services) is deprecated. All exports/imports are handled as packages now and the registration of the services is done in the activator.
  • Felix shell access inside Sling - There is a felix remote shell which can be used to access the felix shell service. It can be installed by just installing a couple bundles (shell, shell remote). This will allow you to run felix shell commands inside Sling via telnet.
I strongly suggest you use code that you know works already when building your first functional bundle. Pick something that does not have a lot of dependencies as well. For the following tutorial steps I will take you through the download, build, activate, and test process with my bundles and then look at some of the code.
  1. Checkout the sample bundle code:
    svn co https://source.sakaiproject.org/contrib/caret/osgi-eb/tags/eb-1.0/ osgi-eb
  2. Build the bundle using maven 2:
    cd osgi-eb
    mvn clean install
  3. Use the Felix console for bundles to install and start the eb-services and eb-rest bundles:
    (should be eb-component/target/eb-services-1.0.jar and eb-webapp/target/eb-rest-1.0.jar)
  4. Verify they started by refreshing and making sure they appear to be running:
    For some reason there are no errors logged to the console by default when a bundle fails to start and no messages appear in the web interface. As a result you simply get a silent failure where it seems to have worked but is actually just installed and inactive. The bundle should say Active next to it (make sure you refresh).
    NOTE: The errors will be logged into SLING_HOME/logs/error.log
  5. Go to the URL that is setup for the bundles (http://localhost:8080/eb), it should load up the description page for the rest webapp portion
For this sample code I have used the OSGi ServiceTracker and ServiceListener with a bit of utilities scaffolding (ServiceTracker2 and ServicesTracker) to setup my bundles and services in the whiteboard pattern. While this works fine for a smaller use case, I would suggest looking into OSGi DS (Declarative Services) and the Maven SCR Plugin when working with a large number of bundles and services.

Here is the start method from the EB services activator showing an example of how services are registered and tracked. This also demonstrates one way to track things which are used by our services. Pre-OSGi Java programming might require the EntityProvider to be manually registered with the EB system using a register method. By leveraging OSGi's ability to lookup services, we can instead simply have developers create their providers and register them with OSGi as services. Then the eb-services bundle picks up the registered providers and handles the registration automatically.
public void start(BundleContext context) throws Exception {
System.out.println("INFO: Starting EB module");

// Create the EB core services
coreServiceManager = new EntityBrokerCoreServiceManager();

// register trackers to handle the optional services
eipTracker = new ServiceTrackerPlus<ExternalIntegrationProvider>(context, ExternalIntegrationProvider.class) {
@Override
protected void serviceUpdate(ServiceEvent event, ExternalIntegrationProvider service)
throws Exception {
coreServiceManager.getEntityBrokerManager().setExternalIntegrationProvider(getService());
}
};

// register a tracker for the entity providers (will find them as osgi services and handle registration)
providerTracker = new ServiceTrackerPlus<EntityProvider>(context, EntityProvider.class) {
@Override
protected void serviceUpdate(ServiceEvent event, EntityProvider service)
throws Exception {
EntityProvider provider = getService(event);
if (event.getType() == ServiceEvent.UNREGISTERING) {
coreServiceManager.getEntityProviderManager().unregisterEntityProvider(provider);
} else {
coreServiceManager.getEntityProviderManager().registerEntityProvider(provider);
}
}
};

// look up a service we optionally want to use but do not require
ServiceTrackerPlus<DeveloperHelperService> dhsTracker =
new ServiceTrackerPlus<DeveloperHelperService>(context, DeveloperHelperService.class);
DeveloperHelperService dhs = dhsTracker.getService();
System.out.println("DeveloperHelperService is currently: " + dhs);

// register the core services from this bundle
ebRegistration = context.registerService( EntityBroker.class.getName(), coreServiceManager.getEntityBroker(), null);
ebManagerRegistration = context.registerService( EntityBrokerManager.class.getName(), coreServiceManager.getEntityBrokerManager(), null);
ebProviderManagerRegistration = context.registerService( EntityProviderManager.class.getName(), coreServiceManager.getEntityProviderManager(), null);
ebEVAPManagerRegistration = context.registerService( EntityViewAccessProviderManager.class.getName(), coreServiceManager.getEntityViewAccessProviderManager(), null);
ebHSAPManagerRegistration = context.registerService( HttpServletAccessProviderManager.class.getName(), coreServiceManager.getHttpServletAccessProviderManager(), null);

System.out.println("INFO: Started EB module and registered services");
}

In this code example, we see the code to handle the start of the EB rest activator. This allows the eb-services bundle to be stopped and started without having to manually do anything to the rest bundle (whose services depend on the ones in the eb-services bundle). It will shutdown the rest services until the core services it requires are available again.
public void start(BundleContext context) throws Exception {
System.out.println("INFO: Starting EB ReST module");
// initialize tracker
this.requiredServicesTracker = new ServicesTracker(context, HttpService.class, EntityBrokerManager.class) {
@Override
protected void requiredServicesReady(Object service, ServiceTracker2 changed,
Map<String, ServiceTracker2> serviceTrackers) throws Exception {
// required services are ready so startup
startServices(getService(HttpService.class), getService(EntityBrokerManager.class));
}
@Override
protected void requiredServicesChanged(Object service, ServiceTracker2 changed,
Map<String, ServiceTracker2> serviceTrackers) throws Exception {
// required services changed so stop and restart
stopServices(getService(HttpService.class));
startServices(getService(HttpService.class), getService(EntityBrokerManager.class));
}
@Override
protected void requiredServicesDropped(Object service, ServiceTracker2 changed,
Map<String, ServiceTracker2> serviceTrackers) throws Exception {
// required services gone so shutdown
stopServices(getService(HttpService.class));
}
};

// optional services trackers
dhsTracker = new ServiceTrackerPlus<DeveloperHelperService>(context, DeveloperHelperService.class) {
@Override
protected void serviceUpdate(ServiceEvent event, DeveloperHelperService service)
throws Exception {
DeveloperHelperService dhs = getService();
if (servlet != null) {
servlet.updateDHS(dhs);
}
}
};
// TODO need to handle the case of the hsapm changing?
hsapmTracker = new ServiceTrackerPlus<HttpServletAccessProviderManager>(context, HttpServletAccessProviderManager.class);

// register the servlet if services are ready
this.requiredServicesTracker.startCheck();
}

Here are some more links to OSGi materials that may be helpful:
http://neilbartlett.name/blog/osgi-articles/
http://www.osgi.org/About/HowOSGi

NOTE: Updated for the graduation of sling and changes in URLs that resulted

Monday, April 06, 2009

High Quality Javascript

Writing good javascript is still more art than science, but there are a growing number of tools out there to help those who are less artsy. I have tried to compile the ones that I use and a few tips which I think are helpful.

jQuery (http://jquery.com/) - a great javascript framework built for developers
If you are still writing javascript without using a framework like jQuery then you are punishing yourself. Stop that! This framework makes javascript work like it probably should have anyway and it is very reliable and widely used. It helps protect you from browser incompatibilities and makes ajax very easy. There are a large number of addons and extensions which provide flashy widgets and things to make your UI look snazzy.

JSLint (http://www.jslint.com/) - this website / tool is similar to findbugs for java and helps with correctness and valid code practices (NOTE: Can enable JSlint in Aptana Studio, see tips below)
JSLint is a JavaScript program that looks for problems in JavaScript programs. Just paste your javascript file into the box and click JSLint. I use the following options (good starting point if you are not sure what to use):
Assume a browser, Disallow undefined variables, Disallow leading _ in identifiers, and Disallow == and !=
You should strive to have no errors indicated. I recommend you avoid the Strict white space and Require parens around immediate invocations options as these are more likely to cause frustration than be helpful. JSLint will also provide you with a list of all functions, members, and Globals in your code which is helpful as a reference.

jqUnit (http://code.google.com/p/jqunit/) - a test writing framework (like jUnit for java)
This is a test framework which allows a developer to write tests which exercise and validate their javascript code. It is compatible with JSUnit but has special handling for jQuery and since you are using jQuery anyway you may as well get the benefits. The tests run in the browser and normally are accessed by loading a page (e.g project/test.html). I won't go into the reasons why you should always have unit tests but if you want high quality code then they are not optional. This will have a huge impact on the reliability and change tolerance of your code and since JS code in general tends to change a lot this is critical.

Fluid Infusion (http://fluidproject.org/products/fluid-infusion/) - framework for accessible javascript
The fluid project is trying to ensure that javascript enabled pages are accessible to everyone. They have a framework which includes widgets which are specially designed to be accessible and also have a lot of documentation for developers.

Aptana Studio (http://www.aptana.com/studio) - IDE for javascript, HTML, DOM, CSS
This is an eclipse based IDE for the web. It provides code completion, formatting, validation, and the things you might expect to get from an IDE. This makes writing javascript a heck of a lot easier and it looks better than textedit/notepad. I just use the eclipse plugin rather than the full product now and if you are web designer or just trying it out then the full option is probably best (as it seems it does not uninstall from eclipse cleanly).

General Tips:
  • Always namespace your javascript functions and vars - http://www.dustindiaz.com/namespace-your-javascript/
  • Write unobtrusive javascript (this basically means avoid writing code in your html except for a few lines at the bottom to run scripts and pass in values as needed) - http://www.onlinetools.org/articles/unobtrusivejavascript/
  • Always use var in front of variables, if you don't they will become globals and when you are namespacing (which you should be) this is considering leaking
  • Declare globals at the top of your JS file in a comment like so:
    /*global jQuery, myGlobal */
  • Always refer to globally used variables at the top of your script so it will fail if the global is missing. This will make JSLint happy and keeps things from appearing to work when globals are not available. Example:
    var $ = $ || function() { throw "JQuery undefined"; };
  • Check to make sure you found something when you use jQuery selectors (if you always except to find something). This protects you from thinking you have actually found something based on an id when there is nothing by that id available. Example:
    var adhocArea = $("#"+adhocAreaId);
    if (adhocArea.length > 0) {
    // do something
    } else {
    throw "failed to find thing with id: "+
    adhocAreaId;
    }
  • Use script blocks to initialize javascript when the DOM it is working with has loaded rather than always (ever?) using onLoad. This will cause the javascript to start quicker on slow loading pages and avoid ugly issues like resetting selections made by the user or clearing fields which had loaded before the whole page loaded and the onLoad executed. For example, if you are adding a simple numeric validator to a field, you should place the script tag to load the JS (from a namespaced variable in a separate file) after the form. This way the script executes as soon as the DOM it will act on is loaded.
  • Be familiar the concept of closures in javascript - http://www.jibbering.com/faq/faq_notes/closures.html
  • Quirksmode is a good reference for javascript cross browser compatibility
  • Enable JSLint validator in Aptana Studio: Preferences -> Aptana -> Editors -> Javascript -> Validation -> Check JSLint Javascript Validator
It is probably a good idea to glance at AJAXian every once in awhile as well.