Refactoring: Biting off more than you can chew?

So as most can assert from reading my blog I have developed a stamp tracking software "suite" of tools. The core of this was based off a JPA implementation which was recently moved to EclipseLink. With some of the new tools available to Eclipse for use with JPA (in particular Dali), I decided to think about reorganizing my services and trying to streamline some of my code (Using MappedSuperClasses as abstract parents for example). I focussed on my StampModel module which had a pretty decent unit test coverage of around 80% (some of the code not covered include error handling and a lot of my upgrade code) and was also where all of my entities existed. I even created a bugzilla entry for this work so that I could track my progress. I estimated it would take me no more than 12 hours. Well, 30+ hours later I finally have repackaged, rewritten or revamped the module (and database tables). For the first time in nearly a week I was able to open my Swing based editor last night and manage my collection. I still have some outstanding issues, but my refactoring is pretty much complete. There are a couple of lessons I took away from this.

1. Never assume how long it will take. Invariably something will come up (and I have to admit the Olympics coverage may have pulled my attention away from eclipse on occasion).
2. When I started refactoring I relied on my unit tests and decided to make updates to the Swing client after refactoring was completed. This was a mistake as often I had to think back to what the update was rather than making the change immediately (I am thinking along the lines of repackaging, changing initialization code etc.) Overall these changes tended to be pretty simplistic (change package X to package Y) but occasionally a DAO that I did not particularly like in the previous module was changed. In one occasion, the new DAO did not really fit and I had to rewrite some controller code (but at least it is better now).
3. Part of my refactoring was to move my entities which were annotated to be orm.xml based. I should have done this first without changing the data-model. Changing the database schema along with moving the entities to orm.xml created a lot of thrashing to work through.

Certifications even for the experienced?

Those of you who know me know that I sometimes shun the world of academia. Not that I do not think education is useful. I do. I am the current holder of two bachelor degrees and numerous certifications and I can honestly say they have proven useful in my career. But too often I see a candidate interviewing at my company who has years of academic training (multiple bachelor degrees, masters degrees, certifications up the wazoo), but they can not solve a simple puzzle put in front of them, or they display a complete lack of communication skills. After I was Java certified in 2003, I decided that I would take additional Java certifications to advance my knowledge. To date this has not happened, largely because my life is considerably busier and I much prefer to tinker and explore technologies than read the official way of doing something. Attending conferences has also been a pleasure to increase my awareness of new technologies.

Recently while being disposed, I was flipping though a copy of the Web Component Developer prep-book put out by Manning some years ago. It was a decent book, and as I flipped through the pages I found myself reading up on the techologies that I used on a daily basis, as well as advise my teams in their use. I feel in general, dispite the maturity of many Web Component pieces (such as Servlet, TAGs and EL), I feel many developers really do not understand why they are using them, or when to use one technology (for example TAGs) over another (such as Servlets). I proposed to the team that reading a book on such a subject matter might prove useful for their benefit. As a technology leader, I therefore felt it only appropriate that I study and become a Sun Certified Web Component Developer.

So this leads me to the conclusion that I suppose I need to engage in the duties of reading a book (for this current challenge I have chosen Head First Servlets & JSP from O'Reilly covering the 2nd edition), preparing myself and actually taking the exam. Based on my study habits, the best motivator I can think of is to schedule the exam now, that way I have to be prepared prior to the actual "writing" of the exam.

Automatically recording Create/Update Timestamps with JPA

Now that I am using Bugzilla to track various bugs and enhancements for my projects, I finally got around to addressing this issue of "Provide the ability to record the create/update timestamp." for my stamp objects. This seems simple enough, and most database applications record this information, however by default this is not something that is automated by the JPA frameworks (unlike the Primary Key with the @Id annotation). Let us first examine the ways we could accomplish this:

1. Provide a database trigger to try and insert the timestamps automatically.
2. Manually set them (or have each persistent service set them) before performing a persist() operation with an EntityManager.
3. Use aspects to dynamically insert the timestamp.
4. Provide an EntityListener which automatically sets the creation/modification timestamp at persist time
.

Obviously the first solution is very database specific and is not really tied to the JPA code. The second solution is likely to be error prone and easily missed. The final solution is the best way to handle his. Daniel Pfeifer provides an excellent walkthrough of this technique in his blog here. I have a few comments on this. First off, the concept of an entity listener, does not follow the normal "implement this interface" convention. An entity listener is any POJO class which contains one or more methods in the format:

public void someMethod( Object obj )

It should be noted that there are javax.persistence annotations for each of the JPA lifecycle states. In the case of create timestamps, annotating a method with @PrePersist will allow it to be used to provide the creation timestamp. For the modify timestamp, annotating a method with @PreUpdate will call this method on persisting an entity which was previously persisted. The entity listener can be registered either in the orm.xml file (as described by Daniel Pfeifer) or one the Entity class itself using the annotation @EntityListeners( class ... ). Personally I prefer this technique as it allows me to programatically tie a class with its behavior (such as storing timestamps on create/update) without having an external configuration. Using an external configuration in several modules, and unit tests quickly becomes muttled in that you forget to update the file under test etc. It also increases the developer's awareness of this association, and to be candid, it is unlikely that I really want to swap out the persistence timestamp handling at package time with a different solution. Since the annotation is applied to an abstract @MappedSuperClass annotated class, any implementing classes will automatically inherit this behavior. I might change my stance on this approach in the future, but at least for this this seems to be the right way to approach this.

Subversion/Bugzilla and Eclipse

I wanted to try and get my newly installed Bugzilla system configured to work with eclipse and subversion. I wanted to be able to reference my Bug IDs on subversion check-in and be able to navigate to them in Eclipse. Mark Phippard did a fantastic job discussing this in his blog here.

Since I was connecting to a bugzilla database which I am allowing anonymous view access to, the bugtraq:url property I used was http://<hostname>/bugzilla/show_bug.cgi?id=%BUGID%. Doing so, meant that when I check in my files and provide a Bug ID, the subversion history will have a record of it, which will be provided as a link to the information page of the bug in eclipse. Pretty neat stuff!

HSQLDB/EclipseLink Bug Filed

Well I got around to filing the uniqueness column constraint issue for HSQLDB and EclipseLink. The Bug ID is: 240618.

EclipseLink Startup Lag

I took a few minutes to look into the startup timing differences between EclipseLink and Toplink Essentials. It looks like with EclipseLink they have changed the way the login information is handled for the decryption of the password. Previously in TopLink Essentials, the password was decrypted through the SecurableObjectHolder, which forced the JCEEncryptor to be initialized only upon creating the new instance. In EclipseLink, rather than initializing on first use, they are pre-initializing the JCEEncryptor through the SecurableObjectHolder and then decrypting the password. Another difference between the EclipeLink and TopLink Essentials encryptor, is with EclipseLink they are creating a separate cypher for encryption and decryption. Having looked at the code, this makes sense from a performance scaling perspective. Having separate encryptors/decryptors means that you do not need to reinitialize them for each encryption or decryption. Of course, on startup that would mean the instantiation of not one cypher (for decrypting the password) but two, which does account for about a 4000 ms* difference in the profile runs. This interesting enough is also the difference between the application startup running either EclipseLink and TopLink Essentials.

* Since I am profiling in Netbeans with All Classes the performance numbers themselves are quite poor. What is more interesting is the ~ 18% performance degradation on startup this causes.

EclipseLink - Initial Impressions

As was suggested by Doug Clarke, I took a look at EclipseLink. I was actually caught a little off guard on the whole subject, as with the birth of my daughter I had pretty much gone under a rock for the past few months. EclipseLink is yet another JPA provider, but there are some interesting aspects to it:

• It was chosen by Sun Microsystems to serve as the reference implementation for Java Persistence 2.0.
• The JPA development community has essentially switched from TopLink JPA right over to EclipseLink.
• EclipseLink is bringing further capabilities with support for MOXy (JAXB), SDO and OSGI to name a few.

There are several good articles on the tool itself from the EclipseLink home page.

I downloaded the most recent release from the Eclipse website, and set about setting up my applications to use it. Generally speaking, I had tried to make only API calls to the javax.persistence APIs, thus I had very little package dependency changes. Since EclipseLink is based off TopLink virtually all the classes from TopLink appear in EclipseLink under a different naming convention. Basically oracle.toplink.essentials became org.eclipse.persistence. Since I have a nice level of unit testing around my services, I was able to quickly identify the places that were failing. In particular, I had some Query Hints that I was applying if the provider was the TopLink provider. I had to replicate this functionality for EclipseLink code paths (fortunately it was only in a few places). I think the biggest impact was on the Upgrade Tooling in which the SQLSchemaUpdateUtility had package dependencies on TopLink Essentials. Instead I roled this into a StatementExecutor interface/implementation and used reflection to call the APIs. Similarly, I did something similar for EclipseLink if I detected it as the persistence provider.

One observation I have noticed is that EclipseLink takes significantly longer to "initialize" than TopLink Essentials. This is most noticeable with the Unit Tests and HSQLDB, where the execution time is approximately ~ 5.0 seconds different. I brought up my JavaSE application in Netbeans 6.1 using the Profiler, and there is a significant time lag building and initializing the EntityManagerFactory. I am not convinced yet that there isn't a setting I am missing which is causing this lag. For example, launching my JavaSE application (which will query to see if there are any upgrades needed, query for all collections and countries, display the Swing UI etc.), almost 56% of the total startup time was spent in the deploy() method of the EntityManagerSetupImpl.

EclipseLink provides a nice compact javax.persistence_1.0.0.jar file which represents all of the J2EE javax.persistence classes required to compile. This is great for application development, and means you can provide your JPA provider at runtime (along with the persistence.xml configuration). Of course, this assumes you are using non-compile dependencies for things like QueryHints etc.

The documentation for EclipseLink seems a little disorganized, but overall there is a lot of information available through the EclipseLink Wiki User Guide.

I am looking forward to using some of the query features (such as fetches for foreign key objects) and it will take some time to really take advantage of the broader features being offered.

Commentary for my HSQLDB and Uniqueness Post

I figured I should post an update about the HSQLDB issue regarding the JPA "unqiue=true" constraint issue. As was suggested by Doug Clarke, I took a look at EclipseLink and it too suffered from the same issue. I will file a bug sometime this weekend with EclipseLink (Last week their Bugs link was broken). As I was thinking about the solution, I think this could affect more than HSQLDB potentially and therefore I think the way to handle this would be add a new method to org.eclipse.persistence.internal.databaseaccess.DatabasePlatform named boolean shouldUniqueKeyBeHandledAsConstraint() and then the code I mentioned earlier could be genericized further with the HSQLPlatform class implementing the aforementioned method with a value of true.

HSQLDB and Toplink : Uniqueness Constraints

After some effort, I was able to figure out a way to place the UNIQUE constraint from the @Column definition on an JPA entity bean, and have it handled properly with the Toplink JPA. I was actually surprised that even with build 40 or Toplink Essentials v2.1 this was still an issue. The problem was, if you defined a value of unique=true in your JPA column annotation, Toplink Essentials would insert the UNIQUE keyword in the create table routine. This would break on HSQLDB, which does not support this keyword during column descriptor creation. The challenge was to fool Toplink into handling the unique attributes like contraints which it would add after the table was created with an ALTER TABLE sequence.

The route I have chosen for now to solve this for my unit testing needs was to provide replacement Toplink-Essentials class files ahead of the Toplink-Essentials JAR file used by my application. Therefore my applications all run with the approved Toplink distributable, however my unit tests run with the instrumented files. There were two small changes I had to make:

(1) Modified oracle.toplink.essentials.tools.schemaframework.FieldDefinition to not write out the unique field keyword if the database was HSQL. From Line 168 of the appendDBString method:

   if(isUnique() && !session.getPlatform().isHSQL()) {
      session.getPlatform().printFieldUnique(writer, shouldPrintFieldIdentityClause);
   }

(2) Modified oracle.toplink.essentials.tools.schemaframework.DefaultTableGenerator to add the unique constraints if the database is HSQL on line 253 of initTableSchema()

 if( dbField.isUnique() && databasePlatform.isHSQL()) {
   tblDef.addUniqueKeyConstraint(dbTbl.getName(),dbField.getName());
 }

Currently I have not posted any information to the Glassfish project with these updates. I am not certain this is the ideal way to achieve this, but from my unit testing perspective I appear to be off to the races. If you are interested in these changes, please contact me and I'll look into working to get them submitted for Glassfish.

JPA Unit Testing

Up until now, I have been testing my stamp services using a test-schema that lives on my MySQL server. This server is remote (well in my basement connected to a 100Mb ethernet). While this has worked well for me, as the number of unit tests have increased in my code, the time of the tests is also increasing. Eskatos's wrote a great little blog article which introduced HSQLDB to my vernacular (see Unit test JPA Entities with in-memory database). Of course in his scenario, hibernate was used in place of Toplink. However I was intrigued by the idea of using a in-memory only database for unit tests. So I set about to get my unit tests to run. This turned out to be tricky to get working using Toplink. The first issue I ran into was that the tables refused to be created on startup. It turns out, this is due to an issue with the toplink.target.database property missing from the persistence.xml file. This was outlined in a useful blog TopLink JPA and HSQLDB Quirk.

Even after making these changes however, I still could not get toplink to properly create the tables. It turns out, I had several entity beans which had name fields defined as unique=true. This caused the UNIQUE keyword to be written in the CREATE TABLE statements by Toplink which appears to be an invalid syntax for the HSQLDB database. After removing this JPA constraint from the affected objects I was able to successfully create the tables and run my tests. I also had some minor refactoring to do in some SQL utilities to leverage the persistenceUnit configuration, but I was very impressed with the speed.

Overall, my test suite went from executing in approximately eighteen seconds down to just four. While eighteen seconds may not seem like a long time, it was sufficiently long to disrupt my work efficiency. I decided to retain my MySQL peristence unit (for occasional "live" DB testing), and have now configured two test targets in Eclipse which take a org.javad.jpa.serviceName environment variable to switch between the hsqldb and toplink-test persistence units.

The final activity I will have left to do, is to determine a way to reinsert the unique statements in my entity beans without have the SQL generated for HSQLDB. There are a few threads out there, so I should be able to come up with something.

Finally, here is my persistence unit configuration for the HSQLDB database:

<persistence-unit name="hsqldb" transaction-type="RESOURCE_LOCAL">
  <provider>oracle.toplink.essentials.PersistenceProvider</provider>
  <class>org.javad.stamp.model.Album</class>
  <class>org.javad.stamp.model.CatalogueNumberReference</class>
  <class>org.javad.stamp.model.Category</class>
  <class>org.javad.stamp.model.Country</class>
  <class>org.javad.stamp.model.Stamp</class>
  <class>org.javad.stamp.model.StampCollection</class>
  <class>org.javad.model.ClassVersion</class>
  <class>org.javad.services.TestEntityWithIdentity</class>
  <properties>
    <property name="toplink.jdbc.user" value="sa"/>
    <property name="toplink.jdbc.password" value=""/>
    <!-- <property name="toplink.logging.level" value="FINEST"/> -->
    <property name="toplink.jdbc.url" value="jdbc:hsqldb:mem:."/>
    <property name="toplink.jdbc.driver" value="org.hsqldb.jdbcDriver"/>
    <property name="toplink.ddl-generation" value="create-tables"/>
    <property name="toplink.target-database" value="HSQL"/>
  </properties>
</persistence-unit>