Sunday, January 20, 2013

Software State of Affairs - Technical Status Quo

I just finished a project to refactor our Web application, written in JSP many years ago. This runs in our EA Server (Ver 5.5) environment. The environment itself is 8+ years old, and the versions are reflective of that time. The technology used was already outdated by the time the software went into production. There were several violations of the modern web programming techniques. Lot of embedded Java coding, at will call/connections to servers and database etc. caused the program to fail during peak loads.

Well, there is a lot to talk about that software and the improvements we did. I will do so in other posts. In this post, I want to focus on the issue I see in every software maintenance projects - the status quo  in keeping the software architecture intact, irrespective how much it changes in business functionality.

I frequently compare software to a car. The comparison arises in that both are built, run, need to be maintained, and tuned up. That's where the comparison ends. We don't keep adding functionalities to a car. We add bells and whistles like tint, color, alarm etc, but never add functionality or change it's function completely; we don't make a car run on square wheels or sideways. Though the engine gets hot, we don't use a it as a toaster or to cook chicken. But, in software we do exactly that.
computer make coffee

When a custom software is designed or purchased, the users typically get all the functions they need. But, over the years users change and so does the business. We keep adding enhancements, bug fixes to the software constantly to meet the (sometimes unreasonable) business requirements that were never dreamed of, when we built the original software.  The software thus gets bloated over a time period. We don't adjust the technical architecture or design to match this new software obesity and it eventually comes to a crawl, at which point everyone blames the software and want to replace it. By then, it may be too late! Many custom software out there have nice front ends, but - pardon my French - crappy code inside. It's like a flash new bus pulled by a mule, no offence to the mule!! I blame this on Software Status Quo! We tend to cater to business so much, we neglect the software itself in the process!!

mule inside

Lot of this is due to the way companies budget for Software projects. Several departments tend to have a "lights-on" policy, which means keeping the software up and running 24X7 without any outage. Companies typically have a lot of budget for "operations" but not real software maintenance. As a consultant, I've seen IT departments struggle to define my work as part of "lights on", so they could continue to get funds for my contract.

Typically, the business side budget pays for the "operations". Except for a handful of savvy business managers, rest of them don't really understand software. To them, the "pretty" screens and "nice" reports are the software. As long as these happen, software is working fine. When you work with such managers, it is not easy to get budget for retrofitting software. "Business" requirements will always take precedence to technical requirements.

But, business is not entirely to blame for this. Recently, when I started introducing the changes in our software, I saw advantages of using more Java in a mostly Powerbuilder environment. There was some awe initially, then reluctance and finally an indifferent acceptance. New architecture means, new problems and tougher maintenance, at least initially. Now, I see why this status quo is continuing. I always hear senior programmers say, if it's not broken and/or users didn't want it, we wouldn't fix it. Where as, this is a safe approach to keep business happy, it hurts the software in the long run.

To end such status quo, business must stop treating software like stationary and recognize and allocate budget for it to be "tuned" regularly. I also feel, real end users (not just business managers) must be regularly consulted as to the performance and usage of the application. I've seen many a software projects fail, in spite of extensive design and funding, because end users simply rejected the software. Developers must also constantly look for ways to change and improve the software itself (not just it's business functionality) and strive to include these in their releases. I see several developers less motivated with the routine business fixes. Adding in technical items, may help to re-energize the developer base.

Many years ago, I worked in a warehouse in NJ where they were using an AS/400 for order tracking, but the software itself was so outdated, the users were improvising. Believe it or not, they would print a report out of this system, then they would input that information into System/36 along with accounting information and export to a file. The next smart user takes that to mainframe to finish up the final entry, where corporate wanted the original shipping and accounts information. Those systems people in Corporate, 3000 miles away conveniently ignored this status quo, that this process actually continued for many years until truly yours stepped in! Ironically, around the same time, corporate headquarters poured in millions to port some of their mainframe applications to flashy client/server tools.

I normally keep notes of all technical issues or a wish list for the software I work on. Then when I get to work on some business requirement in that software, I include any appropriate technical fix. Recently, while working on a business requirement involving Java stored procedures, I rewrote the logging mechanism completely, so we can support it better. Previously, developers were compiling and distributing this code to DBAs completely outside of the build process. If one class had to be updated, just compile it and patch it into the Jar file!!! This was error prone and I found some source code older than compiled code. I introduced eclipse and Ant to reduce human errors. Of course, it took longer to implement the task, but it pays for itself in the long run!

Oh yeah, we did solve the impasse at the warehouse in NJ, but it needed some thinking outside the box. I got the blessing from the local business director and with the help from users and corporate developers, we rolled the users to corporate mainframe (of course, after making necessary changes on the mainframe to accommodate them) and wheeled the AS/400 out, the one I was there to support in the first place! The company saved a bundle and everyone was happy!

Sunday, January 6, 2013

Gotcha: EA Server config. XML file comments

Sybase EA Server (Ver 5.5) provides 2 interfaces for the admin to work with the Server and the Repository. Sybase Jaguar Manager is the GUI interface, that one can use to manage the Server and the repositories.

Jagtool is the command line equivalent of the above manager. There are a lot of things you can do in Jagtool and with host scripting (batch files in windows), you can make real powerful scripts to execute these in batch mode.

EA Server typically configures all elements using properties files. But, it also allows us to use XML files to configure the server. These are available at various levels such as Server, Package, Component, Connection Cache etc.

This gotcha is about writing comments in an XML file meant for configuring EA Server. Single line comments follow the XML/HTML standard.
<!!-- this is a single line comment -->

When you write multiline comment, make sure the tags are on their own line, thus:
<!!-- 
This is a
multi-line
comment
-->

If you include any text in the first line, it will error out.

Note: You only need <!-- to mark the beginning of a comment; I doubled exclamation mark to avoid losing the comment line in view!! Otherwise, Wordpress HTML processor would have hidden it from view.

Ubuntu: File Managers

I am continuing to use Ubuntu as my primary desktop and I am loving it(I still use Windows desktop. It has it's advantages). As I get used to the environment, I am always looking to mimic familiar options from Windows. This time around I was looking to find out about adding applications to Context menu in Ubuntu. Like always, I stumbled on something bigger - File Manager(s).

Earlier I posted about Linux architecture and how different layers of the Operating System can be replaced/added. If you read that post, you will see how I struggled with Unity Desktop that came with Ubuntu 12.04  and eventually learned to replace it with Gnome desktop. Apparently, each desktop has it's own flavor of File Manager and then some.

If you are coming from the Windows world (like I did), here are some terminologies for you. What we call Windows in Windows Operating System (like Windows XP) is really the Desktop. We saw earlier there are many different Desktops including Gnome, KDE, Unity etc. The Desktops have several parts and one of them is a File Manager. This is called Windows Explorer (it used to be called File Manager in versions before Windows 95). There are choices for Explorer, but I've not seen many people replacing the Windows Explorer with another.

In Ubuntu (for that matter entire *nix [Unix, Linux]) world, everything is customizable and/or replaceable. I was able to replace my troubled Unity desktop with Gnome desktop and overcome some issues earlier. When I started researching on Context Menus, I stumbled on Nautilus package, which is the default File Manager in Gnome. Since Ubuntu 12.04 didn't support Gnome desktop natively, it didn't install Nautilus either. It had a File Manager called Thunar instead. And, when I installed Gnome, unknowingly, I had also opted for Nautilus, default file manager for Gnome. Apparently, there are at least 20 different File Managers available. See this blog.

This site has a nice picture of the Linux Architecture.

In Gnome Desktop, you get to File Manager by clicking on Places menu on top.

Workspace 1_057Fig 1: Menu to reach File Manager

Screenshot from 2013-01-06 19:06:37Fig 2: Nautilus File Manager

So what about Context Menu?
Coming back to adding context menu, it's done using Nautilus Actions.I had to install it first. (This you could do it in Ubuntu Software Center or use apt-get command). Once you installed it, you can add context menu items as shown below:

Nautilus Actions ConfigurationFig 3: Nautilus Actions Configuration

And of course, after changing the context menu you need to reload the Nautilus File Manager. To do this, you can type the below command:

<code>killall nautilus</code>

This kills all processes associated with the particular program (here nautilus). For Windows users, this is like going to Windows Task Manager and doing "End Program". See here for more on killall. Incidentally, they also talk about killing nautilus!!

Running KDiff3 from Context Menu
The program I was trying to add to context menu was KDiff3 (it's a great diff utility, try it, if you haven't already. I use it on Windows as well). Nautilus Actions allows you to add programs to the context menu. I select 2 files to be compared and right click to select and open KDiff3 and voilĂ ! KDiff3 opens and diff's the files automatically!

Nautilus Context MenuFig 4: Nautilus Context Menu


kdiff3_074Fig 5: Kdiff3 From Context Menu


References

  1. http://www.tuxarena.com/2011/06/20-file-managers-for-ubuntu/

  2. http://www.techdrivein.com/2010/05/what-is-nautilus-elementary-and-how-to.html

  3. https://live.gnome.org/Nautilus/Screenshots

  4. http://www.tellmeaboutlinux.com/content/linux-architecture

  5. http://en.wikipedia.org/wiki/File_manager

  6. http://askubuntu.com/questions/88480/adding-extra-options-to-right-click-menu

  7. http://www.omgubuntu.co.uk/2011/12/how-to-add-actions-emblem-support-back-to-nautilus-in-ubuntu-11-10/

  8. http://www.linfo.org/killall.html

Monday, December 24, 2012

Merry Christmas!

6d605d6a3345ad0edae720f424704a91

May the Christmas Cheer spread around the world and bring peace, health and wealth to everychristmas_pinguinone!

I just found this nice site to create fancy fonts!

Sunday, December 23, 2012

Software State of affairs - Maintaining OO

Last week, a friend at work was analysing code in Powerbuilder and question came up about function Overloading and we branched into Object Orientation in general. That question got me thinking. Though Object oriented programming has been around for a while, the concepts of OO are not completely understood or followed.

I posted about OO concepts here earlier. To do that I bought and read several books, sifted through the web a lot. There were so much confusion and conflicting ideas. Most of those books use obvious objects like animals, people etc in explaining OO concepts that what we learn in those is not enough to translate in to real life OO programming. I mentioned about some of these problems in my earlier post. More and more languages are adding OO concepts, yet people who are used to traditional (procedural) way of programming, struggle to adapt to it or even completely reject it. Often times, Syntax comes in the way of understanding. For e.g., Java borrowed so much from C, C++, that it's often considered to be just enhanced C, thus OO nature of it is completely ignored. And even if these developers switch to OO eventually, the objects they come up with are just repackaged C code, thus perpetuating the problem!

The application I currently support was written in Powerbuilder using a framework called EAF. The software also has business framework that is built around this, definitely a good extensible architecture. Yet, the software has degenerated over the years. Part of the problem is lack of understanding of the framework and/or OO concepts in general. I remember, when I started here, this framework was considered to be the main reason for all evil in the software! There was not enough documentation about proper framework usage, thus it was easy to blame it!

When new developers came in, they wrote and rewrote functions and objects that completely ignored underlying framework thus code is bloated. Further, supporting business has been utmost priority, so such technical issues were completely ignored. I feel, every maintenance project should include the maintenance of the software itself - in addressing technical issues and retuning it to keep it running good, like we do with our cars. After all, business will eventually be affected, if we didn't take care of such technical issues in time! The very first problem I solved in this software was a technical issue - there was a memory leak (due to improper handling of Object pooling) in the software that kept crashing the server during peak usage in the prior years. Business of collecting taxes, was indeed affected during those crashes. And I cannot imagine how much time, effort and resources were wasted during those times.

Another problem I see in software maintenance, is the tendency to solve code smells and code bloats by adding more resources - more servers, more disk space, more memory. When a car runs sluggish, we can't solve it by just adding more engines or more tires, can we? Why is it OK with software? Prior solution to the memory leak problem I mentioned above, was to add more servers and a script to automate the Server Restarts! Once the memory leak was fixed, no more frequent crashes!!!

To curb some of the issues in the application, I recommended and started on a project to re-factor some of the code in the application earlier this year. Main goal was to improve performance in our web application, but I wanted to clean up and reintroduce some of the framework methodologies.  Though there was some push back initially, other developers started buying into the idea. Yet, I see the reluctance to change existing ways; after all the software works during normal (low?) usage and business is OK (if not happy), why change those? When I first introduced log4J (logging libraries for Java) for logging in our JSP pages, it took the team at least couple of months to buy into the idea, in spite of me showing stats to support the change.

Our application architecture uses EA Server (EAS) as the middle tier. EAS supports Java and Powerbuilder objects naturally.While Powerbuilder is great for developing applications quickly, Java was used for any multi-threaded activities such as caching and logging etc. We even have Java Stored procedures running inside Oracle database! I recently introduced tools like Eclipse, Ant etc. to streamline Java development activities. With such exposure and addition of new Java packages prompted a team member to say to the effect "we are becoming a Java shop whether we like it or not". I've tried to explain in vain that it was all there before and why we even need them!!

Part of the problem is also lack of career planning and/or training. I remember the days when companies invested in training employees to keep up with the industry trends. Now, we seem to be spending more in buying those extra servers to support more memory leaks :)

Tuesday, December 11, 2012

Tools: Web Statistics Tools - Analog

We have a web site used by the tax payers to file their annual taxes. This is used by 100s of thousands of users within a short period during peak season (Feb - Mar) and reasonably used other times. During the peak times, it almost explodes.

I have been working on improving performance of this site. Also, there have been no stats whatsoever to be able to confidently say how many users were served successfully or when it failed where the bottlenecks are. We relied on information stored in the Database to "guess" these stats. I am adding a bunch of tools and (Java) programs to get more stats as well.

I am currently doing load testing on a test version of this site using Apache JMeter. While this tool is great and comes with a suite of reports, and I have added a tons of stats on the server side (an old Sybase EA Server) I needed something outside of these two. So, I started looking at the HTTP Access logs. EA Server includes a light weight Apache server at the core. So, the format of the log file is almost identical to a Apache Server Access logs. (On EA Server the file is called JaguarHttpRequest.log).

As usual, I looked around for tools that will help with analyzing these logs (when the computer can do the work, why do it manually?!). While searching for the tools, initially I looked for Apache log parser. I did find a few, but nothing great. (I found one PHP program though, which is kind of interesting from programming point of view). Eventually, I landed in Web Analytics Software and our good old Wiki (here) came to the rescue. I tried a few software listed there (Webalizer, AWStats) look interesting.

Finally, I tried Analog. Bingo!! This is what I have been looking for. Analog is a very simple tool to install and use. (Installation is just unzipping the zip file). Using the tool is as simple as Double clicking or running from command line, the analog.exe file.

This actually produces a (sample) report with the name Report.html. After reading more on Analog web site, I was able to customize the config file named analog.cfg. Once I tweaked this file, I started parsing the log files in minutes. I was even able to append date to the report file name and thus was able to run it in a loop to capture the stats every few minutes.



Apart from the Summary page, it also has several graphs to show stats for Hourly, Daily, Weekly, Monthly etc.

If you are looking to analyse usage patterns on your web site, this is a great tool. Check it out!

Friday, December 7, 2012

Quick Tip: How to exit from SQL*Plus on command line

This is about running SQL*Plus in a batch mode. Suppose you have a script (shell or batch file) that calls SQL*Plus to run a SQL in a batch mode. I am talking about running it from Command line, thus:

$ sqlplus <user_id/password>@SID @<sql_file_name)

Chances, you will want to exit SQL*Plus as soon as the script is done (EOF file is reached in SQL file), so we can continue our processing.

Typically, we add an EXIT statement at the end of the SQL file itself and this will force SQL*Plus to quit. What if you forgot or couldn't add EXIT (you use the same script in different scenarios). If you don't have an EXIT or QUIT statement at the end of your SQL file, you will end up seeing the SQL Prompt:

SQL>


Here is a quick tip to exit Sql*Plus after it's done with the script:

exit | sqlplus <user_id/password>@SID @<sql_file_name)

(That's it. Essentially piping exit into sqlplus command! When the End of file is reached, SQL*Plus returns to the shell and your shell script can go on!

This tip works on both DOS (Windows command prompt)_ and *nix systems.