This is from 2010, but interesting read
http://www.sciencedaily.com/releases/2010/03/100325100848.htm
Tuesday, January 22, 2013
2012 in review - Auto generated Wordpress Report
The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.
Here's an excerpt:
Click here to see the complete report.
SamV: Wasn't exactly a good year, but this is the year, this blog is turning around. I am finally seeing some visitors. I am still in the process of improving this. If you happened to stop by here and have some ideas as to how to improve the user experience, please feel free add as comments. I appreciate it. Have a wonderful 2013!!!
Here's an excerpt:
600 people reached the top of Mt. Everest in 2012. This blog got about 4,900 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 8 years to get that many views.
Click here to see the complete report.
SamV: Wasn't exactly a good year, but this is the year, this blog is turning around. I am finally seeing some visitors. I am still in the process of improving this. If you happened to stop by here and have some ideas as to how to improve the user experience, please feel free add as comments. I appreciate it. Have a wonderful 2013!!!
Quick Tip: Java Stored Procedures in Oracle - Reading Content
This post is about how to get the contents of a resource file that is loaded into Oracle as part of Java Stored Procedures package.
While working on enhancing some Java procedures, I added a simple Java logger class, fashioned after Log4J. To be able to dynamically configure the logging properties (log levels, threshold), I added in a properties file, bundled into the Jar file itself1. After loading the properties file into the Database (using LoadJava command; loaded as a Java Resource), I wanted a way to "see" what's inside the file. That's when I posted a question on StackOverflow, but eventually googled and found the answer. See my question and my own answer here. And here is the script, I came up with:
This printed the following:
While working on enhancing some Java procedures, I added a simple Java logger class, fashioned after Log4J. To be able to dynamically configure the logging properties (log levels, threshold), I added in a properties file, bundled into the Jar file itself1. After loading the properties file into the Database (using LoadJava command; loaded as a Java Resource), I wanted a way to "see" what's inside the file. That's when I posted a question on StackOverflow, but eventually googled and found the answer. See my question and my own answer here. And here is the script, I came up with:
SET SERVEROUTPUT ON
EXEC DBMS_JAVA.SET_OUTPUT (1000000);
DECLARE
bText CLOB;
len PLS_INTEGER;
offset PLS_INTEGER;
text VARCHAR2(2000);
BEGIN
DBMS_LOB.CreateTemporary(bText, FALSE );
DBMS_JAVA.Export_Resource('LAJavaSP.properties', bText);
len := 1000; -- length of the lob to read.
offset := 1;
DBMS_LOB.Read(bText, len, offset, text);
DBMS_OUTPUT.Put_Line(text);
END;
/
This printed the following:
logLevel=ERROR
dbLogLevel=ERRORFor now, that's all I have in the properties file. Hoping to add more later.
1 Adding the properties file into the jar file makes it easy to read it using
getSystemResourceAsStream. Otherwise, we will have to worry about Directory objects, Permissions etc.(Project) Withdrawal Symptoms
I must admit. I think, I am going through withdrawal symptoms. No, not that kind. I am not an alcoholic, and I don't do drugs. I do drink coffee, but definitely no withdrawal from there! This is a different kind of withdrawal. I think, I just OD'd on work and now I am feeling the project withdrawal symptoms. I did a regular day's work today, it feels like I am not doing any real work. I get out of the building, while there is some light, it feels like I am leaving too early!
I just finished a project of refactoring one of our websites for performance. It took a lot of long days and nights, incessant reading, writing, thinking. Lot of headaches, anxieties, uncertainties. At times, it looked impossible, insurmountable and the team members expressed concerns, doubts. Even the week we were getting ready, there were issues, bugs, failed tests, network issues. After all, we are in the tax (peak) season already, is this a good time to deploy such a big change? Is it going to happen? Is it worth it? It was like writing/reading a mystery novel at the same time. What is the end? I am writing it, but I am also hoping to read it ahead. Couldn't stand the suspense no more.
The number of tasks grew and grew, as I picked up slack for team members that went on vacation. Finally, it's over!! poof! gone! We installed the system Thursday night. No issues!!!!! What? No way! I was looking for issues. But none. It was a good thing, lot of planning and work went into it, yet it felt like a very bland ending of the mystery novel. Phew! No issues! It feels good. But, why does it feel like nothing happened? The rest of the team members haven't said anything yet. Did the implementation really happen? I look at the beautiful log file created by Log4J smiling at me. The changes actually helped us catch an issue, we otherwise wouldn't. Yes, we did it!!
Now, I am catching up with the rest of life. I was here, wasn't I? Calls to be made, photos to share, trips to make, people to catch up with. In the past few months, I would come home late from work, eat and jump right back on the computer, pouring over the source code, popping in and out of forums, go round and round the world wide web, eyes glued to pages of the books and the mountain of pages I printed. Now, it feels like I don't have anything to do. But, it's a nice withdrawal! No hangovers, no issues! Work, life balance? My wife thinks, I am back to life!!! Good to be back!
Monday, January 21, 2013
Software State of Affairs - Tools of Trade
Early man did a lot of things by hand, but soon started inventing tools (Simple machines in physics) to help him do things quicker and easier. Modern tools are way too complex than these early predecessors.
Tools of Trade1
In the modern world, we depend on tools so much, we don't even realize. See this little Physics activity page to see how many you may be using. Without tools, our lifestyles as we know will cease to exist. Every trade has associated tools, which we will call Tools of trade (In legalese it has a more restrictive meaning. See note 1). When we call a plumber, we expect him to come with the tools necessary, like the wrench etc. The electricians use different set of tools of which, voltage tester may be the most important. If they don't have the right tools, we know what will happen. At the very least, you will be looking up Yellow Pages to call the next technician.
Software Tool Kit
IT field borrows a lot of terminologies from other fields. We are engineers and technicians and architects that "build" software products, just that you won't be able to see the finished products except on your computer screen. While building, running, testing software products, we use a lot of tools. Like the product they help to build, these tools are software themselves. We use Compilers, Editors, Linkers, Runtime etc. to build and execute our software. But, unlike a plumber going to Home Depot to get his tools, we could simply download it. At the same time, while a plumber's tool chest may always look alike, our tool chest varies depending on the field and the software we build. And unlike a plumber or an electrician, we have the luxury of building our own tools, more so now in the age of Open Source.
Tools I use
As a consultant, I am constantly exposed to different tools at different sites and for some tasks, I tend to keep my own time tested tools. Here are some Open Source tools, I currently use. Often times, I use more than one tool for a task and combine the results. These are mostly for Windows environment. I will share other tools and for Linux separately.
Windows Grep I use it to search for text in files. Same like what Notepad++ or Textpad would search, but has a little more flexibility like the grep.
Many years ago, I used Examdiff extensively. Lately, WinMerge is the diff tool I use most frequently. WinMerge helps me compare at directory level and helps to merge files quickly. There are some limitations, but I love it. I also use K-Diff3 and CSDiff as each of these offer something unique. These are all free software. Another great diff tool I found was Beyond Compare. My next software purchase would be this. The above tools don't provide a good report for directory compare (Beyond Compare may). To do this, I found diffUtils and a nice script I found on the internet, to format the output in HTML.
Greenshot This is the open source screen capture tool I use. Pikpik is a great tool as well. Again, each of these have some feature that others lack.
OpenCommandPromptHere There are several names like Command Prompt Here, SendtoToys etc. Essentially, creates a context sensitive menu in Windows Explorer, so you can open command prompt from where ever you are in Explorer. (you don't have to CD into any directory manually).
BareTail I recommended this earlier; I use this to tail Jaguar.log etc. It's really good. You can highlight lines with different phrases.
YEd This I use a lot to do diagramming. It's more a graph editor than a Graphics editor. Works for me. While researching on diagramming, I found UMLet to be interesting. Dot, Dotty and GraphViz help me with automatically generating some flow diagrams.
Apart from these I use Ant, Eclipse, JMeter, Xampp etc as part of the my development environment. I also use Libre Office (Open Office's successor) a lot.
Tools I build
I am always looking for ways to make the computer do the work for me, instead of doing things manually. I feel, if a task has to be done more than once, it deserves a script or a tool. When the tools available are not good enough for a particular task, I try to build some on my own as well. But this is also an excellent opportunity to learn new techniques and keep your skills sharp. I do these during down times and/or as part of the main task.
I am sharing some of my tool building experiences here, hoping it would benefit someone else. If you are a Software developer and somewhat less motivated with the stuff you are working on, give tool building a try next time. The software will benefit from it too. All you need is some imagination and some laziness to resist doing anything manually! Gone are the days, when you had to purchase special software to build such tools. You will be amazed to see how much programming capabilities a standard computer can offer you. For e.g., if you have Word or Excel, you can program build powerful macros (VBA programs). Apart from these, you always have Open Source scripting languages like PERL, Python, PHP etc.
If you recognize early enough, the tools you build could become integral part of a project as well. At Velos, while working on the task of upgrading the database (Sybase SQL Anywhere) for a newer specification, I decided convert the specification into an Access Database. Once the new Spec. was in a database, I was able to compare existing database elements with the new ones and voilà!, the database upgrade script could be generated right out of the database itself. They had a Powerbuilder application to design and save screens (datawindows) in database (blob columns). After going through problems with the blob, decided to write a tool to view, edit, correct screens outside of the application. Few more times of using this, I ended up in writing a simple scripting language to perform these same tasks in a batch mode. Necessity is the mother of invention! Several of these tools were actually used by the support person to identify and fix issues at customer sites.
Some of the casual tools I build for one time use, end up as permanent additions to the team's tool chest. At Capital group, one developer was collecting batch job stats on a daily basis, and when he went on vacation, he gave a page long instructions on how to get these details in Excel. The task was, to go to the web pages (someone with tool mentality, pulled the stats from Autosys into the page, using CGI-PERL), copy the job names to a spreadsheet and make adjustments to be presentable. When my turn came, I got tired of copying, pasting and correcting pretty quickly. Before the next time, I added a "Web Query" in Excel to pull up the web pages directly into the spreadsheet and wrote some VBA macros to clean up and present the spreadsheets. Eventually, it became full-fledged product in it's own right, everyone in the team started using it. Page long instructions gone. (Of course, when I left, I left a few pages long document about the macro, but that's another story).
Similarly, in the same company there was another task, developers and/or the analysts had to do. Every now and then company received fund codes from outside (at the time I didn't know much, these are the GICS codes we received from MSCI), that had to be reconciled against codes that existed in the system already. Some codes may be replaced with new ones, some may get new descriptions, some may be discontinued. So, they would enter these "changes" in a spreadsheet or word document and then ask a developer to convert these to SQL (INSERT/UPDATE/DELETE) statements to be put into production. Enter yours truly, the lazy user. I wrote some excel macros that would import the data from the Oracle table (using ODBC), reconcile and generate the SQLs for the developer to run in production. Yes, it did take a little more time to develop and test it, but next time similar request came, it was a breeze.
An Excel macro dubbed "Query Tool", I developed at Capital Group tops it all. It again started as a manual task. Goal was to identify some codes (for e.g., security codes) from another system and see if it existed in our database. This was done manually too. User would extract data from the table into a spreadsheet and look up each code. Sometimes, codes might have been mistyped, so they have to match partially or look at other attributes to identify and correct. At the end of this, they may have gone through a spreadsheet with 1000's of codes in several passes, identifying codes from another list they had. Time consuming, laborious and error prone. If this is not what the computer does best, what else? So, I started coding the excel macros to do the repeated search of the codes they entered. Little by little, it became a full fledged querying tool (using Database Query - through ODBC - in Excel). The tab containing the actual SQLs helped customized the lists further. What resulted was a nice product. The users loved it.
These one-off solutions may also help you save a project. At FHLB, we were working on a data extraction project. This was part of the data reporting requirements of the bank set by FHFB. I created an Access Database as a prototype to a much bigger data-warehouse built in Oracle by a team of ETL developers. I kept this as a tool to validate the data/report from the ETL tools, before we sent the data to FHFB. All was going well, but suddenly one day, we found a flaw in the Data-warehouse design and the multi-million dollar project simply collapsed. Luckily my boss saw this risk early, supported my efforts to spruce up my Access Database. Believe it or not, my little tool eventually became the official "data reporting tool" for the company, at least the first year.
Journey Continues..
Recently, one developer mentioned about how to check a file in Textpad for a record with odd length. This was the idea behind this PERL script. Another opportunity for such a script was where a batch program creates a large file (XML in this case) that had to be split into smaller chunks. Perfect candidate again for a PERL or a Python script. I did a quick prototype, but it died down for lack of interest. I hear they do it with some tools like access DB and a lot of manual "pulling the hair". And then there are tasks like parsing logs and so forth. A well written script can analyze and report stats from these files, in a jiffy.
The other day, we were discussing testing for a Database upgrade. I brought up the topic of monitoring and my boss said, "you can make yet another tool for it"! I know he was kidding, but I probably will or if I am in luck, I may be able to just download something from the internet. The journey continues!
1. Tools of Trade has a different meaning in legalese. It's the tools your livelihood depends on, so in case of bankruptcy, they will not be taken away by creditors.
Tools of Trade1
In the modern world, we depend on tools so much, we don't even realize. See this little Physics activity page to see how many you may be using. Without tools, our lifestyles as we know will cease to exist. Every trade has associated tools, which we will call Tools of trade (In legalese it has a more restrictive meaning. See note 1). When we call a plumber, we expect him to come with the tools necessary, like the wrench etc. The electricians use different set of tools of which, voltage tester may be the most important. If they don't have the right tools, we know what will happen. At the very least, you will be looking up Yellow Pages to call the next technician.
 |  |
Software Tool Kit
IT field borrows a lot of terminologies from other fields. We are engineers and technicians and architects that "build" software products, just that you won't be able to see the finished products except on your computer screen. While building, running, testing software products, we use a lot of tools. Like the product they help to build, these tools are software themselves. We use Compilers, Editors, Linkers, Runtime etc. to build and execute our software. But, unlike a plumber going to Home Depot to get his tools, we could simply download it. At the same time, while a plumber's tool chest may always look alike, our tool chest varies depending on the field and the software we build. And unlike a plumber or an electrician, we have the luxury of building our own tools, more so now in the age of Open Source.
Tools I use
As a consultant, I am constantly exposed to different tools at different sites and for some tasks, I tend to keep my own time tested tools. Here are some Open Source tools, I currently use. Often times, I use more than one tool for a task and combine the results. These are mostly for Windows environment. I will share other tools and for Linux separately.
Windows Grep I use it to search for text in files. Same like what Notepad++ or Textpad would search, but has a little more flexibility like the grep.
Many years ago, I used Examdiff extensively. Lately, WinMerge is the diff tool I use most frequently. WinMerge helps me compare at directory level and helps to merge files quickly. There are some limitations, but I love it. I also use K-Diff3 and CSDiff as each of these offer something unique. These are all free software. Another great diff tool I found was Beyond Compare. My next software purchase would be this. The above tools don't provide a good report for directory compare (Beyond Compare may). To do this, I found diffUtils and a nice script I found on the internet, to format the output in HTML.
Greenshot This is the open source screen capture tool I use. Pikpik is a great tool as well. Again, each of these have some feature that others lack.
OpenCommandPromptHere There are several names like Command Prompt Here, SendtoToys etc. Essentially, creates a context sensitive menu in Windows Explorer, so you can open command prompt from where ever you are in Explorer. (you don't have to CD into any directory manually).
BareTail I recommended this earlier; I use this to tail Jaguar.log etc. It's really good. You can highlight lines with different phrases.
YEd This I use a lot to do diagramming. It's more a graph editor than a Graphics editor. Works for me. While researching on diagramming, I found UMLet to be interesting. Dot, Dotty and GraphViz help me with automatically generating some flow diagrams.
Apart from these I use Ant, Eclipse, JMeter, Xampp etc as part of the my development environment. I also use Libre Office (Open Office's successor) a lot.
Tools I build
I am always looking for ways to make the computer do the work for me, instead of doing things manually. I feel, if a task has to be done more than once, it deserves a script or a tool. When the tools available are not good enough for a particular task, I try to build some on my own as well. But this is also an excellent opportunity to learn new techniques and keep your skills sharp. I do these during down times and/or as part of the main task.
I am sharing some of my tool building experiences here, hoping it would benefit someone else. If you are a Software developer and somewhat less motivated with the stuff you are working on, give tool building a try next time. The software will benefit from it too. All you need is some imagination and some laziness to resist doing anything manually! Gone are the days, when you had to purchase special software to build such tools. You will be amazed to see how much programming capabilities a standard computer can offer you. For e.g., if you have Word or Excel, you can program build powerful macros (VBA programs). Apart from these, you always have Open Source scripting languages like PERL, Python, PHP etc.
If you recognize early enough, the tools you build could become integral part of a project as well. At Velos, while working on the task of upgrading the database (Sybase SQL Anywhere) for a newer specification, I decided convert the specification into an Access Database. Once the new Spec. was in a database, I was able to compare existing database elements with the new ones and voilà!, the database upgrade script could be generated right out of the database itself. They had a Powerbuilder application to design and save screens (datawindows) in database (blob columns). After going through problems with the blob, decided to write a tool to view, edit, correct screens outside of the application. Few more times of using this, I ended up in writing a simple scripting language to perform these same tasks in a batch mode. Necessity is the mother of invention! Several of these tools were actually used by the support person to identify and fix issues at customer sites.
Some of the casual tools I build for one time use, end up as permanent additions to the team's tool chest. At Capital group, one developer was collecting batch job stats on a daily basis, and when he went on vacation, he gave a page long instructions on how to get these details in Excel. The task was, to go to the web pages (someone with tool mentality, pulled the stats from Autosys into the page, using CGI-PERL), copy the job names to a spreadsheet and make adjustments to be presentable. When my turn came, I got tired of copying, pasting and correcting pretty quickly. Before the next time, I added a "Web Query" in Excel to pull up the web pages directly into the spreadsheet and wrote some VBA macros to clean up and present the spreadsheets. Eventually, it became full-fledged product in it's own right, everyone in the team started using it. Page long instructions gone. (Of course, when I left, I left a few pages long document about the macro, but that's another story).
Similarly, in the same company there was another task, developers and/or the analysts had to do. Every now and then company received fund codes from outside (at the time I didn't know much, these are the GICS codes we received from MSCI), that had to be reconciled against codes that existed in the system already. Some codes may be replaced with new ones, some may get new descriptions, some may be discontinued. So, they would enter these "changes" in a spreadsheet or word document and then ask a developer to convert these to SQL (INSERT/UPDATE/DELETE) statements to be put into production. Enter yours truly, the lazy user. I wrote some excel macros that would import the data from the Oracle table (using ODBC), reconcile and generate the SQLs for the developer to run in production. Yes, it did take a little more time to develop and test it, but next time similar request came, it was a breeze.
An Excel macro dubbed "Query Tool", I developed at Capital Group tops it all. It again started as a manual task. Goal was to identify some codes (for e.g., security codes) from another system and see if it existed in our database. This was done manually too. User would extract data from the table into a spreadsheet and look up each code. Sometimes, codes might have been mistyped, so they have to match partially or look at other attributes to identify and correct. At the end of this, they may have gone through a spreadsheet with 1000's of codes in several passes, identifying codes from another list they had. Time consuming, laborious and error prone. If this is not what the computer does best, what else? So, I started coding the excel macros to do the repeated search of the codes they entered. Little by little, it became a full fledged querying tool (using Database Query - through ODBC - in Excel). The tab containing the actual SQLs helped customized the lists further. What resulted was a nice product. The users loved it.
These one-off solutions may also help you save a project. At FHLB, we were working on a data extraction project. This was part of the data reporting requirements of the bank set by FHFB. I created an Access Database as a prototype to a much bigger data-warehouse built in Oracle by a team of ETL developers. I kept this as a tool to validate the data/report from the ETL tools, before we sent the data to FHFB. All was going well, but suddenly one day, we found a flaw in the Data-warehouse design and the multi-million dollar project simply collapsed. Luckily my boss saw this risk early, supported my efforts to spruce up my Access Database. Believe it or not, my little tool eventually became the official "data reporting tool" for the company, at least the first year.
Journey Continues..
Recently, one developer mentioned about how to check a file in Textpad for a record with odd length. This was the idea behind this PERL script. Another opportunity for such a script was where a batch program creates a large file (XML in this case) that had to be split into smaller chunks. Perfect candidate again for a PERL or a Python script. I did a quick prototype, but it died down for lack of interest. I hear they do it with some tools like access DB and a lot of manual "pulling the hair". And then there are tasks like parsing logs and so forth. A well written script can analyze and report stats from these files, in a jiffy.
The other day, we were discussing testing for a Database upgrade. I brought up the topic of monitoring and my boss said, "you can make yet another tool for it"! I know he was kidding, but I probably will or if I am in luck, I may be able to just download something from the internet. The journey continues!
1. Tools of Trade has a different meaning in legalese. It's the tools your livelihood depends on, so in case of bankruptcy, they will not be taken away by creditors.
Sunday, January 20, 2013
Software State of Affairs - Technical Status Quo
I just finished a project to refactor our Web application, written in JSP many years ago. This runs in our EA Server (Ver 5.5) environment. The environment itself is 8+ years old, and the versions are reflective of that time. The technology used was already outdated by the time the software went into production. There were several violations of the modern web programming techniques. Lot of embedded Java coding, at will call/connections to servers and database etc. caused the program to fail during peak loads.
Well, there is a lot to talk about that software and the improvements we did. I will do so in other posts. In this post, I want to focus on the issue I see in every software maintenance projects - the status quo in keeping the software architecture intact, irrespective how much it changes in business functionality.
I frequently compare software to a car. The comparison arises in that both are built, run, need to be maintained, and tuned up. That's where the comparison ends. We don't keep adding functionalities to a car. We add bells and whistles like tint, color, alarm etc, but never add functionality or change it's function completely; we don't make a car run on square wheels or sideways. Though the engine gets hot, we don't use a it as a toaster or to cook chicken. But, in software we do exactly that.
When a custom software is designed or purchased, the users typically get all the functions they need. But, over the years users change and so does the business. We keep adding enhancements, bug fixes to the software constantly to meet the (sometimes unreasonable) business requirements that were never dreamed of, when we built the original software. The software thus gets bloated over a time period. We don't adjust the technical architecture or design to match this new software obesity and it eventually comes to a crawl, at which point everyone blames the software and want to replace it. By then, it may be too late! Many custom software out there have nice front ends, but - pardon my French - crappy code inside. It's like a flash new bus pulled by a mule, no offence to the mule!! I blame this on Software Status Quo! We tend to cater to business so much, we neglect the software itself in the process!!
Lot of this is due to the way companies budget for Software projects. Several departments tend to have a "lights-on" policy, which means keeping the software up and running 24X7 without any outage. Companies typically have a lot of budget for "operations" but not real software maintenance. As a consultant, I've seen IT departments struggle to define my work as part of "lights on", so they could continue to get funds for my contract.
Typically, the business side budget pays for the "operations". Except for a handful of savvy business managers, rest of them don't really understand software. To them, the "pretty" screens and "nice" reports are the software. As long as these happen, software is working fine. When you work with such managers, it is not easy to get budget for retrofitting software. "Business" requirements will always take precedence to technical requirements.
But, business is not entirely to blame for this. Recently, when I started introducing the changes in our software, I saw advantages of using more Java in a mostly Powerbuilder environment. There was some awe initially, then reluctance and finally an indifferent acceptance. New architecture means, new problems and tougher maintenance, at least initially. Now, I see why this status quo is continuing. I always hear senior programmers say, if it's not broken and/or users didn't want it, we wouldn't fix it. Where as, this is a safe approach to keep business happy, it hurts the software in the long run.
To end such status quo, business must stop treating software like stationary and recognize and allocate budget for it to be "tuned" regularly. I also feel, real end users (not just business managers) must be regularly consulted as to the performance and usage of the application. I've seen many a software projects fail, in spite of extensive design and funding, because end users simply rejected the software. Developers must also constantly look for ways to change and improve the software itself (not just it's business functionality) and strive to include these in their releases. I see several developers less motivated with the routine business fixes. Adding in technical items, may help to re-energize the developer base.
Many years ago, I worked in a warehouse in NJ where they were using an AS/400 for order tracking, but the software itself was so outdated, the users were improvising. Believe it or not, they would print a report out of this system, then they would input that information into System/36 along with accounting information and export to a file. The next smart user takes that to mainframe to finish up the final entry, where corporate wanted the original shipping and accounts information. Those systems people in Corporate, 3000 miles away conveniently ignored this status quo, that this process actually continued for many years until truly yours stepped in! Ironically, around the same time, corporate headquarters poured in millions to port some of their mainframe applications to flashy client/server tools.
I normally keep notes of all technical issues or a wish list for the software I work on. Then when I get to work on some business requirement in that software, I include any appropriate technical fix. Recently, while working on a business requirement involving Java stored procedures, I rewrote the logging mechanism completely, so we can support it better. Previously, developers were compiling and distributing this code to DBAs completely outside of the build process. If one class had to be updated, just compile it and patch it into the Jar file!!! This was error prone and I found some source code older than compiled code. I introduced eclipse and Ant to reduce human errors. Of course, it took longer to implement the task, but it pays for itself in the long run!
Oh yeah, we did solve the impasse at the warehouse in NJ, but it needed some thinking outside the box. I got the blessing from the local business director and with the help from users and corporate developers, we rolled the users to corporate mainframe (of course, after making necessary changes on the mainframe to accommodate them) and wheeled the AS/400 out, the one I was there to support in the first place! The company saved a bundle and everyone was happy!
Well, there is a lot to talk about that software and the improvements we did. I will do so in other posts. In this post, I want to focus on the issue I see in every software maintenance projects - the status quo in keeping the software architecture intact, irrespective how much it changes in business functionality.
I frequently compare software to a car. The comparison arises in that both are built, run, need to be maintained, and tuned up. That's where the comparison ends. We don't keep adding functionalities to a car. We add bells and whistles like tint, color, alarm etc, but never add functionality or change it's function completely; we don't make a car run on square wheels or sideways. Though the engine gets hot, we don't use a it as a toaster or to cook chicken. But, in software we do exactly that.
When a custom software is designed or purchased, the users typically get all the functions they need. But, over the years users change and so does the business. We keep adding enhancements, bug fixes to the software constantly to meet the (sometimes unreasonable) business requirements that were never dreamed of, when we built the original software. The software thus gets bloated over a time period. We don't adjust the technical architecture or design to match this new software obesity and it eventually comes to a crawl, at which point everyone blames the software and want to replace it. By then, it may be too late! Many custom software out there have nice front ends, but - pardon my French - crappy code inside. It's like a flash new bus pulled by a mule, no offence to the mule!! I blame this on Software Status Quo! We tend to cater to business so much, we neglect the software itself in the process!!
Lot of this is due to the way companies budget for Software projects. Several departments tend to have a "lights-on" policy, which means keeping the software up and running 24X7 without any outage. Companies typically have a lot of budget for "operations" but not real software maintenance. As a consultant, I've seen IT departments struggle to define my work as part of "lights on", so they could continue to get funds for my contract.
Typically, the business side budget pays for the "operations". Except for a handful of savvy business managers, rest of them don't really understand software. To them, the "pretty" screens and "nice" reports are the software. As long as these happen, software is working fine. When you work with such managers, it is not easy to get budget for retrofitting software. "Business" requirements will always take precedence to technical requirements.
But, business is not entirely to blame for this. Recently, when I started introducing the changes in our software, I saw advantages of using more Java in a mostly Powerbuilder environment. There was some awe initially, then reluctance and finally an indifferent acceptance. New architecture means, new problems and tougher maintenance, at least initially. Now, I see why this status quo is continuing. I always hear senior programmers say, if it's not broken and/or users didn't want it, we wouldn't fix it. Where as, this is a safe approach to keep business happy, it hurts the software in the long run.
To end such status quo, business must stop treating software like stationary and recognize and allocate budget for it to be "tuned" regularly. I also feel, real end users (not just business managers) must be regularly consulted as to the performance and usage of the application. I've seen many a software projects fail, in spite of extensive design and funding, because end users simply rejected the software. Developers must also constantly look for ways to change and improve the software itself (not just it's business functionality) and strive to include these in their releases. I see several developers less motivated with the routine business fixes. Adding in technical items, may help to re-energize the developer base.
Many years ago, I worked in a warehouse in NJ where they were using an AS/400 for order tracking, but the software itself was so outdated, the users were improvising. Believe it or not, they would print a report out of this system, then they would input that information into System/36 along with accounting information and export to a file. The next smart user takes that to mainframe to finish up the final entry, where corporate wanted the original shipping and accounts information. Those systems people in Corporate, 3000 miles away conveniently ignored this status quo, that this process actually continued for many years until truly yours stepped in! Ironically, around the same time, corporate headquarters poured in millions to port some of their mainframe applications to flashy client/server tools.
I normally keep notes of all technical issues or a wish list for the software I work on. Then when I get to work on some business requirement in that software, I include any appropriate technical fix. Recently, while working on a business requirement involving Java stored procedures, I rewrote the logging mechanism completely, so we can support it better. Previously, developers were compiling and distributing this code to DBAs completely outside of the build process. If one class had to be updated, just compile it and patch it into the Jar file!!! This was error prone and I found some source code older than compiled code. I introduced eclipse and Ant to reduce human errors. Of course, it took longer to implement the task, but it pays for itself in the long run!
Oh yeah, we did solve the impasse at the warehouse in NJ, but it needed some thinking outside the box. I got the blessing from the local business director and with the help from users and corporate developers, we rolled the users to corporate mainframe (of course, after making necessary changes on the mainframe to accommodate them) and wheeled the AS/400 out, the one I was there to support in the first place! The company saved a bundle and everyone was happy!
Sunday, January 6, 2013
Gotcha: EA Server config. XML file comments
Sybase EA Server (Ver 5.5) provides 2 interfaces for the admin to work with the Server and the Repository. Sybase Jaguar Manager is the GUI interface, that one can use to manage the Server and the repositories.
Jagtool is the command line equivalent of the above manager. There are a lot of things you can do in Jagtool and with host scripting (batch files in windows), you can make real powerful scripts to execute these in batch mode.
EA Server typically configures all elements using properties files. But, it also allows us to use XML files to configure the server. These are available at various levels such as Server, Package, Component, Connection Cache etc.
This gotcha is about writing comments in an XML file meant for configuring EA Server. Single line comments follow the XML/HTML standard.
When you write multiline comment, make sure the tags are on their own line, thus:
If you include any text in the first line, it will error out.
Note: You only need <!-- to mark the beginning of a comment; I doubled exclamation mark to avoid losing the comment line in view!! Otherwise, Wordpress HTML processor would have hidden it from view.
Jagtool is the command line equivalent of the above manager. There are a lot of things you can do in Jagtool and with host scripting (batch files in windows), you can make real powerful scripts to execute these in batch mode.
EA Server typically configures all elements using properties files. But, it also allows us to use XML files to configure the server. These are available at various levels such as Server, Package, Component, Connection Cache etc.
This gotcha is about writing comments in an XML file meant for configuring EA Server. Single line comments follow the XML/HTML standard.
<!!-- this is a single line comment -->
When you write multiline comment, make sure the tags are on their own line, thus:
<!!--
This is a
multi-line
comment
-->
If you include any text in the first line, it will error out.
Note: You only need <!-- to mark the beginning of a comment; I doubled exclamation mark to avoid losing the comment line in view!! Otherwise, Wordpress HTML processor would have hidden it from view.
Ubuntu: File Managers
I am continuing to use Ubuntu as my primary desktop and I am loving it(I still use Windows desktop. It has it's advantages). As I get used to the environment, I am always looking to mimic familiar options from Windows. This time around I was looking to find out about adding applications to Context menu in Ubuntu. Like always, I stumbled on something bigger - File Manager(s).
Earlier I posted about Linux architecture and how different layers of the Operating System can be replaced/added. If you read that post, you will see how I struggled with Unity Desktop that came with Ubuntu 12.04 and eventually learned to replace it with Gnome desktop. Apparently, each desktop has it's own flavor of File Manager and then some.
If you are coming from the Windows world (like I did), here are some terminologies for you. What we call Windows in Windows Operating System (like Windows XP) is really the Desktop. We saw earlier there are many different Desktops including Gnome, KDE, Unity etc. The Desktops have several parts and one of them is a File Manager. This is called Windows Explorer (it used to be called File Manager in versions before Windows 95). There are choices for Explorer, but I've not seen many people replacing the Windows Explorer with another.
In Ubuntu (for that matter entire *nix [Unix, Linux]) world, everything is customizable and/or replaceable. I was able to replace my troubled Unity desktop with Gnome desktop and overcome some issues earlier. When I started researching on Context Menus, I stumbled on Nautilus package, which is the default File Manager in Gnome. Since Ubuntu 12.04 didn't support Gnome desktop natively, it didn't install Nautilus either. It had a File Manager called Thunar instead. And, when I installed Gnome, unknowingly, I had also opted for Nautilus, default file manager for Gnome. Apparently, there are at least 20 different File Managers available. See this blog.
This site has a nice picture of the Linux Architecture.
In Gnome Desktop, you get to File Manager by clicking on Places menu on top.
Fig 1: Menu to reach File Manager
Fig 2: Nautilus File Manager
So what about Context Menu?
Coming back to adding context menu, it's done using Nautilus Actions.I had to install it first. (This you could do it in Ubuntu Software Center or use apt-get command). Once you installed it, you can add context menu items as shown below:
Fig 3: Nautilus Actions Configuration
And of course, after changing the context menu you need to reload the Nautilus File Manager. To do this, you can type the below command:
<code>killall nautilus</code>
This kills all processes associated with the particular program (here nautilus). For Windows users, this is like going to Windows Task Manager and doing "End Program". See here for more on killall. Incidentally, they also talk about killing nautilus!!
Running KDiff3 from Context Menu
The program I was trying to add to context menu was KDiff3 (it's a great diff utility, try it, if you haven't already. I use it on Windows as well). Nautilus Actions allows you to add programs to the context menu. I select 2 files to be compared and right click to select and open KDiff3 and voilà! KDiff3 opens and diff's the files automatically!
References
Earlier I posted about Linux architecture and how different layers of the Operating System can be replaced/added. If you read that post, you will see how I struggled with Unity Desktop that came with Ubuntu 12.04 and eventually learned to replace it with Gnome desktop. Apparently, each desktop has it's own flavor of File Manager and then some.
If you are coming from the Windows world (like I did), here are some terminologies for you. What we call Windows in Windows Operating System (like Windows XP) is really the Desktop. We saw earlier there are many different Desktops including Gnome, KDE, Unity etc. The Desktops have several parts and one of them is a File Manager. This is called Windows Explorer (it used to be called File Manager in versions before Windows 95). There are choices for Explorer, but I've not seen many people replacing the Windows Explorer with another.
In Ubuntu (for that matter entire *nix [Unix, Linux]) world, everything is customizable and/or replaceable. I was able to replace my troubled Unity desktop with Gnome desktop and overcome some issues earlier. When I started researching on Context Menus, I stumbled on Nautilus package, which is the default File Manager in Gnome. Since Ubuntu 12.04 didn't support Gnome desktop natively, it didn't install Nautilus either. It had a File Manager called Thunar instead. And, when I installed Gnome, unknowingly, I had also opted for Nautilus, default file manager for Gnome. Apparently, there are at least 20 different File Managers available. See this blog.
This site has a nice picture of the Linux Architecture.
In Gnome Desktop, you get to File Manager by clicking on Places menu on top.
Fig 1: Menu to reach File ManagerFig 2: Nautilus File ManagerSo what about Context Menu?
Coming back to adding context menu, it's done using Nautilus Actions.I had to install it first. (This you could do it in Ubuntu Software Center or use apt-get command). Once you installed it, you can add context menu items as shown below:
Fig 3: Nautilus Actions ConfigurationAnd of course, after changing the context menu you need to reload the Nautilus File Manager. To do this, you can type the below command:
<code>killall nautilus</code>
This kills all processes associated with the particular program (here nautilus). For Windows users, this is like going to Windows Task Manager and doing "End Program". See here for more on killall. Incidentally, they also talk about killing nautilus!!
Running KDiff3 from Context Menu
The program I was trying to add to context menu was KDiff3 (it's a great diff utility, try it, if you haven't already. I use it on Windows as well). Nautilus Actions allows you to add programs to the context menu. I select 2 files to be compared and right click to select and open KDiff3 and voilà! KDiff3 opens and diff's the files automatically!
Fig 4: Nautilus Context Menu
Fig 5: Kdiff3 From Context Menu
References
- http://www.tuxarena.com/2011/06/20-file-managers-for-ubuntu/
- http://www.techdrivein.com/2010/05/what-is-nautilus-elementary-and-how-to.html
- https://live.gnome.org/Nautilus/Screenshots
- http://www.tellmeaboutlinux.com/content/linux-architecture
- http://en.wikipedia.org/wiki/File_manager
- http://askubuntu.com/questions/88480/adding-extra-options-to-right-click-menu
- http://www.omgubuntu.co.uk/2011/12/how-to-add-actions-emblem-support-back-to-nautilus-in-ubuntu-11-10/
- http://www.linfo.org/killall.html
Subscribe to:
Posts
(
Atom
)