Source Code Control

Background

I always think, we programmers are like writers and artists. We create software that could be powerful and beautiful. There is no lack of abstraction in our work either!

There is one difference, though. When an Author writes a book, she could revise her work so many times at her desk, but only the final outcome becomes public. Not so with our work! A programmer comes up with several revisions of his/her program before it goes to production. Even so, there may be a bug or something was missed that he/she has to revise it again.

A software could also be "backed out" to a prior version. (Even software giant like Microsoft does this once in a while. Remember when people rejected Windows Vista, their solution was to go back to XP, the prior version of Windows! I myself went back to Gnome desktop on my Ubuntu 12.04 desktop, because of some issues with Unity desktop). And of course, someone might point out that something worked in an earlier version, that you will have to dig up the sources for that version, find out what changed and try to re-incorporate that code back into the new version!! I can't imagine an artist having to go through this type of revising of his/her painting! (I agree a book may have editions, but it's always move forward for them.)

So, if you are a software developer you need to worry about keeping track of the history of the program. Not just the history, but all the files that made up a particular version! And to do this we resort to so many ingenious(?) ways. How many times, have you seen files with .bak, .sav, .org etc extensions on a developer's machines? (guilty!). Some may even organize their files in dated directories, so they can go back to a specific date (guilty as well). But when your program is made up of so many files (gone are the days when you wrote all your code in one file!), such rudimentary methods of keeping track of files won't work! Even more so, if you are working (collaborating) with other developers.

Source Code Control or Version Control

Wouldn't it be better if a Software can track these information at the source level, so it's easy to pull up any version of the software sources and rebuild it? Such a software tool exists for over 40+ years and is known variously as Source Code Control Software (SCCS), Version Control System (VCS), Revision Control System (RCS) and even Software Control Management (SCM). Dozens of tools have been developed with these acronyms in their names. Some of the most popular ones over the years are SCCS, RCS, PVCS, Perforce, Visual Source Safe (VSS), Mercury etc. And in the open source arena, we have had CVS, SVN and Git. Please refer to the wiki site for comparison of many more tools available on the market. The difference is not only in the names, but in techniques and technology involved. Newer tools support concurrency better.

The least a VCS can do is to keep track of history of files and thus offers reversibility. With various versions saved in the repository, it is only natural that we require the VCS to provide a good compare (diff) utility or at least the capability of using external diff tools.  Any type of files can be saved in the repository including binary files. Though, the diff utility is typically geared for text files.

More traditional VCS offers a centralized repository. SCCS, PVCS etc belong to this category. More modern ones offer more distributed approach. With developers from around the world working on open source projects, distributed repositories more sense. And the other characteristics that go with the territory is the concurrency. Again, modern VCS have more and more concurrent support built in. For e.g., CVS refers to Concurrent Version System.

Further Reading

Please refer to the Eric Sink's website (his book is available in PDF format on his site) for detailed discussion on the SCCS tools. This blogger recaps the history of Source Code control in a funny way! And the wiki page like always delivers a nice introduction to this topic. Eric Raymond's paper explains various generations of Version Control Systems nicely.

http://www.ericsink.com/vcbe/index.html
http://www.catb.org/esr/writings/version-control/version-control.html#why_vcs
https://code.google.com/p/pysync/wiki/VCSHistory

Software State of Affairs - Tools of Trade

Early man did a lot of things by hand, but soon started inventing tools (Simple machines in physics) to help him do things quicker and easier. Modern tools are way too complex than these early predecessors.

Tools of Trade¹

In the modern world, we depend on tools so much, we don't even realize. See this little Physics activity page to see how many you may be using. Without tools, our lifestyles as we know will cease to exist. Every trade has associated tools, which we will call Tools of trade (In legalese it has a more restrictive meaning. See note 1). When we call a plumber, we expect him to come with the tools necessary, like the wrench etc. The electricians use different set of tools of which, voltage tester may be the most important. If they don't have the right tools, we know what will happen. At the very least, you will be looking up Yellow Pages to call the next technician.








Software Tool Kit

IT field borrows a lot of terminologies from other fields. We are engineers and technicians and architects that "build" software products, just that you won't be able to see the finished products except on your computer screen. While building, running, testing software products, we use a lot of tools. Like the product they help to build, these tools are software themselves. We use Compilers, Editors, Linkers, Runtime etc. to build and execute our software. But, unlike a plumber going to Home Depot to get his tools, we could simply download it. At the same time, while a plumber's tool chest may always look alike, our tool chest varies depending on the field and the software we build. And unlike a plumber or an electrician, we have the luxury of building our own tools, more so now in the age of Open Source.

Tools I use

As a consultant, I am constantly exposed to different tools at different sites and for some tasks, I tend to keep my own time tested tools. Here are some Open Source tools, I currently use. Often times, I use more than one tool for a task and combine the results. These are mostly for Windows environment. I will share other tools and for Linux separately.

Windows Grep I use it to search for text in files. Same like what Notepad++ or Textpad would search, but has a little more flexibility like the grep.

Many years ago, I used Examdiff extensively. Lately, WinMerge is the diff tool I use most frequently. WinMerge helps me compare at directory level and helps to merge files quickly. There are some limitations, but I love it. I also use K-Diff3 and CSDiff as each of these offer something unique. These are all free software. Another great diff tool I found was Beyond Compare. My next software purchase would be this. The above tools don't provide a good report for directory compare (Beyond Compare may). To do this, I found diffUtils and a nice script I found on the internet, to format the output in HTML.

Greenshot This is the open source screen capture tool I use. Pikpik is a great tool as well. Again, each of these have some feature that others lack.

OpenCommandPromptHere There are several names like Command Prompt Here, SendtoToys etc. Essentially, creates a context sensitive menu in Windows Explorer, so you can open command prompt from where ever you are in Explorer. (you don't have to CD into any directory manually).

BareTail I recommended this earlier; I use this to tail Jaguar.log etc. It's really good. You can highlight lines with different phrases.

YEd This I use a lot to do diagramming. It's more a graph editor than a Graphics editor. Works for me. While researching on diagramming, I found UMLet to be interesting. Dot, Dotty and GraphViz help me with automatically generating some flow diagrams.

Apart from these I use Ant, Eclipse, JMeter, Xampp etc as part of the my development environment. I also use Libre Office (Open Office's successor) a lot.

Tools I build

I am always looking for ways to make the computer do the work for me, instead of doing things manually. I feel, if a task has to be done more than once, it deserves a script or a tool. When the tools available are not good enough for a particular task, I try to build some on my own as well. But this is also an excellent opportunity to learn new techniques and keep your skills sharp. I do these during down times and/or as part of the main task.

I am sharing some of my tool building experiences here, hoping it would benefit someone else. If you are a Software developer and somewhat less motivated with the stuff you are working on, give tool building a try next time. The software will benefit from it too. All you need is some imagination and some laziness to resist doing anything manually! Gone are the days, when you had to purchase special software to build such tools. You will be amazed to see how much programming capabilities a standard computer can offer you. For e.g., if you have Word or Excel, you can program build powerful macros (VBA programs). Apart from these, you always have Open Source  scripting languages like PERL, Python, PHP etc.

If you recognize early enough, the tools you build could become integral part of a project as well. At Velos, while working on the task of upgrading the database (Sybase SQL Anywhere) for a newer specification, I decided convert the specification into an Access Database. Once the new Spec. was in a database, I was able to compare existing database elements with the new ones and voilà!, the database upgrade script could be generated right out of the database itself. They had a Powerbuilder application to design and save screens (datawindows) in database (blob columns). After going through problems with the blob, decided to write a tool to view, edit, correct screens outside of the application. Few more times of using this, I ended up in writing a simple scripting language to perform these same tasks in a batch mode. Necessity is the mother of invention! Several of these tools were actually used by the support person to identify and fix issues at customer sites.

Some of the casual tools I build for one time use, end up as permanent additions to the team's tool chest. At Capital group, one developer was collecting batch job stats on a daily basis, and when he went on vacation, he gave a page long instructions on how to get these details in Excel. The task was, to  go to the web pages (someone with tool mentality, pulled the stats from Autosys into the page, using CGI-PERL), copy the job names to a spreadsheet and make adjustments to be presentable. When my turn came, I got tired of copying, pasting and correcting pretty quickly. Before the next time, I added a "Web Query" in Excel to pull up the web pages directly into the spreadsheet and wrote some VBA macros to clean up and present the spreadsheets. Eventually, it became full-fledged product in it's own right, everyone in the team started using it. Page long instructions gone. (Of course, when I left, I left a few pages long document about the macro, but that's another story).

Similarly, in the same company there was another task, developers and/or the analysts had to do. Every now and then company received fund codes from outside (at the time I didn't know much, these are the GICS codes we received from MSCI), that had to be reconciled against codes that existed in the system already. Some codes may be replaced with new ones, some may get new descriptions, some may be discontinued. So, they would enter these "changes" in a spreadsheet or word document and then ask a developer to convert these to SQL (INSERT/UPDATE/DELETE) statements to be put into production. Enter yours truly, the lazy user. I wrote some excel macros that would import the data from the Oracle table (using ODBC), reconcile and generate the SQLs for the developer to run in production. Yes, it did take a little more time to develop and test it, but next time similar request came, it was a breeze.


An Excel macro dubbed "Query Tool", I developed at Capital Group tops it all. It again started as a manual task. Goal was to identify some codes (for e.g., security codes) from another system and see if it existed in our database. This was done manually too. User would extract data from the table into a spreadsheet and look up each code. Sometimes, codes might have been mistyped, so they have to match partially or look at other attributes to identify and correct. At the end of this, they may have gone through a spreadsheet with 1000's of codes in several passes, identifying codes from another list they had. Time consuming, laborious and error prone. If this is not what the computer does best, what else? So, I started coding the excel macros to do the repeated search of the codes they entered. Little by little, it became a full fledged  querying tool (using Database Query - through ODBC - in Excel). The tab containing the actual SQLs helped customized the lists further. What resulted was a nice product. The users loved it.

These one-off solutions may also help you save a project. At FHLB, we were working on a data extraction project. This was part of the data reporting requirements of the bank set by FHFB. I created an Access Database as a prototype to a much bigger data-warehouse built in Oracle by a team of ETL developers. I kept this as a tool to validate the data/report from the ETL tools, before we sent the data to FHFB. All was going well, but suddenly one day, we found a flaw in the Data-warehouse design and the multi-million dollar project simply collapsed. Luckily my boss saw this risk early, supported my efforts to spruce up my Access Database. Believe it or not, my little tool eventually became the official "data reporting tool" for the company, at least the first year.

Journey Continues..

Recently, one developer mentioned about how to check a file in Textpad for a record with odd length. This was the idea behind this PERL script. Another opportunity for such a script was where a batch program creates a large file (XML in this case) that had to be split into smaller chunks. Perfect candidate again for a PERL or a Python script. I did a quick prototype, but it died down for lack of interest. I hear they do it with some tools like access DB and a lot of manual "pulling the hair". And then there are tasks like parsing logs and so forth. A well written script can analyze and report stats from these files, in a jiffy.

The other day, we were discussing testing for a Database upgrade. I brought up the topic of monitoring and my boss said, "you can make yet another tool for it"! I know he was kidding, but I probably will or if I am in luck, I may be able to just download something from the internet. The journey continues!

---

1. Tools of Trade has a different meaning in legalese. It's the tools your livelihood depends on, so in case of bankruptcy, they will not be taken away by creditors.

Software State of affairs - Maintaining OO

Last week, a friend at work was analysing code in Powerbuilder and question came up about function Overloading and we branched into Object Orientation in general. That question got me thinking. Though Object oriented programming has been around for a while, the concepts of OO are not completely understood or followed.

I posted about OO concepts here earlier. To do that I bought and read several books, sifted through the web a lot. There were so much confusion and conflicting ideas. Most of those books use obvious objects like animals, people etc in explaining OO concepts that what we learn in those is not enough to translate in to real life OO programming. I mentioned about some of these problems in my earlier post. More and more languages are adding OO concepts, yet people who are used to traditional (procedural) way of programming, struggle to adapt to it or even completely reject it. Often times, Syntax comes in the way of understanding. For e.g., Java borrowed so much from C, C++, that it's often considered to be just enhanced C, thus OO nature of it is completely ignored. And even if these developers switch to OO eventually, the objects they come up with are just repackaged C code, thus perpetuating the problem!

The application I currently support was written in Powerbuilder using a framework called EAF. The software also has business framework that is built around this, definitely a good extensible architecture. Yet, the software has degenerated over the years. Part of the problem is lack of understanding of the framework and/or OO concepts in general. I remember, when I started here, this framework was considered to be the main reason for all evil in the software! There was not enough documentation about proper framework usage, thus it was easy to blame it!

When new developers came in, they wrote and rewrote functions and objects that completely ignored underlying framework thus code is bloated. Further, supporting business has been utmost priority, so such technical issues were completely ignored. I feel, every maintenance project should include the maintenance of the software itself - in addressing technical issues and retuning it to keep it running good, like we do with our cars. After all, business will eventually be affected, if we didn't take care of such technical issues in time! The very first problem I solved in this software was a technical issue - there was a memory leak (due to improper handling of Object pooling) in the software that kept crashing the server during peak usage in the prior years. Business of collecting taxes, was indeed affected during those crashes. And I cannot imagine how much time, effort and resources were wasted during those times.

Another problem I see in software maintenance, is the tendency to solve code smells and code bloats by adding more resources - more servers, more disk space, more memory. When a car runs sluggish, we can't solve it by just adding more engines or more tires, can we? Why is it OK with software? Prior solution to the memory leak problem I mentioned above, was to add more servers and a script to automate the Server Restarts! Once the memory leak was fixed, no more frequent crashes!!!

To curb some of the issues in the application, I recommended and started on a project to re-factor some of the code in the application earlier this year. Main goal was to improve performance in our web application, but I wanted to clean up and reintroduce some of the framework methodologies.  Though there was some push back initially, other developers started buying into the idea. Yet, I see the reluctance to change existing ways; after all the software works during normal (low?) usage and business is OK (if not happy), why change those? When I first introduced log4J (logging libraries for Java) for logging in our JSP pages, it took the team at least couple of months to buy into the idea, in spite of me showing stats to support the change.

Our application architecture uses EA Server (EAS) as the middle tier. EAS supports Java and Powerbuilder objects naturally.While Powerbuilder is great for developing applications quickly, Java was used for any multi-threaded activities such as caching and logging etc. We even have Java Stored procedures running inside Oracle database! I recently introduced tools like Eclipse, Ant etc. to streamline Java development activities. With such exposure and addition of new Java packages prompted a team member to say to the effect "we are becoming a Java shop whether we like it or not". I've tried to explain in vain that it was all there before and why we even need them!!

Part of the problem is also lack of career planning and/or training. I remember the days when companies invested in training employees to keep up with the industry trends. Now, we seem to be spending more in buying those extra servers to support more memory leaks :)

To OO or not to OO

I originally wanted to name this post "Object Oriented Concepts - Contd....", but then the Hamletian dilemma came to mind. Every programmer faces this, even today, many years after OO has been widely accepted. Part of the problem is that the OO paradigm requires a change in how we think about a problem.

As we saw earlier, in traditional languages we always think "how to do a task" in a procedural sense. This we could adapt to - after all, we humans are good in giving instructions and making recipes and grocery lists. Programming was a natural extension of this - we told the computer what to do and how to do it. We told the computer what to do while feeding data (nouns) required as parameters to these commands (verbs).

When it comes to OO, we have to start thinking what will be the objects we need and "leave" it to those objects to do the work. Really, it's just repackaging, but it's not easy. We are now forced to think of everything in terms of objects. If it's objects like animals, cars, people etc that are typically used in examples about OO, we can understand. But the moment we try to come up with objects for abstract concepts and purely action oriented tasks, it becomes a daunting task.

Baking a Cake

For e.g., let's look at a simple problem of baking a cake (no disrespect to the chef in the kitchen, but let's abstract it here as simple task here for a minute). A recipe typically talks about how to get, do, heat, bake etc. All verbs. Interspersed will be details about data to be used - how much flour, how much sugar, what temperature, how many minutes etc. These "nouns" are used as parameters to the recipe and off we go. Of course, every now and then people try to "reuse" recipes by switching ingredients.

This simple recipe for a cookie , that highlights these points. It has an instruction with 3 simple steps to make delicious cookies. It has some ingredients (attributes) mentioned - dough, apricot jam, and parameters like temperature at 350 degrees, time for 15 - 18 minutes and 1/3  cup of jam. It is straight forward and it even "reuses" recipe for "basic sugar cookie dough"! We certainly know who is making this cake - you, and the ingredients and implements you will be using.



In our abstraction in a procedural language, we created a program with the instructions to bake a cake. We probably broke down each step into a separate procedure (or function), and created a main function that looked like the recipe and called these functions to get the baking done. We created some variables outside of these functions for the parameters required. This works and even is simple to write. But, we assume a lot of things here, like the main function itself acts like the baker executing the instructions. Controlling the oven may be built in as a built in function, but at some point it has to call something to make it happen.

In OO, the task gets a bit more complicated! First, you would have to start thinking differently. Now, you will have to think about the nouns first. Of course there is cake, the final product. But, there is also the bowl, oven and a fictitious baker - may be you, but object none the less. These other objects were included in our procedural programs, but were not given much importance. Any reusable, replaceable parts of the program will now have to be separate objects. In our example, if we make the Oven and the Baker and bowl objects correctly, they could be reused in making/baking pies as well as switch to make or even pot roast! Guess what, you can even replace the baker easily!! This is what makes OO so reusable and to some "so bloated!".

If you are interested in OO cakes, here is a link on a cake OO.

 

 

 

So, To OO or not to OO?


OO is almost everywhere around us. Language after procedural language (recently PHP, Coldfusion, ABAP, Flash ActionScript) have added OO capabilities in recent years. But then, there are also others that write about the difficulties and shortcomings of OO. The post titled  Debate Still Rages Over Object-Oriented Programming discusses such dilemma. While you are there, don't forget to check out  Marc Funaro's orginal post here: How OO Almost Destroyed My Business. He has articulated very nicely, why he couldn't and didn't have to switch over to OO completely. And see here for a funny take on Java's OO techniques! Irrespective of such views, OO is here to stay. Even if you don't use OO technology currently, chances are, sooner or later you may be exposed to it.

OO methodologies could get complex and could actually distract a developer from the original goal of getting a working program quickly. General rule of thumb I have, is to use procedural for any small (read quick and dirty - most of my utility programs and scripts fall under this) programs and use OO methodology for any larger (business, corporate, enterprise, framework) programs and programs that have potential to grow in the future. (The truth is, even the quick and dirty program/script could outgrow their originally intended lifetime and objects do creep into those every now and then!). OO definitely enforces certain planning ahead of time, so it takes a bit longer to develop it upfront, but if a program lives long enough, re-usability makes up for the cost over time.

In the cake example above, if I am just making a cake for the family on a weekend, simple procedural implementation would be fine! But, if you are going to be making a cake each weekend or thinking of starting a bakery with several branches down the road, then I better start thinking in OO terms, so we don't have to re-invent a lot of stuff later.

General Design Considerations

Irrespective of whether we use OOP (OO programming language) or not, we are bound to use the modern techniques like Abstraction, Encapsulation, Inheritance and Polymorphism in our program designs. Underlying all these, is the code reuse, data hiding and of course modular design. These of course are not OO by themselves, but are the foundations of OO techniques. By nature of it, OO is almost automatically modular.

• Keep your code Clean

If the code is not "cleanly" written,  even you cannot understand it a year later. By clean, I mean well commented (documented) and following some coding standards and best practices.

As for coding standards, what's good for one language isn't always good for another - for e.g, in Java we follow Camel case for naming variables while in PB  identifiers are mostly in lower case. So, if your system includes programs in different languages, make sure to include language specific standards as well. Also, best practices for any language may change over version. For e.g., there may be newer constructs and/or functions may be available in a later version that are more efficient than what was available earlier. Make sure to keep up with those in your programs.

OO Note: OOP languages such as Java promote commenting and documenting of the programs easier, with the Javadoc tool (even more so, with IDEs like Eclipse and Netbeans). A well written framework for Java, actually enforces some coding practices automatically. But we can achieve such coding practice with libraries of well written API. A well commented header file distributed along, could act as the documentation for the API.

• Keep it modular

By breaking down your program into smaller modules, your program becomes more manageable. If your program includes specific modules for various functionalities, it will be easier to replace or rewrite or even add a specific module without affecting the entire program. For e.g., if you isolate all database related functions to a separate module, then if your database changes, then all you have to do is to replace this one module. In our cake example above, we could simply add an object for Microwave and start using it instead of Oven, if oven breaks down!

OO Note: Writing such "modular" programs comes naturally in OO languages as the program is essentially built around objects (or classes). But this concept itself is not new. For e.g., in C, you can achieve this by keeping different modules in separate source files), your program becomes more manageable.

• Make the code Reusable

How many times, have you "cut and pasted" your code from one source file to another to "reuse"? If the program is properly designed, then the reuse would be much easier and automatic.

OO Note: In OO, inheritance promotes reuse naturally. A well written framework or class libraries contain reusable classes and methods. Apart from using the classes in the framework themselves, one could extend those via inheritance and thus more custom classes for others in a company to reuse. For e.g., you can create custom loggers specific for your need, by extending classes in log4J library.

But again, this is not specific to OO. In a procedural language like C, one could build libraries of reusable functions. While C doesn't allow extension through inheritance, reuse can be achieved through composition. An inheritance is typically defined by a is-A relationship, while a composition by a has-A. Here is an excellent article that compares the two.

Hello World!

This is my real first post in Powerbuilder. (I wrote a lot about PB and my association with it before). Like it is customary for any tutorial or book on programming to start with a program to print Hello World, I will try to do the same in Powerbuilder.

Before I jump into it, let's some simple ones: (I am only including the actual lines and not full program code here,  please refer to the really interesting site that prints hello world in various languages - here)

In a Pascal program, you can just say,

[sourcecode language="delphi"]
{Hello World in Pascal}
writeln("Hello world");
[/sourcecode]

(In Delphi which is based on Pascal, it is written as,

[sourcecode language="delphi"]
{Hello World in Delphi}
WriteLn('Hello World');
[/sourcecode]

In C, you can do

[sourcecode language="cpp"]
/*Hello World in C*/
puts("Hello World!");
[/sourcecode]

In Basic Language, (which contributed to  Powerbuilder's Powerscript) you can do

[sourcecode language="vb"]
10 REM Hello World in BASIC
20 PRINT "Hello World!"
[/sourcecode]

Coming to the more Object oriented world,

In C++ you can write,

[sourcecode language="cpp"]
/*Hello World in C++*/
cout << "Hello World!" << endl;
[/sourcecode]

And in Java, this would be,

[sourcecode language="java"]
/*Hello World in Java*/
System.out.println("Hello World!");
[/sourcecode]

Well, almost of all of them print to a console - a text based screen, like in DOS, Unix, Linux. The same task of showing hello world to the user takes a bit more, when it comes to a windowing world (like MS windows, X-windows, Apple mac interface). The presentation becomes little more important. After all, in such an operating environment, everything is shown inside a window, so our program needs to write into one such too! In a windowing environment, simplest way to present a text is using some type of message box, which is a small window showing the text (and may be some question buttons).  Luckily windows API offers functions to create such a messagebox with ease. Let's see how we could do the same in Java, for a windowing environment:

[sourcecode language="java"]
/*Hello World in Java -windowing version*/
javax.swing.JOptionPane.showMessageDialog(null,"Hello world!");
[/sourcecode]

Swing is a package available in Java to create an application for a windowing environment.
Coming back to what we started with, to print Hello World in Powerbuilder, we could do something like above using a messagebox. Luckily PB offers a function for that - MessageBox. So, in the simplest form, we will do,

[sourcecode language="cpp"]
/*Hello World in Powerbuilder-Simple*/
MessageBox("Hi from Powerbuilder", "Hello World!")
[/sourcecode]

We are almost done. Like almost all of the examples above (except BASIC which was interpreted, you just type that one line and it did print it), we need to wrap this line in a program. In the java example above,
we would do,

[sourcecode language="java"]
// Hello World in Java using Swing GUI
class HelloWorldSwing {
static public void main(String args[]) {
javax.swing.JOptionPane.showMessageDialog(null,"Hello world!");
}
}
[/sourcecode]

Notice the main function. That is the entry point in various languages like C, C++, Java. This is the function that is called when the program is executed.

Powerbuilder is built slightly differently. It does have an entry point function - actually an event. This will be the open even of an Application object. Just like the class with main function becomes entry point object in Java, an Application object is the entry point object in Powerbuilder. So in it's simplest form, we simply add the messagebox function in the application open event and just run the application. When run, this is what you will see,





Well, that's for showing the simplest way. But a typical PB application is built not just with application object, but with one or more windows as well. We will see about that in a later post.