Source Code Control

Background

I always think, we programmers are like writers and artists. We create software that could be powerful and beautiful. There is no lack of abstraction in our work either!

There is one difference, though. When an Author writes a book, she could revise her work so many times at her desk, but only the final outcome becomes public. Not so with our work! A programmer comes up with several revisions of his/her program before it goes to production. Even so, there may be a bug or something was missed that he/she has to revise it again.

A software could also be "backed out" to a prior version. (Even software giant like Microsoft does this once in a while. Remember when people rejected Windows Vista, their solution was to go back to XP, the prior version of Windows! I myself went back to Gnome desktop on my Ubuntu 12.04 desktop, because of some issues with Unity desktop). And of course, someone might point out that something worked in an earlier version, that you will have to dig up the sources for that version, find out what changed and try to re-incorporate that code back into the new version!! I can't imagine an artist having to go through this type of revising of his/her painting! (I agree a book may have editions, but it's always move forward for them.)

So, if you are a software developer you need to worry about keeping track of the history of the program. Not just the history, but all the files that made up a particular version! And to do this we resort to so many ingenious(?) ways. How many times, have you seen files with .bak, .sav, .org etc extensions on a developer's machines? (guilty!). Some may even organize their files in dated directories, so they can go back to a specific date (guilty as well). But when your program is made up of so many files (gone are the days when you wrote all your code in one file!), such rudimentary methods of keeping track of files won't work! Even more so, if you are working (collaborating) with other developers.

Source Code Control or Version Control

Wouldn't it be better if a Software can track these information at the source level, so it's easy to pull up any version of the software sources and rebuild it? Such a software tool exists for over 40+ years and is known variously as Source Code Control Software (SCCS), Version Control System (VCS), Revision Control System (RCS) and even Software Control Management (SCM). Dozens of tools have been developed with these acronyms in their names. Some of the most popular ones over the years are SCCS, RCS, PVCS, Perforce, Visual Source Safe (VSS), Mercury etc. And in the open source arena, we have had CVS, SVN and Git. Please refer to the wiki site for comparison of many more tools available on the market. The difference is not only in the names, but in techniques and technology involved. Newer tools support concurrency better.

The least a VCS can do is to keep track of history of files and thus offers reversibility. With various versions saved in the repository, it is only natural that we require the VCS to provide a good compare (diff) utility or at least the capability of using external diff tools.  Any type of files can be saved in the repository including binary files. Though, the diff utility is typically geared for text files.

More traditional VCS offers a centralized repository. SCCS, PVCS etc belong to this category. More modern ones offer more distributed approach. With developers from around the world working on open source projects, distributed repositories more sense. And the other characteristics that go with the territory is the concurrency. Again, modern VCS have more and more concurrent support built in. For e.g., CVS refers to Concurrent Version System.

Further Reading

Please refer to the Eric Sink's website (his book is available in PDF format on his site) for detailed discussion on the SCCS tools. This blogger recaps the history of Source Code control in a funny way! And the wiki page like always delivers a nice introduction to this topic. Eric Raymond's paper explains various generations of Version Control Systems nicely.

http://www.ericsink.com/vcbe/index.html
http://www.catb.org/esr/writings/version-control/version-control.html#why_vcs
https://code.google.com/p/pysync/wiki/VCSHistory

Oracle: Forward Slash in SQL*Plus - Take 2

Take 2?
I posted about Forward slash in Oracle SQL before. I even went back and updated it a few times based on the traffic. Believe it or not, I still get a lot of traffic to this blog site solely based on this one little topic!! So, I thought I would spend a little more time on this.

Wordpress has a nice feature in the Stats page that shows the Search engine terms that drove the user to my page. This is what some enters in say, Google and clicks Search. Here are some sample terms on Slash:

sqlplus forward slash
sql forward slash
forward slash in sql
oracle forward slash
oracle sql forward slash
slash in sql
sql slash
.....
commit vs forward slash
oracle pl/sql semi-colon or forward slash
oracle slash before commit
oracle sql commit slash
forward slash in ddl file

and so many variations of these. One thing that caught my attention (in bold) was the confusion user(s) had with commit and Forward slash. They really don't have anything in common, but I guess Oracle behavior in different scenarios could lead one to that. So, I thought of explaining these one more time.

Then Vendor said: Let there be a Database and a (client) Tool

Let's start from the beginning. A database system is typically run on a remote server. To work with it, the vendors - like Oracle, Sybase etc- provide a client tool. Oracle has always come with SQL*Plus as a client tool. This is a simple command line tool, yet powerful. People use it for both interactive and batch use. Oracle also came up with "better" tools eventually, like iSQL*Plus, SQL Developer etc. Similarly, Sybase has Interactive SQL. Apart from these tools, there are plenty of tools available from outside vendors. Some prominent ones for Oracle are Toad, SQLTools, PLSQL Developer etc. The modern tools are typically GUI based and provide plenty of options for interactive usage. Irrespective of that, SQL*Plus is still going strong. Several DBAs prefer this tool to flashy new GUI tools, so whichever tool you use, you may want to make sure your SQL will run SQL*Plus.

Why those Punctuation?
In a GUI tool, you can type up a SQL and click on a button (or press a key), the tool will get the SQL executed (send it to Oracle server, get the results etc). If there are multiple SQLs, you just highlight one and click and go. Now, SQL*Plus is a command line tool. It's text based and often used in both interactive and batch modes. You couldn't really click a button or press a key to execute SQLs. Especially, if it's running SQLs from a file. Then you need a marker to show the end of each SQL, right? That, my friend, is the semi-colon - the statement terminator. Sort of like the period at the end of an English statement. Semi-colon is mentioned as the statement terminator in the SQL Standards books also.

Unfortunately, SQL Standards came much later. Database vendors already came up with their way of telling their tool to execute the SQLs. For e.g., client tools for Sybase and Microsoft SQL Server use go command. That makes sense. Type in a SQL. Key in go and it goes! (SQL is sent to server to be executed etc). Oracle (SQL*Plus) had a similar command - RUN!! So, if you are running SQLs in a file, you would have a SQL followed by RUN and then the next SQL followed by RUN and so on. This is good. But then Oracle decided to give us the short cuts. Like Forward Slash ("/") for RUN¹. (Those early days, every byte counted. Why type 3 chars for RUN and why store so many of them in a file? Slash will do it for cheaper price!!).

---

1. Actually RUN = Load & Run and the shortcut for it is "R" while "/" is to just "Run" what's already loaded

What does Forward Slash mean?

So, there you have it. Forward Slash executes a SQL before it. If it was left like that, things probably would have been simpler. Oracle also decided to support the statement terminator, semi-colon. (Sybase and other databases did not require this earlier). Now, you have a semi-colon that could get the SQL executed and a Slash that could do the same thing. They are not the same, but served the same purpose of indicating end of a SQL statement in SQL*Plus!! This started the trouble, especially when people started mixing them. Imagine what happens when you have a semi-colon and a forward slash?

Then came PL/SQL. Now, PL/SQL is a programming language in Oracle which can have several embedded SQLs, often separated by you guessed it - semi-colon - the statement terminators. Now you can have a bunch of PL/SQL block in a file, how do you distinguish each block? Don't you need something like a PL/SQL statement terminator? Well, semi-colon is already taken. So, Oracle came up with the ingenious way of reusing a character that already had a purpose - you guessed it, the Forward Slash!!!!! It was meant to be "RUN" the SQL, right? Now, it also has the purpose to terminate PL/SQL. (It could also mean RUN the PL/SQL). That paved the way to all the troubles we face in mixing these up.

Semi-colons and Forward Slashes

That mixture combined with SQL*Plus's own quirks made it worse. In SQL*Plus, you can type a (SQL like) text and end it with semi-colon, it will pass it on to the database. You type a SQL followed by slash ("/"), it will be sent to the server as well. What happens when you have a semi-colon and slash? It gets executed twice. What happens when you have 2 semicolons? You might get an error on the second semi-colon. This is because of the way SQL*Plus works. As soon as you run a SQL, SQL*Plus stores it away in a buffer. Next time you type "/", it re-executes what's already stored in the buffer. Type "/" again, it runs it again. Until you type up a new SQL. You can smell trouble, right? People run into trouble because they don't understand the nuances or simply because they mistyped.

Remember, "/" is a command. So, it has to be on a line by itself. Semi-colon on the other hand, has to be the last character on a line. If there is anything else after that, even a comment (--) or another semi-colon will cause an error!!

DML, DDL etc
When I mentioned SQL above, I meant any type of SQL. SELECT, INSERT, UPDATE, DELETE, CREATE TABLE etc. First 4 deal with data in tables. SELECT is a query. INSERT, UPDATE, DELETE make up the DML. There are whole bunch of SQL statements like CREATE TABLE that actually define the structure of a database object (here TABLE). These are called Database Definition Language (DDL). What we saw above applies to both DMLs and DDLs. You can use semi-colon to end DML statements as well as DDL statements. You can use Forward Slash with DML and DDL statements.

To confuse you a bit, general convention in Oracle world is to use Forward Slash to end DDL statements. This would make sense if you look at the syntax for creating Stored Procedures, Functions etc. These are PL/SQL objects. Remember, I said PL/SQL needs to be ended in Forward Slash? Now that we decided to (have to) use Forward Slash for DDL for Stored programs, why not use it for all DDLs? So, there you have it. This is how we arrived at Semi-colon (";") to end DML statements and Forward Slash ("/") for DDLs.

How did COMMIT enter this discussion about Forward Slash?

Phew! We got through that fine. Now, where does commit come into the picture and why confusion about it? Typically, Oracle has Auto-commit option turned off by default. You will have to issue an explicit COMMIT to commit a transaction. (This helps us to have bunch of related DMLs inside a transaction).

Here again there are some short cuts and quirks in the tool that made it more confusing. If you didn't add an explicit COMMIT and simply exit SQL*Plus, the tool will issue the COMMIT for you! And the DDLs do not require explicit COMMIT, as a DDL always issues implicit commits (See here) before and after the actual DDL statement. So, then with DDL it's automatically committed and DDLs typically end with Forward Slash ("/"). Now, I can imagine why the user searched for a connection between Commit and Slash. He/She is probably running a script that has "/" at the end. The user probably ran the script and when SQL*Plus exited, it probably committed. Are you seeing the connection now? Really, there is no connection between "/" and COMMIT. They just look to be related because of quirks and twists in the tool and the specs.

What was all the blabbering about?

In summary, Forward slash is like go in SQL Server, but use with caution, as it's not only the RUN command, but could be indirectly construed as a statement terminator as well. Forward Slash ("/") does not commit. SQL*Plus may silently COMMIT your transaction when you exit. But, always add COMMIT and ROLLBACK explicitly.

And Remember, all this is specifically applicable to SQL*Plus. But, since it has become a de facto standard for running SQLs from a file, you make sure to follow these standards to avoid surprises.

And that mumbo-jumbo applies only to Oracle SQL or PL/SQL. Forward slash doesn't have this type of special meaning in other databases!!!!

Quick Tip - Oracle 11g Instant Client

Oracle 11g has 2 versions of clients, Instant Client which has minimum required files for connecting to Oracle and the full blown (thick?) client which has many drivers and tools like SQL*Plus etc. The full blown version seems to be more for the developer and the instant client seems to be suitable for end users.

Recently, we upgraded to Oracle 11g database. When our desktop admin made changes to a user's PC to connect to our 11g database, one of the applications (in VB) stopped working. Incidentally, he had only installed Oracle instant client for end users, as it uses less space.

The VB application in question actually used OleDB to connect to Oracle. Apparently, the instant client we installed for this user did not include OleDB related files. This is why the connection failed. To make it work, we did the following:

1. Installed Instant Client for Oracle 11g (32bit) (option 1 during installation)

2. After the install we copied the OraOLEDB.dll files into the %ORACLE_HOME%\bin directory and registered it. (using regsvr32).

3. Created the directory C:\Oracle11g\Network\Admin and move over the tnsname.ora. (By default tnsnames.ora is left in Oracle11g folder. If you don't move the file, you need to add/change TNS_ADMIN environment variable to point to the right directory.)

Once we did this, the VB application was able to connect to Oracle 11g through OleDB.

Note:

1. This is the minimum required client installation for Oracle. We didn't install the full blown client, as it comes with SQL*Plus etc which the end user doesn't need.

2. All the data access drivers are bundled in an optional package called ODAC. We didn't want to install all the drivers for the user. So, copied only OleDB driver from here.

3. Apparently, SQL*Plus can be added on to instant client as well. See here for more information about Oracle instant Client. Question about connection strings has a hint for connection problems similar to ours.

Reference:
http://www.oracle.com/technetwork/database/features/instant-client/index.html

http://www.oracle.com/technetwork/database/features/oci/ic-faq-094177.html

http://www.oracle.com/technetwork/topics/dotnet/downloads/index.html

http://support.dundas.com/Dashboard4.Installing_the_Oracle_Data_Provider_for_dot_NET.ashx

Tools: Notepad++ - Syntax Highlighting

I've posted about Notepad++ (Npp) tool briefly in a different context. I've been using it for a while and the more I use, the more I discover about the tool. This post is about the syntax highlighting in the tool and how you can customize it.

Npp is simply a text editor. It does the same thing Windows' Notepad does and then some. Hence the name Notepad++. With it, you can open almost any text file. Have you ever tried to open a text file from Unix (or Linux) in Windows' Notepad? All lines will be collapsed into a big jumble? This won't happen in Notepad++. It also allows you to adapt various encoding in file (ASCII/ANSI, UNICODE etc) and thus you won't see box characters (well, most of the time). So, it's a great text editor. But that's not all. What makes it a great tool are it's easy-to-use user interface, it's search capabilities (includes Regex out of the box), plugins etc. If you haven't already tried it, download it from here.

As developers we tend to expect a lot more from any tool. I myself have used several editors, starting DOS's EDIT, Unix Vi etc. (I still remember Brief Editor of the early 90's made by a company called Underware!) Each had unique capabilities and weaknesses. One thing they were all missing was the Syntax highlighting. For a long time, this was the feature of IDEs/Editors that came with the language. But in the era of extensible editors like Eclipse, we've come to expect Syntax highlighting as a minimum for any text editor.

I have always wondered how the programming tools color their syntax. I can understand a dedicated tool like Powerbuilder. They know their language, so they can just hard code it in their programs. How does a simple editor like Notepad++ do that without having knowledge about each language it supports? These tools provide some kind of mechanism (proprietary or standard) to define various aspects of the syntax and how to highlight them. But, they just don't know anything about the language itself. That is the syntax highlighting in such tools is based on pure syntax and not the semantics of the language. This is the key to understanding some anomalies in highlighting in these tools.

Notepad++ (Npp) is one of the editors based around a standard editing component called Scintilla. According to their home page, "Scintilla is a free source code editing component. It comes with complete source code and a license that permits use in any free project or commercial product.". More on this later. Npp comes with syntax definitions for several languages. But, if your language of choice is not in the list, no worries; you can simply add another "User Defined Language" (UDL). With the latest version (6.2.x), it's improved and called UDL2.

It's the UDL that I am going to talk about in this post. If you open Npp and click on Language menu, you will see a list of languages available. You can then click "Define Your language" option and get going with adding the definition for your language. I used this define the syntax highlighting for Powerbuilder code.

User Defined Language (UDL)



Fig 1. Npp UDL dialog

Using this dialog, you can create new syntax definitions, import, export and copy from another language definition. There are several tabs here. We will come to Folder & Default in a minute. Keywords Lists is where you put in the keywords in your language. You can separate the keywords in your language into several groups. For e.g., in my example below, I've 2 lists - one for all the reserved words in Powerbuilder and the second list contains all the types. Separating it thus allows you style your keywords distinctly.



Fig 2. Setting up Keyword Lists

When you define an entity - here Keyword list, you can also attach a styler to it. The styler is where you define the fonts and colors for the text for the particular type of entity. For e.g., I've colored PB reserved words in blue as shown in Fig3.



Fig 3. Styler Dialog

Next Tab in the UDL dialog is the "Comment & Number". This is where you can define the commenting symbols in your language. For comments they have 2 styles. Line Style which is a single comments. In PB (C, C++, Java etc) single line comments are identified // at the beginning of the text. Such comments can be at the beginning or end of the line or by themselves.

The second style is C style languages is the multi-line comments, anything in between /* and */. Here is a screenshot of the comment section. As you can see, there is a lot more you can do with it. In the styler for the comments, you can also specify about nested comments.



Fig 4. Setting up Syntax highlighting for comments

Similarly, you can highlight numbers.

The last tab is the Operators & Delimiters. This is where you list out the operators and any delimiters in the language you are defining. Operators are the symbols like +, -, /, * etc. Comma (,), semi-colon(;), colon(:), pipe (|) are examples of delimiters. Again, use the stylers to format them.

I will now get back the first tab - Folders & Default. Default is simple - This the normal text that is not defined in any of the tabs above - i.e., text that is not a keyword, comment, operator or a number. To define this, just click on the Default Style-> Styler button.

The folders section is a bit more interesting. Have you ever seen code folding? Eclipse has it. I believe, Visual Studio does. Powerbuilder itself does not offer it. It's the feature in some editors that let you selectively hide sections of code, so you can see the "big picture". For e.g., if you are working with a multi-level nested IF statements, you can hide inner ones to see what the outer most if/else does. You typically see code folding available for any control structures (let's call it block definers) in the programming language - like IF, CASE, FOR, DO, {, } etc. These typically have an opening and closing marker text or symbol. The code folding also helps in such cases, to see if any block is missing closures.

In this tab, Npp UDL offers 2 styles - one that doesn't require separators and the other that does. This is a bit confusing and I had to try both to understand. The one that doesn't require separator are symbols like {, }, (, ) etc. Even if you have text touching them, they are identifiable. You put them in the first style. The second style, that needs separators (typically space), will contain all the other block definers. For e.g., in PB we have IF...ELSE...END IF to define a IF block. The opener is the IF, END IF is the closer. ELSE happens to be the middle (Earlier versions of UDL only supported open and close. So, upgrade to the latest version of Npp, if you haven't already done so). Notice END IF is actually 2 words; in such cases surround them in Double quotes.

Folding in comments allows you to put specific comments in your code, that you can use for folding the section. We often comment out whole block of codes to test portions of code. Just add marker texts (open, middle, close) defined earlier to such comments, you will be able to just hide the whole block.



Fig 5. Code Folding in Npp

Of course, you can add styles in the stylers.

Here is a sample PB code in Npp, after I created the UDL "Powerbuilder" and applied to the file opened.



Fig 6. Editing PB code in Npp

Note: When I added code folding the syntax highlighting for the keywords in Code folding (IF, END IF, CHOOSE CASE etc) seem to disappear. It may be a bug in the current version of Npp. I will check and post back here.

[gallery include="4872,4873,4874,4875,4876,4877"]

Top N subcategories

Top N subcategories.

Oracle SQL - UNION ALL

Oracle SQL - UNION ALL

Here is another TechTarget article about Oracle SQL. This time about using UNION ALL to combine results of 2 SQLs to sort numbers.

Windows Automation Tools/Scripts

After the successful run with JMeter earlier to load test our web pages, I have been on a mission to find an open source tool to automate our GUI application. So far, I have not found any good (free) open source tool. But, I stumbled upon several nice tools and scripting languages to automate windows GUI application.

JMeter was very good for the Web. Since each web page can be generated (HTML), the tool could simulate page to page navigation well. If only we can do this for the GUI screens? It's all compiled code and a GUI doesn't look like it can be tampered with (until I found below scripts). So, you essentially need to work on it like a black box, so you can automate a GUI application only by capturing it's external interfaces - mouse clicks, keys types and windows messages to and from the application.

Sikuli

I started with Sikuli. This is a great tool/script. It's Python based and lets you capture screen shots of different parts of the screen and drive your script by looking for these images. They call it Computer Vision. You can build a library of standard images (For e.g., Button with the label OK) and reuse them across the application(s). As long as your application has GUI standards, this will work out great. It's a great concept and it seems to work decently.

[caption id="attachment_4847" align="alignleft" width="1152"] Fig 1. Sikuli IDE. Notice the version # in the title r930 is the latest and stable version.[/caption]

Sikuli IDE lets you capture screenshots from within and use these in functions For e.g., wait function, waits for a screen matching the screenshot to appear. After you create your script in IDE and save it, the underlying script is actually saved as a Python script, with the all the images saved.

[caption id="attachment_4848" align="alignnone" width="588"] Fig 2. Corresponding Python Script[/caption]

Sikuli has a full array of built in (global) functions. See documentation for more.

Note: Above script has UDF - user defined functions. When you record script through AutoIt3 Recorder, it will have one long script. I've divided these into functions.

Sikuli is a great tool. It deserves more research and I reserve it for future. I am more of a scripting guy and I like text based scripts. (It's easier to find and replace globally and you can even write a script to generate another). So, I continued my search for my text based scripting language. Also, I had some issue with the installation. Though it worked initially, the runtime (a batch file running Java jar) kept getting corrupted for some reason. Then I tried their portable version, it seemed to work OK so far. But, if you are having issues with Sikuli installation, please see here, here, here, here. I did not see any recorder for Sikuli, though I saw references to one all over the web.

AutoIt3
Then I landed on AutoIt. It's great tool. I've really grown to like it. It has a BASIC kind of language (actually more like PERL, C etc) and lets you program everything from mouse clicks to taking complete control of a GUI application window. Apparently original AutoIt reached version 2 and stopped. There is a new branch called AutoIt3. This version is bigger and better than previous versions. See here for full history of the product.

AutoIt3 scripts can be edited in integrated Scite Editor. All popular editors like Notepad++ also support Autoscript syntax. AutoIt3 comes with a Recorder as well. I was able to run our application and record it into a script that I could use repeatedly. I am just beginning to explore this tool. I will post more as I explore the tool further.

Fig 3 AutoIt3 Script edited in Notepadd++

When you install AutoIt3, the commands are integrated into Windows Explorer menu. You can edit or Run script from here. Edit opens Scite Editor by default.

[caption id="attachment_4850" align="aligncenter" width="300"] Fig 4. Au3 Commands integrated into Windows Explorer menu[/caption]

Autoit3 Samples

AutoIt site has a very good forum with lots of skilled people supporting it. A section called "Example Scripts" provides lots of good sample scripts. For a real dive into AutoIt3, take a look at a script called Control Viewer. The author has taken the pains to develop it in AutoIt3 itself. It shows you how to create GUI in Autoscript, and using this you can identify controls (widgets) on any windows application currently running. (AutoIt3 comes with AutoInfo which has similar functionality, but CV has it better!). For some reason the author has discontinued it, but you can down the EXE and the scripts from the main page.

[caption id="attachment_4851" align="aligncenter" width="209"] Fig 5. Control Viewer window - GUI built in AutoIt3 script![/caption]

AutoHotKey (AHK)
The third product I looked at, but haven't done much with, is AutoHotKey (AHK). Apparently it combines strengths of 2 different programs, AutoIt (older version) and Hotkey. One immediate observation about AHK scripts was that it didn't look as structured as AutoIt scripts. But, I could be wrong. There are debates about AutoIt vs AHK. See here for a comparison, though somewhat old. AutoIt3 wins hands down for me at the moment.

I will post more as I explore any of these tools for use in test automation. If you are interested, here are some interesting links about the topic:

References

http://www.autoitscript.com/site/

http://www.autoitscript.com/autoit3/docs/
http://www.autoitscript.com/autoit3/docs/intro/dev_history.htm
http://www.autoitscript.com/forum/forum/9-example-scripts/

http://www.sikuli.org/
http://www.javaedge.com/pdf/Sikuli-Gennadi-Zimmerman.pdf
http://tux2323.blogspot.com/2011/06/sikuli-gui-test-automation-with-java.html
http://groups.csail.mit.edu/uid/other-pubs/uist2011-sikuli-guide.pdf
http://code.google.com/p/simplesikuli/
http://stackoverflow.com/questions/2125839/has-anybody-used-sikuli-for-testing-their-gui-based-apps

http://www.autohotkey.com/
http://paperlined.org/apps/autohotkey/autoit_and_autohotkey.html

http://www.scintilla.org/SciTERegEx.html