Ubuntu in action... - NDAS

Last week, my Windows PC stopped working. (I have to look into it later). At the moment, I am working with my taxes and I needed some files from the old machine, desperately. Luckily, I've been backing up. For this, I use IOCell Networks (also Ximeta)¹, NetDisk. NetDisk is a Network Direct Attached Storage (NDAS) device. This seems to be a combination of Network Attached Storage (NAS) and Direct Attached Storage (DAS). As a device, NetDisk is really an external hard disk enclosure that is network enabled and you add hard disk (I added a 1TB drive to it). The difference from other external drive enclosures is that this is network enabled, so once plugged into Ethernet, it can be accessed from any PC on the network. And Direct Access because, you can just plug in the device directly to Ethernet, not through any computer or server.  To add security, they have added a software layer (IOCell/Ximeta NDAS) that you need to install on each PC to access the drive. You will have to register the software with the specific serial # of the NetDisk. Thus, only specific PCs with the right software and serial # register can access the drive. I had installed this software on my Vista machine and thus worked like a charm. (This post relates only to the Linux version of this software. If you need help with Windows version, please refer to their user manual).

Now, at this critical moment, when I needed to access the back up, I couldn't. My main PC is dead. My wife's laptop runs Windows 7 and is not capable of running Ximeta NDAS software (It freezes - even the version 2.72 they suggest to fix the issue). Then my other PC, where I am typing this from, is running Ubuntu 12.04 for which Ximeta does not have a driver or programs for NDisk. At least, not any installation binaries!

But there was hope. Luckily, Ximeta provides (open source) source code at https://github.com/iocellnetworks/ndas4linux/tree/master/ for Linux. Downloaded the source code (If you go to zip tab in the above page, it will download a file named ndas4linux-master.zip. The source directory has several versions. Which version to use depends on your kernel version. (To find your kernel's version, you can use uname -r). For Ubuntu 12.04, I used version 3.2.0. Though I work with large systems at work, I never venture to compile big programs at home. This time, it's different. Desperate situation require desperate measures. I ventured on compiling the driver for Linux myself. I was amazed. Huge set of files got compiled while I watched without major issues. Once the source code is compiled, we need to install and load it. Here is a list of things to do from their documents:

1. Download a source tarball.


2. Unpack


3. CD into the right folder and run make


4. Change to root or sudo make install


5. Start ndas as root


6. As root, or by sudo, register and enable ndas devices with ndasadmin

 Apart from the documents provided, each directory inside the main folder has a README. Read these for instructions on how and where to compile. Here is a little better instructions on how to build it.

Building the software

This is really confusing, as there are many directories and many make files. To do this correctly, you need to be in the right directory. Change to ndas4linux-master/3.2.0/doc (I use 3.2.0; change to the right version) and read

how_to_export.txt first. Here is the summary of how to compile:

cd ndas4linux-master/3.2.0
make linux-rel

This creates a new directory, ndas4linux-master/3.2.0/build_x86_linux. linux-rel is one of the options. This is the final version. But, you can also build dev or debug versions. Take a look at the doc files.

Installing the software

Once you compiled, you need to install the driver and start it. Below commands do that.

cd ndas4linux-master/3.2.0/build_x86_linux/ndas-3.2.0.x86
make

The make command compiles so many files and actually builds ndasadmin and "installs" in /usr/sbin. This is the command we will use to mount NetDisk.

ndasadmin has several options. Please take a look here. To run this command, you need to be root or use sudo. If you just type sudo ndasadmin on command prompt, it will display various options available.

Starting ndas Driver

After building and installing the driver software, you need to start the driver before you can use the device.

sudo ndasadmin start

 

Mounting the Device

You also need to register the device serial #, before you can mount the device volumes. The readme file, how_to_use_ndasadmin.txt inside the version folder (3.2.0 in my case) lists the steps to install and run NDAS software. To register the device, you need to find the serial number (and write key) of the NetDisk device. These are available at the bottom of the box.

sudo ndasadmin register <SERIAL_NUMBER> -n NetDisk1
sudo ndasadmin enable -s 1 -o r

brings up the NetDisk volumes. NetDisk1 is just name I gave to the device. This can be anything. Also, register option have a slight variation with Serial # and Write key. See the readme file mentioned above.

First, I didn't see them come up. Then I found them listed in "Places" option in Gnome desktop menu. To see the device listing, you can use the following command:

cat /proc/ndas/devs

To disable/unmount the device, use the following commands:

sudo ndasadmin unregister  -n NetDisk1
sudo ndasadmin disable -s 1 -o r

Further Help

Like I said earlier, the doc folder inside the version you are working with (ndas4linux-master/3.2.0/doc) actually contains all the instructions. Here are some of the files in that directory.

how_to_build.txt

how_to_export.txt

how_to_use_ndasadmin.txt

Note that the software seems to be constantly evolving, so it may not support all the functionalities, the actual device can support. And if you need more help, here are some links if you want to learn more about Iocell Ximeta on Linux.

1. http://ndas4linux.iocellnetworks.com/


2. https://github.com/iocellnetworks/ndas4linux/wiki/How-to-export-NDAS-source-for-different-architectures


3. http://ndas4linux.iocellnetworks.com/trac/index.cgi/browser/ndas4linux/release


4. http://ndas4linux.iocellnetworks.com/kermit/index.cgi/wiki/Usage


5. http://ndas4linux.iocellnetworks.com/kermit/index.cgi/wiki/HowToBuildDEB

---

^{1 IOCell acquired Ximeta's NDAS; so you will see the names used interchangeably here and the web. See here}

Quick Tip - *nix: Back Quote

While researching on an issue with a Java Stored procedure at work recently, I had to go to oracle udump^† directory to find out if there was any errors. To do this, we do grep like,

grep -i -l "nullpointer" *

to list the names of all log files that may contain NullPointer exceptions^‡. -i = ignore case, -l = list only file names.

This tip is to be able to use that list in another command using back-quote (`). This is the character (also called grave accent) before 1 on PC keyboards, if you haven't used it. Back-quote is used for command substitution in *nix (unix, linux) systems. It's available in

For eg, to list those files with full directory listing:

ls -lt `grep -i -l "nullpointer" *`

Or to open the trace files found by grep, all at once in vi:

vi `grep -i -l "nullpointer" *`

(and then use :n in vi to go through the files)

Below line copies the found files out to another directory,

cp `grep -i -l "nullpointer" *` ~/nullsod

(where nullsod is a directory in my home directory.)

Command Substitution (using Back quote)

This process of executing commands inside back-quote is called command substitution. The command inside the back-quote is executed first and the output is plugged into the commandline that contains it. For e.g.,

echo `date`

returns System Date. Notice a back-quoted command is mostly embedded inside another. If you miss the echo there, it will try to return the date string to the shell which it will try to execute as a command which results in "Command not found" error.

Back-quote is often used in scripts to set the result into a variable. For e.g., we can save the date value in a variable called dt. Then we can use this variable in script.

$> export dt=`date +"%m%d%C%y%H%M%S"`
$> echo "Date is $dt"
$> cp output.log output.log."$dt"

#Where, $> is the shell prompt

In the above list, Date is executed with a specific format (mmddyyyhhmiss). This is then stored in a variable called dt. First we echo dt. Then we use dt to rename a file with date suffix, very handy when you are running a script on a regular basis and want to save the output in timestamped logs.

Incidentally, the statements above, show another kind of substitution, a string or variable substitution using double quotes("). Any expression inside a double quote is often evaluated first and substituted in the command that uses it. Notice the difference: double quote simply evaluates a string/variable and plugs in the value. A back-quote on the other hand, actually executes the string inside as a command and returns the  output to the enclosing command.

You can combine many of these expressions, and that's what makes it all powerful. If you haven't tried back-quote yet, try it. You will be glad you did!

Useful links

1. Command Substitution in Unix


2. Unix Shell Substitutions


3. Advanced Bash Scripting

---

† udump - user dump - contains user trace files in Oracle database; all print statements from stored programs in Oracle (Java, PL/SQL) end up here.
‡ udump directory may have a large # of files depending on activity, in which case, grep may not work with wild card (*). You may have to refine the grep parms.

Quick Tip - *nix: ulimit

Last week one of our DBAs called me. She was trying to install Oracle 11g. She obtained zip files for the installation from another co-worker. This had many zip files, probably one per installation disk. She was trying to unzip the files and got some error.

Like always, I got down to business of checking and googling. When I tried to unzip the first file, it gave an error as to missing zip file headers. It suggested that the file could be part of a multi-part zip file. Good old google suggested how to combine a multi-part zip file. Just cat each file into one big file, then unzip it:

cat file1.zip file1.zip file3.zip > fullfile.zip

Unzip failed on the fullfile.zip. When I checked the file size, it had he same size as the 1st file, the largest file. Strangely, they both had the size of about 1GB. More googling revealed role of ulimit!

Resource limits on *NIX systems (ulimit)

All *nix systems (all flavors of Unix and Linux) have a limits on system resources per user^†. The limits include cpu time per process, number of processes, largest file size, memory size, number of file handles etc (file size is defined in number of blocks.). The root user gets unlimited resources, but all others are assigned a finite settings as defined in /etc/security/limits on AIX. (/etc/security/limits.conf in Ubuntu). Here is a good post, as to why you want to limit resources for users. To check the limits, use ulimit command.

ulimit -a

There are Hard limits (ulimit -Ha) set by the root and Soft limits (ulimit -Sa) a user can set herself.  See this blog for details on using ulimit.

Role of ulimit in our problem

Continuing our story about faile Oracle install above, when we did ulimit -a, one particular entry caught our attention. file size was set to 1GB (approx., remember it's set in blocks?). Bingo, this is why when we try to create the fullfile.zip, it got truncated at 1GB, the limit for the user. Apparently, the DBAs are supposed to have unlimited file size. She got onto calling the Unix admin.

To cut the story short, apparently ulimit played a role in getting the original zip files as well. When she downloaded the files, it truncated the biggest of files to 1GB each. Our first file happened to be one and that's why it looked incomplete. To make matters worse unzip and gzip ended in pointing to a multi-part zip file, which they weren't.

Like I said, this is true for all *nix systems. This is very powerful, in controlling runaway processes sucking system resources away. But there may be a legitimate situation like above, where you need to override the settings. So, next time your file seem corrupted or truncated, remember to check ulimit.

References

1. ulimit in IBM AIX


2. ulimit on HP-UX


3. ulimit on Linux - blog


4. ulimit on Unix - blog

---

†Depending on the *nix you are using, this may be variously mentioned as limits per user, per shell and it's descendants. Some of the settings like cpu time is applied to individual processes user/shell is running.