Review of CSM (Compression Storage Manager)
by
Shawn M. Gordon
President
S.M.Gordon & Associates
Introduction
When I first bought a Wintel PC a number of years ago, disk space was still pretty expensive. I was thrilled to have a 210Mb hard drive. I had spent over $1,000 on a 60Mb hard drive a few years before, and that had been a good deal. In any case I immediately invested in a product called Stacker. This was the first, and I still think best, of the disk compression software products. It would take your hard disk, and make a hidden file where all your files resided in a pseudo drive in a compressed state. You could easily double your disk space by just using a hundred dollar piece of software.
So what does this have to do with anything? Well I remember a while back some folks talking about getting at the internals on the HP to maybe make this same type of product available on the 3000. This never happened, and disk drives got very cheap (I just put in a 1.3Gb drive for $160 in my PC). However, SolutionSoft Systems came out with a product called CSM, that will do very high level compression for you, and then decompress the file automatically when it is accessed. This makes it possible to implement a small portion of HSM (Hierarchical Storage Management), by taking older, infrequently accessed files, compressing them in place to keep them accessible, and then automatically, and quickly, decompressing them when they are accessed.
So without further delay, here is what CSM can do for you.
Features
CSM initial purpose in life is to do high speed, in place compression of files. This leaves the file looking exactly the same, but it’s sector count will drop dramatically, see figure 1 for a before and after LISTF of the file, as well as the statistics for the compression. This file was a copy of XL.PUB.SYS, and XL’s aren’t real good candidates for compression, but they do OK. You will notice that it took only 10 seconds to compress the file.
The second thing that CSM must do, is decompress the file for you in a quick and zero maintenance fashion. It does this through the use of Procedure Exits. Essentially, as I understand it, they trap any call to FOPEN and check to see if the file is compressed, if it is, then it is quickly decompressed and passed off to the original process that tried to open it. Some of your better candidates for compression are archival databases, so let’s look at an example.
Take a look at Figure 2. This database had it’s capacities set at about 90% for the detail sets because it’s a static history base. In this case you would think that the compression wouldn’t be quit as effective, but I was able to realize over 60% compression on the base. This is a fairly good size database as you can see, and I was able to decompress the whole thing within QUERY automatically using an FO SETS command, in under 2 minutes. Since your applications are only going to access a set at a time or so, this seems pretty reasonable to me.
Figure 3 shows an example of compressing a base that is mostly empty. In this case I used the PreviewComp command, with the TOTAL option, so I just got the final line statistics. You can see how we are getting compression in the 99% range. So you can see a dramatic difference with the amount of compression. Databases are notorious for having a lot of ‘white’ space in them, which makes them good candidates for compression.
Figure 3 has an example of the commands that are available from CSM. There is a variety of options other than the compress that we talked about. You will notice that many of the commands have options that allow you to select a fileset based on the age of the last access, and/or the size of the file. While this is handy, it would be nice to have a more robust ability to select files. The fact that you can use an indirect file as input means that you could use another tool to select the files you are interested in, write them out to a file, and then use that as input to the CSM command.
There are a few commands and options for displaying information about a compressed file, and if it is compressed at all, but I really like a couple of options that aren’t really associated with the product directly. First there is the ‘TruncateFile’ command. What this does is remove the unused portion of the last extent of a file. The first time you run this on a system, you can usually pick up several million sectors of free space. The only impact it has is that the next time you go to write to the file, it will allocate a new extent. I use this technology weekly on my production systems to free up disk space.
The other neat options is ‘PackFile’. This will compress a file into a target file and include all sorts of handy file label information. This is handy for moving files between machines, you can take advantage of your computing power to reduce the amount of data you are going to transmit with something like FTP, then on the target machine, you can recreate the file in exactly the same format. What’s interesting is that the PACK command compresses ever so slightly more than the COMPRESS command. The drawback is that you have to have the product on both machines.
The product supports some nice, and what I am starting to feel are standard features, such as a redo stack, and the ability to issue MPE commands, from within the program. CSM doesn’t support anything other than a space as a delimiter between directives, and doesn’t appear to like having mulitple options specified at the same time, such as TOTAL and FORCE on a COMPRESS, or PREVIEW command. I don’t know if this was a bug or how it was expected to work, but none of the examples showed more than one option being used at a time, so that’s probably how it was intended to work.
Figure 1
SYS.PUB% :listf xl2,2
ACCOUNT= SYS GROUP= PUB
FILENAME CODE ------------LOGICAL RECORD----------- ----SPACE----
SIZE TYP EOF LIMIT R/B SECTORS #X MX
XL2 NMXL 128W FB 67542 4096000 1 69632 37 *
SYS.PUB% ,.CSM
CSM.PUB.SSS
Compression Storage Manager ---------------------- THU, DEC 5, 1996, 8:04 AM
This CSM copy is licensed for exclusive use by PAY USA
There are 388 days left for this demo copy
(C) SolutionSoft Systems, Inc., 1996. All Rights Reserved. C.01.19 (11/11/96)
CSM>CM XL2.PUB.SYS
FILE: XL2.PUB.SYS
Time: 8: 4:27:20 started
Time: 8: 4:37:90 100% completed
Before: 69632 SECTORS After: 48944 SECTORS Compression ratio: 30%
TOTAL FILES SCANNED: 1
TOTAL FILES COMPRESSED: 1
TOTAL SECTORS BEFORE: 69632
TOTAL SECTORS AFTER: 48944
TOTAL SECTORS SAVED: 20688
COMPRESSION RATIO: 30%
CSM>:LISTF XL2,2
ACCOUNT= SYS GROUP= PUB
FILENAME CODE ------------LOGICAL RECORD----------- ----SPACE----
SIZE TYP EOF LIMIT R/B SECTORS #X MX
XL2 NMXL 128W FB 67542 4096000 1 48944 24 *
CSM>
Figure 2:
CSM>CM MYBASE@
FILE: MYBASE.DATA.SMGA
Time: 8:25:47:10 started
Time: 8:25:47:20 100% completed
Before: 128 SECTORS After: 96 SECTORS Compression ratio: 25%
FILE: MYBASE01.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
FILE: MYBASE02.DATA.SMGA
Time: 8:25:47:30 started
Time: 8:25:47:30 100% completed
Before: 64 SECTORS After: 32 SECTORS Compression ratio: 50%
FILE: MYBASE03.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
FILE: MYBASE04.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
FILE: MYBASE05.DATA.SMGA
Time: 8:25:47:50 started
Time: 8:25:48:00 100% completed
Before: 1520 SECTORS After: 496 SECTORS Compression ratio: 67%
FILE: MYBASE20.DATA.SMGA
Time: 8:25:52:90 started
Time: 8:25:53:40 100% completed
Before: 5056 SECTORS After: 2336 SECTORS Compression ratio: 54%
FILE: MYBASE21.DATA.SMGA
Time: 8:25:53:50 started
Time: 8:25:53:60 100% completed
Before: 208 SECTORS After: 96 SECTORS Compression ratio: 53%
FILE: MYBASE22.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
FILE: MYBASE23.DATA.SMGA
Time: 8:25:53:60 started
Time: 8:25:59:80 100% completed
Before: 24544 SECTORS After: 7792 SECTORS Compression ratio: 68%
FILE: MYBASE24.DATA.SMGA
Time: 8:25:59:90 started
Time: 8:26: 1:50 100% completed
Before: 15168 SECTORS After: 6480 SECTORS Compression ratio: 57%
FILE: MYBASE57.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
FILE: MYBASE58.DATA.SMGA
WARNING! File with less than 8 Kbyte size is ignored.
TOTAL FILES SCANNED: 69
TOTAL FILES COMPRESSED: 50
TOTAL SECTORS BEFORE: 1085728
TOTAL SECTORS AFTER: 390560
TOTAL SECTORS SAVED: 695168
COMPRESSION RATIO: 64%
CSM>
Figure 3
DBTINFO.PROG.SMGA
Begin run of DBTINFO at 07:51:02
Copyright 1992 S.M.Gordon & Associates
Enter ?? for help at any prompt (almost)
Enter ? for lists on sets and items
Base name: MYBAS2
Password:
Mode:
Dataset: ?
Data Set Name Type Entries Curr Cap Init Cap Max Cap Incr Entry
SET3-A A 104 252,759 0 252,759 0
SET5-A A 0 30,011 0 30,011 0
SET6-A A 16,428 25,013 0 25,013 0
SET21-A A 0 1,200,003 0 1,200,003 0
SET2-D D 0 215,884 215,884 1,002,726 8,029
SET20-D D 0 20,017 20,017 3,014,834 8,029
SET29-D D 0 10,000 10,000 226,000 8,000
SET30-D D 16,103 26,040 0 26,040 0
Dataset:
Base name:
End of DBTINFO at 07:51:17
CSM.PUB.SSS
CSM>pc mybas2@ total
Time: 8:39: 8:90 started
Time: 8:45:41:10 100% completed
TOTAL FILES SCANNED: 72
TOTAL FILES COMPRESSED: 67
TOTAL SECTORS BEFORE: 4960080
TOTAL SECTORS AFTER: 10096
TOTAL SECTORS SAVED: 4949984
COMPRESSION RATIO: 99%
CSM>
Figure 4:
CSM>HELP
(CM) CompressFile file_set | ^file_name [age_in_days [max_size]]
[TOTAL [FORCE]]
(DM) DecompressFile file_set | ^file_name [TOTAL]
(DO) Do [redo_stack_index]
(EX) Exit
(HE) Help [command_name]
(LF) ListFile file_set | ^file_name [age_in_days] [TOTAL]
(LM) ListMode [SECTOR | KBYTE | PAGE]
(LR) ListRedo
(OU) Out [LP | file_name | TERM]
(PK) PackFile source_file target_file
(PC) PreviewComp file_set | ^file_name [age_in_days [max_size]]
[TOTAL [FORCE]]
(RE) Redo [redo_stack_index]
(RC) ReportComp group_set [ACCOUNT] [NOTZERO]
(TF) TruncateFile file_set | ^file_name [age_in_days]
(UK) UnpackFile file_name
: mpe_command_line
Usability (also installation)
Software is basically easy to use. You have extensive online help with plenty of examples, just make sure that you only use spaces as delimiters between command tokens since comma and semi-colon don’t appear to be supported. The error messages are clear and descriptive, which is necessary in a command driven environment.
Installation is very simple; RESTORE with a CREATE option, then execute a command file. That takes care of everything, and you are done.
Reliability
In general the software works very well, and very quickly, however I did find at least one bug in the version I had. There is a parameter called FORCE that can be used with compression or preview compression (see figure 2), this allows you to force the compression of files that CSM won’t normally compress, like files in PUB.SYS, and files to small to make a difference. Well in my version, this just plain didn’t work, it would give you the same warning it would normally give, and not to the procedure. I am sure this will be fixed by the time you read this review however.
Performance
As noted earlier, CSM is very very fast. The speed of the compression and decompression was really amazing, at least in my opinion. Other functions such as TruncateFile were much quicker than other utilities I have used that perform the same function. The hit on the CPU is moderate, you can certainly tell that CSM is running by using a tool such as SOS or GLANCE, but in general the users don’t notice the hit.
Supportability (including Doc)
The support from SSS was top rate. I got a hold of someone every time I called, and they were always friendly and responsive. I had no trouble using the software, other than the one bug I found. The documentation is easy to read, and concise. It is written as a reference manual, and includes a table of contents and index, as well as a chapter on frequently asked questions, which is handy as it does seem to address the standard questions. One of the first chapters is a Quick Start Guide, which is very good at showing you the basic operation and features of the software.
Summary
I can’t fault anything the software is currently doing, other than it could use more sophisticated selection criteria for compressing files. CSM goes way beyond the call of duty in supplying some of it’s features, such as PACK and TRUNCATE. Basically I have just a couple of concerns with the product. As mentioned earlier, you can freely save and distribute the decompression program so you can theoretically always get to your data, but what if you store off some data, send it to someone else, or have to recover from a disaster, and you forgot to include the decompression program? Granted, storing the decompression program should always be part of your procedures, but we’re all human.
So here’s the verdict. If you are keeping a fair amount of rarely accessed history data on your computer, and you can cost justify buying this software instead of just buying a few disk drives, or you have maxed out your ability to add disk drives, then I think CSM is a great deal. It would be even better if it would let you read the data in compressed form, like STACKER, then it would be even easier to justify. CSM is a technical marvel, and well worth looking at if you are in a position to need it.
At-a-Glance box
CSM (Compression Storage Manager) version C.01.19
SolutionSoft Systems Inc.
2350 Mission College Blvd., Suite 715
Santa Clara, CA 95054
408-988-7378
408-988-4777 FAX
email wayne@solution-soft.com
Call, write, fax or Email for a free demo. The software costs between $1,500 and $11,00, discounts for mulitple copies are available, with support being between $200 and $700. Includes 1 indexed 44 page manual.