Cache, Blocks and Borders

by daniel on September 9, 2008

Not long ago we had to implement data wiping of existing files. Data wiping is performed by writing nonsense information on the file, once or several times, and the deleting the file.

The solution was straight forward:

  1. open the file
  2. write zeros
  3. close the file
  4. delete the file

The development was done on the developer’s machine, using windows XP and so was the development validation.

When moving to QA stage, the operating system was 2003 and the storage was a managed centralized storage.

Very quickly the QA engineer found out that the solution doesn’t work. The file is deleted, but no data is written over it. The developer looked at the code to try to find a problem, but no problem was detected.  After additional test, no apparent reason was discovered, so I was approached with the problem.

In order to try to identify the problem, the programmer was asked to omit the last step in the sequence. To our surprise, omitting the deletion caused the file to be written with zeros as expected.

The developer was blaming the centralized storage because it had a controller who could modify the writing sequence, but looking at the complete flow, different patterns appear.

The actions requested from the application can be one of two:

  1. block actions (write action)
  2. file action (delete action)

Block actions are transferred as is to the storage, with address mapping by the file system management.

File actions are converted to block actions by the file system management, i.e. NTFS management.

From the analysis above it’s quite clear that the only one who can understand the delete action is the file system management.

Additional research found this document, explaining how Windows Server 2003 Cache operates.

The cause of the seen behavior was write optimization performed by the file system management.

At the end the simple solution was opening the file write through, so no optimization was performed on the block write, and the deletion deleted a file filled with zeros.

This type of system problem solving exhibits one of the most important requirements of system engineering: understanding where the responsibility borders are and to be able to relay that information correctly.

No tags for this post.

Comments on this entry are closed.

Previous post:

Next post: