On Friday, the Republican leadership in the House of Representatives continued its wholly political, and ultimately theatrical investigation of the IRS Tea Party-targeting scandal. Chairman Dave Camp’s(R-MI) committee brought current IRS Commissioner John Koskinen before the committee to testify as to the loss of Lois Lerner’s emails, among other misdeeds. You may argue that Paul Ryan(R-WI) was very aggressive in his examination of the witness, but that entire exchange was mere political theater that will evince nothing at the end of the investigation. Had the Republicans in Congress the first inclination to get to the bottom of this scandal, they would begin by taking the following series of steps:
- Bring before the committee the entire IT staff that supports the IRS, particularly its executives.
- Audit the purchase records and replacement schedule of equipment used to support Lois Lerner’s computer usage. Congress should want to know how old her computer had been when the hard disk “died.”
- Require IT staff managers to testify as to the method of email archiving, email storage, email backups, and the entire email system used by the IRS.
- Seek a federal court order requiring the production of all existing equipment that is currently, or has ever been in use by the IRS in storing email,over the period of the last six years, including particularly SAN devices and servers.
- Seek a federal court order requiring the immediate production of all backup media on which IRS files and email may have been copied.
- Form a select committee with broad investigatory powers to pursue the entirety of this affair, particularly with an eye toward fraud, destruction of government records and data, as well as political influences brought to bear on the IRS from any branch of government or outside interest groups.
- Bring in experts to audit access records for servers and storage devices to discover when anybody interacted with the equipment in question. These devices and servers maintain extensive logs of the commands issued from administrators. Knowing who did what will be a key to cracking this case. The government may well have logging servers to which all events are reported.
For those of you who are less than technically inclined, I will be glad to explain to you why this whole “lost hard drive” claim is a dodge, and for those who may have less than a strong understanding of the politics, I’ll be glad to explain to you how I know the Republicans are playing a game for show, but do not want the truth to come out.
As an information systems professional, who works with storage systems, backup systems, networks, servers, and workstations every day, and who works with the applications and databases which is the purpose of all of that lovely, grotesquely expensive equipment, let me tell you a few things you won’t read in the media. You might even take a moment to learn a bit more about your own computer.
First, the email system the IRS uses is almost certainly an IMAP or MAPI variant. This means that on the most basic level, emails are not stored on the client, except as a temporarily cached copy. Deleting it may cause the email to appear deleted for that user, but the mail archiving functionality will maintain a copy for a period as prescribed by policy, usually determined in applicable statutes and regulations. Most corporate and government environments will not even permit users to store mail in local folders(email folders solely on your local computer) unless they are first archived in the email archiving system, which is generally part of the same overall system. Nevertheless, examining the event viewer in Windows will offer some insight into what may or may not have been done on a given workstation or server. Linux and other operating systems have similar logging facilities. (If you have Windows, you can get an idea by going to your Control Panel, then to Administrative Tools, and Event Viewer. You will be surprised what you can learn about your computer’s routine operations.)
In the second place, the number of servers used for an email system to support an organization the size of the IRS must be quite large. It undoubtedly consists of multiple servers, at multiple server farms, in a redundant scheme of some sort intended to prevent the loss of data. You, the taxpayer, has spent billions upon billions since the advent of email to provide these facilities for our federal bureaucracy.
Third, since email storage in such an environment is bound to be monumental in scale, there are undoubtedly many storage blades of some form, probably Storage Area Networks(SANs) to handle the storage needs of the mail system servers. These are also geographically dispersed for reasons of data security, and what you should know about these technologies is that if your Storage Administrators are doing their jobs, there is virtually no credible fashion in which data of this sort could be lost simply because somebody’s office computer’s hard disk died.
To put it in context, consider one of the leading manufacturer’s systems. Called an ISE2, it’s made by X-IO and it can contain two datapacs that contain what are essentially a stack of hard disks that are effectively “self-healing,” and in common usage, contain more than fourteen terabytes of data in each datapac. By design, such a device already creates a Redundant Array of Inexpensive Disks(RAID) by virtue of its design, permitting the administrator to choose either RAID 1(mirroring) or RAID 5(a form of quasi-mirroring). The way these devices are used is to create storage volumes in the datapacs and attach those volumes to servers. They can be swapped in and out, and they can be mirrored as individual entities across other devices. The servers in question see these volumes as hard-drives, and in effect, they function in precisely that manner. I would be stunned to find that the US Federal government is not using such an arrangement, whomever the vendor, and there are many. Chances are high that wherever the server farm is that operates the IRS email system, there are likely to be many SAN units, or other storage containers that have similar functionality.
Putting of all of this into simplest terms, the series of failures that would be required to disappear Lois Lerner’s emails, along with those of six other IRS executives, is an astonishing string of virtual impossibilities and displays of incompetence and malfeasance that should result in the ouster of every IT manager supporting the IRS. It’s not that I don’t believe there are incompetents working in government, or that I don’t believe there are some slothful folk administering systems for the IRS, but that the totality of this loss of data represents a complete failure at virtually every level and every step of the organization. Even in a clunky, bureaucratic, top-heavy organization like the IRS, there are still some competent people who keep it working despite all the obstacles placed in their way. The manner in which their storage systems and server farms are designed tends to preclude the chance that something so seemingly innocuous as the loss of one person’s email(or seven) is even a remote possibility.
Knowing how such systems work, and knowing that the government spends more money on the core computing technology than any entity on the planet, their claim to have lost the email due to a hard drive failure on a client machine is an absolute farce. To claim even that no data was recoverable on that hard drive is pretty hard to believe too, since I’ve seen data recovered from hard disks that have been in computers essentially destroyed by fires. In fact, given the nature of the data I have handled over the course of my computing career, it is common that when a computer reaches the end of its service life, organizations resell the computers but strip the hard drives out of them for mechanical destruction so that no data may be recovered from them. (In many cases, this involves drilling holes through the platters, using a cutting torch, or other methodology designed to destroy the actual storage media in the drive, which is generally very hard metal platters.)
All of that doesn’t matter in the least, however, as the servers and archive servers and storage devices in the systems are apt to have contained one or more(probably many more) copies of the target emails. Then there are backup tapes or other backup devices. No, ladies and gentlemen, if the administration’s hacks like Mr. Koskinen come forward to tell you in smug tones that the data was irretrievably lost, they are lying. It may have been irretrievably destroyed, but that would require a conspiracy because no one computer technician could possibly have access to all the relevant systems in an organization so large.
The technician who was responsible for maintaining and repairing Lois Lerner’s computer is not the same technician who administers the email system. That administrator is not the same person who operates and maintains the bulk data storage containers, nor is that the same person who operates all backups and certainly not the same person who maintains and administers the network on which all of this computing takes place. It’s not plausible in an organization the size of the IRS. In many cases, data is duplicated and moved off-site for disaster recovery purposes. No, if this data is unrecoverable, it is because it was ordered to be placed in that state. Knowing this, and knowing what would be entailed in literally destroying any trace of these emails, I can only conclude that this administration is lying, and is an active participant in a criminal conspiracy and cover-up of crimes that would tend to place Lerner and her superiors in jeopardy of long jail terms, and this president in the direct path of impeachment proceedings.
At the beginning of this article, I explained to you that I believed that the claim about the emails being “lost” is nonsense, and a lie. I hope I’ve managed to illuminate a few of the reasons why you should not believe such claims, but I also contended that you should not believe that the Republicans are very serious about uncovering the truth, despite their harrumphing to the contrary. You see, if the Republicans in Congress were serious about all of this, they would issue subpoenas to the entire IT staff. They would drag them in, one at a time, starting at the top, and working their way down to the lowest technician. They would have questions, specific technical questions, prepared for them by people like me, or actually those rare birds who designed such systems, and they would begin the grilling. Under oath. Somebody would crack. A lie of this sort cannot be hidden if there is a consistent and tireless effort to uncover it.
The problem may be that to uncover Lois Lerner’s email would reveal something no Representative in that committee hearing room wants you to know: Lerner may have been receiving emails from both sides of the aisle on Capitol Hill urging audits and investigations into Tea Party groups. The IRS was used in this instance to quell a peaceful, political uprising by making the formation of a group so painful and problematic as to frustrate into capitulation all but the most insistent and persistent persons. The Republicans tried first to co-opt the Tea Party phenomenon, making it their own, but when they found they were unable to control the myriad of organizations springing to life around the country, their next motive was to destroy it because they posed a serious challenge to the orthodoxy of establishment power in Washington DC. Most Republicans in Washington DC want the Tea Party buried, some of them more fervently even than the Democrats.
If the Republicans in the House cannot muster a select committee to look into this and other matters of extreme government corruption, it is only because they do no want the truth discovered. If they will not bring an endless string of witnesses to testify as to their role in the email “losses” and the system design of the email and data facilities of the IRS, then they don’t want an answer. Paul Ryan and others can put on one Hell of a show in the committee room, but the truth is that saying “I don’t believe you” in an exchange with an IRS commissioner isn’t going to turn over many stones. If you want the truth, you bring in the subject matter experts and responsible parties, and you grill them and continue to remind them of their oaths. At some point, some junior flunky intern who was told to ditch a hard disk in the Potomac is going to squeal, because he doesn’t want to go to prison. Then you bring back the person who gave him that order, and then the person who issued that order from higher on the food chain. Work your way down to get them on the record, until somebody cracks, and then work your way back up, exposing lies until the scheme is revealed in full.
If the Congress won’t do this, they’re not serious about the matter. It suggests strongly that they don’t want the truth revealed any more than the administration. There are plenty of smart people on Capitol Hill, and they have plenty of contacts who understand such systems and could provide technical advice both in the formation of questions and in the manner by which to challenge the credibility of the answers. Those behind this atrocious abuse of government power must be held accountable and jailed for their crimes. Make no mistake about it: Grievous crimes were committed both as a part of the targeting, as well as during this extended cover-up. If the Republicans now fail to uncover those crimes and see this investigatory process through to a just ending, you can be sure that they hadn’t wanted the truth to be discovered, because their fingerprints are all over this too.