Posts Tagged ‘Camp’

IRS Scandal Follow-up: The Show Must Go On

Sunday, June 22nd, 2014

…But nobody actually asked Koskinen about the Sonasoft contract. Wonder why?

On Saturday, it was revealed that the IRS has been contracting with Sonasoft for the back-up of emails since 2005, and indeed, looking at Sonasoft’s clients list, listed there is the Internal Revenue Service.  Adding to my list of things about which the Republicans should seek testimony (if they’re serious,) the specific details of the performance requirements of this contract must now be considered.  Undoubtedly, in soliciting bids for back-ups, there must have been a policy for back-ups the bidder must have been prepared to fulfill.  These details would have been dominated by a records retention schedule that would have been designed to comply with statutory minimums. In any event, such a contract would have been carefully vetted for specific performance requirements, the methodology by which performance could be verified, and the chain of responsibility for those on the government side of the contract to make sure performance was fulfilled, or to seek remediation if the requirements were not met.  There would be a schedule of audits of the performance, and there should be no excuse for pretending somebody hadn’t known their specific duties, on either side of the contract. Here’s the point: We very likely have an organized criminal conspiracy, and if the Republicans don’t begin to immediately turn over rocks to find the culprits, the evidence will be destroyed, but that may be precisely what the GOP leadership wants.

People continue to question whether I’ve entered the realm of “tinfoil-hat-wearing” conspiracy kooks, because I doubt the seriousness of the intent of the House Republican leadership in pursuing this scandal.  After all, they ask, why would the Republicans seek to cover the scandal?  Let’s be blunt, shall we?  As long as this scandal has been going on without serious investigation, how much evidence has been destroyed in the interim?  It is true that if there is a cover-up, there will always be some evidence of that, because it’s impossible to completely cover the tracks of what has been done.  Permitting a delay of the investigation would allow the culprits to destroy the evidence so that any crimes perpetrated in the original scandal would be hard to substantiate to the satisfaction of a jury, or an impeachment proceeding, even if the evidence of a cover-up would be harder to conceal.  In the end, however, let us imagine that there had been a few Republicans who had wanted to hammer the TEA Party, like John Boehner, or Mitch McConnell.  They’ve said as much in open contempt for the TEA Party.  By permitting the administration and its lackeys to destroy evidence, the evidence of their own complicity would be hidden too, and all that would remain are the allegations and evidence of a cover-up of something, in which the Republican leadership would not be implicated.  After all, they’ve been conducting an investigation, right?

If this sounds too conspiratorial to you, consider that these are the same people who invented voting for a thing before voting against it.  John Cornyn had no problem voting for cloture on the Senate Amnesty bill last year before coming home to Texas to tell voters he had voted against the final bill, which he had.  He repeated the procedure at the time of the government shutdown last October, again voting to bring the bill for a vote, so that he could vote against it thereby claiming “conservative credentials” all the while have enabled the bill to see the light of day in the first place.  They bank on we voters remaining largely ignorant of their scandalous manipulations, so that a less-than-vigorous investigation wouldn’t provide much of a surprise. By the way, and by way of evidence of the establishment’s thesis in operation, John Cornyn won his primary by pretending to be a conservative while relying on the longterm detachment and ignorance of voters.  Still, roughly forty percent of the Republican electorate in Texas was able to see through his nonsense, but not enough to replace him as our Senator.

My point to you is this: It may be too late to salvage the data, because this has been left withering on the vine for much too long.  The list of particulars I provided yesterday should have been exercised more than two years ago, and it should have been done with vigor.  If there is no active complicity by Republican leadership, there is at least gross incompetence verging on the criminally negligent.  Are we to believe that none of the people in leadership had any idea, and that none of their staff had any idea how to approach such a scandal?  Are we to believe they had no access to any person with sufficient technical understanding who would have apprised them of the sort of things that would need to have been done to “disappear” such data?  Are we to believe that those who were conducting the preliminary investigations on behalf of House committees could not imagine to immediately contact people specializing in data recovery?  Why has it taken until yesterday to discover that the IRS had contracted with Sonasoft?  What were these investigators investigating?  Didn’t they look at the IT expenditures and contracts of the IRS for clues?  You see, once you consider all of this, it’s easier to understand how an observer could reasonably conclude that the Republicans didn’t want to investigate, and having been forced into it by public pressure, have done a half-hearted job of it.

How can we be nearly three years into this investigation, and we’re only now finding there had been a back-up company contracted?  I will not be surprised to learn that the IRS contract with Sonasoft required them to hold emails for a period of only three years, so that by now, Lois Lerner’s emails have fallen off the archive due to age.  A serious investigation would have immediately discovered the existence of a contract with Sonasoft, and those records could have been pulled three years ago.  What will we get as a result?  At best, some underlings who were a part of the cover-up will be burned, but the chain of command to the top will be obfuscated, and then we will get some dog-and-pony IRS Reform bill that will require the agency from this date forward to maintain all emails for ten years, or some such thing.  Then it will all go away, and the original participants in the scandal of targeting TEA Party groups and their members will be forgotten, and life will go on in Washington DC, with we being the only victims, now poorer and less free, and deprived of justice.

The questions I’ve posed over the last thirty-six hours are the sort I would expect of a serious investigation.  To date, we’ve had a lot of finger-waggling by Republicans asking questions of witnesses, but we’ve gotten no meat from these bones.  Certainly, it does not help that we have a Department of Justice that is led by a crook and crony, and it does not help that the media covers everything up on behalf of this administration, but if the Republicans had been serious about getting to the bottom of this scandal, they would have taken significantly more exhaustive steps by now, but to date, all they’ve done is generate ominous soundbites that tend to feed the red-meat aspects of politics, yet have resulted in no arrests, no indictments, and no justice.  In three years?  This scandal is well on its way to becoming a cold case, and that’s just how Washington DC likes it.

Update: The Daily Caller reports that the IRS cancelled its Sonasoft contract only weeks after Lois Lerner’s hard-drive “crash.”

Advertisements

John Boehner’s Dog and Pony Show

Saturday, June 21st, 2014

On Friday, the Republican leadership in the House of Representatives continued its wholly political, and ultimately theatrical investigation of the IRS Tea Party-targeting scandal.  Chairman Dave Camp’s(R-MI) committee brought current IRS Commissioner John Koskinen before the committee to testify as to the loss of Lois Lerner’s emails, among other misdeeds.  You may argue that Paul Ryan(R-WI) was very aggressive in his examination of the witness, but that entire exchange was mere political theater that will evince nothing at the end of the investigation.  Had the Republicans in Congress the first inclination to get to the bottom of this scandal, they would begin by taking the following series of steps:

  1.  Bring before the committee the entire IT staff that supports the IRS, particularly its executives.
  2. Audit the purchase records and replacement schedule of equipment used to support Lois Lerner’s computer usage. Congress should want to know how old her computer had been when the hard disk “died.”
  3. Require IT staff managers to testify as to the method of email archiving, email storage, email backups, and the entire email system used by the IRS.
  4. Seek a federal court order requiring the production of all existing equipment that is currently, or has ever been in use by the IRS in storing email,over the period of the last six years, including particularly SAN devices and servers.
  5. Seek a federal court order requiring the immediate production of all backup media on which IRS files and email may have been copied.
  6. Form a select committee with broad investigatory powers to pursue the entirety of this affair, particularly with an eye toward fraud, destruction of government records and data, as well as political influences brought to bear on the IRS from any branch of government or outside interest groups.
  7. Bring in experts to audit access records for servers and storage devices to discover when anybody interacted with the equipment in question. These devices and servers maintain extensive logs of the commands issued from administrators. Knowing who did what will be a key to cracking this case. The government may well have logging servers to which all events are reported.

For those of you who are less than technically inclined, I will be glad to explain to you why this whole “lost hard drive” claim is a dodge, and for those who may have less than a strong understanding of the politics, I’ll be glad to explain to you how I know the Republicans are playing a game for show, but do not want the truth to come out.

As an information systems professional, who works with storage systems, backup systems, networks, servers, and workstations every day, and who works with the applications and databases which is the purpose of all of that lovely, grotesquely expensive equipment, let me tell you a few things you won’t read in the media.  You might even take a moment to learn a bit more about your own computer.

First, the email system the IRS uses is almost certainly an IMAP or MAPI variant.  This means that on the most basic level, emails are not stored on the client, except as a temporarily cached copy.  Deleting it may cause the email to appear deleted for that user, but the mail archiving functionality will maintain a copy for a period as prescribed by policy, usually determined in applicable statutes and regulations. Most corporate and government environments will not even permit users to store mail in local folders(email folders solely on your local computer) unless they are first archived in the email archiving system, which is generally part of the same overall system. Nevertheless, examining the event viewer in Windows will offer some insight into what may or may not have been done on a given workstation or server. Linux and other operating systems have similar logging facilities.  (If you have Windows, you can get an idea by going to your Control Panel, then to Administrative Tools, and Event Viewer. You will be surprised what you can learn about your computer’s routine operations.)

In the second place, the number of servers used for an email system to support an organization the size of the IRS must be quite large.  It undoubtedly consists of multiple servers, at multiple server farms, in a redundant scheme of some sort intended to prevent the loss of data.  You, the taxpayer, has spent billions upon billions since the advent of email to provide these facilities for our federal bureaucracy.

Third, since email storage in such an environment is bound to be monumental in scale, there are undoubtedly many storage blades of some form, probably Storage Area Networks(SANs) to handle the storage needs of the mail system servers.  These are also geographically dispersed for reasons of data security, and what you should know about these technologies is that if your Storage Administrators are doing their jobs, there is virtually no credible fashion in which data of this sort could be lost simply because somebody’s office computer’s hard disk died.

To put it in context, consider one of the leading manufacturer’s systems.  Called an ISE2, it’s made by X-IO and it can contain two datapacs that contain what are essentially a stack of hard disks that are effectively “self-healing,” and in common usage, contain more than fourteen terabytes of data in each datapac.  By design, such a device already creates a Redundant Array of Inexpensive Disks(RAID) by virtue of its design, permitting the administrator to choose either RAID 1(mirroring) or RAID 5(a form of quasi-mirroring).  The way these devices are used is to create storage volumes in the datapacs and attach those volumes to servers.  They can be swapped in and out, and they can be mirrored as individual entities across other devices.  The servers in question see these volumes as hard-drives, and in effect, they function in precisely that manner.    I would be stunned to find that the US Federal government is not using such an arrangement, whomever the vendor, and there are many.  Chances are high that wherever the server farm is that operates the IRS email system, there are likely to be many SAN units, or other storage containers that have similar functionality.

Putting of all of this into simplest terms, the series of failures that would be required to disappear Lois Lerner’s emails, along with those of six other IRS executives, is an astonishing string of virtual impossibilities and displays of incompetence and malfeasance that should result in the ouster of every IT manager supporting the IRS.  It’s not that I don’t believe there are incompetents working in government, or that I don’t believe there are some slothful folk administering systems for the IRS, but that the totality of this loss of data represents a complete failure at virtually every level and every step of the organization.  Even in a clunky, bureaucratic, top-heavy organization like the IRS, there are still some competent people who keep it working despite all the obstacles placed in their way. The manner in which their storage systems and server farms are designed tends to preclude the chance that something so seemingly innocuous as the loss of one person’s email(or seven) is even a remote possibility.

Knowing how such systems work, and knowing that the government spends more money on the core computing technology than any entity on the planet, their claim to have lost the email due to a hard drive failure on a client machine is an absolute farce.  To claim even that no data was recoverable on that hard drive is pretty hard to believe too, since I’ve seen data recovered from hard disks that have been in computers essentially destroyed by fires.  In fact, given the nature of the data I have handled over the course of my computing career, it is common that when a computer reaches the end of its service life, organizations resell the computers but strip the hard drives out of them for mechanical destruction so that no data may be recovered from them. (In many cases, this involves drilling holes through the platters, using a cutting torch, or other methodology designed to destroy the actual storage media in the drive, which is generally very hard metal platters.)

All of that doesn’t matter in the least, however, as the servers and archive servers and storage devices in the systems are apt to have contained one or more(probably many more) copies of the target emails. Then there are backup tapes or other backup devices. No, ladies and gentlemen, if the administration’s hacks like Mr. Koskinen come forward to tell you in smug tones that the data was irretrievably lost, they are lying.  It may have been irretrievably destroyed, but that would require a conspiracy because no one computer technician could possibly have access to all the relevant systems in an organization so large.

The technician who was responsible for maintaining and repairing Lois Lerner’s computer is not the same technician who administers the email system.  That administrator is not the same person who operates and maintains the bulk data storage containers, nor is that the same person who operates all backups and certainly not the same person who maintains and administers the network on which all of this computing takes place.  It’s not plausible in an organization the size of the IRS.  In many cases, data is duplicated and moved off-site for disaster recovery purposes.  No, if this data is unrecoverable, it is because it was ordered to be placed in that state.  Knowing this, and knowing what would be entailed in literally destroying any trace of these emails, I can only conclude that this administration is lying, and is an active participant in a criminal conspiracy and cover-up of crimes that would tend to place Lerner and her superiors in jeopardy of long jail terms, and this president in the direct path of impeachment proceedings.

At the beginning of this article, I explained to you that I believed that the claim about the emails being “lost” is nonsense, and a lie.  I hope I’ve managed to illuminate a few of the reasons why you should not believe such claims, but I also contended that you should not believe that the Republicans are very serious about uncovering the truth, despite their harrumphing to the contrary.   You see, if the Republicans in Congress were serious about all of this, they would issue subpoenas to the entire IT staff.  They would drag them in, one at a time, starting at the top, and working their way down to the lowest technician.  They would have questions, specific technical questions, prepared for them by people like me, or actually those rare birds who designed such systems, and they would begin the grilling.  Under oath.  Somebody would crack.  A lie of this sort cannot be hidden if there is a consistent and tireless effort to uncover it.

The problem may be that to uncover Lois Lerner’s email would reveal something no Representative in that committee hearing room wants you to know:  Lerner may have been receiving emails from both sides of the aisle on Capitol Hill urging audits and investigations into Tea Party groups.  The IRS was used in this instance to quell a peaceful, political uprising by making the formation of a group so painful and problematic as to frustrate into capitulation all but the most insistent and persistent persons.  The Republicans tried first to co-opt the Tea Party phenomenon, making it their own, but when they found they were unable to control the myriad of organizations springing to life around the country, their next motive was to destroy it because they posed a serious challenge to the orthodoxy of establishment power in Washington DC.  Most Republicans in Washington DC want the Tea Party buried, some of them more fervently even than the Democrats.

If the Republicans in the House cannot muster a select committee to look into this and other matters of extreme government corruption, it is only because they do no want the truth discovered.  If they will not bring an endless string of witnesses to testify as to their role in the email “losses” and the system design of the email and data facilities of the IRS, then they don’t want an answer.  Paul Ryan and others can put on one Hell of a show in the committee room, but the truth is that saying “I don’t believe you” in an exchange with an IRS commissioner isn’t going to turn over many stones.  If you want the truth, you bring in the subject matter experts and responsible parties, and you grill them and continue to remind them of their oaths.  At some point, some junior flunky intern who was told to ditch a hard disk in the Potomac is going to squeal, because he doesn’t want to go to prison.  Then you bring back the person who gave him that order, and then the person who issued that order from higher on the food chain.   Work your way down to get them on the record, until somebody cracks, and then work your way back up, exposing lies until the scheme is revealed in full.

If the Congress won’t do this, they’re not serious about the matter.  It suggests strongly that they don’t want the truth revealed any more than the administration.  There are plenty of smart people on Capitol Hill, and they have plenty of contacts who understand such systems and could provide technical advice both in the formation of questions and in the manner by which to challenge the credibility of the answers.  Those behind this atrocious abuse of government power must be held accountable and jailed for their crimes.  Make no mistake about it: Grievous crimes were committed both as a part of the targeting, as well as during this extended cover-up.  If the Republicans now fail to uncover those crimes and see this investigatory process through to a just ending, you can be sure that they hadn’t wanted the truth to be discovered, because their fingerprints are all over this too.