Why I don't buy Uretsky's story, and why that doesn't get the DNC off the hook

[markpkessinger]

I have worked in IT for decades, in areas ranging from database design and management to IT security. When I initially heard Josh Uretsky's story that the queries against the Clinton campaign's proprietary data were performed as part of an effort to investigate the extent of exposure of the proprietary data of the Sanders, it sounded like a perfectly reasonable and even likely explanation. Then I read the specifics of the queries that were run. From CNN's reporting:

The Sanders team, which consisted of four people, ran multiple searches in Iowa, New Hampshire, Nevada, South Carolina and about 10 March states, including Florida and Colorado. In Iowa and New Hampshire, the Clinton campaign has ranked voters on a scale of 1-100 for turnout, enthusiasm and support, the senior Democrats said. The Sanders campaign ran two searches: "Show me all the Clinton people rated higher than 60" and "Show me all the people rated less than 30." This would be a key way of knowing who Sanders should target in the final weeks before voting: Ignore those above 60, while focus on those below 30, because they are looking for a Clinton alternative and might be open to Sanders.

So, this index of turnout/enthusiasm/support, is a database field created by, and the property of, the Clinton campaign. If Uretsky's intent had been merely investigative, he could have included that field in a select query, with no particular selection criteria specified, in order to test whether proprietary fields were exposed. There would have been no need to place any particular selection criteria on that field in the query, much less to filter it in a way that would yield information that could be specifically useful to the Sanders campaign. As for not attempting to cover his tracks, I'm sure Uretsky was aware that it is EXTREMELY difficult to get around a database's audit logs (that is, it is extremely difficult if the designers of the database had even minimal competency), and thus knew better than to even try (as the attempt itself would have raised red flags). Instead, he figured that if the queries came to light, he could pass them off as having been investigative in nature. Uretsky knew the security vulnerability was there. I think he figured that since they had previously reported that vulnerability, and nothing had been done about it, he could get away with exploiting that vulnerability to the benefit of the Sanders campaign, and that if any question should arise, he could claim his intent was investigative, citing the fact that he didn't try to cover his tracks to support that claim.

In the end, it was a monumentally stupid move by a campaign staffer, and he deserved to be fired because of it. When it came to light, the Sanders campaign took immediate, appropriate and effective remedial action. And THAT fact -- i.e., that the Sanders campaign had already taken timely, appropriate and effective remedial action, is what made the DNC's attempt to 'punish' the Sanders campaign so outrageous. The DNC's and Wasserman Schultz's disingenuousness is revealed for exactly what it is by the fact that the DNC was notified of a major security flaw in October, and two months later, no corrective action had been taken. I work in legal IT for a major international law firm. In my world, if a flaw like that had come to light, the vendor relationship would be immediately terminated, because it demonstrates the vendor's rank incompetence in database design. So why had the DNC not compelled NGP VAN to fix the flaw? That's anybody's guess. Why did the DNC not terminate the vendor relationship with NGP Van? Gee, do you think it could possibly have something to do with the fact that Stu Trevelyan, the CEO of NGP VAN, was a '92 staffer in the Clinton-Gore campaign, and a White House staffer during the Clinton presidency?

Wasserman Shultz is correct that an "open door" does not provide cover for someone who exploits it in order to access something they would not otherwise have access to. But there's another part of that analogy that points a finger back at the management of the DNC under Wasserman Schultz. Think of a retail store whose manager one night forgot to lock the doors upon closing, and the store, as it happened, was robbed that night. The owner of the store will certainly want to press charges against the thieves; but that owner will also most certainly fire the person who left the door open in the first place!

Presented with evidence of wrongdoing by a few of its staffers, the Sanders campaign took immediate and effective remedial action. Notified of a major security vulnerability in the DNC's database, Debbie Wasserman Schultz sat with her thumbs up her ass for two months, and then had the gall to self-righteously expound about an "open door." What's her excujse?

[notadmblnd]

1. If you've worked in IT for decades

then you know, that no one performs maintenance on a system or app while users are logged on.

[uponit7771]

2. There could be maintenance done that isn't app hardware critical and takes only...

... a small amount of resources to complete allowing the system to continue on operating within service level agreement boundries

[notadmblnd]

6. I think you are mistaken

The system might have not been taken offline to patch an app, but the app certainly would. Attempting a fix while users were logged in would risk corrupting the entire database especially if a user was adding data or making changes as the fix was being installed without being tested before being brought back online.

[uponit7771]

19. I thought this was maintenance and not a fix... maintenance could be just backing up files...

... a software fix is different operationally than maintenance.

[markpkessinger]

22. You're playing with semantics here . . ..

. . , In terms of a database, the distinction you are making is meaningless. The critical distinction with a database is whether the maintenance/fix/patch or whatever else you may call it is applied at the server or at client level. This was an issue of database permissions, which occur at the server level. The database would need to have been taken offline in order to safely address those issues without corrupting the underlying database.

[uponit7771]

29. The post I replied to mentioned maintenance not me. A fix and maintenance can have two different...

... action paths.

[markpkessinger]

33. If one insists upon the distinction . . .

. . . taking down a database in order to correct permissions settings is technically speaking a matter of maintenance, because incorrectly set permissions are not a software bug or malfunction. The server must be taken offline in order to safely do this.

[uponit7771]

36. May be true, but with Oracle and some other DBs the dbs the DBA can set permissions on users or...

... a user groups without a restart.

My understanding is this system is pretty antiquated though

[notadmblnd]

53. Right, but the sysadmin or dba has to long on and turn those permissions off and on

permissions do not just turn themselves on and off. If permissions do turn themselves on and off- it's a bigger problem.

[PatrickforO]

37. By the way, I like your quote from Benjamin Harrison.

[notadmblnd]

51. any fix is maintenance. One is maintaining also know as upkeep.

Upgrading software would be known as upkeep aka maintenance. Replacing a power supply or a disk drive would be considered maintenance. Rebooting would be known as maintenance. Backups are a form of maintenance too.

[markpkessinger]

3. Yes . . .

. . . and I also know that when critical maintenance is required, a time to perform that maintenance is scheduled, and notice goes out to users that at such and such time, the database will be taken offline for maintenance. And I know that, at some designated interval shortly prior to taking down the database, a message can be broadcast to any users who are logged on that the database will be going offline momentarily, and instructing them to please log off forthwith. AND I know that, should they not log off of their own accord, a DB admin can force a logoff. And finally, I know that in no corporate IT department in this country would the excuse of "users were logged on" be accepted as an excuse for having failed to perform critical maintenance for two months. So what's your point?

[notadmblnd]

7. If it was a critical fix, the corruption of the entire database would have been at risk

resulting in more downtime while a complete restore was being performed.

[markpkessinger]

14. Sorry, but table- or field-level database permissions are not that complicated . . .

And frankly, if proprietary information is being stored in the same table as non-proprietary information, that itself is a serious design flaw -- I mean, amateur-level design flaw.

[KittyWampus]

41. I am going to tuck this tidbit here- how do companies doing IT work for the Govt get hired?

Does the Government hire the best/most up-to-date crew or do they hire the company that has figured out how to jump through all the legal hoops and fill out the huge paper trail necessary to qualify for the job.

Remember that whole thing when Obamacare's website first came online and it sucked.

That's when a few reports came out how it isn't the best company for the job that gets hired. It's the one that know how to go through all the paperwork, certifications etc.

And we're probably both cynical enough to figure some cronyism is involved.

[notadmblnd]

42. Was the security flaw with the field level security? Just a matter of clicking on a few options

Last edited Sun Dec 20, 2015, 07:49 PM - Edit history (2)

Since you say this is a field level security problem- it's just a matter of a sysadmin clicking on a few options- something like that would never ever involve installing a patch at that level, would it? Because in order for access permissions for a user or group to be turned off or on- a system admin would have perform that function each time. If user permissions were being turned off and on with out being managed by the sysadmin, Then it would more than likely be a system flaw and require a Microsoft patch and this more than likely require being installed on the operating system in safe mode, with users logged off along with a reboot after. Correct me if I'm wrong.


On edit: I didn't mean to change my post, I was just trying to clean it up a bit. Please see original if interested

[markpkessinger]

44. It could be addressed in a number of ways

I'm not a big fan of field-level security, and in most cases, in a well-designed relational database, it shouldn't be necessary. I don't know the specifics of how this particular database is set up, but if I were designing it, each candidate's proprietary information would be stored in a separate table (linked by key field to the main voter information table), with only the users of the respective candidate having access to the table. Storing shared and proprietary information in the same table is a reciope for disaster.

[notadmblnd]

50. I'm not a database expert either, but I do know the standards

that are usually followed when performing maintenance.

[PatrickforO]

30. Yeah, that's sure true. I think of my own employer and how it's IT department does stuff.

Like you say, they announce that 'between x and y time on z day, we're going to take the system offline for maintenance. Please plan your work around being logged off at this time.'

And then when the time comes, if you're working there's a warning, and if you're still on, you get logged out.

Does seem like DWS's excuse that 'people were logged on' is pretty weak.

[ejbr]

4. Color me confused

so the flaws should remain because nothing can be done as the campaigns are using the data? Or did the OP indicate that this was the remedy; patching while users logged in? I am not an IT professional, so please keep it simple.

[notadmblnd]

8. No, the application is taken offline

the fix performed and tested for success then the system or app being returned to users.

[stevenleser]

25. The first part of what you wrote will be an interesting part of the puzzle.

I am interested in what constraints were put on the vendor to be able to fix the issue.

On the one hand, NGP VAN is the bargain basement version of the voter data database. Think Free Republic as a forum software as compared to this new version of DU which is out there and available which in the voter database world is offered by PDI (See this article on PDI http://www.washingtonmonthly.com/political-animal-a/2015_12/an_explanation_of_what_bernie059035.php and their website here: http://politicaldata.com/) So... one possibility is that NGP VAN are borderline incompetent and just couldnt figure out how to fix whatever the issue is.

Another possibility is that the DNC told them that they could not take the database offline until after the primaries were decided. Again, I don't know if they did this but I have seen folks not allow vendors to fix much more critical things for reasons that to me did not seem adequate.

And those are only two of a myriad of reasons why this was not fixed.

[PatrickforO]

34. Well, I've got to say that this DWS/NGP VAN rat really stinks.

Because
a) Since it's been a poorly kept secret that the DNC is in the bag for Hillary,
b) the fact the DNC didn't order NGP VAN to take it offline to do maintenance,
c) suggests VERY strongly to me that other searches were quite likely made in Bernie's files by the Clinton campaign.
d) This seems especially true since the Sanders campaign reported the open access TWO months ago.
e) So, Debbie did the 'wink, wink' thing when Clinton's people accessed Bernie's stuff, but when Bernie's staff did it, she
f) went directly to press in a clumsy attempt to ratfuck Bernie's campaign.

Bernie has apologized and fired his people that did it. But it sure would be interesting if searches done by Clinton people could be logged like that and made public. Because as Bill Maher says, "I can't prove it, but I just KNOW" that this 'open' access worked both ways. We're just not gonna hear about Clinton's malfeasance, if any, because the establishment doesn't want to do ANYTHING to endanger her winning over Bernie.

[stevenleser]

35. All database searches were logged so if the Clinton folks did it we would likely already know.

a) is your opinion.
b) we dont know the details of how this played out. That is one possibility.
c) no, it doesnt. All searches and access was logged. That's how Bernies folks were caught.
d) No.
e) No again.
f) And if the situation were reversed and Clintons folks accessed Bernies data and she tried to keep it hush hush, you would be screaming about a conspiracy. Interesting how that works.

[markpkessinger]

39. Even if they felt it was impossible to have the database taken offline for any amount of time . . .

. . . NGP VAN, as the DB developer, surely has a model or test copy to which any modifications could be made and tested, then, when it was time to go live, delete from the test model the dummy data, do a back-end import of records from the previous, still live database into the corresponding tables of the test copy, and then take the old database offline just long enough to replace it with the modified test copy (that now has the current data), and then start it up. It is not a complicated process, and is oen anybody with any experience working with databases knows well.

[stevenleser]

60. You would think so, but NGP VAN seems to be the bargain basement of voting DBs

I'm guessing that many of their IT processes are well below what we would consider to be up to snuff and that explains both why they are so cheap and why their DB app belongs back in the mid 1990s along with Free Republics discussion forum software.

My guess is that this focus on them will unearth a host of, shall we say, the opposite of best practices.

[TexasMommaWithAHat]

5. But that was the type of material the firewall was designed to protect

Ordinary information about people is not proprietary information and that information is available to both candidates, so checking "that" information would not reveal that the firewall was down.

And, no, there is absolutely no excuse for that firewall to be down when they were online. Absolutely none. That kind of carelessness can void contracts.

[stevenleser]

10. The terminology is wrong as another DUer noted yesterday. It's not a "firewall"

It's simply the access control system of the database or app.

And yes, it should have prevented this, but for reasons we don't know yet, could be a bug in the database, the app under which it runs, the OS, we don't know yet, whatever the reason the functionality that kept everyone to their own stuff periodically failed.

[rjsquirrel]

23. As a dB guy myself

I'm being driven slowly crazy by the lack of basic understanding of what a database is or how it works in the media and on DU. The misuse of "firewall" because it makes one sound smart about computers is one of them.

This vendor has proprietary control over a mission critical resource, under contract, and being used by hundreds of campaigns below the presidential level. It's a fantasy that they would have been fired over anything yet come to light. Unless you work in financial network security (and even then) fault tolerance is generally a lot higher than you suggest and complex systems always have bugs and issue queues. You can't just hire another firm and give them the data without a very significant disruption of current operations. In my domain of technology business incompetent individual contractors m, even vendor reps, rarely are simply tossed overboard. A change of vendors on a major long term project comes with huge risks and costs and challenges.

[stevenleser]

32. It also makes the issue sound more trivial and perhaps nefarious.

A standalone firewall device or software is IMHO a much more straightforward proposition to deal with than a large multi-user database which can have a somewhat complex permissions scheme.

A firewall would also generally not need to be taken down to change which services it blocked whereas a database would almost certainly need to be brought down to implement changes to its schema.

You also make a great point. Fire the vendor over a bug? No software vendor would be able to maintain a contract. We also dont know that the fact it wasnt fixed was their fault. Its completely within the realm of possibility that the DNC told them to wait until after primary season to fix it so that the DB would not have to be brought down. We just dont know enough information yet.

[markpkessinger]

38. You make a fair point about the difficult of simply firing a database vendor . . .

. . but I disagree that an issue of who has access to what is a mere "fault." When there is an intent to store both shared, as well as proprietary, information, the failure to protect the proprietary information is more than just a "fault" -- it's a critical design flaw that undermines one of the principle objectives of the database.

[rjsquirrel]

58. But that's basic architecture

Surely this is based on commercial dB ware?

[markpkessinger]

59. Depends on what you mean by "commercial dB ware"

Neither Oracle, nor Microsoft's SQL Server, nor ASE (Adaptive Server Enterprise), arrive out of the box as functioning databases. They are all database <i>management</i> systems. A database developer would have likely used one of these three platforms, all three of which employ SQL (Structured Query Language) in order to create tables and queries on which to build a database for the DNC. If, say, they used something like Microsoft Access (or, God forbid, Filemaker) as both a platform and DBMS, that would have been a shockingly inappropriate decision, as neither Filemaker nor Access are appropriate for a database of the scale of the DNCs voter database, nor is it appropriate for a db shared across a network with multiple parallel user groups, each of which has its own proprietary data along with shared data. With any SQL-based database system, the architecture must be custom-designed in each case (i.e., tables must be defined, with key fields along with other fields, the data type specified in each case, and, depending on the datatype, the field size specified. Then the relationships between the tables, which enable the front end of the database to determine which data records from one table correspond to which data records from another. And finally, user groups must be defined, and their respective permissions set. Each of the major DBMS applications has its own set of tools and templates to assist a developer with design and development of a database, but by themselves cannot be counted upon to address the unique security concerns of a particular database. That part requires very careful planning and implementation by a database professional who knows what the hell he or she is doing.

[markpkessinger]

31. You make an important point about the term "firewall"

In this context, the "firewall" is not a piece of software -- like, say, a Windows firewall or a Norton Internet Firewall, that operates independently of other software. In this case, it is a kind of catch-all phrase to describe the constellation of read/write/edit/change permissions, of the database itself, which can be set at the object level (i.e., database, table, field, query) level, at the user- or user group (e.g., Administrators, a group with top-level access that can alter the structure and permissions of the underlying database, an ALL Users group, and, in this case, groups corresponding to Sanders, Clinton and O'Malley users) level, and, if desired, at the record (row) level. And in a properly designed database with permissions appropriately set, no one will be able to access information they aren't supposed to access.

[markpkessinger]

11. Read my post again . . .

In a "select" query, the user creates a query consisting of the fields of information he or she wishes to see, as well as any filtering criteria for any of those fields. The proprietary field in question was an index of likely turnout/enthusiasm/support, a ranking from 1-100, that was developed by the Clinton campaign. If Uretsky were merely testing whether that proprietary field was exposed to Sanders staffers (thereby verifying that proprietary fields of Sanders were also likely similarly exposed to staffers of the other campaigns), then he could have simply included that index field as one of the fields to be included in the query, without telling it to "Show me all the Clinton people rated higher than 60" and "Show me all the people rated less than 30," criteria that produced output that is clearly potentially useful to the Sanders campaign. He could have tested the visibility of the proprietary field without setting any limiting criteria on it.

The "firewall" this case is not an independent piece of software that can be taken down independently of the database itself. When critical maintenance is required, a time is scheduled for it, users are notified in advance, and when the time comes, if any users remain logged on, the DB Admin can force a logoff. This is standard protocol, and it wouldn't violate any contracts.

[TexasMommaWithAHat]

24. If the firewall is lowered, allowing others to get my info

I would say that you are in breach of contract. "You" didn't force a logoff, obviously.

[stevenleser]

26. There is no firewall. There is the app or databases' access control system. nt

[stevenleser]

9. Agreed about the breach. I've also been in IT a long time as my "other career"

To include stints in IT Security and it is a very common excuse by those who intentionally hacked or accessed data they were not supposed to access that they were just "testing".

Uretsky had a computer concentration to his bioengineering major and I am sure was exposed to all kinds of information about ethical computer use and would have definitely known better than this.

[markpkessinger]

12. I'm not defending Uretsky's actions . . .

. . . but Uretsky's inappropriate and wrongful actions are no excuse for inaction to address a major security flaw on the part of the DNC and NGP VAN.

[stevenleser]

15. I'm reserving my opinions on NGP VAN and the DNC for a number of reasons

So far DWS' and the DNCs actions do not seem egregious or unwarranted to me.i know if the situation were reversed, Sanders supporters would still be screaming bloody murder and would not believe this was an isolated incident. I know this because several of them are still posting about the b.s email server issue. I also know if the situation were reversed and DWS and the DNC tried to keep it quiet and not tell the press, there would be all kinds of allegations of a conspiracy.

And I don't know what the issue was and how difficult it was to fix or what constraints were put in NGP VAN in terms of being able to take the DB offline to implement a fix. For all we know, the DNC told them it was too important for the campaigns to have continuous access to the data to take the app offline. We just don't know.

[rjsquirrel]

27. I noticed that

I was testing the vulnerability and wearing my white hat I swear...

Almost guaranteed he was guilty when I heard that.

[stevenleser]

28. Exactly & mpkessinger's point is even more right on, that queries that verified an issue but did not

reveal actionable data could have been run instead.

[TheProgressive]

13. We need to question the staffer's allegiance...

Was the staffer bribed by the Clinton campaign?

[stevenleser]

16. You mean all four staffers? One now fired and two now suspended and the fourth

we don't know about yet?

[TheProgressive]

17. Start with the original staffer who was fired.

[stevenleser]

20. Start what? nt

[gwheezie]

21. Oh dear that really is funny

I did it. I bribed poor Bernie's staff and made it look like Hillary did it (i am frequently mistaken for her since we own the same pantsuit) because I'm working for OMalley. He got the idea from Howard Dean so Bernie would look bad when it turned out it wasn't Hillary at all.
Sandy Berger was resurrected so we could hide the documents in his pants, Bill drove the getaway car to whitewater and with the help of Ken Starr, who it turns out was really working with dws all along, (to make sarah palin look bad) transferred the incriminating evidence to Benghazi!
I had to take the day off from work but got my doctor to write a sick note, he was in on it too. Should I call Trey Gowdy? Or Alex Jones?

[Agschmid]

45. ...

Good lord.

[TheProgressive]

47. What, you think it is impossible that the staffer could have been bribed?

[Agschmid]

48. In this case yes.

This is rediculous, the attempt to pin the error of someone Sanders fired for their action on Clinton is twisting you guys into pretzels.

Sanders has admitted this was a mistake, he repeated that again on MTP today... This is NOT a conspiracy, and certainly not the one you are painting.

Again...

Good lord.

[TheProgressive]

49. Why did the DNC open up the breach right before the debates?

[Agschmid]

54. The DNC didn't.

The fired staffer is the one you need to direct your anger at, they were a fool.

[mythology]

18. The one problem with your analysis is that multiple staffers were involved

and the Sanders campaign initially misstated the truth calling it a low level staffer and didn't mention the specific nature of the queries.

I still think it's largely a non-issue in terms of the data breach due to how far behind Sanders is nationally, but the initial cover up was really a stupid thing to do.

[KittyWampus]

40. No, Sanders didn't take timely action. He only just fired two more staffers yesterday.

[nc4bo]

43. Oh quit it Kitty..

http://m.huffpost.com/us/entry/sanders-sues-dnc_56748b06e4b06fa6887d883e

The Sanders campaign fired its national data director, Josh Uretsky, on Friday, and said it was investigating whether to take disciplinary action against any other staffers who may have been involved. The DNC and Clinton’s campaign have said that four separate accounts accessed Clinton’s data.

So original breach was Wednesday. The date of this Sanders statement was Friday. Yesterday was Saturday.

He fired more personnel.

I'd say the campaign is doing what it said it was going to do and it took a couple days to perform the investigation.

[Agschmid]

46. It was timely.

[valerief]

52. Without knowing the database schema, wouldn't having no selection criteria

possibly be reckless? (I assume Uretsky knew the schema.) I thought this database housed data for more than just the voters in this election. He wouldn't want to tie things up with a query that could yield a gazillion rows, especially since all he wanted to find out was how exposed Sanders data was.

If Uretsky's intent had been merely investigative, he could have included that field in a select query, with no particular selection criteria specified, in order to test whether proprietary fields were exposed. There would have been no need to place any particular selection criteria on that field in the query, much less to filter it in a way that would yield information that could be specifically useful to the Sanders campaign.

[markpkessinger]

56. He could have applied selection criteria to any of the shared fields . . .

. . . say, by zip code or county, or some other piece of shared data. Merely including the Clinton proprietary index field would have shown the value of the field for those records that matched the selection criteria placed on the shared field. The fact that he applied selection criteria to a field he knew he shouldn't have access to, and those criteria yielded information that could be particularly useful to the Sanders campaign, militates against any suggestion that he was merely "testing."

In any case, an end-user should never need to know, and actually shouldn't know, the database schema. If access to the database and its various tables is properly set, and permissions are set according to user group (rather than to individual users). If any proprietary data belonging to a specific campaign is stored, as it should be, in a separate table, linked to the main, shared table by a key field, then the only users who even need to know that separate table exists are those who are permitted to see that data in it. To the extent a user can do anything "dangerous" in a database, that is a problem of incompetent database design. Period.

[stevenleser]

61. What OP seems to be suggesting is to list one or more field names in certain tables

Not data. I don't think that would yield a very large result.

[valerief]

63. I included in my post what the poster was suggesting.

No selection criteria.

It doesn't matter how many columns are returned. It's how many rows.

[highprincipleswork]

55. There's just no excuse for DWS. She is her own disaster in the making.

She should be fired immediately, with someone less biased and more willing to supervise a fair nomination process put in place, and all this NONE TOO SOON!!!!

[markpkessinger]

57. An added point about the age of the NGP VAN database

Yes, by technology standards, it is "old," dating from the mid-1990s. But the basic principles of good database design and security have been long established and well-documented since at least the late 1980s. So the age of the database fails as an excuse for this kind of design incompetence.

[stevenleser]

62. What I wrote wasn't intended as an excuse but a potential explanation.

There is no excuse for not having good DB design and maintenance processes in 2015. But I have seen all kinds of nonsense in my career from folks trying to cut corners to save money or because they were just plain lazy.