Connect with us


Stephen Jones: Injury Doesn’t Affect Dak’s Contract Negotiation

Emily walpole



Between the time Cowboys QB Dak Prescott’s right leg collapsed under the weight of Giants DB Logan Ryan, Prescott was carted back to the locker room, transported to a local hospital and operated on, there wasn’t much time for many of those involved to contemplate the injury’s long-term ramifications.

At that point, concern was with Prescott, and the focus was on getting him everything he needed in the moment. Which is why Cowboys owner Jerry Jones and COO Stephen Jones left their box the minute the cart pulled off the field—as Dallas’s game against the Giants resumed—and headed for the elevators to meet their quarterback in the bowels of AT&T Stadium. They caught him just before he was loaded into the ambulance.

The conversation was private. But the sentiment was obvious.

“Anytime you see that type of injury that can happen in our game, we’ve had them from everybody from Alex Smith to Joe Theismann, to have it happen to somebody that’s your own—on your own team—it breaks your heart,” Stephen Jones said over the phone Wednesday night. “Dak being our leader in every way, it magnified the situation even more. It’s just very, very difficult, knowing how much he loves the game, how much he loves to compete, how much he wants to win for this team.

“When you saw that, you knew that he was gonna be done for this year, you didn’t need to talk to a doctor or anything like that. There was finality as far as this year is concerned.”

Now, four days later, the reality of the long-term situation can be considered.

The Cowboys and Prescott’s camp have, for 18 months or so, tried to get a long-term deal done. Their failure to do so meant Prescott was playing on the final year of his rookie deal in 2019, at just over $2 million, and on the exclusive version of the franchise tag this year, with the price ballooning to $31.409 million. So no matter what happened with Dallas this fall, Prescott’s contract situation was going to linger over it.

As Jones said, now everyone has finality on that part of it. We know Prescott won’t play again this year. We also know that when, by rule of the franchise tag, negotiations can pick back up after Dallas’s season ends, Prescott will still be rehabbing.

And as Jones and I spoke, I figured that could cloud the situation come January or February.

Instead, Jones told me the team’s approach to Prescott won’t be shaken, even a little.

“Doesn’t change anything,” he said. “We’re all in on trying to get it done. Doesn’t change a thing.”

Then, I raised another tag—which would cost $37.7 million—and the point was driven home.

“Like I said, nothing changes,” he said “There is zero change in his contract negotiation status, how much we want to get him signed. Nothing changes. He didn’t lose value. Nothing. It’s a continually work in progress. Obviously, it’s been a challenge for us, in that we haven’t gotten it done, especially with how bad I know he wants to be a Cowboy.

“And I know how bad we want him to be the leader of this team for the next 10, 12 years.”

So in case you were wondering where the Cowboys stand, there it is.


Week 6 has been retrofitted—we get no Thursday night game, and two Monday night games this week—and we’re here to prepare you for it. Inside this week’s GamePlan …

• How the Titans’ players rallied, and what they showed everyone.

• Who Matt Rhule might be helping with his Panthers’ breakthrough.

• Power rankings!

• Some football that will be played on Thursday night (you’ll like this one).

But we’re starting with the Cowboys, Prescott and where the franchise goes from here.


The next question here then would be fairly obvious: If the Cowboys feel this strongly about Prescott, why didn’t they already sign him?

It’s a fair one, too, and there’s detail here—and this isn’t from my conversation with Jones—that may not be particularly entertaining for the average football fan thinking that contract disputes happen like Rod Tidwell’s did in Jerry Maguire.

Basically, the heart of the problem between the Cowboys and Prescott came down to years. In most other sports, where contracts are fully guaranteed, players want longer-term deals because that means more security. In football, the opposite is true. Because the biggest deals for the brightest stars usually guarantee money through only three years, every year past that becomes a de facto team option.

Accordingly, Prescott and his agents wanted a four-year extension, like the deals that Russell Wilson has done in Seattle—in fact, my understanding is something around Wilson’s deal (money and everything) would’ve been sufficient in the spring. The problem was the precedent Dallas had set in insisting on longer deals with stars like Ezekiel Elliott, Zack Martin, Tyron Smith, Jaylon Smith and DeMarcus Lawrence.

So the sides could never bridge that gap, which had very little to do with what the Cowboys thought of Prescott as a player and a person.

That part of the equation wasn’t a problem. In fact, the Cowboys saw how Prescott handled the impasse—the kind that often can cause friction with players—as further affirmation of who their quarterback is.

“It’s a full body of work,” Jones said. “He continues to build on how much respect he gets, how much we think of him, not only as a player but as a person, as a leader, a face of the organization, it only increases with each month that passes. You just continue to gain respect for him as you go. He handles adversity like no other. He’s just a special man.”

And so Jones has no doubt that whatever Prescott can do to get back better than ever he will do in the coming months. “I don’t want to, in any way, underplay the severity of the injury. But the normal prognosis for something like this is four to six months. Betting on Dak, he always seems to be on the low side of something like that.”

The scene on the field in the aftermath of the injury showed, again, why. The respect that Prescott commands, and has earned, was pretty clear.


And this is where we can go back from the future to this year.

What Dallas is losing was illustrated perfectly out there on the AT&T Stadium turf. It wasn’t just Cowboys teammates going over to see Prescott. It was the Giants, too. And not even just the guys who knew him—New York had more than a couple.

“Obviously, the severity of the injury was very clear,” Jones said. “I know people who saw it, and could see it, just from a box in the stadium. You could tell immediately. And then you take the respect that this league has for Dak. Not even the people who know him, that goes without saying, [Giants and ex-Dallas coaches] Jason Garrett or Marc Colombo or Derek Dooley, the people who know him were there.

“But just the overall respect from people who just admire what he’s done, and what a worker that he is and what a leader that he is, there’s just such respect for him. And that’s the reason.”

Now, taking all of this into account, the flip side is that Dallas has been here before. More important than just that, the Cowboys learned from it.

When Tony Romo broke his collarbone in Week 2 of 2015, a season that started with so much promise went careening into a ditch. There were plenty of factors (Greg Hardy was definitely one), but the biggest issue, without question, in Dallas’s inability to weather Romo’s injury was its depth issue behind the starter. Brandon Weeden got the first crack at it, Matt Cassel was then signed to replace him and Cassel was eventually benched in favor of Kellen Moore, with Romo also making a brief—and failed—bid at returning in the middle of all that.

That was a big reason for the Cowboys to make quarterback depth a focus in 2016 draft, which is how Dallas wound up with Prescott. And the idea of building out the room past the starter was only reaffirmed with Mike McCarthy’s arrival in Dallas.

The ex-Packers coach, Jones explained, made it clear from the start that he wanted to add a veteran and draft a developmental prospect at the position. So when Andy Dalton became available at an affordable rate, the decision to pursue him was academic. “Great news for us, Andy was ready to come back to Texas, so turned out it was in play,” Jones said. “Great opportunity for us, and he wanted to be here. Couldn’t have worked out any better.”

So going forward, the Cowboys have a roster they believed could compete for a title in 2020 with Prescott at the helm—with the deep investment in the core as proof of that feeling—and now go to a quarterback who’s been to the playoffs five times, and is still just 32 years old. That’s why, when I asked Jones whether he feels fortunate to have Dalton, he didn’t even let me finish the question.

“No, very fortunate. Very fortunate,” Jones said. “Same type of guy, class act, leader, been there, done that, obviously led his team to the playoffs. Not unlike the Cowboys as of late, hasn’t won a lot of playoff games. But he certainly got his team there with the opportunity to do it. We’re just fortunate to have that type of guy here ready to step in. I know his teammates have total respect for him and confidence in him.”

And for that reason, he sees the season as far from over.

“[Defensive coordinator] Mike Nolan made some adjustments that help our personnel and certainly our players are embracing and executing his defense better,” he continued. “And certainly, the offense has been good throughout. I think Andy gave you a glimpse there at the end, leading us down the field, I think he was 9-of-11. In the heat of the battle, crunch time, he played well. So I’ve got all the confidence in the world in this team.”

So yeah, a lot did change on Sunday afternoon in Arlington.

But, if you listen to Jones, plenty also stayed the same.



1) Green Bay Packers (4–0): This team is coming off a bye, and I can’t wait for Packers–Bucs on Sunday. Brady–Rodgers is always a fantastic show, and I expect nothing less from this one.

2) Seattle Seahawks (5–0): I’m a little uneasy about the Seahawks’ defense still, but what Russell Wilson is doing on a week-to-week basis is enough to keep Seattle moving up the list.

3) Tennessee Titans (4–0): That was impressive. Buffalo was down its starting corners, but the overall effort from Tennessee, and the style with which it won, says pretty loudly that Mike Vrabel’s crew isn’t going away after the team’s outbreak.

4) Pittsburgh Steelers (4–0): That Browns–Steelers is on the marquee this week is great. It’s been a long, long time since one of the NFL’s best rivalries held the sort of importance in the standings that Sunday’s game will carry.

5) Kansas City Chiefs (4–1): I’m willing to consider the loss to the Raiders as a blip, but concerns on K.C.’s offensive line remain. And the defense, for the first time all year, looked like the old Chiefs’ defenses (not a good thing). The Bills will provide a nice test late on Monday afternoon.



How did the Titans pull THAT off?

We can start here: Football coaches will use just about anything for motivation, so the NFL put this one on a tee for Mike Vrabel. As his players saw it, on Monday night, he drove that thing 300 yards in addressing them in a way that humanized what they’d been through over the two weeks previous.

Indeed, they were dealing with the stress of the team’s COVID-19 outbreak and fingers pointed squarely at the players and coaches for their handling of protocols. And no matter where you fall on how Tennessee has handled this (pictures surfaced of Titans players having a group workout at a local high school, which I wrote about in this space last week), it’s not hard to see how players would buy into the idea that they’d been turned into villains in an ordeal that put their own health in the crosshairs.

“Mike, man, every single game, he tries to find something that gets the guys going,” said veteran safety Kevin Byard. “And obviously, with all the criticism we faced the past few weeks, he pretty much let us know, and not to use the exact words, Hey, it’s us against the world. We have to go out there and prove everybody wrong. Everything that was said about us, prove them wrong. At the end of the day, don’t have to say too much, let our play speak for us.

“I think our play spoke volumes to how we prepared.”

Sure did. After being locked out of team headquarters for 11 days, and having been without practice for two weeks, the Titans got two days of on-field work in, then bludgeoned the previously unbeaten Bills 42–16.

Getting there required a lot of adjustments, obviously. Byard was nice enough the other day to give us a snapshot of his last two weeks.

Managing emotions. Waking up day after day to news of coworkers testing positive and schedules being pushed back can take a toll on a player, to be sure. And in Byard’s case, it took him back to the summer and the fallout was felt all over his house.

“Before the season started, we had the whole deal where guys were able to opt out. Me and my wife, we sat down, we talked about it and we thought it was best that I play this year,” Byard said. “But she was pregnant at the time, I just had a son in August, and I have a one-year-old daughter. Waking up and seeing, positive test, positive test, positive test, you’re just thinking, ‘Hey, am I gonna be next on the list?’

“I have a nanny that works in our house. She was kind of concerned. We didn’t know if she was gonna want to be around the kids. We had to send her home. … Coming home, my daughter’s one year old, she runs up to me, I’m thinking, Should I kiss her? Should I stay back? Should I quarantine and stay upstairs, because we all know the incubation period can take up to 12 days? It was challenging, man.”

Managing his body. Under normal circumstances, almost anything an NFL player would need is there for him under one roof—and so players naturally become reliant on being under that roof. So what happens when the front door is closed?

Byard took his Peloton out. During quarantine, he used the bike to stay in shape, going on challenging HIIT and Tabata rides a few times a week. And then, once camp began, and the rhythm of football season kicked in, he toned it down, but still found time to get in a 20-minute low-impact ride here or a 30-minute hip hop ride there. So over the last two weeks, he just pivoted and ramped that work back up.

“The first week, I treated it like a bye week a little bit, so I did some low-impact rides,” Byard said. “But the second week, right before the facility opened back up, I was doing those HIIT rides, trying to get my body right. Also, just on my own, I went out to a park and got some striders in, because the Peleton works very well to keep your muscles and stuff good, but I had to stretch it out, make sure my legs were good.”

Managing his mindset. There were obvious concerns going into game day physically, without all the work a player would normally get in. And those were addressed in a pretty interesting way, with Byard telling his teammates, “Take it back to them park ball days when you had to wake up Saturday morning, throw your pads on and go play football. It was one of those where guys said, Forget it, we’re just gonna play.”

But beyond that, and maybe more significantly, there was also mental stuff people might not think about that weighed on Byard once he got into the 24 hours leading into kickoff. It takes a lot to be in the right mindset to play an NFL game. Which led the 27-year-old star to YouTube late Monday night.

“I just started watching some Brian Dawkins highlights, some Troy Polamalu, some Ed Reed highlights, just to get my mind set so I could play unconscious. Don’t even think about nothing that happened before, just go out there and try to make as many plays as you can for your team,” Byard said. “And that’s what I preached to the guys before the game. It’s a blessing that we even have the opportunity to play this game. This game almost got, not necessarily snatched from us, but we almost weren’t there because of this COVID thing.

“So I’m like, Take this opportunity for what it is and go out there and try to ball out.”

The Titans did.


So now the page is turned, and the Titans are getting ready for the Texans with, they hope, this episode behind them. But Byard says that, without question, he will be trying to lead in tightening up his locker room’s approach to managing the protocols.

“The way we gotta look at, we used our lifeline already,” he said. “We used our bye week up. At this point, we don’t have any room to be lax on protocols or not take it very seriously. We were the first team to have an outbreak. Obviously, Dennis Kelly, our NFLPA leader, Mike Vrabel and really Jon Robinson, they’ve all been doing a great job making sure that we’re doing what we can—even during practice, spacing out on the sidelines.

“We just have to continue to preach that, myself, all the coaches, all the players, leaders on the team, we used our lifeline. We have no more lifelines left. We have to continue to stay safe and do what we can. This is real.”

Byard stood by the effort made by Vrabel and Robinson, too, saying, “People just trying to take from what they see—we should forfeit a game, the Titans are trying to destroy the NFL season, I think that was pretty unfair.” But he conceded complacency may have crept into places like the lunchroom. With full mask compliance, he hopes that everyone’s gotten the message now.

If nothing else, the last couple of weeks showed the Titans how things can be taken away, just as Tuesday night proved how much the team has to lose if this happens again.

“I will say that [Tuesday] night showed that the guys that we have in that locker room, from the coaches to the staff to everybody around the building, it’s Tennessee Tough, man,” Byard said. “We’ve faced a lot this year. Having the leaders we have on this team, having the coaches, and just us really staying locked in on the goal, we all want to be winners. That’s really the whole deal. We want to be winners by any means necessary.

“You can throw anything at us, we’re gonna be able to take it and keep it rolling. At the end of the day, I think adversity makes the strong stronger. And I think [Tuesday] showed that.”



The potential Matt Rhule Effect.

I’m not crowning the Panthers champions. But what we’ve seen so far? It looks pretty damn good. They were very competitive in battling back from double-digit deficits against Vegas and Tampa in Weeks 1 and 2, have won three in a row since and have done it after spring and summer circumstances that were supposed to make life harder on first-year coaches.

Rhule has also pulled this off without more than a single season as an assistant offensive line coach of previous experience in the NFL. So I figured I’d ask around a little and see whether Carolina’s breaking through so quickly could impact hiring in pro football.

“Absolutely,” said one exec who is well-connected in the college game.

So we’re going to get you a list of guys in college who might be able to transition like Rhule has soon. But for now, it’s worth examining what someone like Rhule brings to the table that would work for any NFL team—and has worked in Charlotte thus far.

Program building. In college, coaches have their hands in the pot of everything, from on-field stuff to player evaluations through recruiting to fundraising and business operations. So someone like Rhule comes thoroughly prepared to manage the football side of an NFL franchise, because he was responsible for so much in his college jobs.

Practice structure. College programs deal with far more stringent rules around the work they get with players, so they have to be creative and efficient. You may remember Chip Kelly’s practices being a thing of legend on the scouting trail 10 years ago, which was a major factor in NFL teams’ interest in him. A similar dynamic existed with Rhule—and he’s told me himself coming from college served him well this summer in that regard (as I wrote at length in August).

Crisis management. Rhule’s experience in this area is unique, of course, because of what he was tasked with at Baylor. But college coaches deal with extracurricular problems on a lot of different levels because, simply, their players are college kids. And so that part of being an NFL coach wouldn’t be foreign to a guy coming from a big-conference college job.

Open-mindedness on scheme. This is fairly simple: Schemes in college lack the depth that NFL schemes do, but are oftentimes more diverse from school-to-school and more creative (if you’re at, say, Vanderbilt, you have to have a little extra against Alabama). Which means someone coming from that level is going to, naturally, be pretty open-minded. Rhule, for example, hired Joe Brady, with whom he had no history, because he liked his creativity.

History with players. This is a short-term benefit, of course, but a benefit nonetheless. A lot of players that Rhule assessed leading up to the draft were guys he’s had his eyes on since they were teenagers (both guys he coached and guys he recruited who went to other schools). That, of course, creates an inherent advantage. And it’s one that Seahawks coach Pete Carroll has cited repeatedly, in discussing Seattle’s success drafting in his early years.

Now, does this mean NFL teams should just wildly swing at whomever the hot college name is come December? Of course not. A huge part of Rhule’s success is who Rhule is, regardless of his background. So nothing is automatic, and the drawbacks of hiring coaches from college are still out there.

But if you’re asking whether Rhule’s success means teams will take a harder look at the college ranks this year than they have previously in looking for coaches, then my answer is yes, absolutely, I believe that’ll happen.



Dying for some Thursday Night Football, with Chiefs–Bills moved to Monday? We’ve got you covered.

Tonight, at 9 p.m. ET on ESPN2, Isidore Newman takes on Booker T. Washington. Why should you care about a Louisiana high school game? Newman’s sophomore quarterback, Arch Manning, has a couple of uncles you may have heard of. And he happens to be a player you’ll be hearing a lot about over the next few years.

So enjoy, and you’re welcome.


The First Few Milliseconds of an HTTPS Connection

Mish Boyka




Convinced from spending hours reading rave reviews, Bob eagerly clicked “Proceed to Checkout” for his gallon of Tuscan Whole Milk and…

Whoa! What just happened?

In the 220 milliseconds that flew by, a lot of interesting stuff happened to make Firefox change the address bar color and put a lock in the lower right corner. With the help of Wireshark, my favorite network tool, and a slightly modified debug build of Firefox, we can see exactly what’s going on.

By agreement of RFC 2818, Firefox knew that “https” meant it should connect to port 443 at

Most people associate HTTPS with SSL (Secure Sockets Layer) which was created by Netscape in the mid 90’s. This is becoming less true over time. As Netscape lost market share, SSL’s maintenance moved to the Internet Engineering Task Force (IETF). The first post-Netscape version was re-branded as Transport Layer Security (TLS) 1.0 which was released in January 1999. It’s rare to see true “SSL” traffic given that TLS has been around for 10 years.

Client Hello

TLS wraps all traffic in “records” of different types. We see that the first byte out of our browser is the hex byte 0x16 = 22 which means that this is a “handshake” record:

The next two bytes are 0x0301 which indicate that this is a version 3.1 record which shows that TLS 1.0 is essentially SSL 3.1.

The handshake record is broken out into several messages. The first is our “Client Hello” message (0x01). There are a few important things here:

  • Random:

    There are four bytes representing the current Coordinated Universal Time (UTC) in the Unix epoch format, which is the number of seconds since January 1, 1970. In this case, 0x4a2f07ca. It’s followed by 28 random bytes. This will be used later on.
  • Session ID:

    Here it’s empty/null. If we had previously connected to a few seconds ago, we could potentially resume a session and avoid a full handshake.
  • Cipher Suites:

    This is a list of all of the encryption algorithms that the browser is willing to support. Its top pick is a very strong choice of “TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA” followed by 33 others that it’s willing to accept. Don’t worry if none of that makes sense. We’ll find out later that Amazon doesn’t pick our first choice anyway.
  • server_name extension:

    This is a way to tell that our browser is trying to reach This is really convenient because our TLS handshake occurs long before any HTTP traffic. HTTP has a “Host” header which allows a cost-cutting Internet hosting companies to pile hundreds of websites onto a single IP address. SSL has traditionally required a different IP for each site, but this extension allows the server to respond with the appropriate certificate that the browser is looking for. If nothing else, this extension should allow an extra week or so of IPv4 addresses.

Server Hello replies with a handshake record that’s a massive two packets in size (2,551 bytes). The record has version bytes of 0x0301 meaning that Amazon agreed to our request to use TLS 1.0. This record has three sub-messages with some interesting data:

  1. “Server Hello” Message (2):

    • We get the server’s four byte time Unix epoch time representation and its 28 random bytes that will be used later.
    • A 32 byte session ID in case we want to reconnect without a big handshake.
    • Of the 34 cipher suites we offered, Amazon picked “TLS_RSA_WITH_RC4_128_MD5” (0x0004). This means that it will use the “RSApublic key algorithm to verify certificate signatures and exchange keys, the RC4 encryption algorithm to encrypt data, and the MD5 hash function to verify the contents of messages. We’ll cover these in depth later on. I personally think Amazon had selfish reasons for choosing this cipher suite. Of the ones on the list, it was the one that was least CPU intensive to use so that Amazon could crowd more connections onto each of their servers. A much less likely possibility is that they wanted to pay special tribute to Ron Rivest, who created all three of these algorithms.
  2. Certificate Message (11):

    • This message takes a whopping 2,464 bytes and is the certificate that the client can use to validate Amazon’s. It isn’t anything fancy. You can view most of its contents in your browser:
  3. “Server Hello Done” Message (14):

    • This is a zero byte message that tells the client that it’s done with the “Hello” process and indicate that the server won’t be asking the client for a certificate.

Checking out the Certificate

The browser has to figure out if it should trust In this case, it’s using certificates. It looks at Amazon’s certificate and sees that the current time is between the “not before” time of August 26th, 2008 and before the “not after” time of August 27, 2009. It also checks to make sure that the certificate’s public key is authorized for exchanging secret keys.

Why should we trust this certificate?

Attached to the certificate is a “signature” that is just a really long number in big-endian format:

Anyone could have sent us these bytes. Why should we trust this signature? To answer that question, need to make a speedy detour into mathemagic land:

Interlude: A Short, Not Too Scary, Guide to RSA

People sometimes wonder if math has any relevance to programming. Certificates give a very practical example of applied math. Amazon’s certificate tells us that we should use the RSA algorithm to check the signature. RSA was created in the 1970’s by MIT professors Ron Rivest, Adi Shamir, and Len Adleman who found a clever way to combine ideas spanning 2000 years of math development to come up with a beautifully simple algorithm:

You pick two huge prime numbers “p” and “q.” Multiply them to get “n = p*q.” Next, you pick a small public exponent “e” which is the “encryption exponent” and a specially crafted inverse of “e” called “d” as the “decryption exponent.” You then make “n” and “e” public and keep “d” as secret as you possibly can and then throw away “p” and “q” (or keep them as secret as “d”). It’s really important to remember that “e” and “d” are inverses of each other.

Now, if you have some message, you just need to interpret its bytes as a number “M.” If you want to “encrypt” a message to create a “ciphertext”, you’d calculate:

C ≡ Me (mod n)

This means that you multiply “M” by itself “e” times. The “mod n” means that we only take the remainder (e.g. “modulus”) when dividing by “n.” For example, 11 AM + 3 hours ≡ 2 (PM) (mod 12 hours). The recipient knows “d” which allows them to invert the message to recover the original message:

Cd ≡ (Me)d ≡ Me*d ≡ M1 ≡ M (mod n)

Just as interesting is that the person with “d” can “sign” a document by raising a message “M” to the “d” exponent:

Md ≡ S (mod n)

This works because “signer” makes public “S”, “M”, “e”, and “n.” Anyone can verify the signature “S” with a simple calculation:

Se ≡ (Md)e ≡ Md*e ≡ Me*d ≡ M1 ≡ M (mod n)

Public key cryptography algorithms like RSA are often called “asymmetric” algorithms because the encryption key (in our case, “e”) is not equal to (e.g. “symmetric” with) the decryption key “d”. Reducing everything “mod n” makes it impossible to use the easy techniques that we’re used to such as normal logarithms. The magic of RSA works because you can calculate/encrypt C ≡ Me (mod n) very quickly, but it is really hard to calculate/decrypt Cd ≡ M (mod n) without knowing “d.” As we saw earlier, “d” is derived from factoring “n” back to its “p” and “q”, which is a tough problem.

Verifying Signatures

The big thing to keep in mind with RSA in the real world is that all of the numbers involved have to be big to make things really hard to break using the best algorithms that we have. How big?’s certificate was “signed” by “VeriSign Class 3 Secure Server CA.” From the certificate, we see that this VeriSign modulus “n” is 2048 bits long which has this 617 digit base-10 representation:

1890572922 9464742433 9498401781 6528521078 8629616064
3051642608 4317020197 7241822595 6075980039 8371048211
4887504542 4200635317 0422636532 2091550579 0341204005
1169453804 7325464426 0479594122 4167270607 6731441028
3698615569 9947933786 3789783838 5829991518 1037601365
0218058341 7944190228 0926880299 3425241541 4300090021
1055372661 2125414429 9349272172 5333752665 6605550620
5558450610 3253786958 8361121949 2417723618 5199653627
5260212221 0847786057 9342235500 9443918198 9038906234
1550747726 8041766919 1500918876 1961879460 3091993360
6376719337 6644159792 1249204891 7079005527 7689341573
9395596650 5484628101 0469658502 1566385762 0175231997
6268718746 7514321

(Good luck trying to find “p” and “q” from this “n” – if you could, you could generate real-looking VeriSign certificates.)

VeriSign’s “e” is 216 + 1 = 65537. Of course, they keep their “d” value secret, probably on a safe hardware device protected by retinal scanners and armed guards. Before signing, VeriSign checked the validity of the contents that claimed on its certificate using a real-world “handshake” that involved looking at several of their business documents. Once VeriSign was satisfied with the documents, they used the SHA-1 hash algorithm to get a hash value of the certificate that had all the claims. In Wireshark, the full certificate shows up as the “signedCertificate” part:

It’s sort of a misnomer since it actually means that those are the bytes that the signer is going to sign and not the bytes that already include a signature.

The actual signature, “S”, is simply called “encrypted” in Wireshark. If we raise “S” to VeriSign’s public “e” exponent of 65537 and then take the remainder when divided by the modulus “n”, we get this “decrypted” signature hex value:

FFFFFFFF00302130 0906052B0E03021A 05000414C19F8786
871775C60EFE0542 E4C2167C830539DB

Per the PKCS #1 v1.5 standard, the first byte is “00” and it “ensures that the encryption block, [when] converted to an integer, is less than the modulus.” The second byte of “01” indicates that this is a private key operation (e.g. it’s a signature). This is followed by a lot of “FF” bytes that are used to pad the result to make sure that it’s big enough. The padding is terminated by a “00” byte. It’s followed by “30 21 30 09 06 05 2B 0E 03 02 1A 05 00 04 14” which is the PKCS #1 v2.1 way of specifying the SHA-1 hash algorithm. The last 20 bytes are SHA-1 hash digest of the bytes in “signedCertificate.”

Since the decrypted value is properly formatted and the last bytes are the same hash value that we can calculate independently, we can assume that whoever knew “VeriSign Class 3 Secure Server CA”’s private key “signed” it. We implicitly trust that only VeriSign knows the private key “d.”

We can repeat the process to verify that “VeriSign Class 3 Secure Server CA”’s certificate was signed by VeriSign’s “Class 3 Public Primary Certification Authority.”

But why should we trust that? There are no more levels on the trust chain.

The top “VeriSign Class 3 Public Primary Certification Authority” was signed by itself. This certificate has been built into Mozilla products as an implicitly trusted good certificate since version 1.4 of certdata.txt in the Network Security Services (NSS) library. It was checked-in on September 6, 2000 by Netscape’s Robert Relyea with the following comment:

“Make the framework compile with the rest of NSS. Include a ‘live’ certdata.txt with those certs we have permission to push to open source (additional certs will be added as we get permission from the owners).”

This decision has had a relatively long impact since the certificate has a validity range of January 28, 1996 – August 1, 2028.

As Ken Thompson explained so well in his “Reflections on Trusting Trust”, you ultimately have to implicitly trust somebody. There is no way around this problem. In this case, we’re implicitly trusting that Robert Relyea made a good choice. We also hope that Mozilla’s built-in certificate policy is reasonable for the other built-in certificates.

One thing to keep in mind here is that all these certificates and signatures were simply used to form a trust chain. On the public Internet, VeriSign’s root certificate is implicitly trusted by Firefox long before you go to any website. In a company, you can create your own root certificate authority (CA) that you can install on everyone’s machine.

Alternatively, you can get around having to pay companies like VeriSign and avoid certificate trust chains altogether. Certificates are used to establish trust by using a trusted third-party (in this case, VeriSign). If you have a secure means of sharing a secret “key”, such as whispering a long password into someone’s ear, then you can use that pre-shared key (PSK) to establish trust. There are extensions to TLS to allow this, such as TLS-PSK, and my personal favorite, TLS with Secure Remote Password (SRP) extensions. Unfortunately, these extensions aren’t nearly as widely deployed and supported, so they’re usually not practical. Additionally, these alternatives impose a burden that we have to have some other secure means of communicating the secret that’s more cumbersome than what we’re trying to establish with TLS (otherwise, why wouldn’t we use that for everything?).

One final check that we need to do is to verify that the host name on the certificate is what we expected. Nelson Bolyard’s comment in the SSL_AuthCertificate function explains why:

/* cert is OK. This is the client side of an SSL connection.
 * Now check the name field in the cert against the desired hostname.
 * NB: This is our only defense against Man-In-The-Middle (MITM) attacks! 

This check helps prevent against a man-in-the-middle attack because we are implicitly trusting that the people on the certificate trust chain wouldn’t do something bad, like sign a certificate claiming to be from unless it actually was If an attacker is able to modify your DNS server by using a technique like DNS cache poisoning, you might be fooled into thinking you’re at a trusted site (like because the address bar will look normal. This last check implicitly trusts certificate authorities to stop these bad things from happening.

Pre-Master Secret

We’ve verified some claims about and know its public encryption exponent “e” and modulus “n.” Anyone listening in on the traffic can know this as well (as evidenced because we are using Wireshark captures). Now we need to create a random secret key that an eavesdropper/attacker can’t figure out. This isn’t as easy as it sounds. In 1996, researchers figured out that Netscape Navigator 1.1 was using only three sources to seed their pseudo-random number generator (PRNG). The sources were: the time of day, the process id, and the parent process id. As the researchers showed, these “random” sources aren’t that random and were relatively easy to figure out.

Since everything else was derived from these three “random” sources, it was possible to “break” the SSL “security” in 25 seconds on a 1996 era machine. If you still don’t believe that finding randomness is hard, just ask the Debian OpenSSL maintainers. If you mess it up, all the security built on top of it is suspect.

On Windows, random numbers used for cryptographic purposes are generated by calling the CryptGenRandom function that hashes bits sampled from over 125 sources. Firefox uses this function along with some bits derived from its own function to seed its pseudo-random number generator.

The 48 byte “pre-master secret” random value that’s generated isn’t used directly, but it’s very important to keep it secret since a lot of things are derived from it. Not surprisingly, Firefox makes it hard to find out this value. I had to compile a debug version and set the SSLDEBUGFILE and SSLTRACE environment variables to see it.

In this particular session, the pre-master secret showed up in the SSLDEBUGFILE as:

4456: SSL[131491792]: Pre-Master Secret [Len: 48]
03 01 bb 7b 08 98 a7 49 de e8 e9 b8 91 52 ec 81 ...{...I.....R..
4c c2 39 7b f6 ba 1c 0a b1 95 50 29 be 02 ad e6 L.9{......P)....
ad 6e 11 3f 20 c4 66 f0 64 22 57 7e e1 06 7a 3b .n.? .f.d"W~..z;

Note that it’s not completely random. The first two bytes are, by convention, the TLS version (03 01).

Trading Secrets

We now need to get this secret value over to By Amazon’s wishes of “TLS_RSA_WITH_RC4_128_MD5”, we will use RSA to do this. You could make your input message equal to just the 48 byte pre-master secret, but the Public Key Cryptography Standard (PKCS) #1, version 1.5 RFC tells us that we should pad these bytes with random data to make the input equal to exactly the size of the modulus (1024 bits/128 bytes). This makes it harder for an attacker to determine our pre-master secret. It also gives us one last chance to protect ourselves in case we did something really bone-headed, like reusing the same secret. If we reused the key, the eavesdropper would likely see a different value placed on the network due to the random padding.

Again, Firefox makes it hard to see these random values. I had to insert debugging statements into the padding function to see what was going on:

wrapperHandle = fopen("plaintextpadding.txt", "a");
fprintf(wrapperHandle, "PLAINTEXT = ");
for(i = 0; i < modulusLen; i++)
    fprintf(wrapperHandle, "%02X ", block[i]);
fprintf(wrapperHandle, "rn");

In this session, the full padded value was:

00 02 12 A3 EA B1 65 D6 81 6C 13 14 13 62 10 53 23 B3 96 85 FF 24
FA CC 46 11 21 24 A4 81 EA 30 63 95 D4 DC BF 9C CC D0 2E DD 5A A6
41 6A 4E 82 65 7D 70 7D 50 09 17 CD 10 55 97 B9 C1 A1 84 F2 A9 AB
EA 7D F4 CC 54 E4 64 6E 3A E5 91 A0 06 00 03 01 BB 7B 08 98 A7 49
DE E8 E9 B8 91 52 EC 81 4C C2 39 7B F6 BA 1C 0A B1 95 50 29 BE 02
AD E6 AD 6E 11 3F 20 C4 66 F0 64 22 57 7E E1 06 7A 3B

Firefox took this value and calculated “C ≡ Me (mod n)” to get the value we see in the “Client Key Exchange” record:

Finally, Firefox sent out one last unencrypted message, a “Change Cipher Spec” record:

This is Firefox’s way of telling Amazon that it’s going to start using the agreed upon secret to encrypt its next message.

Deriving the Master Secret

If we’ve done everything correctly, both sides (and only those sides) now know the 48 byte (256 bit) pre-master secret. There’s a slight trust issue here from Amazon’s perspective: the pre-master secret just has bits that were generated by the client, they don’t take anything into account from the server or anything we said earlier. We’ll fix that be computing the “master secret.” Per the spec, this is done by calculating:

master_secret = PRF(pre_master_secret, 
                    "master secret", 
                    ClientHello.random + ServerHello.random)

The “pre_master_secret” is the secret value we sent earlier. The “master secret” is simply a string whose ASCII bytes (e.g. “6d 61 73 74 65 72 …”) are used. We then concatenate the random values that were sent in the ClientHello and ServerHello (from Amazon) messages that we saw at the beginning.

The PRF is the “Pseudo-Random Function” that’s also defined in the spec and is quite clever. It combines the secret, the ASCII label, and the seed data we give it by using the keyed-Hash Message Authentication Code (HMAC) versions of both MD5 and SHA-1 hash functions. Half of the input is sent to each hash function. It’s clever because it is quite resistant to attack, even in the face of weaknesses in MD5 and SHA-1. This process can feedback on itself and iterate forever to generate as many bytes as we need.

Following this procedure, we obtain a 48 byte “master secret” of

4C AF 20 30 8F 4C AA C5 66 4A 02 90 F2 AC 10 00 39 DB 1D E0 1F CB
E0 E0 9D D7 E6 BE 62 A4 6C 18 06 AD 79 21 DB 82 1D 53 84 DB 35 A7
1F C1 01 19

Generating Lots of Keys

Now that both sides have a “master secrets”, the spec shows us how we can derive all the needed session keys we need using the PRF to create a “key block” where we will pull data from:

key_block = PRF(SecurityParameters.master_secret,
"key expansion",
SecurityParameters.server_random +

The bytes from “key_block” are used to populate the following:


Since we’re using a stream cipher instead of a block cipher like the Advanced Encryption Standard (AES), we don’t need the Initialization Vectors (IVs). Therefore, we just need two Message Authentication Code (MAC) keys for each side that are 16 bytes (128 bits) each since the specified MD5 hash digest size is 16 bytes. In addition, the RC4 cipher uses a 16 byte (128 bit) key that both sides will need as well. All told, we need 216 + 216 = 64 bytes from the key block.

Running the PRF, we get these values:

client_write_MAC_secret = 80 B8 F6 09 51 74 EA DB 29 28 EF 6F 9A B8 81 B0
server_write_MAC_secret = 67 7C 96 7B 70 C5 BC 62 9D 1D 1F 4A A6 79 81 61
client_write_key = 32 13 2C DD 1B 39 36 40 84 4A DE E5 6C 52 46 72
server_write_key = 58 36 C4 0D 8C 7C 74 DA 6D B7 34 0A 91 B6 8F A7

Prepare to be Encrypted!

The last handshake message the client sends out is the “Finished message.” This is a clever message that proves that no one tampered with the handshake and it proves that we know the key. The client takes all bytes from all handshake messages and puts them into a “handshake_messages” buffer. We then calculate 12 bytes of “verify_data” using the pseudo-random function (PRF) with our master key, the label “client finished”, and an MD5 and SHA-1 hash of “handshake_messages”:

verify_data = PRF(master_secret,
"client finished",
MD5(handshake_messages) +
) [12]

We take the result and add a record header byte “0x14” to indicate “finished” and length bytes “00 00 0c” to indicate that we’re sending 12 bytes of verify data. Then, like all future encrypted messages, we need to make sure the decrypted contents haven’t been tampered with. Since our cipher suite in use is TLS_RSA_WITH_RC4_128_MD5, this means we use the MD5 hash function.

Some people get paranoid when they hear MD5 because it has some weaknesses. I certainly don’t advocate using it as-is. However, TLS is smart in that it doesn’t use MD5 directly, but rather the HMAC version of it. This means that instead of using MD5(m) directly, we calculate:

HMAC_MD5(Key, m) = MD5((Key ⊕ opad) ++ MD5((Key ⊕ ipad) ++ m)

(The ⊕ means XOR, ++ means concatenate, “opad” is the bytes “5c 5c … 5c”, and “ipad” is the bytes “36 36 … 36”).

In particular, we calculate:

seq_num +
TLSCompressed.type +
TLSCompressed.version +
TLSCompressed.length +

As you can see, we include a sequence number (“seq_num”) along with attributes of the plaintext message (here it’s called “TLSCompressed”). The sequence number foils attackers who might try to take a previously encrypted message and insert it midstream. If this occurred, the sequence numbers would definitely be different than what we expected. This also protects us from an attacker dropping a message.

All that’s left is to encrypt these bytes.

RC4 Encryption

Our negotiated cipher suite was TLS_RSA_WITH_RC4_128_MD5. This tells us that we need to use Ron’s Code #4 (RC4) to encrypt the traffic. Ron Rivest developed the RC4 algorithm to generate random bytes based on a 256 byte key. The algorithm is so simple you can actually memorize it in a few minutes.

RC4 begins by creating a 256-byte “S” byte array and populating it with 0 to 255. You then iterate over the array by mixing in bytes from the key. You do this to create a state machine that is used to generate “random” bytes. To generate a random byte, we shuffle around the “S” array.

Put graphically, it looks like this:

To encrypt a byte, we xor this pseudo-random byte with the byte we want to encrypt. Remember that xor’ing a bit with 1 causes it to flip. Since we’re generating random numbers, on average the xor will flip half of the bits. This random bit flipping is effectively how we encrypt data. As you can see, it’s not very complicated and thus it runs quickly. I think that’s why Amazon chose it.

Recall that we have a “client_write_key” and a “server_write_key.” The means we need to create two RC4 instances: one to encrypt what our browser sends and the other to decrypt what the server sent us.

The first few random bytes out of the “client_write” RC4 instance are “7E 20 7A 4D FE FB 78 A7 33 …” If we xor these bytes with the unencrypted header and verify message bytes of “14 00 00 0C 98 F0 AE CB C4 …”, we’ll get what appears in the encrypted portion that we can see in Wireshark:

The server does almost the same thing. It sends out a “Change Cipher Spec” and then a “Finished Message” that includes all handshake messages, including the decrypted version of the client’s “Finished Message.” Consequently, this proves to the client that the server was able to successfully decrypt our message.

Welcome to the Application Layer!

Now, 220 milliseconds after we started, we’re finally ready for the application layer. We can now send normal HTTP traffic that’ll be encrypted by the TLS layer with the RC4 write instance and decrypt traffic with the server RC4 write instance. In addition, the TLS layer will check each record for tampering by computing the HMAC_MD5 hash of the contents.

At this point, the handshake is over. Our TLS record’s content type is now 23 (0x17). Encrypted traffic begins with “17 03 01” which indicate the record type and TLS version. These bytes are followed by our encrypted size, which includes the HMAC hash.

Encrypting the plaintext of:

GET /gp/cart/view.html/ref=pd_luc_mri HTTP/1.1 
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv: Gecko/2009060911 Minefield/3.0.10 (.NET CLR 3.5.30729) 
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 
Accept-Language: en-us,en;q=0.5 
Accept-Encoding: gzip,deflate 
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 
Keep-Alive: 300 
Connection: keep-alive 

will give us the bytes we see on the wire:

The only other interesting fact is that the sequence number increases on each record, it’s now 1 (and the next record will be 2, etc).

The server does the same type of thing on its side using the server_write_key. We see its response, including the tell-tale application data header:

Decrypting this gives us:

HTTP/1.1 200 OK 
Date: Wed, 10 Jun 2009 01:09:30 GMT 
Server: Server 
Cneonction: close 
Transfer-Encoding: chunked

which is a normal HTTP reply that includes a non-descriptive “Server: Server” header and a misspelled “Cneonction: close” header coming from Amazon’s load balancers.

TLS is just below the application layer. The HTTP server software can act as if it’s sending unencrypted traffic. The only change is that it writes to a library that does all the encryption. OpenSSL is a popular open-source library for TLS.

The connection will stay open while both sides send and receive encrypted data until either side sends out a “closure alert” message and then closes the connection. If we reconnect shortly after disconnecting, we can re-use the negotiated keys (if the server still has them cached) without using public key operations, otherwise we do a completely new full handshake.

It’s important to realize that application data records can be anything. The only reason “HTTPS” is special is because the web is so popular. There are lots of other TCP/IP based protocols that ride on top of TLS. For example, TLS is used by FTPS and secure extensions to SMTP. It’s certainly better to use TLS than inventing your own solution. Additionally, you’ll benefit from a protocol that has withstood careful security GFN.

… And We’re Done!

The very readable TLS RFC covers many more details that were missed here. We covered just one single path in our observation of the 220 millisecond dance between Firefox and Amazon’s server. Quite a bit of the process was affected by the TLS_RSA_WITH_RC4_128_MD5 Cipher Suite selection that Amazon made with its ServerHello message. It’s a reasonable choice that slightly favors speed over security.

As we saw, if someone could secretly factor Amazon’s “n” modulus into its respective “p” and “q”, they could effectively decrypt all “secure” traffic until Amazon changes their certificate. Amazon counter-balances this concern this with a short one year duration certificate:

One of the cipher suites that was offered was “TLS_DHE_RSA_WITH_AES_256_CBC_SHA” which uses the Diffie-Hellman key exchange that has a nice property of “forward secrecy.” This means that if someone cracked the mathematics of the key exchange, they’d be no better off to decrypt another session. One downside to this algorithm is that it requires more math with big numbers, and thus is a little more computationally taxing on a busy server. The “Advanced Encryption Standard” (AES) algorithm was present in many of the suites that we offered. It’s different than RC4 in that it works on 16 byte “blocks” at a time rather than a single byte. Since its key can be up to 256 bits, many consider this to be more secure than RC4.

In just 220 milliseconds, two endpoints on the Internet came together, provided enough credentials to trust each other, set up encryption algorithms, and started to send encrypted traffic.

And to think, all of this just so Bob can buy milk.

UPDATE: I wrote a program that walks through the handshake steps mentioned in this article. I posted it to GitHub.

Continue Reading


Investors Extracted $400 Million From a Hospital Chain That Sometimes Couldn’t Pay for Medical Supplies or Gas for Ambulances

Mish Boyka



ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up to receive our biggest stories as soon as they’re published.

In the decade since Leonard Green & Partners, a private equity firm based in Los Angeles, bought control of a hospital company named Prospect Medical Holdings for $205 million, the owners have done handsomely.

Leonard Green extracted $400 million in dividends and fees for itself and investors in its fund — not from profits, but by loading up the company with debt. Prospect CEO Sam Lee, who owns about 20% of the chain, made $128 million while expanding the company from five hospitals in California to 17 across the country. A second executive with an ownership stake took home $94 million.

The deal hasn’t worked out quite as well for Prospect’s patients, many of whom have low incomes. (The company says it receives 80% of its revenues from Medicare and Medicaid reimbursements.) At the company’s flagship Los Angeles hospital, persistent elevator breakdowns sometimes require emergency room nurses to wheel patients on gurneys across a public street as a security guard attempts to halt traffic. Paramedics for Prospect’s hospital near Philadelphia told ProPublica that they’ve repeatedly gone to fuel up their ambulances only to come away empty at the pump: Their hospital-supplied gas cards were rejected because Prospect hadn’t paid its bill. A similar penury afflicts medical supplies. “Say we need 4×4 sponges, dressing for a patient, IV fluids,” said Leslie Heygood, a veteran registered nurse at one of Prospect’s Pennsylvania hospitals, “we might not have it on the shelf because it’s on ‘credit hold’ because they haven’t paid their creditors.”

In March, Prospect’s New Jersey hospital made national headlines as the chief workplace of the first U.S. emergency room doctor to die of COVID-19. Before his death, the physician told a friend he’d become sick after being forced to reuse a single mask for four days. At a Prospect hospital in Rhode Island, a locked ward for elderly psychiatric patients had to be evacuated and sanitized after poor infection control spread COVID-19 to 19 of its 21 residents; six of them died. The virus sickened a half-dozen members of the hospital’s housekeeping staff, which had been given limited personal protective equipment. The head of the department died.

The litany goes on. Various Prospect facilities in California have had bedbugs in patient rooms, rampant water leaks from the ceilings and what one hospital manager acknowledged to a state inspector “looks like feces” on the wall. A company consultant in one of its Rhode Island hospitals discovered dirty, corroded and cracked surgical instruments in the operating room.

These aren’t mere anecdotes or anomalies. All but one of Prospect’s hospitals rank below average in the federal government’s annual quality-of-care assessments, with just one or two stars out of five, placing them in the bottom 17% of all U.S. hospitals. The concerns are dire enough that on 14 occasions since 2010, Prospect facilities have been deemed by government inspectors to pose “immediate jeopardy” to their patients, a situation the U.S. Department of Health and Human Services defines as having caused, or is likely to cause, “serious injury, harm, impairment or death.”

Prospect has a long history of breaking its word: It has closed hospitals it promised to preserve, failed to keep contractual commitments to invest millions in its facilities and paid its owners nine-figure dividends after saying it wouldn’t. Three lawsuits assert that Prospect committed Medicare fraud at one of its facilities. And ProPublica has learned of a multiyear scheme at a key Prospect operation that resulted in millions of dollars in improper claims being submitted to the government.

Leonard Green and Prospect, which have operated hand-in-glove throughout this period, both declined requests for interviews. (Near the end of the reporting for this article, Prospect’s CEO, Lee, spoke to ProPublica on the condition that he not be quoted.) Leonard Green and Prospect responded to ProPublica’s questions in written statements through Sitrick and Company, a crisis PR firm jointly retained on their behalf. They maintain that they’ve kept their commitments, abided by the law, provided good patient care and invested hundreds of millions of dollars, saving many failing hospitals and preserving thousands of jobs. “Prospect Medical Holdings is a healthcare system that provides compassionate, accessible, quality healthcare and physician services,” the statement asserted.

The question of whether profits and good medical care can coexist is not a new one in the United States. But that tension is particularly acute in the case of Leonard Green and Prospect, where private equity has extracted hefty profits from a business that acquires struggling hospitals and relies on Medicaid and other government programs to pay the bills for its impoverished patients.

“It’s such a brutal case of unabashed greed,” said Rosemary Batt, a professor at Cornell University’s School of Industrial and Labor Relations, who has studied private equity’s involvement in health care. “We’re talking here about safety-net hospitals that are serving the poor, the unemployed, disproportionately people of color. They’re just doing this immoral sucking out of resources. That is beyond the pale.”

A storage area in January at the radiology department of Prospect Medical’s hospital in Culver City, California.
(Photo courtesy of SEIU-UHW)

Prospect’s story is also a bleak omen for the future of America’s health care system — and a particularly telling one because the company is effectively on its second tour through the private equity system. The business model for private equity firms like Leonard Green involves stripping cash out of the organization, loading down operations with debt and reducing every conceivable expense. After that is accomplished, firms then usually resell the operation to another buyer within five years.

The saga of Leonard Green and Prospect embodies a broader trend. Starting around 2010, giant private equity firms like Cerberus Capital Management and Apollo Global Management rushed into the hospital business, buying up facilities and assembling chains. Their moves intensified a shift to for-profit ownership among the nation’s 5,200 general hospitals: from about 15% for-profit in 2000 to 25% for-profit in 2018, the most recent year for which data is available. The biggest corporations still own more hospitals than private equity firms: HCA, for example, owns about 180; Apollo claims 89 and Cerberus had 37 at its peak, before selling this year.

Almost as quickly as it rose, private equity firms’ ardor for hospitals has “substantially cooled” in the past few years, said Lisa Phillips, editor of, which tracks private equity health care deals. “I’ve seen the whole M&A market for hospitals dry up.”

Making quick profits from operating hospitals proved daunting. “There’s so many other places to put their money in health care that they can flip faster,” Phillips said. (The firms have lately turned their sights to outpatient clinics or staffing emergency rooms.) “Private equity really wants to see growth fast and get out,” Phillips said. “They’ve squeezed it as dry as they can.”

Those actions have made it hard for the firms to sell hospitals, according to Eileen Appelbaum, senior economist at the Center for Economic and Policy Research, who studies private equity. “They’re loaded with debt and anybody sensible is not prepared to buy them.”

Indeed, Leonard Green is now on its third attempt to sell Prospect. Other firms, facing growing losses, have placed some hospitals into bankruptcy and closed others, offering up their real estate while seeking to sell the rest of their medical operations at bargain-basement prices.

The exodus isn’t necessarily good news, according to Batt and other experts. As they see it, this is merely the latest stage in a slow descent to the bottom. Given the cash and assets that private equity owners have already taken out of hospitals, their new owners will be left with heavy debt and limited resources — as the saga of Leonard Green and Prospect demonstrates. Faced with that financial plight, these hospitals will be compelled to cut costs even further, making it ever harder to deliver quality care.

Just over two decades ago, Sam Lee and the private equity firm where he then worked were among the first such firms to invest in hospitals — and it started almost by accident. In 1998, most private equity firms avoided health care. The industry was complicated and highly regulated, both anathema to private equity. Kline Hawkes, the young Los Angeles firm where Lee worked, had made only one previous health care investment, in a medical instruments company. The founder of Kline Hawkes was an investor named Frank Kline, who told ProPublica that no person named Hawkes was ever involved with the firm. Kline picked a British-sounding name to add a dash of gravitas.

In 1998, Kline Hawkes was approached by David Topper, a veteran hospital marketing executive who was seeking funding to buy eight struggling little hospitals in the LA area (one would be immediately sold) and assemble them into a company called Alta Healthcare Systems. Then 49, Topper started Alta after recovering from a fire at his home that left him on a respirator, with third-degree burns over 70% of his body.

Kline was skeptical. He relented only after a bit of salesmanship: Topper surprised him by turning up at a dinner meeting with 15 doctors who promised to send patients to Alta’s hospitals. It convinced Kline that Topper could deliver growing revenues. He decided to invest, putting up $3 million in equity toward the $34 million purchase price. Alta borrowed the rest.

Kline assigned Lee, then 32, to oversee the investment. Lee’s experience was in finance, not medicine. Born in South Korea and raised in Tampa, Florida, he was an industrial engineering graduate of Georgia Tech. Lee had worked for Andersen Consulting (“as a grunt,” he later explained in a deposition) and a Florida software company before getting an MBA at Harvard and joining Kline Hawkes.

Lee was whip smart, could be charming when he wanted to and preferred to operate behind the scenes. He hasn’t been quoted in the press in more than 20 years. (In recent years, his wife’s Facebook account has shown him celebrating holidays with her and their three college-age sons; family vacations in Maui, Aspen and Las Vegas; and pilgrimages to the Super Bowl and the American Music Awards.)

Sam Lee in 2014.
(via Facebook)

Lee became “super-involved” in overseeing Alta even as Kline Hawkes quickly found an exit, according to Kline. Just three years after investing $3 million, the firm cashed out in 2001 with $5.3 million, a 73% profit. Already deeply indebted, Alta had to borrow more to pay the $5.3 million.

For his part, Lee decided to stay. He sensed a major opportunity, according to a corporate history his spokesman provided: “While the cost of healthcare was growing at three times the rate of the US GDP, hospitals as a group were inefficient in delivering quality care.” Lee quit the private equity firm in 2000 and joined Alta full time, becoming its co-president and a 50-50 partner with Topper. Lee became the primary decision-maker. Topper’s main role was to be Alta’s salesman, schmoozing doctors and nursing home administrators to feed Medicaid and Medicare patients into their small community hospitals, located in low-income neighborhoods.

From the beginning, Lee and Topper brought the cost-slashing philosophy of private equity firms to Alta and its hospitals, according to interviews with former executives and multiple lawsuits. The effects were felt almost immediately.

Critical medical equipment and supplies, including drugs and tracheotomy kits, were “routinely unavailable” at Alta’s hospitals because bills hadn’t been paid, according to a breach of contract suit later filed by a former Alta chief operating officer named Michael White. According to the suit, the company regularly “changed vendors to avoid payment” and “bounced checks as part of its regular cash management process.” (White’s suit was later settled.) The portrait offered by White was affirmed by other executives, including Paul Smith, a former vice president for finance at Alta, who told ProPublica he recalled “having to switch vendors sometimes because we would get cut off.” Emergency room staff in at least one Alta hospital lacked chemical reagents needed to perform critical enzyme tests on heart attack patients, according to another former Alta executive who sued the company. Employees sometimes had to spend their own money to buy toilet paper for patients.

The stringent penny-pinching wasn’t enough to generate profits at first. Some of Alta’s hospitals, according to company filings with the California health department, were averaging 30% occupancy. According to White’s lawsuit, Alta lost a cumulative total of $35 million through the end of 2002. In April 2003, Lee and Topper abruptly shut down two of their hospitals, placing them into bankruptcy (and eventually liquidation), while selling a third. Lee disputed some of White’s claims, but acknowledged in the company’s written responses that this was “a difficult time,” resulting in “some bounced checks and some payables being missed.” He insisted that “ultimately, all the vendors were paid.”

Shedding those money-hemorrhaging operations helped Alta turn a financial corner. By cutting costs and maximizing government reimbursements at its remaining facilities, Alta started to eke out profits from its four remaining hospitals. “Their model was really about just bare minimum,” said Mike Heather, who later helped Prospect acquire Alta and served as Prospect’s CFO from 2004 through 2013. Alta’s facilities “were sort of war-zone hospitals. They were very, very dirt cheap in every respect.”

Things began looking up for the business. Occupancy climbed, and individual hospitals began reporting growing profits — though perhaps not as much as Alta’s financial reports suggested. “When you looked on paper, it was a beautiful turnaround,” said Jack Lahidjani, who was Alta’s CFO from 2003 to 2006. The reality, he said, was that Lee was “putting out aggressive financial statements.”

Lee “fought tooth and nail” to hike Alta’s reported profits in 2006 by booking inflated estimates for forthcoming Medicaid revenues, according to Michael Bogert, who prepared Alta’s audited financial statements for Moss Adams LLP, the company’s accounting firm. “He had our partners convinced I was being too conservative,” said Bogert, now executive vice president for corporate finance at Prime Healthcare, a Prospect rival. Lee convinced a Moss Adams senior partner to overrule him — something that had never happened, Bogert said, during more than 300 previous hospital audits. (Moss Adams declined to comment.)

The rosy numbers helped attract a buyer for Alta in 2007: Prospect Medical Holdings, a small, publicly traded company that managed 10 physician groups. The deal paid off Alta’s debt and netted Lee and Topper $50 million each in cash and Prospect stock. Those shares were enough to give Lee and Topper control of Prospect. Their ambitions were only growing.

The merger nearly wrecked Prospect. Just weeks after the deal closed, Prospect’s audit firm, Ernst & Young, discovered inflated revenues and profits on Alta’s books. (The E&Y senior manager assigned to examine Alta’s financials told ProPublica the misstatement was “very easy” to find.) As a result, Prospect was unable to complete its Securities and Exchange Commission filings, forced to cancel its annual shareholder meeting, delisted from the American Stock Exchange and defaulted on its loans, triggering millions in lender penalties. In April 2008, Alta restated its 2006 revenues, lowering them by about $4 million. In the restatement, filed with the SEC, Moss Adams explained that Alta had misused and ignored “factual information that existed” at the time it compiled the inflated financial statements. (Prospect told the SEC the company’s investigation had found no “intentional wrongdoing.” Lee, in his statement to ProPublica, dismissed the significance of the Alta restatement and said the bigger problem at the time was that Prospect was in far worse shape than he’d been led to believe.)

Despite the turmoil, Lee became CEO of Prospect and consolidated power. He acquired a moldering flagship hospital, the 420-bed Brotman Medical Center, in Culver City, California, out of bankruptcy; replaced Ernst & Young; fired and sued the company’s outside law firm; and ousted Prospect’s 74-year-old founder, Dr. Jacob Terner. A year later, Lee halted payment on Terner’s exit package. (Terner, who has since died, sued and won the full $1 million he was due, plus legal fees, in court.)

Mold broke through a wall this year near a nursing station at Prospect’s hospital in Culver City.
(Courtesy of SEIU-UHW)

Michael Terner worked as an executive vice president at Prospect for five years and departed around the same time as his father. He says his dad covered for Lee after the merger by soft-pedaling Alta’s accounting problems only to have Lee turn on him. “You’ll find,” Terner said, “if you go through the history of Sam Lee, there’s a lot of corpses.”

Indeed, the trail of litigation, unpaid bills and accusations was already lengthy. Two former senior executives at Alta claimed that Lee and his longtime partner had cheated them out of a promised equity stake. Minority investors in Brotman accused Lee of cooking its books to defraud them. Dozens of lenders, executives, doctors, staffing agencies and hospital vendors filed lawsuits and court claims over unpaid debts and broken agreements. Three law firms hired by Alta later sued for unpaid bills. Lee professed his innocence and fought the actions, typically settling for discounted amounts. The pattern would continue at Prospect.

Lee was demanding and unrelenting, according to people who worked for him. “One day you’re like a superstar and the future of the company,” said Steve Aleman, who became Prospect’s CFO in 2013. “The next day you’re absolutely in the doghouse.”

(Last fall, Lee abruptly terminated Aleman, who then filed suit claiming he is owed for unpaid compensation and canceled company stock options. Aleman is now CFO of Prime Healthcare. In its responses for this story, Prospect made an array of unsubstantiated allegations about Aleman’s workplace conduct during his 12 years at the company. A Prospect lawyer also wrote Aleman, accusing him of making “false and defamatory” statements to ProPublica. In a letter responding to the company, Aleman’s lawyer denied that his client made any defamatory statements. Aleman confirmed to ProPublica the accuracy of his comments in this article. In addition, Prospect made accusations about the conduct or character of seven other former executives and employees critical of the company, including two other former CFOs of Prospect or Alta. Aleman called the charges an “offensive smear campaign that Prospect is attempting against myself and others who are no more than victims of Lee’s broader plan to enrich a few and hurt many.”)

Lee churned through executives and could turn brutal, screaming at subordinates or grilling them over a tiny issue. “He’d go through three hours of literally just peeling the skin off somebody,” Aleman said. Lee would make executives cry, recalled former Alta CFO Lahidjani, who counted himself in that category.

Meanwhile, the CEO whittled costs to the bone by finding cheap sources for medical supplies; through “real-time” monitoring of hospital staffing; and slow-walking every vendor payment. “He was very proud of making it impossible to get a dollar out,” former CFO Heather said. “He would just not pay people as a way to negotiate. He would shut off things you’d say it was crazy to shut off.”

Through its spokesman, Prospect said “we do not have a slow-pay policy at Sam Lee’s or anyone else’s direction.” It said the company’s implementation of a new financial system over the past 12 months has caused a number of vendor payment delays and “credit holds.”

Prospect was far less obsessive about patient care issues, according to former company executives. “That quality component was always lax in my GFN,” one said. “It’s always the bottom line.” In public testimony a few years ago, Prospect executives acknowledged the point. “As an organization, we had delegated the role of the quality program to a local level,” Senior Vice President Von Crockett testified, “without the proper oversight at a corporate level.”

By 2010, the investing trends had changed. Big private equity firms were flooding into the hospital business. Leonard Green, a firm known for its investments in marquee consumer brands like Whole Food Markets and Neiman Marcus, joined the rush.

Prospect’s business, which involved spending as little as possible and squeezing profits out of Medicare and Medicaid reimbursements, while using Prospect’s physician groups to generate patients, didn’t fit the pattern. But Leonard Green viewed Prospect’s approach as one that could be applied widely and used to acquire more hospitals and reap more profit. Lee was eager to expand, too, confident that his business model could be applied to many more struggling hospitals, multiplying the company’s revenues from about $470 million in 2010 to several billion.

Leonard Green struck a deal that aligned Lee’s financial interests with its own. In addition to more than $2 million a year in salary and bonus, he would get 20.2% of Prospect’s shares (and dividends). Topper received a 14.9% stake, while Leonard Green got 61.3%. The rest was distributed in the form of stock options to Prospect’s top executives, to whom Lee dangled the possibility of a big future payoff.

Leonard Green’s point man for the Prospect stake was a former investment banker named John Baumer, a graduate of Wharton and Notre Dame, where his father had worked as the university’s comptroller. At Notre Dame, Baumer and his wife have endowed the lacrosse team’s head coaching position ($3 million) and funded a new men’s dormitory ($20 million). The Baumers live about 30 minutes from the firm’s Santa Monica offices in a large oceanfront property on Manhattan Beach, purchased, through a corporate entity he set up, for $18.4 million.

Baumer and two Leonard Green colleagues, who together made up a majority of Prospect’s five-member board, left day-to-day health care operations to Lee. The private equity board members focused on profits — and wasted little time in beginning to reap returns.

In 2012, Prospect paid Leonard Green and its investors a total of $188 million in two rounds of dividends. Prospect raised the money by issuing junk bonds. Only two years in, the private equity fund had made back most of its $205 million investment.

As Prospect cranked up its ambitious expansion plans, it consistently told the targets of its acquisitions and the government regulators who needed to approve them that it was in the business of saving troubled hospitals. “We haven’t closed hospitals, and we don’t close services,” Dr. Mitchell Lew, Prospect’s president, said at a Connecticut public hearing in March 2016. “We’re in this for the long term, OK?”

Lee’s first out-of-state acquisition would erase that claim. In 2012, Prospect paid $48 million for San Antonio’s Nix Health System. Nix included a 208-bed downtown hospital, an inpatient psychiatric center and multiple outpatient clinics.

Nix was an unusual acquisition for Prospect. It was profitable and had a higher federal quality rating, with four out of five stars, than any other Prospect hospital. Yet Prospect claimed the role of savior. In a press release announcing the deal, Lee said the company “will help ensure the long-term success of Nix.”

That success didn’t last long. Prospect removed Nix’s longtime CEO in 2015 and established control from headquarters in LA, while cycling through four more CEOs in the next four years. Doctors who had long relationships with Nix stopped referring patients. After decades of profits, Nix began losing money.

In 2019, after repeatedly promising to keep at least part of the system open, Prospect shut it all, laying off nearly 1,000 employees. The company sold Nix’s downtown building to a hotel chain and exited with a big loss. “It was mismanaged at the corporate level,” Aleman said. “It went from making about $20 million to losing money. It was an absolute disaster.”

In its responses to ProPublica, Prospect blamed the failure on a “catastrophic” broken water pipe in 2016 that flooded “the entire hospital infrastructure,” forcing doctors and patients to go elsewhere for months. “Volume and physicians,” the company said, “never returned to pre-flood levels.”

Prospect is now poised to shutter another acquisition it eagerly pursued: East Orange General, outside Newark, New Jersey. In late 2015, Prospect outbid two other companies with a $44 million offer for the 196-bed hospital, then in bankruptcy and losing more than $2 million a month.

Prospect vowed to spend $52 million on capital improvements and keep the hospital open for “no less than five years.” Three years into that vow, with losses still running about $1 million a month, Prospect’s warnings that it wanted to sell or close the hospital spurred state lawmakers to hand the company an “emergency” $15 million grant.

Lee couldn’t find a buyer, Aleman said: “They would have just given East Orange away — literally handed over the keys. They wanted to get rid of it at all costs.” In its statement, Prospect said it will keep East Orange open into 2021 while it continues to seek a buyer and thus “will surpass our five-year commitment of operating the hospital.” The company also said it has met its $52 million capital-spending promise under provisions of its purchase agreement that allow it to count debt payments and routine maintenance costs toward that total.

In Rhode Island, Prospect was welcomed as a savior in 2013 when it agreed to pay $45 million for controlling ownership of two money-losing Providence-area hospitals: 220-bed Roger Williams and 359-bed Our Lady of Fatima. Eager to save jobs, Fatima’s powerful United Nurses & Allied Professionals union endorsed Prospect’s bid.

State regulators, who had to approve the sale, had two big concerns. The first was the $188 million in dividend payouts previously made to Leonard Green and other investors. Those payments raised fears that Prospect wouldn’t fulfill its pledge to spend $90 million on capital improvements over four years. No problem, Prospect responded; it wouldn’t pay out any more dividends. “Prospect’s management and representatives have given assurances that this was a one-time event and that there are no plans to make a similar distribution in the foreseeable future,” the Rhode Island attorney general noted in his written findings on the hospitals’ sale in 2014.

Employee pensions was the other issue. The retirement plan for Our Lady of Fatima, which 2,700 past and current hospital employees were counting on, had been woefully underfunded since 2008. The problem had escaped federal ERISA oversight because of the hospital’s affiliation with the Catholic Diocese of Providence, making its pension system a legally exempt “church plan.”

The size of the problem was a secret. During negotiations over the sale, Prospect was repeatedly briefed on actuarial studies showing that even after a $14 million contribution that Prospect agreed to make, the plan would run out of money by 2036, while still owing about $98 million in retirement benefits.

After learning this, Prospect negotiated contract language freeing it from any future pension liability. The retirement system, and its massive funding deficit, would become the responsibility of a nonprofit community board, which had no reliable source of income.

Prospect officials never disclosed the plan’s dire straits during the state approval process. Instead, retirees nervous about Prospect’s purchase were shown a PowerPoint presentation stating that Prospect’s one-time contribution would “stabilize plan assets.” Lee attested in writing that the payment would “assure that the pensions and retirement of many former employees, who reside in the community, are protected.” Prospect told the attorney general that any necessary future payments would “be made based on recommended annual contribution amounts as provided by the Plan’s actuarial advisors.” Remarkably, no one addressed who would actually make such payments. Rhode Island approved the purchase in 2014.

Over the three years that followed, neither Prospect nor anyone else paid a penny into the pension plan. In 2017, the system was declared insolvent and placed in receivership. The court-appointed receiver has filed multiple lawsuits accusing Prospect and the diocese of “omissions and half-truths actionable as fraud,” demanding that they help make the pension whole. The cases are all pending.

In its statement, Prospect noted that its purchase agreements for the hospitals “clearly spell out” that the company had “no responsibility” for funding the pension plan. It also said it would have been “economically impossible” for Prospect to take over liability for the retirement system and called the receiver’s allegations about the company’s actions “false and unsubstantiated.” Both Prospect and the diocese deny concealing the pension system’s condition.

Meanwhile Prospect sought to cut costs by reducing the workforce, trimming benefits and tightly monitoring each hospital’s patient count throughout the day from its LA headquarters, sending nurses and aides home whenever possible in mid-shift.

After hearing about a consultant’s 2017 report describing dirty and damaged operating room instruments, the union at Fatima requested documents about this and other problems revealed by various inspections. Prospect refused, and failed to turn over any of the materials, despite an order to do so from the National Labor Relations Board in April 2019, affirmed by the 1st U.S. Circuit Court of Appeals in March 2020.

Prospect asserts that it promptly addressed the consultant’s concerns about dirty and damaged surgical instruments, but that it viewed the report as “proprietary” and thus “availed itself of the court system.” The company added: “As we recently received a ruling from the Federal Court to produce the document, we have complied with the order.”

Prospect has yet to hand over any documents, according to the union. “Prospect is lying in claiming that they’ve complied with the order,” union general counsel Chris Callaci said. Dealing with the company, he added, has been “a parade of horribles.”

Many of these problems had yet to emerge by 2015, as Prospect struck rapid-fire deals to double the company’s size. That’s when it reached agreements to spend more than $500 million to buy hospital systems in three states: East Orange General, in New Jersey; three community hospitals in Connecticut; and a four-hospital system in suburban Delaware County, Pennsylvania, west of Philadelphia. Prospect promised to spend hundreds of millions more on pension and capital improvements.

Prospect was reporting revenues of about $1 billion in 2015, with operating profits of $108 million. After digesting the acquisitions in the pipeline, the company projected, revenues and profits would surely soar.

Leonard Green was now ready to fully cash in and exit its investment. In October 2015, the firm hired Morgan Stanley to find a new private equity buyer for Prospect. The company’s 92-page “confidential information memorandum,” prepared for prospective acquirers and obtained by ProPublica, promoted the company’s “cost-effective care” model, including daily “flex” management of hospital staffing, use of low-cost sources for medical supplies and a focus on high-profit programs for treating the seriously mentally ill.

Bain Capital and CVC Capital Partners were the two final bidders. Both made offers around $1.2 billion, according to sources familiar with the talks. Then, in early 2016, U.S. capital markets tightened amid fears of a recession, dashing the company’s hopes to get even more. Lee decided to hold off on a sale. Aleman said the discussed reasoning was that Prospect could bring a far richer price after mining its pending acquisitions for bigger profits.

But a new problem had emerged behind the scenes during this period: improper Medicare billing. The issue, described in internal documents obtained by ProPublica and interviews, involved “unsupported” reimbursement codes submitted by Prospect’s physician-management business, whose dramatically increased profits the company had promoted to potential buyers. The problem was discovered in August 2015 during a routine compliance audit by nurses with Inter Valley Health Plan, a California HMO that sent Medicare Advantage patients to Prospect doctors and, as a result, had shared in the improper windfall (unknowingly, according to Inter Valley).

Inter Valley promptly notified Prospect, which expressed skepticism that anything was wrong, according to Inter Valley chief operating officer Susan Tenorio. “They really didn’t take us seriously,” she said. “The response was: ‘We do this all the time. Nobody has questioned it.’ That’s when I went back to our CEO and said, ‘There’s a problem here.’”

Inter Valley began investigating, with help from a law firm and outside consultant. It found that Prospect had submitted an estimated $22.6 million in potentially improper charges, which the federal government had already paid. Several million dollars more in improper claims, not yet processed, had to be canceled, according to Inter Valley. Inter Valley’s CEO and its chief compliance officer then sent a letter detailing their findings to the Centers for Medicare and Medicaid Services in August 2016.

The letter, obtained by ProPublica, reported that Prospect had submitted thousands of claims dating back to 2013 that were “not supported by audited medical charts.” It added: “In many instances, diagnosis codes were submitted for dates of service for which there was no evidence in a medical chart confirming that the [Prospect physician] had a face-to-face visit with the beneficiary.” Most of this “upcoding” involved claims that individual patients had made two visits to Prospect doctors on the same day. “We reported everything,” Inter Valley CEO Mike Nelson said. Everyone on Inter Valley’s board, he added, accepted that its organization had been reimbursed for false charges. “Making it right is what we should do,” Nelson said.

Prospect, Inter Valley and a hospital used by the plan’s patients had to repay the federal government for the improper income they’d received. Nelson said the three parties set aside a combined $22 million to cover the reimbursements while CMS completed its still-unfinished audit of how much is due. (CMS did not respond to requests for comment.)

Prospect’s own consultant, Alvarez & Marsal, largely confirmed Inter Valley’s findings in September 2016 in a confidential draft document reviewed by ProPublica. Alvarez & Marsal was also concerned the problem extended far beyond what Inter Valley had discovered: that Prospect had submitted bogus claims for more than 20 other Medicare Advantage plans, including United Healthcare and Blue Shield.

Another of the Medicare Advantage plans that received payments because of Prospect’s improper claims, CalOptima, said in a statement that Prospect first informed it in March 2016 of an “inadvertent and isolated” billing error from a single month in 2015. Months later, Prospect acknowledged the problem was far more widespread. It eventually turned out there were 3,847 “erroneous” claims over four years, requiring $2.8 million in repayments to CMS, including $1.7 million from Prospect. Because the improper claims were eventually self-reported, the government has taken no action against Prospect.

The ultimate total cost to Prospect from the improper billing episode, including expected income the company lost as a result, was in the tens of millions, Aleman estimated.

Prospect asserted that its cost was actually $8.5 million and that “management was unaware” of any inappropriate billing until after the fact. The company blamed the episode on the vice president who had presided over all reimbursement submissions, who was fired. In an interview, the woman, who asked not to be identified, told ProPublica, “They blamed me for something I didn’t do.” Inter Valley’s Tenorio called her “a scapegoat.”

Meanwhile, three lawsuits have charged Prospect with different allegations of billing fraud at its flagship hospital in Culver City. According to a pending suit filed by Charles Harper, a 28-year employee who served as director of cardiopulmonary therapy, the hospital fraudulently billed Medicare for individual respiratory therapy while regularly requiring its staff to treat two patients at the same time, a practice known as “stacking.” Harper claims he was fired for complaining about the wrongdoing. (Prospect denies any improper billing and says Harper’s job was eliminated because of diminished demand for respiratory services.)

A second lawsuit filed in federal court claimed the hospital inflated Medicaid revenues at its Miracles detox center by admitting financially needy or homeless patients with “no medical reason for being hospitalized for chemical dependency.” The plaintiff, a former nurse there who sought whistleblower status for the suit, alleged that some patients were admitted so often, without undergoing standard addiction screenings, that the staff referred to them as “frequent fliers.” Federal prosecutors ultimately declined to join the case but allowed it to proceed as a private action against Prospect and the hospital under the False Claims Act. Prospect settled in 2017, agreeing to pay $275,000 while asserting that the claims were “wholly without merit.”

The third suit alleged an “illegal patient procurement scheme” to generate fraudulent Medicare and Medicaid claims. Christina DeMauro, an emergency room nurse at Culver City for six years, asserted that a special team of hospital “marketers” generated a stream of about 20 elderly patients a day, most suffering from chronic dementia, who were admitted through the ER despite having no problems that required hospitalization.

According to her suit, these patients were brought from nursing homes and other senior facilities, “many well over 100 miles away,” when their Medicare benefits there, capped at 100 days, were about to expire. After an unnecessary hospital admission requalified them for Medicare benefits, the patients were then returned to their facilities, according to the suit, boosting government billings for both Prospect and the senior facilities. Filed in 2018, the case remains pending in Los Angeles. DeMauro alleges that “unlawful retaliatory conduct” she faced after complaints about these practices forced her to resign.

Don Andrews, a seasoned administrator who worked as emergency department director during part of this period, backed these claims in an interview with ProPublica. Andrews said Prospect marketers insisted that elderly mental health patients “from nowhere near Culver City” be admitted through the emergency room even when no psychiatric beds were available in the hospital. He says this routinely resulted in a handful of patients being held for days in a crowded ER “overflow” area with no beds or privacy — just chairs and a single bathroom — serving as a sort of “bootleg inpatient psychiatric unit.” A few years before Andrews got there, one 79-year-old man suffering from dementia disappeared after being left unattended in the overflow area, according to a state inspection report and a lawsuit by his family. His body was later found on a beach 7 miles away; the man had drowned. The “overflow” area remained in use until about 2018, when it was permanently locked, hospital employees said.

Prospect’s spokesman denied DeMauro’s allegations but declined to address specifics because her litigation is pending.

This same hospital, the company’s largest, is also the most visible monument to Prospect’s neglect. Long called Brotman Medical Center, it is best known for its burn center, which treated Michael Jackson in 1984 after his hair and jacket caught fire during the filming of a Pepsi commercial.

In 2013, four years after buying the hospital, Prospect grouped Brotman with two of its other hospitals, renaming it Southern California Hospital at Culver City. The move, made to qualify for extra government subsidies for treating low-income patients, helped Brotman generate profits.

But Brotman has continued to deteriorate. In 2015, inspectors shut down all elective surgery at the hospital for eight days, citing a “widespread pattern” of poor infection control and sterility; the problems resulted from inadequate heating and cooling systems. That episode, as well as the death of the ER-overflow patient whose body was found on the beach, resulted in immediate jeopardy findings.

That same year, state health inspectors cited the hospital after a broken refrigeration system in its morgue caused a woman’s corpse to decompose so badly it produced a “noticeable stench,” making it impossible for her family to have an open-casket funeral. Meanwhile, one of the hospital’s elevators has been out of order for 10 months. Patients needing MRI scans must be taken outdoors and down an alley, past dumpsters and into a hospital parking lot, where the scan is done in a rented trailer.

Prospect said it quickly resolved all immediate jeopardy findings, something a hospital is required to do to remain eligible for federal reimbursements. It said it is “working with state and local officials to expedite” the broken elevator’s replacement. And it said it is “not uncommon for hospitals to utilize a mobile MRI,” but plans are underway to relocate the MRI inside a nearby building.

When it rains in Culver City, water drips from ceilings throughout the hospital’s two buildings, forcing staff to relocate patients and plant orange buckets in the hallways. In 2014, a patient’s wife filed suit after soaked ceiling tiles fell and struck her in the head while she was sitting in the hospital lobby. This January, a giant brown mold formation burst through the wall near a fourth-floor nurses’ station. Noted the resulting complaint to the California health department: “There are mushrooms growing out of the wall (which they cut off and patched back up). There is leakage from the ceiling when it rains you can taste the mold in the air.”

A water leak at Prospect’s hospital in Culver City.
(Courtesy of SEIU-UHW)

Employees told ProPublica the problem has persisted for years and provided photographs and videos documenting numerous leaks as well as the mold growth. Prospect asserted, by contrast, that “all leaks are identified and fixed as they occur.” The company said roof replacement has begun on the main patient building.

A 2018 state inspection found the pharmacy staff at the Culver City hospital had for months ignored findings of “fungal air growth,” “bacterial organisms” and mold in equipment used to mix patient medications in a sterile environment. According to the report, this resulted in the dispensing of about 21,000 doses of “adulterated dangerous drugs” to patients over a nine-month period. In September 2019, California’s attorney general formally charged Prospect executives, including Lee, the hospital and its supervising pharmacists, with “gross negligence,” initiating proceedings to revoke or suspend the hospital’s pharmacy permit. The matter remains pending. Prospect asserted that “no patient harm occurred” from the “error,” which has been corrected, and said the pharmacy is now “fully operational.”

Eventually, word of Prospect’s practices spread, causing alarm when the company sought to acquire new hospitals in other states. As Connecticut in 2016 weighed whether to approve Prospect’s purchase of three hospitals, the state sent a team to California to investigate five recent immediate jeopardy findings, which had placed one Prospect hospital license on a “termination track” for cutoff of Medicare and Medicaid funding.

Prospect executives tap danced, alternately denying problems and explaining away the repeated findings of imminent threat. SVP Crockett testified, for example, that “there was no specific patient harm” that occurred, while insisting Prospect acted aggressively to address the “allegations,” including by creating new posts for a “chief quality officer” and a “vice president of regulatory affairs and patient safety.”

The company claimed it would act differently in the new states it was entering. “What happened in California certainly is concerning,” Prospect’s president, Lew, acknowledged at the hearing. “… And so, we’re not bringing California’s quality program to Connecticut,” he said. “If you want to look at us as performing an ‘F’ on the test in California, that student is not coming here to tutor Connecticut on quality, OK?”

As it turns out, Prospect hasn’t earned stellar grades in Connecticut either. State officials approved the company’s acquisitions on a conditional basis in 2016, while imposing a three-year monitoring regime that health department officials describe as unprecedented. Before the monitoring period expired, two of Prospect’s newly acquired Connecticut hospitals were slapped with immediate jeopardy findings.

This time, two patient deaths triggered the jeopardy findings. In 2018, Manchester Memorial Hospital mishandled two high-risk pregnancies: One woman died after delivering a stillborn baby; a second gave birth to an infant with severe encephalopathy, a form of brain damage, after an emergency cesarean section was performed too late. Waterbury Hospital was found to have failed to properly monitor two suicidal patients on a single day in March 2019. In one case, staff returned a belt to an “actively suicidal” psychiatric patient who then used it to hang himself in his hospital bathroom. After his death, the hospital failed to notify police. A second patient attempted suicide by tying hospital socks around his neck after being left unwatched while a nurse went to lunch.

The Joint Commission on Hospital Accreditation responded by initially denying Waterbury’s accreditation, required to receive Medicare and Medicaid funding, after an inspection that found 42 quality standards “out of compliance.” In December 2019, Connecticut regulators extended the monitoring of the state’s three Prospect hospitals until May 2021.

More failures appeared in the company’s biggest purchase yet, agreed to in late 2015: the four-hospital Crozer-Keystone system in Pennsylvania. Prospect paid $300 million. It made other promises as part of the deal: to spend an additional $200 million in capital improvements within five years; to keep all the hospitals open for a decade; to fund $171 million in pension benefits within five years; and to endow a community health care foundation for $53 million.

Almost immediately, Prospect began contesting the agreement. Always eager to delay and reduce a big outlay, Prospect deferred $21.5 million of the foundation funding for 90 days — and then refused to make the payment altogether, challenging how much it owed.

The foundation sued, eventually extracting Prospect’s agreement to submit the matter to arbitration while putting the money into escrow. When Prospect then missed the escrow deadline, the foundation began garnishing the company’s accounts and sought to have a receiver appointed over all its financial transactions. Prospect finally paid, 18 months late, after the arbitrator awarded the foundation $23.7 million, including interest. (Prospect’s spokesman said the matter was “referred to the court” because efforts to resolve the amount of the payment were unsuccessful.)

At Crozer-Keystone, as elsewhere, Prospect has aggressively moved to lower costs. It sought, unsuccessfully, to reduce nurses’ accrued vacation time and to cut pension benefits for all employees who didn’t work full time. The company has also waged a four-year battle to halve the tax assessments on all its hospital properties. (Prospect says it believes the assessments are excessive and will pay “once a final ruling is given as to what is fair and proper.”)

In November 2018 came yet another immediate jeopardy finding. This one stemmed from patient-safety violations in a mental health ward at 300-bed Crozer-Chester Medical Center, the system’s largest hospital. According to state health department inspectors, video monitors at a nurses’ station for maintaining watch over suicidal patients were turned off or ignored; an activity room was left unattended as psychiatric patients milled about; patients were placed in restraints or in seclusion without proper documentation; and facilities in the locked unit treating elderly psychiatric patients, some of them suicidal, presented multiple hanging hazards.

Hospital workers have regularly reported staffing shortages, sometimes forcing delays of scheduled medical procedures. Two medical employees at Delaware County Memorial Hospital are lead plaintiffs in a national class action against Prospect, claiming insufficient staffing regularly forces hospital employees to work, unpaid, through meal breaks. The company denies the allegations, including that any of its hospitals suffer from staffing shortages.

As elsewhere, Prospect’s failure to pay bills on time has delayed repairs and resulted in supply shortages. At Delaware County Hospital, veteran nurse Angela Neopolitano said a call-bell system in one unit, which patients use to summon help from nurses, has been broken for more than two years. “Creditors would not come in to fix things because the hospital owed them money,” she said. “Then we suffer and the patients suffer.”

Paramedics have repeatedly gone to fuel up ambulances using a hospital credit card, only to have it rejected, according to Larry Worrilow, assistant chief for the Crozer-Keystone EMS system. “It’ll be fine for six or eight months. And then, all of a sudden, boom — you can’t get fuel,” said Worrilow, who has worked there since 1977. “After you rattle their chains, they pay part of the bill, get their credit hold lifted, and you can get fuel.” (Prospect said the card was rejected because it placed a charge limit on it as a security measure, and “when it was brought to our attention that the account was reaching the credit limit frequently, we increased the credit limit to ensure there was not disruption of services.”)

The system’s eight ambulances are so old — two have more than 275,000 miles on them — that they frequently break down, according to Worrilow. “There’s plenty of times when we went to go on an emergency call and the ambulance wouldn’t start,” he said. “You have to send the next closest ambulance. Or you get to the scene and the ambulance won’t run.”

COVID-19 caught many of America’s top medical centers by surprise. But Prospect’s penchant for scrimping on staff and medical supplies left its hospitals with little margin for error.

In Rhode Island, for a time in March, hospital employees at Our Lady of Fatima were threatened with discipline for wearing their own masks, even though the hospital didn’t have enough to give them. Nursing assistant Doreena Duphily, who worked in the geriatric mental-health ward, where 19 of 21 patients were infected, was out sick for three weeks with COVID-19 herself. Duphily blames the hospital’s frequent rotation of its limited staff to different floors for spreading infection. Six members of the environmental services staff, responsible for cleaning patient floors, also got sick. On May 1, department supervisor Jerald Ferreira, 63, died of COVID-19.

“We were probably about three weeks behind every other hospital in getting just the basics,” said Fatima RN Lynn Blais. “All of a sudden COVID comes in, everybody should have surgical masks, and we don’t have two days’ worth of surgical masks, much less two months of surgical masks. We were caught with our pants down. That germ was all over the floor.”

An employee in Culver City wearing plastic bags earlier during the pandemic because protective booties were not available.
(Courtesy of California Nurses Association)

In Culver City, nurses unable to get proper protective gear for a time donned plastic garbage bags. ER secretary Chudi Long says she became infected after being denied a mask despite working in close quarters with COVID-19 patients. After her breathing grew weak while she was battling the virus at home, Long was rushed to another hospital’s ER, where she lost two front teeth during an emergency intubation, and spent seven days on a ventilator.

Prospect denied it ever lacked PPE at any of its hospitals.

Leonard Green may not have been involved in Prospect’s day-to-day management, but it has popped up periodically to make sure it gets a return on its investment. In 2018, less than four years after assuring a state attorney general that it had no plans to seek new dividends, Prospect attempted to do just that. It began preparing to issue a $600 million dividend. As always, the plan envisioned funding that payment through debt.

Moody’s, the ratings agency, was dismayed by Prospect’s soaring debt. It lowered the company’s credit rating in response. As a result, Prospect reduced the dividend to $457 million. In a letter to Rhode Island officials, Prospect insisted it didn’t violate the pledge it made back in 2014 because “in 2014, no dividends were planned.”

That $457 million raised the total in dividends extracted from Prospect since Leonard Green acquired it to $645 million. Roughly $386 million had gone to Leonard Green’s investors and the firm (which gets 20% of all fund profits); $128 million to Lee; $94 million to Topper; and the remaining $37 million was divided among other Prospect executives. (Another $14 million in fees went to the private equity fund.)

Having collected that cash, Leonard Green made a second attempt to exit the investment in June 2018. By this point, Prospect had grown to 20 hospitals. Detailed management presentations to the two 2015 finalists were followed by dinners in Beverly Hills, leading to informal discussions with CVC, which was contemplating a considerably richer offer this time, according to Aleman. (CVC declined to comment.)

But once again, the sale collapsed. As Prospect headed toward the September close of its 2018 fiscal year, its business began deteriorating rapidly, torpedoing the projections it had given potential buyers. Recognizing that the bad numbers would surely blow up the deal, Leonard Green and Lee decided to hold off again. (The statement from Green and Lee denies they tried to sell the company in 2018.)

The situation grew dire. By January 2019, Prospect had so little cash that it needed an emergency $41 million loan from Leonard Green, Lee and Topper to assuage auditor fears that the company might not remain “a going concern” and to avoid violating loan covenants, according to Aleman. In March, Moody’s downgraded Prospect’s debt a notch deeper into junk territory, citing the company’s “very high financial leverage, shareholder-friendly financial policies, and a history of failing to meet projections.”

Eager to raise capital, Prospect sold its land and buildings last fall in a sale-leaseback transaction that allowed the operations to remain in the facilities. The company raised $1.55 billion. Prospect used much of the cash to pay off its loans. It had effectively replaced its debt payments with rent payments.

The sale of the land and buildings brought in much needed cash and stabilized the company. But it also meant that Prospect had shed by far its biggest asset, sharply reducing the value of the company. When Leonard Green made its third attempt to exit, the nominal price was a pittance.

In October, the private equity firm agreed to sell the firm’s 60% stake to Lee and Topper for $12 million in cash plus the assumption of $1.3 billion in lease obligations. The $12 million was to be paid by Prospect, not the two executives. As Prospect and Lee put it in their statement for this article, “In effect, the company’s money is their money.”

To Lee’s management team, who dreamed of stock option riches, it was an outrage. The low cash price would value their shares and options at a pittance, dashing their expectations of a windfall. A “drag-along” provision of the agreement would force all shareholders to sell immediately, rather than wait and hope for a better price. In February, Aleman, who’d been stripped of his stock options when he was suddenly fired last fall, filed suit in California, seeking restoration of his shares and payment of his 2018 bonus. (Under agreements Prospect makes virtually all employees sign, the case is scheduled to go to arbitration.)

For Leonard Green, the exit made a certain sense. As of this year, when the firm hopes to close the sale, Green has retained its Prospect stake for 10 years; indeed, the $5.3 billion fund that holds that and other investments was launched in 2007, making it venerable in private equity years. That fund has doubled in value overall, according to data on its investors’ websites. All told, for all investments in the fund over 13 years, ProPublica estimates Leonard Green has made more than $1.5 billion for itself from fees and its share of the fund’s profits. (Through its spokesman, Leonard Green said this figure was wrong and that the firm would “not respond to inaccurate guesses.”)

It is leaving a mess behind at Prospect. The company has little cash, weighty pension debts and lease commitments, and uncertain future earnings.

Some current and former management shareholders, working with Aleman, contemplated trying to recruit another buyer who would pay a far higher price. But when an email exploring this effort was accidentally sent to Prospect, the company responded by dispatching a letter to Aleman’s attorney, accusing the former CFO of “colluding with others in an attempt to interfere with a Company transaction.” It demanded that he “immediately cease and desist.”

Leonard Green’s sale to Lee and Topper requires approval from state officials in Rhode Island, since it involves hospitals there. The officials have postponed their decision until November, saying there are missing documents and unanswered questions. And opponents there are making a stand. The Private Equity Stakeholder Project, a union-backed research group, has produced detailed reports criticizing Green’s history with Prospect. It has lobbied public-pension investors and members of Congress to press the firm to return its dividends to the company, saying its profiteering has put Prospect’s safety-net hospitals at risk. The Fatima union and the pension fund’s receiver have opposed the sale too. “I don’t know the answer, but I think there’s something wicked going on here,” Max Wistow, the receiver’s special counsel, told a public hearing. Citing Leonard Green’s history with the hospital company, the Rhode Island state treasurer has said he will block any future investments by his state, which sunk $20 million into the fund that owns Prospect, in the private equity firm’s funds.

Leonard Green defended the transaction to state officials and a Rhode Island congressman, writing that the sale price reflects Prospect’s “future obligations” and was agreed to by “sophisticated investors” who wanted “to not burden the company with additional debt.” The firm added: “We reject any implication that we have managed Prospect in a financially irresponsible fashion or that we have put our own financial interests ahead of the interests of the hospital system. Prospect today is at no risk of financial failure.”

Given Sam Lee’s prowess at squeezing cash out of ailing institutions, Prospect undoubtedly will find profits left to extract. What it will have to offer patients is less clear.

Correction, Oct. 12, 2020: An earlier version of this article misspelled the name of Doreena Duphily.

Continue Reading


Building Kubernetes Native SaaS applications: iterating quickly by deploying in-cluster data planes

Mish Boyka



At Pixie, we are working on a Kubernetes native monitoring system which stores and processes the resulting data entirely within a user’s cluster. This is the first in a series of posts discussing techniques and best practices for effectively building Kubernetes native applications. In this post, we explore the trade-offs between using an air-gapped deployment that lives completely within a cluster and a system which splits the control and data planes between the cloud and cluster, respectively.

One benefit of building for the Kubernetes platform is that it simplifies the process of deploying applications to a user’s environment, often requiring only a few simple steps such as applying a set of YAMLs or installing a Helm Chart. Within minutes, users can easily have a running version of the application on their cluster. However, now that these applications are running entirely on prem, it becomes difficult for the developer to manage. In many cases, rolling out major updates or bug fixes relies on having the user manually update their deployment. This is unreliable for the developer and burdensome for the user.

Diagram of a connected on-prem architecture.

To address this problem, we propose a connected on-prem architecture which delegates the responsibility of managing the data and control planes of the application to the deployment running in the cluster and a developer-managed cloud environment, respectively. More concretely, the application deployed in the user’s cluster is solely responsible for collecting data and making that data accessible. Once the foundation of this data layer is established, the logic remains mostly stable and is infrequently updated. Meanwhile, a cloud-hosted system manages the core functionality and orchestration of the application. As the cloud is managed by the developer themselves, they are freely able to perform updates without any dependency on the users. This allows the developer to iterate quickly on the functionality of their system, all while maintaining data locality on prem.

This split-responsibility architecture is common in many hardware products, since external factors may make it challenging to deploy updates to software running on physical devices. For instance, despite these physical limitations, Ubiqiti’s UI is able to offer a rich feature-set by delegating functionality to their cloud and keeping their physical routers within the data plane. Similarly, WebRTC is a standard built into most modern browsers for handling voice and video data. Although browser updates are infrequent, having the separated data and control layers allows developers to freely build a diverse set of applications on top of WebRTC. This architecture is still relatively uncommon in enterprise software, but has been adopted by popular products such as Harness, Streamsets, and Anthos.

However, designing a connected on-prem architecture is easier said than done. When building such a system, one challenge you may encounter is how to query data from an application running on the user’s cluster via a UI hosted in the cloud. We explore two approaches for doing so:

  1. Making requests directly to the application in the cluster
  2. Proxying requests through the cloud

For brevity, we will refer to the application running on the user’s cluster as a satellite.

Approach 1: Making Requests Directly to the Application in the Cluster

The simplest approach for executing the query on a satellite is to have the UI make the request directly to the satellite itself. To do this, the UI must be able to get the (1) status and (2) address of the satellite from the cloud, so that it knows whether the satellite is available for querying and where it should make requests to.

Diagram of Non-Passthrough Mode where the UI makes requests directly to the satellite agent itself.

Step 1: Heartbeating

A common technique to track the status of a program is to establish a heartbeat sequence between the program (the satellite) and the monitoring system (the cloud). This is typically done by having the satellite first send a registration message to the cloud. During registration, the satellite either provides an identifier or is assigned an identifier via the cloud, which is used to identify the satellite in subsequent heartbeat messages.

Following registration, the satellite begins sending periodic heartbeats to the cloud to indicate it is alive and healthy. Additional information can be sent in these heartbeats. In our case, we also attach the satellite’s IP address. Alternatively, the IP address could have been sent during registration, if it is not subject to change. The cloud records the satellite’s status and address so that it can be queried by the UI.

Now, when the UI wants to make a request to a satellite, it first queries the cloud for the address, then directly makes the request to that address.

Great! That wasn’t too bad. In many cases, many cloud/distributed satellite architectures already communicate via heartbeats to track satellite state, so sending an additional address is no problem. However… If your UI is running on a browser and your satellite is responding over HTTPS (likely with self-signed certs), you are not done yet…

Step 2: Assigning Satellites a Domain Name

The browser is blocking our requests because of the satellite’s SSL certs! A user could go ahead and navigate directly to the satellite’s address, where the browser prompts the user with whether or not they want to bypass the invalid cert.

However, this would need to be done per satellite and is disruptive to the user’s overall experience. It is possible to generate SSL certs for IP addresses, but this is uncommon and isn’t available with most free Certificate Authorities. This approach is also complicated if the satellite’s IP address is subject to change.

Diagram of SSL certification flow for Non-Passthrough Mode.

To solve this problem, we used the following solution:

  1. Pre-generate SSL certs under a subdomain that you control, for instance: <uuid> This step is easy to do with any free Certificate Authority and can be safely done if the subdomain has a well-known DNS address. You should make sure to generate more SSL certs than the number of expected satellites.
  2. When an satellite registers with the cloud, it should be assigned an unused SSL cert and associated subdomain. The SSL cert should be securely sent to the satellite and the satellite’s proxy should be updated to use the new cert.
  3. When the cloud receives the satellite’s IP address from its heartbeats, it updates the DNS record for the satellite’s subdomain to point to the IP address.
  4. When executing queries, the UI can now safely make requests to the satellite’s assigned subdomain rather than directly to its IP address, all with valid certs!

In the end, making requests directly to the satellites turned out to be more complicated (and hacky) than we’d originally thought. The solution also doesn’t scale well, since the SSL certs need to be pre-generated. Without having a fixed number of satellites, or an upperbound on the number of satellites, it isn’t long before all the certs have been assigned and someone needs to step in and manually generate more. It is possible to generate the certs and their DNS records on the fly, but we’ve found these operations can take too long to propagate to all networks. It is also important to note that this approach may violate the terms of service for automated SSL generation and is susceptible to usual security risks of wildcard certificates.

When a satellite is behind a firewall, it will only be queryable by users within the network. This further ensures that no sensitive data leaves the network.

Approach 2: Proxying Queries through the Server

Diagram of Passthrough Mode where UI requests are proxied through the cloud.

As seen in the previous approach, it is easiest to have the UI make requests to the cloud to avoid any certificate errors. However, we still want the actual query execution to be handled by the satellites themselves. To solve this, we architected another approach which follows these general steps:

  1. User initiates query via the UI.
  2. The cloud forwards the query to the appropriate satellite.
  3. Satellite send its responses back to the cloud.
  4. Cloud forwards responses back to the UI.

The cloud must be able to handle multiple queries to many different satellites at once. A satellite will stream batches of data in response, which the server needs to send to the correct requestor. With so many messages flying back and forth, all of which need to be contained within their own request/reply channels, we thought this would be the perfect job for a message bus.

The next question was: which message bus should we use?

Choosing a Message Bus

We built up a list of criteria that we wanted our message bus to fulfill:

  • It should receive and send messages quickly, especially since there is a user waiting at the receiving end.
  • It should be able to handle relatively large messages. An satellite’s query response can be batched into many smaller messages, but the size of a single datapoint can still be non-trivial.
  • Similarly, since an satellite’s response may be batched into many messages, the message bus should be able to handle a large influx of messages at any given time.
  • It should be easy to start new channels at any time. We may want to create a new channel per request or per satellite, all of which we have no fixed number.

We briefly considered Google Pub/Sub, which had strict quota requirements (only 10,000 topics per Google project), and other projects such as Apache Pulsar. However, we primarily considered two messaging systems: Apache Kafka and NATS. General comparisons between Kafka and NATS have been discussed at length in other blogs. In this blog post, we aim to compare these two systems based on our requirements above.

We relied heavily on benchmarks that others have performed to judge latency based on message size and message volume. These results lean in favor of NATS.

We also wanted to test each system on our particular use-case, and performed the following benchmark to do so:

  1. Through a WebSocket run on the browser, send a message to the server.
  2. A service running on the server, called RequestProxyer, receives the message and puts it on topic A.
  3. A subscriber of topic A reads the message, and publishes it onto topic B.
  4. RequestProxyer reads the message on topic B, and writes a response back out to the WebSocket.
Diagram of the benchmark we used to test various message bus options.

In this case, the latency recorded for the benchmark is the time from which the websocket message is received in the RequestProxyer, to the time in which the server receives the response message from the subscriber.

These benchmarks were run on a 3-node GKE cluster with n1-standard-4 nodes, with a static 6-byte message. These results may not be generalizable to all environments. We also acknowledge that these systems were not built for this particular use-case.

Self-Hosted Kafka

Kafka benchmark results

NATS Streaming

NATS Streaming benchmark results

The Winner

We ended up choosing NATS as our messaging system. Benchmarks performed by others and our own benchmark above showed that NATS is capable of efficiently handling our request and response messaging patterns. We also found it was extremely easy to create topics on-the-fly in NATS, whereas creating topics on Kafka can be fairly complicated since partitioning must be determined before start-up. Given that we will support many short-lived queries, we want to avoid any topic creation overhead. These points, paired with the lower operational complexity of NATS made it the clear winner for our case. It is important to note that Kafka’s system is built to provide additional guarantees and has many positives, which may be necessary for other use cases.

The Implementation

Implementation for Passthrough Mode where UI requests are proxied through the cloud.

The actual implementation of our query request pipeline looks very similar to the benchmark case we ran above.

  1. The user initiates the query request through the UI.
  2. The cloud service responsible for handling the query requests receives the message and starts up a RequestProxyer instance in a new goroutine.
  3. The RequestProxyer generates an ID for the query and forwards the query and its ID to the correct satellite by putting a message on the satellite/<satellite_id> NATS topic. It waits for the response on the reply-<query-ID NATS topic.
  4. The service responsible for handling satellite communication (such as heartbeats) is subscribed to the satellite/* NATS topic. It reads the query request and sends it to the appropriate satellite via its usual communication channels. The satellite streams the response back to this service. The service then puts these responses on the reply-<query-id> NATS topic.
  5. The RequestProxyer receives the responses on the reply-<query-id> topic and sends them back to the UI.

It is worth noting that in this approach, since data is now funneled through the cloud rather than directly from the satellite to the browser, there may be additional network latency.

In clusters behind a firewall, proxying the request through the cloud will allow data access to out-of-network users. This can be both a positive and negative, as it makes the application easier to use but relies on potentially sensitive data exiting the network.

We use both approaches in Pixie, and have found either method allows us to efficiently and reliably query data from our customer’s clusters. By providing both options, customers have the flexibility of choosing the architecture that best meets their security needs. We believe these techniques can be useful for any on-prem connected architecture, and the particular approach should be chosen depending on the overall use-case and constraints specific to the system itself.

Overall, designing an split data/control plane architecture for Kubernetes native applications will help developers iterate quickly despite the on-prem nature of Kubernetes deployments.


Continue Reading

US Election

US Election Remaining