The Joe Sullivan Verdict – Unfair? – Which Part? (cybertheory.io)
Fujitsu Details Non-Ransomware Cyberattack (webpronews.com)
5 Key Questions CISOs Must Ask Themselves About Their Cybersecurity Strategy (thehackernews.com)
Sizable Chunk of SEC Charges Vs. SolarWinds Dismissed (darkreading.com)
CrowdStrike CEO apologizes for crashing IT systems around the world, details fix | CSO Online
Summary:
Cybersecurity Updates: Uber’s Legal Trouble, SolarWinds SEC Outcome, and CrowdStrike Outage
In Episode 273 of the Defensive Security Podcast, Jerry Bell and Andrew Kalat discuss recent quiet weeks in cybersecurity and correct the record on Uber’s CISO conviction. They delve into essential questions CISOs should consider about their cybersecurity strategies, including budget justification and risk reporting. The episode highlights the significant impact of CrowdStrike’s recent updates causing massive system crashes and explores the court’s decision to dismiss several SEC charges against SolarWinds. The hosts provide insights into navigating cybersecurity complexities and emphasize the importance of effective communication and collaboration within organizations.
00:00 Introduction and Banter
01:52 Correction on Uber’s CISO Conviction
04:07 Recommendations for CISOs
09:28 Fujitsu’s Non-Ransomware Cyber Attack
12:13 Key Questions for CISOs
32:47 Corporate Puffery and SEC Charges
33:15 Internal vs External Communications
33:52 SolarWinds Security Assessment
36:36 CrowdStrike CEO Apologizes
37:16 Global IT Systems Crash
37:57 CrowdStrike’s Kernel-Level Issues
40:55 Industry Reactions and Lessons
42:58 Balancing Security and Risk
49:26 CrowdStrike’s Future and Market Impact
01:03:46 Conclusion and Final Thoughts
Transcript:
defensive_security_podcast_episode_273 ===
jerry: [00:00:00] All right, here we go. Today is Sunday, July 21st, 2024, and this is episode 273 of the Defensive Security Podcast. My name is Jerry Bell, and joining me tonight as always is Mr. Andrew Kalat.
Andy: Good evening, Jerry. I’m not sure why we’re bothering to do a show. Nothing’s happened in the past couple of weeks.
Andy: It’s been really quiet.
jerry: Last week was very quiet.
Andy: Yeah, sometimes You just need a couple quiet weeks.
jerry: Yeah. Yeah, nothing going on so before we get into the stories a reminder that the thoughts and opinions We express on this podcast do not represent andrew’s employers
Andy: Or your potential future employers
jerry: or my potential future employers
Andy: as you’re currently quote enjoying more time with family end quote
jerry: Yes, which by the way Is highly recommended if you can do it.
Andy: You’re big thumbs up of being an unemployed bum.
jerry: It’s been amazing. Absolutely [00:01:00] amazing. I I forgot what living was like.
jerry: I’ll say it that way.
Andy: Having watched your career from next door ish, not a far, but not too close. I think you earned it. I think you absolutely earned some downtime. My friend, you’ve worked your ass off.
jerry: Thank you. Thank you. It’s been fun.
Andy: And I’ve seen your many floral picks. I don’t, I’m not saying that you’re an orchid hoarder, but some of us are concerned.
jerry: I actually think that may be a fair characterization. I’m not aware of any 12 step programs for for this disorder here.
Andy: There’s a TV show called hoarders where they go into people’s houses who are hoarders and try to help them. I look forward to your episode.
jerry: I yes, I won’t say anymore. Won’t say anymore. So before we get into the new stories, I did want to correct the record on something we talked about on the last episode [00:02:00] regarding. Uber’s CISO that had been criminally convicted. Richard Bejtlich on infosec. exchange actually pointed out to us that it was not failure to report the breach that was the problem. It was a few other issues, which is what Mr. Sullivan had actually been convicted of. So I’m going to stick a story into the show notes. That has a very very extensive write up about the issues and that is from cybertheory. io. And in essence, I would distill it down as saying again, I guess he was convicted so it’s not alleged. He was convicted of obstruction of an official government investigation. He was convicted of obstructing the ongoing FTC investigation about the 2013 slash 2014 breach, [00:03:00] which had been disclosed previously.
jerry: The FTC was rooting through their business and were asking questions and unfortunately apparently Mr. Sullivan did not provide the information related to this breach in response to open questions. And then furthermore, he was he was convicted of what I’ll summarize as concealment.
jerry: He was concealing the fact that there was a felony. And the felony was not something that he had done. The felony was that Uber had been hacked by someone and was being extorted. But because, he had been asked directly, Hey, have you had any, any issues like this?
jerry: And he said, no, that becomes a concealment, an additional concealment charge. And so the jury convicted him on both of those charges, not on failure to disclose a breach.
Andy: Yeah, it’s we went down the wrong path on that one. We were a little, we put out some bad info. [00:04:00] We were wrong.
jerry: So I’m correcting the record and I certainly appreciate Richard for for getting us back on the right track there.
jerry: This article, by the way, does have a couple of interesting recommendations that I’ll just throw out there. One of them is hopefully these are fairly obvious. Do not actively conceal information about security incidents or ransomware payments, even if you’re directed to do so by your management.
Andy: Yeah. I think, let’s put it out for a second. If you’re in that situation, what do you do? Resign?
jerry: Yes. Or do you,
Andy: yeah, I think that’s,
jerry: I mean you either resign or you have to become a whistleblower.
Andy: Yeah, that’s true. Your career has probably ended there at that company either way. Most likely. But it’s better than going to jail.
jerry: It’s a lot better than going to jail. I think what I saw is he Sullivan is up for four to eight years in prison, depending on how he’s sentenced.
Andy: Feds don’t like it when you lie to them. They really don’t like it.
jerry: No, they don’t. Next recommendation is if you’re, if your company’s under investigation, get help and potentially [00:05:00] that means getting your own personal legal representation to help you understand what reporting obligations you may have for any open information requests. And I say that because. In this instance, Sullivan had confirmed with the CEO of Uber at the time about what they were going to disclose and not disclose and the CEO signed off on it. And he also went to the chief privacy lawyer, who by the way, was the person who was managing the FTC investigation and the chief privacy lawyer also signed off on it.
Like the joke goes, the HR is not, it’s not your friend. Your legal team may also not be your friend. At some point if you’re in a legally precarious position, you may need your own council, which is crappy.
Andy: That is crazy. How much is that going to cost? And wow, that’s it. I don’t [00:06:00] one more reason to think long and hard before accepting a role as CISO at a public company.
jerry: Yeah, this, by the way I’m skipping over all sorts of good stuff in this story. So I invite everybody to read it. And it’s a pretty long read.
jerry: It, it talks about the differences between the Directors of companies and officers of companies and the different obligations and duties they have related to shareholders and customers and employees and whatnot. And what was very interesting. The point they were making is that CISOs don’t have that kind of a responsibility, right?
jerry: They don’t, they’re not corporate officers in the same way. And so what they, what, when you read the article, and I apologize for not sending it to you. I just realized, when you read the article it was very clear that there The author here was pointing out that the government and I suspect with, at the behest of Uber, was really specifically [00:07:00] going after Sullivan, right?
jerry: Because in exchange for testimony, people got immunity in order to testify against Sullivan. And that kind of went all up and down, including You know, it’s some of the lawyers. So I, by the way, I think he clearly had some bad judgment here. But, also, he wasn’t the only one. This was a a family affair, but he’s the one who’s really taken taken the beating. Next recommendation was paying a ransom in return for a promise to delete copies of data, not disclosed data does not relieve your responsibility to report the issue in many global laws and regulations.
jerry: So just because you’ve gotten an assurance that the, after you’ve paid a ransom that the data has been destroyed, you still in, in almost all cases are going to have a responsibility to report. And, one of the things the the author here says is you really should let everybody know, there’s vehicles to [00:08:00] inform at least in the U S CISA and the FBI, and I’m sure there’s similar agencies in different countries. To help insulate yourself do not alter data or logs to conceal a breach or other crime. That seems pretty self evident, but I think the implication is that.
jerry: That’s what happened here. And then also lastly, do not create documents that, contain false information.
Andy: Shocking.
jerry: Yes. So again, not, nothing in there that is like earth shattering but it’s a good reminder,
Andy: yeah. And I, I don’t know if but our good friend Bob actually got out of the South American prison he’s been in for a while, and I heard from him, and he’s doing well, he’s got three new tattoos and lost two fingers, but otherwise he’s doing well. He was telling me that he once worked for a CISO that actually fabricated evidence for an internal auditor.
Andy: And thought it was a fun [00:09:00] game
Andy: and how he had a tough time knowing how to handle that.
jerry: And the ethics of how to disclose that, right?
Andy: Especially because as he described it, it was a very powerful CISO who had a reputation for retaliatory behavior to those who did not bow before him. Damn. So
jerry: yeah, Bob has all the best stories.
Andy: He does. He does. I look forward to hearing more about his South American prison stint.
jerry: All right. Our next story today comes from web pro news. com. And the title here is Fujitsu details, non ransomware cyber attack. It feels like it’s been so long since we’ve talked about something that wasn’t ransomware.
Andy: I feel like these bad guys just, lost a good ransomware opportunity.
jerry: Clearly they did. So there’s not a huge amount of details. But basically Fujitsu was the victim of some sort of [00:10:00] data exfiltrating worm that crawled through their network. They haven’t published any details about who or how, or, why, what was taken, but was, what was most interesting to me is that, the industry right now is very taken by ransomware or, more pedestrian hacks of things to mine cryptocurrency or send spam or, do those sorts of things.
jerry: It’s been a while since I’ve. I can think of the last time we actually had a, like a a destructive or, something whose job was not. To be immediately obvious that it’s in your environment.
Andy: Yeah. If I had to, again, the details are very sketchy, but if I had to guess, maybe this was some sort of corporate espionage or some sort of, it appears the way they described it, which again, the details are sparse.
Andy: It was low and slow and very quiet [00:11:00] trying to spread throughout their environment. It didn’t get very far. They said, what, 49 systems? 49. And they had a lot of interesting, you caveats of it didn’t get to our cloud this and it didn’t do that. So there’s a lot of things that didn’t do.
Andy: They didn’t tell us much about what I did do. But if I had to guess, maybe some sort of corporate espionage. Yeah, maybe that’s, or just random script kitties being like, you can never always attribute motivation. So I’ll say
jerry: this way, intellectual property theft, the motivations for that, I guess this is an exercise left to the reader, but.
jerry: They did say that data was exfiltrated successfully. They didn’t say what data but I, my guess is, they were after some sort of intellectual property theft. The reason for bringing this up is not that this has a whole lot of actionable information, but more that, that there are other threats out there still, it’s not all, it’s not all ransomware and web shells and that sort of stuff.[00:12:00]
Andy: Indeed, but to be fair that is majority of it. Protect your cybers. You know what helps? A solid EDR. It’s a little foreshadowing for a future story.
jerry: We’ll get there. We’ll get there. All right. The next story comes from thehackernews. com and the title here is five key questions CISOs must ask themselves about their cybersecurity strategy.
Andy: Apparently, we need to add a sixth one, which is, Am I going to go to jail?
jerry: So the key questions here, number one, how do I justify my cyber security? Actually, you know what, I’m going to back up for a second, because there were a couple of other salient data points in here. And the first one was they pointed out that only 5 percent of CISOs report directly to the CEO , then two thirds of CISOs are two or more levels below the CEO in the reporting chain. And that, those two facts indicate a potential lack of high level influence to [00:13:00] use their words. I will tell you the placement of the CISO in an organization isn’t necessarily an indicator of how much power they have. Somebody who reports to the CEO is going to be more influential for sure, but there are lots of different organizational designs especially when you go into larger companies.
Andy: Sure. I would say also if they’re highly regulated, that CISO has a lot of inherent authority because of the regulations that are being enforced upon that organization. So by external third parties.
jerry: The Ponemon or Pokemon Institute found that only 37 percent of organizations think they effectively utilize their CISOs expertise.
jerry: I kind of wonder who are they asking that? Are they asking the CISOs or are they asking, I, anyway I am curious about the [00:14:00] methodology behind that study. It doesn’t necessarily surprise me. Just moving somebody up in a different, into a different place in the organization doesn’t necessarily mean that they’re going to more fully use the talents of or expertise of a CISO.
Andy: Yeah. If it’s anything in most organizations, it’s. They delegate to that CISO, not like what the assumption, is that the boards of the executive teams would be asking deep cyber questions of the CISO, which is an odd expectation.
jerry: It is an odd expectation. And similar related to what you’re saying, gartner finds that there are only 10 percent of boards. that have a dedicated cybersecurity committee overseen by a board member.
Andy: The way I would look at it, both of those stats is more, how much influence does CISO have on the company operating in a less risky or more risky methodology, right?
Andy: It’s not about leveraging their expertise. It’s about how influential are they to [00:15:00] guide the company away from risk and what those trade offs are.
jerry: It also comes down to what the company’s value. This is a financial risk management. And
Andy: the flip side is I think a lot of executives think of CISOs as constantly calling for the skies falling to get better budgets and build their empire and more people. And as this is a black hole of money we’re throwing money into that we can’t, which this article goes into, we can’t justify it.
Andy: We can’t prove the ROI on.
jerry: Yes, exactly. So the the key questions to ask yourself is number one, how do I justify my cybersecurity budget? And that is a I think that’s a perennial challenge that anybody in security leadership has. How do how do you justify, or demonstrate that you are spending the right amount of money?
jerry: You’re not spending too much. You’re not spending too little. Generally [00:16:00] speaking, and this is like a, one of those mass psychosis. episodes. You do that by often benchmarking yourself against your competitors.
Andy: It’s a safe answer.
jerry: And they do it by benchmarking themselves against their competitors.
Andy: You’ve got the theory of the wisdom of crowds, right? What’s if I’m around the average, I must be doing fairly close to correct, but not all companies are the same. Not all companies have the same risk tolerance. Not all companies have the same, corporate structure in the same financial situation. So I get it. That’s where my mind goes. What percentage of G&A is spent on cyber in the, my industry? That’s what I’m going to go ask for.
jerry: Number two is how do I master the art of risk reporting, which by the way, I think is not entirely disassociated from the last one, right? Because part of your budget in I dare say a major part of your budget is intended to address [00:17:00] risk. And and what they’re really pointing out here is how do you communicate to the senior leadership team, the board of directors and so on, the level of risk that you cyber risk that you have in your organization in terms that make sense to them,
Andy: That’s an incredibly challenging question, honestly.
jerry: Yeah. I, so something that was very interesting is I was, to me, at least, is I was reading this because look, I struggle with all these things too, right? I’ll. Five of these things that we haven’t got to all of them yet, but they resonated with me and he’s super interesting is we all have to make this up on our own.
Andy: You didn’t go through that section of the CISSP?
jerry: There’s not like a GAAP, in, in in accounting, you have the GAAP generally accepted accounting principles. There’s really a gap type methodology for this in risk reporting. And [00:18:00] perhaps there should be.
Andy: This is why we are often accused of being an immature industry from other well trodden business leaders who have a shared language.
Andy: We’re wizards and witches walking in speaking spells that they don’t understand out of black boxes that don’t make sense.
jerry: So I, I think this is an area that we can certainly mature. So I would love to hear from anybody in the audience who thinks that there’s a, a common methodology that people can adopt here. I’d love to talk about that in a future episode. All right. Number three is how do I celebrate security achievements?
jerry: I have a problem with the way some of the, this was worded public recognition of attacks that were deflected. This is in quotes, by the way, public recognition of attacks that were deflected can simultaneously deter attackers and reassure stakeholders of the organization’s commitment to data [00:19:00] protection.
jerry: So I’m reminded of when I read that I, I immediately thought of Oracle’s unbreakable Linux or unhackable, what do you call it?
Andy: Yeah.
jerry: It’s like putting a chip on your shoulder and Begging someone to come in.
Andy: If I really dug into this, define what an attack is, define when I’ve deflected it. Like every firewall drop, log entry, is that an attack I stopped?
Andy: Like I’ve seen that kind of shenanigans. Or is it more, hey, we had an incident that started and we contained it. Or is it, I don’t know, every time my email security tool stopped a phishing attack? There’s all those sorts of metrics you can run, but is it valuable?
jerry: There’s all you get into like how many spams did I reject?
jerry: How many phishing emails did I reject? Which we make fun [00:20:00] of, right? Because they’re metrics. They’re not achievements.
Andy: But you’re trying to prove a negative here. This is, this has been the fundamental problem from day one with the industry is you’re spending money to stop something. How do you know if you hadn’t spent that money, that things would have happened?
jerry: The only thing I can say is if you take a more capability focused view rather than a metrics focused view, I think that’s perhaps where the opportunity lies. We had a gap in. We had a gap in our authentication scheme because we didn’t have multi factor authentication.
jerry: We, we implement a multi factor authentication. We closed a huge hole. Yeah. Yes. Super simplistic example. Yeah. But I will say, there is a there’s another aspect of this that you have to be aware of. And perhaps I worked alongside too many lawyers, [00:21:00] but one of the, one of the pitfalls of taking credit for doing some security thing is that you’re tacitly admitting that you weren’t doing it before.
jerry: Yeah,
Andy: that’s true.
Andy: Our new version no longer does X. Wait, you were doing X before? Don’t worry about that. The fact is we’re not doing it now.
jerry: We implemented multi factor authentication. Oh so wait a minute,
Andy: right? It’s a tough one. Yeah. I, but I also, You also can never be, if you’re completely risk zero and completely safe, you’ve either way overspent, or you’ve added so much friction to business, or you’ve inhibited the ability for people to do the jobs that you’re now breaking the business in a different way.
Andy: You’re not going to get to risk zero. So what’s the right balance?
jerry: Yeah. And the business doesn’t want you to get to, I remember working effectively as the CIO for a company that [00:22:00] we both worked for once. And the COO told me he was he pitched it in the form of a question. Now what is your approach to passing audits, Jerry?
jerry: Do you want to, like, how do you you want to do really well? And I said, yeah, I think you should do really well. And he said, no, I said, if you fail audits, you’re going to get fired. And if there are no issues ever found, you’re probably going to get fired because you’re spending too much money.
jerry: So you got to find the right balance because that’s what the business wants. If you’re, if you are. Spending enough money to do perfect and everything that’s coming at the expense of other things that the business could be investing in and the return, the rate, I think his point was not.
jerry: Except trying to accept too much risk, but that to do things perfectly, as you continue to move up the [00:23:00] maturity ladder, it gets more and more expensive. And the, the marginal utility starts to decline.
jerry: Sure.
jerry: Anyhow, I, all that said it is very important from a morale perspective, if for nothing, no other reason from a morale perspective to celebrate. But you’ve got to be smart about it.
Andy: I wouldn’t do it publicly, frankly.
jerry: I wouldn’t either.
Maybe internally Somewhat company wide maybe, or at least departmental wide, you need to understand what motivates your people and what their reward systems are like and work to those. But I am very much against putting that bullseye on your back by saying you’re not hackable or, you’re about to get a free audit.
jerry: Oh, yes. Oh, yes. So number four is how do I collaborate with other teams better at this, by the [00:24:00] So again this whole article is. Aimed at CISOs and CISOs are almost always an executive level position. And one of the, I learned a lot, right? I ended my career. I don’t know if that’s the end or if there’s more to come as an executive.
jerry: And I learned a lot. I learned a lot about what it, what that means. And one of the most important aspects is that. You do partner with those other people. That’s less, an intrinsic part of being an executive.
Andy: Yeah. It becomes about working well with other departments. And that means sometimes you’ve got to give and take and be willing to lose a battle to win the war, as they say.
jerry: So it’s super important, but I think this is not. It’s not security centric. This is a fundamental [00:25:00] tenet of what it means to be a leader in an organization.
Andy: And I think we, as technology people often get promoted up with a background that isn’t well suited for that, to be completely honest to the point where many technical people score in those quote unquote soft skills.
Andy: But if you want to get to that level of organization, it is required.
jerry: Yes. Yes, absolutely. No, they point out that there are tangible security benefits. So for example, building bridges with HR allows you to do things like Integrate security requirements into the onboarding and offboarding processes and whatnot.
jerry: And also having those relationships throughout the organization are very key, especially when in times of crisis, like in an incident or what have you, you’ve got to, you’ve got to have the trust of the team and the team needs to have trust in you.
jerry: Last [00:26:00] one is how do I focus on what matters most? This is a hard one. And I think in large measure, it’s because there are so many variables, every company values things differently, they have different risk appetites. They’re in different industries. They move at different speeds. They have, different idiosyncrasies. They they like different technologies or they don’t like different technologies. And in many instances, companies hire people, they hire a CISO based on who that person is, like what, not necessarily what they need. They hire them based on, on the reputation, Jerry’s Jerry’s an incident response focused person.
jerry: We have lots of incidents.
Andy: Because of Jerry?
jerry: Maybe.
Andy: Oh, that’s fair. Job security.
jerry: I think you’ve got to take a step back and, as you’re figuring out how to focus on what matters most, you’ve got to, [00:27:00] first define what is it that matters most.
Andy: Yeah. . The other thing I would add, it’s, I don’t know how to integrate this, but there’s a cultural aspect of a company. The culture matters. So I may want to do a highly impactful security initiative. Let’s say something like, I don’t know, DLP with a document classification system, and I may work for a large financial who’s completely on board with that and very comfortable with that.
Andy: And great. You work for a small startup. You’re getting a lot of pushback on that because that’s not their culture. It’s not something that, from my perspective, and obviously I haven’t been a CISO but I’ve been senior level and currently a director, you can’t go faster than the culture of the company will allow you to go if you’re implementing potentially friction inducing security controls.
Andy: And so that I think can help determine your priorities. What’s acceptable to the business from an [00:28:00] impact perspective.
jerry: That’s a good, it’s a good point. I’ll tag on to that and say, one of the things I’ve learned over the years is that. Regardless of how much money you’re given, there is only so much change than an organization can undergo.
Andy: Oh yeah. That’s a good point.
jerry: In a certain period of time. So someone comes to you and says, you have a blank check. I need you to implement DLP, replace all our firewalls, a bunch of additional, very disruptive things. You’re not going to be able to do it.
jerry: Even if you have the money to do it there’s a finite capacity for change in an organization that, you know, that, that threshold is going to be different in different organizations. And that’s one of the challenges that you as a leader have to figure out is, where that threshold is. So you don’t cross it because if you cross it, Things start to fall apart and you don’t actually make progress on anything.
jerry: And I’ve seen that [00:29:00] happen time and time again, where, we, we, Hey, we’re all in we are going to blow the doors off the budget. We’re going to, we’ve got a lot of things that have to change.
Andy: Especially after a breach.
jerry: Yeah. And you end in having, you end having not accomplished anything.
jerry: You’ve burned through all the money. But you’ve not accomplished fully anything. And so you’ve got to be very very measured in that.
Andy: And the flip side, if you’re an aggressive go getter as a leader and you commit to some aggressive schedule and then you don’t get it done or you don’t get there as fast as you want, then you potentially have executives looking at you like you’re ineffective.
Andy: So it’s a careful balance.
jerry: And that’s it. Leadership 101. Like you’ve got to, you’ve got to meet your commitments.
jerry: Alright. So the, they do talk a little bit about communication, effective communication between the CISOs and [00:30:00] and up. I think that, permeated the la those five items. But, there, there is a, this maybe goes back to the whole gap that, the idea of the gap style reporting. I think we as a security community, often do a pretty bad job of.
jerry: Communicating in non technical terms we talk about CVEs and and DDoS volumes and things like that, but, translating that into business impact is really what the business needs from you.
Andy: When it’s also a likelihood percentage, not a guarantee. Correct.
Andy: Correct. And it’s at best a guess. Based on best practices and observing what happens to other companies and, lots of inputs and data, but there’s no guarantees.
jerry: I think one of the struggles when I read articles like this, they, they often talk about [00:31:00] things like how many fewer incidents did you have or how many fewer breaches did you have?
jerry: And and whatnot. And using that as, Is how you communicate the effectiveness and of your program or what you need to improve and I think the reality is that breaches tend to be like, very Transformational pivotal incidents. They’re often not like countable you don’t you don’t Stay in a ciso role and have you know So many breaches that you can show trends over time, right?
jerry: It’s just you’re, if you’re in that position and you have that kind of data, like something’s wrong.
Andy: Yeah. We need to learn very, we need a little, we need to learn from other people’s breaches, right?
jerry: Exactly.
jerry: All right. Moving on, the next one comes from dark reading here. And the title is sizable chunk of SEC’s charges against solar winds tossed out of court. So I will [00:32:00] admit I have not read. All 107 pages of the judge’s ruling, so shame on me for that.
Andy: You’re an unemployed bum, what else do you have to do?
jerry: Absolutely nothing. The SEC filed a lawsuit against SolarWinds and SolarWinds CISO, alleging lots of things. Everything was dismissed except for the statements that the CSO had made about the security program at SolarWinds prior to the breach. So there were inaccuracies in their 8k, which for those of you don’t know, 8k is a form that you have to fill out in a, in the wake of a breach as required by the SEC that apparently had some inaccuracies.
jerry: And so that was. Part of the case there were other statements made post breach that the judge I did find in a different article, described it as corporate puffery that is not [00:33:00] actionable. I thought that was pretty funny.
Andy: That is pretty funny. I think that needs to be a thing. I got to work that into more and more conversations.
jerry: It’s interesting that, A lot of the reaction to this, which means that there are apparently other implications in the ruling. A lot of a lot of the, post judgment discussion has been, Oh gosh, this is really a good thing because it allows teams internally to communicate amongst themselves without fear of what you write being used against you.
jerry: However, that, that actually. isn’t obvious as part of what the SEC was charging them with. I’ve got to go, I really want to go read that 107 pages to understand, what exactly the SEC was alleging. But in some regards it’s neither here nor there. What is most interesting though, Is the charges that do remain, which are those that [00:34:00] basically said before the breach, the solar winds CISO had come in and performed an assessment and found lots of problems and a documented those problems, but then we would go externally into customers and perhaps investors and made claims about the robustness of their security program.
jerry: And that is what. The SEC is still going after and that is what the judge is allowing them to continue pursuing.
Andy: Because the theory that the SEC I’m assuming is going after is accurate information should be disclosed to the investing public. And so they know how to appropriately measure the risk of investing in a company and or the board, point, all that stuff that comes with being a public company.
Andy: They want, they’re very particular. about making sure that the information that is disclosed is accurate and not misleading. We [00:35:00] see all sorts of stuff about misleading, just going back to Elon Musk’s tweets about Tesla getting him in a lot of trouble with the SEC and that sort of thing.
Andy: Like they take that stuff very seriously.
jerry: Oh yes. Yes, indeed. So more, probably more to come on this after I have a chance to read the the court decision, but I would definitely say, have a measured approach to communicating, especially if you’re aware that there are security gaps or weaknesses in your environment.
jerry: If you end up in a position where you are radically representing things differently in internal communications versus external communications, you should probably. Take a step back and ask yourself what you’re doing. I guess it probably won’t be a problem if they’re, if you’re not breached, but if you are like, that’s going to be exhibit a.
jerry: Yep. And as we’ve now [00:36:00] seen, like the company isn’t going to have your back. They’re not going to, they’re not going to stand in front of you and take the bullet, they’re going to be like, Oh, look at God, our CISO, he was a terrible guy.
Andy: Which by the way, is why people get so frustrated as statements put out by companies sounding like legalese and business ease because they’re protecting themselves with very specific language for these sorts of circumstances.
jerry: Yes.
jerry: So there’s one more story. It’s a small thing. It’s not probably not even worth talking about. I, I wasn’t even sure if we were going to get to it. Yeah, it’s not really even, you know what? Let’s talk about it. So this is the one comes from CSO online and the title is CrowdStrike CEO apologizes for crashing it systems around the world and details fix.
Andy: Yeah, it was it was a thing.
jerry: It was a thing. I have to tell you, I I woke up. on Friday to a text from my wife who had [00:37:00] already been up for hours asking me if I was happy that I was no longer in corporate IT. And I, what?
Andy: What just happened?
jerry: What just happened? And so of course I jumped on to infosec. exchange and quickly learned that 8. 5 million. window systems around the world had simultaneously blue screened and would not come back up without intervention.
Andy: Like on site physical intervention.
jerry: Yes. Yeah. Yes. Apparently though, if you could reboot it, Some, somewhere between three and 15 times, and it might come back on its own.
Andy: I
Andy: heard
Andy: that
Andy: too.
Andy: I have no idea how accurate that is.
jerry: I don’t know either.
Andy: But that’s going to be the new help desk joke. Have you tried rebooting it 15 times?
jerry: Yes. Yes, it is already it is already meme fodder for sure.
Andy: Insane amount of memes. So what happened?
jerry: CrowdStrike, I think most people know is a, it’s an [00:38:00] EDR agent that runs on.
jerry: I think it’s probably 10 to 15 percent of corporate systems around the world. It’s a, it’s a significant number. They deliver these content updates which are, I would say roughly equivalent to what we used to think of as antivirus updates. They delivered one on Friday. And these, by the way, are multi time a day updates. So these aren’t like new versions of the software. These are like, fast quickly delivered things. And so what happened was on Friday, CrowdStrike pushed out a change, an update in how it processes or how it. analyzes named pipes and the definition file that they pushed out had some sort of error that the nature of the error hasn’t been [00:39:00] disclosed that I’ve seen at least and that error caused the windows to crash basically.
jerry: Like crash
Andy: hard. Crash hard
jerry: but with blue screen basically.
Andy: Yeah
jerry: and put in a blue screen loop at that point. And so yeah, because it was, because, Of how CrowdStrike integrates with Windows, it would, it would blue screen again, immediately, as part of the startup process. So you would, as you described that you would up in a blue screen loop.
jerry: And so the only option you had was to go into safe mode. And remove a file. And people came up with all sorts of creative ways of doing that with scripts and even Microsoft released a an image on a thumb drive that you could boot now where it went horribly wrong for some people. And I think in Azure, it was ironically, like one of the most problematic places where you have disk encryption, [00:40:00]
Andy: right?
Andy: You’re probably most often using boot boot locker. And if you don’t have your recovery key. You can’t access the disk without fully booting.
jerry: So lots of lots of IT folk got a lot of exercise over the past weekend. And by the way, if they’re looking a little, they’re dragging this week, but go buy him a donut, hi and say, thank you.
jerry: Because they’ve had a, they’ve had a
Andy: bad couple of days. Yeah, no kidding. It’s also amazing how many people have turned into kernel level programming experts. On social media in the past three days,
jerry: 100%,
jerry: they’re like, they’re formerly political scientists and constitutional scholars and trial attorneys and epidemiologists and climate scientists and whatnot.
jerry: So it does not surprise me that they are also kernel experts.
Andy: There’s been a lot of really intense finger pointing and [00:41:00] debate going on when we, when Honestly, don’t even fully know the entire story yet.
jerry: No. It, there’s a whole lot of hoopla about, layoffs of QA people and returned the impact of return to office, but we don’t, we really don’t know what happened.
jerry: What I find most interesting is that this is a process that happens multiple times a day for the most part. And. Hasn’t happened before. So something went horribly wrong. And I don’t know if that was because they skipped the process or because there was a gap in coverage, like this, the set of circumstances that arose here, we’re just not ever accounted for, like that nobody thought that was a possibility,
Andy: which by the way, it happens at almost every engineering discipline.
Andy: We learn through failure. I’m not. Okay. Let me back way up and say, I am not in any way a CrowdStrike apologist. In fact, I’m not a huge fan of CrowdStrike. [00:42:00] However, I’m seeing a whole lot of holier than thou, you should have XYZ’d on social media. That puts me in a contrarian mood to counter those arguments with a cold dash of reality of, there’s a whole lot of reasons Companies do what they do in the way they do it.
Andy: And anyway, I don’t want to get in my entire soapbox when there’s more done back here, but I think it’s very easy to point out failures without weighing it against benefits.
jerry: Yeah, absolutely. CrowdStrike works quite well for a lot of people. And I dare say it has saved a lot of asses and a lot of personal data.
Andy: So for everybody, there’s certain people like somebody we know who’s very commonly commenting on these things who were in a couple of articles saying that, this just proves that automatic updates are a bad idea. I don’t know that’s true. I would say you can’t. Say that unless you [00:43:00] measure the value of automatic updates that have stopped breaches and stopped problems because those updates were so rapid and so aggressive against this outage, right?
Andy: You have to balance both sides of that scale. So just look at this and say, yeah, this was a massive screw up. It’s caused massive chaos and huge amount of loss of income for a lot of people disrupted a lot of people’s lives. Okay. That’s bad, but weigh that against For those using tools with automatic updates, how many problems were solved and avoided, which is so difficult to measure, but must be thought of by those rapid updates that were automatic.
jerry: I think what is the most problematic aspect of that is that the the value is amortized and spread out over, thousands or tens of thousands of customers. Over, over the, over a long period of time, but this failure was a, a one, impacted everybody at the [00:44:00] same time and caused mass chaos, mass inconvenience, mass outages.
jerry: All at the same time. And so completely agree with you. But I think that’s what’s setting everybody’s alarms bells off is that, Oh my gosh we have this big systemic risk, which, realistically hasn’t happened perhaps as often as you might think. And I know by the way, there’s a lot of people who are also talking smugly about how good it is to be on Linux and not on Windows because, this issue only impacted Windows, not Linux. But I will tell you, as the former user of a very large install base of CrowdStrike on Linux, it had problems. And a lot of them. We had big problems. I don’t know that there was ever one like as catastrophic as this where it happened all at the same time, [00:45:00] but, it, it’s not Linux the CrowdStrike agent on Linux hasn’t also had issues.
Andy: Sure. It’s the question comes, what problems would you had if you hadn’t run it? What value did it bring?
jerry: That’s obvious. I think that for many organizations, this is the way they identify that they’ve been in intruded on
Andy: yeah I mean ensure cyber insurance companies basically mandate you have EDR.
Andy: For ransomware containment. There’s it’s For good or ill it’s not table stakes So and those by the way, those cyber security companies out there who are currently casting stones At CrowdStrike saying we don’t do things this way and we would never have this problem. Yeah, good luck. Yeah, you are the definition of glass houses.
Andy: We’ve, this is, interestingly, that 8. 5 million stat apparently was, according to Microsoft, less than one per seat, 1 percent of the Windows fleet in the world, which I find fascinating that only 1 percent cause this much [00:46:00] chaos. So I wonder how many of those are second order impacts and other sort of, not direct, but secondary fallout of some critical system somewhere going down, but we’ve seen problems like this before.
Andy: It’s just on smaller scales. Even Windows Defender has had somewhat similar outages that have caused problems. This is the whole debate of Windows update, do you automate, do you trust it, do you test it? And I go back to, okay, you may have a problem. But is that problem worth the efficiency and speed of getting a patch out there before you get hit with that exploit for whatever recently patched problem or, you can’t just look at one side of the equation, which is so frustrating to me.
Andy: And so many people are out there clout farming right now, just being, I told you so’s or you guys are just dumb and they’re not looking at the big picture at all.
jerry: So on the converse side. There [00:47:00] were huge impacts, and I think that we do have to do better. But that said, I don’t think it’s a wise idea to run out and uninstall your CrowdStrike agenT.
jerry: there are other technologies, other ways of linking in that perhaps are less risky, but not no risk, for sure.
Andy: and are you talking about the kernel level? Yeah. Integration. Yeah. Versus non kernel level.
jerry: Yeah, so like E-E-B-P-F versus, kernel modules and whatnot. But those that by the way is a it solves some problems, but it creates other problems and so we’ve not seen big failures with those other companies But have we not seen them because they’re just small or because they don’t happen
Andy: And look, let me be very clear. I am not a coder. I don’t understand what i’m talking about here I am just going off of like Rough understanding of trying to get my arms around this issue.
Andy: So take everything i’m about [00:48:00] to say with a grain of salt but my understanding is that the advantage of being at the kernel is you’ve got much deeper level access that’s faster, more efficient, and more TAP resistant than running in user space. And as a security control that’s trying to stop things like rootkits, which were a big problem back in the day, not so much now, I think that’s where that came from. Like we’ve seen a number of vendors who are saying running at the kernel level is just, it’s just irresponsible. We don’t do that. And here’s why we don’t do that. We’re differentiated because of XYZ. Okay, cool. But there must be a trade off and Yeah, you can’t crash the system like that.
Andy: But are you also as capable at detection and what is your resource impact? And I honestly don’t know, right? And when you listen to these other vendors who don’t use kernel mode, they of course have very compelling arguments and they drink their own Kool Aid and they believe their own marketing.
Andy: And maybe they’re right. I don’t know, but I also don’t think CrowdStrike is just completely irresponsible for running a [00:49:00] kernel. Some people are saying. I think that has a benefit. Which is why they did it. Now, whether that benefit is worth it is the question, but I don’t think they’re just malicious.
Andy: So I don’t know. I just see a lot of people going off on this. And I admittedly, I don’t know enough to probably really participate in that conversation, but
Andy: it’s a tough,
jerry: I do think that CrowdStrike, my understanding at least is that CrowdStrike was, or is moving in the direction of a similar strategy, but they just aren’t there yet of not using curl mode. Yeah. Yeah. Yeah. Not using a kernel module. So don’t misunderstand. Like they’re, they are using this thing called EBPF.
jerry: It’s a way, it’s a very uniform way of getting visibility into what’s happening in the kernel without actually loading your own driver into or module into the [00:50:00] kernel. One of the big problems I had with CrowdStrike was this constant churn of, do I patch my kernel or do I leave CrowdStrike running because I can’t do both.
jerry: And of sequence
Andy: on supporting each other.
jerry: Yeah. And that, by the way, it comes back to the fact that they have to they have to create a kernel module. It’s tailored, to, to different kernel versions. And depending on how changes in the kernel, how the kernel changes from one version to the next.
jerry: Sometimes you don’t, sometimes it’s fine. Most of the time it’s not fine. And so you end up with this problem. So less of a problem on windows because windows kernels are pretty stable. And I think they do, they have a lot more interlocking with vendors like CrowdStrike than Linux does. But in any event, I.
jerry: I I agree with your thesis there that like, this is something that we have. [00:51:00] We are benefiting from technology like this and assuming that it could never go wrong. And that’s probably not a good assumption as we’ve now seen, but at the same time, like I do think this could have gone better.
Andy: Absolutely. But where the assumption of this could go wrong at some point is. Planning for this to go wrong again, a wise use of time and money versus all the other problems you’re likely to deal with. Is this a black swan event that isn’t likely to happen enough to bother to build mitigations in for?
jerry: I, I, so it was standing in line for dinner, my wife and she asked me this question. Cause at the time a lot of flights were still canceled. And she asked the question and I didn’t have a good answer. What can companies do to avoid that? They prepare in some kind [00:52:00] of disaster recovery way to do that?
jerry: And the reality is, I don’t think you can. And so you could say you know what? I’m going to get rid of CrowdStrike and I’m going to go with Sentinel one or somebody else, but you’re taking the leap of faith that they won’t also have a problem or that they won’t have a problem that says that they’re.
jerry: They’re blind to some kind of attack that CrowdStrike could have seen.
Andy: You could duplicate your infrastructure with failover and run two different EDRs on each of the backups, but then so much more cost complexity. Are you going to run them as well? Are you going to have the mastery of two different vendors?
Andy: Like that introduces a whole lot of complexity. That’s easy to just say, go do it. But it’s very complicated for a maybe that happens. We talk about uptime and percentages. And we look at, I always want to say, again, I’m not a CrowdStrike defender. This is the funny part. I’m just, I’m frustrated with the thought leaders out there who are just blowing smoke on, just being, [00:53:00] pounding the table with, look how bad this is without putting it in context.
Andy: If we look at the percentage of how many of these, Channel file updates went out without a problem versus the ones that do. Is that a fair estimation of success versus failure rate? And we holding CrowdStrike to five, nine, six, nines, three nines, two nines, There’s no perfect solution. And by the way, the closer you get to perfect, the more expensive it gets.
Andy: So when we talk about like uptime and systems of being five nines, that’s a hell of a lot more expensive than two nines or three nines. So are we being unrealistic in, in saying that this should never happen and CrowdStrike should go out of business and okay, fine, then how much more are you willing to pay for a system that’s not perfect?
Andy: That never has this problem. And how slow are you willing to accept updates against new novel techniques, which is what they said they were pushing out to make sure this never happens? Because they’ve got this tension between getting these updates out fast versus doing all the checks in QA we all want to see.
Andy: So what are you willing to trade [00:54:00] off? Cost, complexity, time, risk, and the risk of if you don’t get the update fast enough, and then that self propagating ransomware hits you before you got it and Oh, sorry, we were in QA at the time. Are you willing to accept that? We just, we’ve got to be adults in the room and look at this with all sides of the equation and not just point fingers at somebody when they screw up without realizing the other side of the equation.
Andy: And again, I am not a CrowdStrike apologist here. I’m frustrated with the mindset of I’m going to build my thought leadership by just pointing out the bad things without ever balancing it against what the good is. Sorry, I’m a little frustrated.
jerry: It’s what we’ve built our industry around. I know.
Andy: Am I wrong? I’m not, am I wrong? Not to put you on the spot, but
jerry: The problem I have with the situation is that it reliably crashed every Windows computer it landed on.
Andy: I’m not sure that’s a hundred percent true. And [00:55:00] the only reason I say that as I’ve seen some social media imagery of a bunch of like check in kiosks at a airport and only one of five was down. I don’t know why. I have no idea why. It’s very flimsy evidence, but it’s I would say let’s get a bit more root cause analysis before we can completely say that’s true, but it’s obviously very highly effective, right?
Andy: And very immediately impactful to a super high percentage of the machines. This was installed on.
jerry: So I guess my question, my concern is. Did this happen as a, cause I’m assuming, and I feel pretty confident that they have a pipe, some kind of testing pipeline where before they push it out, it goes through some, some standard QA checks and I’m assuming what got missed.
jerry: Yeah. I’m assuming something didn’t happen and like it didn’t crash their [00:56:00] version of windows. In their test pipeline or did crash and it wasn’t detected or somebody skipped that step altogether. I don’t know, my concern lies there. Like what was the failure mode that, that happened? And, hopefully they’ve, hopefully they’ll come out of this better than they were in, in general, like we, as an industry, we we advance by kicking sand in other people’s faces.
jerry: That’s not
Andy: wrong. And there’s not, it’s by the way, I’m not trying to say, let’s just go up, up. Let’s suck. Let’s move on. Obviously we have to learn from this and we have to understand the implications, but I. And we have to adapt to it. I’m sure that every other vendor doing similar things is very curious what went wrong, right?
Andy: And hopefully we can all learn from it. Hopefully Construct will be very transparent. There’s no guarantee, but hopefully. And mind you, they’re going to get massively punished in the market for this as they should. There’s going to be a lot of people who are not [00:57:00] renewing CrowdStrike over this incident, and I have no problem with that.
Andy: And that’s that’s how the industry works. And there’s all, they, how they handle this incidents is going to be highly impactful to how well they keep their customer base. But they’re also massively highly deployed. So that’s, one people say they’re too highly deployed. I don’t know, 14 percent of the industry.
Andy: I think I heard somebody say, I don’t know how accurate that is. They’re one of the big boys in the EDR space. And frankly, having met a number of their people they know it and they have some swagger, so maybe that’ll knock them down a notch. Admittedly, when I first heard this, I was like, ah, couldn’t have happened to a better company.
Andy: But, the other aspect is Microsoft, man, they’re taking a bunch of crap over this because it was their systems, but it was really not their fault. As far as we know at all, but they’re trying to step up. Like they’re putting out recovery information. They’re trying to put out tooling.
Andy: Like they’re trying to help, which I appreciate, but [00:58:00] it’s certainly a mess. Don’t get me wrong. And I know I went down a ranty rabbit hole of just the stuff I didn’t like, because, partially because I’m assuming people who listen to the show know the details, right? So we’re trying to, there’s no reason for us to rehash all the basics.
Andy: It’s, trying to get into what we think, people care about that, but.
jerry: Yeah, I guess the net point is, shit happens. I think we have to be pragmatic in that EDR is. An incredibly important aspect of our controls. And I think auto update is as well. The whole point of this is to be as up to date as you can be because the adversaries are moving very fast.
Andy: Yeah. And we as an industry are not moving away from auto update AI is the antithesis of manual updating guys. Yeah. Buckle up. If you don’t like auto updating, you’re not going to AI much.
jerry: So we [00:59:00] have to find a way, I think, to to co exist and I don’t have a lot of magic words to say, and, like this happened to CrowdStrike, but it has happened, it happened to McAfee and it’s, it is, an incredible coincidence that the CEO of CrowdStrike was also the CTO of McAfee when that happened but it’s happened to Microsoft and it’s happened. I think it’s happened to Symantec and I think it’s happened to Microsoft multiple times.
jerry: Now I think the difference between those and this again, is the time proximity, like all of these things have all of these eight and a half million systems went down roughly at the same time And contrasted with a lot of the others, these were almost all corporate or, slash business systems.
jerry: Because you don’t run CrowdStrike for, actually Amazon’s been trying to sell me CrowdStrike out for my home computer, but generally you don’t run CrowdStrike on, on your your home PC.
jerry: You run them on, on, the stuff that you care [01:00:00] about, because it’s really freaking expensive.
jerry: And when those systems go down. The world notices and when eight and a half million of them go down all at the same time, it becomes big news. And lots of people consternate over it. And look, again, as an industry there’s so much clamoring for airtime and we love a crisis.
jerry: We love to talk about, why you shouldn’t be using kernel modules, why you shouldn’t have auto updates, why you shouldn’t do this, why you shouldn’t do that. It’s what we do, and it’s annoying. And I think it’s sometimes you can cross the threshold of causing more problems than you’re solving, because if you’re trying to solve a problem that may never happen again in your career by, ripping things out, it’s going to or duplicating, your EDR environment it’s just not necessarily cost effective.
jerry: So I don’t have a great. Look, this is, this has been an industry wide problem [01:01:00] and I don’t even think everybody’s fully recovered. I think there’s still people, those, I don’t know how many of the eight and a half million systems are still sitting there with a blue screen, but it’s not zero. I’ll tell you that.
Andy: Delta still canceling flights. This is just as one small, tiny example.
jerry: Yeah. I had packages delayed UPS is saying, your shipment is delayed because of a technology failure. So this is, it’s far reaching, but I think we have to be thoughtful and not knee jerk reaction to what happened here.
jerry: I think CrowdStrike has a lot to answer for. I do think. One of the quotes in this article is hilarious. I’ll read it here. It’s quoting the CEO, quote, the outage was caused by a defect found in a Falcon content update for Windows hosts, Kurtz said, as if the defect was a naturally occurring phenomenon discovered by his [01:02:00] staff.
Andy: I go back to probably about eight, 80 lawyers are all over every single sentence being uttered right now. Oh yeah. Yeah. You never
jerry: first, first law is you don’t accept responsibility until at least until more. Yeah. But they, look, they don’t I’m guessing the fog of war is thick over there right now.
jerry: Yeah. I’m sure they know what happened. By now. And like you said, hopefully they will be transparent about it, but
Andy: holy cow. It was interesting is if you get into their blog about the technical details, 409 UTC is when they put out the bad file, it was fixed by 527 UTC. So an hour, 21 minutes, 79 or 80,
jerry: 80 minutes.
Andy: Yeah. So that’s pretty fast, right? Obviously they knew they had a problem pretty fast. But it’s too late. Once it’s out there, especially because the machine’s down, you can’t push a fix to them. It’s a worst case scenario for them, in that they couldn’t [01:03:00] push out an autofix. Yeah, it’s yeah, as a side thing, we’re also seeing a lot of bad actors starting to jump on, trying to send out malicious content under the guise of helping with CrowdStrike issues.
Andy: So that’s always fun.
jerry: Dozens, maybe hundreds of domain names registered, many of which were some many of which were malicious. Some of which were parodies and yeah. Oh yeah. Yeah. Good times. Anyway not a lot happened on Friday or last week. Hopefully it’ll be another quiet week.
Andy: It is for you, unemployed bum.
jerry: I don’t, it’s interesting, I don’t know how I ever had time to work.
Andy: I, I think you need to put some air quotes around work, but sure.
jerry: Ouch.
Andy: All right, we have gone pretty long today, maybe we should wrap this bad boy
jerry: up. Yes, indeed. I appreciate everybody’s time. Time. Hopefully this was interesting to you. If you like the show, you can find it [01:04:00] on our website, www. defensivesecurity. org. You can find this podcast in 272 that preceded it on your favorite podcast app, except for Spotify.
jerry: I’m still working on that. And you can find Lurg, where?
Andy: I’m on x slash Twitter at L E R G and also on infosec. exchange. at L E R G, LERG.
jerry: You can find me at Jerry on InfoSec. Exchange and not so much on X anymore. And with that, we will talk again next week. Thank you, everybody.
Andy: Have a great week. Bye
jerry: bye.
Our TOPPODCAST Picks 
Stay Connected
