AI Governance and Board Accountability: Jeff Saviano on Responsible AI

Jeff Saviano:

We've been working with many companies to understand where they are on their AI journey have them ask that one question of their leader teams. What would we not tolerate from our AI systems? What won't we tolerate? And I was with the board of a big bank just last week and they had not addressed that. We have to be sure that companies aren't developing AI systems that are harming the world, harming their stakeholders. What is the red line that we will not cross? As companies do that and ensure that that learning then transfers to the teams that are building AI solutions, that is one outcome that we're particularly focused on.

Jeff Kavanaugh:

A few months ago, I was on a stage in Naples, Florida, talking about what it means to generate insights with empathy. The person who put that evening together is my guest today. Jeff Saviano has spent the last several years building a governance framework for ethical AI, starting at Harvard, advancing it at MIT, and now publishing it with Wiley under the title Boundaries of Tolerance. If you're a board member, C-suite leader, or anyone accountable for AI deployment, this conversation is for you. Jeff, it's great to have you on the Knowledge Institute podcast.

Jeff Saviano:

Thank you, Jeff. It's so good to be here. So good to see you again.

Jeff Kavanaugh:

Your book opens with a sharp question. What happens when boards and executives are asked to integrate one of the most transformative technologies in history without the frameworks or regulatory clarity to lead responsibly? Let me ask, how bad is this gap between where AI deployment is and where the governance required?

Jeff Saviano:

I appreciate the question, Jeff. I think the gap is stark. We've seen this directly with companies. They're racing to produce new AI systems. Sometimes they're for internal efficiency gains. Oftentimes they're looking for a competitive edge. And as they're racing forward like that, let's say they're often not leading with governance. We've seen it too. If you believe in the Gartner hype cycle, I love how Gartner tracks the trajectory of emerging technology adoption. And they show both generative AI and agentic AI as, of course, making significant inroads, but the governance is lagging. So we get it through views from Gartner, but we get it importantly from companies. And because of that fear of obsolescence, they're afraid of being left behind by their competitors. So they put the edict out in companies to build something fast and get it out into the world. And I think that's creating the gap.

Jeff Kavanaugh:

Before we get into the framework, I have to ask about Rome. You spoke on AI ethics at the Sala Regia, at the Vatican's Apostolic Palace. That room has held many consequential forums over the centuries. What does it say to you that AI governance is now being discussed there and what was the room's reaction?

Jeff Saviano:

As you said, it was in the Sala Regia, and that's translated as the Regal Room. It was commissioned by Pope Paul the Third and so it was completed way back in fifteen seventy-three.

One quick anecdote about the day. When I was in the room, looked to my right and we noticed these incredible doors. They must have been 50 feet tall. And I asked one of the members of the Vatican who were there, he said, that's the door that goes to right to the Sistine Chapel.

So it put it in perspective, I think settings matter. I think it matters where we have these conversations. We often bring companies onto campus. We've done it at Harvard, we've done it at MIT. It matters. It gets people out of their regular environments and I think it can lead to a more conducive and important conversation.

And so having the session at the Vatican, I'm a lifelong Catholic. Before I went, my Irish Catholic mother said it's the most important presentation you'll ever make. So that that raised the stakes for me a bit.

But I could feel it in the room, could feel it from people who were present that they realized how important AI systems and consequential the outcomes are for humanity, not just about enterprise risk management, but about humanity. And to hear we open the session with a proclamation from Pope Leo, delivered by a monsignor in the church. And that set the stage for everybody to understand that the stakes are high.

And so I was so appreciative of the opportunity to gather there to contribute to that conversation.

Jeff Kavanaugh:

You anchored the book in fiduciary law. Caremark, Boeing, McDonald's. Most AI ethics conversations stay in the abstract. Why did you decide that legal accountability was the right way to reach boards and executives?

Jeff Saviano:

We started this work and we heard from so many boards and so many business leaders, Jeff, that you need to meet us where we are. Start where we are. We often start, this is a board director perhaps speaking, saying that we often start with our fiduciary duties.

Grounded, there was an important decision back in the 1990s involving Caremark. And what came out of that is a very what seems like a very simple rule is that boards need to have an information system. How does information flow up to the board? And also, if there's a red flag, they have to act on it.

Since that case, we've had other key important decisions involving companies from Boeing to McDonald's, and they help to guide what the fiduciary duties are in a variety of contexts. What our team did... our lead researcher at the time was a grad student at Harvard Law School. We spent the first six months of our work diving into those cases and applying the principles and the holding of those cases to AI systems and what it means for boards to govern it.

We've got a whole chapter in the book that's dedicated to fiduciary duty. And I think it's an important aspect. We hear from so many business leaders, we want to understand what this means for our fiduciary duties first.

Jeff Kavanaugh:

Well, it also sounds like you're going beyond the hype and what people fear and dealing with something that they can actually be held responsible for. And it's also in the same kind of language they're used to dealing with other parts of their responsibility.

Jeff Saviano:

There was a case I mentioned the case involving Boeing, and Boeing is such an important decision. I don't think enough companies and boards are talking about it. The board was sued, it was a shareholder derivative suit. And the holding though was so important. The court said that they agreed that the board could have done more to protect the shareholders. How often these cases begin, shareholders are suing the board for a reason. The court went beyond that and said, we also don't think you did enough to protect your stakeholders. In that case, stakeholders were passengers, were people on the ground.

And what's important that came out of it, I think those key principles, you can draw a straight line to AI governance. We have to be sure that companies aren't developing AI systems that are harming the world, harming their stakeholders. And I think it's one that as we talk to dozens and dozens of boards, they need to connect those dots and ensure that they're looking beyond just the bottom line and ensure that they're also thinking about stakeholder rights as well.

Jeff Kavanaugh:

Boundaries of Tolerance is a deliberately precise phrase. Walk me through what it means and why you use that framing.

Jeff Saviano:

We stole the terminology. Let me just start right from that point that I did not develop that phrase. I kept hearing it on campus. And as you said, this effort for me started with an appointment at the Harvard Safra Ethics Center three years ago. And in the travels around campus, we heard this terminology. And it has its roots in bioethics.

And we thought that the words matter and it's directly applicable, the importance for businesses to determine the guardrails. What are the guardrails that they will put on these AI systems? And the reason, Jeff, that we think this is so important is because government regulation will be lacking. One of the first places our research team went, we chronicled government examples of how they either regulated or in many cases decided not to regulate emerging technologies. And we found that that regulation is often thin. Governments are afraid, they don't want to stifle innovation. We kept hearing that terminology. We don't want to stifle innovation within our countries.

So as a result of that, regulation, it's my own belief, will never have a globally consistent, robust set of AI regulation. And that puts more pressure back on boards to determine what the guardrails are within their organizations.

Jeff Kavanaugh:

The first pillar of your model is the ethical maturity stack, a progression from noncompliance at the base all the way to ethical vanguard at the top. When you work with organizations, where do most of them land? And are leaders honest with themselves about where they really are?

Jeff Saviano:

That's a hard question. Well, let me first say that we developed the stack because companies needed a common language. They needed some way that they could determine what is the extent of the maturity of ethical practices from an AI context. And we looked and looked for frameworks and we couldn't find one. And so we decided to develop it.

In fact, if it's okay, can I just tell you a quick story about how this came about? We had 25 business leaders on campus at Harvard Business School back in the summer of 2024. And as we were getting started, Jeff, you've been to hundreds and hundreds of conferences like this. We had an icebreaker, people were explaining where they were from and what their roles were.

And we had a director of one of the biggest banks in the world, stood up and said, I'm here for a specific reason. We had an AI solution team come to the board because they were stuck. They developed a solution that was biased against 5% of the population. And they couldn't fix it and they didn't know what to do. They went to management. Management wasn't sure. They said we have a board. Boards should address tough questions. So they brought the issue to the board. The board listened. Then they went into an executive session. Immediately, one of the directors said, well, it's only five percent. What do we care? It's immaterial. Thankfully, other directors said, well, wait a minute. If it's we have a solution that's biased against even one person, that's a problem.

The important part of the story though, Jeff, is that they were stuck. The board was stuck. They didn't know what to do. And the light bulb that went off for many of us at that session, where frankly was the origin of this framework, is that what that company hadn't done yet, they were ill-equipped to address. This is a high-functioning board, but they were ill-equipped to address the intricacies of an ethical issue of an AI system because they hadn't first done the homework to determine how do you align the company values to the system? What will we tolerate and what won't we tolerate? And so that's the origin for how we came up with Boundaries of Tolerance.

Jeff Kavanaugh:

Speaking of other phrases, two that jumped out at me from the book guard bands of safety and prudent vigilance. They sound deceptively simple. What do they require of a leadership team in actuality?

Jeff Saviano:

Yeah, these are emerging practices. So in addition to the ethical maturity stack, we chronicle three leading practices and three emerging practices, which you've highlighted, Jeff. Those are two of the emerging practices. Let me start with guard band of safety.

Guard band of safety comes from a practice that was developed by Intel, company Intel. They were really, really concerned about antitrust litigation. They were so concerned about it that they instituted this policy that they call by this name guard band of safety. Think of it like a super compliance layer. They were so concerned about tripping and about being sued for antitrust purposes that they instituted this guard band to ensure that they would never come close to the compliance layer.

And when we saw it, we thought that it was a terrific system that we could apply to AI solution development. If you're concerned about the outcomes from these systems causing harm in the world, put a guard band in place as regulation is still thin and developing around the world, put a guard band in place that falls short of where the current compliance obligations are, to be sure that your systems will never cross that threshold.

I think it's a great metaphor, but also practice that Intel use for a very, very different purpose, but I think it applies here. So that's the first. That's guard band of safety.

The second, prudent vigilance really spoke to me. It's a principle on how to approach this tension between innovation and risk management. I've had lots of innovation roles in my career and I think it's inherent for innovators that ultimately they'll have some tension with risk management.

Innovation is about pushing forward and growth and developing new systems. And of course, risk managers are protecting the organization. What had always concerned me about that is that was always viewed as a very binary question. Do we move forward with a new innovative solution? Or do we not? Do we assume that there's too much risk involved? It was either a yes or no question. What prudent vigilance does is offers a middle ground. And it's a practical approach when the innovators are fighting with the risk managers and vice versa. It offers a middle ground. And what the middle ground says is to move forward, but put guardrails in place, continuous oversight.

When we studied, and one element of our research, we studied dozens and dozens of AI failures. Many of those failures happen after deployment. There are breakdowns in governance after deployment. You understand what happens. You develop a new solution. Oftentimes the teams get excited, they pop the champagne corks, and they move on to the next project. What prudent vigilance says is, well, hold on. We're going to launch it. We're going to let you go. We're going let you move forward, but we're going to put guardrails and oversight in place. So you're watching what happens and ensure that there are no harms that are caused by the system.

So, I think there are many who will be watching this podcast and listening to this podcast today that will have faced that binary question, yes or no. And it doesn't have to be so stark. There is a middle ground, and that's what prudent vigilance is all about.

Jeff Kavanaugh:

What about speed versus accountability and maybe transparency versus privacy? In your experience working with executives, which of these has been the most difficult? And any particular comments on them?

Jeff Saviano:

This has been one of the hardest aspects of the research that oftentimes these ethical issues with AI, you're dealing with competing priorities and they're both compelling principles. The one that has, I think, caused the most consternation within companies is the tension between explainability and the robustness of your AI system.

Of course, many companies want the most robust. They want the strongest algorithms. They want the best AI systems. Oftentimes, though, those are black boxes. We don't know. We don't, we can't explain the outcomes. Nobody can explain the outcomes, but we know that the outcomes are powerful. And so it creates this tension, this tension with the ethical principle of explainability.

If an algorithm, if an AI system is recommending a course of action, then many people would say that they adopt the principle of explainability. They want to know how it reached that decision. Sometimes the law says you have to explain it. The truth in lending law is not an AI law, but applying the truth in lending law to financial services, if an AI, if an algorithm was recommending that somebody should be approved or denied of a loan, they need to know why.

So I think that's a board-level decision. Which do you want? Do you want the most powerful AI systems, or do you want to be able to explain the outcomes? I don't think oftentimes you can have your cake and eat it too. And I think that it really invites having a conversation like this at the board level, the senior management level, to determine what are the values of our organization? Do we demand explainability? If so, let's be upfront about it and let's recognize we may not have the most powerful AI systems in the world.

Jeff Kavanaugh:

You built this framework at Harvard's Ethics Center, then advanced it at MIT. What was the hardest challenge in moving from the rigorous research of academia to something a board could use?

Jeff Saviano:

I think for me at the time I just retired last year from a 33-year career in consulting. I've spent my life, my business career as a consultant helping companies with their business problems and with opportunities. So the heart of this, the reason I took this appointment, Jeff, was because there was no framework. We were working with companies every day in the my last 18 months in the consulting firm we met with over a hundred boards and C-suite teams and we kept hearing the same refrain. We needed a framework. We needed guidance. So that's what was fueling our work. That's why I mention the origin of the framework.

We brought companies in. We would regularly bring companies onto both campuses, MIT and Harvard, to hear from them, to test it. We've been testing this in executive ed sessions and testing with companies that come on campus and that has grounded this work. In fact, we had a academic report ready to release to the world and we decided to halt it and to convert it into this more traditional nonfiction book because we wanted to reach more people.

Jeff Kavanaugh:

The risk that you catalog, biased algorithms, reputational damage, shareholder litigation, loss of public trust, you know what a list. They're already visible. But organizations keep getting caught flat footed. What's the root cause? Is it awareness, incentives, capability, or something else?

Jeff Saviano:

I think awareness can still be somewhat of an issue, although frankly, I'm not so sure it's the root cause anymore. Business leaders very quickly they are realizing that AI has inherent risks included, the gap is shifting from basic knowledge to the need for some actionable governance systems and structures.

I think part of the issue comes from that what's different about this AI, this AI is ubiquitous. There are so many powerful free systems that oftentimes employees are using AI tools and their leaders may not know how and if they're actually using it. I think that's creating a lot of tension on governance. It's creating tremendous concern within companies when AI use perhaps is unsanctioned. Some people call it shadow AI. So that's causing lots of issues, this decentralized adoption of AI systems.

But another issue is the lack of accountability. I used to have a boss who would say when we would launch a new project, she would say, okay, now I need to know whose throat do I choke if something goes wrong? And it always made me uncomfortable just hearing those words. But now I get it. It's hard to have accountability by committee. And many companies are forming these cross-functional AI teams, and that's important because of the reach within the organization. But when you do that, it's so important. Who is that one person who is accountable if something goes wrong? And I think that still has created issues within companies that haven't quite established those clear accountability roles and responsibilities across their leader teams.

Jeff Kavanaugh:

You recommend establishing board level technology or AI committees, oversight committees. But a lot of boards are still catching up to just cybersecurity. What does a well-functioning AI governance committee actually look like? What's the single most important thing it needs to do?

Jeff Saviano:

This has been perhaps a bit controversial. And as I've talked to so many leaders about this, I think most advisors would say the answer to the question, do we consider adopting an AI or technology committee at the board level? Most would say it depends. I don't think that's the case. I think almost universally, of course, there's always some exceptions, but I think almost universally the answer should be yes.

One of the reasons why. A clear outcome from going deep into the fiduciary duty cases that we talked about earlier was that how so many courts, when evaluating if boards were liable for certain risks that materialized, they will comment on if there was a committee at the board level that was actively looking at the issue. So the courts are very focused on this. If you do it for no other reason than to help ensure that you can successfully defend a lawsuit down the road. That's perhaps one reason.

But more importantly, it helps. You can, one example, you can bring experts in from outside the organization. You can you can use an existing committee. Many companies have asked their audit committee to assume that responsibility. I love this structure. I've seen this in three different places. Form an AI or a technology subcommittee to the audit committee. Don't take it away from the audit committee. But form a separate committee, give it some separate life, shine a bright light on it, bring others in, ensure that the organization knows any new AI deployments and risk will have to come to that committee before it gets to the board. I think it's important.

One caution. The last point I'll make on this, Jeff, I've seen this happen in a few places. Companies will form a technology or AI committee at the board level, they'll let it operate for a bit and then they'll shut it down. The reason they'll shut it down is that if a board director is not on the committee, sometimes they'll check out of the conversation. The average age of a board director right now is 67 years old. Many of them are not digital natives. Sometimes if they say, well, somebody else has it handled, that's probably not the best answer for the board. So you've got to be careful and ensure that it's not replacing every director has a responsibility to understand technology and understand how AI is impacting the company.

Jeff Kavanaugh:

Here's a challenge. Can you describe a company that's doing AI governance right? Not just for compliance reasons, but leading. What distinguishes them?

Jeff Saviano:

Yeah, that's a great question. We've seen some of these same characteristics. I've mentioned a few of them already. I think number one, there's a functioning AI inventory. They know where it's operating within the company. And this is another distinguishing feature of this AI, of generative AI, agentic AI systems, is that that oftentimes it can be driven directly by employees. And many times leaders don't have their a grasp on where AI is even being deployed within the company. So number one, that they know where AI is operating within the enterprise and what the risks are associated with that.

Another is that there is clear ownership, there's clear accountability. You've defined who is accountable for different systems. And if there's handoffs, that those handoffs are clear. It seems simple, but it doesn't happen all the time.

The continuous monitoring. New solution development teams that that tend to walk away. And I think I've seen really successful boards shining a brighter light on systems once they're deployed for that reason. They're worried that people will take their eye off of the ball.

And then lastly, I want to talk a second about the ethical vanguard layer. You mentioned it earlier, Jeff. It's the highest level on our ethical maturity stack. To reach that level, it's not just about your company. You're influencing your peer group, you're influencing the world, you're contributing to standards that may exist. You're working with regulators. You're treating governance as part of your competitive and reputational positioning. Ethics of AI becomes more of a strategic asset because of how focused you are on it. I think that's important. And we've seen just a few of these companies but operating at that layer where it's not just about you and your organization, but you're contributing to others, that's a sign that you're doing something right.

Jeff Kavanaugh:

We shared a stage at TEDx Naples a few months ago. You organize an event, it's a fantastic event. And I spoke on this concept of insights with empathy. The idea that analysis without empathy misses some of the most important things. I'm curious whether that's relevant in the governance context about how do empathy dimensions, you know, stay prevalent?

Jeff Saviano:

Yeah. I thought a lot about that, Jeff, as you and I got to know each other and your excellent, excellent talk. It's been so much fun to see how it is influencing the world. The importance of empathy.

And I think even in the conversation that we're having today, the importance of understanding that there are real people on the other side of these AI systems. There are people whose loans are not being approved, people whose health decisions are being made and recommended by AI systems. AI systems determining whether the organization should hire them or not. There's real people on the other side.

I felt that, I really felt that at the Vatican. I thought I understood it, frankly, before I went to Rome, but it was that was so impactful to me and to others who were there to understand that that's the focus of the church. It's the focus of many. To ensure that we're protecting humanity.

And especially you mentioned the recent news about Anthropic, and we know that we don't have globally consistent regulations yet, so the burden falls on companies to ensure that they understand and act on that.

Jeff Kavanaugh:

Jeff, one last question. What's one thing you'd want a business leader listening to this to do tomorrow? And what are you doing to keep evolving your own understanding of this rapidly changing area?

Jeff Saviano:

We've been working with many companies to understand where they are on their AI journey have them ask that one question of their leader teams. What would we not tolerate from our AI systems? What won't we tolerate? Have they had that conversation? And I was with the board of a big bank just last week and they had not addressed that. And it was so interesting to sort of raise that and then take a step back as an outsider and just watch the board address those questions. And not at the superficial level, not at the level of we're against discrimination. Of course, you're against discrimination, you're against bias. But we hope the impact that our framework will have is that it provides these different lenses to look through to determine where the organization stands today from, for example, leadership priorities, how it works with their stakeholders, how it works across the ecosystem.

So I would have them ask that one question, what won't we tolerate? What is the red line that we will not cross? As companies do that and ensure that that learning then transfers to the teams that are building AI solutions, that is one outcome that we're particularly focused on.

As far as what's next, we are actively trying to get the word out. So thank you again. Opportunity to come on your show today. We want to get the word out to people as our book it's available now on Amazon, but it doesn't ship until October. And so, it's our aims that we want not just boards but senior management teams, really anybody that is working with AI systems within public or private organizations to take this framework and apply it to determine what those guardrails are. Lots more work to do in this space.

We've just got going deeper into agentic AI systems, and we're finding it's putting even more pressure on ethics. Of course, the nature of agentic AI is that you can have less human eyeballs on it and less human actors, and that puts tremendous pressure ensuring that we have enough human oversight of these systems. And so that's one of the areas that our team is now focused on.

Jeff Kavanaugh:

Jeff Saviano, author of Boundaries of Tolerance, published by Wiley, available for pre-order now. If you lead an organization that's deploying AI, and at this point that's nearly everyone, this book is the governance grounding you've been looking for.

Jeff, thank you so much for joining us. Thanks for putting together an evening in Naples that none of us will ever forget.

Jeff Saviano:

Thank you, Jeff. Appreciate the opportunity to have this conversation today. Thank you so much.

Jeff Kavanaugh:

Yulia De Bari and Christine Calhoun produce this podcast. Dode Bigley is our audio engineer. I'm Jeff Kavanaugh from the Infosys Knowledge Institute. And until next time, keep learning and keep sharing.