Venky Iyengar on Quality Engineering as the AI Trust Layer

Jeff Kavanaugh:

I’m Jeff Kavanaugh here at Connect, the Infosys conference here in Los Angeles. I’m with Venky Iyengar.

Venky Iyengar:

Hi Jeff, good to see you.

Jeff Kavanaugh:

Head of quality engineering at Infosys. Thank you. You know, I tell people that quality engineering always gets, not a bad name, but is misunderstood because it doesn’t sound exciting, and yet it's the foundation of almost everything that goes on. And what better area to talk about unlocking value from AI because so much of this value is kind of hidden or beneath the surface. What are you seeing out there with companies in the area of quality engineering when you apply AI to generate value?

Venky Iyengar:

Jeff, that's a great question. There are broadly two dimensions of how AI applies in the world of quality engineering. One is, how do you leverage AI to do quality engineering for the traditional types of testing that we do. Software and testing… Which are deterministic in nature.

Jeff Kavanaugh:

Deterministic.

Venky Iyengar:

Yes. But the second dimension, which is even more exciting, is how do you test this probabilistic AI-infused applications or systems? So, you know, it's a very interesting paradigm and it's a huge opportunity for all of us. It's, in fact, a tale of two cities. Using AI for QE to test the Deterministic applications are going to bring down our services revenue, but at the same time, testing AI systems to make sure that they are trustworthy, they are quality… of great quality, and they have idempotent characteristics that the enterprises need is super important. And that business is going to grow tremendously. So, as you can imagine, the biggest barrier in adopting anything in life is trust. So how do you trust AI? And that's where the quality engineering comes in and reshapes the entire...

Jeff Kavanaugh:

You took the words right out of my mouth because I was thinking while quality engineering might not be the first thing you think about, trust sometimes is. And trust is only earned because you believe in what’s behind it. What is one of the biggest benefits you've seen from AI developing that trust in the data.

Venky Iyengar:

Yeah. So there are, you know, when we look at what we have to look at from a trust perspective, we have critical thinking as humans to really see what could go wrong. Right? But, you know, we spend a lot of time typing in what is there in our mind using keyboard, typing things up. And we spend a lot of time. You know, that's the time that we can save. That's the...

Jeff Kavanaugh:

And you lose something in translation too, when you try to type something that's in your mind.

Venky Iyengar:

Yes, exactly. So now the AI has given us the power to, you know, focus on the critical thinking aspect and make sure...

Jeff Kavanaugh:

Critical thinking, judgment, ethics, empathy.

Venky Iyengar:

Absolutely. All the things that could potentially make sure that the systems are trustworthy. You can think about it. You can get the AI to do the task of typing and creating the kind of test that you need to do.

Jeff Kavanaugh:

You know, you just make me think, quality engineering is a much tighter connection to responsible AI than I thought.

Venky Iyengar:

Absolutely. We have done from my team, we have looked at all the failures of AI around the different enterprises, different business domains. So, we’ve looked at what can fail in an AI system.

Jeff Kavanaugh:

So, you’re looking at failure modes, Six Sigma, design experiments, all those tools still apply in this world.

Venky Iyengar:

Absolutely. We have created a taxonomy, sort of an anthology of what could fail, why it would fail, and how to make sure it doesn’t happen to our customers. And so we have created… the developers build these AI systems with some data in mind. But in real world, the data is very different. It's dirty, it's flowing, it's messy, ambiguous. So, we have created a system that will allow the enterprises to make sure that they can test under all edge conditions and they can make sure that the system that they can put into production is trustworthy, is of great quality.

Jeff Kavanaugh:

What’s an example of this at work at one of your clients?

Venky Iyengar:

So, for example, take the example of a local customer, one of our utility companies.

Jeff Kavanaugh:

Utility?

Venky Iyengar:

Utility company. They have built a utility adviser AI agent and when somebody calls the AI agent and asks why is my bill so high, now a human person would have taken several hours to dig through the details to find out what was the reason, but now there’s an AI agent that travels through different systems, makes the judgement and provides a response to the person calling. How do you know whether that is accurate? Is it because it was summer, is it because there were some guests at home, so you have to simulate all these conditions, and that’s what we did and prevented many of the wrong information going back to the customer and assured that the AI is doing the right thing that it is supposed to. So that’s a great example of how it is useful in the real world.

Jeff Kavanaugh:

That’s great for utilities. Now, within Infosys itself, has it been an effective efficiency tool?

Venky Iyengar:

It has been. I know if you look at what we did just a couple of weeks back. Within my organization, we looked at the top hundred clients and we looked at how people are leveraging AI to do the normal business as usual work. Roughly, believe it or not, about 25% throughput increase efficiency has been gained in the last about 12 months. So that means the same number of people are able to deliver more work for the customers than they were doing in the past.

Jeff Kavanaugh:

Well, let’s dig a little deeper on that one. What was it that caused that 25% improvement?

Venky Iyengar:

So, the team is able to build what we call as digital brain or context mesh. So, they look at the enterprise landscape, they have a lot of this tacit knowledge about how the systems work and so they are able to bring it all together in a knowledge graph for the enterprise, for their customer.

Jeff Kavanaugh:

So, you’re making knowledge graphs for the company and that's simplifying all their work.

Venky Iyengar:

Yeah, so that make sure that when we use AI, the kind of test that we develop, the kind of scripts that we build is very contextual to the customer and it is accurate. And it reduces the token consumption but it more importantly produces accurate results. Otherwise, you know, the AI can produce hundreds of tests which are not needed.

Jeff Kavanaugh:

Is that a good mix, the token consumption reduction? I mean that's got to be real money saved. Is that more or less than the other efficiencies?

Venky Iyengar:

It is, when we say token consumption reduction, what is very important is, it's important to reduce not only the overall token consumption, but to make sure that the input token versus the output token ratio is maintained. So if output token is more, then it costs more. Whereas if it is input token is more, the accuracy increases, but the cost is less. So you need to really look at… More input versus output.

Jeff Kavanaugh:

Yes. And your quality engineering approach is able to adjust that ratio in the right way?

Venky Iyengar:

It is. We are able to adjust it because of the knowledge graph that we have created that makes sure that when a person asks a question, it goes through the context mesh, is able to take good input, and then able to produce quality limited output.

Jeff Kavanaugh:

Using AI, are you now able to more easily create a knowledge graph than before?

Venky Iyengar:

100 percent. It's able to...

Jeff Kavanaugh:

Leading question, but I mentioned that because when Rafee and I co-wrote The Live Enterprise, knowledge graphs were still kind of new. And you can only do it for something really big, like Facebook had the social graph, LinkedIn had their employment one. But if we can make a knowledge graph for every company, wow, nothing like it.

Venky Iyengar:

Absolutely, we are right now working with two of our customers, where we have built these knowledge graphs already. Because as quality assurance people, we are the bridge between business and development.

Jeff Kavanaugh:

You view yourself as the bridge between the business folks and the developers?

Venky Iyengar:

Because the developers, they build components. Business knows the end-to-end picture. And there is somebody who is required in between to make that bridge connection between the developer and…

Jeff Kavanaugh:

So quality engineering also includes architecture and all the models.

Venky Iyengar:

Absolutely. We look at the architecture, we look at the code, we look at the standard operating procedures, we look at the whole picture to make sure from an end-to-end business perspective, nothing is missed.

Jeff Kavanaugh:

I feel better about quality engineering after talking with you. I have one question for you. What do you see in the next six to twelve months that companies should really look for as maybe the next horizon of improvement?

Venky Iyengar:

So we, from an AI for QE perspective, that means leveraging AI in doing quality engineering for deterministic applications perspective, we believe there will be hundreds of AI agents that will help us accelerate that process. But at the same time, if you do not build the context mesh, many customers are not there today, and they have to build that context mesh.

Jeff Kavanaugh:

So context mesh, you got to get it right, otherwise the agents are going to run wild. Right.

Venky Iyengar:

And they're going to drift in production.

Jeff Kavanaugh:

There may be an uprising.

Venky Iyengar:

Absolutely. And therefore, building a good control plane, creating an observability for how the tokens are getting consumed, how the agents are performing is going to be super critical.

Jeff Kavanaugh:

New times, new terms and new challenges, but sounds like a fantastic opportunity. Venky, thanks so much.

Venky Iyengar:

Thank you, Jeff.

Jeff Kavanaugh:

You bet. I'm Jeff Kavanaugh, until next time, keep learning and keep sharing.