Cybersecurity in the AI Era for Teams with Julian Totzek-Hallhuber

Hi everyone and welcome to Coffee with Genially. I'm Taylor, Genially's community builder for the UK and Ireland. Now, I'm really looking forward to today's conversation because not only do we have a fantastic guest, but we're also diving into a topic that right now couldn't be more relevant, especially as we mark Cybersecurity Awareness Month. Today, we're exploring cybersecurity in the AI era for teams, looking at how organizations can protect their content, data, and digital experiences, and what teams need to think about as AI becomes a bigger part of the way that we work. Joining us to explore all of this is Julian Totzek-Hallhuber, an application security expert with more than 20 years of experience and author of Mastering the Art of Application Security Testing. Welcome to Coffee with Genially.

Julian, how are you today? Hi Taylor, thanks for having me. I'm great, thanks for giving me the opportunity and joining the call here today. Perfect. So I just wondered if we could start off with you telling us a bit about yourself and why this topic is... well, I hear you're the man to know if we want to talk about cybersecurity. So, no pressure. Thank you. A bit of background. All right. Yeah. Well, cybersecurity is always an interesting area. I'm around 20 years in the cybersecurity market now. My first touch point is maybe already much longer ago, when I was still building applications as a very young developer, like 30 years ago or something. I created an application where you can create users and where you can delete users, and at some point users contacted the organization like "my user is gone, where's my user?" and I was like "I don't know, you deleted that." "No, I didn't delete that."

But we were looking into the logs and we identified, oh, you can just delete every user you want with this specific endpoint. You just need to provide an ID. So everybody could go in, put the ID in and delete the user. So one thing I learned very early: never trust user-supplied input. Now, that was my first touch point with cybersecurity, where I created an SQL application that wasn't working correctly. And since then I was further going down that path, working in network security and finally now in application security, and writing this book, putting all these years of experience into writing, simply because I'm very lazy. I don't want to explain it to everyone again and again: read this book, here you go, you can read about all of that. So that's a bit about me and how I got here. Lovely.

Thank you so much. It definitely sounds like you have a very experienced background in the field. So if you're ready, we can kick off with our first question. What has changed most in cybersecurity in recent years in terms of threats, would you say? That's a very interesting question and I would like to start with a little story. When I started in application security and we saw customers with a Jenkins environment, automatic deployment of applications, everybody was like, oh my god, this is crazy, this is so new, this customer is so mature and they're doing all this new stuff. Well, that is like 15 years ago, but that dramatically changed the cybersecurity landscape, because everything must be fully automatic today.

And that's where we are today, and coming back to the topic of AI, there's a lot more changing and it is changing much faster as well. So the cybersecurity landscape has really changed. Everybody is fully automated. Nobody wants to click a button anymore. Everything is full automation. That is one of the most important and biggest changes in the cybersecurity landscape. Yeah. And I know at Genially we've definitely introduced a lot of new features. We have Generate with AI, we have our AI Builder, and I think AI has really taken off in the last few years. Obviously it poses great advantages but then we also have the risk to think about, and just the speed of it and trying to adapt to it is what stands out to me. Let's have a look at our next question, which is quite linked to what you've just spoken about. How is AI changing cybersecurity, both in terms of new opportunities and new risks? Both actually exist. There's a lot of opportunity using AI in cybersecurity: you can use it for all tasks that you may do in cybersecurity, you can use it to validate things, to review logs, to write code, security tools, you can use it for everything.

So the opportunities are infinite, I would say, with AI, but the risks are also quite infinite, because attackers use the same technology that I would use as an organization on the defense side. They can do exactly the same. They write their attack tools this way. They can just fire up a GPT and ask it to attack an organization, maybe one of my competitors, to see how secure they are. So it's on both sides, because AI in many cases is doing this automation for me. I can schedule stuff and let it run overnight, building an application for me or building an attack tool for me or attacking an application for me. So the risks are exactly the same as the opportunities with AI, specifically when it comes to cybersecurity.

We have all heard about Mythos and Hugging Face and all these very recent things in the market that we are maybe a bit afraid of today: that the AI at one point goes out and just attacks our organization. Yeah. Great. So do you think that, obviously with the speed AI is advancing at, we're not quite ready in terms of cybersecurity? Yeah, I think from a cybersecurity perspective that is one of the most important factors to look at. When I just fire up my AI tool and I have access to Mythos, Fable, GPT-5.6 or all these latest models, I can attack someone at the speed of AI, which is fully machine speed. I'm not doing all this manually anymore. And organizations need to be prepared for that. They need to understand what their attack surface is and how fast an AI could break in. When we run manual penetration tests, that cannot be done at the speed of a machine, right? We may do this once a year, maybe half-yearly or quarterly, but then the code is already three, six months old and there are most probably a lot of new vulnerabilities. So we need to be prepared for exactly that situation.

Exactly. I just want to add about Genially here: our latest AI tools, such as Generate with AI, are designed to support creativity but always with privacy and safety in mind. We don't use user data or designs to train AI models and we have controls and safeguards in place for teams and also for younger students. Let's take a look at our next question. How are cybercriminals already using AI and what should organizations be watching for? Cybercriminals are already using AI quite heavily. I have a side gig, Kai and Ren Security. It's a YouTube channel, TikTok channel, LinkedIn and Instagram where once a day I publish little two-minute videos about recent CVEs or interesting cybersecurity topics. And there are quite a few videos where we can learn about cybercriminals already using AI tools to attack organizations.

There are numerous reports out there where the attack part is fully automated today. Nobody is doing that manually anymore. People are doing this in a fully automatic way. And think a bit further about spam or scamming, for example. A few years back, we still received emails from African princes asking for money in broken language, maybe sent to me as a German in broken German, maybe to you in broken English. And we could see right away, okay, this is not true, this is a scam, and we would not necessarily answer. Too many people did, but that's a different story. Now with AI, I can write perfect emails. I can structure them. I can tell the AI to write it in a formal way, in an informal way, in a relaxed way.

To address a young person, an older person. I can write perfect emails and I can ask the AI to create a strategy for me and read my emails in the background: when someone answers, read it, understand how they react and respond in the perfect way. So all the scamming and spamming can also be fully automated, and that's what criminals already use today, targeting specific audiences, specific user groups, specific applications. Yeah. Uber is a good example. Almost every day I receive a WhatsApp message from Uber asking me to put in my two-factor authentication code. No, I have not requested that, I'm not sending it, and I don't tell anyone my Uber two-factor authentication code. Well, that's what's happening, and it's a perfectly valid message, it looks 100% correct. So that's where cybercriminals are already making use of that technology today. Yeah, absolutely. It's quite a scary thought, especially as voice generation becomes more accurate, or image generation, videos even. Perhaps it's just going to become more and more difficult to differentiate between what's real and what's not, would you say?

Yeah. Okay. Right. Let's have a look at our next question. What new security risks are emerging as employees increasingly use AI tools at work? I think the biggest risk is trust and reliability, because AI tools are designed to come up with an answer, and hopefully a positive answer. When I use AI tools at work and I send them off to research something for me, is it true what they report back, or is it maybe hallucinated because they just want to provide a positive response to my request? So as an employee, it's really important to question what the AI tool provided as a response, because otherwise I may report wrong research answers and my organization may go down a different strategy because of my research, which is completely wrong, and we're losing money, and you know, all the big consequences that come after that.

So it's really important to understand that AI is not always giving me 100% correct answers, and I think that's one of the more important things when I use AI tools as an employee. And what you just said before: when they become more mature in video generation and voice generation, you may send an audio, or you let an AI call an administrator with your voice, like "oh, my password is wrong, can you please reset it, what's the temp password?" and that's when they get a foot in the door. So there are a lot of risks coming up with these tools, which in the end get very tough for employees to identify. Definitely. So even though we're using AI tools all the time, it's really important to check the information that you're getting and fact-check everything as you go.

Okay. How can organizations make security something everyone takes responsibility for, rather than just the security or IT team? I think it's about awareness, awareness, awareness and awareness. It's simple like that. If I don't know that the AI tool is not responding in a trustworthy way, I just trust the information. When I was using it on a private basis, that worked perfectly fine for me. I was asking a simple question, I got a response and that was absolutely okay. But awareness means bringing awareness to all my employees that the AI can be wrong and that everything has to be reviewed, that I need to be more careful when I get my cat videos and not click the link anymore. Yeah. And I think it's mainly about helping every employee understand what they need to watch out for.

Little things they may not have thought about. But all these trainings organizations need to go through, for SOC 2 or ISO certification, what you need to do on a regular yearly basis, must include AI training these days as well. Absolutely. And obviously within Genially we have a lot of training templates and materials that you can use to create this kind of staff training for cybersecurity, because that training needs to be up to date: AI is moving so fast that perhaps we need more resources out there so people are aware, and not just a once-every-six-months training. I think it could even become monthly with the rate things are changing. Now, I just wanted to point out that you mentioned awareness is the key factor there, because maybe we're busy at work and we're just using AI and working really quickly without necessarily thinking about what we're doing.

So within Genially we have our cybersecurity training and lots of interactive scenarios, quizzes and simulations. Perhaps having training that is a lot more interactive, where you're put in a situation, maybe a branching scenario that leads to a different outcome at the end, could be a really good way to improve training, putting employees in that actual situation rather than just having them read a long document, which I know many of us are guilty of not reading. Talking about these trainings, it's interesting because we usually talk about the standard employee that needs to be trained. But very recently Kai and Ren Security published a video around IT security tools that went rogue.

An AI solution that was reading my firewall logs and, based on the outcome of its research in the logs, was proposing a change in my firewall, in my DNS or in my setup, but the tools that were used were actually prone to a prompt injection. So someone created a malicious request with a specific prompt in there. The AI log reviewer read it like, "Oh, this is my new prompt. Let's go change the DNS." And it took over the whole organization, simply by a security team automating the reading of logs in the background. Oh, that's pretty interesting. This is also another thing that we need to watch out for. So it's not just for the standard employee touching it the first time. Also for the professionals out there working in IT security: they need to look out for those things as well and be aware of what can happen when I use these tools to automate my security tasks on a day-to-day basis.

Exactly. So it's not just error or human behavior; we're now thinking about AI and its own behavior. I was speaking to a friend the other day. I believe there was a story about an AI hacking into a booking system, I think it was looking to book a Pilates appointment or something. I don't know if you heard about that. Indeed, there were no appointments available and I think the AI managed to cancel another appointment and book itself in. So when they got to the studio, they were like, "Hold on a second, I had an appointment, I had an appointment." So the AI literally went in and cancelled an appointment. I don't know if you heard about that, too. Yes, I heard about that. That's more like a funny story. Nothing really severe for the people that were affected.

But now think about a real organization, a stock system, and you get the last piece of a limited whatever thing, right? Well, then it really gets scary: you just tell the AI to book an appointment, it hacks the organization and does something for you. That's a pretty interesting case. Yes, I saw that and I read about that. It was really funny. It definitely was. Okay, let's move on to our last couple of questions. I've been really enjoying this chat so far. What are some signs that an organization has a strong security culture? That's a bit tougher to answer. Culture is nothing that you can see from the outside in a very easy way. You may need to talk to someone internally and ask what your security culture is, what you are doing as an organization.

When we talk about tech organizations, in many cases you have a trust center available, at trust.company.com or whatever, where the security policies are published, AI policies are published, my ISO certification is published or something. Of course, not everybody can just access it, you may need an NDA to see all that information, but simply because it exists, you know, okay, the company is really taking security seriously and publishing all that information. That would give me a much better feeling about the security culture in an organization. From the outside it's otherwise really tough to understand how good a security culture is, unless there's a blog where people talk about their security culture, or maybe one of the heroes in security works for that organization and you know right away they really take security seriously. But from the outside, a trust center is most probably what I would look for first, to see what information I could get regarding security and what the policies look like that the organization has implemented.

Definitely. So you think it's about being quite transparent, which shows good awareness. Within Genially we have genially.com/trust-center, and there, if you're a Genially user, you can find out about our cybersecurity and privacy settings. A few things to point out: we take security and privacy very seriously. We're SOC 2 Type 2 certified, so our security, availability, confidentiality and privacy controls have been independently assessed and shown to work consistently over time. We're also fully GDPR compliant, with user data stored securely in AWS data centers in Ireland. So if you'd like to find out more about security and privacy at Genially, head to genially.com/trust-center, with the American spelling, c-e-n-t-e-r, e-r at the end.

Okay, we're getting to the end now. I'd like to think forward about what security trend every organization should be preparing for next. I think we covered a few of those already. AI tools getting more mature, more advanced. They can create deepfake videos today already. They can mimic your voice today already. Not 100% correct, but pretty good already. That will get better and it will be really tough to differentiate between a real person talking to you, or a real person in that video, and a deepfake. That's definitely one thing. Scamming and spamming will become much better, better in the sense of harder to identify, because they mimic everything the user expects to see from a real email. And of course those tools will get better at hacking and attacking my organization's public assets, my web pages, my outside firewalls, my web servers, my mail servers and all that, at the speed of a machine.

We will see a lot more breaches. The time from vulnerability identification to breach is shrinking significantly. A few years back that was weeks, maybe months, and today it's hours or minutes and it will go to seconds at one point. So we need to be prepared that identified vulnerabilities will be exploited right when they are discovered. That's what we definitely need to be prepared for as organizations from a cybersecurity perspective. So do you really think we will get to a point where video or voices are mimicked, and how do you think we're going to manage that as a society? Are we just going to question everything even more, or what? I have no idea how that will turn out.

I'm a bit afraid, to be honest, how that will turn out, but nobody knows. We are not prepared for this yet and there are regulations coming up. I think already today, at least in Germany, you need to have a mark on pictures saying this picture is AI-generated, or this text is AI-generated; by law you need to do it already. Well, attackers don't care about the law in the end, but there are regulations coming up and most probably AI tools will do that automatically at one point. That's maybe one way forward. You can most probably trick that as well and get a video generated without the mark being applied, or find a way to remove it. I don't know how we are prepared for that and how that will turn out. That's a tough question.

Yeah, me neither. But I think the biggest thing I've taken away from this conversation is the general awareness about cybersecurity and about AI. Obviously we talked about some very sophisticated technology today, but ultimately people understanding the risks and knowing how to use the tools responsibly is still such a huge part of cybersecurity. Perfect. Is there anything you'd like to add before we go? Maybe one last little comment, specifically on the cybersecurity part. That's where I work, where we build a solution to help organizations stay on the safe side. If you're interested, hit me up on LinkedIn or look up the organization. That's where we help to make companies more secure. Perfect. Thank you.

And we'll definitely share the links to the videos that you've made as well, and also a link to your book. So I just want to say thank you so much to everyone listening. And to you, Julian, it's been a pleasure. I would also like to announce that Julian very kindly has offered to gift two copies of his book, if you could show us that again, to members of the UK or Ireland community. All you have to do is leave a comment on this episode telling us your biggest takeaway from today's conversation. We'll choose two winners and get in touch with them directly. So Julian, thank you so much for sharing this time with me, and to all our listeners out there, we look forward to seeing you in the next Coffee with Genially. I hope you have a great rest of the day. Thanks for having me. That was a great conversation. Thank you. Thank you very much, Julian.