Qalqan sector 10
Human & AI Cognitive Synergy
Protecting people, businesses and digital infrastructure from the risks of developing, deploying and using artificial intelligence: AI agents, large language models, autonomous systems, corporate AI services and generative AI.
Key threats
- Leaks of confidential data through AI services
- Trade secrets handed over to public models
- Prompt injection and manipulation of AI agents
- Unauthorised actions by autonomous AI agents
- Wrong or fabricated model answers
- Deepfakes, voice cloning and synthetic identities
- AI phishing and automated social engineering
- Tampering with or poisoning a model’s data and knowledge
- Use of insecure third-party models and AI services
- Excessive permissions of AI agents
- Wrong automated decisions
- Breaches of confidentiality and personal data protection
- Uncontrolled use of AI by employees — shadow AI
- New AI attacks on information systems
- Reputational, legal and operational risks of using AI
Typical tasks
- AI security audit of an organisation
- Audit of employees’ use of generative AI
- Audit of AI agents and their permissions
- Corporate policy for the safe use of AI
- Classification of data that may and may not be shared with AI systems
- Threat modelling of AI systems
- Review of AI solutions before deployment
- Red teaming of AI models and AI agents
- Testing resilience to prompt injection
- Human-in-the-loop for critical operations
- Managing AI agents’ access to corporate systems
- AI incident response procedures
- Training executives and staff in the safe use of AI
- Assessment of AI solution vendors
- Expert assessment of new AI threats
See also
Qalqan sector
Information & Cybersecurity
Digital threats: from phishing to AI-driven attacks
Open sectionQalqan sectorIntegrated Technical Security
Technology that closes a risk instead of simulating protection
Open sectionQalqan sectorPsycho-emotional & Mental Security
Resilience to manipulation and psychological pressure
Open section