Security competency assessment and tracking

You can't improve what you don't measure. Before launching security training, you need to understand your team's current skill level. After training, you need to verify it worked. Ongoing assessment ensures skills don't decay.

This article covers practical methods for assessing developer security knowledge: quizzes, code review exercises, CTF competitions, and building competency maps that visualize team strengths and gaps.

Three-part series

This is Part 2 of the developer security training series:

1. Curriculum and resources — what to teach
2. Assessment and competency mapping (this article) — how to evaluate skills
3. Implementation guide — how to roll out the program

Why assessment matters

Identify gaps before training. Generic training wastes time on topics people already know. Assessment reveals what each developer actually needs.

Measure training effectiveness. Did the training work? Compare pre- and post-training scores to know for sure.

Track improvement over time. Security knowledge decays without practice. Regular assessment catches skill regression early.

Justify the investment. Leadership wants to know if training is working. Assessment provides concrete metrics.

Motivate developers. Clear competency levels give developers targets to work toward. Improvement is visible and rewarding.

Assessment methods

Method 1: Security knowledge quiz

The simplest approach. Create multiple-choice and short-answer questions covering key security topics.

Quiz design principles:

Principle	Why it matters
Role-relevant	Don't quiz backend devs on iOS security
Difficulty progression	L1 (basic) → L2 (intermediate) → L3 (advanced)
Practical focus	Code examples over definitions
No trick questions	Test knowledge, not reading comprehension
Explain answers	Learning opportunity even during assessment

Assessment structure:

Duration: 45–60 minutes. Format: Online, proctored or honor system.

Sections:

1. Core security concepts (10 questions) — all roles
2. OWASP Top 10 (10 questions) — all roles
3. Language-specific (10 questions) — based on role
4. Authentication/Authorization (5 questions) — backend/fullstack
5. Security testing (5 questions) — all roles

Scoring: 90%+ = L3 (Expert) · 70–89% = L2 (Practitioner) · 50–69% = L1 (Aware) · Under 50% = Training required

Method 2: Secure code review exercise

More practical than quizzes. Give developers code with intentional vulnerabilities and ask them to find issues.

How it works:

1. Prepare code samples with 5-10 vulnerabilities
2. Set time limit (30-45 minutes)
3. Ask developer to identify and explain issues
4. Score based on vulnerabilities found and explanation quality

Example exercise:

# Code Review Exercise: Find the security issues
# Time limit: 30 minutes
# Identify all security vulnerabilities and suggest fixes

from flask import Flask, request, render_template_string
import sqlite3
import pickle
import os

app = Flask(__name__)
SECRET_KEY = "super_secret_key_12345"  # Issue #1

@app.route('/login', methods=['POST'])
def login():
    username = request.form['username']
    password = request.form['password']
    
    conn = sqlite3.connect('users.db')
    cursor = conn.cursor()
    # Issue #2
    query = f"SELECT * FROM users WHERE username='{username}' AND password='{password}'"
    cursor.execute(query)
    user = cursor.fetchone()
    
    if user:
        return f"Welcome {username}!"  # Issue #3
    return "Invalid credentials"

@app.route('/profile/<user_id>')
def profile(user_id):
    # Issue #4
    user_data = open(f'profiles/{user_id}.json').read()
    return user_data

@app.route('/search')
def search():
    query = request.args.get('q', '')
    # Issue #5
    template = f"<h1>Results for: {query}</h1>"
    return render_template_string(template)

@app.route('/load-session')
def load_session():
    data = request.cookies.get('session_data')
    # Issue #6
    session = pickle.loads(bytes.fromhex(data))
    return str(session)

@app.route('/execute')
def execute():
    cmd = request.args.get('cmd')
    # Issue #7
    result = os.popen(cmd).read()
    return result

if __name__ == '__main__':
    app.run(debug=True)  # Issue #8

Answer key:

#	Vulnerability	Location	Severity
1	Hardcoded secret	Line 8	High
2	SQL injection	Line 18	Critical
3	Potential XSS	Line 21	Medium
4	Path traversal	Line 26	High
5	Server-Side Template Injection	Line 32	Critical
6	Insecure deserialization	Line 38	Critical
7	Command injection	Line 43	Critical
8	Debug mode in production	Line 47	Medium

Scoring:

• Found vulnerability: 1 point
• Correctly explained impact: +1 point
• Suggested valid fix: +1 point
• Maximum score: 24 points

Method 3: Capture The Flag (CTF)

Gamified assessment where developers exploit vulnerable applications to find "flags" (hidden strings).

Benefits:

• Engaging and fun
• Tests practical skills
• Encourages learning through exploration
• Can be team-based for collaboration

CTF platforms:

Platform	Best for	Cost	Link
CTFd	Self-hosted custom CTF	Free	ctfd.io
OWASP Juice Shop	Pre-built challenges	Free	owasp.org/www-project-juice-shop
HackTheBox	Realistic machines	Paid	hackthebox.com
PicoCTF	Beginner-friendly	Free	picoctf.org

Running internal CTF:

Duration: 2–4 hours (can be spread over days).

Setup: Deploy OWASP Juice Shop or custom challenges. Create teams of 2–3 people, mix experience levels, prepare hints for stuck teams.

Challenges by difficulty:

• Easy (1 star): 5 challenges — basic XSS, robots.txt
• Medium (2–3 stars): 5 challenges — SQLi, auth bypass
• Hard (4–5 stars): 3 challenges — deserialization, SSRF

Scoring: Points based on difficulty. Bonus for early solves. No penalty for hints — encourages learning over competition.

After the event: Solution walkthrough for all challenges. Recognition for top teams. Individual skill notes based on participation.

Method 4: Vulnerability injection exercise

Realistic assessment — you intentionally add vulnerabilities to a PR and see if developers catch them during review.

How it works:

1. Create a branch with intentional vulnerabilities
2. Ask developer to review PR
3. Track which vulnerabilities they find
4. Provide feedback on missed issues

Example vulnerabilities to inject:

# Intentional vulnerabilities for testing

# 1. SQL Injection (obvious)
cursor.execute(f"SELECT * FROM users WHERE id = {user_id}")

# 2. Command injection (less obvious)
os.system(f"convert {filename} output.pdf")

# 3. SSRF (requires understanding)
requests.get(user_provided_url)

# 4. Hardcoded secret
API_KEY = "sk_live_123456789"

# 5. Insecure deserialization
data = pickle.loads(user_input)

# 6. Path traversal
file_path = f"uploads/{user_filename}"
with open(file_path, 'r') as f:
    content = f.read()

Creating quiz questions

Using AI to generate questions

You can use AI assistants to help create assessment questions. Here's a prompt template:

Create 5 multiple-choice security questions for [ROLE] developers 
working with [TECHNOLOGY STACK].

Requirements:
- Include code examples where applicable
- Vary difficulty (2 L1, 2 L2, 1 L3)
- Focus on [SPECIFIC TOPIC: e.g., SQL injection, authentication]
- Provide correct answer and explanation
- Make wrong answers plausible but clearly incorrect

Format each question as:
**Q[N] ([Level]):** Question text
[Code if applicable]
a) Option A
b) Option B
c) Option C
d) Option D

**Answer:** [Letter]
**Explanation:** [Why this is correct and others are wrong]

Example questions by topic

SQL Injection (Backend, Data)

Q1 (L1): Which of these is vulnerable to SQL injection?

# Option A
cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))

# Option B
cursor.execute(f"SELECT * FROM users WHERE id = {user_id}")

# Option C
User.objects.get(id=user_id)

a) A only b) B only c) A and B d) All of them

Answer: b Explanation: Option B uses f-string interpolation, directly embedding user input into the query. Option A uses parameterized queries (safe). Option C uses Django ORM (safe).

---

Q2 (L2): Is this code safe from SQL injection?

if user_id.isdigit():
    User.objects.raw(f"SELECT * FROM users WHERE id = {user_id}")[0]

a) Yes, still vulnerable to SQL injection b) No, isdigit() validation makes it safe c) No, Django ORM prevents SQL injection d) Depends on the database type

Answer: b (but discuss edge cases) Explanation: isdigit() ensures only digits, making injection impossible. However, this pattern is fragile — better to use parameterized queries as defense in depth. raw() with f-strings is a code smell.

---

Q3 (L3): Review this parameterized query. Is it safe?

table_name = request.args.get('table')
cursor.execute(f"SELECT * FROM {table_name} WHERE id = %s", (user_id,))

a) Yes, parameterized queries prevent SQL injection b) No, table name is still injectable c) Yes, the placeholder protects all inputs d) Depends on database driver

Answer: b Explanation: Parameterization only works for values, not identifiers (table/column names). The table_name is still vulnerable. Solution: whitelist allowed table names.

XSS (Frontend, Backend)

Q1 (L1): What does XSS stand for? a) Cross-Site Scripting b) Cross-Server Security c) Client-Side Scripting d) Cross-Site Security

Answer: a

---

Q2 (L2): In React, which is vulnerable to XSS?

// Option A
<div>{userInput}</div>

// Option B
<div dangerouslySetInnerHTML={{__html: userInput}} />

// Option C
<input value={userInput} />

a) A only b) B only c) A and B d) All of them

Answer: b Explanation: React escapes content in JSX by default (A and C are safe). dangerouslySetInnerHTML bypasses this protection and is vulnerable if userInput contains malicious HTML/JavaScript.

---

Q3 (L2): Which Content-Security-Policy header best prevents XSS? a) Content-Security-Policy: default-src * b) Content-Security-Policy: default-src 'self' c) Content-Security-Policy: default-src 'self'; script-src 'unsafe-inline' d) Content-Security-Policy: script-src 'none'

Answer: b Explanation: 'self' allows only same-origin resources. Option a allows everything (no protection). Option c allows inline scripts (XSS risk). Option d blocks all scripts including your own.

Authentication (All roles)

Q1 (L1): Where should you store JWT tokens in a browser? a) localStorage b) sessionStorage c) HttpOnly cookie d) URL parameters

Answer: c Explanation: HttpOnly cookies can't be accessed via JavaScript, protecting against XSS token theft. localStorage/sessionStorage are accessible via JavaScript. URL parameters are logged and cached.

---

Q2 (L2): What's wrong with this password storage?

import hashlib
password_hash = hashlib.md5(password.encode()).hexdigest()

a) MD5 is too slow b) MD5 is cryptographically broken and no salt is used c) Should use base64 instead of hexdigest d) Nothing, this is correct

Answer: b Explanation: MD5 is fast and has collision vulnerabilities — unsuitable for passwords. Passwords should use bcrypt, Argon2, or scrypt with automatic salting.

---

Q3 (L3): Review this JWT validation. What's the security issue?

const decoded = jwt.verify(token, secret);
if (decoded.userId) {
  // Grant access
}

a) No issue — jwt.verify checks everything b) Missing algorithm specification — vulnerable to algorithm confusion c) Should use jwt.decode instead d) Missing audience check

Answer: b Explanation: Without specifying algorithms, attacker could craft token with "alg": "none" or switch from RS256 to HS256 using public key as secret. Always specify: jwt.verify(token, secret, { algorithms: ['HS256'] })

Authorization (Backend)

Q1 (L1): What is IDOR?

a) Insecure Direct Object Reference — accessing resources by guessing IDs b) Internal Data Object Routing — internal API pattern c) Integrated Development Object Repository — version control term d) Indirect Object Reference Design — secure coding pattern

Answer: a

---

Q2 (L2): This endpoint retrieves user data. What's missing?

@app.route('/api/users/<user_id>/profile')
def get_profile(user_id):
    user = User.query.get(user_id)
    return jsonify(user.to_dict())

a) Input validation on user_id b) Authorization check — verifying requester can access this user's data c) Rate limiting d) HTTPS requirement

Answer: b Explanation: Any authenticated user could access any other user's profile by changing the user_id. Need to verify: if current_user.id != user_id and not current_user.is_admin: abort(403)

---

Q3 (L3): Which is the most secure authorization pattern?

a) Check authorization in controller after retrieving data b) Filter data in repository/ORM layer based on current user c) Use middleware that applies authorization to all endpoints d) Check authorization in service layer before data access

Answer: b Explanation: Filtering at repository level ensures unauthorized data never leaves the database. Other approaches risk data exposure if check is missed. Defense in depth: use multiple layers.

Building developer security competency maps

Competency maps visualize skills across the team and identify gaps.

Individual competency map

Developer: Jane Smith | Role: Senior Developer | Level: L2

Area	Score	Status
Secure coding	80%	✓
Authentication	90%	✓
Input validation	100%	✓
Cryptography	40%	⚠ Gap
Threat modeling	50%	—
Security testing	70%	✓
Dependency management	80%	✓
Incident response	40%	⚠ Gap

Recommended: Cryptography course, IR training

Team competency heatmap

Name	Secure coding	Auth	Input valid.	Crypto	Threat model	Testing	Deps
Jane S.	✓✓	✓✓	✓✓	✗	✓	✓✓	✓✓
John D.	✓✓	✓✓	✓✓	✓✓	✓✓	✓	✓
Alice K.	✓	✓	✓✓	✗	✗	✓	✓
Bob M.	✓✓	✓✓	✓✓	✓✓	✓✓	✓✓	✓✓
Carol L.	✓	✓	✓	✗	✗	✗	✓
Team avg	80%	75%	90%	45% GAP	40% GAP	60%	70%

✓✓ = Strong (70%+) · ✓ = Developing (40–70%) · ✗ = Gap (under 40%)

Tracking competency over time

Q1 2024 Assessment

Area	Score	Target	Status
Secure coding	80%	80%	Met
Authentication	75%	80%	In progress
Cryptography	45%	60%	Gap
Threat modeling	40%	60%	Gap
Security testing	60%	70%	In progress

Actions:

• Authentication module completed
• Threat modeling workshop (scheduled February)
• Cryptography training for team (scheduled March)

Q2 goals: Cryptography above 60% · Threat modeling above 60% · All developers pass intermediate assessment

Competency tracking tools

Approach	Best for	Effort
Spreadsheet	Small teams (under 10)	Low
Notion/Confluence	Documentation-heavy orgs	Low
Skills management platform	Large teams	Medium
Custom dashboard	Specific needs	High

Simple spreadsheet template:

Developer	Role	OWASP Top 10	Auth/AuthZ	Crypto	Testing	Threat Model	Last Assessment
Jane S.	Senior	4/5	5/5	2/5	3/5	3/5	2024-01-15
John D.	Mid	3/5	3/5	4/5	2/5	4/5	2024-01-15
...	...	...	...	...	...	...	...

Assessment platforms comparison

Free platforms

Platform	Strengths	Limitations
Google Forms	Easy setup, free	No code execution, basic analytics
Microsoft Forms	Office 365 integration	Similar to Google Forms
Typeform	Better UX	Limited free tier
Self-hosted CTFd	Full control, gamified	Requires setup and maintenance

Paid platforms

Platform	Best for	Approximate cost
Secure Code Warrior	Enterprise security training	$$$$
HackEDU	Developer-focused training	$$$
Avatao	Hands-on challenges	$$$
AppSec Engineer	Cloud-native training	$$
Kontra	Interactive lessons	$$

DIY assessment

For small teams with limited budget:

1. Create questions in Markdown — version control, easy updates
2. Use Google Forms for delivery — free, easy sharing
3. Track results in spreadsheet — simple competency mapping
4. Run CTF with Juice Shop — free, comprehensive

Measuring improvement

Metrics to track

Metric	How to measure	Target
Average assessment score	Quiz results	Improvement each quarter
Vulnerability density	SAST findings per KLOC	Decrease over time
Time to fix	JIRA tracking	Decrease for security bugs
Security bugs in production	Bug tracker	Fewer each release
Code review catch rate	Track review comments	More security issues caught
Training completion	LMS or spreadsheet	100% for mandatory

Before/after analysis

Training Impact Report: Q2 2024

Assessment scores before and after training:

Area	Pre-training	Post-training	Change
SQLi prevention	65%	88%	+23%
Auth best practices	70%	85%	+15%
Input validation	60%	82%	+22%

Real-world impact over the same period:

Metric	Q1	Q2	Change
SAST findings	145	89	−39%
Security bugs in production	12	5	−58%
Average fix time (days)	8	4	−50%

Training is showing measurable ROI. Continue quarterly assessments and focus training on current gaps.

Common mistakes

1. Testing knowledge, not skills — Include practical exercises, not just theory
2. One-size-fits-all assessment — Customize for roles and tech stacks
3. Testing once and forgetting — Regular assessment catches skill decay
4. Punishing low scores — This discourages honest assessment
5. Not acting on results — Assessment without training is pointless
6. Making it too long — 45-60 minutes maximum for quizzes

Self-check questions

1. What are the four methods for assessing developer security skills?
2. Why is code review exercise more practical than a quiz?
3. How do you calculate competency levels from assessment scores?
4. What metrics should you track to measure training effectiveness?
5. How often should developers be assessed?
6. What's the benefit of CTF-style assessment?

Conclusion

Assessment only matters if you act on the results. A skill gap you've measured and ignored is worse than one you haven't measured — you knew, and did nothing.

Run the baseline. Find the gaps. Use them to prioritize what you train on first.

What's next

Next: implementation guide — step-by-step plan for rolling out developer security training across the team.