IT and security need the same view of operations

I’ve seen this pattern in a few places. IT operations are trying to keep services up. Security is trying to reduce risk. Both want the same thing in practice, but they often end up pulling in different directions.

What it looks like

These are the signs the two teams are not working from the same view:

• Emergency change windows keep turning up at odd hours. The logs show change requests going round change control:
  2025-03-12T03:12:05Z change: APPLY: patch-20250312 by svc-ops - bypass=TRUE
• There are two ticket queues for the same incident. One says service restored. The other says the vulnerability is still open.
• Alerts get ignored because they flood the ops channel.

When operations and security are split, the usual fallout is obvious enough:

• Hosts on the perimeter stay unpatched because ops staged patches outside the security maintenance window.
• Configuration drifts across environments. The expected state is SSH only from the admin subnet on the management VLAN. The actual state is SSH open to 0.0.0.0/0 on several hosts.
• Incident response slows down because every handoff is manual.
• Audits get messy because evidence shows split responsibility and no clear owner for controls.

Where it happens

Network and infrastructure teams usually own change windows and firewalls. Security handles vulnerability scanning and risk scoring. If they do not share maintenance windows and severity definitions, the two sides collide.

The same thing happens between application ops and security testing. Ops want uptime. Security wants fixes that sometimes need downtime.

The process problems are usually familiar too:

• Change management runs without shared gating. A ticket moves to done because the service came back, but the security artefacts are missing.
• Patch management is split across tools. Ops does the patching, while security tracks it in a separate spreadsheet.

A synthetic but typical failure looks like this: a patch goes to a web cluster without cert rotation. The service throws TLS handshake failed: remote error: tls: bad certificate. I’d check with:

ss -tulpn | grep 443

The expected state is LISTEN 0.0.0.0:443 with the right cert fingerprints. The actual state is a live listener with an expired cert. The root cause is simple: ops skipped the cert step to get the service back quickly. The fix is to add cert validation into the deployment pipeline and stop the change closing until that check passes.

Find the cause

The root causes are usually the same:

• Measures point in different directions. Ops gets judged on uptime. Security gets judged on how many findings are closed. Neither side is measured on joint outcomes such as time-to-secure.
• The tooling is split. Separate CMDBs, separate ticketing, separate logging.
• Blame creeps in after incidents. Ops hide quick fixes. Security escalates the tone. That kills honest handovers.
• The teams use different language. Ops talk about deploys and rollbacks. Security talks about CVE severity and exploitability. Without a shared glossary, the friction never really goes away.

Another common failure is inventory drift. The log says:

Job failed: vulnerability-scan: critical CVE-2024-XXXX found on host-12, status=UNACKED

Ops then marks the host as patched, but the scanner never runs against it because the tag is wrong. I’d check with:

sudo tail -n 200 /var/log/vulnscan.log

The expected line is scan complete host-12: no critical findings. The actual line is host-12 not in scope.

Then I’d inspect the inventory source:

git show deploy/hosts.yml | grep host-12

The expected entry is the correct inventory tag. The actual entry shows host-12 stuck in a legacy group. That is inventory drift between ops and security tooling. The fix is a single inventory source and an automated rescan.

Fix it

1. Use shared measures. Drop the separate scorecards and track joint numbers instead: MTTR for exploitable vulnerabilities, patch-to-deploy time, and emergency rollbacks.
2. Keep one source of truth for inventory and assets. Use one CMDB, or one automated inventory service that the pipelines read from.
3. Gate changes on both service health and security artefacts before closure.

Automation helps because it cuts out the handoffs where people make mistakes:

• Run static analysis and a container CVE scan in CI. Fail the pipeline on high-severity findings. For example:

docker scan myimage:latest

The expected output is no critical issues. The real value is that the pipeline blocks and creates a triage ticket instead of letting the problem drift.

• Rotate certs as part of patching. The deployment should only complete when the cert fingerprint matches the known good value. After that, TLS failures during deployment stop showing up.

Training helps too:

• Run joint war games. Simulate a patched-but-broken deployment with a security finding. Ops drains traffic, security scores exploitability, and both sides confirm the fix.
• Swap people between teams for a short period. A security engineer on ops on-call, and an ops engineer sitting with security, both learn the same language.
• Write runbooks together. Put in the exact commands, the expected output, the actual output, and the escalation path.

For recurring problems, map the root cause to a fix:

• Inventory drift: authoritative inventory plus daily reconciliation.
• Separate ticket states: link the change and vulnerability tickets, and block closure until both are in the agreed state.

Check it is fixed

Track whether the setup is actually improving:

• MTTR for incidents caused by bad configuration and by exploited vulnerabilities.
• Time from patch release to verified patch in production.
• Count of change closures that both ops and security have signed off.

Keep rescanning after changes. Make the rescan part of the final deployment step and block change closure if the scan fails:

curl -sS https://scanner.local/api/scan/host-12/status

The expected result is {"status":"PASS"}. The failure case is {"status":"FAIL","reason":"CVE-XXXX"}.

Centralised logging helps as well. If a change has broken auth, this sort of check will show it:

sudo grep "authentication failure" /var/log/auth.log | grep "$(date +%F)"

The expected result is no burst after deployment. If there is a spike, I’d roll back and look at the change.

After incidents, write the review up properly. Keep the exact error lines, the commands used, the root cause, and the fix that went in. Review the joint metrics monthly. If something slips, fix the process rather than arguing about the symptom.