In the beginning, security was a technology problem solved by adding tools that created boundaries between an organization’s environment and the outside world. When cloud technologies entered the picture, it became equal parts a technology and people problem as people used passwords to connect to remote resources and stolen credentials became increasingly prevalent.
Today, security is a data problem. Most organizations collect and ingest data into a SIEM from a complex collection of security tools, like endpoint detection and response (EDR), identity and access management (IAM), or configuration management databases (CMDB). To cost-effectively aggregate this data, many organizations implement a security data lake or warehouse. However, these only solve the storage solution. They fail to solve the bigger problem.
Data security is a data problem. Every workflow is data movement wearing a detection costume. Despite this, 70% of security organizations lack a data orchestration layer. For APIs and AI to work correctly, organizations need standardized formats and query languages to transmit and correlate data across complex systems.
Lack of Data Orchestration Creates Severe Stress for Security Engineers and Data Analysts
Without data orchestration, organizations rely on systems to determine the data’s value. Sometimes the systems work. Sometimes it fails. Without normalized data, organizations have to focus on the underlying systems they use rather than the business problem they want to solve.
Difficulty managing ad hoc issues
Security engineers and data analysts are under severe stress. Even in an AI security operations center (SOC) solution, running ad hoc taste requires knowing the threats facing the organization and the current compliance state.
However, the data that answers these questions may be pulled from various security tools, like:
- EDR
- Asset management repository
- Vulnerability management system
- Governance, risk, and compliance (GRC) solution
Security practitioners need to know what the problems are and the steps to take to remediate compliance issues. However, without a data orchestration layer, they need to sift through siloed data across multiple systems.
Time-consuming manual processes
Many organizations still maintain data in spreadsheets. A security practitioner may need a list of critical vulnerabilities, but searching through spreadsheets can take hours. Organizations want to push data into a SIEM or other centralized location so they can enrich the data.
However, to appropriately mitigate risk, they need to build an ad hoc query or agent that creates the ad hoc query then pull data into the spreadsheet. Then, they normalize it and access all the systems that collect and store vulnerability information. Finally, the workflow may require manually creating a ticket for it.
Data and operational silos
Across the average enterprise, every team has its own controls, governance mechanisms, and environments. For example, an organization may have a security, DevOps, and AI operations teams who all need similar data but use specialized tools. This complexity exists before looking into business applications and infrastructure.
These various teams have no way to work across systems. Even if they did, they would require different permissions. At a minimum, collaboration requires a platform that can pull data from these disparate systems, normalize the data, and forward it to where users can access it. A giant database is inefficient.
Platformization Relies on Data Orchestration
Realistically, organizations need to adopt platformization because they need to connect their IT, DevOps, and business applications because all provide different data points related to security. Platformization is nothing new. Back in 2008 to 2010, McAfee, Symantec, Cisco, and IBM all provided platformatized solutions that covered most security requirements. Then, the cloud expanded the attack surface and the number of security tools exploded.
Organizations went from using 10 to 12 different tools to between 80 and 100. Looking at spend data, the top 40 platforms remain approximately 60% of sales, leaving the other 40% for new and best of breed technologies.
Dynamic workflows
Data orchestration enables security practitioners to create dynamic workflows where a security engineer or security analyst can interact with systems without knowing what those underlying systems are. Whether through an agent or API, they can ask their core questions.
A data orchestration layer normalizes and maps security data into a vendor-agnostic format, like the Open Cybersecurity Schema Framework (OCSF). By mapping various data formats to a standardized one, security teams have usable data flowing between systems so that they have easy access to all the information they need without worrying about the underlying system providing it.
Rapid tool onboarding
As threats evolve, organizations deploy new detection and response technologies To gain that tool’s benefit, they need out-of-the-box integrations that allow them to drop the solution into their broader ecosystem.
With a data orchestration layer, they have the option of a common, industry standard data model or their own data model. The orchestration layer accelerates data integration and reduces implementation delays by enabling the team to change, expand, or modify the data model.
Considerations for Data Orchestration
According to a Palo Alto and IBM study, the average organization has 83 different security solutions from 29 different vendors. To manage these tools, many organizations take a platform approach, with 96% of executives who have adopted platformization saying that security is a source of value. For organizations to move security from a necessary cost to a value generator, they need a data orchestration layer that enables data movement across complex systems. Whether looking to build or buy a data orchestration tool, organizations should consider these fundamental requirements.
Vendor Relationship
Owning the vendor relationship at the platform level removes the months of partnership overhead that otherwise stand between a requirement and a working integration.
- Established Technology Alliance Program agreements and NDAs already in place with target vendors
- Not-for-Resale license access secured on your behalf, not negotiated per integration
- A named technical contact and support path at each vendor for production issues
- No per-customer or per-deal renegotiation required to add a new target
Normalization Layer
A single, unified data model means every new connector extends what already exists instead of requiring its own bespoke mapping work.
- Alignment to an open standard, like the OCSF, rather than a proprietary schema
- One consistent data shape across vendors within the same category
- New providers added to a category without changes to your existing integration code
- A schema that can be extended or re-mapped without a rebuild
Built Environments
Provisioned, ready-made sandboxes let engineering start building the moment a requirement is identified, rather than waiting on environment access.
- Sandboxed test environments with mock data available for every supported target
- Environment access decoupled from your own vendor relationship or approval process
- Consistent environment provisioning across cloud, on-prem, and hybrid deployments
- No dependency on production customer data to begin development
Testing & Validation
Going beyond a vendor’s reference API to validate against real, varied customer environments and determine whether an integration actually holds up in production.
- Active testing against live vendor APIs, not just published documentation
- QA coverage across a broad range of real-world customer configurations
- Ongoing monitoring for schema drift and API changes, not one-time validation
- A defined process for catching and absorbing breaking changes before they reach customers
Security Controls
Sitting in the middle of credentials and customer data, an orchestration layer needs security controls that hold up under enterprise scrutiny not just provide partial answers.
- Credentials encrypted at rest and never stored in plain text
- Role-based access control enforced at the tenant level
- True data isolation between customers in multi-tenant deployments
- Credential lifecycle management, including rotation and revocation, without manual intervention
Confidence & Visibility
Continuous, centralized monitoring enables organizations to create a proactive approach to managing integration health and preventing service disruptions.
- Real-time visibility into which integrations are live, healthy, and in use
- Centralized monitoring across all targets, not siloed per-integration dashboards
- A clear, published SLA for uptime and issue resolution
- Change and break detection that surfaces issues before customers notice them
Security Mesh Architecture Is the Future
With APIs as the user interface and a data orchestration layer normalizing data, organizations can implement a security mesh architecture that breaks down organizational silos. However, for it to fulfill its promise, vendors must maintain APIs in a way that allows them to handle updates without causing system disruptions. As organizations move toward this model, they need to consider the different ways that agents, custom agents, and data orchestration solutions work together for improved data sharing that reduces engineering costs, improves compliance, and speeds up threat detection and response.
