How Cloud Elemental helped a large European energy provider modernise its log analytics platform using Amazon OpenSearch Service, enabling secure, scalable, and compliant observability across hybrid environments.
from fragmented monitoring to resilient log analytics.
AWS OpenSearch Implementation
The Client
Our client is a large European energy organisation operating critical infrastructure across multiple generation assets.
As a public-facing utility provider, the organisation manages large volumes of operational and application log data. This data must be retained securely and remain accessible for audit, regulatory compliance, and operational analysis.
As the organisation’s platform footprint expanded, the existing on-premise logging and analytics platform struggled to meet requirements for scalability, security governance, and operational resilience.
Cloud Elemental was engaged to design and implement a modern, cloud-native log analytics platform using Amazon OpenSearch Service.
The Challenge
The organisation’s legacy log analytics platform presented several operational and architectural limitations. As data volumes grew and compliance requirements increased, the existing environment created operational risk and governance gaps.
Key challenges included:
Scalability & Platform Resilience
The legacy search infrastructure struggled to scale with growing log volumes and lacked built-in fault tolerance, increasing the risk of downtime and reduced operational visibility.
Limited Governance & Access Control
Broad network-level access exposed sensitive operational logs. Without identity-based access controls, the organisation could not enforce least-privilege access or meet governance requirements.
Infrastructure & Storage Constraints
On-premise infrastructure limited storage capacity and performance, making it difficult to support long-term log retention required for compliance and audit investigations.
Operational Overhead & Maintainability
The platform required frequent manual maintenance and lacked automated lifecycle management, increasing operational effort and reducing the team’s ability to respond quickly to incidents.
The CE Approach
Cloud Elemental delivered a structured engagement to evaluate the current environment and design a scalable OpenSearch-based solution aligned with enterprise governance practices.
Cloud Readiness Assessment
Assessed existing logging infrastructure and operational workflows
Identified platform gaps and compliance risks
Defined technical and organisational requirements for a modern logging platform
Solution Blueprinting
Designed an Amazon OpenSearch Service architecture for secure, scalable log analytics
Defined security, access management, and retention models
Established operational patterns aligned with cloud best practices
Feasibility & Compliance Validation
Evaluated the architecture against governance and regulatory expectations
Refined retention models and security controls
Ensured the solution aligned with enterprise operational policies
Delivery Planning
Produced an implementation roadmap and architecture documentation
Defined a scalable operational model for the new logging platform
Provided guidance for platform adoption and operational integration
Our Solution
Multi-AZ Resilient Architecture
The OpenSearch cluster was deployed across multiple AWS Availability Zones to ensure high availability and resilience.
Key architectural characteristics included:
Multi-AZ cluster deployment for fault tolerance
Dedicated master nodes to maintain cluster stability
Automated failover capabilities in the event of infrastructure issues
Support for blue/green deployments enabling safe upgrades and rollback
This architecture ensures consistent uptime and operational visibility, even during maintenance events or infrastructure failures.
Tiered Storage for Cost Efficiency
To balance performance, cost, and compliance requirements, the solution implemented tiered log storage using Index State Management (ISM) policies.
Logs automatically transition between storage tiers based on age:
Hot Tier
Retention
- 0-30 days
Purpose
- High-performance indexing for active monitoring and debugging
UltraWarm Tier
Retention
- 30-90 days
Purpose
- Cost-efficient storage for historical investigation
Cold Tier
Retention
- 90+ days
Purpose
- Long-term retention for regulatory and audit requirements
This automated lifecycle policy ensured:
- Cost-optimised storage management
- Rapid access to recent operational data
- Compliance with long-term retention policies
- Reduced manual operational overhead
Granular Access Control with Azure AD (Entra ID) and SAML
To strengthen governance and security, the solution integrated identity-based access control using Azure Active Directory and SAML federation.
Single Sign-on Integration
Users access OpenSearch Dashboards via Azure AD authentication, enabling centralised identity management.
Role-based Access Control
Fine-grained permissions were mapped to Azure AD groups, restricting access to sensitive operational logs.
Secure Governance Model
Access events are linked to verified user identities, supporting auditability and compliance.
This identity-aware access model ensures that:
- Sensitive log data is visible only to authorised users
- Access is centrally governed through existing identity systems
- Compliance and audit requirements are met
Our Results
Enterprise-Scale Log Analytics
A centralised Amazon OpenSearch platform capable of handling growing volumes of operational and application logs while supporting future platform expansion.
High Availability & Resilience
Multi-AZ deployment with built-in redundancy ensures continuous log visibility and reliable access to operational data during infrastructure failures or upgrades.
Improved Security & Governance
Identity-aware access control integrated with Azure AD enables role-based permissions, ensuring sensitive log data is protected and access is fully auditable.
Cost-Optimised Log Retention
Automated tiered storage policies allow logs to transition through hot, warm, and cold tiers, balancing performance, compliance requirements, and long-term storage costs.
Looking to optimise visibility?
Discover how Cloud Elemental partners with organisations to deliver secure, resilient, and future-ready cloud solutions.
