Challenge :
Banking customer was looking for solution to handle the huge volume of data & data processing issues they had in the on premises infrastructure.
Solution :
We implemented a cloud-based architectures. The key components of our solution included
Data Sources:
Data and Jobs from EDP moved to S3
Log Collection:
Aurora DB used for Log collection.
Data Ingestion:
From S3, the data are ingested using the Snow pipe to Glue Workflow & EMR . AWS Glue crawler to populate the AWS Glue Data Catalog with databases and tables.
Data Processing:
snow pipeline is used for the data processing
Visualization and Reporting:
BI tools such as Tableau and PowerBI were used to create dashboards and reports, leveraging snowflake powerful querying capabilities for both real-time and historical data.
Outcome:
By leveraging a snow flake hybrid architecture, we ensured that real-time data was stored and processed
Benefits:
- Real-time monitoring and incident response with on-premises storage.
- Efficient long-term storage and querying of historical data .
- Scalable solution capable of handling large volumes of security events.