Log Data Engineer (Contract – Hybrid – Owings Mills, MD)

A prominent client in the Financial Industry is seeking a highly skilled Log Data Engineer (Contract – Hybrid – Owings Mills, MD) to join their team. This contract position, based hybrid in Owings Mills, Maryland (requiring two days in the office per week), offers a unique blend of infrastructure and data engineering, operating at the critical intersection of DataOps. The ideal candidate will possess a high degree of autonomy, the ability to prioritize work with minimal oversight, strong analytical skills, and excellent communication abilities, all essential for successful alignment within the organization’s dynamic and highly regulated environment.

Log Data Engineer (Contract – Hybrid – Owings Mills, MD)

Location: Owings Mills, MD – This is a Hybrid role, requiring two days in the office per week.

Employment Type: Contract (W2 or C2C)

Pay Range: $60/hr W2, $72/hr C2C

Industry: Computer and Mathematical / Financial Industry

What’s the Job? Engineering Scalable Log Data Platforms for Financial Insight

As a Log Data Engineer, you’ll operate at the vital intersection of infrastructure and data engineering (DataOps), playing a pivotal role in building, maintaining, and optimizing the platforms that process vast amounts of log data. Your work is absolutely critical for providing actionable insights into security, operations, and compliance, thereby ensuring the reliability and integrity of core financial systems around the clock. This role demands a blend of deep technical expertise in both infrastructure and data management.

• Develop Data Processing Pipelines: You will be responsible for meticulously developing robust data processing pipelines. These pipelines, built using programming languages such as Java, JavaScript, and Python on Unix server environments, are designed to efficiently extract, transform, and load (ETL) log data from a multitude of disparate sources. This involves ingesting various types of logs (e.g., security audit logs, application performance logs, network flow data, user activity logs, compliance-related audit trails) from diverse systems across the financial enterprise. Your coding expertise ensures that raw, often unstructured or semi-structured, log data is accurately cleansed, normalized, and prepared for analysis and operational use, laying the groundwork for critical insights.
• Implement Scalable, Fault-Tolerant Solutions: You will architect and implement scalable, fault-tolerant solutions for data ingestion, processing, and storage. This involves designing resilient architectures that can efficiently handle massive data volumes (terabytes to petabytes) and gracefully manage unexpected failures or spikes in data. You’ll ensure that log data is continuously captured, processed, and stored without loss or degradation, which is paramount for real-time monitoring, historical analysis, and forensic investigations in the highly sensitive financial sector. Your solutions will be built with redundancy and high availability in mind, ensuring uninterrupted data flow.
• Support Systems Engineering Lifecycle Activities: You will provide comprehensive support for systems engineering lifecycle activities pertaining to data engineering deployments. This spans the entire process, including:
  • Requirements Gathering: Collaborating with security, operations, audit, and compliance teams to understand their specific log data needs.
  • Design: Creating detailed architectural diagrams, data flow diagrams, and technical specifications for robust data solutions.
  • Testing: Rigorously testing their functionality, performance (e.g., latency for real-time ingestion), and scalability.
  • Implementation: Overseeing and executing their deployment into production environments.
  • Operations: Managing their ongoing health, performance, and maintenance.
  • Documentation: Maintaining thorough, auditable documentation that covers all aspects of the data platforms. Your involvement ensures the end-to-end quality and integrity of data platforms crucial for a regulated industry.
• Automate Platform Management Processes: You will drive significant operational efficiency by automating platform management processes through Ansible or other scripting tools/languages (e.g., Python scripts, Shell scripts). This includes developing automation for infrastructure provisioning (e.g., spinning up new data ingestion nodes), configuration management (e.g., deploying standard log forwarders), software deployments (e.g., updating pipeline components), and routine maintenance tasks (e.g., log rotation, storage optimization). This reduces manual effort, minimizes human error, and enhances operational consistency and speed.
• Troubleshoot Incidents Impacting Log Data Platforms: You will be directly responsible for troubleshooting incidents impacting the log data platforms. This involves rapidly diagnosing issues related to data ingestion failures, processing bottlenecks within the pipelines, storage accessibility problems, or inaccuracies in reporting. Your problem-solving skills will be crucial for minimizing downtime, ensuring the continuous flow of critical log data for security operations, compliance auditing, and maintaining the overall integrity of financial systems. This often requires deep dives into distributed logs and metrics across various components.
• Collaborate on Data Requirements and Scalable Solutions: You will foster strong relationships and collaborate with cross-functional teams across the organization, including security operations centers (SOC), compliance officers, fraud detection teams, and application development groups. The goal is to deeply understand their specific log data requirements. Leveraging these insights, you will then design scalable solutions that effectively meet their business needs, ensuring that the log data platforms provide the necessary insights and support for critical decision-making without compromising performance or cost-efficiency.
• Develop Training and Documentation Materials: You will play an active role in creating essential training and documentation materials for the log data platforms. This includes developing clear user guides, operational runbooks for incident response, detailed architectural diagrams, and troubleshooting guides. High-quality documentation is vital for knowledge transfer, onboarding new team members, enabling self-service for data consumers, and ensuring consistent operational practices for the complex log data platforms, which is critical for auditability in finance.
• Support Log Data Platform Upgrades and Coordinate Testing: You will provide crucial support for log data platform upgrades. This involves meticulously planning upgrade processes, conducting thorough testing of new versions (e.g., performance testing, regression testing, user acceptance testing), and coordinating these testing efforts with key users of the platform. Your aim is to ensure minimal disruption, successful transitions to newer versions, and continuous compliance with evolving security and regulatory standards.
• Gather and Process Raw Data from Multiple Sources: You will be adept at gathering and processing raw data from multiple disparate sources. This includes writing custom scripts to extract data from various endpoints, calling diverse APIs for real-time data streams, and writing complex SQL queries to retrieve information from different databases and legacy systems. Your efforts transform raw, often unstructured or semi-structured, data into a format suitable for analysis and ingestion into the log platforms, preparing it for actionable insights.
• Enable Analytical Processing Solutions: You will contribute significantly to enabling both batch and real-time analytical processing solutions leveraging emerging technologies. This includes designing and implementing systems that allow for the efficient analysis of large volumes of log data, supporting various use cases from real-time security incident detection and operational performance monitoring to historical analysis for audit and forensic purposes, all crucial for a financial institution.
• Participate in On-Call Rotations: You will be a key member of the team, participating in on-call rotations to address critical issues and ensure the reliability of data engineering systems around the clock. This commitment is vital for maintaining the continuous availability and integrity of critical log data platforms in a 24/7 financial environment, supporting security and operational continuity.

Required Technical Expertise & Experience: Your Core Qualifications for DataOps Excellence

To excel as a Log Data Engineer, you’ll need extensive, hands-on experience in AWS cloud services, data lake development, scripting, and a strong background in IT service management and advanced troubleshooting.

• AWS Expertise and CI/CD for Log Ingestion: You must possess expertise in AWS (Amazon Web Services) and the implementation of CI/CD (Continuous Integration/Continuous Delivery) pipelines specifically designed for log ingestion. This indicates deep knowledge of how to build automated, reliable workflows for bringing diverse log data into AWS environments using services like Kinesis, Firehose, S3, and Lambda, managed through CloudFormation or Terraform.
• Expertise with AWS Computing Environments: You have demonstrated expertise with various AWS computing environments. This includes practical experience with ECS (Elastic Container Service) for container orchestration, EKS (Elastic Kubernetes Service) for managed Kubernetes clusters, EC2 (Elastic Compute Cloud) for virtual servers, and Lambda for serverless computing. This wide-ranging experience ensures you can design and manage diverse compute resources optimized for log processing and analytics.
• Data Lake Design, Development, and Deployment (3-5 years): You must have 3-5 years of experience in designing, developing, and deploying data lakes using AWS native services. This includes proficiency with AWS S3 (Simple Storage Service) as a scalable object storage layer, AWS Kinesis Firehose for real-time data streaming, IAM (Identity and Access Management) for secure and granular access control, and Terraform for Infrastructure as Code (IaC) to provision and manage data lake components effectively.
• Data Pipeline Orchestration Platforms: You have practical experience with data pipeline orchestration platforms such as Apache Airflow, AWS Step Functions, or Prefect. This indicates your ability to design, schedule, monitor, and manage complex, interdependent data workflows for log processing and delivery across distributed systems.
• Ansible/Terraform Scripting and IaC Expertise: Expertise in Ansible/Terraform scripts and overall Infrastructure as Code (IaC) scripting is required. This demonstrates your proficiency in automating the provisioning, configuration, and management of infrastructure components that support log data platforms, ensuring consistency, repeatability, and efficient change management.
• Version Control and CI/CD for Data Engineering: You must implement version control (e.g., GitLab, GitHub, Bitbucket) and CI/CD practices for data engineering workflows. This ensures reliable and efficient deployments of data pipelines and infrastructure changes, promoting collaborative development and adherence to best practices in software delivery.
• Proficiency in Distributed Linux Environments: You possess strong proficiency in distributed Linux environments. This includes administering Linux servers, understanding their networking, storage, and process management in a scaled-out, clustered context, which is common for hosting log agents or processing engines.
• Proficiency in Monitoring, Logging, and Alerting for Data Infrastructure: You demonstrate strong proficiency in implementing monitoring, logging, and alerting solutions specifically for data infrastructure. This includes using tools like Prometheus for metrics collection, Grafana for data visualization and dashboarding, and leveraging native AWS services (e.g., CloudWatch) to ensure proactive identification and resolution of issues within the log data pipelines.
• Experience Writing Data Pipelines for Log Ingestion: You have proven experience writing data pipelines to ingest log data from a variety of sources and platforms. This is a core hands-on skill, covering diverse log formats (e.g., JSON, XML, unstructured text) and ingestion methods (e.g., agents, APIs, streaming services).
• Implementation Knowledge in Data Processing Pipelines (Java, JavaScript, Python): You possess implementation knowledge in data processing pipelines using programming languages like Java, JavaScript, and Python to extract, transform, and load (ETL) data. This confirms your ability to code the actual logic for data cleansing, enrichment, and transformation within the pipelines.
• Data Model Creation and Maintenance: You are skilled in creating and maintaining data models, ensuring efficient storage, retrieval, and analysis of large datasets of log information. This includes designing schemas that optimize for performance, usability, and compliance.
• Troubleshoot Data Processing, Storage, and Retrieval Issues: You are adept at troubleshooting and resolving issues related to data processing, storage, and retrieval specific to log data. This demonstrates your ability to diagnose complex data-related problems across the entire data lifecycle, from source to consumption.
• Experience in Data Extraction, Ingestion, and Processing Systems: You have proven experience in the development of systems for data extraction, ingestion, and processing of large volumes of data. This highlights your capability to build robust data infrastructure from the ground up, tailored for log analytics.

General Skills and Experience:

• Complex Issue Troubleshooting: You have the ability to troubleshoot and diagnose complex issues across various technical domains, from network connectivity to application-level errors impacting data flow.
• Supporting Technical Users and Requirements Analysis: You are able to demonstrate experience supporting technical users and effectively conduct requirements analysis for data-focused solutions, translating business needs into technical specifications.
• Independent Work with Minimal Guidance: You can work independently with minimal guidance and oversight, demonstrating strong autonomy, self-motivation, and the ability to manage your workload efficiently.
• IT Service Management Familiarity: You have practical experience with IT Service Management (ITSM) and familiarity with Incident & Problem management processes.
• Performance Bottleneck Identification and Resolution: You are highly skilled in identifying performance bottlenecks, recognizing anomalous system behavior, and resolving root cause of service issues for complex data platforms, ensuring optimal system health.
• Cross-Team Influence: You have a demonstrated ability to effectively work across teams and functions to influence design, testing, operations, and deployment of highly available software, fostering collaboration and alignment.
• Knowledge of Methodologies: You possess knowledge of standard methodologies related to security, performance, and disaster recovery for data infrastructure.

If this Log Data Engineer role in Owings Mills, MD, aligns with your expertise in AWS, data pipeline development, scripting, and your passion for ensuring the reliability of critical log data platforms in the financial industry, we encourage you to learn more about this exciting hybrid contract opportunity. This is a fantastic chance to make a significant impact on core financial systems through DataOps.

Ready to leverage your combined infrastructure and data engineering skills to drive operational excellence and secure financial insights?