Senior Software Engineer

About the Role

We are looking for a skilled Big Data Administrator with 4-7 years of experience to support and manage cloud-based big data infrastructure using AWS managed services and Databricks. The ideal candidate will have strong expertise in Linux, automation, and operational support of services like EMR, Kafka, MSK, and Databricks clusters.

We are also looking for a Senior Software Engineer to work on a groundbreaking multichannel SaaS Digital Marketing Platform that focuses on uniquely identifying customer patterns, effectively interacting with them across channels, and achieving a positive return on marketing investment (ROMI). The platform helps consolidate and integrate features and functionality typically found in stand-alone services and channel-specific messaging platforms to give marketers a tightly integrated, easily orchestrated, insights-driven, cross-channel marketing capability. The primary role of the Senior Software Engineer is to envision and build internet-scale services on Cloud using Python and distributed technologies.

Responsibilities

For Big Data Administrator:

• Administer and support Linux-based big data environments hosted on AWS Cloud and on-prem environments.
• Manage and maintain AWS-managed big data services including Amazon EMR, Amazon MSK, and Amazon Managed Kafka.
• Administer HBase clusters, ensuring optimal performance, scalability, and data integrity.
• Manage and monitor Databricks All-Purpose Clusters, Job Clusters, and SQL Warehouses, ensuring efficient resource utilization and high availability.
• Perform cluster upgrades and patching for EMR and MSK to enhance system stability and performance.
• Implement automation for cluster provisioning, configuration, and operational tasks using Ansible and Shell scripting.
• Monitor job performance and resource usage across data pipelines, taking corrective actions as needed.
• Collaborate with data engineers and developers to support ETL workflows and optimize pipeline reliability.
• Configure IAM roles, networking, and security policies to secure big data environments.
• Implement proactive monitoring, logging, and alerting using CloudWatch, Databricks metrics, or third-party tools.
• Troubleshoot cluster failures, job execution issues, and data inconsistencies across services.
• Maintain backups and support disaster recovery strategies for critical data and metadata stores.
• Document infrastructure components, operational runbooks, upgrade processes, and best practices.
• Analyze and optimize performance and cost-efficiency of cloud-based big data workloads.
• Stay updated with emerging big data and cloud technologies and contribute to architectural improvements.

For Senior Software Engineer:

• Be responsible for development and maintenance of applications with technologies involving Enterprise Python and Distributed technologies.
• Collaborate with developers, product manager, business analysts, and business users in conceptualizing, estimating, and developing new software applications and enhancements.
• Assist in the development and documentation of software's objectives, deliverables, and specifications in collaboration with internal users and departments.
• Collaborate with QA team to define test cases, metrics, and resolve questions about test results.
• Assist in the design and implementation process for new products, research, and create POC for possible solutions.
• Develop components based on business and/or application requirements.
• Create unit tests in accordance with team policies & procedures.
• Advise and mentor team members in specialized technical areas as well as fulfill administrative duties as defined by support process.
• Create Value-adds that would contribute to Cost Optimizations/Scalability/Reliability/Secure solutions.

Qualifications

For Big Data Administrator:

• Bachelor's degree or equivalent in computer science.
• Strong experience in Linux system administration.
• Hands-on expertise with AWS services: EMR, MSK, Kafka, S3, IAM, and VPC.
• Experience managing HBase in distributed environments.
• Deep understanding of Databricks clusters, job scheduling, and SQL Warehouse monitoring.
• Proficiency in Ansible and Shell scripting for automation and orchestration.
• Experience with EMR/MSK upgrades, tuning, and troubleshooting.
• Strong analytical, communication, and problem-solving skills.

For Senior Software Engineer:

• (Qualifications are not explicitly listed in the provided text for the Senior Software Engineer role, other than the implied skills from 'What you will do' and 'Tech Stack')
• Solid understanding of software security practices including user authentication and authorization.
• Data-savvy.
• Ability to leverage best practices in design patterns and design algorithms for software development that focus on high quality and agility.
• Good understanding of Agile Methodologies like SCRUM.

Tech Stack (Relevant to Senior Software Engineer)

• Fusion of data and technology with SaaS offerings developed as a Cloud-first approach.
• Technologies involving Enterprise Python and Distributed technologies.