Manager -- Big Data Application

Technical Oversight

• Provide technical leadership for large-scale distributed data processing systems built on Apache Spark.
• Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.
• Establish best practices for Spark application development, optimization, observability, and operational excellence.
• Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls.
• Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.
• Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.

Data Platform & Spark Engineering

• Design and oversee development of batch and real-time data processing pipelines using Apache Spark.

Collaborate on architecture involving Spark, Iceberg, Hive, Hadoop, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.

• Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows.
• Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.
• Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.

Stakeholder Collaboration

• Work closely with product managers, architects, data scientists, platform teams, and business stakeholders.
• Translate business and data processing requirements into executable engineering plans.
• Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.
• Partner with cross-functional teams to deliver data-driven products and platform capabilities.

People Management

• Conduct performance reviews, coaching, and regular feedback sessions.
• Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.
• Build technical depth within the team through knowledge sharing and career development.
• Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.

Process & Quality Management

Ensure Agile, SDLC and engineering governance practices are followed.

• Track delivery, quality, reliability, and operational metrics.
• Improve engineering productivity through automation, standardization, and platform improvements.
• Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.

Required Knowledge And Experiences

Required Knowledge and Experience

Technical Expertise

• Strong hands-on experience withApache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).
• Proven experience building and operating large-scale distributed data processing systems.
• Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.
• Experience with one or more programming languages such asScala, Java, or Python.

Experience with modern data ecosystems includingHadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.

• Experience designing and maintaining batch and streaming data pipelines.
• Strong understanding of distributed systems, cloud-native architectures, and scalability principles.
• Hands-on experience deploying and managing Spark workloads onAWS and/or GCP.

Leadership Experience

• Proven experience leading engineering teams and delivering complex technical initiatives.
• Ability to balance hands-on technical leadership with people management responsibilities.
• Experience driving cross-functional collaboration and influencing technical direction.
• Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.

Scope & Positioning

• Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.
• Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.
• Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.

Preferred Qualifications

• Experience withAWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.
• Experience withGCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.
• Experience managing terabyte-to-petabyte scale data processing environments.

Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.

• Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.

TransUnion Overview:

At TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development— while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence.

Be a part of our Workforce for Good – you’ll work with great people, pioneering products and cutting-edge technology.

TransUnion Job Title

Manager II, Applications Development