Job Title: Big Data Developer (Scala/Spark)
Location: New York City, NY
Job Description: We are seeking a skilled Big Data Developer with expertise in Scala and Spark to join our dynamic team. The ideal candidate will have a strong background in big data technologies and experience working on large-scale data processing projects. You will be responsible for designing, developing, and optimizing data pipelines and analytics solutions using Scala and Spark.
Responsibilities:
- Design, develop, and optimize scalable data pipelines and analytics solutions using Scala and Spark.
- Work closely with data engineers and data scientists to implement data processing and machine learning algorithms.
- Perform data ingestion, transformation, and aggregation tasks on large datasets using Spark RDDs, DataFrames, and Datasets.
- Optimize Spark jobs for performance, scalability, and reliability.
- Collaborate with cross-functional teams to define data requirements, architecture, and design.
- Troubleshoot and debug issues that arise during development and in production environments.
- Stay up-to-date with emerging technologies and best practices in big data and distributed computing.
Requirements:
- Bachelor's degree in Computer Science, Engineering, or a related field.
- Proven experience as a Big Data Developer, with at least [X years] of experience.
- Strong proficiency in Scala programming language.
- Hands-on experience with Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and MLlib.
- Experience with distributed computing frameworks such as Hadoop, HDFS, and YARN.
- Proficiency in SQL and experience with relational databases such as PostgreSQL or MySQL.
- Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Excellent problem-solving skills and attention to detail.
- Ability to work independently and collaboratively in a fast-paced environment.
- Excellent communication and interpersonal skills.
Nice to Have:
- Experience with other big data technologies such as Kafka, HBase, or Cassandra.
- Knowledge of functional programming concepts and libraries in Scala.
- Familiarity with containerization technologies such as Docker and Kubernetes.
- Certifications in Scala or Spark.
