Job Title: Data Engineer
Job Summary
We are seeking a skilled Data Engineer with 5-9 years of experience to join our dynamic team. The ideal candidate will have a deep understanding of data engineering principles and be proficient in designing, building, and maintaining data pipelines. This role requires expertise in Python, Apache Spark, Hive, SQL, and Databricks to effectively manage and optimize our data systems.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Apache Spark and Databricks.
- Implement data models and ensure data quality and integrity across various data sources.
- Collaborate with data scientists and analysts to understand data needs and translate them into technical specifications.
- Optimize SQL queries for performance and scalability to handle large datasets.
- Utilize Hive for data storage and retrieval, ensuring efficient data management processes.
- Monitor and troubleshoot data pipeline issues, providing timely resolutions to ensure data availability.
- Document data processes and architecture to maintain clear communication within the team.
- Stay updated with industry trends and emerging technologies to continually improve data solutions.
Required Skills
- Strong proficiency in Python for data manipulation and analysis.
- Hands-on experience with Apache Spark for big data processing.
- Proficient in SQL and experience with database management systems.
- Experience with Hive for data warehousing solutions.
- Familiarity with Databricks for collaborative data engineering.
- Strong problem-solving skills and attention to detail.
- Excellent communication skills, both written and verbal.
Preferred Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Knowledge of data visualization tools like Tableau or Power BI.
- Understanding of machine learning concepts and frameworks.
- Familiarity with data governance and compliance standards.
Benefits
- Competitive salary and performance-based bonuses.
- Comprehensive health, dental, and vision insurance.
- Generous paid time off and holiday schedule.
- Opportunities for professional development and continuous learning.
- Flexible work hours and remote work options.
- A collaborative and inclusive work environment.