Full-Stack Data Engineer

We are seeking a highly skilled Full-Stack Data Engineer to lead the design, development, and management of a data-driven solution that integrates diverse car-related data sources and ensure that all integrated data is validated, structured, and accurate, serving as a single source of truth for all car-related data services.

This role combines expertise in back-end engineering, data science and knowledge in front-end development to build a scalable and efficient, and maintainable platform. You will be responsible for building APIs, implementing data aggregation pipelines, creating a rule engine for intelligent data unification and routing, and extending a monitoring/admin panel for solution management.

The ideal candidate has a strong understanding of AWS services, data classification and data modeling techniques as well as fundamental web development knowledge.

Key Responsibilities

Data Integration and Processing

• Design, implement and maintain pipelines to gather data from various sources, including Web sites, APIs, DBs, FTPs, etc.
• Classify and categorize data into a well-structured taxonomy for use in analytics and decisionmaking systems.
• Develop and configure a rule engine to unify incoming data and make intelligent routing decisions based on predefined rules.

Backend Development

• Design and maintain a robust database schema in PostgreSQL (hosted on AWS RDS) to store and retrieve data efficiently.
• Build scalable RESTful APIs using AWS API Gateway and AWS Lambda.
• Implement a data aggregation pipeline using the ELK Stack for analytics.

Full-Stack Development

• Extend and maintain an intuitive and responsive Admin Panel using Node.js and JavaScript for solution monitoring, managing data pipelines, rules, and configurations.

Cloud Infrastructure and Deployment

• Utilize AWS services to deploy and manage APIs, ensuring scalability and security.
• Implement AWS S3 + CloudFront for efficient storage and delivery of image assets.
• Extend and implement data security and access control measures using tools like IAM, API Keys, and AWS Cognito.

Collaboration

• Work closely with data scientists, product managers, and other developers to ensure the platform meets business requirements.
• Document processes, workflows, and designs for team collaboration and knowledge sharing.

Required Skills & Qualifications

Technical Skills :

• Experience with PostgreSQL, including schema design, query optimization, and AWS RDS deployment.
• Experience with classification and categorization of complex datasets.
• Expertise in data aggregation tools like the ELK Stack (Elasticsearch, Logstash, Kibana).
• Familiarity with AWS services, including API Gateway, Lambda, S3, RDS, and CloudFront.
• Knowledge of building and using rule engines for data unification and routing.
• Knowledge in Python (for data gathering and pipeline development).
• Knowledge of Node.js and JavaScript (for the admin panel extension)

Soft Skills:

• Strong analytical and problem-solving skills.
• Ability to work independently and collaboratively in a team environment.
• Excellent communication and documentation skills.

Nice-to-have:

• Familiarity with VIN decoding and car industry-specific datasets.
• Experience with some of the modern data engineering tools like Apache Airflow, AWS Glue, or Step Functions

Interested in leading the design, development, and management of a data-driven solution that integrates diverse car-related data sources, ensuring they are validated, structured, and accurate? 

Apply Now! Send us your CV in English to Ivanina.Ivanova@easyconsult.eu