Middle Data Engineer ID86295
AgileEngine
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a Middle Data Engineer to help modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers. This role also uses AI tools like Claude and GitHub Copilot to speed up development.
WHAT YOU WILL DO
- Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
- Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers.
- Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
- Use Claude or Github Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation and prototyping solutions.
- Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
- Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
- Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
- Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
- Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.
MUST HAVES
- 3+ years of professional experience in data engineering , featuring direct expertise with Apache Spark and cloud-based data architectures.
- Strong hands-on experience building data pipelines with Databricks, Apache Spark (PySpark), and Delta Lake .
- Advanced SQL and Python , with strong data modeling skills across dimensional and Lakehouse patterns.
- Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs .
- Experience with workflow orchestration ( Databricks Workflows, Airflow, or Azure Data Factory ).
- Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems.
- Strong problem-solving, collaboration, and communication skills.
- Familiarity with Unity Catalog , data governance, access control, and PII handling.
- Experience with dbt or an equivalent transformation framework.
- Familiarity with secure coding standards and industry security best practices.
- Experience delivering production data platforms at scale.
- Upper-intermediate English level.
NICE TO HAVES
- Experience with Infrastructure as Code (IaC) using Terraform and CI/CD using Azure DevOps.
- Experience working with relational databases (specifically PostgreSQL) and data persistence concepts.
- Familiarity with logging and monitoring tools (e.g., Dynatrace, CloudWatch, Databricks system tables).
- Experience working in Agile or team-based development environments preferred.
PERKS AND BENEFITS
- Growth without limits : build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
- Competitive compensation : get recognition that reflects your skills and impact, with regular performance and compensation reviews
- Flexibility : work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
- Meaningful, modern projects : build impactful products using modern technologies alongside global teams and leading brands
- Collaborative culture : join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
- Well-being & support : access local well-being programs and people-focused support tailored to your location
- ...ABOUT THE ROLE We are looking for a Full Stack Software Engineer to build and maintain web applications for a tax research and workpapers... ...in React and back-end services in Node.js, C#, and Python, with data managed across PostgreSQL and DynamoDB. This is a short-term...
- ...make an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE We are looking for a Senior Quality Engineer to drive quality throughout the software development lifecycle by partnering with Product Managers, Software Engineers, and...
- ...meet you! ABOUT THE ROLE We are looking for a Senior Data Scientist to drive rigorous statistical execution, multi-goal... ...flaws before deployment. - Collaborate daily with internal engineering cross-functions via Slack and Google Meet to rapidly resolve ambiguous...
- ...ABOUT THE ROLE We are looking for a ServiceNow Discovery Engineer to join an enterprise-scale Discovery Engineering squad that executes... ...including MID Servers, Patterns/Probes, and CMDB/CSDM data quality. WHAT YOU WILL DO - Continuously assess the health...
- ...of accuracy and consistency. MUST HAVES - Strong attention to detail and accuracy. - Comfortable performing repetitive data-entry and back-office tasks. - Ability to work efficiently across multiple browser-based applications and websites. - Good general...
- ...program, or industry experience strongly preferred; familiarity with accounting ledger concepts, real-time decisioning systems, or rules engines is a plus. - Upper-intermediate English level. NICE TO HAVES - Familiarity with UX research and best practices. -...
- ...Quote-to-Cash and Order-to-Cash processes, connecting financial data to real business decisions. This person turns operational... ...partnering closely with Technical Delivery Managers (TDMs) and engineering teams who architect the technical solutions. - Ensure complex...
- ...standard modeling techniques such as BPMN, swimlane diagrams, and data flow diagrams. - Facilitate requirements review sessions... ...considerations when scoping AI/ML initiatives. - Exposure to prompt engineering concepts or evaluating generative AI tools for business process...
- ...behind them. WHAT YOU WILL DO - Act as the Squad Lead, driving the team to execute on deliverables while maintaining high engineering standards. - Spend a significant portion of your time designing and implementing robust backend services using Java and...
- ...world sustainably . The Opportunity We are looking for an engineer or a professional with a related background to join our... ...Inventor, Fusion 360, SolidWorks, etc.) – Not mandatory . Data management and analysis (Excel, Power BI, Visual Basic) – Not...
- ...working experience. ~ Technical or Bachelor degree in Renewable Energies, Architecture, Industrial Design, Electrical or Civil Engineering or related field. ~ You interest in learning and evolving to other technical roles. ~ You will use our ERP SAGE software for...
- ...pipelines, and ensure a positive recruitment experience. This position offers the opportunity to influence hiring outcomes through data-driven decision-making, strong stakeholder partnerships, and a commitment to operational excellence, innovation, integrity, collaboration...
Emerson
Sorocaba, SPhá 4 dias atrás - ...an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE We are looking for a Senior Full Stack Engineer to build robust server-side solutions for a SaaS platform operating at scale, working across Java, React, SQL, Redis, and Elasticsearch...
- ...Rails . - Strong command of API design patterns and best practices in Rails . - Solid understanding of protocols and JSON data structures . - Experience with relational databases, e.g. PostgreSQL . - Familiarity with Git and collaborative...
- ...grow, make an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE We are looking for a Senior Data Engineer to own ingestion, pipeline orchestration, and the Single Source of Truth architecture for new product modules. You will build...
- ...technical direction across our product portfolio. You will design scalable architectures, guide feature implementation, and uphold engineering best practices across the codebase. Working closely with Product Management, you will translate requirements into practical...
- ...Cloud and related operational systems. - Work across active engineering workstreams including AI Case Summarization, bulk email/... ...and case orchestration, agentic feedback sync, and Salesforce-to-data-platform integrations. - Rotate into the DFR role to triage...
- ...demonstrated production AI agent and enterprise SaaS platform engineering expertise. WHAT YOU WILL DO - Design, build, test, and... ...Glean, AI assistants, and enterprise systems. - Develop APIs, data connectors, webhooks, search integrations, and RAG/semantic-...
- ...initiatives, and translating it into product and operational decisions. - Collaborative stakeholder management across product, design, engineering, operations, and service teams. Strong written and verbal communication, including concise research readouts and executive-ready...
- ...functionality. The role requires close collaboration with product, QA, and engineering teams with active overlap in EST hours. WHAT YOU WILL DO... ...and extend Python backend services (APIs, integrations, data processing). - Collaborate closely with product, QA, and...
- Estágio em Customer Success Data Analytics | Saneamento 4.0 Local: Sorocaba/SP | Híbrido Carga horária: 6 horas/dia | Horário a combinar Na Stattus4 , buscamos uma pessoa estagiária para atuar em Customer Success Data Analytics , apoiando o relacionamento com...
- ...across multiple vendors and systems to ensure reliable lead flow, data accuracy, and system connectivity. - Identify, design, and... ..., requirements, and operational solutions with minimal engineering dependency. - Experience supporting both legacy systems and...
- ...capabilities within the People Foundation Pod. - Investigate and resolve both functional (configuration) and technical (integration and data-flow) production issues affecting Core HCM, Workday Help, and related worker lifecycle processes. - Troubleshoot failed or...
- ...make an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE We are looking for a Senior Backend Engineer to build and maintain production-grade microservices and distributed systems, developing backend features that support AI-enabled...
- ...you! ABOUT THE ROLE We are looking for a Senior Backend Engineer to build and maintain the application services and REST APIs that surface creator insights and AI-generated scores for a data analytics platform, integrating third-party sources including YouTube...
- ...-scale Salesforce environments. - Familiarity with complex data models and large-scale implementations. - Experience with Copado... ...or automation using REST APIs. - Understanding of prompt engineering concepts or applying generative AI tools within business...
- ...teams through implementations, mentor developers, partner with sales on pre-sales calls, and help clients turn Shopify into a growth engine, all within a remote-first environment. WHAT YOU WILL DO - Architect and oversee Shopify stores, applications, and custom...
- ...Distribution Network SCADA Engineer Main Responsibilities This position is based in Lima, Peru, focusing on providing full-process technical support and external coordination for overseas projects related to distribution automation and SCADA systems. It requires...
NARI Brasil Holding Ltda
Sorocaba, SPhá 10 dias atrás - ...Industries company, is a leading global supplier of catalysts, engineered materials and fine chemicals. We provide innovative products, technologies... ...employee relations. ~ Proficiency in utilizing HR systems, data analysis and reporting capabilities. ~ Strong professional...
- ...grow, make an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE We are looking for a Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural...
Deseja receber mais vagas?
Assine e receba vagas semelhantes a Middle Data Engineer ID86295. Seja o primeiro a se candidatar!
