Databricks Architect (Lead Data Platform Architect)
New York, United States
USD 180K-247K (estimate) Senior-level Full Time
Tasks
- Architect batch and real time processing
- Architect data governance frameworks
- Architect structured and unstructured workloads
- Build data ingestion frameworks
- Collaborate on networking security and storage strategies
- Conduct design reviews
- Define Delta Lake data storage standards
- Define job orchestration standards
- Define workspace design and cluster strategy
- Design ETL and ELT pipelines
- Design data modeling standards
- Design lakehouse architecture end to end
- Enforce architecture governance
- Ensure data privacy compliance
- Guide Spark and PySpark optimization best practices
- Implement Databricks SQL
- Implement MLflow
- Implement Spot instances
- Implement Unity Catalog
- Implement autoscaling
- Implement cost governance strategies
- Implement data lineage and cataloging
- Implement fine grained access control
- Implement job clusters
- Implement medallion architecture
- Implement metadata management
- Integrate Databricks with BI tools
- Integrate Databricks with ML pipelines
- Integrate Databricks with data sources
- Lead Databricks platform deployment and optimization
- Mentor engineering teams
- Optimize cluster utilization
- Optimize query performance
- Optimize storage efficiency
Perks/Benefits
- N/A
Skills/Tech-stack
ABAC | AWS | Airflow | Apache Spark | Autoscaling | Azure | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Cost Optimization | Data Factory | Data Governance | Data Ingestion | Data Lineage | Data Modeling | Data Vault | Data mesh | DataOps | Databricks | Databricks SQL | Delta Lake | Dimensional Modeling | ELT | ETL | Event Hubs | GCP | GenAI | Job Clusters | Kafka | MLOps | MLflow | Metadata Management | Photon | PySpark | RBAC | SQL | Spot instances | Star Schema | Unity Catalog
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R13d ago
-
Azure DevOps | Data Modeling | Data integration | ETL | GitHubSenior-level ContractSaint Paul, United States8h ago
-
Access Control | Backup and Recovery | Change Management | Data Cleansing | Data QualitySenior-level ContractLansing, United States8h ago
-
Senior-level Full TimeNew York-TONAWANDA9h ago
-
Airflow | Apache Kafka | Apache Spark | Bash | CI/CDSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL10h ago
-
AI & Data Architect - Trust and Safety USD 155K-200KAI Agents | Data Architecture | Data Modeling | Data Products | Data integrationSenior-level Full TimeSan Jose, California, United States10h ago
-
AI & Data Architect - Trust and Safety USD 155K-200KData Architecture | Data Modeling | Embeddings | Explainability | Feature EngineeringSenior-level Full TimeNew York, United States10h ago
-
Senior Data Architect - Supply Chain USD 140K-190KAI Agents | Agent Orchestration | Analytics engineering | Anomaly Detection | Apache SparkSenior-level Full TimeSunnyvale, CA11h ago
-
Technical Deployment Lead, Google Cloud USD 183K-266KAPI Integration | Agent Design | Agent collaboration | Agentic Workflows | Artificial IntelligenceEmployee benefits | Equity | Performance bonus target | Travel opportunitiesSenior-level Full TimeAustin, TX, USA; Atlanta, GA, USA11h ago
-
Associate Cloud Data Engineer II USD 78K-97KAgile | Airflow | Apache Beam | BigQuery | Cloud ComposerOnsite workMid-level Full TimeAda, MI, United States, 4935515h ago
-
Data Engineer USD 61K-141KAgile | Amazon Redshift | C++ | Cassandra | ELTDependent care | Paid leave | Professional development | Tuition assistance | Work-life programsMid-level Full TimeUSA, HI, Honolulu (999 Bishop St), …22h ago
-
Data Engineer - Reinvention Centers USD 80K-266KAI Foundry | APIs | Azure AI | Azure AI Foundry | Azure Data401k plan | Dental insurance | Life insurance | Long-Term Disability coverage | Medical insuranceMid-level Full TimeDallas, 5205 N OConnor Las Colinas, …22h ago
-
Solutions Architect - AI USD 26K-26KAI Observability | AIOps | AWS | Adversarial Attacks | Agentic WorkflowsSenior-level Full TimePhiladelphia, PA 19106, United States22h ago
-
Enterprise Data Engineer USD 73K-111KAmazon Web Services | Azure Data | Azure Data Factory | Cloud platform | Data FactoryPaid time offEntry-level Full TimeAvera Downtown Building-Sioux Falls, United States22h ago
-
Senior-level Full TimeAtlanta, US22h ago
-
Data Engineer USD 93K-168KApache Spark | Change Management | Customer Data | Customer data platforms | Data GovernanceMid-level Full TimeIrving, TX, USA; Dallas, Texas, UNITED …22h ago
-
Durability Data Analytics Principal Engineer P4 (Onsite) USD 107K-204KData Engineering | Data Visualization | Databricks | ETL | Machine Learning401k match | Dental insurance | Employee assistance program | Flexible work schedule | Health insuranceSenior-level Full TimeUS-CT-EAST HARTFORD-OBG ~ 400 Main St …22h ago
-
AWS | Agile | Amazon Redshift | Cassandra | Google CloudSenior-level Full TimeMcLean, VA, United States22h ago
-
Palantir Data Engineer USD 77K-129KArchibus | Data Governance | Data Pipelines | Data Validation | Data Wrangling401k retirement plan | Adoption Assistance | Company-Paid Holidays | Disability coverage | Emergency back-up childcare programMid-level Full TimeHome Office: Macomb, MI, United States22h ago
-
Sr Data Engineer USD 91K-163KAI/ML | Azure DevOps | CI/CD | Code review | Data Compliance401k contribution | Career development opportunities | Comprehensive benefits | Incentive and recognition programsSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R22h ago
-
APIs | Batch Processing | BigQuery | CI/CD | Cloud RunSenior-level Full TimeUS010 PA Horsham - 800/850 Ridgeview …22h ago
-
Staff Data Engineer USD 200K-260KApache Airflow | Apache Beam | Apache Spark | BigQuery | DBT401k plan | Dental insurance | Disability insurance | Equity | Flexible time offSenior-level Full TimeSunnyvale, CA22h ago
-
APOC | Agentic AI | Airflow | Apache Iceberg | AuraDBSenior-level Full TimeTX, United States23h ago
-
[C] Data Engineer, Safeguards USD 380K-485KAccess Control | Amazon Kinesis | Amazon Redshift | Apache Airflow | Apache SparkFlexible work schedule | Flexible working hours | Generous vacation | Parental leaveSenior-level Full TimeSan Francisco, CA | New York …23h ago
-
Data Owner Lead-Vice President USD 150K-200KAmazon Web Services | Business Intelligence | Data Governance | Data Lake | Data ManagementBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site wellness centersSenior-level Full TimeWilmington, DE, United States23h ago