Osmos AI Data Engineer on the Databricks Data Intelligence Platform

The AI Engineer That Understands Code and Data

I remember firing up my Borland IDE circa 1989, writing code and actually being able to step through the code one line at a time. Seeing variables change in real-time while stepping through the code was absolutely magical and a massive increase in my productivity.

Prior to this innovation, debugging code meant littering it with debug statements trying to figure out where things were failing.

We are seeing another seismic shift in the industry where the human’s role is quickly evolving from writing code to orchestrating agents to accomplish tasks. Tell a code gen tool what you want, and it spits out code. It’s magical (just like my Borland IDE) … at least that’s the promise.

Yes, many of today’s AI and agentic code generation tools can help, but they fall short for data engineering problems. In order to meet a new business need, a data engineer doesn’t just need to understand their codebase, they need to understand their data architecture.

Translating this need to an AI-first world requires data-aware agentic AI that can take an engineering task from problem to solution by updating code and schemas.

Osmos Data Engineer delivers the agentic promise to those who are responsible for managing large data estates. Using it gives me that same warm, magical feeling that I got when I was able to step through code in an IDE.

I give Osmos Data Engineer a task, point it at code and data, and it does the work for me - writing code, validating it, and even preparing scripts to make necessary database changes - without having to step through any code at all!

That’s not science fiction. That’s Osmos Data Engineer for Databricks.

“Data-driven innovation is essential to enterprise success. AI-driven solutions like Osmos supercharge data engineers and empower business analysts to tackle complex data challenges. By leveraging the full power of the Databricks platform, Osmos helps organizations unlock the true potential of both their data and their people. Together, Osmos and Databricks are helping organizations transform data into a catalyst for innovation.”
‍
— Bryan Smith Head of Industry Solutions, Consumer Industries, Databricks‍

‍How it Works: From Task to Validated Changes, Fast

You tell Osmos what you want to accomplish just like you’re creating a ticket. Maybe it’s cleaning up a set of ETL jobs or surfacing data in a new table.

Point the Osmos Data Engineer to your code and data, give it some instructions and leave the rest to Osmos.

Here are the details:

1. Setup an Osmos credential. This allows Osmos Data Engineer to access your Databricks instance safely by respecting catalog and workspace access defined by your Databricks admin.
2. Setup an Osmos Data Engineer. Name your Osmos Data Engineer, select the code branch you want it to work off of, and choose the Databricks cluster you want it to use.
3. Create a task. Define the task - the more detailed the better.
4. Task execution. The Engineer will make a copy of Code Branch to work on and make shallow clones of any tables that it needs to write to. It will then update code, update its copy of tables, and validate its work.
5. Task completion. When changes are ready for review, you see a summary of all changes and review code changes in Databricks. You merge the code when you are happy with it and run the Osmos generated script to make any changes to production tables.

More Than Just Code Generation

In recent months we’ve seen various LLM-to-code tools flood on to the scene.

Osmos Data Engineer goes one step further than your average coding tool: it understands your data as deeply as it understands your code.

If a task requires changes to data fields, Osmos will prepare a script to make those updates. This way, your application logic and your data model stay in sync, no broken queries, no mismatched schemas, no surprises.

It’s not just AI assisting a human in writing code. It is a data aware, agentic AI solving an engineering task end-to-end.

‍Safe by Design

Working in Databricks means playing in a shared, high-powered environment - and the stakes are high. Osmos respects your Unity Catalog permissions and never undermines your security model.

Every change starts in an isolated code branch, so your production environment stays pristine until you explicitly push the changes live. Data integrity is maintained by making shallow copies of impacted data tables, without ever touching production data.

This allows Osmos Data Engineer to validate its changes in a safe environment while you retain control of the decision to push to prod and update production tables.

Real World  Use Cases

This secure, agentic approach is already driving impact for enterprise teams.

A global auditing firm is now enabling its non-technical engagement teams to ingest and validate financial data in minutes. Osmos AI Agents automate the transformation workflows - saving thousands of hours each year and slashing the IT overhead that comes with slow, expensive data ops.

A major CPG brand is replacing more than 100,000 data pipelines with Osmos AI Agents. These agents ingest retail data from over 100 retail partners, transform and model it, then store it in Lakehouse tables for analytics and AI. The impact: fully automated, accurate data ingestion to power Revenue Growth Management,  freeing up engineering teams freed up to focus on higher-value work.

Why This Matters Now

The rapid emergence and adoption of coding agents has made for an unprecedented improvement in developer productivity. However, data engineers need more than code gen, they need tools that are data-aware and able to adjust data schemas to meet ever-evolving business needs.

Ready to Try It?

Osmos Data Engineer for Databricks is available now in private preview. We’re working closely with early customers to refine the experience and make it the fastest, safest way to get from idea to deployed changes in Databricks.

If you’re ready to see how far development has come - and how fast you can move when your AI teammate understands your code and your data - book a demo.