Field notes 007 - if you give AI a computer...
We are on the cusp of some incredible shit, let me show you
I’m a software engineer. I’ve noticed that I become a better engineer when I’m writing my learnings down. In this section of my newsletter I post daily learnings as I work to become a fantastic software engineer.
I have a really sick demo to show you guys
This is a video where I use vers, the product I’m working at HDR, to control a computer using natural language. In addition to writing code, LLMs can now use a computer like a human does - by moving a mouse around, left clicking, and typing. We are building the infrastructure that lets you give the LLM a computer. You can deploy thousands of these agents to do your bidding.
Here, we’ve given the LLM a Linux virtual machine. I instruct the LLM to read the NYT, grab the headlines, and send me an email with the summary. It’s just a prompt! No custom code. Some insane stuff you can see in this video:
The model (Claude by Anthropic) decides to open Firefox on the remote machine, visit the NYT, and click on various sections. This is not Puppeteer - it’s using the browser like a human would.
Then, it uses its bash tool generate and run the code needed to send an email. I haven’t specified or given it example Python code to write.
In fact, it actually gets the code wrong - and corrects itself.
Today, we have the Universal API for computers. AI can now perform any action on a computer - it can hit traditionally programmatic APIs, as well as use the Human API - a user interface.
I have even cooler demos to show you. I’ve gotten vers to log in to websites, scrape data, open the developer console and run commands - all controlled with natural language. HUGE game changer. I basically think that if you are building agents and need them to do anything, our platform will arm your agents with the ability to use computers to accomplish those tasks.
A LOT of amazing stuff is coming.
Side note, our cofounder Matilde is holy shit so good at design. It’s such a powerful skill to be this good. It’s a little cringe to be this starstruck ig, hi matilde if you’re reading this. But I’m genuinely in awe. How do I get this good? It probably took a while to develop not just the skills, but a personal taste and voice - like in writing.
I’m a hacker let’s gooo
OK technically I’m a scraper, but it felt AMAZING yesterday to successfully get around some auth restrictions and pull data (that I’m still authorized to access. Still allowed into the building, just used a side door).
Once again, last two days I’ve been able to get really far in debugging by writing things down and wiggling around pieces of the system to isolate issues to a specific chunk. And if I’m stumped on a chunk, I can ask AI - the smaller the chunk, the better chance of success.
When trying to set up scraping, watch the Network tab and look for JSON responses to see when data is actually being sent back. Then, look for the payloads that returned that data.
GraphQL is pretty cool, I discovered that for the first time. I guess it’s a great system if you have data that needs to be queried in complex ways. Is it pretty much the standard, or are there other ways to do it?
Learnings about product engineering
Over the past few weeks at HDR I’ve been really happy with how I’ve been shaping my role. We’re building an infrastructure platform to deploy and discover agentic workflows, and trying to discover the value we can provide along the way. Instead of trying to blindly come up with features, I’ve decided to become my own user and build solutions for customers that could be more effective and more capable thanks to our platform. I’m becoming my own user, so I can discover what needs to be built.
The features that interest you as an engineer are not always the features that will keep the business alive. Yesterday I spent extra time trying to get a return array formatted just right, but I realized it was completely irrelevant when I went to build a demo for a customer. Instead, a context issue in our LLM was the real blocker to solving their problem well.
It’s really rewarding and really clarifying. Solving a problem for a customer quickly makes it clear what to build. And I’m grateful for my engineering skill. When building my demo, I found a logging issue that made the product basically unusable… and I was able to put up a PR to fix it. Big upgrade from the PM life.
Big tip here if you’re at larger companies: work in the revenue-generating part of the business! Much more opportunity for growth, solving important problems, and your team is more likely to be shoulder to shoulder tackling external challenges, rather than infighting for resources or influence.
I’d like to get better at being a product engineer. I notice that when having a conversation with a potential customer, I was really clingy to my own expectations, and really clingy to a particular outcome (I convince them to buy our product). The conversation sputtered for a variety of reasons. I sent my friend the transcript and he sent back some great tips and feedback for talking to customers:
1. Get to the problem more quickly - it felt like “what sucks most right now?” was the last question you asked when it should’ve been the first or second. Maybe ask one framing discovery question before “what sucks.” A good question after “what sucks” may be “if you had a VA right now - what would you have them do?”
2. Frame how you can solve it - only IF YOU THINK YOU CAN if not it’s no problem just say you don’t think you can. Expectation management matters so much here
3. Figure out how you can provide and value and be helpful immediately if you think you can - set up tangible expectations of what comes next. It projects competence + holds you to a pace.
Thanks Mark.
Other musings
I would like to return to clowning. It’s so important to me to continue to deepen my connection with myself, and learn how to connect more deeply with others. I just… need to continue that process, even as I work really hard at my jobs.
Can’t believe the picture caption is I’m gonna nut