LLMs are the squishy middle between human and machine
They can transform structured data really fast
Structure is power
Our ability to structure the world of data defines what we can do with computers. If you take a picture of a traffic intersection and ask a computer to determine if it's safe to drive through, the computer will have no idea how to respond.
But if you label the picture and say "there are 5 pedestrians, and 2 cars, and these are their positions, and this is the traffic light, and it's green, and green means legal to go" - you've turned the unstructured data (pixels of a raw image) into structured data (physical and legal conditions relevant to a safety evaluation). The computer can make an assessment, and indeed this is how self-driving cars are built.
We've historically needed humans to structure data: to label an image as a banana or a cookie, determine from a transcript if a support call was a success or failure, or gauge an essay's style as New Yorker or InfoWars.
This is why LLMs have been transformative - because they can automate the process of structuring text-based data. They can make semantic judgments, so they actually can assess a phone call's outcome and the style of an essay.
This is also why structured extraction is such a powerful concept. How we structure information defines what we can do with it.
At ESAI, we use AI to help students with the college admissions process. Previously, when our AI tools would output a long block of text with college application advice, we couldn't select and save pieces of those responses. Even if we asked the LLM to generate "three recommendation cards," from the machine's perspective, there was no distinction - it was all just text.
LLMs can transform data really fast
Human-structured. a computer does not know these are three cards:
Intermediary. an LLM transforms the data through semantic understanding:
Machine-structured. a computer does know these are three separate cards:
In this example, both plaintext and JSON are structures readable by humans. For the plaintext, if you asked someone to point to the title of each card they would point to the title. If you asked someone to point to the description they would point to the description.
But to a computer, the plaintext is unstructured. If you pasted that raw text into Notepad and ⌘-F searched for “title” or “description,” you would get no results. On the other hand, JSON is a structure that a computer understands. So if you pasted the JSON into Notepad and ⌘-F searched for “title” or “description,” the computer could point you to each one. The computer could also give you interesting new data, like how many titles and descriptions there are and how many characters in each.
The crazy thing is that the data on the left was transformed into the data on the right by an LLM! So an LLM isn’t quite a computer because it can understand unstructured data, and it’s not quite a human because it doesn’t have values of its own - I have to tell it what structure I want. I think LLMs are an interface between humans and computers that allow us to squishily transform data that is human-structured to data is that is machine-structured, so we can do really powerful things with that data.
And once we have the original dataset, it’s powerful to use LLMs to transform that data into various forms really fast:
Fast transformation of unstructured data is how you solve problems for users
Today at ESAI we use structured extraction to solve UX problems. The output of our Story Strategist tool previously outputted a single wall of text:
We wanted to break up how the outputs looked to users. To “teach” our frontend to render each the content for each card separately, we had to give it structured data with a defined list of cards and their titles, descriptions, and tool links. Now we’re moving into letting users save individual cards, so we’re “teaching” our backend what a card is and what a bucket is. Next, structured extraction can enable even more UX wins, like generating and rendering links to multimedia within our cards.
Here, structured extraction allowed us to teach our system about concepts like “card 1,” “card 2,” “card 3,” “title,” and “description.” Our frontend could treat these as separate entities and display them differently. Now that our backend knows what a “card” is, we can also give the user the ability to save individual cards. Next, structured extraction can enable even more UX wins, like generating and rendering links to multimedia within our cards.
If you expand structured extraction to include “provide some code here if the user asks to pull data from the Common App,” we can imagine integrating “agentic” behavior in our tools to give our LLMs the ability to pull live data from public or private APIs. This is actually super easy nowadays with tools like the Vercel AI SDK.
We can easily picture the long-term vision of ESAI, which allows our users to transform their histories into unique, shareable artifacts. LLMs are the translation layer between their stored historical data and the structure they want to transform it into - whether that’s a resume, a pitch deck, or a video.
What we can structure defines what we can do with computers. As software engineers we create hierarchies and ontologies for defining the world. Now, we don’t have to process unstructured data into hierarchies ourselves - we can have the LLM do it for us, and solve a whole new class of problems for users.
Idk, maybe in a few years storing data in tables will be rare. All data will be stored as a huge plaintext or JSON blob somewhere, and LLMs will reorganize that data into our desired hierarchies on the fly. Structure on-demand, courtesy of our alien LLM translators.