What I learned building an app for the ChatGPT Store

It can be hard to grasp how the ChatGPT store and app builder work unless you’ve gone hands-on with them. So I did.

What happened: Last week, OpenAI launched its long-awaited GPT Store, where developers can share chatbots they’ve built.

• Rather than writing code, developers create apps using text-based prompts in GPT Builder, which is powered by the GPT-4 large language model.

Why it matters: GPT Builder has been available since November, but the ability to browse and share GPTs is being compared to Apple’s app store launch, attracting more outside developers to its platforms and showing users the more refined experiences that are possible.

• It may also help drive ChatGPT Plus memberships, which you need to access both GPT Builder and GPT Store.

What I did: I set into GPT Builder with two app concepts: Peak Dragon, which evaluates ideas for a new startup or product, and Insightful Peak Explorer, which pulls insights and simple overviews of news stories from previous Peak and Peak Tech newsletters.

• I came up with Peak Dragon’s name, but Insightful Peak Explorer was suggested by ChatGPT after I gave my initial description of the app.

After describing my apps, specifying how friendly they should be, detailing how many questions they should ask, and getting some DALL-E-generated logos, I got down to building. Here’s what I learned:

You are programming a user experience: This is not app development in the traditional sense — you won’t be able to create anything that ChatGPT can’t already do. Most of what you are doing is setting parameters and guides so the user doesn’t have to, crafting an experience that gets them to what they want.

• Most of my “programming” was prompting the bots on what information to get from a user, how to use that info, what to include or exclude from responses, and how it should be asking questions.

You have to be very detailed: There were times when the whole process felt like talking to a genie — it will give you what you want, but you have to predict every possible loophole and possibility when you ask, or else you’ll get some surprises.

• That’s likely why many are predicting that “prompt engineering” will become an in-demand job skill, if not a distinct role unto itself.
   
• My first run-through pitching Peak Dragon on a delivery app for pet products resulted in it asking if I’d like ideas for a different app I could make, which stung.

Be ready to take a lot of breaks: ChatGPT Plus members have a limit of 40 messages every three hours, which I burned through very quickly, several times. It seems like that’s inevitable in a coding system that’s based on sending messages to both build and test an app, as well as constant follow-up messages to correct and refine the experience.

• GPT Builder has an area where you can directly type out and edit conversation-starter prompts and high-level instructions, but many things I did there would change on their own after other prompts.

It can be forgetful: Just like regular coding, glitches and things you don’t want to happen can come up, even when you think you’ve fixed the problem.

• Despite telling it several times to spread out questions, Insightful Peak Explorer seemed determined to ask multiple questions about what someone wanted to know about news stories all at once.
   
• I wrote a very detailed prompt that would get Peak Dragon to generate a cool startup name, but it simply would not remember to ask the user if they wanted one.

You can teach it things yourself: The builder gives you the option of uploading “Knowledge,” a file it can pull info from when completing tasks.

• The first time I asked Insightful Peak Explorer to explain Wi-Fi 7, it pulled a description I would have gotten from a Bing search. After uploading one of last week’s Peak Tech editions, the bot explained Wi-Fi 7 using nothing more than our explainer, matching the focus and writing style.
   
• ChatGPT has said “Knowledge” will only be used in the GPT it was added to. It will not train current or future models, and users are blocked from downloading the data.
   
• You can also turn off the bot’s ability to search the web for more info, which is useful for controlling the bot’s answers and — as was the case for Explorer — limit how often it might pull in info from unverified (or uninteresting) sources.

Bottom line: The apps I created were fine, I guess, but they lacked nuance. Insightful Peak Explorer had trouble picking out the interesting parts of a story, and tended to leave out important details. Peak Dragon gave advice that seemed accurate, but was ultimately no different than what someone might be able to source from a few web searches.

• Sourcing and organizing that info could still be useful for some users, though, and it seems possible for someone with time and motivation to refine the experience with more organized Knowledge documents and focused prompting.

Yes, but: Using GPT Builder, browsing the GPT Store, and even using a chatbot someone else has created requires a ChatGPT Plus membership. When people still aren’t sure if they even want to use chatbots, a closed-off, paid experience isn't likely to deliver on the "app store for chatbots" promise or move the needle on adoption any time soon.

If you happen to have a ChatGPT Plus membership, feel free to try Peak Dragon and Insightful Peak Explorer for yourself.