Just Box It (Typescript Edition)

Jan 21

GPT-4 is stupid. There, I said it. (It’s either that or I’m really really bad at prompting, but come on). It’s incredible to me that there is no way to reliably generate structured JSON like JSONformer lets you do with open models. Calling functions is such a core part of any useful workflow that goes beyond generating text. In fairness, this seems much more difficult to do at OpenAI scale since you’re serving models to 1000s of people. Still I’m shocked at how inflexible the whole setup is today.

Anyway, here are a couple of useful tricks I’ve learned:

Use one call to pick which function to use, and the next call to execute that function. You can’t force GPT-4 to use a function and not text, the only options are auto (might output text), never (never use a function) or function_name (only use this function). A simple workaround is to create a meta-function that acts as an action selector whose output space is constrained to the names of the other functions.
Use Typebox. I don’t know what kind of abstraction magic is in the water in South Korea, but sinclairzx81 has created an incredibly clean library that lets me generate JSON schema automagically while also giving me the right Typescript types. It feels even better than Pydantic to me because Typescript has a type system I rarely have to wrangle with.

Let’s put this all together with an example:

import { Type, type Static } from '@sinclair/typebox';
// ...
const ReACTSchema = Type.Object({
  thought: Type.String(),
  action: StringEnum(Object.values(ActionNames)), // ActionNames is the Typescript enum of actions.
});
type ReACTSchema = Static<typeof ReACTSchema>;
// ...
// Now the function call parameter looks like this. So clean!
  {
    type: 'function',
    function: {
      name: ToolNames.ReACT,
      description: 'Think and select an action.',
      // You need a Strict type to remove all the Typebox specific metadata.
      parameters: Type.Strict(ReACTSchema),
    },
  },

You can even validate the params that OpenAI returns to you oh-so easily:

import { Value } from '@sinclair/typebox/value'
// ...
const res = await openai.chat.completions.create(...)
const functionArgs = res.choices[0].message.tool_calls[0].function.arguments
const toolParamsParsed: Static<typeof toolParamSchema> = JSON.parse(functionArgs);
// toolParamsSchema can be ReACTSchema for example.
if (!Value.Check(toolParamSchema, toolParamsParsed)) {
  throw `Tool params ${JSON.stringify(toolParamsParsed, null, 2)}\n did not match schema ${JSON.stringify(toolParamSchema, null, 2)}`;
}

You might have noticed the StringEnum object (defined here) I’m using above. I discovered that StringEnum worked better than TypeBox’s default T.Enum for reliable action generation. StringEnum uses the enum field of JSONSchema whereas T.Enum uses anyOf: perhaps GPT-4 is more optimized for the former. As a final bonus to this long log, here’s how you define validation on StringEnum:

import { TypeRegistry, Kind } from '@sinclair/typebox';

const StringEnum = <T extends string[]>(values: [...T]) => Type.Unsafe<T[number]>({
  type: 'string',
  enum: values,
  [Kind]: 'StringEnum'
})

TypeRegistry.Set('StringEnum', (schema, value) => {
  // @ts-ignore
  if (schema.type === 'string' && schema.enum && schema.enum.includes(value)) {
    return true;
  }
  return false;
});