Build an AI scheduling agent for Google Calendar and Outlook

Why agents double-book

Most reports look alike. A Vapi support thread is titled Google Calendar checkAvailability tool not preventing double bookings. On the n8n forum, a builder writes about "my VAPI AI double booking appointments via Google Calendar".

The model is rarely the cause. The tools around it are. Four mistakes account for most double bookings, and each one is fixable in your code.

The agent checks the wrong calendars

Google's free/busy call answers only for the calendars you name. Its items field is the "List of calendars and/or groups to query." If the person keeps appointments on a second calendar and the tool asks about primary alone, that time comes back free.

Decide for each user which calendars count as busy, query all of them, and treat a calendar you couldn't read as busy, not free. The free/busy guide covers both providers' calls and what each one counts as busy.

Time zones the model has to guess

A model doesn't know today's date or where the caller is unless you tell it. "Thursday at three" needs both.

The providers don't agree on defaults either. Google requires an offset on an event's dateTime "unless a time zone is explicitly specified in timeZone". Microsoft Graph's event resource says of start: "By default, the start time is in UTC." You get local times only if you send Prefer: outlook.timezone. An agent that reads a Graph time as local time reads the wrong hour back to the caller.

Put the current time and the user's IANA time zone in the system prompt. Make your tools take and return UTC timestamps ending in Z, and convert to local time only when the agent speaks or displays a time. Pyas gives every event startsAt and endsAt in UTC, plus startsAtLocal and an IANA timezone, whatever the provider stored. Test a booking in the week the clocks change, such as the first Sunday in November in the US.

Nothing stops a write into a busy slot

Neither Google's events.insert nor Microsoft Graph's create event has a parameter that refuses an overlapping event. The create succeeds whether the time is free or not, so the check is yours, and it has to be close to the write.

On a phone call, a minute can pass between the agent offering three times and the caller choosing one. Someone else can book the same slot on your website in that minute. Pyas has no lock either: two people can pick the same slot at once. Check that one slot again immediately before creating, and if it's gone, offer the next one.

Reschedules have the same race. When the agent moves an event, pass ifMatch: event.version so the update is refused if someone changed the event after the agent read it.

Retries create duplicates

Voice platforms and workflow tools retry calls that time out, and models sometimes call the same tool twice. Each repeated create is another event on the calendar.

The providers handle this differently. Google lets you choose an event's ID on insert: base32hex characters, 5 to 1024 long, "unique per calendar". Microsoft has transactionId, "A custom identifier specified by a client app for the server to avoid redundant POST operations in case of client retries to create the same event."

Pyas uses one rule for both. The SDK sends an Idempotency-Key on every write and reuses it on retries. Pass your booking id as the key and the API replays the first answer for 24 hours, so a retried create never makes a second event. The idempotency docs have the details.

Google and Outlook in one agent

Many agent templates cover only one of the two. Supporting both by hand means two free/busy calls with different answers (Google returns busy times only; getSchedule returns statuses and working hours), two event formats, and two sign-in setups.

With Pyas, the same tools work on either. meetingTimes.find takes Google and Outlook accounts in one call. Meeting links follow the calendar: Google Meet on Google, Teams on Outlook work accounts, and Zoom on either when the host's Zoom account is connected. Teams on a personal Outlook.com account is refused with an error that says why, so the agent can offer another option. Users connect through Pyas Auth, so you skip Google's OAuth review.

Let the agent act only through tools you control

The model should never choose whose calendar it touches. Resolve the user on your server and build the tools from that, as with pyasTools({ subjectId: user.id }). Don't take an account id from the conversation, and don't hand the model your API key or a generic HTTP tool.

Separate reading from writing. Each Pyas MCP tool is marked as reading, changing or deleting, and --read-only offers only the ones that change nothing. A receptionist that only answers "when is Dr. Lee free?" never needs delete_event. For the tools that do write, have the agent read back the time, in the caller's time zone, and the attendee before it creates anything.

Voice receptionists on Vapi, Retell or n8n

Pyas has no integration with Vapi, Retell, n8n or any voice platform. A voice or workflow tool calls Pyas through its own HTTP or function-calling features.

Point that tool at an endpoint on your own server rather than straight at Pyas. Your secret key stays on your server, and your code decides which calendar the call may read and keeps the answer short enough to speak.

// Called by your voice platform's HTTP tool. Map its payload to from and to.
app.post('/tools/:practiceId/find-times', async (req, res) => {
    const practice = await practices.get(req.params.practiceId)
    const slots = await pyas.meetingTimes.find({
        accountIds: [practice.calendarAccountId],
        from: req.body.from,
        to: req.body.to,
        durationMinutes: 30,
        timezone: practice.timezone,
        limit: 3,
    })
    res.json({ slots: slots.map((slot) => ({ startsAt: slot.startsAt, endsAt: slot.endsAt })) })
})

When the agent creates the event, Google or Outlook sends the caller the calendar invitation. Any other confirmation or reminder, by email or text, comes from your app. Pyas also doesn't offer booking pages or round-robin; your agent decides who gets the meeting.

Test against the sandbox first

Every Pyas app has a sandbox: sample people with real-looking calendars and simulated Google, Outlook and Zoom accounts, reached with pyas_sk_test_... keys that never touch a real calendar. Run the conversations that break agents before a real customer does:

• "Next Friday", said on a Friday.
• A booking during the week clocks change.
• Two conversations picking the same slot at once.
• A create that times out and is retried.
• Teams requested on an Outlook.com account.

The MCP tools list_recent_requests, get_request and explain_error show what the agent actually sent and why a call failed, which is faster than reading the model's own account of what it did.