Semantic Poetry with Google Apps Script

Imagine this: you recall a document about an intriguing subject but can’t pin down a specific term. It’s a common scenario where traditional search methods in Gmail or Google Docs often fall short, relying heavily on exact terms. Enter the realm of ‘semantic’ search, powered by advanced language models. ‘Semantic’ isn’t just a fancy word; it’s about understanding the meaning and context behind your words. Instead of a frustrating keyword hunt, these models interpret your descriptions, no matter how vague, to find that needle in the digital haystack. Ready to see how semantic search transforms your quest for information?

Hello, I’m Riël Notermans, and today I’m taking you on a unique journey with a twist – we’re exploring language models through a series of poems I have created (well, I had them created by my favorite AI poet). This is more than just a technical demonstration; it’s a practical look at how these advanced models can handle nuance and understand contexts. I hope you appreciate the demonstration, seeing how language models can be applied in innovative ways.

It gave me tons of inspiration how we could help our customers forward. I hope it will give you as well. As a Google Workspace Developer Expert, my go-to tool for tackling such challenges is our beloved Google Apps Script. The reason I wrote this document is because I have been really impressed.

I’ll guide you through the workflow that enhances a Language Model’s responses by integrating external text corpora. We’ll leverage semantic information retrieval techniques to accurately answer questions, utilizing the Semantic Retriever and Attributed Question & Answering (AQA) APIs within the Generative Language API framework. To illustrate these concepts, I’ve prepared some demo material for practical application.

On a rocky cliff, where the sea eagles cry,
A seagull’s wings refused to fly.
Grounded and lonely, watching others soar,
A dream unfulfilled, to explore.

With patience and time, and feathers that mend,
He took to the skies, with the wind as a friend.
Over oceans and cliffs, in the vast blue expanse,
In flight’s freedom, he found his chance.

What are we going to do?

Setup the endpoint and API calls
Defining and creating a corpus
Creating a document
Load chunks by traversing our Google Drive folder
Send chunks to our document and combine the above functions in a start() function.
Lets go!
Bonus!
Final remarks

Imagine this scenario: You have a vague memory of a poem that deeply moved you. It had something to do with a thirsty animal and the scorching heat, but the exact words escape you. In our demonstration, I’ll guide you through the process of rediscovering that elusive poem using our semantic search tool. With this technology, you can find the poem by describing it in just the way you remember it. No need for exact keywords; simply ask as if you’re talking to a friend. Ready to embark on this journey of rediscovery? Let’s begin.

Prerequisites
An apps script project: https://script.new
A Google Cloud project: https://console.cloud.google.com/projectcreate
Take an existing one, or create a new one. Get the project code (the number)
Configure your OAuth screen: https://console.cloud.google.com/apis/credentials/consent/edit.
Just the bare minimum is enough here.
Enable the API’s for this project.
Generative Language API: https://console.cloud.google.com/apis/api/generativelanguage.googleapis.com
Drive API: https://console.cloud.google.com/apis/api/drive.googleapis.com
Point your Apps Script project to the cloud project you created.
Copy the following appsscript.json into your script:

				
					{
 "timeZone": "Europe/Brussels",
 "dependencies": {
   "enabledAdvancedServices": [
     {
       "userSymbol": "Drive",
       "version": "v3",
       "serviceId": "drive"
     }
   ]
 },
 "exceptionLogging": "STACKDRIVER",
 "runtimeVersion": "V8",
 "oauthScopes": [
   "https://www.googleapis.com/auth/cloud-platform",
   "https://www.googleapis.com/auth/documents",
   "https://www.googleapis.com/auth/script.external_request",
   "https://www.googleapis.com/auth/generative-language.retriever",
   "https://www.googleapis.com/auth/drive.readonly"
 ]
}

1) Setup the endpoint and API calls

Here’s how you would set up the main variables and functions to achieve this.

This is a simple function to make our calls to the Generative Language API v1beta1 endpoint.

“The Semantic Retriever API lets you perform semantic search on your own data. Since it’s your data, this needs stricter access controls than API keys. Authenticate with OAuth with service accounts or through your user credentials.”

It uses your scripts’ OAuthtoken to authenticate! All too easy!

				
					const BASE_URL = 'https://generativelanguage.googleapis.com/v1beta/';

/**
 * Sends a request to a specified path with the given payload.
 * @param {string} path - The API endpoint path.
 * @param {string} method - POST or GET.
 * @param {Object} payload - The payload to be sent in the request.
 * @returns {Object} The response from the request.
 */
const doRequest = (path, method, payload) => {
 const options = {
   contentType: 'application/json',
   method: method,
   muteHttpExceptions: true,
   headers: {
     authorization: `Bearer ${ScriptApp.getOAuthToken()}`
   }
 };
 if (method === "POST") {
    options.payload = JSON.stringify(payload);
 }
 const response = JSON.parse(UrlFetchApp.fetch(BASE_URL + path, options ).getContentText());
 return response;
};

The first thing we want to do, is loop trough our folder structure. You maybe already enjoyed the fine work of poetry you found there. You maybe even remembered one or two now. Anyway, we need to get this data into the API. We do this by:

2) Defining and creating a corpus

The semantic retriever works with chunks of data (text), that are stored within documents, that on their turn are part of a corpus.
A corpus contains the documents and chunks. You can have 5 corpora per project. There are two ways to retrieve the data, by querying the corpus, or by querying a specific document. All depends on your use case. You will always get chunks and their metadata as a result. In this example, we create one document in one corpus where we store our chunks.

To create the corpus, we only need to give it a display name. The API will generate a corpus name by itself, so we do not send that key. We get this generated name back in the response. We need to use that name for further calls. (You can always set your own name, but I found this this easiest way to use it).

For each call here, I do search for the corpus by displayName, so in production environments, you could prevent that step by providing the name key at once.

				
					
/**
 * The corpus name this project is aimed at.
 */
const CORPUS_NAME = 'Semantic Poetry';

/**
 * Retrieves a specific corpus by name or creates a new one if it does not exist.
 * This function first attempts to find the corpus with the specified name.
 * If it's not found, a new corpus with that name is created.
 * 
 * @param {string} name - The name of the corpus to retrieve or create.
 * @returns {Object} The corpus object.
 */
function getOrCreateCorpus() {
  let corpus;
  const response = doRequest('corpora');  
  if (response.corpora) {
    corpus = response.corpora.find(c => c.displayName === CORPUS_NAME);
  }
  if (!corpus) {
    corpus = doRequest('corpora', 'POST', { displayName: CORPUS_NAME });
  }
  return corpus;
}

3) Creating a document

A document is a collection of chunks. As we mentioned in our case, we create one single document that we will call something like ‘Our poems’. We can attach custom metadata to a document as well, for later usage after retrieval, or to keep track of its relevance, or updates. In our code, we use the function below to create a document. As with the corpus, we only enter the displayName, and we will use the returned name later to add our data (chunks).

				
					/**
 * Creates a new document within a given corpus with specified title and metadata.
 * This function sends a POST request to create the document and logs the result.
 * 
 * @param {Object} corpus - The corpus object where the document will be created.
 * @param {string} documentTitle - The title of the document to be created.
 * @param {Object[]} documentMeta - An array of metadata objects for the document.
 * @returns {Object} The created document object.
 */
function createDocument(corpus, documentTitle, documentMeta) {
  const payload = {
    displayName: documentTitle,
    customMetadata: documentMeta
  };
  const document = doRequest(`${corpus.name}/documents`, "POST", payload);
  console.log(`Document created, current document ${document.name}`);
  return document; 
}

4) Load chunks by traversing our Google Drive folder

We can load a total of 1 million chunks in a corpus. A chunk can be 2043 tokens big. This means, in many cases we need to split our data. In this example, we ‘split’ our data into the single Google Doc files we have, and add them as chunks into our document. We need text data, so if you have any other file type then Google Docs, you should find a method to extract the relevant text parts. That goes beyond our scope, but there are many ways to do that. I have made it easy for us, by just creating Google Docs.

To read all Google Doc text content, we need a function to iterate over all our folders, fetch the Google Doc, and get the body as text.
Afher that, we create the chunk-array as described in the API documentation.

❕We save the folder name, url, and filename as customMetaData, which enables us to specify a filter for this folder in the search query! You can add 20 custom metadata keys.

The next functions take care of all this. I have 3 folders in the dataset, so this gives us the opportunity to specify a folder to our query later. This will help narrow down results if you want to.

				
					/**
 * Retrieves chunkdata for documents from a drive folder.
 * @returns {Object[]} An array of objects containing chunk data.
 */
function getDocumentChunks(document) {
  const rootFolder = DriveApp.getFolderById("1k_Nt-w4Tx_6EsVOCgpxv4ynsnHjc8HRR");
  const chunkData = chunkFilesInFolder(rootFolder, [], document);
  return chunkData;
}

/**
 * Recursively gets all files in a given folder and its subfolders.
 * @param {GoogleAppsScript.Drive.Folder} folder - The folder to process.
 * @param {Object[]} chunkData - Array to store chunk information objects.
 * @param {document} document - corpus document
 */
function chunkFilesInFolder(folder, chunkData, document) {
  var files = folder.getFiles();
  while (files.hasNext()) {
    var file = files.next();
    var chunks = [DocumentApp.openById(file.getId()).getBody().getText()];

    // we can split the fileContent in 2 chunks for demo-purposes
    // a chunk can be 2043 tokens big.
    // const chunks = fileContent.split(/\s{2}/g).map(chunk => chunk.trim());
    // our script expects chunks to be an array.

    chunks.forEach(chunk => {
      // the object the API wants to see;
      chunkData.push({
        parent: document.name,
        chunk: {
          data: { string_value: chunk },
          customMetadata: [
            { key: 'filename', string_value: file.getName() },
            { key: 'foldername', string_value: folder.getName() },
            { key: 'url', string_value: file.getUrl() },
          ]
        }
      })
    })
  }

  var subFolders = folder.getFolders();
  while (subFolders.hasNext()) {
    chunkFilesInFolder(subFolders.next(), chunkData, document);
  }
  return chunkData;
}

To illustrate what the chunking method should produce, chunkData is an array of objects like this:

				
					const chunkData = [{
    "parent": "document.name",
    "chunk": {
        "data": {
            "string_value": "chunk"
        },
        "customMetadata": [{
                "key": "filename",
                "string_value": "[filename]"
            },
            {
                "key": "foldername",
                "string_value": "[foldername]"
            },
            {
                "key": "url",
                "string_value": "[url]"
            }
        ]
    }
}]

5) Send chunks to our document and combine the above functions in a start() function.

We are almost there! Since we gathered all our data with our script, lets send it to the document. We load them using the batchCreateChunks, so we can send all 30 chunks in one call. At this point, lets run all we have. Call the separate functions from start() and we will do the steps we just created:

Get our corpus;

Create a document;

Load the chunks;

Send the chunks to the API.

				
					/**
 * Initializes the process of creating and processing a document within a corpus.
 * This function first retrieves or creates a corpus with a specified name. 
 * It then creates a document within that corpus, retrieves its chunks, 
 * and sends a batch request to add these chunks to the document.
 * 
 * @returns {void} This function does not return a value.
 */
function start() {
  let corpus = getOrCreateCorpus(CORPUS_NAME);
  
  const documentMeta = [
    { key: 'author', string_value: "AI" },
  ];

  const documentTitle = "My Poems 1.0";
  
  const document = createDocument(corpus, documentTitle, documentMeta);

  const chunks = getDocumentChunks(document);
  console.log("Loaded "+chunks.length+" chunks from folders.");

  const chunkResult = doRequest(`${document.name}/chunks:batchCreate`, "POST", { requests: chunks });
  console.log(`${chunkResult.chunks.length} chunks added.`);
}

In your script editor, select the ‘start’ function and run it! Traversing the docs might take a minute, but after that you should have a logging like this:

Information 	Document created, current document 
		corpora/semantic-poetry-icyu5vsbq8fa/documents/my-poems-10-fl90o4hxhtfz
Information	Loaded 30 chunks from folders.
Information	30 chunks added.

There we go, our corpus is ready to go!

6) Lets go!

Now it is time to use our created corpus! The question about that thirsty animal keeps itching me, so it is time to send in that question: “What was that poem, about a thirsty animal and a lot of heat?”

This is how you can query the corpus. In this code, I log the result and relevance in the console.

❕The interesting part is that you can work with the relevance here. You can try out what relevance value still makes sense for you, and have your application respond to that. I found that in our poem set, a relevance above 60% always gives me the proper answer for questions I have about my poems!

				
					/**
 * Test function to query
 * 
 */
function testMe() {
  const query = "What was that poem, about a thirsty animal and a lot of heat?";
  const corpus = getOrCreateCorpus(CORPUS_NAME);
  const result = queryCorpus(corpus.name, query, 5);
  console.log("Here are your results:")
  result.relevantChunks.forEach(c => {
    console.log("["+Math.round(c.chunkRelevanceScore * 100)/100 + "] "+ c.chunk.customMetadata.map(meta => meta.stringValue).join(" | "))
  })
}



/**
 * Queries a specific corpus with a provided query.
 * @param {string} corpus - The corpus name 'corpora/name'.
 * @param {string} query - The query to be sent.
 * @param {number} results - The amount of results to receive.
 * @returns {Object} The result of the query.
 */
const queryCorpus = (corpus, query, results) => {
  const result = doRequest(`${corpus}:query`, "POST", {
    query,
    resultsCount: results
  });
  return result;
};

The Semantic Retriever will respond to you with the best documents that fit into this query. So what do we get?

"What was that poem, about a thirsty animal and a lot of heat?"


Information	Here are your results:
Information	[0.62] The Elephant's Discovery | set_3 | 
https://docs.google.com/document/d/1miSjHxn6OO7AAOProt7W1bM5lSF0Xnx3eY5tr6XG7sE/edit?usp=drivesdk
Information	[0.61] The Camel's Oasis | set_1 |
https://docs.google.com/document/d/180NiREu7-DN09ZLWovXvXkfAEHDAl-V7hNYs7DgWzEs/edit?usp=drivesdk
Information	[0.6] The Lion's Hunt | set_1 | 
https://docs.google.com/document/d/1cWFvo-1uQsVk1qCFFaAhPD_HThRc7FojA-3vnMIgP4w/edit?usp=drivesdk

🎉

And, is ‘The Elephant’s Discovery’ indeed about a thirsty animal suffering from warmth?

Here it is:

In the vast savanna, where the grasses sway,
An elephant trumpeted in dismay.
The waterhole dried, under the sun’s relentless beat,
A life source vanished, in the heat’s defeat.

With memory’s guide, and her herd in tow,
To a hidden spring, they did go.
In the savanna’s heart, a new hope found,
Where water and life, once more abound.

Amazing, isn’t it? This is a very useful way to really get a lot of value out of your data. Now, with the above script, it should be an easy step to use your own data. Experiment with metaData, with more documents and chunks as well.

7) Bonus!

Since LLM and Generative AI is quite popular, we can leverage this as well. Instead of returning our documents, just have the AI explain what is going on in the poems!

Add the following code in your script:

				
					/**
 * Tests the generation of answers from a given corpus based on a query.
 * This function creates or retrieves a corpus and then uses it to generate an answer
 * for the specified query. It logs the result, including the answer's content and its probability.
 */
function testGenAnswers() {
  const query = "Why was the seagull sad?";
  const corpus = getOrCreateCorpus(CORPUS_NAME);
  const result = generateAnswerAqa(corpus, query);

  if (result.answer) {
    console.log(`I got an answer for you, with probability ${result.answerableProbability}`);
    console.log(result.answer.content.parts[0].text);
  } else {
    console.log(result.error);
  }
}

/**
 * Generates an answer for a given query from a specified corpus.
 * This function constructs a payload with the query and corpus information, then
 * sends a POST request to the 'models/aqa:generateAnswer' endpoint to retrieve the answer.
 * 
 * @param {Object} corpus - The corpus object from which the answer is to be generated.
 * @param {string} query - The query string for which the answer is sought.
 * @returns {Object} The result object containing the generated answer or an error message.
 */
const generateAnswerAqa = (corpus, query) => {
  const payload = {
    "contents": {
      "parts": [{ "text": query }]
    },
    "semanticRetriever": {
      "source": corpus.name,
      "query": {
        "parts": [{ "text": query }]
      },
      "minimumRelevanceScore": 0.5 //play with it
    },
    "answerStyle": "VERBOSE"
  };
  const result = doRequest("models/aqa:generateAnswer", "POST", payload);
  return result;
}

"Why was the seagull sad?"

The seagull was sad because he was grounded and unable to fly. He watched other seagulls soar through the air and longed to be able to join them.

Final remarks

The use of Language Models (LLMs) and embeddings, which form the backbone of this technology, marks a significant advancement in how we interact with data. By harnessing these tools, we’re able to search with our natural language, adding incredible value to data management and automation processes.

You’ll find the complete script at the link provided at the beginning of this article. Below I am including a function for removing all your corpora. If you’ve discovered other impressive use cases or have insights to share, I’d love to hear about them in the comments!

Improvements?

In this case, I suspect that the system does not really know what the individual poems are. If I ask “are there any poems that are not about animals” it says they all are. If I ask how many poems there are, it gives me the amount of 4. So while we definitely can ask about individual situations of the subject in the stories, it seems the system interprets it as one single story. If you have any tips to prevent this, I will be more then happy. I would try things like: “I have a rule that a poem must include an animal. What poems are not about animals?”. This would be a great use case in the real world.

Thank for staying so long!

				
					/**
 * Removes all corpora for this project
 * Removes all documents before deleting the corpus.
 */
const removeCorpora = () => {
  const result = doRequest("corpora");
  result.corpora.forEach(corpus => {
     console.log("Removing corpus "+ corpus.displayName);
    let more = false;
    do {
      const documents = doRequest(`${corpus.name}/documents?pageSize=20`);
      if (documents && documents.length) {
        documents.documents.forEach(doc => {
          doRequest(`${doc.name}?force=true`, "DELETE");
        })

        console.log("Removed " + documents.documents.length + " documents...");
        more = documents.documents.length == 20;
      }
    } while (more);
    let c = doRequest(`${corpus.name}`,"DELETE");
    console.log("Removed corpus "+ corpus.displayName);
    return c;
  })
  return result;
};