Before ChatGPT: What My Master's Thesis Taught Me About Meaning in Text

On 10 November 2021, I submitted my master’s thesis: Implementation, Evaluation, and Comparison of the GLDA Topic Model. At the time, I thought I was writing about a fairly specific piece of natural language processing research. Looking back in the ChatGPT era, it feels more like a small snapshot of the moment when language models were learning to treat words as geometry.

The thesis was about Gaussian latent Dirichlet allocation, or GLDA. That name is not exactly designed to attract readers. The underlying question, however, is still a good one:

Can we discover what a collection of documents is about without being told the topics in advance?

That question sits underneath search, recommendation systems, document exploration, and many of the systems we now casually call “AI”. The interesting part is not only whether a model produces useful output. It is how it represents meaning, what it groups together, and what it loses along the way.

From a bag of words to a space of meanings #

The classic answer to topic modeling is latent Dirichlet allocation (LDA). A document is treated as a mixture of topics, and a topic is a probability distribution over words. A news article might be 40% politics, 30% economics, and 30% international affairs. The model is not given those labels; it infers them from word usage across the corpus.

LDA is elegant, but it treats words as discrete identities. The word car is one item in a vocabulary, and automobile is another. Unless their surrounding usage causes the model to connect them indirectly, there is no built-in sense that they are related.

Word embeddings offered a different intuition. Instead of representing a word as an ID, represent it as a point in a high-dimensional space. Words that occur in similar contexts tend to end up near each other. Meaning becomes something that can be approximated with distances and directions.

That was the central change in GLDA: replace LDA’s distributions over discrete word types with Gaussian distributions over word-embedding vectors. A topic was no longer just a list of vocabulary items. It became a region in an embedding space, with a mean and a covariance describing its shape.

This sounds like a small modification. It is not. Once words become 50- or 100-dimensional vectors, the inference problem becomes much more expensive. Probability mass functions become probability densities. Covariance matrices enter the calculation. Numerical linear algebra becomes part of what used to look like a straightforward topic model.

The part where probability meets engineering #

The thesis had two practical goals. First, I implemented two inference methods for GLDA—collapsed Gibbs sampling and stochastic variational inference—in the InfoFormalizer framework. Second, I built an experiment pipeline to compare GLDA with a conventional LDA implementation under the same conditions.

The framework mattered as much as the model. Comparing machine-learning algorithms is surprisingly easy to do unfairly. Different preprocessing, different corpora, different evaluation tools, or an undocumented parameter can make a result look much better than it really is. I wanted the data preparation, training, metrics, and downstream tasks to live in one reproducible pipeline.

The implementation also made the computational trade-off impossible to ignore. LDA was comparatively cheap in these experiments: with count-based topic-word statistics, its common inference operations avoided the repeated full-covariance linear-algebra work required by GLDA. GLDA repeatedly evaluates distributions in the embedding space and works with covariance information. In my straightforward implementation, the costly operations involving full covariance matrices—such as matrix factorization, inversion, and determinant evaluation—scaled cubically with the embedding dimension.

On the laptop I used for the experiments, the difference was very concrete: LDA experiments generally finished in minutes, while some GLDA experiments on the movie-review data took five to ten hours. More semantic information is not free.

That lesson has aged well. Modern language models hide enormous computational costs behind a friendly chat box, but the basic question remains: is the extra compute buying us a capability we actually need?

A topic model can be understandable without being useful #

I evaluated the models on three very different corpora:

• BBC News, with recognizable categories such as sport, business, politics, and technology.
• NIPS papers, a narrower and more technical collection.
• The Large Movie Review Dataset (ACL IMDB), where the labels describe sentiment rather than subject matter.

There were two ways to judge the topics. The first was qualitative: could a person look at the top words and understand a theme? The second was quantitative: did metrics such as topic coherence or a downstream classifier suggest that the model had found something useful?

Those answers did not always agree. GLDA often produced human-readable topics on the news corpus, but it also found broad, recurring topics made up of words such as verbs, names, or temporal expressions. Because similar words in embedding space can appear in similar contexts, GLDA sometimes grouped words that were semantically related in a broad sense but not especially informative about the corpus.

The difference was particularly visible in the sentiment task. Distributional embeddings can place antonyms close together because they often occur in similar contexts. For example, “the actor was good” and “the actor was bad” have similar grammatical structure. That is useful if the question is about distributional similarity. It is less useful if the question is to separate positive from negative reviews.

The classification results made the point more sharply. On BBC News, LDA and the 50-dimensional GLDA model performed similarly. With 60 topics, LDA reached 90% accuracy and GLDA-50 reached 91% in my experiment. On ACL IMDB, however, LDA reached 76.2% while GLDA-50 reached 64.0% at the same topic count. The model that captured more general semantic similarity was not automatically the better feature generator for sentiment.

This was one of the most useful conclusions of the work: a model can find structure without finding the structure that your application needs.

The timing is hard to ignore #

I handed in the thesis before ChatGPT existed as a public product. GPT-3 was already part of the conversation, and word embeddings were well established, but the current LLM era had not yet made language models a daily tool for millions of people.

Since then, the center of gravity in NLP has moved. Static embeddings such as the GloVe vectors I used assign one vector to a word type. Transformer language models build contextual representations: the word “bank” can be represented differently in “river bank” and “bank account”. Large models can also generate explanations, summaries, labels, and even plausible names for the topics they discover.

That progress changes how I think about the thesis, but it does not make the underlying problem obsolete.

An LLM can summarize a corpus, but its answer is not necessarily a transparent map of the corpus. It may confidently invent a category, merge distinct themes, or describe the most salient documents rather than the most representative structure. A topic model is much less eloquent, but its assumptions and measurements are easier to inspect. In conventional LDA, you can inspect topic-word distributions. In GLDA, you can inspect the Gaussian topic parameters and the words most representative of each topic, alongside document-topic vectors, preprocessing, and the evaluation pipeline.

There is also a useful distinction between representation and explanation. Embeddings help a model organize language in a geometric space. An LLM can turn that organization—or its own internal representation—into a fluent explanation. Fluency is valuable, but it is not the same thing as faithfulness.

Today I would probably revisit the experiments with contextual document and token representations, stronger baselines, and better hyperparameter tuning. I would also ask people to judge the topics directly instead of assuming that a coherence metric—such as the UMass or cPMI measures I used—corresponds to human understanding. Those are natural extensions of the thesis’s unfinished work.

What stayed with me #

The biggest surprise was not that GLDA was slower than LDA. The equations made that fairly predictable. The surprise was how difficult it was to define “better” for an unsupervised model.

A higher score on a coherence metric does not necessarily mean a more useful topic. A better classifier does not prove that the model discovered the most interesting structure. A topic that appears in many documents might be noise, or it might be the background context that makes the specialized topics meaningful.

The experiments also reinforced a principle I still use in software engineering: comparisons are only meaningful when the path to the result is visible. A model name, a benchmark number, or a polished demo is not enough. We need to know what data entered the system, what assumptions were made, what was measured, and where the method fails.

That is why I am glad I worked on GLDA before the current wave of LLMs. The project was small compared with today’s models, but it forced me to look closely at the machinery behind a claim about language and meaning. It taught me that putting words into a vector space is powerful, but it does not remove ambiguity. It only gives ambiguity a coordinate system.

And sometimes, before asking a model to explain a corpus, it is still worth asking a simpler question first: what structure is actually there?