Wondering if Amazon Q is worth it? In this second part of our blog post series, I dive into the Amazon Q user experience.

And in case you missed it, I already covered Amazon Q basics, set up, and business and development use cases in part 1 of this article series, where I exposed the good, the bad, and the ugly!

Using Amazon Q Business for Internal Chatbots

Tailored to a non-technical audience, Amazon Q Business is probably the aspect of Q that interests the majority of people. This set of tools and services enables you to build chatbots based on your organization’s data, allowing you to easily customize an LLM. It comes with various data import adapters, a rudimentary GUI editor, and hosting—all of which are accessible via a web interface.

I found Q Business to be the most impressive part of Amazon Q—not because the LLM was exceptionally smart or eloquent, but because of the nice suite of tools AWS has built around it. With today’s AI hype, everyone wants to have a chatbot on their website, but getting data into those systems is no easy feat. 

AWS provides “data sources” (as they call the importers for Q Business) for all kinds of information storage. This includes low-level data sources for popular databases like Postgres and high-level data sources like a web crawler. 

If you already have a blog or documentation hosted somewhere, you can quickly hook it up to a chatbot. Just make sure to check your billing statements. While AWS claimed Q would be free until the end of July 2024, we ended up getting billed around $50 for our chatbot, with no clear insights as to how we racked up these costs.

I tried to build a Q chatbot with the IOD blog, which only took a few minutes. The configuration was easy and flexible. With URL filters, I was even able to exclude other pages from our website to ensure that only the blog posts were included.

The GUI editor wasn’t up to par with the quality of the data ingestion tooling, but it was an okay tool for quickly creating a basic web front for the chatbot to quickly get something up and running. 

It’s enough to build an FAQ based on your organization’s internal data so your teams can get answers fast. It even includes the sources for the answers so everyone can verify their correctness.

But the real cherry on top is Q Business’ guardrails feature, as it allows you to filter out answers. You can use it to protect part of the crawled information from your data source or to prevent answers on topics that could result in liabilities (e.g., health or security).

Finally, since Q is essentially another AWS service, it integrates well within IAM, and you can manage access to your chatbot just like with any other AWS service. Q Business also has an API. Everything that can be done in the AWS Console can be done with the API and more. 

For example, you can generate a chatbot with the Console and then use the API to initiate chats via another frontend after testing it with the no-code web experience Q Business provides out of the box. The API also enables direct control over the indices of your RAG retrievers, which isn’t possible in the Console. This enables more granular control of your Q Business applications.

The API is rather complex, but if you’re used to AWS APIs, this shouldn’t stop you. The documentation is also extensive.

Getting Programming Support from Amazon Q Inside an IDE

I wasn’t as impressed with Q’s IDE integration and programming support as I was with the chatbot tooling, but it was okay. It’s a basic programming companion, just like all the others that have cropped up since the LLM hype started. It’s not bad, though not exceptionally good either.

The plumbing around Q Developer gets better with every update. You can mark code in your editor and ask Q in the chat for a refactor. It will generate and explain the refactored code, and you can add it with one click on “Insert at cursor.” 

For example, in Figure 1, you can see how I asked Q to refactor a function from an iterative style to use recursion instead.

The autocomplete is okay. But like with any of these programming companions, it just seems to regard the current file when suggesting. This can be jarring when you have already “taught” Q your way of writing things in one file, then you start the next one, and it gives you a suggestion that requires you to start from zero.

Q Developer would benefit from a bigger context. The conversational memory seems to only include the file you’re currently active in, so it forgets your coding style or functions from files you have closed. This doesn’t pose a problem with basic style issues your formatter can solve after completion; but if you want to implement your algorithms with functional programming and Q is always showing you an imperative solution first, it can drive you crazy after a few files.

However, the IDE integration has gotten better with recent updates. While you can’t move the chat window from the sidebar (Figure 1), at least you can now add code snippets from the chat with one click.

Implementing a Blog with Q Developer Agents

For me, the developer agents were the most intriguing feature of Amazon Q but also the most disappointing.

Agents are more or less autonomous instances of AI. Instead of just answering your questions, they can: 

• • Plan actions to complete a task

• • Write code in your environment

• • Start other agents to help them with their work

They come in two flavors: via IDE or CodeCatalyst integration, but both present the same issues.

Q Developer Agents in the IDE

The most straightforward way to use the agents is via the IDE extension. As with Q’s summary, explanation, and refactoring features, you can start a developer agent session, i.e., a new chat window where you explain to Q what you want it to build. 

Unlike other development features, the agents can write your code files, so you don’t have to copy and paste around your repository, which is a plus. 

The process of telling the agents what to do is simple—in theory. You explain your goal, and the agent generates a plan showing how it would accomplish this goal. You can discuss this plan with the agent, and if everything is good, you tell the agent to implement the solution. 

Then, you can discuss the solution with the agent to eliminate issues; and in the end, the agent will put the new files into your codebase. It even uploads your codebase to Q before generating the plan to analyze your code before making suggestions. This is much better than the auto-complete feature that only accounts for the code from the file you’re currently typing in.

However, the process doesn’t work in practice. After the planning and implementation, the discussion steps are a good idea, but the moment you tell the agent that its plan or code requires changes, it forgets everything you talked about before. 

I tasked it with a REST API for a blog, so it generated a plan. When I saw it forgot authentication, I asked it to include this in the plan, but then it generated a plan solely for the authentication. 

I expected it to use the new plan with the old one, but in the end, it just generated code for the authentication. This happened with each of my additions.

I also told Q to ensure it added unit tests; it only wrote tests, not the API. I asked it to add a simple test frontend to show API usage; it only generated the frontend, not the API. 

Every change suggestion in the process destroys everything that comes before it, which doesn’t make much sense. And this is just a tangentially LLM-related issue. Q simply doesn’t ensure that the prompts for the agent include the previous plan.

I created more sophisticated prompts to include everything I could have missed so I didn’t have to tell the agent later to update its plan (which would have led it to forget everything we had discussed in the session). But the code it created was often incomplete. 

Strangely enough, it had some inner consistency when leaving out code. For example, it would create two files, server.js and database.js, both of which would only include code for authentication and reading articles, but not for creating, updating, or deleting them.

The developer agents can solve bigger tasks than the inline completion or the developer chat. The agent implemented a working authentication for my blog, including SQLite integration. For that, the agent had to select the right packages and create multiple files, which was certainly helpful, even if I had to update the packages and clean the code up later. But with the main functionality missing, it was quite disappointing. 

It’s also hard to find the right task size for the agents. Too big, and it will omit code; too small, and you can get faster results with inline completion. Right now, I would avoid that feature unless you have time to experiment until you find something that works.

Q Developer Agents in Amazon CodeCatalyst

CodeCatalyst is one of AWS’ CI/CD services. It is similar to GitHub but from AWS. With it, you can create Git repositories, issues, deployments, etc.

CodeCatalyst was the only CI/CD service integrated with Q Developer at the time of writing this article, so I wasn’t left with much of a choice when using Q Developer outside of an IDE.

The CodeCatalyst integration was a bit more complex to set up since it isn’t that well integrated with other AWS services. You need an AWS Builders ID, which differs from your regular AWS account. However, when you create a new CodeCatalyst space, you’re asked to link it to your AWS account.

Integrating a developer agent into CodeCatalyst makes more sense than integrating it into an IDE. It takes some time to do its work, and in contrast to that IDE chat window, CodeCatalyst’s issue tracker is better suited for asynchronous conversations. 

I recorded a short video showing the workflow with Q Developer in CodeCatalyst. I created a React project and wanted Q to implement an About page.

The video contains the following steps:

1. 1.  I create a new issue and assign it to Q.
      
      
   2. Q analyzes the issue and creates two tasks.
      
      
      • The first task is to create the About page component.

         

      • The second task is to integrate the component into the routing.

         

   3. I assign the first task, the creation of the component, to Q.

       

   4. Q analyzes the task and suggests a solution, which I accept.

       

   5. Q implements the solution and sends pull requests, which I merge.

Everything worked pretty well until this point. The new component looked legit, and I decided that Q should work on the second task. However, the proposed solution for the second task didn’t make much sense. 

I went ahead with it to see the end result (and because Q’s conversational memory doesn’t work well between comments). But it only changed a workflow file, despite mentioning other changes in the pull request description.

I don’t know what Q’s problem was in the second task, but after countless tries, it seemed that it somehow liked to change workflow files, even if it had nothing to do with the task. If I prevented Q from accessing workflow files, Q complained that it needed access and wouldn’t continue its work.

In my experiments, Q created a workflow file with a shell script to write the TypeScript code I had requested. I had to execute the workflow to generate the actual code. But the code workflow file was broken too. Q could have used the workflow validator to check its generated code, but it could not access it. Again, not an AI problem, just an immature solution.

Final Thoughts

Amazon Q is a mixed bag. While Q Business is a nice product that even non-technical people can use, Q Developer is, at best, an average developer AI. 

See Table 1 and Table 2 below, where I sum up the pros and cons along with a brief summary for each.

Amazon Q Business brings real benefits to organizations that need chatbots and want them well integrated into their AWS-based stack. It’s especially useful for non-technical folks, as it comes with basic GUIs and a comprehensive API. However, it’s difficult to set up for a non-technical person; the GUIs aren’t very customizable, and the API is quite complex for a simple chatbot.

Table 1: Amazon Q Business pros and cons

Amazon Q Developer is an okay service. The autocomplete works like every developer’s AI service and delivers acceptable results. The chat is a nice extra that allows you to discuss refactorings right where you write your code. The CodeCatalyst integration is certainly interesting, and as long as you don’t assign Q complex coding tasks, it can handle basic issues like a human would. Yet, it isn’t obvious where the limits of the developer agents lie. Often, they deliver incomplete or wrong results if the task is too complex.

Table 2: Amazon Q Developer pros and cons

Stay tuned for part 3 of our AWS Q series, where we examine AI market trends, AWS Q maturity, competition, alternative tools, and the future of products that support GenAI.

Need AWS-related blog content?

Since its founding, IOD has been serving AWS Partner Network clients, delivering technical content at scale written by experts with deep hands-on experience in cloud, security, DevOps, AI, data engineering, and more. 

Cut AWS review times and achieve impressive ROI.