Skip to main content

How do you deploy generative AI on AWS?

Generative AI on AWS explained: Amazon Bedrock versus SageMaker, retrieval-augmented generation over your own data, guardrails, access control and cost.

short answer

Short answer

Most companies deploy generative AI on AWS through Amazon Bedrock, which gives API access to foundation models inside your AWS account. You connect the model to your own documents with retrieval-augmented generation, restrict access with IAM, add guardrails, and log every request. Amazon SageMaker is the alternative when you need to train or host your own model.

By the Celestibia Solutions engineering team · Published

Bedrock or SageMaker

  1. Amazon Bedrock

    A managed service that exposes foundation models from several providers through one API. There are no servers to run. It suits assistants, document question answering, summarization and agents, which is where most projects start.

  2. Amazon SageMaker

    A platform for training, fine-tuning and hosting models on infrastructure you size yourself. It suits teams with their own models, strict latency targets or a need for full control of the serving stack.

Connecting the model to your data

A foundation model knows nothing about your company. Retrieval-augmented generation (RAG) fixes that without retraining: your documents are split into passages, indexed in a vector store, and the passages most relevant to each question are passed to the model along with the question. The model answers from that material and can cite it.

On AWS this is commonly built with Amazon Bedrock Knowledge Bases over documents in Amazon S3. The quality of the answers depends mostly on the quality and freshness of the indexed documents, so the data pipeline deserves as much attention as the prompt.

Security controls to put in place

AWS states that Amazon Bedrock does not use your prompts or outputs to train its base models and does not share them with model providers. Confirm the current terms in the Bedrock data protection documentation before relying on this for a regulated workload.

  • Keep traffic private: reach Bedrock through VPC interface endpoints instead of the public internet.
  • Control access with IAM: decide which roles may call which models, and scope retrieval so users only get documents they are already allowed to read.
  • Encrypt data at rest with AWS KMS keys, including the vector store and stored prompts.
  • Log model invocations and API calls with CloudWatch and CloudTrail so usage can be audited.
  • Apply guardrails to filter harmful content, block named topics and redact personal data in prompts and responses.

What drives the cost

  • Model choice: larger models cost more per token. Many tasks run well on a smaller one.
  • Prompt size: retrieved passages are billed as input tokens, so retrieve fewer, better passages.
  • Volume and pattern: steady high volume can justify provisioned capacity; occasional use suits on-demand pricing.
  • The surrounding system: the vector store, data pipelines and logging also cost money.

From pilot to production

  1. Pick one use case

    Choose a task with a clear owner and a way to tell good answers from bad ones.

  2. Build an evaluation set

    Collect real questions with expected answers. Without this, nobody can say whether a change improved the system.

  3. Add the controls

    Access rules, guardrails, logging and cost alerts go in before wider rollout, not after.

  4. Monitor in use

    Track answer quality, latency and spend, and review logged conversations for failure patterns.

Key takeaways

  • Start with Amazon Bedrock unless you have a specific reason to host your own model.
  • RAG over your own documents is usually a better first step than fine-tuning.
  • Access control, guardrails and logging are part of the first release.
  • An evaluation set is what turns a demo into something you can improve.

Related questions

Is my data used to train the model on Amazon Bedrock?

AWS states that Bedrock does not use customer prompts or outputs to train base models and does not share them with model providers. Check the current Bedrock data protection documentation for your case.

Do we need to fine-tune a model?

Usually not at first. Retrieval-augmented generation gives a model access to your documents without retraining and is easier to keep current. Fine-tuning helps when you need a specific style or task behaviour that prompting and retrieval do not achieve.

Does Celestibia build generative AI systems on AWS?

Yes. Celestibia delivers secure LLM deployment, retrieval-augmented generation, Amazon Bedrock implementations, AI agents and governance, as described on the generative AI service page.

How Celestibia helps

Further reading

Want this applied to your environment?

A Celestibia engineer can review your setup and turn this guidance into a concrete plan. The first assessment is free.

What happens next

  1. Answer six questions about your cloud setup, spend and priorities. It takes about three minutes.
  2. A Celestibia engineer reviews your answers within one business day.
  3. You get tailored recommendations and the AWS programs you are likely to qualify for.

Monday to Friday, 10 AM to 7 PM IST