AI - When a machine or system mimics a human intelligence
ML - computer learns on its own, without being told programmatically what to do
GenAI - Generative AI - is a type of machine learning that produces new data that is similar to the training data. It is a branch of AI that focuses on the creation of new content (like for example, text, images, videos, audio), rather than the analysis of existing content.
LLM - Large Language Model - A type of ML model that is trained on a massive amount of text data, which allows the model to learn to perform a variety of tasks, such as generating text, translating languages, and writing different kinds of creative content.
PLM - Pre-trained Language model.
Prompt - a short piece of text that is given to the LLM as input, and it can be used to control the output of the LLM in a variety of ways.
Prompt Engineering - process of crafting the text prompt that is given to a language mode in order to elicit a desired response.
Prompt supervised fine-tuning - technique to improve performance of language model on a specific task, by leveraging additional training on a specific, labeled dataset.
PET - Parameter efficient tuning - technique for fine tuning LLK's using a smaller number of parameters. Example algorithms: Adapter-Tuning, Prompt-Tuning, Diff-Pruning, LoRa, and BitFit.
Hallucination - Refers to the output of LLM's not being factual, accurate, or realistic. Can happen when a the LLM is trained on data not representative of the real world, or when the LLM is asked to generate text on a topic that is not familiar with.
RLHF - Reinforcement Learning from Human Feedback. Facilitates the alignment of LLM's with human preferences, which significantly enhanced the quality of interactions between humans and LLM's.
Vertex AI LLM Variables
Temperature
Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that expect a true or correct response, while higher temperatures can lead to more diverse or unexpected results. A temperature of 0 is deterministic: the highest probability token is always selected. For most use cases, try starting with a temperature of .2.
Token limit
Token limit determines the maximum amount of text output from one prompt. A token is approximately four characters. The default value is 256.
Top-K
Top-k changes how the model selects tokens for output. A top-k of 1 means the selected token is the most probable among all tokens in the model’s vocabulary (also called greedy decoding), while a top-k of 3 means that the next token is selected from among the 3 most probable tokens (using temperature). The default top-k value is 40.
Top-P
Top-p changes how the model selects tokens for output. Tokens are selected from most probable to least until the sum of their probabilities equals the top-p value. For example, if tokens A, B, and C have a probability of .3, .2, and .1 and the top-p value is .5, then the model will select either A or B as the next token (using temperature). The default top-p value is .8.
ML & AI Resources
Vertex AI Workbench
Vertex AI Managed Notebooks:
VertexAI User-managed Notebooks
Github GenAI Examples
Ideation
Question & Answering
Text Classification
Text Extraction
Text Summarization
Langchain
Tuning
Generative AI for Developers
Training
Generative AI Learning Path - no cost training
Introduction to Generative AI
Introduction to Large Language Models
Introduction to Responsible AI
Generative AI Fundamentals
Introduction to Image Generation
Encoder-Decoder Architecture
Attention Mechanism
Transformer Models and BERT Model
Create Image Captioning Models
Introduction to Generative AI Studio
Additional AI/ML training - varying learning credits required to complete on Google Cloud Skills Boost
Introductory level
[Course with completion badge] How Google Does Machine Learning (1 day)
[Course with completion badge] MLOps: Getting Started (1 day)
[Skill badge] Get started with TensorFlow on Google Cloud (8 hours)
[Skill badge] Perform foundational ML, AI and data tasks in Google Cloud (7 hours)
[Course with completion badge] Language, Speech, Text, and Translation with Google Cloud APIs (5 hours)
Intermediate/Multi-level
[Learning Path] Machine Learning Engineer (Collection of 15 video courses and labs)
[Skill badge] Build and Deploy Machine Learning Solutions on Vertex AI (1 day)
Advanced
[Training + certification exam] Google Cloud Professional ML Engineer Certification (varied time)
[Skill badge] Machine Learning with TensorFlow in Vertex AI (90 minutes)
[Course with completion badge] Natural Language Processing in Google Cloud (1 day)
Security & Data
Considerations
Google's Secure AI Framework
Google Cloud Privacy Controls
VPC-SC
(VPC Service Controls)
AXT
(Access Transparency Logs)
DRZ
(Data Residence Controls)
CMEK
(Customer Managed Encryption Keys)
Ephemerality: In-use customer core content destroyed upon expiry without undue delay.
Data Governance and Generative AI
Terminology and data classification
Foundation Models
(Trained on data not owned by GCP customers)
Owned by Google
Prompt datasets
(Datasets provided by GCP customers)
Owned by Customer
Fully-tuned or fine-tuned models
(Trained on prompt dataset data provided by GCP customers)
Owned by Customer
Adapter models
(Trained on prompt dataset provided by GCP customers, but simpler and less costly to train & serve)
Owned by Customer
Industry resources
MITRE | ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence System
OWASP Top 10 List for Large Language Models version 0.1
AI Vulnerability Database
ML and AI
Products and Solutions
PaLM 2
Domain-specific variants
Med-PaLM 2
Sec-PaLM 2
Sizes
Gecko
(smallest)
Otter
Bison
Unicorn
(largest)
Products & Services
Enterprise
AI for Data Scientists
Vertex AI
Our new unified machine learning platform will help you build, deploy and scale more effective AI models.
Challenges
High-quality AI requires a lot of data
Customers can customize Google’s data models to get started with APIs and AutoML.
AI expertise is in high demand
BigQuery ML allows data analysts or data scientists to build and operationalize machine learning models directly within BigQuery, using simple SQL.
Getting value of ML requires a modern data platform
Google’s data cloud embraces the full data lifecycle and allows customers to unify data across the entire organization.
Activating ML requires surfacing AI into decision UIs
The data and ML infrastructure is integrated to provide real-time, personalized ML.
Operationalizing ML is hard
Vertex AI is one comprehensive end-to-end platform for everything AI – unified ML and data science.
State-of-the-art changes rapidly
Google Cloud can help customers leverage AI more effectively.
Business value proposition
Build on the best of Google
Products and solutions built on the foundation of Google’s pioneering AI research means customers can take advantage of continuous enhancements.
Vertex AI, our comprehensive ML platform, brings the best of Google’s AI technologies in computer vision, conversation, language, and structured data, including a foundation of product excellence into one scalable unified experience.
Accelerate time to value
Customers can accelerate their ability to deploy more models in production by materially increasing the rate of experimentation.
Vertex AI makes sophisticated AI more accessible with flexible tools for streamlined and scalable collaboration across all levels of technical expertise – data scientists don’t need to be ML engineers!
Enhanced MLOps capabilities make practitioners’ jobs easier with robust, self-service workflow management.
Trust and responsibility
Responsible AI
All products must follow Google's mandatory and comprehensive AI principles review process so customers can trust that the platform is built with ethical governance.
“Google has taken a clear thought leadership position in the area of AI explainability and responsibility,” according to the Gartner Magic Quadrant, 2021 Data Science & ML.
Security
Customers minimize their security risk by working with the leader among data security portfolio vendors.
Google Cloud powers organizations whose core businesses rely on data privacy and trust, such as Palo Alto Networks and The Mayo Clinic.
Sustainability
Our AI products leverage our environmentally responsible infrastructure. On average, a Google data center uses 50% less energy than a typical data center.
Technical value proposition
Unified data and AI platform
Product integrations help us deliver a complete data-to-value journey:
Vertex AI Workbench is a one-stop surface for data science encompassing data and AI workloads.
There is consistent workflow with minimal data movement within Vertex AI, BigQuery, (the most popular store of data on Google Cloud) and Spark – all AI tasks that work with data directly in BigQuery.
MLOps tooling for BigQuery ML brings MLOps to the data and tools with which users are most comfortable.
End-to-end MLOps
Vertex AI’s end-to-end MLOps enables data scientists and ML engineers to efficiently and responsibly manage, monitor, govern, and explain ML projects throughout the entire development lifecycle. Vertex AI gives customers total flexibility in the tools they use to complete their ML journeys. Managed Pipelines let data scientists spend their time building ML solutions rather than building the infrastructure needed to get those solutions into production, and quickly take their experiments and convert them into production ready workflows by:
Providing familiar Python SDKs
Leveraging GCP’s suite of managed services to make pipelines both scalable and cost effective
Providing the key capabilities needed to integrate pipelines into more complex business systems
This allows data scientists to iterate faster, get more of their work into production, and work more independently.
With Vertex AI Experiments, customers can track and compare multiple experiment runs and analyze key model metrics. Vertex AI provides a host of products to monitor and govern models, all of which help drive both successful and responsible AI deployment. Customers are also able to reveal the ‘why’ behind their model and predictions with Explainable AI.
Open and scalable AI infrastructure
With Vertex AI, customers can accelerate the velocity of models to production while maintaining the flexibility of ML frameworks and compute options. We also provide flexibility while delivering scalable infrastructure for AI at whatever level is right for our customers. We continue to invest across the stack to ensure customers don’t have to change the way they work to realize the full benefits of our AI infrastructure with Vertex AI.
With Vertex AI training, customers can run and scale their code with high availability. Once trained, customers use Vertex AI prediction to serve their models, with both online and batch prediction available. Vertex AI Training Reduction Tool, which helps customers optimize distributed GPU training for synchronous data parallel algorithms, introduces an additional worker role, a reducer, which requires little computing power and can run on inexpensive compute nodes. By doing this, customers achieve both faster training time at a lower cost. Reduction Tool supports TF, Pytorch, and JAX.
State-of-the-art AI
Choosing a data and AI provider is a long term investment and key to an enterprise’s strategy and digital transformation over the next decade. To provide assurance that Google and Vertex AI is a sound investment, we offer customers:
Neural Architecture Search that lets them discover model architectures while optimizing for performance and efficiency
Vizier, a black-box optimization service to tune hyperparameters in complex ML models
Matching Engine that offers vector similarity search and matching
TabNet, a high-performance, explainable tabular learning tool
Vertex AI Tabular Workflows that are easy to use, pre-built pipelines and components for a variety of structured data tasks for AutoML, exclusive model training, feature search and more
Vertex AI Forecast that offers the highest accuracy forecasts with state-of-the-art ML models
Vertex AI Workbench
The single development environment for the entire data science workflow.
AI for Developers
AutoML
Train high-quality custom machine learning models with minimal effort and machine learning expertise.
Cloud Natural Language
Derive insights from unstructured text using Google machine learning.
Dialogflow
Create conversational experiences across devices and platforms.
Media Translation (Beta)
Add real-time audio translation to your content and applications.
Speech-to-Text
Accurately convert speech into text using an API powered by Google's AI technologies.
Text-to-Speech
Convert text into natural-sounding speech using an API powered by Google's AI technologies.
Timeseries Insights API (Preview)
Large-scale time series forecasting and anomaly detection in real time.
Translation AI
Make your content and apps multilingual with fast, dynamic machine translation.
Video AI
Enable powerful content discovery and engaging video experiences.
Vision AI
Derive insights from your images in the cloud or at the edge with AutoML Vision or use pre-trained Vision API models to detect objects, understand text, and more.
Background on AI and
connected cameras
Today, there are about a billion cameras worldwide in cities, factories, retail stores, cars, satellites, drones and other places that are changing the way we look at and operate in the world. Video streams have evolved over time to be pliable with AI, which not only records images but also learns, makes assessments and in a basic manner ‘thinks’ about the images and videos it records.
For example, AI can be used on video streams to perform advanced functions such as vehicle detection, people counting, missing object detection, traffic counting, personal protective equipment detection and more.
Cameras can record video footage and feed the captured information to an analytic (AI) layer. The AI, in real-time, processes millions of options of video footage to help organizations make fast, informed decisions.
Sensors for AI cameras are expected to grow, and so is the data they create. That means even more new developments are coming in AI with the migration to higher sensor resolutions, and communications from devices happening at 5G speeds.
Components
Vertex AI Vision Studio
Ingest service
Analyze service
Vertex AO Vision Warehouse
Store service
Search service
Services
Vertex AI Vision Streams: a geo-distributed managed endpoint service for ingesting video streams & images. Easily connect cameras or devices from anywhere in the world and let Google handle ingestion and scaling
Vertex AI Vision Applications: a serverless orchestration platform for video models & services enabling developers to stitch together large, auto-scaled media processing and analytics pipelines
Vertex AI Vision Models: a new portfolio of specialized pre-built vision models for common analytics tasks including occupancy counting, PPE detection, face-blurring, retail product recognition and more. Additionally, users can build and deploy their own custom models
Vertex AI Vision Warehouse: a serverless rich-media storage that provides the best of Google search combined with managed video storage. Perfect for ingesting, storing, and searching PBs of video data.
Capabilities
Ingest media from live cameras or existing media data
Process media using either state-of-the-art pretrained AI models, or custom-built AI models
Store data in Vision Warehouse, which comes with powerful search capabilities
Analyze data and serve meaningful, actionable business insights to its users.
Business values
Optimized for real-time video analytics
With Vertex AI Vision, customers can ingest media from live cameras or existing media data.
Move quickly for better decision making
Vertex AI Vision reduces the time to build vision AI applications from days to minutes. Customers can then analyze data and serve meaningful, actionable business insights to their users.
Technical values
Privacy
By incorporating features to protect privacy and security, like person blur, and choosing not to offer any kind of personal identification features such as facial recognition, Vertex AI Vision offers a high level of privacy when it comes to end customers where these apps may be deployed.
End-to-end development environment
Customers can process media using either state-of-the-art pretrained AI models, or custom-built AI models. For example, the occupancy analytics models include line crossing counting, active zone counting, and dwell time features. These models, combined with the person blur model, deliver important customer analytics relevant across industries while helping retain privacy.
Integrated with Google Cloud services
Vertex AI Vision is fully integrated with BigQuery and Vertex AI, and comes with a simple and easy to use user interface.
AI Infrastructure
Deep Learning Containers
Preconfigured and optimized containers for deep learning environments.
Deep Learning VM Image
Preconfigured VMs for deep learning applications.
GPUs
High-performance GPUs on Google Cloud for machine learning, scientific computing, and 3D visualization.
TensorFlow Enterprise
Reliability and performance for AI applications with enterprise-grade support and managed services.
TPUs
Train and run machine learning models fast.
Business solutions
Contact Center AI
(CCAI)
Transform your contact center with AI technology (Dialogflow CX, Agent Assist, Contact Center AI Platform, and CCAI Insights). Increase operational efficiency and personalized customer care from the first “Hello.”
Background
Components
Virtual Agents
Customers access 24/7 conversational self-service, with seamless handoffs to live agents for more complex issues.
Agent Assist
Agents are empowered with real-time, step-by-step support during their calls to improve accuracy and reduce handle time. AI-driven summarization automates call wrap-up.
Insights
Contact center managers see where to start and how to improve outcomes with key call drivers surfaced from all customer self-service and live agent interactions.
CCAI Platform
Until recently, customers could only consume our AI services via third party contact center solutions like Genesys, Avaya, and Cisco. We now offer our own Google end-to-end cloud native contact center as a service solution, called CCAI Platform, which adds additional customer and agent experience benefits and simplifies deployment.
Challenges addressed
Unhappy, frustrated customers
Reason: Friction created by poor self-service and long wait times, along with potential unresolved issues
High contact center
agent turnover rates
High contact center agent turnover rates
Frustrated contact center
and experience owners
Reason: Unable to balance operational efficiency with customer satisfaction due to lack of contact center insights and being hindered by existing systems that cannot support a unified customer experience
Business value proposition
Improves customer self-service
CCAI has a broad impact at the front door of contact center interactions by using conversational steerage to recognize the reason for customer contact, and then routes the customer to the right self-service or live agent. Traditional IVRs use a more complicated structure that consists of many rare branches that attempt to handle typical customer questions, so when follow-up questions are asked, customers attempt to navigate through the paths to arrive at their answer or they run into a situation where their follow-up question lies outside the questions the static system was designed to handle. The result is a frustrated customer. CCAI uses a stateful system that is dynamic, where rare branches can be eliminated. Customer follow-up questions can be seamlessly answered anywhere in the flow, and the customer is gracefully brought back to continue. This allows businesses to create dynamic customer journeys that are personalized and prepared for many additional customer follow-up questions. Also, CCAI Platform offers a CRM-centric design that empowers seamless channel blending where customers can combine a phone call with text and images in a single interaction.
Reduces agent frustration and turnover
CCAI superpowers customer agents by providing context at the start of a call and guidance during the call. Guidance includes surfacing live transcription, recommended responses and articles based on the real-time conversation with the customer. Many customers save 10% handle time, and are able to handle 28% more chat conversations concurrently and respond to chats 15% faster.
Delivers a high quality service for an acceptable cost in a timely manner
Contact center owners can enable transcription of calls and use topic modeling to identify areas to focus on injecting AI. For every live agent interaction, automated summarization of the call or chat can be populated to their CRM. For high volume call topics, Google has playbooks and prebuilt components to build virtual agents for customer self-service. Customers can inject AI on their current platform or migrate to Google CCAI Platform. CCAI Platform offers a turnkey core contact center solution out of the box for faster time to production, and lower custom development overhead.
Document AI
Document AI solutions suite includes pre-trained models for data extraction, Document AI Workbench to create new custom models or uptrain existing ones, and Document AI Warehouse to search and store documents.
Features
Manage the entire unstructured document lifecycle in one unified solution
Reduce manual document processing, minimize setup costs, and accelerate deployment
Ensure a high level of accuracy with Google's AI and Human-in-the-Loop (HITL) reviews
Use their own data to gain new insights about their products and meet customer expectations
Human interpretation
to connect data
Capabilities
Specialized or pretrained models
Translation support
Human in the loop
Customized Processors
Uptraining
Challenges
Document processing
technology challenges
Pre-processing – preparing documents for processing by splitting, rotating, etc.
Post-processing – error checking and quality assurance
Rules management – vigilantly analyzing and revising core rules
Template management – creating bounding boxes for every new version
Business challenges
Cost
Accelerate business insight
Accuracy
Added value
Business value proposition
Improve speed and document processing time
Improves time to insight
Reduce costs
Technical value proposition
Cloud-native managed service
Best semantic search
Extensibility
Understand text
Understand layout and fields of interest
Refine
Digital manual
labor challenges
Digital manual labor challenges include:
Validating that documents are the correct ones
Checking that documents are complete
Verifying that all documents are present
Sorting documents into different workflows
Keying in data from documents to systems
Copy-pasting data from one system to another
Cross-checking data across documents
Normalizing data in spreadsheets and databases
How it works
Understand
(Document AI Workbench)
Components
Pretrained Models
Out-of-the-box; Uptrain - Invoice, Expense, Identity, Contracts, W2, Forms and more
Workbench
Custom model - Label, test, train, deploy, HITL
OCR
Extract test from images
and documents
Capabilities
Extract text, layout and generic fields
Extract Industry and business specific fields
Split and classify documents
Review documents to improve accuracy with built-in (expert) human-in-the-loop (HITL) reviews
Organize
(Document AI Warehouse
Components
Integrated with Document AI models
Semantic Search, Faceted Search
Managed service - no infrastructure to maintain
Capabilities
Storage schema of extracted documents
Full text, semantic, faceted search with Google’s best
HR: Drivers licenses, task cards and forms, resumes
Supply Chain: Bills of lading
Government: Case filings, codes, procedures
Health Care and Life Sciences: Research and publications
Specialized Solutions
Document AI for lending
Document AI for procurement
Document AI for contracts
Discover AI for Retail
Increase conversion rate across digital properties with AI solutions that help brands to deliver personalized consumer experiences across channels.
Duet AI for Google Workspace
Anti Money Laundering AI
Consulting services
AI Readiness Program
Our AI Readiness Program is a 4-6 week engagement designed to accelerate value realization from your AI efforts. Our experts will work with you to understand your business objectives, benchmark your AI capabilities, and provide tailored recommendations for your needs.
Consumer
Google Bard
MakerSuite
PaLM API
(Generative AI for Developers)
GenAI on Google Cloud
Generative AI support on Vertex AI
Build applications with generative AI
(Generally Available)
Watch the demo
Generative AI Studio
Test models using prompt samples.
Design and save your own prompts.
Tune a foundation model.
Convert between speech and text
AI model API's and
Model Garden
PaLM API
PaLM API for text is fine-tuned for language tasks such as classification, summarization, and entity extraction.
Language Models
Text
The Text Embedding API generates vector embeddings for input text. You can use embeddings for tasks like semantic search, recommendation, classification, and outlier detection.
Chat
PaLM API for chat is fine-tuned for multi-turn chat, where the model keeps track of previous messages in the chat and uses it as context for generating new responses.
Code
The Codey APIs generate code. The Codey APIs include three models that generate code, suggest code for code completion, and let developers chat to get help with code-related questions
Image Models
Speech Models
Convert text to Speech
Convert Speech to text
Prompt design
Responsible AI
Data Governance
Generative AI App Builder
Create generative AI powered experiences in minutes
(Available to trusted testers)
Watch the demo
Duet AI for Google Workspace
Create, connect, and collaborate like never before
(Available to trusted testers)
Watch the demo
Duet AI for Google Cloud
Boost your productivity with our always-on AI collaborator
(Available to trusted testers)
Watch the demo
Google Cloud Consulting
Create with Generative AI
Discover with Generative AI
Summarize with Generative AI
Automate with Generative AI
AI partner ecosystem
Enjoy the flexibility of our open approach to tools and models
Gen AI Enterprise use cases
Complex data, intuitively accessible
(Analyst)
Public Website Navigation
Effectively find information from a website via multi-modal inputs and conversational queries
Customer Service Automation
Effectively service customers requests for information and service provisioning
Product / Content Catalog Discovery
Effectively find the most relevant Products / Content listings from an inventory catalogue
Product / Content Recommendation
Recommend personalized Product / Content / Next Best Action from a catalogue
Online interactions made conversational
(Customer Service)
Intra-Knowledge Q & A
Conversationally query questions for answers from internal knowledge sources
Document Search & Synthesis
Effectively find the most relevant documents and summarize their contents
Business Process Automation
Automating the information retrieval and recommendation step of a recurring business process
Creative Assistance
Empower creative teams to create bespoke images and creative content for campaigns and editorial content
Content generation at the click of a button
(Creative)
Regulatory Compliance Automation
Interpret regulatory policy / documents to identify potential violations relative to operating procedures
Research Acceleration
Find complex subject domain information across many disparate sources and synthesis the findings
Documentation Generation
Write new documentation based on summarization of other documents & software code
Developer Efficiency
Complete and augment code to make your engineering team more efficient and effective
Customize Foundational models
(AI Practitioner)
Customize large models and incorporate state of the art generative capabilities natively into your own internal MP operational platforms